Page 1 Report on the Usability of Publisher Wildfire Infographic Publisher Team 2: Mark Johnson, Sarah Lynn, Pierce Trey,

Morgan Weir, Crystal Zarate Introduction The purpose of this document is to detail the methods and results of a usability test on the effectiveness of two infographics regarding wildfires, which may be found in Appendix A. Both were designed to increase readers knowledge regarding wildfire suppression, but the data in one was presented in a gains frame1 while the second presented the same data in a losses frame. The goal of this usability test was to determine the efficacy of these infographics on readers. The test was designed using methodological triangulation and will be further detailed in the methods section of this document. Methods Each group member tested one gains and one losses frame of the infographic for a total of ten participants. These selections were not random, but based upon people group members knew (for instance, roommates). To protect anonymity, participants were assigned a number which was written at the top of all their test material. They were granted an unlimited amount of time to view the infographic assuming that when viewed in regular life, viewers will not have a time restriction. Additionally, the amount of time spent viewing the infographic may yield insight toward its effectiveness and appeal. This test was analyzed using methodological triangulation which uses three independent data platforms to collect data then analyze the results. Through this, the effectiveness of the infographic was determined. This group used surveys, talkaloud-protocol, and interviews as the triangulation points. The experimental methods implemented in each of these are detailed below. Surveys Surveys were conducted before and after viewing the infographic. The pre-survey consisted of two parts; the first was composed of questions regarding participants backgrounds (Appendix B) while the second inquired about their current views regarding wildfire management (Appendix C). The post-survey was completed after reading the infographic and asked the same questions as part two of the pre-survey. Because the surveys were completed on a scale of negative three to three, it was
1

Framing refers to the emotional tone of the message being conveyed. Framing refers to the words chosen, the images alongside them, and the colors used. A gains frame presents the message such that the benefits are listed while a losses frame shows the negative effects. Oftentimes it is unintentionally done by the messages creator and examples can be found in everyday life. For example, the dentist tells you to brush your teeth everyday to keep your mouth healthy versus the dentist tells you to brush your teeth everyday or your teeth will rot out. The first is a gains frame while the second is a losses.

Page 2 possible to sum participant responses on both surveys. The sum of the responses will hereafter be referred to as the score. Knowledge gain was measured by calculating the between the pre- and post-surveys. Talk-Aloud Protocol The talk-aloud portion of the test occurred simultaneously with reading the infographic. This measurement required viewers to tell the testers what they were thinking and where they were looking whilst viewing, much like thinking aloud. Assuming people do not usually do this while reading, participants were asked to complete a practice talk-aloud with a different infographic before viewing the infographic being tested. The practice infographic is attached as Appendix D. Interview Interview data was the final data point collected from this usability test. The interview was conducted after the post-survey and the questions focused on how the infographic affected the participant and what the readers felt could be improved. The interview was conducted one-on-one in a casual setting. The interview questions are attached as Appendix E.

Page 3 Results The overall goal of this usability test was to determine the effectiveness of two infographics. The effectiveness was determined by participant feedback regarding the design as well as knowledge gain. The concluding result was that the infographics designs were insufficient in attracting viewers and being credible. The results also show that while participants did acquire knowledge from the infographic, it did not motivate most to take further action.

Figure 1. Interview Responses. Credibility and Motivation to Action Figure 1 displays interview responses when participants were asked the following three questions: Did you find the information credible and presented in a manner which encouraged you to take the matter seriously? Did the infographic motivate you to take future action? Do you feel you learned anything? As seen in Figure 1, 75 percent of participants indicated that they learned information between the pre- and post-surveys. However, 60 percent of participants felt the infographic lacked credibility and 70 percent of participants were not motivated to take further action.

Page 4

Figure 2a. Knowledge Gain for Losses Infographic. Note that participants seven and ten did not change any answers between the pre- and post-survey and therefore measured no knowledge gain.

Figure 2b. Knowledge Gain for Gains Infographic.

Page 5 Knowledge Gain Figures 2a and 2b display the knowledge gain of each participant. This data was compiled by comparing the data from pre-survey and post-survey questions as detailed in the methods section of this document. As seen in Figure 2a, the losses frame, participants two and four displayed slight changes in knowledge while participant six gained a significant amount. However, participants seven and ten did not display any change in score indicating no knowledge gain. Despite this, the average result was a positive knowledge gain. Figure 2b shows that although all those who viewed the gains infographic showed a change in knowledge, the average result was negative. In both frames, the average knowledge gain was very small indicating most participants did not gain a significant amount of knowledge.

Figure 3a. Average Effectiveness Rating of Prevention Techniques (Gains Frame)

Page 6

Figure 3b. Average Effectiveness Rating of Prevention Techniques (Loss Frame) Effectiveness The final section of the surveys contained a series of five statements recommending preventive actions against wildfires. Participants were asked to rate the effectiveness of these actions on a scale of negative three to three (three being highly effective). Figures 3a and 3b display the number of each statement and the average participant rating of it. As seen in Figure 3a, the gains infographic, all of the participants felt that the actions were at least somewhat effective both before and after viewing the infographic. The effectiveness ratings were low, averaging between one and two. In the postsurvey, questions two and four had the same rating as in the pre-survey. Questions one and three decreased in effectiveness after viewing the infographic while question five increased. According to Figure 3b, in the pre-survey participants viewing the losses infographic found statements two, three, and five effective. However, question one was neutral and question four was slightly ineffective. In the post-survey questions three, four, and five displayed the same effectiveness rate as seen in the pre-survey. Question two decreased in effectiveness, while question one made a slight increase.

Page 7

Figure 4a. Average Likelihood of Performing Prevention Technique (Gains Frame)

Figure 4b. Average Likelihood of Performing Prevention Technique (Losses Frame) Likelihood of Taking Preventative Action In addition to stating prevention techniques, the final portion of the surveys asked participants how likely they would be to perform the action. The overall purpose of the usability test was to determine if people are motivated to take action against forest fires

Page 8 after viewing the proposed infographics. As displayed in Figure 4a, most participants who viewed the gains frame responded that they would be the most likely perform the actions described in statements two, three and five. However, statements one and four are not likely to be performed by viewers. There was no change in the responses between the pre- and post-survey regarding questions three and five. Questions one and two increased in likelihood while question four decreased. Figure 4b displays the same likelihood of performing the prevention technique when it was presented in the losses frame. On average, participants found themselves likely performing the prevention techniques in statements two, three, and five. As in the gains frame, viewers were unlikely to perform the indicated prevention technique described in statements one and four. After viewing the infographic, responses to statements three and four remained the same. Responses to statements two and five indicated an increased likelihood of performing the prevention technique. Statement one had the most significant change increasing in likelihood from an average rating of -1.2 to 0. Analysis In general, the results provide information about aspects of the infographic that should be changed rather than positive attributes of the infographic. They present an overall need for change in the infographic content. Using methodological triangulation of data collection, some conclusions can be drawn regarding the infographic. Many participants changed only a few answers between the pre- and post-surveys, suggesting that the infographic had little or no effect on them. However, 60 percent reported knowledge gain from the infographic, as seen in Figure 1. Average knowledge gain measured marginally higher from the losses frame than from the gains as shown in Figures 2a and 2b. Despite the reported knowledge gain, 60 percent of viewers found the information not credible, also seen in Fig. 1. This might suggest that the material is elementary or not well referenced. Neither frame appears to have a significantly greater influence on the participants than the other; they both proved equally ineffective. As noted before, the losses frame proved to convey more knowledge than the gains frame (Figures 2a, 2b). Also, the participants that viewed the infographic with the gains frame found the prevention techniques described in the final portion of the survey more effective than those that viewed the losses infographic. However, those that viewed the infographic in the losses frame were overall more likely to perform the actions. As seen in Figure 1, the infographic was not successful in motivating the participants to pursue further action. There were complaints regarding the credibility of the information and the lack of specific wildfire data which may be part of the reason participants were not motivated toward further action. As motivating viewers was the primary goal of the infographic, this indicates the infographic would benefit from adding more motivating

Page 9 elements as well as additional data. The talk aloud protocol information gathered from the usability test produced little helpful information. General information such as approval of the colors used in the infographic and expressing boredom with the infographic was collected from this data point. Discussion This infographic needs many revisions. The flow of the visual layout seems to be generally acceptable, but the content of the document needs to be brought to a higher level, as it was perceived more often than not as common sense material. The preventative techniques presented on the infographic were perceived by the test subjects as being ineffective and unimportant. The presentation of the information was responded to as being not serious enough and even described as something you would present to a childboth because of the physical design, and the simplicity of the preventative techniques. The framing was shown to be irrelevant, as there was no significant change in data between frames. Both frames failed to produce a significant amount of knowledge gain when the answers of the post-survey were compared with those of the pre-survey. The data from the losses frame showed a small improvement over the gains frame in terms of knowledge gain, but the difference was not enough to indicate a clear lead, especially considering the small size of the data set. Due to the lack of effectiveness of this infographic, it is recommended that the current infographics not be used for a large scale study. The changes required to increase the effectiveness of these documents to a useful level may be more effort than it is worth; it would be better for entirely different graphics to be used. The physical design of the infographics have good aesthetic appeal (though it was considered too cartoony by a few participants, which correlated to a decreased credibility of the data in their minds), but it failed to convey any useful information. In addition to the need to revise the infographics, some of the survey questions need to be revised. Some questions supplied for the study (appendix B) were phrased in an unclear manner, and a number of participants expressed confusion over them and found it difficult to assign a numerical value to their answer (and mentioned that their answers may have changed between the surveys because they were not quite sure how the question was posed). In order for the survey to be effective, these questions need to be revised. The talk-aloud protocol used in this study failed to provide any information relevant to the framing of the infographic, and mostly just confirmed that people tend to read top to bottom, left to right. Since the layout of the information did not change between the gains and losses frames, this data on layout provides no helpful insight into the framing

Page 10 of data used. It is important to keep in mind that the infographics tested in this study were designed by students with no experience in design or the framing of information. This lack of professional design experience could explain why there was not much data pertinent to the framing of information. In order for a large scale study on framing to be conducted, the infographics need to be properly framed. This study also sampled much too small of a population for the data to provide definitive evidence for the effectiveness of framing.