4
Communicating Forecast Uncertainty

Communication is the critical link between the generation of information about forecast uncertainty (Chapter 3) and how information is used in decision making (Chapter 2). This chapter discusses issues at the interface of generation and use. It builds on the foundation laid in Chapter 2, which describes the theoretical aspects of uncertainty in decision making, and focuses on practical aspects of communicating uncertainty in hydrometeorological forecasts.

This chapter addresses the committee’s third task: identifying sources of misunderstanding in communicating forecast uncertainty, including vulnerabilities dependent on the means of communication, with recommendations on improvements in the ways used to communicate forecast uncertainty. It explores the roles of graphics, animation, and language; consistency; dissemination technologies; and the media in uncertainty communication. In addition, it presents ideas on refinements to NWS’s product development process and education and research needs to support NWS and Enterprise-wide progress on communicating uncertainty information. The chapter is supported by an annex with examples of approaches and products with (and without) an uncertainty component.

As noted in Chapter 2, there is an extensive and rich literature on uncertainty communication in a variety of fields, including medicine, health, and hazards. Given the breadth of this literature, it is beyond the scope of this report to comprehensively review the general topic of communicating uncertainty. Nonetheless, Chapter 2 summarizes aspects from other fields that are central to this report, and for an introduction to the broader literature on uncertainty communication, the reader is referred to Morgan and Henrion (1990) and Morgan et al. (2002) in addition to references in Chapter 2. Because this literature is rapidly evolving, NWS and the rest of the Enterprise will need to entrain expertise on communicating uncertainty from outside the hydrometeorological community on a regular basis to effectively use this knowledge. Chapter 2 presents a process by which NWS could learn to utilize relevant expertise from within the Enterprise and from other disciplines to improve communication of uncertainty information. The present chapter draws on lessons from these other disciplines as needed to support recommendations for improving uncertainty communication in hydrometeorology.

4.1
BACKGROUND

Full disclosure of forecast uncertainty information is consistent with—and in fact fundamental to—NWS’s established vision for communicating information (Box 4.1). This vision emphasizes dissemination of a wide range of NWS information. As discussed in Chapters 2 and 3, this means not only NWS forecasts and products but also the fundamental supporting information (such as verification and past performance) that is central to improving uncertainty communication throughout the Enterprise.

Beyond NWS’s philosophy of information availability, though, there are practical considerations on how to effectively communicate uncertainty information that help set the context of this chapter. For instance, the National Research Council (NRC) workshop on “Communicating Uncertainties in Weather and Climate Information” (Box 4.2) observed that understanding, communicating, and explaining uncertainty should be an integral and ongoing part of what forecasters do and are essential to delivering accurate and useful information.

4.2
COMMUNICATING UNCERTAINTY IN EVERYDAY AND HAZARDOUS WEATHER FORECAST PRODUCTS

Forecast uncertainty can be communicated in such products as maps (Figure 4.1), graphs, tables, charts, flip books, images, and written or oral narrative (see Annex 4 for a range of examples). Selecting an appropriate product type and carefully crafting its content can substantially reduce the likelihood of misunderstandings. Each approach to communicating uncertainty will inherently have strengths



The National Academies | 500 Fifth St. N.W. | Washington, D.C. 20001
Copyright © National Academy of Sciences. All rights reserved.
Terms of Use and Privacy Statement



Below are the first 10 and last 10 pages of uncorrected machine-read text (when available) of this chapter, followed by the top 30 algorithmically extracted key phrases from the chapter as a whole.
Intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text on the opening pages of each chapter. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.

Do not use for reproduction, copying, pasting, or reading; exclusively for search engines.

OCR for page 66
Completing the Forecast: Characterizing and Communicating Uncertainty for Better Decisions Using Weather and Climate Forecasts 4 Communicating Forecast Uncertainty Communication is the critical link between the generation of information about forecast uncertainty (Chapter 3) and how information is used in decision making (Chapter 2). This chapter discusses issues at the interface of generation and use. It builds on the foundation laid in Chapter 2, which describes the theoretical aspects of uncertainty in decision making, and focuses on practical aspects of communicating uncertainty in hydrometeorological forecasts. This chapter addresses the committee’s third task: identifying sources of misunderstanding in communicating forecast uncertainty, including vulnerabilities dependent on the means of communication, with recommendations on improvements in the ways used to communicate forecast uncertainty. It explores the roles of graphics, animation, and language; consistency; dissemination technologies; and the media in uncertainty communication. In addition, it presents ideas on refinements to NWS’s product development process and education and research needs to support NWS and Enterprise-wide progress on communicating uncertainty information. The chapter is supported by an annex with examples of approaches and products with (and without) an uncertainty component. As noted in Chapter 2, there is an extensive and rich literature on uncertainty communication in a variety of fields, including medicine, health, and hazards. Given the breadth of this literature, it is beyond the scope of this report to comprehensively review the general topic of communicating uncertainty. Nonetheless, Chapter 2 summarizes aspects from other fields that are central to this report, and for an introduction to the broader literature on uncertainty communication, the reader is referred to Morgan and Henrion (1990) and Morgan et al. (2002) in addition to references in Chapter 2. Because this literature is rapidly evolving, NWS and the rest of the Enterprise will need to entrain expertise on communicating uncertainty from outside the hydrometeorological community on a regular basis to effectively use this knowledge. Chapter 2 presents a process by which NWS could learn to utilize relevant expertise from within the Enterprise and from other disciplines to improve communication of uncertainty information. The present chapter draws on lessons from these other disciplines as needed to support recommendations for improving uncertainty communication in hydrometeorology. 4.1 BACKGROUND Full disclosure of forecast uncertainty information is consistent with—and in fact fundamental to—NWS’s established vision for communicating information (Box 4.1). This vision emphasizes dissemination of a wide range of NWS information. As discussed in Chapters 2 and 3, this means not only NWS forecasts and products but also the fundamental supporting information (such as verification and past performance) that is central to improving uncertainty communication throughout the Enterprise. Beyond NWS’s philosophy of information availability, though, there are practical considerations on how to effectively communicate uncertainty information that help set the context of this chapter. For instance, the National Research Council (NRC) workshop on “Communicating Uncertainties in Weather and Climate Information” (Box 4.2) observed that understanding, communicating, and explaining uncertainty should be an integral and ongoing part of what forecasters do and are essential to delivering accurate and useful information. 4.2 COMMUNICATING UNCERTAINTY IN EVERYDAY AND HAZARDOUS WEATHER FORECAST PRODUCTS Forecast uncertainty can be communicated in such products as maps (Figure 4.1), graphs, tables, charts, flip books, images, and written or oral narrative (see Annex 4 for a range of examples). Selecting an appropriate product type and carefully crafting its content can substantially reduce the likelihood of misunderstandings. Each approach to communicating uncertainty will inherently have strengths

OCR for page 66
Completing the Forecast: Characterizing and Communicating Uncertainty for Better Decisions Using Weather and Climate Forecasts BOX 4.1 National Weather Service Vision for Communicating Information The NWS vision of communicating information to users is to Make a wide range of information readily available to a diverse user community; Disseminate all NWS information nationwide; Disseminate broad user community-specific information using a subset of NWS information; and Deliver critical information to the public, the hazards community, and other users. SOURCE: NWS, http://www.nws.noaa.gov/om/disemsys.shtml. and weaknesses, and each may best communicate a different type of uncertainty to a different user group. Products can be tailored to specific user needs, but when communicating with a diverse audience such as the public, one product is unlikely to meet all users’ needs or to be readily understandable to all subgroups (Chapter 2). When such a broad audience is anticipated, a mix of products will likely be most useful (Chapter 2). In addition, an NWS National Digital Guidance Database (recommendation 3.6) would help support this mix of products by providing users and intermediaries with data and tools for customizing communication of uncertainty information. NWS and other members of the Enterprise generate a variety of textual, verbal, and visual products that communicate uncertainty (Annex 4). However, most weather forecasts specifically generated for the public contain little or no useful uncertainty information; they are simplified and deterministic. Members of the public have been conditioned to these deterministic forecasts and have been given little objective information on the inherent errors in these simplified predictions. Instead, users in the public have developed their own informal methods of estimating the uncertainty. This highlights the need for user education as the Enterprise transitions to probabilistic forecasting. One major example of a predominantly deterministic product is the NWS public weather forecasts produced by the Interactive Forecast Preparation System (IFPS) and distributed as the National Digital Forecast Database (NDFD). IFPS/NDFD’s strength is that it allows forecasters to generate, present, and communicate forecasts of multiple weather elements as a digital database, both for the local area and as a nationally unified grid. Its main weakness is that the forecasts contain limited information about uncertainty. Most variables are estimated and presented as “point forecasts” BOX 4.2 Suggestions for Improving Communication of Uncertainty Information The following practical suggestions were made during an NRC workshop to improve information delivery (NRC, 2003b): View communicating uncertainty to all information users as a key part of the decision-making process. Communicate why information is uncertain, not just the fact that it is uncertain. Communicate why information about uncertainty is important. Use multiple measures of uncertainty and ways of communicating uncertainty to reach diverse audiences. Use both qualitative and quantitative forms to communicate uncertainty. Effectively communicating uncertainty and its context shifts the burden and responsibility of decision making to the information user. The following suggestions from the NRC workshop could improve communications to decision makers: The careful and strategic use of context (a tie to a past experience) in the face of complexity and uncertainty frequently makes the meaning of the uncertainty tangible. Comprehensively communicate what is known rather than only what it is thought the decision maker needs to know. Perceived success or failure of forecasts and the portrayal of forecasts by the media and decision makers guide opinions and help determine the credibility of future forecasts. The following actions were suggested: Expect misinterpretation. Make an effort to correct problems as soon as possible. Feedback from users is critical. Provide a “measuring stick” to decision makers to guide their evaluation of forecasts and forecast uncertainty. Avoid overselling or overinterpreting the science. Provide follow-on information about forecast quality to help ensure the credibility of future communications. This information is particularly important following the forecast of significant events (e.g., when a forecast was successful despite a large uncertainty or when a forecast was highly credible and failure resulted). SOURCE: NRC, 2003b.

OCR for page 66
Completing the Forecast: Characterizing and Communicating Uncertainty for Better Decisions Using Weather and Climate Forecasts FIGURE 4.1 Probability of wind speeds of greater than 39 mph (tropical storm force) during the specified 120-hour period. The colored bar at the bottom of the figure gives the probability scale in percent. SOURCE: Experimental NWS product generated by the Tropical Prediction Center (TPC). that appear as deterministic values or graphs for the next 7 days, with no change in format. As discussed in Chapter 3, these deterministic digital values for days into the future are not scientifically valid and could be highly inaccurate and misleading. In addition, the system issues forecasts of precipitation type and thunderstorm risk using vague uncertainty terms such as “slight chance,” “chance,” “likely,” and “occasional.” As discussed in Chapter 2 and developed later in this chapter, research has shown that these terms are interpreted by users as communicating a wide range of probabilities. The NDFD enables a user to select a site-specific forecast and to extract tailored forecasts from the database. The drawback, though, is that these forecasts include no “qualifier” text or statistical ranges that provide the user with uncertainty information to aid decisions. Fundamentally, IFPS and NDFD are also designed from a deterministic framework (other than the “Probability of Precipitation” component) and thus cannot be easily modified to incorporate communication of forecast uncertainty information. The provision of single-valued forecasts without uncertainty information (such as error bars on a meteograph) not only exposes a significant limitation of the NDFD/IFPS process but is also fundamentally inconsistent with the science (Chapter 1). Moreover, these digital systems may generate machine-derived text forecasts of “partly cloudy” skies for several days in a row—in essence representing a wide range of weather conditions—and therefore do not effectively communicate the complexity or uncertainty of future weather. With the importance of digital dissemination of forecasts through the Internet, incorporating uncertainty information into NDFD and IFPS would be advantageous to the public, intermediaries, and specialized users. Many methods of communicating uncertainty are available. Choosing the most effective method (or methods) will require research and two-way interactions with users (see Sections 4.4 and 4.5). Possible methods to consider include displaying skill scores or the standard forecast variance for each forecast variable at different times or providing confidence intervals. Another possibility, to improve consistency, is to communicate cloud cover not as scattered or broken but rather in categories (such as high, medium, and low) or as a percentage (as is currently done for probability of precipitation type and probability of thunderstorms in Model Output Statistics [MOS]). Finding: The public weather forecasts from the IFPS and distributed as the NDFD are one of NWS’s primary forecast products. The system is unable to provide probabilistic forecasts for most fields, and it cannot access probabilistic guidance from the National Centers for Environmental Prediction (NCEP) or other ensemble systems. With the incorporation and communication of uncertainty in most

OCR for page 66
Completing the Forecast: Characterizing and Communicating Uncertainty for Better Decisions Using Weather and Climate Forecasts forecast parameters, IFPS and NDFD can reach their full potential as forecast products that meet the NWS vision for communicating information. Development efforts are under way to provide initial probabilistic fields by “dressing” IFPS forecasts with historical error statistics, but making such capabilities operational is years away. Recommendation 4.1: The NWS should expedite development of the IFPS toward a system that can access, produce, and communicate uncertainty guidance for most forecast parameters. Such a revised system should be able to access deterministic and ensemble prediction systems, historical error statistics, and statistically post-processed forecast information (e.g., MOS) to allow production of uncertainty information with varying levels of subjective and objective contributions. The system should be capable of preparing probabilistic products to communicate probability density functions and other types of uncertainty information (e.g., probability of temperature less than freezing or wind speed greater than 26 knots). Most of the above discussion focuses on communication of uncertainty in public forecasts and outlooks. Communicating uncertainty in hazardous weather situations, particularly for short-fuse, possibly life-threatening events, presents additional challenges. Yet even in these situations, communication of simple uncertainty information may be advantageous (Box 4.3). 4.3 IMPORTANT ASPECTS OF COMMUNICATING UNCERTAINTY 4.3.1 Use of Language and Graphics Users’ interpretations of forecast information can lead to misunderstandings that affect their decisions, sometimes with catastrophic consequences (e.g., Box 4.4). In particular, words or images that a forecaster or scientist thinks are clear may be interpreted differently by users (and differently among users). For example, interpretations of the term “possible” span most of the probability spectrum (Chapter 2). When this term is used to communicate forecast uncertainty, some users will inevitably misinterpret what the forecaster intended to convey. In addition to reducing the likelihood of misinterpretation in the use of language to characterize uncertainty, NWS can also increase the clarity and accessibility of its uncertainty products. Two examples have immediate potential, the Area Forecast Discussion (AFD) and the Climate Prediction Center’s (CPC’s) monthly and seasonal forecasts. 4.3.1.1 Area Forecast Discussion The NWS AFD is one of the most commonly accessed products on NWS Web sites. Notwithstanding the challenge BOX 4.3 Communication of Forecast Uncertainty in Short-Term Warnings Communication of uncertainty information within short-term, high-risk weather events such as tornados, flash floods, or severe storms presents a dilemma for forecasters. Research in risk communication suggests that motivating action requires clear, consistent messages that warn of the approaching hazardous event and recommend specific responses, as in current NWS tornado, severe thunderstorm, and other warnings. Adding uncertainty information to these forecasts may confuse the message and possibly delay life-saving actions. Yet every forecast contains some uncertainty, and many members of the public have experience with categorical forecasts of short-fuse hazardous weather events that have not occurred as forecasted. Such experiences can lead people to interpret warnings according to their own perceptions of forecast uncertainty, which may be substantially different than the uncertainty in the actual weather situation. This, too, can delay decisions to take action. Thus, at a minimum, including some consistent information about confidence in short-term forecasts and warnings may help people evaluate the uncertainty in the situation and, in doing so, benefit their decisions. of effectively using words to convey uncertainty (Chapter 2), these discussions provide one of the few available assessments of forecast uncertainty generated by a human forecaster. They are particularly useful to meteorologists and to specialized users who understand the meteorology. However, the AFDs (and other NWS forecast discussion products) still have a major weakness: although some forecast discussions are now written in easy-to-understand terms for the general user, many are still difficult to understand (e.g., Figure 4.4), making the forecast discussion not as widely useful as it could be. Given the AFD’s wide popularity, the forecast discussions might also be adapted into an easily accessible narrative public product that communicates forecast uncertainty to nonmeteorologists. Finding: AFDs are popular NWS products that were designed as technical discussions to enhance collaboration among NWS offices and to convey uncertainty to a specialized audience. AFDs are now routinely accessed by broad user community and could be even more widely read and utilized if they were written for the even larger nonspecialist audience. Recommendation 4.2: The NWS should release the AFD only in layperson English to facilitate its broad use and understanding. For more sophisticated users, NWS could

OCR for page 66
Completing the Forecast: Characterizing and Communicating Uncertainty for Better Decisions Using Weather and Climate Forecasts BOX 4.4 Communication of Forecast Information During the Red River Flood of 1997 in Grand Forks, North Dakota Unclear communication of uncertain forecast information can hinder decision making and have significant negative consequences. An example is the 1997 flood in Grand Forks, North Dakota (Figure 4.2). Although NWS prepared flood stage outlooks months in advance, and forecasters were aware that they were predicting a record-breaking, uncertain event, the outlooks were issued as just two deterministic numbers (expected flood stage and low stage). Members of the community interpreted this range of numbers in different ways, generally not realizing that a significantly higher flood was possible (Pielke, 1999; NRC, 2003b). As it turned out, the NWS flood crest forecasts were too low by several feet, until a few days before the flood crest (Figure 4.3, left panel). Although Grand Forks had made significant preparations based on the early outlook, the city was not adequately prepared for the higher flood, and the city experienced major flood damage. Many people blamed NWS for a blown forecast. According to Ken Vein, Grand Forks city engineer (May 4, 1997), “With proper advance notice we could have protected the city to almost any elevation … if we had known [the final flood crest in advance], I’m sure that we could have protected a majority of the city.” A river stage forecast that communicated uncertainty more fully and clearly, such as the probabilistic product in Figure 4.3, right panel (which is a more recent NWS hydrologic product), may have led to better flood management decisions. In fact, Pielke found that the actual flood crest in this case was within the error range one would expect for such a forecast which could have been—but was not—communicated along with the forecast. FIGURE 4.2 Headline from The Forum newspaper, April 24, 1997. SOURCE: Forum Communications Company. FIGURE 4.3 Left: Deterministic forecasts issued by NWS prior to the Red River flood of 1997. Right: Probabilistic river stage forecast from AHPS. SOURCE: NWS.

OCR for page 66
Completing the Forecast: Characterizing and Communicating Uncertainty for Better Decisions Using Weather and Climate Forecasts FIGURE 4.4 Two examples of area forecast discussions, both containing technical terms and abbreviations that limit their communication of information to users without significant meteorological training or experience. SOURCE: NWS. provide more detailed technical information linked to the AFD. 4.3.1.2 Climate Prediction Center Monthly and Seasonal Outlooks Near the middle of every month, the CPC provides predictions of temperature and precipitation probability anomalies for the coming month, as well as seasonal (3-month) forecasts out to 12.5 months. Monthly outlooks are also updated at the end of the month. These predictions (or “outlooks”) are formulated as probability anomalies for three equally probable classes (below normal, near normal, and above normal). The anomalies now specify the probability assigned to the most likely class. The user must further examine the CPC Web site to determine the rules for distributing probability among the other two classes. The three classes are determined by dividing the normal distribution (for temperature), fitted to observations made over 1971-2000, into three equally likely classes (terciles). Because the underlying distribution for precipitation can be highly skewed, the observations are transformed into a normal distribution prior to dividing into terciles. The primary mode for providing these forecasts is maps depicting the probability value associated with the most likely category at each location. Areas with anticipated above or below normal values are labeled and color coded according to the strength of the probability anomaly; where none of the forecast tools has demonstrated statistically significant skill, the forecasters label the non-colored area as “EC” (Equal Chances; see Figure 4.5). The EC areas can be ambiguous because they may also indicate the forecasters’ belief that each of the categories truly is equally likely. Other aspects of the maps can also be difficult for users to understand (e.g., what exactly are the meanings of the terms “above average,” “below average,” “equal chances,” and “normal”?). Morever, the maps do not convey all of the available or needed information. In particular, they prespecify the thresholds for each class (e.g., above average, below average, and normal), which limits users’ ability to obtain the information that may be most useful. The EC areas are especially problematic as they provide no information about the likely distribution of values. To help users understand the rationale for specific forecasts, the CPC provides a discussion of the anomalies, which describes the sources of information and uncertainty used to develop the predictions. Technical discussions are also provided on topics such as the current state and evolution of the El Niño/Southern Oscillation. In addition, terminology definitions are provided. However, these discussions are likely not read or understood by many users. The probability anomaly maps provide an indication of where conditions are likely to be in one of the four classes (e.g., above normal, below normal, normal, and equal

OCR for page 66
Completing the Forecast: Characterizing and Communicating Uncertainty for Better Decisions Using Weather and Climate Forecasts FIGURE 4.5 Sample seasonal outlook maps for temperature and precipitation. SOURCE: CPC, http://www.cpc.ncep.noaa.gov/products/predictions/90day/. chance), but they do not directly indicate the expected median precipitation and average temperature values or the expected distribution of values. Thus, the CPC also produces maps of the “most likely anomaly” for 3-month temperature and precipitation forecasts1 (see Figure 4.6). These maps present contours of both the average anomalies as well as the climatologically average values. To supplement these maps, exceedance distribution graphics are available for each climate division (Figure 4.7). These plots present cumulative distribution functions for average conditions, as well as shifted distributions of precipitation and temperature (if a shift from normal is predicted) based on the outlook. The anomaly values for precipitation are based on the difference between the medians of the “normal” and “final forecast” distributions. Thus, anomalies indicated on the precipitation outlooks are not really “most likely.” Rather there is a 50 percent chance that the anomaly will be greater or less than the indicated value. The forecast distribution envelope is based on the expected sampling variability of the climatological probability of exceedence using 45 years of data.2 Thus, the anomaly distribution plots present the most complete information about the uncertainty in the forecasts by providing a complete distribution of possible values. The anomaly distribution plots provide sufficient information that a user can specify the precipitation or temperature threshold that is relevant for their use and obtain the associated probability. Some of this information is also available in tabular form.3 Finding: The graphics conveying monthly and seasonal outlooks are difficult for many users to understand and do not convey all the information (both graphical and tabular) that is available or needed. Exceedance probability distributions provide the most complete information about the climate probabilities at particular locations. These distributions do not rely on pre-specified categories or definitions of “normals.” Overall, more research is needed regarding user needs for these graphical and tabular formats, as well as more forecaster-user interactions to provide two-way feedback on this and other products. Recommendation 4.3: The CPC should provide full exceedence probability distributions of the projected monthly and seasonal temperature and precipitation values in both graphical and tabular forms. A straightforward graphical presentation of this information should be developed that is understandable to relevant user groups. 4.3.1.3 Icons and Text Modifiers Weather icons and text modifiers are becoming widespread within the Enterprise as a method of communicating forecast information. Understanding how users interpret these graphics is therefore important in the context of communicating uncertainty. According to one study (Box 4.5), users’ interpretations of icons may even introduce perceived uncertainty when none is intended. In another example, from a local NWS office (Figure 4.8), it is unclear how users will interpret the message conveyed 1 http://www.cpc.ncep.noaa.gov/pacdir/NFORdir/HOME3.shtml. 2 http://www.cpc.ncep.noaa.gov/pacdir/NFORdir/INTR.html. 3 For example, http://www.cpc.ncep.noaa.gov/pacdir/NFORdir/HUGEdir2/cpcllftd.dat.

OCR for page 66
Completing the Forecast: Characterizing and Communicating Uncertainty for Better Decisions Using Weather and Climate Forecasts FIGURE 4.6 Sample plot of most likely temperature and precipitation anomalies. SOURCE: CPC (http://www.cpc.ncep.noaa.gov/products/predictions/90day/). in the product. For example, are the icons in the Tuesday and Thursday forecasts confusing given the accompanying text? More generally, there seems to be little knowledge of how weather forecast icons will be interpreted by users, and insufficient incorporation of users into the icon development process. Fortunately, there is knowledge outside NWS and from other fields on how people interpret language and graphics (Chapter 2), and there are many ideas on how to use language and graphics to communicate uncertainty (e.g., Figure 4.9). Incorporating this knowledge into NWS and Enterprise efforts to communicate forecast uncertainty will create efficiencies and enable faster adoption of new methods. 4.3.2 Consistency Conflicting messages can increase uncertainty or confusion, hampering decision making. This section discusses the lack of consistency in use of uncertainty words and images. As noted in the preceding section, icons have the capability to communicate complex information in an easily

OCR for page 66
Completing the Forecast: Characterizing and Communicating Uncertainty for Better Decisions Using Weather and Climate Forecasts FIGURE 4.7 Example anomaly distribution for a 3-month temperature prediction for southwest Arizona. SOURCE: CPC. BOX 4.5 Internet Survey of Icon Interpretations In 2001, NBC4 in Washington, DC conducted an Internet-based surveya showing a cloud with one snowflake and a cloud with four snowflakes and asked respondents: What does picture B mean to you, in relation to picture A?b Overall, about half of the respondents thought the symbol with four flakes meant more snow, and slightly less than half thought it meant the forecast was more certain it would snow. In other words, many respondents interpreted a deterministic icon as if it were conveying uncertainty. This suggests that forecast providers cannot necessarily predict how users will interpret graphics without careful and thorough study of user interpretations and needs.    aResults of online surveys, while interesting, are not necessarily representative of the entire user population (see section 2.4.1).    bSOURCE: NBC Universal. Any reuse of this material requires the express written consent of NBC Universal.

OCR for page 66
Completing the Forecast: Characterizing and Communicating Uncertainty for Better Decisions Using Weather and Climate Forecasts FIGURE 4.8 Example of potentially confusing icons and accompanying text. SOURCE: NWS. FIGURE 4.9 Symbols for communicating uncertainty. SOURCE: Reprinted with permission from Human Factors, Vol. 47, No. 4, 2005. Copyright 2005 by the Human Factors and Ergonomics Society. All rights reserved.

OCR for page 66
Completing the Forecast: Characterizing and Communicating Uncertainty for Better Decisions Using Weather and Climate Forecasts accessible way. However, the icons that accompany the digitally generated forecasts on NWS homepages are sometimes inconsistent with the accompanying numerical/text forecast. In some instances, these “icon forecasts” can be more confusing than helpful; for example, the same icons are sometimes used for a variety of forecasts (Figure 4.10) and, indeed, significantly different forecasts. Uncertainty words are used inconsistently within NWS and, more generally, across the Enterprise. This inconsistency makes it challenging for users to calibrate the meaning of uncertainty forecasting terms based on experience. In addition, such inconsistency is sometimes evident in different products from the same NWS office during the same period. Box 4.6 contains an extended example from one local NWS office. Local innovation and individual forecaster creativity within NWS is important to help the agency serve local and national needs. But by relying on evolving, ad hoc, and experimental systems without more extensive, consistent, and scientifically valid communication techniques, NWS’s communication of uncertainty information may be interpreted differently by users looking at different products from the same NWS forecaster. A variety of products is needed to effectively communicate uncertainty to the broad range of users that NWS serves. But these need to be consistent across all regions, platforms, and product language and communication methods. Related to this, the NRC Fair Weather report (NRC, 2003a) recommends that “NWS headquarters and regional managers should develop an approach to managing the local forecast offices that balances a respect for local innovation and creativity with greater control over the activities that affect the public-private partnership, especially those that concern the development and dissemination of new products or services.” Finding: A variety of products is needed to communicate uncertainty to a broad range of users. Consistency of language, icons, and graphical representations of uncertainty among all these products is critical for the effective communication of uncertainty information. A necessary first step toward ensuring consistency is understanding users’ interpretations. Recommendation 4.4: To ensure consistency in the communication of uncertainty information and user comprehension, NWS should more fully study and standardize uncertainty terms, icons, and other communications methods through all pathways of forecast dissemination. 4.3.3 Dissemination Technologies The main channels through which NWS distributes information directly to the user are the NWS home pages (such as those of the Storm Prediction Center, Tropical Prediction Center, Hydrometeorological Prediction Center, and the various Weather Forecast Offices), National Oceanic and Atmospheric Administration (NOAA) Weather Radio (NWR), and the Emergency Managers Weather Information Network (EMWIN). In addition, NWS distributes information to intermediaries through the NOAA Weather Wire Service (NWWS), Interactive Weather Information Network, NOAAPORT, and Family of Services. NWR and NWS home pages have formats in which NWS can and already does incorporate uncertainty information into forecasts. Systems such as the NWWS and/or EMWIN, which emergency managers and some private-sector entities (e.g., utilities, transportation industry) use to receive NWS forecasts and special weather statements, can also be used to communicate uncertainty information through text descriptions. In addition, both NWWS and EMWIN allow display of graphical products that could communicate uncertainty. Dissemination technologies are evolving rapidly, affecting how NWS and the other members of the Enterprise approach communicating uncertainty information. Communication devices such as cell phones, personal digital assistants (PDAs), portable MP3 players, computer toolbar “bugs,” and pagers have become commonplace, and the Enterprise is moving closer to being able to communicate information to nearly anyone, anytime, anywhere. These developments present both opportunities and challenges. The opportunity is that the Enterprise can now reach more people in more places and at more times than ever before. One challenge is to understand the strengths FIGURE 4.10 Examples of graphics used to communicate different Probability of Precipitation (PoP) forecasts in NWS public forecasts. SOURCE: NWS Web pages.

OCR for page 66
Completing the Forecast: Characterizing and Communicating Uncertainty for Better Decisions Using Weather and Climate Forecasts representative to WMO), could help regions with weather risks worldwide. 4.6 EDUCATION AND TRAINING NEEDS The research and development aspects of communicating uncertainty will include and lead to education and training of all parties participating in generating, communicating, and using hydrometeorological forecasts. Here “education” is used in a broad sense and involves communication, understanding, and learning. Implementation of this report’s recommendations will change how the Enterprise operates and will lead to adoption of new forecast techniques, products, and communication tools by all sectors. But this will only happen if all Enterprise partners are actively involved in “turning the ship” and are working in a mutually supportive framework. Education initiatives will need a strong commitment by all sectors of the Enterprise and include a wide variety of participants—from elementary school teachers and students to emergency managers, media managers, and communicators. Such initiatives include undergraduate education of future hydrometeorological professionals and continuing education and training of all who communicate hydrometeorological information and forecasts (especially those working in the media). And these initiatives will rely on a two-way interaction that involves effectively communicating new information while also soliciting feedback to further improve communication and understanding. Most forecasters begin their education in an undergraduate program such as a meteorology program. Current standards for undergraduate meteorology programs established by the federal government and the American Meteorological Society (Smith and Snow, 1997) have no requirement to cover uncertainty, use of probabilistic information, and how forecast-related information is used in decisions. Without this material in their curriculum, many meteorology students are not adequately prepared for future careers in generating, communicating, or using hydrometeorological foreasts that include uncertainty information. Training courses are a critical vehicle for forecasters’ continuing education, once they have graduated. These courses convey the latest insights and techniques that enhance forecast generation and communication. Such courses could deliver relevant information and training on communicating uncertainty information. With respect to the training component, academia and government laboratories could partner on developing coursework that addresses forecast uncertainty. A hydrometeorological forecast is often only one piece of a broader spectrum of information being integrated into a decision (Chapter 2). As the human role in conveying probabilistic forecast information becomes increasingly focused at the interface between forecast systems and user decisions (e.g., functioning as the “science integrator” who takes what is known about the science and communicates it to decision makers), academic and other training programs will need to adjust their content accordingly. This adjustment may entail adding material into existing courses or by offering elective courses that focus on communication, probability, and decision issues facing weather- or climate-sensitive decision makers. Because members of the public receive most of their forecasts from the media, the media will play a critical role in helping the public understand and use new uncertainty products. Take, for example, PoP forecasts (Box 4.7). The value derived from PoP forecasts is in no small part due to the long-term efforts of the Enterprise, especially media meteorologists and weathercasters, in educating users about PoP. Even if many members of the public do not know the exact meteorological definition of PoP, many still consider this uncertainty information useful (e.g., Figure 4.17). The broad familiarity with the hurricane track probability forecast (Section 4.3.4) is another case in which the media played a critical role in facilitating acceptance of an uncertainty product and educating the public about its meaning. Including probabilities within the cone presents an opportunity for improving public understanding of uncertainty forecasts. SUMMARY Even the “best” uncertainty information will not serve its ultimate purpose—helping users make better decisions that enhance socioeconomic value—unless that information is effectively communicated. Although a variety of forecast uncertainty products are available from NWS and others in the Enterprise, some of these products do not communicate uncertainty as effectively as they could. Moreover, most publicly available forecast products often communicate little or no uncertainty information. Changing from the current paradigm of primarily deterministic forecast communication will be a major shift, requiring a concerted, coordinated effort by NWS in partnership with others in the Enterprise. The use of uncertainty information in decision making is complex. Learning to communicate uncertainty effectively will therefore require consideration of three factors. First, effective communication must incorporate an understanding of user needs for uncertainty information and how users will apply it. Such understanding must be based on social science research and close interactions with users starting early in the product development process. Second, effective communication requires considering and preventing potential user misunderstanding and confusion that can result from inconsistent communication and ineffective use of uncertainty language and graphics. Third, effective communication of uncertainty requires understanding the key roles that dissemination mechanisms and technologies and the media play in conveying forecasts. This chapter provides several recommendations to help NWS and the Enterprise shift to a new paradigm of clear, effective communication of forecast uncertainty that is consistent with scientific understanding

OCR for page 66
Completing the Forecast: Characterizing and Communicating Uncertainty for Better Decisions Using Weather and Climate Forecasts FIGURE 4.17 TV/Internet survey results of the user utility of PoP in graphic forecasts. SOURCE: Courtesy of NBC Universal. Any reuse of this material requires the express written consent of NBC Universal. of the atmosphere and hydrosphere and knowledge of how uncertain information is used in decision making. These include developing comprehensive education and training efforts and a dedicated, long-term research and development program to improve uncertainty communication in hydrometeorological forecasts. ANNEX 4 EXAMPLES OF UNCERTAINTY COMMUNICATION APPROACHES AND PRODUCTS This Annex presents examples of the range of uncertainty communication approaches and products. The committee sought examples from a variety of sources, including NWS management and individuals from NWS, other government agencies, the private sector, and academia. These examples included operational, experimental, and proposed products, primarily from weather, climate, and hydrological forecasting, but also from other fields. Maps are useful for communicating spatial distributions of forecasted variables and their uncertainty. They can represent forecasts at a specific time, over a specific period, or as a coherent weather feature (such as a hurricane or winter storm) evolves and moves. One way of using maps to communicate forecast uncertainty is to portray the spatial distribution of the likelihood of an event (e.g., tornado) or of a parameter exceeding a specified threshold (e.g., precipitation greater than one inch). Figure 4A.1 shows two examples, one containing numerical probabilities and the other containing qualitative likelihoods. A second way of using maps to communicate forecast uncertainty is to portray the spatial distribution of minimum/ mean/maximum expected (or 10/50/90 percent exceedance) values of a parameter (Figure 4A.2). Maps can also be used to portray the likelihoods of different scenarios in different regions (Figure 4A.3). Another method of communicating uncertainty using maps is to overlay a map of mean or expected values with a map of uncertainty or confidence (Figure 4A.4). The example maps above primarily use contours to represent values. When values are depicted using numbers or other symbols, uncertainty can be portrayed using different symbol sizes or colors (Figure 4A.5).

OCR for page 66
Completing the Forecast: Characterizing and Communicating Uncertainty for Better Decisions Using Weather and Climate Forecasts FIGURE 4A.1 (a) Forecast of tornado probabilities at different locations (SOURCE: Operational NWS product generated by SPC). 4A.1 (b) Forecast of likelihood of significant river flooding at different locations. SOURCE: Operational NWS product generated by HPC.

OCR for page 66
Completing the Forecast: Characterizing and Communicating Uncertainty for Better Decisions Using Weather and Climate Forecasts FIGURE 4A.2 Upper bound (90 percent exceedance) and lower bound (10 percent exceedance) 48-hour forecasts of temperature at 2 m in the northwestern United States. SOURCE: MURI research group at University of Washington, http://www.stat.washington.edu/MURI/.

OCR for page 66
Completing the Forecast: Characterizing and Communicating Uncertainty for Better Decisions Using Weather and Climate Forecasts FIGURE 4A.3 8- to 14-day temperature outlook. SOURCE: Operational product generated by NOAA CPC. Maps can also be used to communicate how uncertainty associated with a moving feature, such as a hurricane or winter storm, evolves with time. One such type of map depicts uncertainty in a feature’s location along its track; an example for hurricane track forecasts was shown in Figure 1.5, whereas Figure 4A.6 shows an example for midlatitude low-pressure systems. Uncertainty in the weather associated with a feature (e.g., wind) at different times along its track can also be depicted (Figure 4.1). Graphs communicating uncertainty can take many forms. One type of forecast uncertainty graph depicts the temporal evolution of a quantity of interest, with uncertainty represented using an ensemble of multiple temporal trajectories, box and whisker plots at each time, or probabilities of exceedance of one or more thresholds at each time. The example shown in Figure 4A.7 uses box and whisker plots to communicate how uncertainty in two forecast parameters increases and evolves with time. A related type of graph is the temporal evolution of the probability of a certain event (such as precipitation) or multiple events (rain, snow, and ice; Figure 4A.8). Another type of graph, commonly used by scientists but probably less easily understood by many members of the public, is a probability density function (PDF) of a variable at a specific location and time (Figure 4A.9). As noted in Section 4.3.3, with Internet technology and in sophisticated decision-support systems, these types of maps and graphs can be combined. For example, a general map or graph can be presented first, allowing users to click on a location or time of interest and obtain a more specific graph or PDF. Most maps and graphs used to communicate uncertainty in hydrometeorological forecasts are two-dimensional. However, three-dimensional representations can also be used (see, e.g., the NRC’s Board on Mathematical Sciences and their applications workshop “Toward Improved Visualization of Uncertain Information”), as well as movies. Tables and charts can communicate uncertainty using numbers, words, icons (symbols), or a combination. Two examples are shown in Figures 4A.10 and 4A.11 (see also Figure 1.4). Narratives can be used to communicate uncertainty orally or through written text. Three examples of narrative forecasts are the NWS forecast discussions written by WFO, HPC, TPC, CPC, and other NWS/NOAA forecasters;

OCR for page 66
Completing the Forecast: Characterizing and Communicating Uncertainty for Better Decisions Using Weather and Climate Forecasts FIGURE 4A.4 500-mb day 3 forecasts generated by HPC (green lines) and NCEP ensemble mean (black lines), overlaid with NCEP ensemble spread (filled contours). SOURCE: Experimental NWS product generated by HPC. FIGURE 4A.5 Example of how symbol size can be used to communicate level of uncertainty. SOURCE: Presentation to the committee by Ed O’Lenic, NWS.

OCR for page 66
Completing the Forecast: Characterizing and Communicating Uncertainty for Better Decisions Using Weather and Climate Forecasts FIGURE 4A.6 Low-pressure system forecast tracks: preferred tracks and track uncertainty. SOURCE: Experimental NWS product generated by HPC. NWR; and TV forecasters presenting a forecast. Figure 4.4 shows an example of the NWS AFD. Often, but not always, narratives accompany one or more maps, graphs, or tables/ charts. Narratives are versatile; they can be used to describe uncertainty in ways ranging from indications of forecaster confidence to scenarios of different ways that weather events might evolve. As noted in Chapter 2, however, uncertainty words are often ambiguous, meaning that using words to convey uncertainty can result in ineffective communication or even miscommunication. As illustrated above, NWS and other members of the Enterprise issue a number of forecasts that include uncertainty information. Nevertheless, as discussed in Chapter 1, most forecasts received by the public and many users still contain little or no information about uncertainty. A prime example is NWS’s public weather forecasts produced by the IFPS and distributed as NDFD (Chapter 3). The only element within the IPFS and NDFD operational system that provides uncertainty information is the PoP. Variables such as temperature, dew point, and sky cover are generated as single (deterministic) values out to 7 days, with no change in format as lead time (and thus uncertainty) increases (Figure 4A.12). The basic suite of NWS public forecasts are now automatically generated from NDFD by IFPS, with limited time for forecaster editing. Thus, these forecasts, too, contain no information about uncertainty other than PoP. Another example of an NWS product that does not convey uncertainty is the quantitative precipitation forecast (QPF) forecast issued by the HPC (Figure 4A.13). This product is accompanied by a forecast discussion that often discusses forecast uncertainty. This uncertainty information is not integrated into the QPF product, however, and thus is likely not seen by many users. Many other products issued publicly by NWS and others in the Enterprise are similar, with limited communication of uncertainty information in ways accessible to those outside the hydrometeorological community.

OCR for page 66
Completing the Forecast: Characterizing and Communicating Uncertainty for Better Decisions Using Weather and Climate Forecasts FIGURE 4A.7 Wind and temperature forecast for days 1-10, including forecast uncertainty. For box-and-whisker plots, the top and bottom of the box represent the 75th and 25th percentile, respectively, while the top and bottom of the lines represent the maximum and minimum. SOURCE: UK Meteorological Office. FIGURE 4A.8 Conditional probability of precipitation forecast type product, from Maintenance Decision Support System. SOURCE: Federal Highway Administration/NCAR.

OCR for page 66
Completing the Forecast: Characterizing and Communicating Uncertainty for Better Decisions Using Weather and Climate Forecasts FIGURE 4A.9 PDF for the temperature forecast for a specific time and location, summarized into a table of categorical exceedances on right-hand side. SOURCE: UK Meteorological Office. FIGURE 4A.10 Experimental probability of snowfall amount product. SOURCE: Mount Holly, NJ (Philadelphia area) NWS forecast office.

OCR for page 66
Completing the Forecast: Characterizing and Communicating Uncertainty for Better Decisions Using Weather and Climate Forecasts FIGURE 4A.11 Flood stage forecast for different locations along the Red River. SOURCE: AHPS, NWS Grand Forks office. FIGURE 4A.12 Public weather forecast generated by IFPS from NDFD. SOURCE: Operational NWS product generated by forecast offices.

OCR for page 66
Completing the Forecast: Characterizing and Communicating Uncertainty for Better Decisions Using Weather and Climate Forecasts FIGURE 4A.13 Day 2 quantitative precipitation forecast. SOURCE: Operational NWS product, generated by HPC.