Communicating Forecast Uncertainty
Communication is the critical link between the generation of information about forecast uncertainty (Chapter 3) and how information is used in decision making (Chapter 2). This chapter discusses issues at the interface of generation and use. It builds on the foundation laid in Chapter 2, which describes the theoretical aspects of uncertainty in decision making, and focuses on practical aspects of communicating uncertainty in hydrometeorological forecasts.
This chapter addresses the committee’s third task: identifying sources of misunderstanding in communicating forecast uncertainty, including vulnerabilities dependent on the means of communication, with recommendations on improvements in the ways used to communicate forecast uncertainty. It explores the roles of graphics, animation, and language; consistency; dissemination technologies; and the media in uncertainty communication. In addition, it presents ideas on refinements to NWS’s product development process and education and research needs to support NWS and Enterprise-wide progress on communicating uncertainty information. The chapter is supported by an annex with examples of approaches and products with (and without) an uncertainty component.
As noted in Chapter 2, there is an extensive and rich literature on uncertainty communication in a variety of fields, including medicine, health, and hazards. Given the breadth of this literature, it is beyond the scope of this report to comprehensively review the general topic of communicating uncertainty. Nonetheless, Chapter 2 summarizes aspects from other fields that are central to this report, and for an introduction to the broader literature on uncertainty communication, the reader is referred to Morgan and Henrion (1990) and Morgan et al. (2002) in addition to references in Chapter 2. Because this literature is rapidly evolving, NWS and the rest of the Enterprise will need to entrain expertise on communicating uncertainty from outside the hydrometeorological community on a regular basis to effectively use this knowledge. Chapter 2 presents a process by which NWS could learn to utilize relevant expertise from within the Enterprise and from other disciplines to improve communication of uncertainty information. The present chapter draws on lessons from these other disciplines as needed to support recommendations for improving uncertainty communication in hydrometeorology.
Full disclosure of forecast uncertainty information is consistent with—and in fact fundamental to—NWS’s established vision for communicating information (Box 4.1). This vision emphasizes dissemination of a wide range of NWS information. As discussed in Chapters 2 and 3, this means not only NWS forecasts and products but also the fundamental supporting information (such as verification and past performance) that is central to improving uncertainty communication throughout the Enterprise.
Beyond NWS’s philosophy of information availability, though, there are practical considerations on how to effectively communicate uncertainty information that help set the context of this chapter. For instance, the National Research Council (NRC) workshop on “Communicating Uncertainties in Weather and Climate Information” (Box 4.2) observed that understanding, communicating, and explaining uncertainty should be an integral and ongoing part of what forecasters do and are essential to delivering accurate and useful information.
COMMUNICATING UNCERTAINTY IN EVERYDAY AND HAZARDOUS WEATHER FORECAST PRODUCTS
Forecast uncertainty can be communicated in such products as maps (Figure 4.1), graphs, tables, charts, flip books, images, and written or oral narrative (see Annex 4 for a range of examples). Selecting an appropriate product type and carefully crafting its content can substantially reduce the likelihood of misunderstandings. Each approach to communicating uncertainty will inherently have strengths
National Weather Service Vision for Communicating Information
The NWS vision of communicating information to users is to
SOURCE: NWS, http://www.nws.noaa.gov/om/disemsys.shtml.
and weaknesses, and each may best communicate a different type of uncertainty to a different user group. Products can be tailored to specific user needs, but when communicating with a diverse audience such as the public, one product is unlikely to meet all users’ needs or to be readily understandable to all subgroups (Chapter 2). When such a broad audience is anticipated, a mix of products will likely be most useful (Chapter 2). In addition, an NWS National Digital Guidance Database (recommendation 3.6) would help support this mix of products by providing users and intermediaries with data and tools for customizing communication of uncertainty information.
NWS and other members of the Enterprise generate a variety of textual, verbal, and visual products that communicate uncertainty (Annex 4). However, most weather forecasts specifically generated for the public contain little or no useful uncertainty information; they are simplified and deterministic. Members of the public have been conditioned to these deterministic forecasts and have been given little objective information on the inherent errors in these simplified predictions. Instead, users in the public have developed their own informal methods of estimating the uncertainty. This highlights the need for user education as the Enterprise transitions to probabilistic forecasting.
One major example of a predominantly deterministic product is the NWS public weather forecasts produced by the Interactive Forecast Preparation System (IFPS) and distributed as the National Digital Forecast Database (NDFD). IFPS/NDFD’s strength is that it allows forecasters to generate, present, and communicate forecasts of multiple weather elements as a digital database, both for the local area and as a nationally unified grid. Its main weakness is that the forecasts contain limited information about uncertainty. Most variables are estimated and presented as “point forecasts”
Suggestions for Improving Communication of Uncertainty Information
The following practical suggestions were made during an NRC workshop to improve information delivery (NRC, 2003b):
Effectively communicating uncertainty and its context shifts the burden and responsibility of decision making to the information user. The following suggestions from the NRC workshop could improve communications to decision makers:
Perceived success or failure of forecasts and the portrayal of forecasts by the media and decision makers guide opinions and help determine the credibility of future forecasts. The following actions were suggested:
SOURCE: NRC, 2003b.
that appear as deterministic values or graphs for the next 7 days, with no change in format. As discussed in Chapter 3, these deterministic digital values for days into the future are not scientifically valid and could be highly inaccurate and misleading. In addition, the system issues forecasts of precipitation type and thunderstorm risk using vague uncertainty terms such as “slight chance,” “chance,” “likely,” and “occasional.” As discussed in Chapter 2 and developed later in this chapter, research has shown that these terms are interpreted by users as communicating a wide range of probabilities.
The NDFD enables a user to select a site-specific forecast and to extract tailored forecasts from the database. The drawback, though, is that these forecasts include no “qualifier” text or statistical ranges that provide the user with uncertainty information to aid decisions. Fundamentally, IFPS and NDFD are also designed from a deterministic framework (other than the “Probability of Precipitation” component) and thus cannot be easily modified to incorporate communication of forecast uncertainty information.
The provision of single-valued forecasts without uncertainty information (such as error bars on a meteograph) not only exposes a significant limitation of the NDFD/IFPS process but is also fundamentally inconsistent with the science (Chapter 1). Moreover, these digital systems may generate machine-derived text forecasts of “partly cloudy” skies for several days in a row—in essence representing a wide range of weather conditions—and therefore do not effectively communicate the complexity or uncertainty of future weather.
With the importance of digital dissemination of forecasts through the Internet, incorporating uncertainty information into NDFD and IFPS would be advantageous to the public, intermediaries, and specialized users. Many methods of communicating uncertainty are available. Choosing the most effective method (or methods) will require research and two-way interactions with users (see Sections 4.4 and 4.5). Possible methods to consider include displaying skill scores or the standard forecast variance for each forecast variable at different times or providing confidence intervals. Another possibility, to improve consistency, is to communicate cloud cover not as scattered or broken but rather in categories (such as high, medium, and low) or as a percentage (as is currently done for probability of precipitation type and probability of thunderstorms in Model Output Statistics [MOS]).
Finding: The public weather forecasts from the IFPS and distributed as the NDFD are one of NWS’s primary forecast products. The system is unable to provide probabilistic forecasts for most fields, and it cannot access probabilistic guidance from the National Centers for Environmental Prediction (NCEP) or other ensemble systems. With the incorporation and communication of uncertainty in most
forecast parameters, IFPS and NDFD can reach their full potential as forecast products that meet the NWS vision for communicating information. Development efforts are under way to provide initial probabilistic fields by “dressing” IFPS forecasts with historical error statistics, but making such capabilities operational is years away.
Recommendation 4.1: The NWS should expedite development of the IFPS toward a system that can access, produce, and communicate uncertainty guidance for most forecast parameters. Such a revised system should be able to access deterministic and ensemble prediction systems, historical error statistics, and statistically post-processed forecast information (e.g., MOS) to allow production of uncertainty information with varying levels of subjective and objective contributions. The system should be capable of preparing probabilistic products to communicate probability density functions and other types of uncertainty information (e.g., probability of temperature less than freezing or wind speed greater than 26 knots).
Most of the above discussion focuses on communication of uncertainty in public forecasts and outlooks. Communicating uncertainty in hazardous weather situations, particularly for short-fuse, possibly life-threatening events, presents additional challenges. Yet even in these situations, communication of simple uncertainty information may be advantageous (Box 4.3).
IMPORTANT ASPECTS OF COMMUNICATING UNCERTAINTY
Use of Language and Graphics
Users’ interpretations of forecast information can lead to misunderstandings that affect their decisions, sometimes with catastrophic consequences (e.g., Box 4.4). In particular, words or images that a forecaster or scientist thinks are clear may be interpreted differently by users (and differently among users). For example, interpretations of the term “possible” span most of the probability spectrum (Chapter 2). When this term is used to communicate forecast uncertainty, some users will inevitably misinterpret what the forecaster intended to convey.
In addition to reducing the likelihood of misinterpretation in the use of language to characterize uncertainty, NWS can also increase the clarity and accessibility of its uncertainty products. Two examples have immediate potential, the Area Forecast Discussion (AFD) and the Climate Prediction Center’s (CPC’s) monthly and seasonal forecasts.
Area Forecast Discussion
The NWS AFD is one of the most commonly accessed products on NWS Web sites. Notwithstanding the challenge
Communication of Forecast Uncertainty in Short-Term Warnings
Communication of uncertainty information within short-term, high-risk weather events such as tornados, flash floods, or severe storms presents a dilemma for forecasters. Research in risk communication suggests that motivating action requires clear, consistent messages that warn of the approaching hazardous event and recommend specific responses, as in current NWS tornado, severe thunderstorm, and other warnings. Adding uncertainty information to these forecasts may confuse the message and possibly delay life-saving actions. Yet every forecast contains some uncertainty, and many members of the public have experience with categorical forecasts of short-fuse hazardous weather events that have not occurred as forecasted. Such experiences can lead people to interpret warnings according to their own perceptions of forecast uncertainty, which may be substantially different than the uncertainty in the actual weather situation. This, too, can delay decisions to take action. Thus, at a minimum, including some consistent information about confidence in short-term forecasts and warnings may help people evaluate the uncertainty in the situation and, in doing so, benefit their decisions.
of effectively using words to convey uncertainty (Chapter 2), these discussions provide one of the few available assessments of forecast uncertainty generated by a human forecaster. They are particularly useful to meteorologists and to specialized users who understand the meteorology. However, the AFDs (and other NWS forecast discussion products) still have a major weakness: although some forecast discussions are now written in easy-to-understand terms for the general user, many are still difficult to understand (e.g., Figure 4.4), making the forecast discussion not as widely useful as it could be. Given the AFD’s wide popularity, the forecast discussions might also be adapted into an easily accessible narrative public product that communicates forecast uncertainty to nonmeteorologists.
Finding: AFDs are popular NWS products that were designed as technical discussions to enhance collaboration among NWS offices and to convey uncertainty to a specialized audience. AFDs are now routinely accessed by broad user community and could be even more widely read and utilized if they were written for the even larger nonspecialist audience.
Recommendation 4.2: The NWS should release the AFD only in layperson English to facilitate its broad use and understanding. For more sophisticated users, NWS could
Communication of Forecast Information During the Red River Flood of 1997 in Grand Forks, North Dakota
Unclear communication of uncertain forecast information can hinder decision making and have significant negative consequences. An example is the 1997 flood in Grand Forks, North Dakota (Figure 4.2). Although NWS prepared flood stage outlooks months in advance, and forecasters were aware that they were predicting a record-breaking, uncertain event, the outlooks were issued as just two deterministic numbers (expected flood stage and low stage). Members of the community interpreted this range of numbers in different ways, generally not realizing that a significantly higher flood was possible (Pielke, 1999; NRC, 2003b). As it turned out, the NWS flood crest forecasts were too low by several feet, until a few days before the flood crest (Figure 4.3, left panel). Although Grand Forks had made significant preparations based on the early outlook, the city was not adequately prepared for the higher flood, and the city experienced major flood damage. Many people blamed NWS for a blown forecast. According to Ken Vein, Grand Forks city engineer (May 4, 1997), “With proper advance notice we could have protected the city to almost any elevation … if we had known [the final flood crest in advance], I’m sure that we could have protected a majority of the city.”
A river stage forecast that communicated uncertainty more fully and clearly, such as the probabilistic product in Figure 4.3, right panel (which is a more recent NWS hydrologic product), may have led to better flood management decisions. In fact, Pielke found that the actual flood crest in this case was within the error range one would expect for such a forecast which could have been—but was not—communicated along with the forecast.
provide more detailed technical information linked to the AFD.
Climate Prediction Center Monthly and Seasonal Outlooks
Near the middle of every month, the CPC provides predictions of temperature and precipitation probability anomalies for the coming month, as well as seasonal (3-month) forecasts out to 12.5 months. Monthly outlooks are also updated at the end of the month. These predictions (or “outlooks”) are formulated as probability anomalies for three equally probable classes (below normal, near normal, and above normal). The anomalies now specify the probability assigned to the most likely class. The user must further examine the CPC Web site to determine the rules for distributing probability among the other two classes. The three classes are determined by dividing the normal distribution (for temperature), fitted to observations made over 1971-2000, into three equally likely classes (terciles). Because the underlying distribution for precipitation can be highly skewed, the observations are transformed into a normal distribution prior to dividing into terciles.
The primary mode for providing these forecasts is maps depicting the probability value associated with the most likely category at each location. Areas with anticipated above or below normal values are labeled and color coded according to the strength of the probability anomaly; where none of the forecast tools has demonstrated statistically significant skill, the forecasters label the non-colored area as “EC” (Equal Chances; see Figure 4.5). The EC areas can be ambiguous because they may also indicate the forecasters’ belief that each of the categories truly is equally likely. Other aspects of the maps can also be difficult for users to understand (e.g., what exactly are the meanings of the terms “above average,” “below average,” “equal chances,” and “normal”?). Morever, the maps do not convey all of the available or needed information. In particular, they prespecify the thresholds for each class (e.g., above average, below average, and normal), which limits users’ ability to obtain the information that may be most useful. The EC areas are especially problematic as they provide no information about the likely distribution of values.
To help users understand the rationale for specific forecasts, the CPC provides a discussion of the anomalies, which describes the sources of information and uncertainty used to develop the predictions. Technical discussions are also provided on topics such as the current state and evolution of the El Niño/Southern Oscillation. In addition, terminology definitions are provided. However, these discussions are likely not read or understood by many users.
The probability anomaly maps provide an indication of where conditions are likely to be in one of the four classes (e.g., above normal, below normal, normal, and equal
chance), but they do not directly indicate the expected median precipitation and average temperature values or the expected distribution of values. Thus, the CPC also produces maps of the “most likely anomaly” for 3-month temperature and precipitation forecasts1 (see Figure 4.6). These maps present contours of both the average anomalies as well as the climatologically average values. To supplement these maps, exceedance distribution graphics are available for each climate division (Figure 4.7). These plots present cumulative distribution functions for average conditions, as well as shifted distributions of precipitation and temperature (if a shift from normal is predicted) based on the outlook. The anomaly values for precipitation are based on the difference between the medians of the “normal” and “final forecast” distributions. Thus, anomalies indicated on the precipitation outlooks are not really “most likely.” Rather there is a 50 percent chance that the anomaly will be greater or less than the indicated value. The forecast distribution envelope is based on the expected sampling variability of the climatological probability of exceedence using 45 years of data.2 Thus, the anomaly distribution plots present the most complete information about the uncertainty in the forecasts by providing a complete distribution of possible values. The anomaly distribution plots provide sufficient information that a user can specify the precipitation or temperature threshold that is relevant for their use and obtain the associated probability. Some of this information is also available in tabular form.3
Finding: The graphics conveying monthly and seasonal outlooks are difficult for many users to understand and do not convey all the information (both graphical and tabular) that is available or needed. Exceedance probability distributions provide the most complete information about the climate probabilities at particular locations. These distributions do not rely on pre-specified categories or definitions of “normals.” Overall, more research is needed regarding user needs for these graphical and tabular formats, as well as more forecaster-user interactions to provide two-way feedback on this and other products.
Recommendation 4.3: The CPC should provide full exceedence probability distributions of the projected monthly and seasonal temperature and precipitation values in both graphical and tabular forms. A straightforward graphical presentation of this information should be developed that is understandable to relevant user groups.
Icons and Text Modifiers
Weather icons and text modifiers are becoming widespread within the Enterprise as a method of communicating forecast information. Understanding how users interpret these graphics is therefore important in the context of communicating uncertainty. According to one study (Box 4.5), users’ interpretations of icons may even introduce perceived uncertainty when none is intended.
In another example, from a local NWS office (Figure 4.8), it is unclear how users will interpret the message conveyed
in the product. For example, are the icons in the Tuesday and Thursday forecasts confusing given the accompanying text?
More generally, there seems to be little knowledge of how weather forecast icons will be interpreted by users, and insufficient incorporation of users into the icon development process. Fortunately, there is knowledge outside NWS and from other fields on how people interpret language and graphics (Chapter 2), and there are many ideas on how to use language and graphics to communicate uncertainty (e.g., Figure 4.9). Incorporating this knowledge into NWS and Enterprise efforts to communicate forecast uncertainty will create efficiencies and enable faster adoption of new methods.
Conflicting messages can increase uncertainty or confusion, hampering decision making. This section discusses the lack of consistency in use of uncertainty words and images.
As noted in the preceding section, icons have the capability to communicate complex information in an easily
Internet Survey of Icon Interpretations
In 2001, NBC4 in Washington, DC conducted an Internet-based surveya showing a cloud with one snowflake and a cloud with four snowflakes and asked respondents: What does picture B mean to you, in relation to picture A?b
Overall, about half of the respondents thought the symbol with four flakes meant more snow, and slightly less than half thought it meant the forecast was more certain it would snow. In other words, many respondents interpreted a deterministic icon as if it were conveying uncertainty. This suggests that forecast providers cannot necessarily predict how users will interpret graphics without careful and thorough study of user interpretations and needs.
accessible way. However, the icons that accompany the digitally generated forecasts on NWS homepages are sometimes inconsistent with the accompanying numerical/text forecast. In some instances, these “icon forecasts” can be more confusing than helpful; for example, the same icons are sometimes used for a variety of forecasts (Figure 4.10) and, indeed, significantly different forecasts.
Uncertainty words are used inconsistently within NWS and, more generally, across the Enterprise. This inconsistency makes it challenging for users to calibrate the meaning of uncertainty forecasting terms based on experience. In addition, such inconsistency is sometimes evident in different products from the same NWS office during the same period. Box 4.6 contains an extended example from one local NWS office.
Local innovation and individual forecaster creativity within NWS is important to help the agency serve local and national needs. But by relying on evolving, ad hoc, and experimental systems without more extensive, consistent, and scientifically valid communication techniques, NWS’s communication of uncertainty information may be interpreted differently by users looking at different products from the same NWS forecaster. A variety of products is needed to effectively communicate uncertainty to the broad range of users that NWS serves. But these need to be consistent across all regions, platforms, and product language and communication methods. Related to this, the NRC Fair Weather report (NRC, 2003a) recommends that “NWS headquarters and regional managers should develop an approach to managing the local forecast offices that balances a respect for local innovation and creativity with greater control over the activities that affect the public-private partnership, especially those that concern the development and dissemination of new products or services.”
Finding: A variety of products is needed to communicate uncertainty to a broad range of users. Consistency of language, icons, and graphical representations of uncertainty among all these products is critical for the effective communication of uncertainty information. A necessary first step toward ensuring consistency is understanding users’ interpretations.
Recommendation 4.4: To ensure consistency in the communication of uncertainty information and user comprehension, NWS should more fully study and standardize uncertainty terms, icons, and other communications methods through all pathways of forecast dissemination.
The main channels through which NWS distributes information directly to the user are the NWS home pages (such as those of the Storm Prediction Center, Tropical Prediction Center, Hydrometeorological Prediction Center, and the various Weather Forecast Offices), National Oceanic and Atmospheric Administration (NOAA) Weather Radio (NWR), and the Emergency Managers Weather Information Network (EMWIN). In addition, NWS distributes information to intermediaries through the NOAA Weather Wire Service (NWWS), Interactive Weather Information Network, NOAAPORT, and Family of Services.
NWR and NWS home pages have formats in which NWS can and already does incorporate uncertainty information into forecasts. Systems such as the NWWS and/or EMWIN, which emergency managers and some private-sector entities (e.g., utilities, transportation industry) use to receive NWS forecasts and special weather statements, can also be used to communicate uncertainty information through text descriptions. In addition, both NWWS and EMWIN allow display of graphical products that could communicate uncertainty.
Dissemination technologies are evolving rapidly, affecting how NWS and the other members of the Enterprise approach communicating uncertainty information. Communication devices such as cell phones, personal digital assistants (PDAs), portable MP3 players, computer toolbar “bugs,” and pagers have become commonplace, and the Enterprise is moving closer to being able to communicate information to nearly anyone, anytime, anywhere.
These developments present both opportunities and challenges. The opportunity is that the Enterprise can now reach more people in more places and at more times than ever before. One challenge is to understand the strengths
Consistency in Communicating Forecast Uncertainty Using Words and Graphics
Four forecast products containing uncertainty words or graphics are used in this example: the text forecast (Figure 4.11) for March 5 to 11 (Sunday to Saturday), the hourly weather graphs for the Sunday to Tuesday in that period (Figure 4.12) and Friday and Saturday of that period (Figure 4.13), and the digital IFPS product for the Sunday through Wednesday (Figure 4.14). For the same period of time on the Sunday afternoon, and for the same region, different NWS products generated by the same local office present the short-term forecast for the snow event as any one of
Users looking at different products are likely to have different interpretations, and users looking at multiple products are likely to be confused. In addition to this inconsistency, Figure 4.12 indicates no precipitation in its lower panel concurrently with showing the PoP at 21 percent (middle panel), whereas the same percentage PoP triggers “chc” or chance of rain in Figure 4.13. Furthermore, this product (Figure 4.13) generates the category “occasional” for a PoP of 100 percent on Sunday, March 5. This could confuse users who are not familiar with the official meteorological definition of PoP. Other products communicating uncertainty in the same forecast are also available, such as the Hydrometeorological Prediction Center’s (HPC’s) snow product (Figure 4.15). Users could derive value from this product, but many are unlikely to be aware of it because it is not linked from the local forecast.
The local Area Forecast Discussion for the same region and time as in Figures 4.11 to 4.14 (not shown here) provides sophisticated users with additional insights into the forecast situation. For the March 5 snow event, the forecaster also prepared a simpler discussion, called a “regional synopsis” (Figure 4.16). The technical discussion gives useful additional information about the forecast situation, including the forecaster’s insight into uncertainty. This includes mention of possible rain/snow mix and slush—potentially important information for a range of users—whereas the public text forecast (Figure 4.11) is only for “snow.” The regional synopsis describes the coming event as a “mixed bag” (a phrase with unclear meaning) and uses “possible,” this time to describe snow accumulations.
and weaknesses of each new communications platform, with respect to conveying uncertainty information, and to optimize the strengths and minimize the weaknesses. For example, cell phones and PDAs have a limited capacity for receiving and conveying information; nonetheless, their usage continues to expand and bandwidth limitations will be reduced. Thus, a strategic approach to utilizing these and other evolving technologies is to recognize the current limitations, track the actual and potential evolution of these technologies, and, most importantly, identify and tailor communications products for them because they enable broad and rapid dissemination of uncertainty information. As the Enterprise considers the incorporation of uncertainty information into products, these opportunities and challenges will have to be understood and incorporated into the product development processes. Finally, even as more—and soon the majority of—forecast uncertainty information is provided to users through “new” media, NWS and the Enterprise will still need to serve users who receive their forecasts through long-standing “traditional” media including television, newspapers, and radio.
The dramatic recent increase in Internet usage has also changed how NWS and Enterprise disseminate forecast information. For example, the Enterprise now routinely conveys complex forecasts and supporting information, including graphical and visual information, through the Internet (e.g., Box 4.5). Indeed, NOAA and NWS home pages and information sites are already the most frequently visited government sites on the Internet.4 Looking ahead, Internet technology could help NWS meet the needs of both specialist and nonspecialist users alike by allowing individual users to select the format and detail of uncertainty information most appropriate for them. Such technology could also help disseminate the increased amount of forecast uncertainty information. For example, the NWS home page could present limited, relatively simple uncertainty information, such as forecaster confidence, high/low temperature ranges, and PoP and precipitation type, with a one-sentence summary of the weather situation. Visitors to the site could have the option to click on the basic information to view graphics that give more detailed uncertainty information about different aspects of the forecast situation, a text discussion of the uncertainty in the forecast situation, and so on. In addition, the information could be viewed in different presentations (e.g., as a map, as a graph of temporal evolution at one location, or as a pdf at one location and time). Additional “clicks” would allow specialist visitors to drill down to obtain more detailed uncertainty information or to download digital information that they could incorporate into their own analyses or value-added products. Such a capability would be facilitated by the development of a database of uncertainty information, such as the National Digital Guidance Database (Chapter 3) and the modifications to IFPS proposed in recommendation 4.1.
The Role of the Traditional Media
The majority of weather information in the United States is communicated to users through intermediaries, particularly the media industry. This industry is composed of two parts—the new media (e.g., Internet, cell phones, pagers, bugs) and the traditional media (e.g., television, radio, newspapers). Weather forecasts and products are one of the primary sources of “hits” and “page views” on the Internet, and although weather information on the Internet and other new media continues to rapidly expand, television (including cable) is still the public’s primary source of weather information.
The media industry as a whole can be considered both “gatekeeper” and a principal partner with NWS in the communication of forecasts, warnings, and uncertainty information. As such, intermediaries, and specifically the media, play a critical role in communicating forecast uncertainty and addressing the challenges presented in doing this effectively. To fully understand the implications of this situation, it is helpful to understand the business fundamentals of the media industry.
The vast majority of the revenue generated in media comes from advertising. Advertisers pay to have their message seen by as many potential customers as possible. Advertisers also pay a premium to have their message delivered to certain “target” customers. When the media communicates hydrometeorological information, the quality of the information content itself is only one of several factors affecting the recipient’s overall perception of quality of the product. Other environmental components that directly affect audience attraction and retention include the personalities presenting the information, the aesthetics of the visuals, and the effectiveness of promotion.
The media industry is also highly competitive. Advertising budgets are limited and any advantage can make a critical difference in where those dollars are spent and the profitability of media entities. Similarly, any disadvantage is often very costly. Including uncertainty information in a forecast may be viewed by some media industry managers and advertisers as a demonstration of weakness, hedging, lack of credibility, or lack of skill instead of as providing a better, scientifically sound, and more useful product. In fact, this is probably one of the main drivers of what might be called the “pretend determinism” that exists in many media presentations today. On the other hand, savvy media entities could regard the inclusion of uncertainty information in forecasts as a competitive advantage.
Changing this deeply embedded legacy will be difficult and require significant time and determination. Nonetheless, there is basis for hope. For example, the news media adopted the hurricane cone of uncertainty (Figure 1.6), which dramatically illustrates uncertainty in the hurricane track forecast, not long after NWS’s Tropical Prediction Center developed it. Similarly, many media presentations now include the
Probability of Precipitation
PoP forecasts have been provided for several decades and are an example of uncertainty forecasts that the public and other users have accepted and used. The public’s understanding of the technical meaning of the forecasts has been debated for years, yet the public does appear to understand PoP sufficiently well to find it useful (Murphy et al., 1980; Gigerenzer et al., 2005; NOAA Technical Memorandum NWS AR-44). Numerous presenters of forecasts to the public have found that whereas news management may want simple deterministic pronouncements, the public does use the PoP in their decision-making process.
PoP (Box 4.7) within text or graphics as they have found users find the probability useful to their decision making even though they may not understand exactly the correct definition of “PoP.” Products that effectively communicate uncertainty will be adopted by the media and advance the public understanding of uncertainty and utility in decision making. Effective and ongoing partnerships between the media and other parts of the Enterprise, along with education of the public and the media, will be critical for acceptance of uncertainty by the user community. NWS is positioned to catalyze this process.
ROLE OF USERS IN THE PRODUCT DEVELOPMENT PROCESS
The shift from a deterministic to a probabilistic approach to communicating forecasts represents such a fundamental change in the presentation of information that user perceptions and opinions will be needed throughout the product development process. Thus, it is important to consider venues for user consultation by NWS, the approach to product development at NWS, and ways in which collaboration with users could be improved.
The need for attention to user interaction on product development is recognized in several earlier NRC reports. In A Vision for the National Weather Service: Road Map for the Future (1999b), the NRC recommended that NWS “routinely examine and anticipate the needs of primary customers and ultimate users.” In Making Climate Forecasts Matter (1999c), the NRC emphasized that there was a need “to bring scientific outputs and users’ needs together,” specifically through the use of surveys and structured discussions. In Fair Weather: Effective Partnerships in Weather and Climate Services (2003a), NRC recommended that NWS establish an independent advisory committee to gather feedback from users, representing all sectors, on weather and climate matters. Discussions within NOAA on this option are under way.
NWS uses a variety of mechanisms for interacting with users and getting feedback on its products. These include formalized workshops and local meetings with public officials, emergency managers, the media, and other weather-sensitive groups. In addition, and usually at the local level, direct informal one-on-one communications take place. For example, NWS staff at the Weather Forecast Offices or Regional Forecast Centers (RFCs) work regularly and directly with users, especially larger customers.
Other venues for user interaction are national meetings of professional organizations (e.g., the American Meteorological Society), annual NWS partner meetings, NOAA data and information user workshops, and annual meetings of target user communities (e.g., the space weather community5). Such interactions can also lead to new forecast uncertainty products; for example, a new probabilistic seasonal forecast product originated from six user workshops held at the RFCs during 1994.6
In addition, NWS gathers user feedback through its Web pages and through Customer Satisfaction Surveys. Although such feedback does provide a snapshot of user issues, such surveys may be insufficient to fully understand and address user needs because they may lack detail, may not be a representative sample, and may not be designed to develop a thorough understanding of how to more effectively communicate uncertainty information. Such surveys are not a substitute for formal research or for effectively collaborating with users (Chapter 2).
Last, NWS has stated7 that it can collect user advice through the NOAA Science Advisory Board (SAB) and the NRC (e.g., this committee). However, neither SAB nor NRC can provide the dedicated, continuous, and long-term forum that user interaction deserves. In addition, SAB may not be the appropriate venue for dealing with the details and nuances of forecast uncertainty information and how it is presented and disseminated.
Although NWS uses a range of mechanisms for gathering user perspectives, these perspectives are not formally or consistently sought throughout the product development process. New products can appear to users to be more technologically driven than scientifically or user driven. NWS’s policy for developing new and enhanced products and services is given by NWS Directive 10-102 (approved August 28, 2002; Box 4.8).8 The NOAA Partnership Policy (Box 1.3) also bears directly on the product development process.
Although Directive 10-102 discusses user needs and requires user feedback, it does not require or discuss sustained feedback from the broader Enterprise from the earliest stage of the product development process. In the first stage, new or amended NWS products can be proposed at any level within NWS. These products must solely meet a requirement of mission connection to NWS. The subsequent experimental phase involves customer feedback and comment on preferences, but does not include objective evaluation of how the product will be interpreted by different users, or how users might make better decisions using the product. Evaluation of NWS products is largely conducted by outside survey organizations that seek to measure customer satisfaction (which may not necessarily mean understanding of or best decision making with the product, especially with respect to communicating uncertainty; Chapter 2). In addition, and with the recent exception of a product developed in the Hydrologic Services Division,9 products are rarely scientific evaluated from perspectives outside the atmospheric and related physical sciences (e.g., by social scientists) prior to product implementation. In fact, social science expertise within NOAA is presently underutilized.10
Finding: The official NWS process for developing new products does not formally engage the user throughout the product development process. Rather, it seeks feedback when the product already has gained significant momentum. Moreover, the feedback obtained often fails to rigorously and comprehensively evaluate the product’s effectiveness.
Recommendation 4.5: NWS should extend NWS Directive 10-102 to require collaboration with users on product development throughout the development process. Moreover, users’ comprehension and interpretation of products should be formally evaluated at several stages during the product development process.
Processes for collaborating with users throughout the product development process will need to accommodate the complexities involved in disseminating and communicating forecasts to a wide variety of users, and how those users incorporate forecasts into their decision making. Such complexities highlight the need for a rigorous and sustained effort that draws on a broad range of expertise, including from the social sciences. These complexities, for example, include users being reached directly or through intermediaries who add value to NWS products; the diversity in technical sophistication and ability to utilize products among users; and the constant evolution of technology and user needs, as well as the user population itself.
NWS Directive 10-102
NWS Directive 10-102 describes in detail the process of, and framework for, developing a new product or service or changing an existing one, including the development of an internal Product/ Service Description Document (PDD). The directive requires internal sponsors of a new product to discuss its intended use and audience; its Appendix B states that “[n]ew products should be developed to satisfy valid user needs/requirements.” The directive also discusses seeking external review and comment upon proposed products and services, stating that “NWS will seek ongoing user feedback on official [operational] products.” Moreover, the directive dictates that NWS will include a feedback statement with each product (one that identifies an actual person responsible for collecting feedback) or otherwise provide a feedback notice to the public.a Feedback must be reviewed at least annually.
The procedures outlined in Directive 10-102 clearly mandate the involvement of users in product development, depending, of course, on how well this directive is actually implemented in practice. It does not, however, require or even mention the involvement of users in concept development; rather, users are not formally involved in new product development until the experimental stage, when a product is nearly complete. Further, the PDD itself, the key document that initiates the product development process within NWS, does not require any user input or external review.
Fortunately, there are several examples where units of NWS have worked, or are working, effectively with users. These efforts could serve as models as NWS refines its product development process. Possible case studies include the development of the AHPS, the NWS Regional Climate Services (for example, in the Central Region), and the Regional Integrated Sciences and Assessment (RISA) program (e.g., the RISA in Arizona). There are also many models to draw from outside NWS—at public institutions (e.g., the NCAR-RAL partnerships with the Federal Aviation Administration and Federal Highway Administration) and in the private sector.
RESEARCH AND DEVELOPMENT PROGRAM TO IMPROVE COMMUNICATION OF FORECAST UNCERTAINTY
Without a stronger knowledge base, NWS and broader Enterprise efforts to improve communication of forecast
uncertainty will be inefficient and perhaps ineffective. A sustained research and development program to improve communication of uncertainty in hydrometeorological forecasts is therefore needed (see also recommendation 4.4). Research on communicating uncertainty is ongoing in many disciplines and sectors outside NWS. For example, there is a body of knowledge about communication of uncertainty and risk in weather, climate, and hydrology, as well as in areas such as medicine and natural and technological hazards (see Chapter 2). Much of the existing research identifies “don’ts” and important factors to consider when communicating uncertainty. Yet many questions remain unanswered, particularly about how to better communicate uncertainty (“dos”), how to do so in the hydrometeorological forecasting arena, and how to do so in ways that meet different users’ needs.
The topic of communicating uncertainty is broad, including all sectors of the Enterprise, and there are many important research questions to be addressed. Moreover, the forecast and communication environments are continuously evolving. Thus, rather than identify specific prioritized research questions, the committee recommends a process for NWS, in partnership with the Enterprise and others, to identify and address key research questions. Examples of possible initial directions include understanding (1) the needs of users for uncertainty information and incorporating these needs into communication techniques; (2) how to effectively partner with the media and other intermediaries to improve uncertainty communication; (3) whether, when, and how to communicate uncertainty in short-term warnings for hazardous weather-related events; (4) what is the relative effectiveness of communicating uncertainty in different ways to users (e.g., as forecast confidence versus different forms of probabilistic forecasts); and (5) how to effectively design icons and other visual tools.
To select research directions and develop them into focused research and development efforts, NWS will need to partner with members of the academic, public, and private sectors on a regular basis to survey existing knowledge, specify priority areas, and develop implementation strategies. One mechanism for doing so is through testbeds (see Chapter 3). To gather a full picture of existing knowledge and needs within NWS, the process of selecting and developing research directions will benefit from communication between NWS headquarters, national centers, and regional and local offices.
A key component of this broad program is for NWS to acquire core in-house expertise in relevant social sciences (recommendation 2.4). This in-house expertise is needed to (1) conduct research, particularly in response to short-term needs; (2) help NWS identify priority research questions and appropriate methods for answering them; (3) help NWS identify and engage relevant external social science or other expertise; and (4) assist with product development. In addition, there is a large external community, particularly in academia, with relevant expertise that could significantly advance this effort. Because most of these researchers rely on funding support, they tend to focus on topics for which more funding is available. One way, then, to engage the academic research community in helping NWS and the Enterprise as a whole improve communication of uncertainty is to draft Requests for Proposals (RFPs) on questions of overlapping interest. These RFPs could be developed and put out by NWS/NOAA on its own, or to leverage additional funding to address research questions of mutual interest, through NWS/NOAA partnerships with other entities. For example, questions that have fundamental as well as applied aspects might be addressed through joint efforts with the National Science Foundation (in addition to linking with the private sector), whereas questions that focus on communication with specific user groups served by NWS might best be addressed through joint efforts with other federal mission agencies, also in partnership with the private sector as appropriate. Other mechanisms for NWS and the Enterprise to put relevant questions about uncertainty communication on the national and international social science agendas include funding visiting scientist programs, student internships, and dissertation and post-doctoral fellowships. Such programs can also train future researchers in this area and train future forecasters and users in key aspects of uncertainty communication. In all, because of the complex, interdisciplinary nature of the topic, this research and development effort will need to employ a mix of strategies to be successful and enduring.
Questions on communication of uncertainty in areas beyond hydrometeorological forecasting are of critical interest to a variety of communities, both within the Enterprise and more broadly. Thus, NWS interests in this area have significant overlap with a number of other entities across society. This has several implications. First, there are groups already studying and implementing more effective communication of uncertainty that NWS will benefit from engaging. Second, there are ample opportunities for NWS to partner with other groups in developing and implementing joint initiatives, leveraging available funding. Third, the results of a program on communication of forecast uncertainty can be used in other areas of risk communication such as medicine, terrorism, and other disasters.
Last, methods of communicating hydrometeorological uncertainty are being explored or implemented in several other countries (see examples in Annex 4). In developing and testing methods for uncertainty communication, NWS will benefit from regular consultation with foreign hydrometeorological services to share experiences and lessons learned. From a global perspective, the World Meteorological Organization (WMO) can be a useful venue for international dialog on experiences with communication of uncertainty, research initiatives, and “best practices.” In addition, WMO has training programs for weather presenters and communicators in many developing countries. Effective communication of forecast uncertainty, built on international dialog on such matters (perhaps led by NWS and the U.S. permanent
representative to WMO), could help regions with weather risks worldwide.
EDUCATION AND TRAINING NEEDS
The research and development aspects of communicating uncertainty will include and lead to education and training of all parties participating in generating, communicating, and using hydrometeorological forecasts. Here “education” is used in a broad sense and involves communication, understanding, and learning.
Implementation of this report’s recommendations will change how the Enterprise operates and will lead to adoption of new forecast techniques, products, and communication tools by all sectors. But this will only happen if all Enterprise partners are actively involved in “turning the ship” and are working in a mutually supportive framework. Education initiatives will need a strong commitment by all sectors of the Enterprise and include a wide variety of participants—from elementary school teachers and students to emergency managers, media managers, and communicators. Such initiatives include undergraduate education of future hydrometeorological professionals and continuing education and training of all who communicate hydrometeorological information and forecasts (especially those working in the media). And these initiatives will rely on a two-way interaction that involves effectively communicating new information while also soliciting feedback to further improve communication and understanding.
Most forecasters begin their education in an undergraduate program such as a meteorology program. Current standards for undergraduate meteorology programs established by the federal government and the American Meteorological Society (Smith and Snow, 1997) have no requirement to cover uncertainty, use of probabilistic information, and how forecast-related information is used in decisions. Without this material in their curriculum, many meteorology students are not adequately prepared for future careers in generating, communicating, or using hydrometeorological foreasts that include uncertainty information.
Training courses are a critical vehicle for forecasters’ continuing education, once they have graduated. These courses convey the latest insights and techniques that enhance forecast generation and communication. Such courses could deliver relevant information and training on communicating uncertainty information. With respect to the training component, academia and government laboratories could partner on developing coursework that addresses forecast uncertainty.
A hydrometeorological forecast is often only one piece of a broader spectrum of information being integrated into a decision (Chapter 2). As the human role in conveying probabilistic forecast information becomes increasingly focused at the interface between forecast systems and user decisions (e.g., functioning as the “science integrator” who takes what is known about the science and communicates it to decision makers), academic and other training programs will need to adjust their content accordingly. This adjustment may entail adding material into existing courses or by offering elective courses that focus on communication, probability, and decision issues facing weather- or climate-sensitive decision makers.
Because members of the public receive most of their forecasts from the media, the media will play a critical role in helping the public understand and use new uncertainty products. Take, for example, PoP forecasts (Box 4.7). The value derived from PoP forecasts is in no small part due to the long-term efforts of the Enterprise, especially media meteorologists and weathercasters, in educating users about PoP. Even if many members of the public do not know the exact meteorological definition of PoP, many still consider this uncertainty information useful (e.g., Figure 4.17). The broad familiarity with the hurricane track probability forecast (Section 4.3.4) is another case in which the media played a critical role in facilitating acceptance of an uncertainty product and educating the public about its meaning. Including probabilities within the cone presents an opportunity for improving public understanding of uncertainty forecasts.
Even the “best” uncertainty information will not serve its ultimate purpose—helping users make better decisions that enhance socioeconomic value—unless that information is effectively communicated. Although a variety of forecast uncertainty products are available from NWS and others in the Enterprise, some of these products do not communicate uncertainty as effectively as they could. Moreover, most publicly available forecast products often communicate little or no uncertainty information. Changing from the current paradigm of primarily deterministic forecast communication will be a major shift, requiring a concerted, coordinated effort by NWS in partnership with others in the Enterprise.
The use of uncertainty information in decision making is complex. Learning to communicate uncertainty effectively will therefore require consideration of three factors. First, effective communication must incorporate an understanding of user needs for uncertainty information and how users will apply it. Such understanding must be based on social science research and close interactions with users starting early in the product development process. Second, effective communication requires considering and preventing potential user misunderstanding and confusion that can result from inconsistent communication and ineffective use of uncertainty language and graphics. Third, effective communication of uncertainty requires understanding the key roles that dissemination mechanisms and technologies and the media play in conveying forecasts. This chapter provides several recommendations to help NWS and the Enterprise shift to a new paradigm of clear, effective communication of forecast uncertainty that is consistent with scientific understanding
of the atmosphere and hydrosphere and knowledge of how uncertain information is used in decision making. These include developing comprehensive education and training efforts and a dedicated, long-term research and development program to improve uncertainty communication in hydrometeorological forecasts.
EXAMPLES OF UNCERTAINTY COMMUNICATION APPROACHES AND PRODUCTS
This Annex presents examples of the range of uncertainty communication approaches and products. The committee sought examples from a variety of sources, including NWS management and individuals from NWS, other government agencies, the private sector, and academia. These examples included operational, experimental, and proposed products, primarily from weather, climate, and hydrological forecasting, but also from other fields.
Maps are useful for communicating spatial distributions of forecasted variables and their uncertainty. They can represent forecasts at a specific time, over a specific period, or as a coherent weather feature (such as a hurricane or winter storm) evolves and moves. One way of using maps to communicate forecast uncertainty is to portray the spatial distribution of the likelihood of an event (e.g., tornado) or of a parameter exceeding a specified threshold (e.g., precipitation greater than one inch). Figure 4A.1 shows two examples, one containing numerical probabilities and the other containing qualitative likelihoods.
A second way of using maps to communicate forecast uncertainty is to portray the spatial distribution of minimum/ mean/maximum expected (or 10/50/90 percent exceedance) values of a parameter (Figure 4A.2). Maps can also be used to portray the likelihoods of different scenarios in different regions (Figure 4A.3). Another method of communicating uncertainty using maps is to overlay a map of mean or expected values with a map of uncertainty or confidence (Figure 4A.4).
The example maps above primarily use contours to represent values. When values are depicted using numbers or other symbols, uncertainty can be portrayed using different symbol sizes or colors (Figure 4A.5).
Maps can also be used to communicate how uncertainty associated with a moving feature, such as a hurricane or winter storm, evolves with time. One such type of map depicts uncertainty in a feature’s location along its track; an example for hurricane track forecasts was shown in Figure 1.5, whereas Figure 4A.6 shows an example for midlatitude low-pressure systems. Uncertainty in the weather associated with a feature (e.g., wind) at different times along its track can also be depicted (Figure 4.1).
Graphs communicating uncertainty can take many forms. One type of forecast uncertainty graph depicts the temporal evolution of a quantity of interest, with uncertainty represented using an ensemble of multiple temporal trajectories, box and whisker plots at each time, or probabilities of exceedance of one or more thresholds at each time. The example shown in Figure 4A.7 uses box and whisker plots to communicate how uncertainty in two forecast parameters increases and evolves with time.
A related type of graph is the temporal evolution of the probability of a certain event (such as precipitation) or multiple events (rain, snow, and ice; Figure 4A.8). Another type of graph, commonly used by scientists but probably less easily understood by many members of the public, is a probability density function (PDF) of a variable at a specific location and time (Figure 4A.9). As noted in Section 4.3.3, with Internet technology and in sophisticated decision-support systems, these types of maps and graphs can be combined. For example, a general map or graph can be presented first, allowing users to click on a location or time of interest and obtain a more specific graph or PDF.
Most maps and graphs used to communicate uncertainty in hydrometeorological forecasts are two-dimensional. However, three-dimensional representations can also be used (see, e.g., the NRC’s Board on Mathematical Sciences and their applications workshop “Toward Improved Visualization of Uncertain Information”), as well as movies.
Narratives can be used to communicate uncertainty orally or through written text. Three examples of narrative forecasts are the NWS forecast discussions written by WFO, HPC, TPC, CPC, and other NWS/NOAA forecasters;
NWR; and TV forecasters presenting a forecast. Figure 4.4 shows an example of the NWS AFD. Often, but not always, narratives accompany one or more maps, graphs, or tables/ charts. Narratives are versatile; they can be used to describe uncertainty in ways ranging from indications of forecaster confidence to scenarios of different ways that weather events might evolve. As noted in Chapter 2, however, uncertainty words are often ambiguous, meaning that using words to convey uncertainty can result in ineffective communication or even miscommunication.
As illustrated above, NWS and other members of the Enterprise issue a number of forecasts that include uncertainty information. Nevertheless, as discussed in Chapter 1, most forecasts received by the public and many users still contain little or no information about uncertainty. A prime example is NWS’s public weather forecasts produced by the IFPS and distributed as NDFD (Chapter 3). The only element within the IPFS and NDFD operational system that provides uncertainty information is the PoP. Variables such as temperature, dew point, and sky cover are generated as single (deterministic) values out to 7 days, with no change in format as lead time (and thus uncertainty) increases (Figure 4A.12). The basic suite of NWS public forecasts are now automatically generated from NDFD by IFPS, with limited time for forecaster editing. Thus, these forecasts, too, contain no information about uncertainty other than PoP.
Another example of an NWS product that does not convey uncertainty is the quantitative precipitation forecast (QPF) forecast issued by the HPC (Figure 4A.13). This product is accompanied by a forecast discussion that often discusses forecast uncertainty. This uncertainty information is not integrated into the QPF product, however, and thus is likely not seen by many users. Many other products issued publicly by NWS and others in the Enterprise are similar, with limited communication of uncertainty information in ways accessible to those outside the hydrometeorological community.