Summary
All prediction is inherently uncertain and effective communication of uncertainty information1 in weather, seasonal climate, and hydrological forecasts benefits users’ decisions (e.g., AMS, 2002; NRC, 2003b). The chaotic character of the atmosphere, coupled with inevitable inadequacies in observations and computer models, results in forecasts that always contain uncertainties. These uncertainties generally increase with forecast lead time and vary with weather situation and location. Uncertainty is thus a fundamental characteristic of weather, seasonal climate, and hydrological prediction, and no forecast is complete without a description of its uncertainty.
Nonetheless, for decades, users of weather, seasonal climate, and hydrological (collectively called “hydrometeorological”) forecasts have been conditioned to receive incomplete information about the certainty or likelihood of a particular event. But this has not always been the case. As early as the 19th century, some predictions included qualitative probabilistic2 expressions of uncertainty and were actually called “probabilities” rather than forecasts. By the 20th century, meteorology evolved into what was thought to be a more exact science and predictions became deterministic3 with no expression of uncertainty. The advent of numerical weather prediction around 1950 and its early successes strengthened this deterministic viewpoint, as did improvements in satellite observations and modeling methods in the 1970s and 1980s. Users became comfortable with single-valued forecasts4 and applied their own experience in determining how much confidence to place in the forecast. The evolution of the media as the primary vehicle for conveying weather information in the United States compounded this trend. The inclusion of uncertainty information in a forecast was viewed by some as a weakness or disadvantage instead of supporting a more scientifically sound and useful product. Most forecast products from the weather and climate enterprise (the Enterprise5), including those from the National Oceanic and Atmospheric Administration’s (NOAA’s) National Weather Service (NWS), continue this deterministic legacy. Decisions by users at all levels, but perhaps most critically those associated directly with protection of life and property, are being made without the benefit of knowing the uncertainties of the forecasts upon which they rely.
Fortunately this situation can be improved. NWS and others in the Enterprise have recognized the need to view uncertainty as a fundamental part of forecasts. By partnering with other segments of the Enterprise to understand user needs, generate relevant and rich informational products, and utilize effective communication vehicles, NWS can take a leading role in the transition to widespread, effective incorporation of uncertainty information into hydrometeorological predictions.
This study explores how to improve the generation, communication, and potential use of uncertainty information for hydrometeorological forecasts and makes recommendations for improvements. The study was requested by NWS in response to ideas in Fair Weather: Effective Partnerships in Weather and Climate Services (NRC, 2003a). In particular,
NWS asked the committee to (1) provide guidance on how to identify and characterize needs for uncertainty information among various users of forecasts; (2) identify limitations in current methods for estimating and validating forecast uncertainty, relating these limitations to users’ needs, and recommending improvements or new methods and approaches; and (3) identify sources of misunderstanding and recommend improvements in the methods used to communicate forecast uncertainty.
Recognizing the breadth and depth of this task, NWS advised the committee at its opening meeting to “teach us how to fish as opposed to giving us a fish.” The committee approached the task accordingly. Relative to the first component of the task, the report reviews how decision makers interpret and use uncertainty information with the aim of helping NWS and others in the Enterprise understand key relevant concepts in decision making under uncertainty. Building from these concepts, the committee recommends a process by which NWS can develop an effective system of user–provider interactions that will lead to more effective products. Relative to the second component of the task, the committee takes the view that generating comprehensive uncertainty information to support all forecasts is central to the mission of NWS and will benefit all users. Such information must be made easily accessible and include all raw and post-processed products as well as verification and measurement information.6 The committee addresses the third component of the task by exploring the roles of graphics and language, dissemination technologies, and the media in communication of uncertainty information. In addition, the committee proposes refinements to NWS’s product development processes and highlights the need for education and research to support Enterprise-wide progress on communication of uncertainty information.
NWS asked the committee to recognize the diverse roles of participants in the Enterprise and the varied needs of forecast users. In addition, NWS requested that the committee’s recommendations focus primarily on the NWS mission, but may also address other components of NOAA or seek to guide other relevant government agencies and nongovernmental entities. In cases where a recommendation states that “NWS should …” it is the committee’s intention that the recommendation also applies to any relevant group or activity within NOAA, such as the Office of Oceanic and Atmospheric Research (OAR). The committee met five times between April 2005 and February 2006. One of its meetings was held in parallel with the annual American Meteorological Society Numerical Weather Forecasting and Broadcast Meteorology Conference, and another was held at the National Center for Atmospheric Research. NWS provided significant informational input to the process, both at meetings and in responses to questions posed by the committee.
OVERARCHING FINDINGS AND RECOMMENDATIONS
Moving toward effective estimation and communication of uncertainty information has broad and deep implications for the Enterprise and the community it serves. Because of the immense breadth and depth of this challenge, detailed solutions are beyond the reach of a single committee. Consequently, this report provides general ideas for consideration by NWS and the entire Enterprise.
The committee presents nine overarching recommendations all with equal priority. In addition, detailed recommendations appear in Chapters 2, 3, and 4 that add further specificity and breadth. All recommendations should be considered in the context of NOAA’s Policy on Partnerships in the Provision of Environmental Information.7
Enterprise-wide Involvement
Finding 1:8 Hydrometeorological services in the United States are an Enterprise effort. Therefore, effective incorporation of uncertainty information will require a fundamental and coordinated shift by all sectors of the Enterprise. Furthermore, it will take time and perseverance to successfully make this shift. As the nation’s public weather service, NWS has the responsibility to take a leading role in the transition to widespread, effective incorporation of uncertainty information into hydrometeorological prediction.
Recommendation 1: The entire Enterprise should take responsibility for providing products that effectively communicate forecast uncertainty information. NWS should take a leadership role in this effort.
Product Development Incorporating Broad Expertise and Knowledge from the Outset
Finding 2:9 Understanding user needs and effectively communicating the value of uncertainty information for addressing those needs are perhaps the largest and most important tasks for the Enterprise. Yet, forecast information is often provided without full understanding of user needs or how to develop products that best support user decisions. Parts of the Enterprise (e.g., within the private sector and academia) have developed a sophisticated understanding of user needs. In addition, there is a wealth of relevant knowledge in the social and behavioral sciences that could be more effectively incorporated into product research and development. Currently, this variety of resources is not being fully tapped by
6 |
Forecast verification is the means by which the quality of forecasts is assessed. Forecast post-processing converts model output into human-comprehensible information and corrects for model biases. |
7 |
See http://www.nws.noaa.gov/partnershippolicy/ and also Box 1.2. |
8 |
See Section 1.5 for further discussion on this topic. |
9 |
See Sections 2.4, 4.2.6, and 4.2.7. |
NOAA,10 and user perspectives are not incorporated from the outset of the product development process.
Recommendation 2: NOAA should improve its product development process by collaborating with users and partners in the Enterprise from the outset and engaging and using social and behavioral science expertise.
Education on Uncertainty and Risk Communication
Finding 3:11 Enhanced Enterprise-wide educational initiatives will underpin efforts to improve communication and use of uncertainty information. There are three critical areas of focus: (1) undergraduate and graduate education, (2) recurrent forecaster training, and (3) user outreach and education.
Recommendation 3: All sectors and professional organizations of the Enterprise should cooperate in educational initiatives that will improve communication and use of uncertainty information. In particular, (1) hydrometeorological curricula should include understanding and communication of risk and uncertainty; (2) ongoing training of forecasters should expose them to the latest tools in these areas; and (3) forecast providers should help users, especially members of the public, understand the value of uncertainty information and work with users to help them effectively incorporate this information into their decisions.
Ensembles
Finding 4:12 The ability of NOAA to distribute and communicate uncertainty information is predicated on the capacity to produce post-processed probabilistic model guidance on a variety of spatial scales. Currently, NOAA maintains long-range (global) and short-range ensemble13 prediction systems. However, the short-range system undergoes no post-processing and uses an ensemble generation method (breeding) that may not be appropriate for short-range prediction. In addition, the short-range model has insufficient resolution to generate useful uncertainty information at the regional level. For forecasts at all scales, comprehensive post-processing is needed to produce reliable (or calibrated) uncertainty information.
Recommendation 4: NOAA should develop and maintain the ability to produce objective uncertainty information from the global to the regional scale.
Ensuring Widespread Availability of Uncertainty Information
Finding 5:14 NWS, through the National Centers for Environmental Prediction (NCEP), produces a large amount of model output from its deterministic and ensemble numerical weather prediction models. The ensemble forecasts and output from statistical post-processing (i.e., Model Output Statistics) already produce a wide variety of uncertainty information. However, both the model output and statistical information regarding its skill15 are difficult to access from outside NCEP. Thus, NWS is missing an opportunity to provide the underlying datasets that can drive improved uncertainty estimation and communication across the Enterprise.
Recommendation 5: To ensure widespread use of uncertainty information, NWS should make all raw and post-processed probabilistic products easily accessible to the Enterprise at full spatial and temporal resolution. Sufficient computer and communications resources should be acquired to ensure effective access by external users and NWS personnel.
Broad Access to Comprehensive Verification Information
Finding 6:16 To make effective use of uncertainty products, users need complete forecast verification information that measures all relevant aspects of forecast performance. In addition, comprehensive verification information is needed to improve forecasting systems. Such information includes previous numerical forecasts, observations, post-processed uncertainty information, and detailed verification statistics (for raw and post-processed probabilistic forecasts).
Recommendation 6: NWS should expand verification of its uncertainty products and make this information easily available to all users in near real time. A variety of verification measures and approaches (measuring multiple aspects of forecast quality that are relevant for users) should be used to appropriately represent the complexity and dimensionality of the verification problem. Verification statistics should be computed for meaningful subsets of the forecasts (e.g., by season, region) and should be presented in formats that are understandable by forecast users. Archival verification information on probabilistic forecasts, including model-generated and objectively
10 |
Recognizing that private-sector entities gain a competitive advantage through knowledge of user needs, there is, nonetheless, some opportunity for information sharing that could significantly improve the effectiveness and efficiency of product development. |
11 |
See Section 4.2.8. |
12 |
See Chapter 3. Production of objective uncertainty information is covered in Sections 3.1 through 3.3. |
13 |
An ensemble is a collection of forecasts, each starting from a different initial state. The variations in the resulting forecasts can be used to estimate the uncertainty of the prediction. |
14 |
|
15 |
Skill measures how well a forecast performs relative to a naïve standard of comparison, such as climatology or persistence. |
16 |
See Section 3.5. |
generated forecasts and verifying observations, should be accessible so users can produce their own evaluation of the forecasts.
Effective Use of Testbeds17
Finding 7:18 Testbeds are emerging as a useful mechanism for developing and testing new approaches and methodologies in estimating, communicating, and using uncertainty information.19 The effectiveness of testbeds is limited when all appropriate sectors of the Enterprise are not included.
Recommendation 7: To enhance development of new methods in estimation, communication, and use of forecast uncertainty information throughout the Enterprise, and to foster and maintain collaboration, confidence, and goodwill with Enterprise partners, NOAA should more effectively use testbeds by involving all sectors of the Enterprise.
Enterprise Advisory Committee
Finding 8:20 Only through comprehensive interaction with the Enterprise will NWS be able to move toward effective and widespread estimation and communication of uncertainty information. One mechanism for engaging the entire Enterprise on this and related topics is an independent NWS advisory committee with broad representation. Such a committee is under consideration by NOAA in response to a recommendation in the Fair Weather report (NRC, 2003a).
Recommendation 8: The committee endorses the recommendation by the National Research Council Fair Weather report to establish an independent advisory committee and encourages NOAA to bring its evaluation of the recommendation to a speedy and positive conclusion.
Uncertainty Champion
Finding 9:21 Incorporating uncertainty in forecasts will require not only the attention but also the advocacy of NWS management. Given the scope of this challenge, the level of effort involved will demand a “champion” within the NWS leadership—an individual who can effectively organize and motivate NWS resources and engage the resources and expertise of the entire Enterprise.
Recommendation 9: NWS should dedicate executive attention to coordinating the estimation and communication of uncertainty information within NWS and with Enterprise partners.
17 |
Testbeds are multipartner collaborations that create prototypical environments where innovative approaches can be tested before being applied more generally. They allow the community to evaluate new modes of cooperative research, development, training, and operations. |
18 |
|
19 |
For example, Joint Hurricane Testbed (http://www.nhc.noaa.gov/jht/), WRF Developmental Testbed Center (http://www.dtcenter.org/index.php), NOAA Climate Testbed (http://www.cpc.ncep.noaa.gov/products/ctb/), NOAA Hydrometeorology Testbed Program (http://hmt.noaa.gov/). |
20 |
See Section 4.2.6, overarching recommendation 2, and Chapter 5. |
21 |
See Section 1.5 and Chapter 5. |