. "3 Estimating and Validating Uncertainty." Completing the Forecast: Characterizing and Communicating Uncertainty for Better Decisions Using Weather and Climate Forecasts. Washington, DC: The National Academies Press, 2006.
The following HTML text is provided to enhance online
readability. Many aspects of typography translate only awkwardly to HTML.
Please use the page image
as the authoritative form to ensure accuracy.
Completing the Forecast: Characterizing and Communicating Uncertainty for Better Decisions Using Weather and Climate Forecasts
Diagnostic verification information can be considered base-level forecast uncertainty information. For example, a straightforward way for NWS to provide forecast uncertainty information is to augment existing forecast products with the error bars implied by historical verification statistics. This approach has been used effectively by the National Hurricane Center37 (Figure 1.6). The radius of the uncertainty circle at each lead is the average historical track error over the observational record. This product has (1) provided useful information to the public and the Enterprise, (2) stimulated research to provide improved uncertainty information, and (3) generated debate about what information the public and emergency managers require. A more sophisticated approach that utilizes ensemble information is given by the “dressing” technique described by Roulston and Smith (2003).
Finally, many of the issues considered in Chapter 4 regarding the communication of uncertainty information are relevant for the communication of verification information. Thus, careful consideration is needed for the way this information is presented to users. In addition to the verification measures and approaches considered here, it also is important that the components that went into the verification be made readily available to all forecast users (see Chapter 5, recommendation 6) to allow specialist users to perform their own post-processing and verification.
Finding: Verification drives forecast system development and affects the use of forecast information. By focusing on providing meaningful information to users about forecast quality and by being more explicit about its choice of verification measures for the forecast development process, NWS will enable open Enterprise debate about the choice of verification measures and the implied NWS role and values. Such debate will allow user interests to directly influence the development of NWS forecasting systems. Application of a broad set of diagnostic approaches, including new approaches developed through verification research, and incorporation of statistical standards (e.g., stratification into meaningful subsets, use of confidence intervals, comparison to a naïve standard) will allow the provision of information that is needed by a broad spectrum of users.
Recommendation 3.15: NWS should expand its verification systems for ensemble and other forecasts and makemore explicit its choice of verification measures andrationale for those choices. Diagnostic and new verification approaches should be employed, and the verificationshould incorporate statistical standards such as stratification into homogeneous subgroups and estimation ofuncertainty in verification measures. Verification information should be kept up to date and be easily accessiblethrough the Web.
In spite of the variety of time and space scales, the differences in quality of numerical models, the range of different forcings, and the assortment of phenomena under consideration, four themes emerge relating to estimation and validation of uncertainty of weather, climate, and hydrologic forecasts within NWS.
There is a need for the production of guidance databases that include raw and post-processed probabilistic information that can be interrogated by all users of hydrometeorological information, including NWS forecasters, the private sector, and members of the public. There is also a strong need for the construction and maintenance of databases of historical forecasts and the associated observations for the purpose of post-processing and verification.
Before such a database can be usefully constructed, improvements are needed in post-processing efforts for the production of objective probabilistic guidance for all parts of NWS.
An increased emphasis on verification is needed across all parts of NWS. A wide range of verification measures that are appropriately applied with a valid statistical basis are necessary to properly assess forecasts and provide meaningful information to users. In addition, diagnostic verification information provides a simple approach for adding uncertainty information to forecasts. Because the choice of verification drives forecast system development, verification measures should be carefully chosen.
The Enterprise, and in particular the academic community, is a vast resource that is underutilized by NWS. Testbeds are one way in which productive links can be forged among NWS, the academic and private-sector communities, and the users they serve, but only if sufficient emphasis is given and NWS buys into the testbed concept.