2
Major Overarching Comments
Characterizing scientific uncertainty about climate change, effectively communicating that uncertainty to decision makers, and incorporating it in the decision making process are important tasks. The climate research community recognizes and understands the importance of characterizing uncertainty in research and assessment efforts, and decision-makers can benefit greatly from improved communication of uncertainty by the research and assessment communities. Thus the Climate Change Science Program’s (CCSP) Synthesis and Assessment Product (SAP) 5.2 will potentially be very beneficial to all stakeholders of climate change science. The committee commends CCSP and the National Oceanic and Atmospheric Administration (NOAA) for emphasizing the need to address this important topic.
This chapter provides an enumerated discussion of the major issues that, from the point of view of the review committee, the authors should strongly consider addressing in the revised version of SAP 5.2. In some cases, findings are simply noted without explicit recommendations. In other cases, the committee provides either a direct recommendation or alternatives for the authors to consider as they address the review findings. In subsequent chapters of this report, the committee provides further overarching thoughts on the draft document and then findings and recommendations specific to individual chapters of the draft. The major overarching comments follow.
-
The draft provides a good treatment of cognitive challenges and expert elicitation issues. The committee finds that the draft SAP 5.2 provides a well-written and concise synthesis of some of the key issues regarding the characterization of scientific uncertainty vis-à-vis climate decision making. The draft provides a particularly good synthesis of the issues regarding cognitive challenges on an individual basis to characterizing uncertainty, the potential implications of those challenges for CCSP and other assessment efforts, and a method of characterizing uncertainty when conventional methods are not practical or adequate (expert elicitation).
-
All of the audiences outlined in the prospectus are not addressed. The committee finds that the draft is written largely for an audience of those people involved in assessment efforts. Indeed, that is one key audiences for SAP 5.2. The intended audiences as outlined in the prospectus also include those people engaged in scientific research, the media, policymakers, and members of the public. The committee suggests that the authors might also acknowledge and discuss the unique needs of the private sector, which are quite different from those of researchers, academia, and the assessments
-
community. The private sector and policy and decision-makers in the public sector (e.g., congressional staff) fall into the category of users of assessments and need to understand the implications of uncertainty, in contrast to the research science community, who are generators of assessments and associated uncertainty information. The draft provides relatively little information for an audience of users, particularly information that could be used as guidelines for effective communication techniques.
-
The range of best practices for characterizing uncertainty is not represented. The document fails to review the range of methods used to characterize uncertainty as called for in the study prospectus. Instead, it focuses almost exclusively on expert elicitation for use in a subjective Bayesian analysis. This focus neglects assessments, including other Synthesis and Assessment Products, associated with the observational record. There is a need to discuss more traditional frequentist methods, which remain dominant in scientific work, and objective Bayesian methods based on non-informative prior distributions. By focusing exclusively on the subjective Bayesian approach, the document also fails to elucidate ‘Best Practices’ for characterizing uncertainty as called for in the study prospectus. The committee understands this elucidation to involve a description of alternative approaches and a discussion of the strengths and weaknesses of each. The addition of a statistician to assist with the elucidation of traditional scientific methods would address a significant weakness in the report. This addition to the authorship team should be strongly considered, regardless of whether the current document (and its authorship team) is greatly expanded to address the additional audiences and issues described in the prospectus.
-
The influence of social context and emotional factors is absent. Although some of the important cognitive factors in understanding and evaluating uncertainty are discussed in Chapter 3, this discussion is incomplete in two senses. First, the draft SAP 5.2 neglects the social context in which such understandings and evaluations are made; even within the narrow focus of discussing expert elicitation, responses will be influenced by how the questions are asked, the context of the interview, the expectations and knowledge of experts about what their peers are saying, and the cultural set of norms that attend the social groups (scientific institutions, universities and departments, etc.) to which respondents belong. Moreover, emotions have been shown to play scientifically measurable roles in estimations of uncertainty. Second, the discussion of cognitive biases, and the missing discussion of the social context and emotive factors in evaluating uncertainty are highly relevant to the communications and decision making chapters but are almost entirely absent in these too-brief chapters. In the communications chapter, the emphasis is on presentation of materials to an amorphous non-technical audience rather than on understanding the needs of multiple audiences within their social contexts, as specified in the prospectus. The decision making chapter neglects the large literatures on decision making in institutional settings where scientific uncertainty is only one of many factors influencing decisions.
-
Introductory material is lacking. The draft would be improved if the brief paragraph at the top of page 1 was expanded into a formal introduction section that provides framing and context for the rest of the document. The authors could define who climate decision-makers are and discuss the importance of characterizing,
-
communicating, and incorporating scientific uncertainty for the decision making process. This could be done using an example of decision making under uncertainty, where the optimal decision depends on the characterization of uncertainty, and also an example of scientific inference, where uncertainty needs to be characterized to go beyond point estimation (i.e., to test an hypothesis or provide a confidence interval)." In its current form, the transition to technical material is far too abrupt. The introduction section could also outline the charge to the authors as they perceived it, and clearly define the goals and objectives of the document. As an alternative, this material could be included in a foreword. The foreword or introduction could also state explicitly what the document does not address vis-à-vis the broader contexts of scientific uncertainty and climate decision making.
-
An executive summary is essential but has not been prepared. The committee finds that the lack of an executive summary hinders its accessibility to the audiences named in the prospectus. A concise and readable summary of the document, including key findings and recommendations, would enable all audiences -- producers of synthesis and assessment products, scientific researchers, decision-makers, media, and the general public -- to glean the main points and to locate further information that may be of interest to them. The summary should not be descriptive, but informative on the main points of the document.