Recommendations 3.1, 3.2, and 3.3—Incorporate Quality-of-Evidence Measures, Represent Risk Severity Separately from the State of the Mitigation Strategy or Countermeasure, and Use Standard Uncertainty Analysis Techniques to Quantify Risk Uncertainty
The committee recommends that NASA (1) determine, and incorporate into the BR, measures of the quality of the evidence that form the basis for defining risks and the assessments associated with each risk; (2) structure the BR to represent separately the severity and likelihood of each risk and the state of the mitigation strategy or countermeasures associated with each risk; and (3) whenever possible, restructure the BR to include a quantification of the uncertainty of risks using standard uncertainty analysis techniques (e.g., frequentist, Bayesian, or possibility theory and approximate reasoning) that will provide uncertainty distributions or ranges in addition to point estimates. This will help contribute to the subsequent definition of operating bands.
The way in which risk-related information is represented in the BR and communicated to users is important to its overall effectiveness as a program management tool. One widely used format for representing risks not currently incorporated into the BR is the NASA-wide Continuous Risk Management Program, and the committee encourages its continued use because it is widely recognized and understood throughout NASA. The committee believes that the BR is better thought of and designed as a dynamic database of information relative to risk definition and assessment, from which a document or set of alternative documents can be derived at any time and incorporated into a risk management program. The web-based on-line version of the BR is an important step in this direction.
It is fundamentally important that configuration control methods be established and implemented to keep the BR up-to-date as new knowledge and technologies develop. This process can be facilitated by identifying an “owner and manager” within NASA for each set of related BR risks and establishing a regular review cycle that should occur not less than once a year. Where there is a desire to combine published research data with “expert opinion” from stakeholders, methods such as Bayesian updating and elicitation of expert opinion are available.