Michael Stamatelatos began his comments by noting that risk is a combination of likelihood and severity of consequence, and that types of mission risk include technical (safety, performance) and programmatic (cost, schedule). He asserted that there is no such thing as qualitative risk assessment, because any meaningful qualitative statement about risk has, either explicitly or implicitly, some quantitative basis. Terms like “high”, “medium,” and “low” risk can be interpreted differently by different people in the absence of a quantitative reference. Qualitative risk assessment has value mainly as a simple way of communicating risk results that are obtained quantitatively. He also noted that perceived risk changes with exposure to consequences, even though quantitative risk may not have changed.
Stamatelatos described the use of probabilistic risk assessment at NASA, pointing out that it is done for shuttle upgrades, construction in space (e.g., the ISS), safety compliance issues (e.g., those associated with Prometheus or Mars sample return), and design (e.g., Prometheus). He agreed that there is a need to improve risk awareness within NASA and to develop in-house expertise to understand probabilistic risk requirements, because risk assessment is a decision support tool and, as such, it cannot be effectively used if decision makers do not understand its methods and findings. He also commented on the need for risk awareness and for management decisions to be informed by risk but not be risk-based—that is, they should not rely solely on risk assessment.
In response to a question, Stamatelatos said that his office was involved with the President’s initiative only at a conceptual level, but that once the vision is more concrete, more rigorous risk assessment can be conducted. In response to a question about how important it is to start risk assessment early in program planning, Stamatelatos said that it should start sooner rather than later; however, if you wait until you have all the information, you will not need a risk assessment. Quantitative risk assessment can inform decisions about where to put money and can identify where the largest risks are. Operational risk can also be reduced once a system has been designed and built, since specific components and technologies can be evaluated.
Donna Shirley asked how, for a broad vision such as ASTRA, does losing one technology (or not achieving maturity for it) affect the risk for the entire system? She contended that if a specific amount of risk is decreed at the highest level, program managers would be inspired to suppress information to try to achieve that decreed risk. This is not productive. Barry commented that since as far back as the Apollo program NASA has never adopted a probabilistic risk assessment culture. The CAIB thought that NASA should use probabilistic risk assessment as a tool to inform decisions, but did not think that the agency should be a slave to the process. Barry went on to say that repeated untoward occurrences (e.g., foam falling off the booster rocket, the erosion of the O-rings) should have signaled potential problems. Although some disagree that probabilistic risk assessment would have correctly identified the foam as a risk, the point is that repeated loss of foam should have been a sign to engineers of a potential problem. Panelists and audience members agreed that the management culture and institutional barriers at NASA still need to be addressed.