Considerations Regarding the BR Process
Several issues contribute to the complexity of the risks that the Bioastronautics Roadmap (BR) addresses, including the number of identified risks, the heterogeneity of risk types, and the interdependence among risks. In addition to the challenge of complexity, the determination of risk regarding activities for which there is little or no operational experience always involves an element of uncertainty, and the degree of uncertainty should be incorporated into the BR if it is to be a useful tool for decision making. Once identified and characterized, the method by which risk-related information is stored and made available to users also affects the overall effectiveness of the BR. This chapter reviews and comments on the methods used to construct the BR and on the format and approach used to represent and communicate its content.
As noted in the introduction to this report, efforts to understand and manage the risks associated with human space flight have been ongoing at the National Aeronautics and Space Administration (NASA) for many years, and specific activities related to the development of a roadmap began in the early 1990s. The process of risk identification that resulted in the BR commenced in 1997, in brainstorming sessions involving NASA, the National Space Biomedical Research Institute (NSBRI) staff and collaborators, and non-NASA experts who rated risks within their own discipline
areas. With guidance from National Academies’ (see Appendix A) and other advisory reports, 150 risks were identified. More recently, and after several iterations, the list was culled to the 45 risks that are the focus of the current BR (http://bioastroroadmap.nasa.gov).
The current set of risks and related research and technology questions were identified through an iterative process that included input from the discipline teams, the Bioastronautics Science Management Team, the Chief Health and Medical Officer, the Astronaut Office, flight surgeons, and NASA research management.
In the characterization of risks contained in the BR, risk assessment criteria included the determination of the likelihood of occurrence; the severity of consequences of each risk in terms of the crew’s health, safety, and ability to perform mission objectives; and the state of the mitigation strategy. Relative risk priorities were derived from that assessment. Each risk has a set of associated research and technology questions. The answers to these questions are intended to lead to (1) risk assessment and quantification, (2) the development of countermeasures to prevent or mitigate the deleterious effects of space flight, (3) an improved basic understanding of underlying processes, and (4) medical diagnostic and treatment capabilities. This risk-based approach was devised to enable the development of a more rigorous decision-making process for allocation and implementation of resources, risk prioritization, access to facilities, operational requirement implementation, and crew time, as well as for development of cost-effective countermeasures and the design and implementation of effective advanced life support technology.
The committee agrees that NASA’s decision to draw on expert opinion in identifying and ranking risks is a reasonable strategy, given the broad array of topics addressed in the BR and the need, in some cases, to characterize risks by extrapolation from current experience. However, the committee believes there are weaknesses in the current risk assessment process related to (1) lack of information regarding the quality of evidence that informs risk assessment, (2) the obscuring of risk that results from “lumping” both the risk and its associated mitigation into a single value, and (3) lack of a quantitative measurement of uncertainty related to the risk. These areas can, and should, be enhanced, and they are discussed in the sections leading to Recommendations 3.1, 3.2, and 3.3. These recommendations flow from an understanding that expert opinion in health care and the life sciences is influenced both by systematically derived data and by heuristics, or “rules of thumb,” that are derived from personal and group experience
(McDonald, 1996) and an awareness that the risks in the BR range from theoretical concerns (e.g., virus-induced lymphomas and leukemia) to practical issues (e.g., nutrition, motion sickness, and bone and muscle loss). In addition, some risks are specific (e.g., renal stone formation), whereas others are general (e.g., ambulatory care).
The committee believes that risk assessment should primarily be evidence-based wherever possible, and where evidence does not exist, research should be directed to acquiring the evidence needed (see example on bone fracture risk associated with prolonged exposure to microgravity, Appendix D). Rating the quality of each published source of information should be a component of the BR. Quality ratings should follow commonly used criteria such as those of the Cochrane Collaboration (Starr and Chalmers, 2005) or the Agency for Healthcare Research and Quality (AHRQ, 2002) National Guideline Clearinghouse, recognizing that quality rating methods based only on the published literature will have to be modified to accommodate other sources such as conclusions from workshops and planning meetings. For risks that have more than one information source, the quality of the most robust source should be noted.
The committee recommends that NASA determine, and incorporate into the BR, measures of the quality of the evidence that forms the basis for defining risks and the assessments associated with each risk.
Disaggregation of Risk from Mitigation of Risk
Currently, the status of risks is represented in the BR as the aggregation of the importance of the risk (probability of occurrence × severity of consequences) combined with the effectiveness of the current state of counter-measures. In this model, a “severe” risk to the crew—such as loss of breathable atmosphere—would be represented as being minimal if the state of life support technology readiness or life support capability readiness to mitigate the risk was high. The committee notes that it is important to explicitly and separately assess and represent the importance of each risk and the state of the mitigation strategy or countermeasures that address the risk.
Disaggregation of mitigations is important for several reasons. First, it is important because aggregated risk values obscure the risk itself and can lead to false confidence or concern, depending on the status of counter-
measure or technology development. Second, the functioning of many parts of the space flight system is closely linked to the functioning of other parts of the system, and approaches that are used to mitigate one risk may have a positive or negative impact on other risks or mitigations. Finally, as the space flight system evolves, changes in systems may have a positive or negative impact on the mitigations. Failure to track risk separately from risk mitigation could well lead to failure to focus on the inherent relationships among risks and mitigations. For example, the risk of high “g’s” for a fighter pilot can be mitigated adequately by a “g-suit” system that is inflated by air pressure, and disaggregation of the mitigation from the risk helps ensure that system designers are aware that any changes that may affect the ability of the system to inflate the g-suit could affect the mitigation of the risk of g-induced incapacity. The committee observes that this disaggregation will also help maintain a clear understanding that the notion of “retiring risks” due to the availability of effective countermeasures is seldom an accurate depiction of the state of operational readiness, since the underlying health and safety issues (e.g., loss of breathable atmosphere) remain a concern and are not “retired” by the existence of a life support system.
The committee recommends that NASA structure the BR to represent separately the severity and likelihood of each risk and the state of the mitigation strategy or countermeasures associated with each risk.
The determination of risk always involves an element of uncertainty. To fully communicate the likelihood of occurrence of an event, it is necessary to communicate the extent of uncertainty in the assessment. The uncertainty associated with a risk may be represented by objective measures such as statistical confidence intervals and sensitivity analyses (NCRP, 1996, 1997; Warren-Hicks and Moore, 1998; Grogan et al., 2000; NCI– CDC, 2003) or by less objective but potentially useful techniques such as approximate reasoning (Hayes et al., 1979) or possibility theory using fuzzy sets (Zadeh, 1978). It is also possible to state uncertainty using narrative descriptions of the risk, such as expert opinion obtained in focus group settings (Cacuci, 1988; Lash and Silliman, 2000). The current printed and on-line versions of the BR do not include any expression of uncer-
tainty in terms of risk estimates, reported confidence intervals, or narrative discussion.
Variability of Opinion About BR Risks
Heuristic solutions result from an informed set of principles or rules that an individual or group uses for decision making. Although such rules may rely on an empirically derived knowledge base, it is important to understand that experience, judgment, personal philosophies, and external pressure can affect a heuristic, causing different groups to reach different conclusions. Given that it is formulated from a set of risks derived from discussions among different teams of disciplinary experts, the BR is not immune to such effects. However, ongoing discussions among the Bioastronautics Science Management Team, the Office of the Chief Health and Medical Officer, the Astronaut Office, flight surgeons, and research management have led to views of the risks of space flight that are generally similar, but not identical, within and outside NASA.
For example, the Institute of Medicine (IOM, 2001) panel that authored Safe Passage: Astronaut Care for Exploration Missions selected four risks that earned the panel’s highest rating of “severe” for a flight to Mars:
Trauma and acute medical problems
Carcinogenesis caused by radiation
Human performance failure because of poor psychosocial adaptation
Acceleration of age-related osteoporosis
The version of the BR originally supplied to the IOM for review (NASA, 2004) used similar language, listing the most serious risks for a Mars mission as the following:
Addressing the requirements for autonomous medical care
Providing radiation protection
Maintaining behavioral health and performance
Bone loss–related issues
Advanced human support technology
Flight surgeons and astronauts provided the BR review panel with a narrower, operationally focused set of priorities for exploration (Baker et
al., 2004). Analogous sentiments are echoed in the most recent version of the Bioastronautics Roadmap (NASA, 2005), which states that “actual risks must be operationally based, not research based.” Flight personnel identified four high-priority items:
Medical care diagnosis and treatment
Behavioral health and performance
Bone loss was considered a lesser, but still important, item. The difference between this highest-priority list and preceding ones is that flight personnel place higher value on the ability of an astronaut to pilot a spacecraft during nominal and/or contingency operations. At the same time, they expressed the belief that successful countermeasures for musculoskeletal deconditioning are imminent.
An excellent example of the different values applied to the BR comes from the results of a consensus workshop held in May 2004 in Houston. Representatives of the Astronaut Office, Space Medicine Office, and the Bioastronautics Science Management Team (BSMT) met to review the BR and assess its suitability for reducing the risk of human space travel to Mars. Participants were asked to rate their concern about the subset of BR human risks, the need for future research in each area, and modifications to any of the risk set. Figure 3.1 summarizes data supplied to the committee.
Although the median “worriness” rating for the high risk category in the BR exceeded that of the medium risk category (median values 6.0 vs. 2.5; p = 0.015), the considerable overlap of “worriness” scores among the high, medium, and low risk BR categories (dark, medium and light gray dots, respectively in Figure 3-1) suggests that different heuristics to evaluate risk for human space exploration operate within NASA. Those communities most directly affected by the risk of a human space flight to Mars (i.e., astronauts and flight physicians) did not share all concerns raised by the disciplinary experts responsible for developing the BR, and in several cases felt that some risks did not belong in the Bioastronautics Roadmap (i.e., “worriness” score of 0).
The committee recommends that whenever possible, NASA restructure the BR to include a quantification of the uncertainty
of risks using standard uncertainty analysis techniques (e.g., frequentist, Bayesian, or possibility theory and approximate reasoning) that will provide uncertainty distributions or ranges in addition to point estimates. This will help contribute to the subsequent definition of operating bands.
The operating bands and exposure limits coming from the BR can be used as the basis for integrating bioastronautics risks into full-scope risk analyses for each of the reference missions. These larger mission risk analyses should be scenario-based, quantitative risk analyses. This approach would allow the evaluation and prioritization of all types of risk within a common framework.
DETERMINATION OF ACCEPTABLE RISK
The committee believes that the BR can be a tool to support both identification and management of risks such that each risk reaches an “acceptable” status for the relevant mission by the intended launch time for that mission. Several potentially useful risk management systems currently exist (e.g., NASA Continuous Risk Management Program [NASA, 1999], U.S. Navy Virginia Class Submarine [Kulez, 2003]). The committee does not recommend a specific implementation system but observes that a roadmap that supports risk management will have to contain elements that support operations in addition to those that point to needs for further research.
In this regard, although research in most fields may continue ad infinitum, the BR should attempt to identify “what is good enough” for the launch of a given category of mission. Researchers in virtually all fields are reluctant to declare total success, since this would be tantamount to forfeiting future funding. In the conduct of exploration, leaders cannot wait until every detail is resolved definitively, but only until the collective risk is mitigated adequately or otherwise reduced to permit a high enough level of optimism to justify mission initiation. This by no means suggests that research in a field should be terminated when sufficient progress has been made for launch, only that the mission should be “cleared” and further research dissociated from the operational aspects of the mission.
The presentation of information relative to risk management will have to be linked to the Design Reference Missions. The three different missions under consideration—1 year aboard the International Space Station (ISS), 1 month on the lunar surface, and a 30-month mission to Mars and back—will have different detailed risk management requirements in each risk category. A risk currently considered unacceptable for the Mars mission may very well be acceptable for the ISS mission. The fact that four Russian individuals (Titov, Manarov, Polyakov, and Avdeyev) have already spent a continuous year or more each in low Earth orbit suggests that the ISS mission category should be rephrased to either (1) identify additional detailed (scientific) mission objectives over and above mere survival for a year in orbit or (2) call for qualifying humans for routine and/or repetitive mission durations of 1 year in the ISS.
The manner in which risk-related information in the BR is represented and communicated to users is important to its overall effectiveness as a program management tool. The committee has identified issues of representation and communication that would help support the full range of BR stakeholders, notably NASA medical operations personnel and astronauts, in addition to NASA-funded researchers and contractors engaged in countermeasure development. Central among these issues is that the BR has been presented as a document for review. The committee believes that the BR should be thought of and designed as a dynamic database of information relative to risk definition and assessment from which a document or set of alternative documents can be derived at any point in time and incorporated into a risk management program. Databases lend themselves to the creation of multiple, different views of subsets of information in the database, and this capability parallels the need for many different groups of stakeholders to view the overall BR from different perspectives and at different levels of detail. Viewing and structuring the BR as a database will also facilitate keeping it current as new knowledge and new technologies emerge. The web-based on-line version of the BR—with its searchable interface and alternative views for aggregating BR information—is an important step in this direction.
The current BR contains for each risk a section titled “Important References,” and many of these include hypertext links to the citation with abstract. It is not possible, however, to determine which references are linked to the specific elements of the BR structure (e.g., risk description, risk rating, current countermeasures, research and technology questions). The availability of links to citations is helpful and would be enhanced by links to the full text of publications, wherever feasible and within the constraints of copyright law.
A variety of alternative methods for scoring and communicating risk information may help with communication to different BR user communities. One widely used format for representing risks not currently incorporated into the BR is the NASA-wide Continuous Risk Management Program (NASA, 1999), and the committee encourages its continued use because it is widely recognized and understood throughout NASA. NASA developed the Continuous Risk Management Program in 1996 to help project managers continuously identify, analyze, and manage risk throughout the life cycle of a project and for use as a proactive tool for managers to monitor resource allocation and ensure that critical project milestones are
achieved within an acceptable level of risk. The use of the Continuous Risk Management Program results in a set of actionable risks that can be assessed with regard to the probability and consequences of occurrence. This information could be used to plan mitigation measures indicating that all risks have been reduced to acceptable levels by the projected launch date, to inform cost–benefit analyses and prioritization efforts, and to help NASA obtain adequate resources (funding, time, expertise) to carry out these measures. Representation of BR risks in this format, in addition to the current formats, may be an effective supplemental way of communicating the elements of the BR throughout the organization.
To enhance effective communication of the content of the BR, the committee recommends that the BR be designed and utilized as a dynamic database of information relative to risk definition and assessment, from which a document or set of alternative documents can be derived at any time and incorporated into a risk management program.
KEEPING THE BR CURRENT
The period of time over which the Design Reference Missions will be planned and executed is decades. Thus, it is fundamentally important that configuration control methods be established and implemented for keeping the BR up-to-date as new knowledge and technologies develop. In this regard, the committee observed that new literature relative to risks in the BR that became available over the course of committee deliberations has not yet been incorporated into the BR. A mechanism is needed for periodic searches of the literature for information related to risks—including research conducted in space or in analog environments—as well as literature on the status of validation of existing countermeasures. This process can be facilitated by identifying an “owner and manager” within NASA for each set of related BR risks and establishing a regular review cycle. Given the rate of publication of new literature, it seems prudent to conduct reviews for updating not less than once annually.
Where there is a desire to combine published research data with “expert opinion” from stakeholders, methods such as computer modeling (White et al., 2003), Bayesian updating, and elicitation of expert opinion are available (see Appendix E for a description of the Bayesian update pro-
cess). When relevant, digital modeling and simulation should be used to integrate and extrapolate findings. Regardless of the combination of research data and expert opinion, the long time frame of the space initiative makes it imperative that new knowledge and technologies be incorporated into the BR.
The committee recommends that the references cited in the BR be updated to reflect more recent research on both risk identification and countermeasure development. Moreover, a mechanism should be established for the ongoing review of the current best evidence contained in the research literature, and methods should be developed to integrate new findings from the literature with the expert opinion of key stakeholders, including those from operations and the research community.
INDEPENDENT HEALTH AND MEDICAL AUTHORITY
The Columbia Accident Investigation Board’s (CAIB’s) report recommended the establishment of an Independent Technical Engineering Authority “that is responsible for technical requirements and all waivers to them” in order to “build a disciplined, systematic approach to identifying, analyzing, and controlling hazards through the life cycle of the Shuttle System” (CAIB, 2003). The Office of the Chief Engineer has now been established as the NASA Independent Technical Authority (ITA), whose mission includes the following: developing and maintaining technical standards; serving as the sole waiver-granting authority for technical standards; conducting risk analyses; serving as owner of the failure mode, effects analysis, and hazard reporting systems; deciding what is and is not an anomalous event; and independently verifying launch readiness.
The committee finds that the decisions that will have to be made with respect to health and human safety aspects of the Design Reference Missions are similar in complexity to those relating to the Shuttle as a system of systems. Additionally, there may be a number of human health issues for which there are not sufficient data to perform a definitive risk analysis or perform conclusive testing or simulation prior to the mission, and there may be a need to accept a substantial amount of unmitigated risk to the crew. It seems reasonable to conclude that establishment of an Independent Health and Medical Authority (IHMA) with a scope of authority and re-
sponsibility analogous to the ITA, but focused on matters of crew health, safety, and effectiveness, is an appropriate management construct within the agency. Just as the ITA is the owner of technical standards, risk analysis, waiver granting, and verifying launch readiness for systems, an IHMA would do the same for human health and medical care issues. The involvement of the IHMA in assessing the ongoing activities of the BR—especially as they change and evolve over time—would foster a “closed-loop” approach for monitoring progress in risk reduction in each of the BR risk areas. The IHMA could delegate authority appropriately through a system of warrant holders such as that used by the ITA, with cooperation between discipline experts in the development and maintenance of standards, especially in areas of potential overlap such as environmental monitoring and human factors. Warrant holders would be expected to coordinate closely with each other when deviances to standards are considered.
The committee endorses the principle that critical decisions regarding safety and health should be made by an authority that is independent of programmatic costs and schedules. Given the importance and complexity of health and human safety issues, the committee acknowledges and endorses the creation of the Independent Health and Medical Authority (IMHA), analogous to the Independent Technical Authority, and recommends that the IHMA be given responsibility, authority, and accountability for the health and safety decisions that relate to risks identified in the BR.
AHRQ (Agency for Healthcare Research and Quality). 2002. Systems to Rate the Strength of Scientific Evidence. Summary, Evidence Report/Technology Assessment: Number 47. AHRQ Publication No. 02-E015. Rockville, MD: Agency for Healthcare Research and Quality. On-line [available: http://www.ahrq.gov/clinic/epcsums/strengthsum.htm]. Accessed 7/25/05.
Baker E, Barratt M, Duncan M, Clark J, Cintron N. 2004. Operational Priorities for Medical Research for Exploration. Presentation to committee, NASA/Johnson Space Center, Houston, TX, November 17.
Cacuci DG. 1988. The forward and the adjoint methods of sensitivity analysis. Ch. 3 in Ronen Y (ed.) Uncertainty Analysis, Boca Raton, FL: CRC Press.
CAIB (Columbia Accident Investigation Board). 2003. Columbia Accident Investigation Board Report. On-line [available: http://www.caib.us]. Accessed 4/18/05.
Grogan HA, Sinclair WK, Voilleque PG. 2000. Assessing Risks from Exposure to Plutonium. Final Report. Part of Task 3: Independent Analysis of Exposure, Dose and Health Risk to Offsite Individuals. Radiological Assessment Corporation (RAC) Report No. 5, Revision 2, February. Neeses, SC.
Hayes J, Michie D, Mikulich LI (eds.). 1979. A Theory of Approximate Reasoning, Machine Intelligence. New York: Halstead Press.
IOM (Institute of Medicine). 2001. Safe Passage: Astronaut Care for Exploration Missions. Board on Health Science Policy, Institute of Medicine. Washington, DC: National Academy Press.
Kulez J. (September 4-5, 2003). Risk Integration Approaches and Results for the Virginia Class Submarine. [Online]. Available: http://www.atc.nasa.gov/hostedEvents/rmc4/presentations/Day%201%209-4-03/3%20pm%20Kulesz.ppt#5 [Accessed January 10, 2006].
Lash TL, Silliman RA. 2000. A sensitivity analysis to separate bias due to confounding from bias due to predicting misclassification by a variable that does both. Epidemiology 11(5): 544-549.
McDonald CJ. 1996. Medical heuristics: the silent adjudicators of clinical practice. Ann. Intern. Med. 124(1 Pt 1): 56-62.
NASA (National Aeronautics and Space Administration). 1999. NASA Continuous Risk Management Program. NASA. Online [available: http://satc.gsfc.nasa.gov/support/ASM_FEB99/crm_at_nasa.html]. Accessed 4/30/05.
NASA. 2004. Bioastronautics Critical Path Roadmap—An approach to risk reduction and management for human space flight: extending the boundaries. National Aeronautics and Space Administration. Rev 1 draft, 4/2/2004.
NASA. 2005. Bioastronautics Roadmap—a risk reduction strategy for human space exploration. On-line [available: http://ston.jsc.nasa.gov/collections/TRS/-techrep/Sp-2005-6113.pdf]. Accessed 1/6/2006.
NCRP (National Council on Radiation Protection and Measurements). 1996. A guide for uncertainty analysis in dose and risk assessments related to environmental contamination. NCRP Commentary No. 14. Bethesda, MD: NCRP.
NCRP. 1997. Uncertainties in fatal cancer risk estimates used in radiation protection. NCRP Report No. 126. Bethesda, MD: NCRP.
NCI (National Cancer Institute). 2003. Land CE, Gilbert E, Smith J, Hoffman FO, Apostoiae I, Thomas B, Kocher DM. Report of the NCI–CDC Working Group to Revise the 1985 NIH Radioepidemiological Tables. NIH Publication No. 03-5387. Bethesda, MD: U.S. Department of Health and Human Services, National Institutes of Health, National Cancer Institute.
Starr M, Chalmers I. 2005. The evolution of the Cochrane Library, 1988–2003. Update Software: Oxford. On-line [available: www.update-software.com/history/clibhist.htm]. Accessed 4/30/05.
Warren-Hicks WJ, Moore DRG (eds.). 1998. Uncertainty Analysis in Ecological Risk Assessment. Pensacola, FL: SETAC Press.
White RJ, Bassingthwaighte JB, Charles JB, Kushmerick MJ, Newman DJ. 2003. Issues of exploration: human health and wellbeing during a mission to Mars. Adv. Space Res. 31: 7–16.
Zadeh L. 1978. Fuzzy sets as a basis for a theory of possibility. Fuzzy Sets and Systems 1: 3–28.