C

2012 Review and Prioritization Methodology

The steering committee and panels that authored the 2012 report by the National Research Council (NRC), NASA Space Technology Roadmaps and Priorities, used a two-step process to prioritize technologies in NASA’s 2010 draft roadmaps. First, they identified 83 high-priority technologies. The steering committee then examined those 83 technologies in more detail to identify technologies that should be considered to be of the highest priority. This appendix describes the prioritization process using text taken from Chapters 2 and 3 of the 2012 report.

2012 NRC REPORT: PROCESS TO IDENTIFY THE HIGH-PRIORITY TECHNOLOGIES

A set of criteria was established by the steering committee to enable the prioritization of technologies within each and, ultimately, among all of the technology areas of the NASA technology roadmaps. These criteria were chosen to capture the potential benefits, breadth, and risk of the various technologies and were used as a guide by both the panels and the steering committee to determine the final prioritization of the technologies. In addition to the primary criteria used to prioritize the technologies, an additional set of secondary descriptive factors were also assessed for each technology. These descriptive factors were added to provide a complete picture of the panels’ assessments of the technologies and assisted in the evaluations.

Broad community input was solicited through a public website, where more than 240 public comments were received on the draft roadmaps using the established steering committee criteria and other descriptive factors. The public and panels were given the same rubrics to evaluate the technologies so that the various inputs could be more fairly compared against each other. These views, along with those expressed during the public workshops, were taken into account by the panel members as they assessed the technologies. The panels then came to a consensus view for each criterion for each technology.

In evaluating and prioritizing the technologies identified, the steering committee made a distinction between technology development and engineering development. Technology development, which is the intended focus of the draft roadmaps, addresses the process of understanding and evaluating capabilities needed to improve or enable performance advantages over current state-of-the-art space systems. Technologies of interest include both hardware and software, as well as testing and evaluation of hardware (from the component level to the systems level) and software (including design tools) at various levels of technology readiness for application in future space systems. In contrast, engineering development, which generally attempts to implement and apply existing or available technology, is understood for the purposes of this study to be hardware, software, design, test, verification, and

validation of systems in all phases of NASA’s acquisition process. The high-priority technologies do not include items for which engineering development is the next step in advancing capabilities.

Top Technical Challenges

When the 2012 report was prepared, the NASA design reference missions were not available, so as a substitute the panels identified a number of challenges for each technology area that should be addressed for NASA to improve its capability to achieve its mission objectives.¹ These top technical challenges were generated to provide some focus for technology development and to assist in the prioritization of the level 3 technologies. The challenges were developed to identify the general needs NASA has within each technology area, whereas the technologies themselves address how those needs will be met. Once the top technical challenges were identified, the panels then determined the relative importance of the challenges within each technology area to put them in priority order.

Descriptive Factors

The steering committee identified three descriptive factors that helped characterize each technology. Although these factors were not primary in the determination of technology prioritization, they did assist in generating a better understanding of the current status or state of the art of the technology.

• Technology Readiness Level (TRL): This factor describes the current state of advancement of the technology using NASA’s TRL scale.² It was determined that TRL should not be a basis for prioritizing technologies, because NASA should be investing across all levels of technology readiness. In assessing TRL levels, the panels were directed to evaluate the most promising developments that should receive attention. For example, electric propulsion systems are commonly used today, so as a whole, they would be assessed as TRL 9; however, the promising area of advancement of high power electric propulsion is less advanced, and thus 2.2.1 Electric Propulsion was assessed as TRL 3.
• Tipping Point: The tipping point factor was used to determine whether the technology was at a state such that a relatively small additional effort (compared to that which advanced the technology to its current state) could produce a significant advance in technology readiness that would justify increasing the priority associated with this technology.
• NASA Capabilities: This factor captured how NASA research in this technology aligns with the expertise, capabilities, and facilities of NASA and/or other organizations cooperating with NASA in this area. It also indicated how much value NASA research in this technology would add to ongoing research by other organizations. This was not a primary consideration in assessing which technologies should be prioritized. Instead it was a factor in considering whether the technology should be developed by NASA, or whether NASA should support other current efforts. The factor also addressed whether NASA should invest in improving its own capability for pursuing the high-priority technologies.

___________________

¹ Design reference missions in the 2015 NASA roadmaps appear in the first volume of the roadmaps, NASA, 2015, NASA Technology Roadmaps: Introduction, Crosscutting Technologies, and Index, May 2015 Draft, http://www.nasa.gov/offices/oct/home/roadmaps/index.html, accessed June 29, 2016, pp. i-46.

² NASA’s technology readiness levels are as follows:

Evaluation Criteria

The steering committee identified three main criteria on which the technologies were to be judged for evaluation. The three criteria were benefit, alignment with NASA’s goals and objectives, and technical risk and challenge. Each of these is described in further detail below. For the latter two criteria, three further subcriteria were created to assist in evaluating the technologies.

For each evaluated criterion or subcriterion, a set of four (or in one case five) grades or bins were established, and the public and panel members were asked to determine what grade each technology should receive for that criterion. For consistency, a set of definitions were generated for each grade. The grading definitions were provided as guidelines to help the panel and steering committee members assign an appropriate range of grades necessary to prioritize the technologies in question. They were generated such that most technologies would be placed into one of the middle bins, while placement at the upper/lower bounds would need significant justification. The grades were assigned numeric scores on a nonlinear scale (e.g., 0-1-3-9) to accentuate the spread of the summed final scores. Higher numeric scores implied greater ability to meet NASA’s goals. Negative numbers indicated characteristics that were not desirable.

Benefit: Would the technology provide game-changing, transformational capabilities in the timeframe of the study? What other enhancements to existing capabilities could result from development of this technology?

1. The technology is unlikely to result in a significant improvement in performance or reduction in life cycle cost of missions during the next 20 years. Score: 0
2. The technology is likely to result in (a) a minor improvement in mission performance (e.g., less than a 10 percent reduction in system launch mass); (b) a minor improvement in mission life cycle cost; or (c) less than an order of magnitude increase in data or reliability of missions during the next 20 years. Score: 1
3. The technology is likely to result in (a) a major improvement in mission performance (e.g., a 10 percent to 30 percent reduction in mass) or (b) a minor improvement in mission life cycle cost or an order of magnitude increase in data or reliability of missions during the next 20 years. Score: 3
4. The technology is likely to provide game-changing, transformational capabilities that would enable important new projects or missions that are not currently feasible during the next 20 years. Score: 9

Alignment: Three subcriteria were created to evaluate the alignment with NASA’s goals and objectives criterion.

Alignment with NASA Needs: How does NASA research in this technology improve NASA’s ability to meet its long-term needs? For example, which mission areas and which missions listed in the relevant roadmap would directly benefit from development of this technology, and what would be the nature of that impact? What other planned or potential missions would benefit?

1. Technology is not directly applicable to NASA. Score: 0
2. Technology will impact one mission in one of NASA’s mission areas. Score: 1
3. Technology will impact multiple missions in one of NASA’s mission areas. Score: 3
4. Technology will impact multiple missions in multiple NASA mission areas. Score: 9

Alignment with Non-NASA Aerospace Technology Needs: How does NASA research in this technology improve NASA’s ability to address non-NASA aerospace technology needs?

1. Little or no impact on aerospace activities outside of NASA’s specific needs. Score: 0
2. Impact will be limited to niche roles. Score 1
3. Will impact a large subset of aerospace activities outside of NASA’s specific needs (e.g., commercial spacecraft). Score: 3
4. Will have a broad impact across the entire aerospace community. Score: 9

Alignment with Non-Aerospace National Goals: How well does NASA research in this technology improve NASA’s ability to address national goals from broader national perspective (e.g., energy, transportation, health, environmental stewardship, or infrastructure)?

1. Little or no impact outside the aerospace industry. Score: 0
2. Impact will be limited to niche roles. Score: 1
3. Will be useful to a specific community outside aerospace (e.g., medicine). Score: 3
4. Will be widely used outside the aerospace community (e.g., energy generation or storage). Score: 9

Technical Risk and Challenge: Three subcriteria were created to evaluate the technical risk and challenge criterion. In this criterion, the grades created were not as straightforward as those for benefit and alignment. They were developed to capture the steering committee’s view on the appropriate risk posture for NASA technology developments.

Technical Risk and Reasonableness: What is the overall nature of the technical risk and/or the reasonableness that this technology development can succeed in the timeframe envisioned? Is the level of risk sufficiently low that industry could be expected to complete development of this technology without a dedicated NASA research effort, or is it already available for commercial or military applications? Regarding the expected level of effort and timeframe for technology development: (a) are they believable given the complexity of the technology and the technical challenges to be overcome; and (b) are they reasonable given the envisioned benefit vis-à-vis possible alternate technologies?

1. The technical risk associated with development of this technology is very low, such that it is feasible for industry or a specific NASA mission office to complete development (without additional NASA technology funding if a mission need arises). Score: 1
2. The technical risk associated with development of this technology is low, and the likely cost to NASA and the timeframe to complete technology development are not expected to substantially exceed those of past efforts to develop comparable technologies. Score: 3
3. The technical risk associated with development of this technology is moderate to high, which is a good fit to NASA’s level of risk tolerance for technology development, but the likely cost to NASA and the timeframe to complete technology development are expected to substantially exceed those of past efforts to develop comparable technologies. Score: 3
4. The technical risk associated with development of this technology is moderate to high, which is a good fit to NASA’s level of risk tolerance for technology development, and the likely cost to NASA and the timeframe to complete technology development are not expected to substantially exceed those of past efforts to develop comparable technologies. Score: 9
5. The technical risk associated with development of this technology is extremely high, such that it is unreasonable to expect any operational benefits over the next 20 years without unforeseen revolutionary breakthroughs and/or an extraordinary level of effort. Score: 1

Sequencing and Timing: Is the proposed timing of the development of this technology appropriate relative to when it will be needed? What other new technologies are needed to enable the development of this technology, have they been completed, and how complex are the interactions between this technology and other new technologies under development? What other new technologies does this technology enable? Is there a good plan for proceeding with technology development? Is the technology development effort well connected with prospective users?

1. This is an extremely complex technology and/or is highly dependent on multiple other projects with interfaces that are not well thought out or understood. Score: –9
2. The development of this technology is just roughly sketched out and there are no clearly identified users (i.e., missions). Score: –3

1. There is a clear plan for advancing this technology. While there is an obvious need, there are no specifically identified users. Score: –1
2. There is a clear plan for advancing this technology, there is an obvious need, and joint funding by a user seems likely. Score: +1

Time and Effort to Achieve Goals: How much time and what overall effort are required to achieve the goals for this technology?

1. National endeavor: Likely to require more than 5 years and substantial new facilities, organizations, and workforce capabilities to achieve; similar to or larger in scope than the Shuttle, Manhattan Project, or Apollo Program. Score: –9
2. Major project: Likely to require more than 5 years and substantial new facilities to achieve; similar in scope to development of the Apollo heat shield or the Orion environmental systems. Score: –3
3. Moderate effort: Can be achieved in less than 5 years with a moderately sized (less than 50 people) team (e.g., Mars Pathfinder’s airbag system). Score: –1
4. Minimal effort: Can be achieved in a few years by a very small (less than 10 people) team (e.g., graduate student/faculty university project). Score: 0

Evaluation Methodology

The individual panels were tasked with binning the individual technologies into high, medium, and low priority for level 3 technologies. This was done primarily by grading the technologies using the criteria described above. The panels generated a weighted decision matrix based on quality function deployment (QFD) techniques for each technology area. In this method, each criterion was given a numerical weight by the steering committee, described below. By multiplying the panel grades by the criteria weighting factor and summing the results, a single score was calculated for each technology.

The steering committee based the criteria weighting on the importance of the criteria to meeting NASA’s goals of technology advancement. It determined that the potential benefit of the technology was the most important factor in prioritizing, with the risk and challenges being second, and alignment being third in importance of the three main criteria. To allow for weighting at the subcriteria level, the steering committee assigned a total weighting of 9 to alignment, 18 to risk and challenges, and 27 to benefits. It then divided those values among the subcriteria to generate the values shown in Table C.1.

This method provided an initial assessment of how technologies met NASA’s goals via the criteria evaluation. After each panel came to a consensus on the grades for all criteria for each technology, a total QFD score was computed for each technology. Consider the example shown in Figure C.1. The QFD score for technology 1.1.1, Propellants, is computed using the score for each criterion and the corresponding multiplier as follows:

The technologies were then sorted by their total QFD scores. In Figure C.1, technology 1.3.1, TBCC, has the highest score, and thus it is the highest priority of the three technologies shown.

Once the panels had ordered the technologies by their total scores, they then divided the list into high-, medium-, and low-priority technology groups.³ This division was subjectively performed by each panel for each technology area for which it was responsible, seeking where possible natural break points. For instance, in the case of the

___________________

³ The panels were tasked with designating each technology as high, medium, or low priority only. Chapter 2 contains a figure for each technology area that lists technologies by QFD score, in descending order; this sequencing may be considered a rough approximation of the relative priority of the technologies within each technology area. Also, this ordering places the override technologies (which were designated as high priority despite their relatively low QFD scores) as least among the high-priority technologies, although that is not necessarily the case.

TABLE C.1 Numerical Weighting Factors Given to Evaluation Criteria in Panel Assessments

assessment of TA1, the panel decided that the split between high- and medium-priority technologies should occur at a score of 150, and that the split between medium- and low-priority technologies should occur at a score of 90.

To add flexibility to the assessment process, the panels were also given the option of identifying key technologies that they believed should be high priority but that did not have a numerical score that achieved a high priority rank. These override technologies were deemed by the panels to be high priority irrespective of the numerical scores. As such, by allowing the panels to use this override provision, the numerical scoring process could be used effectively without the evaluation becoming a slave to it. In the summary tables for each technology area, the override technologies are designated by “H*”.

Based on the raw QFD scoring of the 295 level 3 technologies, 64 were initially classified as high priority, 128 as medium priority, and 103 as low priority. The panels subsequently decided to override the QFD scores to elevate 18 medium-priority technologies and 1 low-priority technology (6.4.4 Remediation) to the high-priority group. The final result was to have 83 high-priority technologies, 110 medium-priority technologies, and 102 low-priority technologies. The steering committee believes that the results of the panel scoring validate the design of the QFD scoring process and the decision to allow the panels to override those scores as appropriate.

The panels also assessed which of the technologies have the greatest chance of meeting the identified top technical challenges. While many of the technologies within a technology area could potentially address one or more of the challenges, the panels only labeled those where investment would have a major or moderate impact.

This assessment was used to verify the proper identification of the high-priority technologies and occasionally as validation for using the override option.

2012 NRC REPORT: PROCESS TO IDENTIFY THE HIGHEST-PRIORITY TECHNOLOGIES

In prioritizing the 83 technologies evaluated as high-priority by the panels across all 14 draft roadmaps, the steering committee established an organizing framework that addressed balance across NASA mission areas; relevance in meeting the highest-priority technical challenges; and expectations that significant progress could be made in the next 5 years of the 30-year window of the roadmaps. Furthermore, the steering committee constrained the number of highest-priority technologies recommended in the final list in the belief that in the face of probable scarce resources, focusing initially on a small number of the highest-priority technologies offers the best chance to make the greatest impact, especially while agency mission areas, particularly in exploration, are being refined and can be shaped by technology options. Within this organizing framework, technology objectives were defined by the steering committee to address the breadth of NASA missions and group related technologies.

Technology Objectives

The 2011 NASA Strategic Plan⁴ states:

The steering committee interpreted this formulation of NASA’s strategic vision as the need to assess the technologies by the measure of how well they supported NASA’s various missions.

The question became one of identifying the totality of NASA’s missions that were all-inclusive of the agency’s responsibilities and yet easily distinguished by the type of technologies needed to support them. The steering committee defined the following technology objectives to serve as an organizing framework for prioritization of technical challenges and roadmap technologies.

Technology Objective A, Human Space Exploration: Extend and sustain human activities beyond low Earth orbit.

Supporting technologies would enable humans to survive long voyages throughout the solar system, get to their chosen destination, work effectively, and return safely.

This objective includes a major part of NASA’s mission to send humans beyond the protection of the Van Allen belts, mitigate the effects of space radiation and long exposure to the microgravity environment, enable the crew to accomplish the goals of the mission (contained in Technology Objective B), and then return to Earth safely. This objective includes using the International Space Station (ISS) for technology advancement to support future human space exploration, providing opportunities for commercial companies to offer services to low Earth orbit and beyond, and developing the launch capability required for safe access to locations beyond low Earth orbit.

Technology Objective B, In Situ Measurements: Explore the evolution of the solar system and the potential for life elsewhere.

Supporting technologies would enable humans and robots to perform in situ measurements on Earth (astrobiology) and on other planetary bodies.

This objective is concerned with the in situ analysis of planetary bodies in the solar system. It includes the detailed analysis of the physical and chemical properties and processes that shape planetary environments and

___________________

⁴2011 NASA Strategic Plan, NASA, 2011, p. 4.

the study of the geologic and biological processes that explain how life evolved on Earth and whether it exists elsewhere. It involves development of instruments for in situ measurements and the associated data analysis. This objective includes all the in situ aspects of planetary science; measurement of interior properties, atmospheres, particles, and fields of planets, moons, and small bodies; and methods of planetary protection.

Technology Objective C, Remote Measurements: Expand our understanding of Earth and the universe in which we live.

Supporting technologies would enable remote measurements from platforms that orbit or fly by Earth and other planetary bodies, and from other in-space and ground-based observatories.

This objective includes astrophysics research; stellar, planetary, galactic, and extra-galactic astronomy; particle astrophysics and fundamental physics related to astronomical objects; solar and heliospheric physics; and magnetospheric physics and solar-planetary interactions. This objective also includes space-based observational Earth-system science and applications aimed at improving our understanding of Earth and its responses to natural and human-induced changes. This objective includes all space science activities that rely on measurements obtained remotely from various observational platforms.

These objectives are not independent and are often shared by a single mission (e.g., humans to explore planetary bodies or to service observatories, as was the case with the Hubble Space Telescope), and there are technologies that support more than one of these objectives (e.g., multifunctional structures, electric propulsion, GN&C). Yet this taxonomy is a useful way to categorize NASA’s responsibilities as described in its strategic plan and serves to prioritize the various technologies and technical challenges identified in this study.

Grouped Technologies

The steering committee determined that, in several instances, technologies on the original list of 83 high-priority technologies that were highly ranked in the final prioritization process were also highly coupled. During the prioritization process, these highly coupled technologies were grouped together and considered as one unit. There are a total of five grouped technologies (designated X.1 through X.5). Each one consists of 3 to 5 original technologies as follows:

Prioritizing Technologies Across Roadmaps

Utilizing the panel results, which established a high degree of correlation between high-priority level 3 technologies and the respective technical challenges for each roadmap, the steering committee was able to relate high-priority technologies that aligned with each of the three technology objectives. This organizing principle in turn helped categorize similar technologies with similar drivers (i.e., technologies driven by keeping humans alive, able to be productive, and transported; in situ measurements; and remote measurements) and enabled prioritization among them on a meaningful basis.

The process followed by the steering committee was as follows: First, the steering committee considered only the 83 high-priority level 3 technologies as selected by the panels. These 83 technologies are listed in Table C.2. Next, following the correlation procedure used by the panels, the steering committee mapped those technologies against the top technical challenges (See Table C.3) that it had identified for each of the three objectives. The correlation matrix for the technologies that were ultimately determined to have the highest priority and the top technical challenges for Technology Objectives A, B, and C are shown in Tables C.4, C.5, and C.6, respectively.

In many cases there is little correlation between particular technologies and the top technical challenges for one or more technical objectives. For example, technologies from roadmaps relating to human exploration or life support would have little correlation with Technology Objective C, which is focused primarily on remote measurements from observational platforms, except if servicing is done by astronauts. The correlation information was then used by the steering committee as it voted on the priority of technologies against the three objectives. Each steering committee member voted on the importance of each technology to each objective using a weighted scale:

0 = Not relevant;

1 = Minor importance;

3 = Significant; and

9 = Essential.

The total of the members’ scores assigned to each technology was then summed to create a rank-ordered list of technologies for each technology objective. There were several iterations of voting and discussion first to develop an interim list of 11 to 13 technologies per objective (see Table C.7), followed by another iteration of voting and discussion to obtain a consensus on the final list of 7 or 8 technologies per objective (see Table C.8).

The robustness of the final results was tested by the steering committee in numerous ways. The steering committee used other weighting schemes (such as voting on top five technologies rather than using a 0-1-3-9 weighting factor) and other voting schemes (such as voting to remove technologies rather than voting to include them). Initially the steering committee had removed from the voting any technologies that were uncorrelated to any technical challenge; to make certain all technologies were properly considered, that constraint was relaxed and all 83 technologies were voted upon. In all cases, however, the changes to the methods had little or no impact on the final outcome.

The final short list of the highest-priority individual and grouped technologies is shown in ranked order in Table C.8, showing three columns with 16 technologies. The steering committee that authored the 2012 report assumed that NASA would pursue enabling technology related to all three objectives in a balanced approach, and the steering committee did not recommend or advocate support for one objective over another.

TABLE C.2 The 83 High-Priority Level 3 Technologies from the 2012 NRC Report

^a Fault detection, isolation, and recovery.

^b Guidance, navigation, and control.

NOTES:

1. Technologies are listed by roadmap/technology area (TA 1 through TA 14; there are no high-priority technologies in TA 13). Within each technology area, technologies are listed in descending order by the quality function deployment (QFD) score assigned by the panels that helped to author the 2012 report. This sequencing may be considered a rough approximation of the relative priority of the technologies within a given technology area.

2. Except for the five new technologies, the name of each technology in this table is as it appears in the original list of 83 high-priority technologies in the 2012 NRC report. In some cases, the names have been slightly revised for the 2015 TABS (see Appendix B). Two technologies have been deleted and do not appear in the 2015 TABS: 8.2.4, High Contrast Imaging and Spectroscopy Technologies, and 8.2.5, Wireless Spacecraft Technologies. Three technologies have been renumbered: 5.4.3, 11.2.4a, 12.5.1, above, have become 5.4.2, 11.2.4, and 12.4.5, respectively, in the 2015 TABS.

TABLE C.3 Top Technical Challenges for Technology Objectives A, B, and C

TABLE C.4 Linkages Between Highest-Priority Technologies and Top Technical Challenges for Technology Objective A, Human Space Exploration

TABLE C.5 Linkages Between Highest-Priority Technologies and Top Technical Challenges for Technology Objective B, In Situ Measurements

TABLE C.6 Linkages Between Highest-Priority Technologies and Top Technical Challenges for Technology Objective C, Remote Measurements

TABLE C.7 Interim List of Highest-Priority Technologies, Ranked by Technology Objective, Comprising a Total of 27 Individual and Grouped Technologies, with 11 or 12 per Technology Objective

NOTE: Shaded items do not appear in the final list in Table C.8.

TABLE C.8 Final List of Highest-Priority Technologies, Ranked by Technology Objective, Comprising a Total of 16 Individual and Grouped Technologies, with 7 or 8 per Technology Objective