This chapter recommends a methodology for conducting independent reviews of future updates to NASA’s space technology roadmaps. This methodology takes into account the extent of changes expected to be implemented in the roadmap from one generation to the next and the amount of time since the most recent comprehensive independent review of the roadmaps.
The chapter reviews the path that led to the recommended methodology by discussing (1) the methodology used during the previous study as documented in the 2012 National Research Council (NRC) report NASA Space Technology Roadmaps and Priorities: Restoring NASA’s Technological Edge and Paving the Way for a New Era in Space,1 (2) the methodology used for this report, and (3) the NASA Office of the Inspector General report NASA’s Efforts to Manage Its Space Technology Portfolio, published December 15, 2015. This review provides the foundation for understanding the value of an independent review and the suggested future methodology for such reviews.
In June 2010, Robert Braun, then NASA’s chief technologist, requested that the NRC conduct a study of 14 space technology roadmaps that NASA had drafted. In response to this request, the NRC appointed an 18-member steering committee and six study panels with a total of 56 additional experts. The six panels covered various subsets of the 14 roadmaps. The steering committee and the panels met for the first time in January of 2011. The steering committee held three additional meetings between January and September of 2011. During the same time frame, each of the six panels held a 1-day public workshop and two additional meetings for each roadmap it was reviewing. Public input was also solicited from a website where 144 individuals provided 244 comments on the draft roadmaps. All of the gathered data allowed the prioritization all of the level 3 technologies in each roadmap, and those detailed analyses are provided as appendixes to the 2012 report. These data were then synthesized by the steering committee and documented in the main body of the report. An interim report was provided in late 2011,2 and the final report
1 NRC, 2012, NASA Space Technology Roadmaps and Priorities: Restoring NASA’s Technological Edge and Paving the Way for a New Era in Space, The National Academies Press, Washington, D.C.
2 NRC, 2011, An Interim Report on NASA’s Draft Space Technology Roadmaps, The National Academies Press, Washington, D.C.
was published in early 2012. This significant effort was completed in roughly a year, which is rapid for a study by the National Academies of Sciences, Engineering, and Medicine (by contrast most NASA science decadal surveys take nearly 2 years to complete).
The methodology from the NRC’s 2012 review is described in Appendix C of this report. Briefly, the individual panels were tasked with categorizing the level 3 technologies into high-, medium-, and low-priority groups. The panels generated a weighted decision matrix based on quality function deployment (QFD) techniques for each technology area. In this method, each criterion and subcriterion was given a numerical weight by the steering committee. The weighting was based on the importance of the criteria to meeting NASA’s goals of technology advancement.
NASA’s technology roadmaps and the review of the roadmaps by the Academies are just two steps in the overall effort to define and execute NASA’s technology investment portfolio. The complete cycle is shown below.
- FY 2010—Space Technology Roadmaps—revised every 4 years
—140 challenges, 320 level 3 technologies, 20-year horizon
- FY 2011—NRC Study—requested every 4 years
—Prioritization: 100 top technical challenges; 83 high-priority technologies (roadmap specific), 16 highest of high-priority technologies (looking across all roadmaps)
- FY 2012—Development of the Strategic Space Technology Investment Plan (SSTIP)—revised every 2 years
—Updated space technology roadmaps: incorporated NRC study results
—Developing a Strategic Space Technology Investment Plan: current investments, current priorities of NASA’s mission directorates and offices, opportunities for partnerships, gaps vs. current budget and capacities, 20-year horizon with a 4-year cadence
- FY 2013—Execution
—Investment portfolio: NASA Technology Executive Council uses SSTIP to make decisions
—Must accomplish: mission needs and commitments, push opportunities, affordability, technical progress, programmatic performance
As can be seen above, NASA intends to revise the roadmaps every 4 years, followed by an independent review, which then would be used to update the SSTIP, which would in turn guide the execution of the “investment portfolio.” The 2010 roadmaps covered all NASA space technologies. The draft 2015 roadmaps also include a roadmap for aeronautics, as well as an additional volume: TA 0, Introduction, Crosscutting Technologies, and Index.
The current review is more limited than the prior comprehensive review. This review is limited to technologies that appeared in the 2015 roadmaps but did not appear in the roadmaps in the 2012 NRC report (see Appendix B for the comparison between the Technology Area Breakdown Structure [TABS] in the 2010 roadmaps, the revised TABS from the 2012 NRC report, and the 2015 roadmap TABS). The review was designed to use the same methodology as the NRC’s 2012 study (see Appendix C) to determine whether any of the new technologies should be added to the list of 83 high-priority technologies and the subset of 16 highest-priority technologies in the 2012 report. The QFD scores were compared with those in the 2012 report to verify that they were consistent.
When the 2012 report was prepared, the NASA design reference missions (DRMs) were not available, so as a substitute the panels identified a number of challenges for each technology area that served to drive the individual technology capabilities. These challenges were generated to provide a focus for the technology development and to assist in the prioritization of the level 3 technologies. For the current 2015 NASA technology roadmaps, instead
of using the technical challenges, NASA used a newly produced set of DRMs, which are described in the first volume of the roadmaps.3 The 2015 review did not include the following items:
- TA 0 Introduction, Crosscutting Technologies, and Index. This document includes the topics that cross multiple technology areas; the categorization of technologies as enabling or enhancing for each DRM; the technologies identified to support campaigns, such as the Evolvable Mars campaign; and the new crosscutting technology structure provided by NASA that built upon what was suggested in the 2012 NRC report.
- TA 15 Aeronautics roadmap. Because there was no TA 15 roadmap in the set of 2010 draft roadmaps that the earlier NRC study reviewed, there is no baseline against which to assess the aeronautics technologies.
The NASA Office of the Inspector General (OIG) performed an audit of NASA’s technology portfolio, the results of which were published in December 2015.4 The OIG profiled the top 15 space technology projects by fiscal year 2015 funding in the following programs: Technology Demonstration Missions Program, Game Changing Development Program, Advanced Exploration Systems Program, and the Science Mission Directorate’s Research Divisions. The report found that deficiencies in NASA’s management processes and controls may limit its efforts to effectively manage its portfolio of space technology investments. The issues cited included a delayed revision of the SSTIP (the one cited frequently in this report was prepared in 2012), an unclear process for initiating new space technology projects, and an inconsistent process for measuring technology projects’ return on investment. One of the recommendations was a further prioritization of “core” and “adjacent” technologies in a revised SSTIP.
During the present study, NASA researchers presented information about the new technologies to the committee, including their evaluation of the technologies’ value using the QFD methodology in the 2012 NRC report. It became clear that the researchers struggled to assign objective grades to their technologies—in almost every case, the QFD scores they assigned were the highest possible. These high scores often overstated the technology’s value owing to the researchers’ understandable bias in favor of their technology and or their limited understanding of broad technological needs. An independent review would provide an objective evaluation of individual scoring and also better captures the alignment to non-NASA aerospace needs, as well as with non-aerospace national goals.
The first volume of the 2015 NASA technology roadmaps includes lists of all level 4 research tasks that are designated as either enabling or enhancing for each DRM.5 An informal review of the lists indicates that there may be a tendency to overstate the case for “enabling” versus “enhancing.” Also, since the DRMs as a whole comprise all possible missions that NASA might carry out rather than a smaller, budget-constrained set that is more likely to be executed, it is difficult to assess the value of technologies based on their ability to support the DRMs. NASA has acknowledged that the existing set of DRMs might be too large, and it has been developing a smaller set. Since the DRMs are a significant new feature in the 2015 roadmaps, a more detailed review of the DRMs and their relationship to the development of the NASA technology portfolio is merited. An independent review of the relationships between the DRMs and the technologies that would enable or enhance them would strengthen the understanding of mission pull and technology push. DRMs tend to change with political cycles, especially for human missions, plans—as has occurred with the last two administrations—so an independent review of the DRMs when an administration changes might be merited.
The addition of TA 15 Aeronautics roadmap also merits an independent review. The Aeronautics thrusts that are used in place of the DRMs actually resemble the 2012 NRC report technical challenges more closely than do
3 NASA, 2015, NASA Technology Roadmaps: Introduction, Crosscutting Technologies, and Index, May 2015 Draft, http://www.nasa.gov/offices/oct/home/roadmaps/index.html, accessed June 29, 2016, p. i-46.
4 NASA Office of the Inspector General, 2015, NASA’s Efforts to Manage Its Space Technology Portfolio, Report No. IG-16-008, Washington, D.C.
5 NASA, 2015, Technology Roadmaps, Introduction, Crosscutting Technologies, and Index, Appendix E.
the DRMs. Future reviews will need to address this inconsistency. Isolating aeronautics from the other 14 roadmaps eliminates the opportunity to assess possible synergies that exist between NASA’s space and aeronautics technology portfolios in areas such as materials, electronics, and propulsion, to name just a few examples.
Given the dynamic nature of technology development organization and management, the pace of technology advances, NASA missions, NASA organization, and so on, and because each iteration of the roadmaps and each independent review will result in new lessons learned, it is not useful to come up with a long-range plan for future reviews. In addition, future review plans will always be subject to change. Accordingly, there is little value in having one independent review make recommendations for more than one subsequent review.
Taking into account lessons learned from the current and prior review, as well as the recommendations from the NASA OIG report, the following methodology is proposed for the next review:
Recommendation 1. An independent review of a roadmap should be conducted whenever there is a significant change to the roadmap. NASA’s technology roadmap revision cycle is expected to be performed every 4 years, but significant changes in NASA’s direction might necessitate more frequent reviews. A review should be one of two types: either a comprehensive review of the complete set of roadmaps (including TA 15), such as the one performed in 2012, or a focused review, such as the one in this report. A focused review can be conducted using fewer resources because it addresses only a subset of the total technology portfolio. In making recommendations about the review methodology, each future independent review should focus on the methodology to be used for the next review rather than on a long-range plan covering multiple reviews.
NASA Roles in the Review
A NASA internally generated prioritization of the technologies across all roadmaps would greatly improve the speed and efficiency of future independent reviews. This prioritzation could be done using either the same methodology as the NRC (see Appendix C) or some other process of NASA’s devising. A key aspect of this effort is that it be a comprehensive prioritization to promote not only the top technologies in each roadmap, but also across all roadmaps.
The NASA Technology Executive Council (NTEC) and the Center Technology Council (CTC)6 have the following responsibilities:
Strategic Integration manages and coordinates the NASA Technology Executive Council (NTEC) meetings. These meetings are chaired by the NASA Chief Technologist. Council membership includes the Mission Directorate Associate Administrators, the NASA Chief Engineer, and the NASA Chief Health and Medical Officer. The function of NTEC is to perform Agency-level technology integration, coordination, and strategic planning. NTEC’s responsibilities include:
- Review, from an Agency perspective, the progress of each project level technology activity, against the baseline performance milestones.
- Assess the program level budget and schedule adequacy of the Agency’s technology development activities to meet Agency strategic goals.
- Assess the Agency-level technology gaps, overlaps, and synergies between the Agency’s technology programs.
- Assess the technology maturation progress against the Mission Directorate’s goals, objectives, missions, and timelines, as well as the Agency technology roadmaps and strategic goals.
- Assess the balance and prioritization of the Agency’s technology investment portfolio.
- Develop and review decisional recommendations regarding the Agency’s technology investment plans.
The Center Technology Council (CTC) is organized and chaired by the Office of the Chief Technologist (OCT). Council membership includes the center chief technologist from each NASA Center (including Jet Propulsion Laboratory) and a representative from the Office of the Chief Engineer and is observed by a representative from each Mission Directorate. The CTC focuses on institutionally funded activities and development of the programs of the OCT. The responsibilities of the CTC include:
- Assess the Agency technology roadmapping and technology prioritization activities from a bottoms-up, institutional perspective and provide these assessments to NTEC.
- Provide NTEC with recommended changes in technology program scope, prioritization, and roadmapping from the Centers’ perspective.
- Provide NTEC with “beyond-program” technology inputs for potential future development.
- Develop Center reports on the performance of the innovation and technology development activities at each Center.
- Identify inter-Center technology leveraging opportunities.
- Develop technology reports (i.e., have the function to look outside the walls of NASA for technology opportunities).
As noted above, the responsibilities of the NTEC include prioritization of NASA technology investments, and the CTC is charged with assisting the NTEC in this effort. The 2012 NRC report notes that prioritization of technologies would be facilitated by the use of systems analysis (see the recommendation on systems analysis in Appendix E).
Recommendation 2. Before the next independent review, the NASA Technology Executive Council and the Center Technology Council (NTEC/CTC), in accordance with their charters, should prioritize the technologies that will be examined in the review. The NTEC/CTC should present the results and rationale for the priorities to the next independent review committee. The prioritization process should take into account the factors included in the prioritization process described in Appendix C. It should also be supported by additional factors such as linkage of technologies to a concise list of design reference missions (DRMs), including an assessment of the technologies as enabling or enhancing; the use of systems analysis to establish the technology’s benefit to the mission relative to the benefit of alternative technologies; and correlation of technology priorities with both expected funding and required development schedule.
The 2012 NRC report included the following recommendation:
Cooperative Development of New Technologies. OCT should pursue cooperative development of high-priority technologies with other federal agencies, foreign governments, industry, and academic institutions to leverage resources available for technology development.
The resources available for development of NASA technologies are inadequate to support the development of the broad array of technologies in the roadmaps. One approach for improving the allocation of technology development resources would be to use a modified version of an approach applied by the Army Research Laboratory (ARL). ARL has classified each of the technologies in its 2015-2035 Science and Technology Campaign Plans7 as falling into one of three categories: Lead, Collaborate, or Watch. The current study committee has modified the
definitions and added one category: Park. These four categories can help NASA determine the level of cooperative development with others and thus reduce their technology development expenditures.
- LEAD: NASA’s needs and timing for a given technology are so unique that advancing the technology will require NASA investment without substantial shared investments by others. Maintaining in-house expertise and infrastructure for this technology is critical to unique NASA needs.
- COLLABORATE: NASA establishes an interdependent partnership with other organizations (government, industry, academia, or international partners) to pursue a technology using shared investments. This collaboration can take several forms. A common example is NASA and another government agency coordinating research and development and communicating the results to each other. Another form is a public-private partnership in which NASA provides part of the funding with cost sharing by the industry partner. NASA can also provide its research partners with access to unique infrastructure, technological advances, and in-house expertise that significantly influence the direction of the collaboration. Collaborating allows NASA’s in-house technical experts to develop technologies that they may not have otherwise been afforded the opportunity to do so.
- WATCH: NASA maintains high vigilance monitoring emerging technologies and corresponding efforts within industry, academia, and international markets. Technologies in this category will most likely achieve advancement outside of NASA because of substantial interest and investment by outside organizations and the technology is not unique to NASA missions. It is important that NASA stay actively engaged in the national and international scientific dialog to remain poised to react to developments that make the technology a viable approach for NASA needs. One means of staying actively engaged in the national and international scientific dialog is the attendance at and the participation in scientific conferences by NASA researchers.
- PARK: Pursuing technology advancement requires better definition of mission or operational requirements before proceeding. The roadmap milestones need to be readjusted to achieve just-in-time rather than just-in-case delivery of value. NASA would minimize effort for technologies in this category until better definition is achieved.
Example of a Technology for Lead Status
Radiation protection and mitigation is well suited for a Lead designation (see technology group X.1, Radiation Protection and Mitigation for Spaceflight, in the group of highest-priority technologies). It was cited as the highest-priority technology for human spaceflight in the 2012 NRC report, it was one of the three highest priority technical capabilities identified in the 2014 NRC report on human spaceflight,1 and it is well represented in NASA’s SSTIP under several core technology investments such as Lightweight Space Structures and Materials, ECLSS, Space Radiation Mitigation, and Scientific Instruments and Sensors. Radiation hazards include both prompt and cumulative damage from ionizing radiation from the sun (the solar wind), from solar particle events (SPEs), and from galactic cosmic rays (GCRs). Shielding in the form of lightweight materials and structures can reduce the exposure of humans and sensitive components to ionizing radiation and SPEs during space travel and in surface habitats, but a satisfactory approach for mitigating GCRs has yet to be determined. GCRs have such high energies that they produce secondary radiation when they interact with shielding or other spacecraft and habitat materials. This secondary radiation can increase the radiation hazard to humans and equipment. Electrostatic deflecting shields have been proposed, but such systems would be heavy, require substantial electrical power, and could themselves pose a threat to human health.
In addition to investments in radiation protection technologies, investments in technologies that are unique to NASA’s needs would also be required for long-term space missions. These needs include (1) smart dosimeters for tracking cumulative doses from all three forms of space radiation both within and external to spacecraft and protective habitats, (2) mitigating biomedical approaches such as dietary regimens and drugs, (3) sophisticated risk-assessment models that can model and simulate radiation risks due to changes in the space radiation environment during all phases of a mission, and (4) sensors and models to predict changes in the space radiation environment.
Examples of Technology for Collaborate Status
NASA and General Motors have partnered to codevelop robots that can work side by side with people to assist in space missions and to enhance safety and productivity of automotive manufacturing. Further collaboration on this topic is encouraged, especially as it relates to technology 4.4.3, Proximate Interaction. Such collaboration is important and valuable, particularly during the current phase of fast-paced adoption of new proximate interaction technologies for industrial and assistive robotics. Investment in collaborative and co-development projects enable NASA to influence the directions of new development so that the technologies better align with NASA needs. As proximate interaction will be an important component of future human space exploration missions, collaborations are also necessary to build and strengthen in-house expertise, allowing NASA to take a lead role in this technology area when it becomes necessary.
Examples of Technologies for Watch Status
Examples of Watch technologies are 11.4.6, Cyber Infrastructure, and 11.4.8, Cyber Security. The use of these important technologies as they are developed by other government and nongovernment organizations is expected to increase within the NASA infrastructure. It is possible that in the future cyber-security needs within NASA flight segments could elevate this technology to the Collaborate category as specific cybersecurity elements are incorporated into flight systems. Another example of a Watch technology is 11.3.5, Exascale Simulation. Several different countries and companies are working toward exascale computing (1,000 petaflops), but that target is not expected to be achieved before 2022. In the United States, the recently announced National Strategic Computing Initiative is expected to provide an extra incentive to reach this goal. NASA will certainly make use of exascale computing, and by watching the development of these computers it will be ready to use them effectively without needing to engage in their development. As exascale computing moves closer to reality, this technology could move from the Watch status to Collaborate status.
Example of a Technology for Park Status
An example of a Park technology is 7.4.4, Artificial Gravity, which is produced by spinning a spacecraft. The requirements for and the efficacy of this technology are unclear at the moment, and the likelihood of its need is dependent on the effectiveness of other gravity countermeasures outlined in 6.3.2, Long-Duration (human) Health, including research task 18.104.22.168, Artificial Gravity, which is produced by spinning individual astronauts using apparatus installed within a spacecraft. It is possible that difficulties in achieving the goals of Long-Duration Health, combined with a near-term need for a deep-space-capable habitation system, would require the posture on 7.4.4, Artificial Gravity, to change from Park to Lead at some future date.
Recommendation 3. As part of its prioritization process, NTEC/CTC should classify each technology to be examined by the next independent review (at TABS level 3 or level 4) as Lead, Collaborate, Watch, or Park. In addition, the Office of the Chief Technologist (OCT) should update NASA’s electronic technology database, TechPort, so that it, too, indicates for each technology whether NASA is pursuing it as Lead, Collaborate, Watch, or Park. For collaborative efforts, OCT should include in TechPort details on the nature of the collaboration, including facilities, flight testing, and the development of crosscutting technologies.
Design Reference Missions
Finding 3. A more concise list of design reference missions (DRMs) produced by NASA that more closely resembles a budget-enabled set of missions would result in better prioritization of “enhancing” and “enabling” technologies in the roadmaps. Whenever there is a substantial change to NASA mission
plans and the DRMs are updated, technologies could be reprioritized by rescoring their benefit and relevance to NASA
The Next Independent Review
Recommendation 4. The next independent review should be comprehensive if there have been major changes to the roadmaps and/or the DRMs, or it should be a focused review and examine only the new technologies if they are few in number. The review should cover the following:
- The prioritization of technologies previously completed by the NTEC/CTC and the process used to conduct the prioritization.
- Roadmap for TA 15 Aeronautics.
- The first volume of the technology roadmaps, TA 0 Introduction, Crosscutting Technologies, and Index.
- The relevance of technologies to the DRMs as either enabling or enhancing.
- Recommendation for the methodology to be used for the review that in turn follows it.
In summary, the committee reviewing the 2015 NASA Technology Roadmaps has formulated a methodology for future independent reviews that will reduce their time and cost by (1) having the NASA NTEC/CTC do a preliminary prioritization of technologies based on the DRMs and (2) configuring the review based on the extent to which the technologies and/or the DRMs have changed. Sorting the level 3 technologies or level 4 research tasks into Lead-Collaborate-Watch-Park categories will help NASA identify technologies suitable for collaboration and will conserve technology development resources.