Interim Report of NRC Review of NASA's Pioneering Revolutionary Technology Program (2003)

Advisers to the Nation on Science, Engineering, and Medicine

At the request of the National Aeronautics and Space Administration, the National Research Council established the Committee for the Review of NASA’s Pioneering Revolutionary Technology (PRT) Program and three supporting panels. The membership of the committee includes a cross section of senior executives, engineers, researchers, and other aerospace professionals. The committee’s purpose is to provide a review of the technical quality of NASA’s PRT program. The committee’s initial assessment, which is summarized in this letter, is based on its collective wisdom as well as inputs from NASA researchers and program managers. The committee was supported by three panels, one for each program of PRT—a total of 37 volunteers participated. The attachments provide a list of committee members and other study participants (Attachment A) and the committee’s statement of task (Attachment B). The work of the committee is ongoing, and late next year it will issue a detailed assessment of the PRT program.

The committee and the three panels met at NASA Ames June 10-13, 2002, for an overview of the PRT program and its various elements. Subgroups of panel members subsequently participated in laboratory site visits, teleconferences, and other information-gathering activities throughout the summer. NASA researchers submitted completed questionnaires describing individual research tasks funded within the program. A total of 379 tasks were reviewed and 13 site visits made. In September 2002, each panel met in Washington, DC, to come to a consensus on observations, findings, and recommendations to be highlighted for the parent committee and to engage in an interactive dialogue with NASA program managers. Panel draft working reports were then submitted to the committee. The final committee meeting was held in Washington, DC, on November 6-8, 2002. A portion of this meeting was dedicated to holding a dialogue between NASA managers representing the PRT program and committee members. This letter report provides an overview of the salient points made in the dialogue with NASA management. The details of the committee’s review will be presented in the final peer-reviewed report to be

prepared after some tasks are revisited and additional information is obtained—necessary steps that are important to the final determinations of the committee.

The preliminary observations and recommendations presented here are offered as an interim report to NASA on overarching issues that will remain critical regardless of near-term programmatic decisions. This interim report primarily emphasizes issues that were deemed to be of the greatest significance and interest to NASA management in light of the current particulars of each program area. Issues discussed in this report (and presented at the November dialogue session) are those the committee believes are not subject to change despite programmatic revisions being planned in various PRT programs between the November 2002 feedback session and the planned Spring 2003 re-visits. Overall consideration and assessment of balance and other crosscutting issues for the PRT program as a whole will be discussed in the final report after the committee completes it ongoing work. The committee notes that it refrained from drawing any conclusions on matters of budget or making recommendations for increases in budget levels. While some areas may suffer from a lack of critical mass, recommendations for increased resources to address the problem are of little value to management and thus are not made.

The committee found that the vast majority of the PRT program consisted of good, solid work that is important to NASA and the nation. Specifically, the committee judged that 90 percent of the PRT program fell into this category. Of those projects, the committee singled out 10 percent for special recognition. This 10 percent was work of the highest quality, representing truly world-class endeavors. The remaining 10 percent of the program was recommended by the committee for discontinuation or transition. Projects marked for transition were typically of high quality but involved technologies ready to be funded by a NASA mission or external partners. Projects marked for discontinuation were identified primarily based on a judgment about the quality of the work, although some of the tasks themselves appeared to be of little value to NASA or were poorly aligned with the stated goals of the PRT program.

The committee’s overall assessment of the research within PRT was made based on the individual assessments of the three supporting panels. Tasks judged by the committee to be outstanding met the following criteria: (1) evidence of productivity (publications, software, presentations, patents, mission-accepted technology); (2) strong linkage at the task level to actual flight projects, flight engineers, or science customers; (3) connectivity with other research communities external to NASA; and (4) external recognition of the research group as an authority in the subject matter. In some cases, excellence was also observed when basic research, facilities, systems analysis, flight integration, and test and evaluation were co-located or programs had achieved success over a period of 10 to 15 years and continued to do so. Areas of excellence and areas of concern, which will be addressed in the final report in more detail, are highlighted in the following paragraphs.

A separate panel of individuals was assigned to review each of the three programs within PRT: Computing, Information, and Communications Technology (CICT), Engineering for

Complex Systems (ECS), and Enabling Concepts and Technologies (ECT). Each section below addresses technical quality, portfolio content and its importance to NASA, gaps in technology investment, relevance, and balance. Other criteria included qualifications of researchers; adequacy of facilities, equipment, and tools; and connectivity to other efforts both internal and external to NASA.

The Computing, Information, and Communications Technology program contains four projects:

• Space Communications (SC),

• Intelligent Systems (IS),

• Information Technology Strategic Research (ITSR), and

• Computing, Networking, and Information Systems (CNIS).

The committee found that a majority of the work within CICT was good, solid, NASA-focused research that should continue. Seventeen tasks were highlighted by the committee as examples of outstanding and exceptional work. These occurred in the IS, ITSR, and SC projects. The committee notes that no truly outstanding tasks were found in CNIS. Four areas (comprising multiple tasks) were emphasized by the committee as being world-class: autonomous robots; planning and scheduling; software validation and verification; and space communications hardware.

The committee also identified nine tasks that, for various reasons, were ready for transition out of the research and development funding line, were complete and should be discontinued, or should no longer be pursued. It found that the PRT research in nanotechnology and human-centered computing could be improved through better cross-agency coordination and new research portfolio components. The committee notes that there are other investments in microelectromechanical systems (MEMS, the general area that includes nanotechnology) research within NASA and the Aerospace Technology Enterprise (Code R). However, NASA researchers did not mention related NASA work and how such efforts might tie in with the CICT work. Coordination of PRT work in nanotechnology, microsensors, distributed and microspacecraft, and intelligent systems with related work in other NASA missions is needed to solidify the overall MEMS research effort. There was little evidence of the use of cognitive human factors assessments in the human-centered computing area.

The committee noted two main gaps in the CICT research portfolio. NASA scientists and missions generate terabytes of data that must be distributed and analyzed throughout the country, yet the review panel saw little or no R&D effort directed toward the management of massively distributed data. The panel found no work in the new software architectures needed for highly distributed processing (both in real-time and information systems applications). There may be work in these areas conducted in other parts of NASA, but this was not presented to the review panel.

To effectively manage and prioritize the PRT technology portfolio, NASA should establish a clear research program architecture. Such a program architecture should help in identifying gaps and overlaps in technology investments. It would also improve communication between top-

level management and task principal investigators so that principal investigators are aware of high-level context, goals, and the ultimate applicability of their research. The committee observed a lack of connectivity between the nanotechnology, microsensors, distributed and microspacecraft, and intelligent systems work in the PRT program. NASA should also take actions to ensure adequate and value-adding communication from within CICT to related groups in the Enabling Concepts and Technologies and Engineering for Complex Systems programs.

Each task, project, and program should have clearly defined and realistic, yet challenging, measures of technical success and impact. NASA managers should develop a set of criteria to evaluate CICT performance. Suggested criteria include demonstrated NASA applicability, researchers’ productivity, technology transfer to an application, and acknowledged authority in specific subject matter.

The qualifications of CICT’s technical staff are very good. NASA should continue to maintain subject matter expertise in areas critical to NASA’s mission, such as autonomous robots, space communications hardware, planning and scheduling, and software validation and verification. These are areas in which current PRT program efforts were highlighted as worldclass by the committee. They are also core areas of competency that NASA should strive to maintain a lead in by employing high-quality researchers. The committee also considered the facilities and working environment to be in a good state of repair and on a par with those of other government laboratories and facilities.

The committee was troubled by the inconsistencies in researcher’s awareness of, and collaboration and cooperation with, others working outside the PRT program and outside NASA. For example, the high-performance computing research within CICT does not appear to exploit outside work. However, the software verification and validation team showed good awareness of external NASA work. Similarly, outside researchers have an inconsistent understanding of NASA’s work, in part because NASA researchers do not publish their results in peer-reviewed journals often enough. NASA’s robotics and software verification and validation teams are well known outside the agency; however, efforts at NASA on parallel programming tools are not well known. CICT managers should continue to encourage close connections between researchers and the external research community. This could be achieved through peer-reviewed publication of research results, participation in major conferences and workshops, organization of technical workshops, involvement as reviewers and editors for journals, and other similar efforts.

One impediment to possible collaboration is that dissemination of NASA data and software is often restricted. For example, parallel programming tools and benchmark test cases for problems on the scale of a NASA mission are not made available outside NASA. The committee recommends that CICT streamline its process for distribution of research reports, benchmark data, and software in a manner that accommodates export control and other restrictions.

The Engineering for Complex Systems program has been in state of flux and is still in the early stages of developing a critical mass of research in risk assessment and risk management. The efforts to redirect the program are appropriate given its central importance to NASA’s mission. The ECS program is divided into three projects:

• System Reasoning and Risk Management (SRRM),

• Knowledge Engineering for Safety and Success (KESS), and

• Resilient Systems and Operations (RSO).

ECS work in individual tasks is, in general, considered solid. Three of the tasks within the ECS program are examples of outstanding or exceptional work: Organizational Risk Perception and Management in KESS; Virtual Iron Birds in KESS; and Advanced Software Verification and Testing Tools in RSO. The ECS program appears to address the right problems through a multidisciplinary research approach; however, there are also gaps that weaken the ECS portfolio. The committee was concerned about several tasks in the ECS program. The Probabilistic Analysis of International Space Station (ISS) Power Systems task is complete and should either be transitioned to another part of NASA or discontinued. The specific content of the Socio-Technical Approach for Identifying Ground Processing Risk task is of questionable value and should be discontinued, although the general subject is very important. In addition, the committee has continuing concerns regarding the state of flux within the SRRM portfolio. Achievement of SRRM goals is essential for NASA, but the project could be improved in many ways.

SRRM research is critical to future NASA missions and, if managed and implemented effectively, has the potential to achieve cross-NASA applicability and national importance. The ECS panel, however, had serious concerns about the SRRM project. The approach to tackling such important topics needs clarification, especially given the state of flux within the entire SRRM portfolio. As presented to the committee in June 2002, the SRRM portfolio appeared to be based mostly on internal work and knowledge. Little leveraging of external work in risk management was evident. In September 2002, NASA managers reported that changes were being made in the portfolio, although the details were not available for review before the writing and publication of this report.

Despite this state of flux, the committee offers several recommendations for improvement of the SRRM project. Research interconnectivity and technical leveraging should be increased in the SRRM project both across NASA communities and across the broader community external to NASA. A wealth of useful, applicable information is available and would only enhance the SRRM project. SRRM results should be applicable to all NASA projects and missions. The SRRM project should also ensure that the early designs of the Conceptual Risk Tool (risk workstation) under development address the major NASA mission-system design areas. The committee recognizes the difficulty of this undertaking. Incomplete design trade-offs may produce misleading or incorrect information, resulting in a false sense of security in the final product. An integrated top-down systems approach is crucial for risk-management application development.

In the KESS project, developing the much-needed models is challenging, and current efforts are commended by the committee. However, the milestones used to gauge research progress may in several cases be too demanding to effectively foster quality research. The RSO project has top-quality researchers working on problems, but the committee has concerns about whether the right, NASA-specific tasks are being pursued.

The committee also recommends that the ECS program increase the use of benchmarks— quantitative goals or expectations that serve as technical measures of success—at the lowest practical level in order to measure progress. Since risk management is such an important field of

research for NASA-specific applications, NASA should continue to establish and strengthen its national-level leadership in the development and application of risk management methods.

Finally, the ECS program should explore the use of non-conventional software research approaches, including dependable computing and static analysis, to help NASA reduce unproductive overlap in current approaches. This need is especially apparent in the Resilient Software Engineering task within RSO.

The primary focus of the Enabling Concepts and Technologies (ECT) program is the development of spacecraft bus and payload technologies, with an emphasis on essential flight hardware. The program is divided into three main projects:

• Energetics;

• Advanced Spacecraft and Science Components, which includes four elements— Space Environmental Effects (SEE), Advanced Measurement and Detection (AMD), Resilient Materials and Structures (RMS), and Distributed and Microspacecraft (D&MSC); and

• Advanced Concepts, which includes the Technology Assessment Analysis (TAA), NASA Institute for Advanced Concepts, and Revolutionary Aerospace Systems elements.

The committee found that much of the portfolio was inherited from previous programs in a “bottom-up” manner. There was no evidence that the current portfolio was developed according to a comprehensive strategy. The committee does note that NASA managers plan to develop future portfolios within this program using strategic planning tools and processes, such as the prospective Technology Assessment Analysis. The committee supports such an approach.

Most of the tasks within the ECT program were deemed either good or excellent on an individual basis. ECT panel members judged approximately 20 percent of the ECT program as outstanding. The Energetics project had seven outstanding tasks (27 percent of the Energetics tasks). The AMD element had eight outstanding tasks (24 percent of the AMD tasks). Two revolutionary areas of research noted by the panel within the AMD element are radio-frequency/terahertz (RF/THz) and focal planes. Other areas of excellence within these elements and projects have been successfully transitioned to missions, including the microshutter and microthermopile array in AMD and electric propulsion within Energetics. Within the RMS element, one task was found to be of outstanding quality, and within the D&MSC element three tasks were considered excellent. The SEE element, though not revolutionary in nature, was judged by the committee to provide a unique and much-needed service to the spacecraft design community.

Conversely, the panel determined that several projects within the ECT program should be considered for discontinuation. In the AMD element, the All Aluminum Lightweight Optics and Structures task has developed a mature technology that should be transitioned into a mission-oriented code at NASA. Three other tasks were considered to be of low quality or low value to NASA: (1) Optimized STAR Structures within the AMD element, (2) Large Area Membrane

Fabrication and Deployment within the RMS element, and (3) Chemical Propulsion work (BiPropellant, Monopropellant, and Micro-propulsion) within Energetics.

The committee did not make a specific judgment on the TAA element within the Advanced Concepts project of the ECT program, given its early stage of formulation. However, there is concern that although this area of research is crucial to the PRT program and possibly all of NASA, it is not receiving the emphasis and appropriate technical direction it needs. Strengthening is recommended in this area of research.

ECT researchers are well qualified technically. However, the panel noted instances of their appearing to be overburdened with marketing and advocacy activities that competed with valuable time and resources for completing research and exploring new areas of research, although keeping a program “ sold” is a recognized necessity. Consistently lacking within the ECT program was a priority emphasis on researchers publishing their results in peer-reviewed journals. The committee suggests that NASA maintain an environment that nurtures and rewards intellectual leadership and technical excellence. Individual expectations should be aligned with metrics (e.g., acceptance of work in refereed publications, receipt of patents) of excellence and leadership within the broader technical community. This should be encouraged in addition to, not in place of, metrics for measuring progress toward technology maturation and transition. The highest-quality tasks managed to do both.

The facilities used by the ECT program are good. NASA should strive to maintain several that are world-class, including the E-beam lithography laboratory at the Jet Propulsion Laboratory (JPL), the Polymer Rechargeable Battery Laboratory at NASA Glenn Research Center, and the Electric Propulsion and Photovoltaic test facilities at NASA Glenn. Panel members also observed that the co-location of basic research, systems analysis, engineering, test and evaluation, and flight qualification improves quality and keeps research focused. This was evident in both the AMD element and Energetics project. The committee recommends that researchers, test facilities, and systems analysis capabilities be vertically (or virtually) integrated wherever possible.

Connectivity of the ECT program to other areas within NASA and to the broader technical community varied from project to project. There were specific examples of good teaming between NASA researchers and external partners in the SEE element and Energetics project. The committee recommends that this type of teaming and collaboration be encouraged and expanded whenever possible. About 40 percent of the ECT program is funded through the Cross-Enterprise NASA Research Announcement (NRA). While the committee views this type of competitive solicitation as a valuable incubator for technology development, the rules in the NRA solicitation prevented quality opportunities for teaming between NASA researchers and the NRA winners. Upon formation of the ECT program, NRA management was transferred from the Space Science Enterprise to the Aerospace Technology Enterprise. This management change, coupled with the broad focus of the announcement, has led to a general lack of integration of the NRA research with NASA programs and centers. This effect may also be due in part to the competitive environment set up between the awardees and NASA researchers who did not win the contracts. The panel also observed a general lack of “ownership” of the NRA work on behalf of NASA. This can most likely be attributed to two factors: (1) allowing NASA centers to compete for awards, and (2) absence of a clear mechanism for evaluation of progress during the award’s duration. The research performed under the NRA should be managed as an integral part of the in-house ECT research activities, with individual program elements taking

responsibility for the performance of the NRA contract, including contract deliverables and milestone monitoring.

The committee noted five recurring themes cutting across the entire PRT program that, if addressed, would strengthen the program. These are systems analysis, benchmarking and metrics, external peer review, stability and continuity, and investment portfolio.

A crucial part of portfolio management, systems analysis includes competitive task selection, ongoing refinement, and redirection as technical progress is made in a program. Systems analysis also leads to an awareness of system-level impacts of individual technologies under development. The committee observed gaps in system-level awareness and systems analysis capability throughout the PRT program, from top to bottom. Methods for risk assessment were not widely used or understood. Yet, pockets of systems analysis were found within the program, typically in the areas of excellence. For example, the Energetics project within the ECT program has effectively used high-quality systems analysis for much of its work to guide research efforts toward the critical highest-payoff technical challenges on the system level. It is the committee’s understanding that the Technology Assessment Analysis process within the ECT program is being developed to address a portion of this need; however, at this point TAA is a capability that does not yet exist.

Systems analysis capability that covers a range of fidelity—from back-of-the-envelope to parametric excursions of specific point designs—should be employed. Awareness of system-level impacts should be encouraged down to the level of individual projects and researchers as a mechanism for ensuring that research goals retain their original desired relevance. Such analyses should vary in complexity; in some cases, a simple calculation suffices, but in others a more advanced state-of-the-art analysis is needed.

Benchmarking establishes quantitative goals or expectations that serve as technical measures of success. These objective targets are expressed at the discipline, component, subsystem, and system levels, tied together by systems analysis. Excellent projects and tasks within the PRT program always developed methodologies and goals from meaningful technical benchmarks and subjected their research progress to external assessment with appropriate metrics. These benchmarks were supported by analyses, where appropriate, and developed from basic principles (of mechanics, physics, and so on). The Space Communications project within CICT is an excellent example of how setting and using proper metrics can enhance a research program. The project’s tasks had clearly defined goals for even the most basic research. Both the Advanced Measurement and Detection element and the Energetics project within ECT also exemplify this characteristic.

Each program element and task should, in conjunction with element and program managers, establish technical benchmarks that are supported by analyses from basic principles. These metrics should be tempered with realistic engineering considerations and should be used to devise research methodologies based on consistent scientific methodology. Used correctly, these metrics can enable a useful assessment of long-term progress and results in the context of their application.

Interaction with external peers comes in a number of different forms, all of which should be encouraged throughout the research life cycle. Before research is initiated, external peer reviews are used fairly effectively in the competitively selected external portion of the PRT program, but apparently not at all in competitively selecting in-house research projects. Furthermore, as in-house research is being conducted, there is limited involvement of external peers in evaluating the content and output of projects. Finally, as mentioned in the previous sections on each program, there is an inconsistent priority placed on encouraging publication in peer-reviewed technical journals. As observed by the panels, there is a clear correlation between high-quality work and tangible results presented in peer-reviewed publications and manifested in deliverable flight hardware and software. For example, in the Resilient Materials Structures element within ECT about 80 percent of the publications are from two of the nine tasks. These two tasks were judged by the panelists to be of high quality.

The PRT program should institutionalize an external peer review process in all aspects of the research and technology enterprise: task selection (including the in-house portion of the program), ongoing progress reviews, and final assessment of results. This peer review process would:

• Increase the quality of program planning processes,

• Increase communication across groups within NASA,

• Increase communication with researchers outside the agency, and

• Reduce unintentional overlaps of research with ongoing academic and commercial research.

It is important for the credibility and success of such a review that an appropriate number of non-advocate reviews and reviewers be used.

Changes in priority, organization, and funding will always occur and should be expected in a dynamic research program. However, the PRT program has undergone frequent and sometimes disruptive restructuring and reorganization, often based merely on advocacy hype. Some of these changes appeared to be destructive rather than a reallocation of resources as a natural part of research progress and maturation. For example, portions of the program have been managed by five different enterprises within NASA during the past 10 years. A link can be made between the stability of a project in this regard and the project’s technical performance over a long time

horizon. This is especially so for the more fundamental and challenging research tasks, in which fundamental advances in science and engineering are required.

The committee recognizes that certain current program time spans are imposed by the Office of Management and Budget. However, these program constraints apply 5-year time horizons, whereas the program at hand has experienced reorganization at 1- and 2-year intervals. NASA should strive to redirect programs based on sound technical issues and progress. It should avoid organizational churning and stutter-step reprogramming motivated by advocacy or external pressure. NASA management and the technical team must share responsibility in providing stability and continuity for the management of inevitable change.

Projects with the PRT program provide examples of why a process for effectively managing change is needed. The Advanced Measurement and Detection element within ECT is exemplary in its well-structured process for selecting and maturing technology through instrument development and transition to application. This process has led to the successful integration of instruments in NASA missions despite the management and organizational change endured by the element. This element within ECT and other programs of excellence within NASA have internally adopted management practices that can accommodate the more frequent reorganizations at the top. They have achieved progress in spite of the reorganizations, not because of them.

The committee recognizes that a large portion of the PRT program appropriately contains evolutionary technology. Only a few stretch, high-risk research efforts were observed—those that when successful disrupt conventional thinking and open up the possibility of new approaches, missions, and systems. Although the program is investing in some so-called revolutionary areas (such as nanotechnology and quantum computing), the committee notes that the selection of a research topic perceived as revolutionary does not necessarily mean that the NASA research done on the topic is of high quality or high potential relevance to NASA. Also, the committee noted that there was some high-quality research, very relevant to NASA missions, that was more evolutionary or even pedestrian by some accounts. These efforts support a core competency of technical expertise within NASA-unique capabilities and needs. For this reason, the committee urges NASA to ensure that it selects research projects on the basis of the quality of the research and its relevance to NASA, independent of whether it is perceived as revolutionary. That said, the committee also believes that the PRT portfolio should exhibit more tolerance for taking on stretch goals that yield potentially high-payoff results in areas where NASA can have a unique impact.

The committee also observed that some useful technology becomes caught between the end of PRT support (at a lower technology readiness level (TRL)) and the start of user support (at a mid to high TRL). Every effort should be made to work with the user enterprises of NASA and industry to develop continued co-sponsored funding to facilitate a smooth transition of technology to mission applications. As successful research efforts mature, specific transition funding should be allocated jointly between Code R (PRT) and the user enterprise. The committee notes that a few projects within PRT have an effective process for making this transition of new technology to the successful production of mission hardware. The JPL autonomous robotics work in CICT and the Advanced Measurement and Detection element in

the ECT program both transition technology successfully. For AMD, an enduring, well-defined process exists that allows a natural transition through mid-TRL instrument development programs such as the Planetary Instrument Definition and Development Program (PIDDP) and the Instrument Incubator Program (IIP). Recent examples include the microshutter array that is now baselined for the future James Webb telescope and the microthermopile array for the Mars Climate Sounder instrument on the Mars Reconnaissance Orbiter. Cost-sharing of transitional research is a goal of the ECT program and is used quite frequently. This should be continued and expanded beyond ECT.

These are some of the top-level observations and recommendations based on the committee’s review of NASA’s PRT program. These highlights, summarized in our meeting with you on November 8, 2002, are offered here as interim documentation of that oral feedback. As mentioned above, a detailed report will be issued late in 2003 by the NRC.

Sincerely,

Raymond S. Colladay,

Chair

Committee for the Review of NASA’s Pioneering Revolutionary Technology (PRT) Program