Decadal Survey of Civil Aeronautics: Foundation for the Future (2006)

The study began in September 2005 with a joint kick-off meeting between the steering committee and its supporting panels in order to hear directly from NASA and other federal entities the primary purpose of the study. Committee and panel members were also briefed on a recent report of the National Institute of Aerospace (NIA, 2005) and an earlier NRC study, Aeronautical Technologies for the Twenty-first Century (NRC, 1992).

At the second steering committee meeting, in November 2005, representatives from industry and academia were consulted in a roundtable discussion. The steering committee then developed a framework for the study, Strategic Objectives, and guidelines for the panels. It developed a quality function deployment (QFD) process, described below, for the panels to use. It identified, defined, and weighted the Strategic Objectives as well as the Why NASA? criteria. Finally, it outlined some basic rules and conventions for completing the prioritization process, all of which are summarized below.

From November 2005 through January 2006, each panel held a series of meetings. The panels identified and consulted a broad range of experts with backgrounds in industry, government, and academia. Many of them were able to attend panel meetings (see Appendix I). Working among themselves, the panel members then developed lists of research topics, called R&T Challenges. In some cases, these lists were very long, exceeding 100 items for a single R&T Area. Because it was not feasible to describe and prioritize so many Challenges in detail, the panels winnowed their lists, first by dropping those that seemed to have very little relevance to the Strategic Objectives. The number of Challenges was further reduced (to a total of 89 among all five Areas) by increasing the breadth of many of them, so that several very specific R&T topics could be collected into a single Challenge. Each panel, working under the oversight of the Steering Committee, then used the QFD methodology to relate these Challenges to the Strategic Objectives, generating a list of the 10 highest-priority Challenges within their Area.¹ All five panels considered issues related to subsonic, supersonic, and hypersonic flight regimes; infrastructure; transformation of the air transportation system; workforce; and education.

At the final meeting of the steering committee, in February 2006, it compiled inputs from the panels, vetted the prioritized list for each R&T Area, resolved conflicts in scoring among panels that had considered similar technologies, identified common themes among R&T Challenges from more than one R&T Area, and reached consensus on the overall content of the report, including summary findings and recommendations.

The steering committee directed the panels to use a modified QFD approach to rank the R&T Challenges they identified. QFD is a group decision-making methodology often used in product design. It is very useful for evaluating choices given a specific set of values. Cross-cutting research tends to rank highly, because it helps achieve multiple Objectives. The QFD scores described in this report for each R&T Challenge have no absolute, quantitative value. Rather, the QFD process serves as an organizational system that consistently evaluates each R&T Challenge and clearly conveys the rationale for the priority assigned to it. It is a qualitative process that utilizes the judgments and wisdom of informed experts to achieve a collective ranking of disparate objects.

The QFD process for this study used a matrix like the one shown in Table 2-1. The primary evaluation criteria are the six Strategic Objectives.² The R&T Challenges to be prioritized appear in the left-hand rows.³ Each panel, as a group, scored each R&T Challenge with respect to the individual Objectives, based on its relevance and impact. Possible scores are limited to 1, 3, or 9. As is often done in QFD exercises, a nonlinear scale is used to magnify the differences in technologies to help delineate the most critical ones. A score of 1 implies that the Challenge has little or no relevance to the Objective. A 3 implies that the Challenge has moderate relevance and impact. A 9 implies that the Challenge has major relevance and impact.

The steering committee assigned each of the six Strategic Objectives a weight of 1, 3, or 5 to convey its relative importance to U.S. civil aeronautics research. The committee believes that the first two Objectives, capacity and safety and reliability, are the most critical because of their broad impact on the air transportation system as a whole, the vital importance of safety, and need to meet growing demand, and assigned them a weight of 5. The next two Objectives, efficiency and performance and energy and the environment, directly affect certain stakeholders and indirectly affect the public as a whole through their secondary effects on capacity and safety and reliability. They are considered to be slightly less important overall and are assigned a weight of 3. Finally, synergy with national and homeland security and support to space are assigned a weight of 1. Neither of these Objectives falls directly under the purview of civil aeronautics. Even so, security and the space program are important to the nation, and all other things being equal, civil aeronautics research that also provides benefits for these two Objectives should be of somewhat higher priority than comparable research that does not provide benefits for them.

The weight for each Strategic Objective (1, 3, or 5) is multiplied by the relevance and impact score (1, 3, or 9), which describes the impact on that Objective of research in a particular R&T Challenge. The sum of those products for each R&T Challenge then becomes the national priority score for that R&T Challenge. That score is a measure of the relative overall value to the nation of conducting research to overcome that particular R&T Challenge.

Every R&T Challenge that has a high national priority does not necessarily become a high priority for NASA’s civil aeronautics research program. To determine the NASA priority scores, each R&T Challenge is given a Why NASA? score, which is multiplied by the national priority score to arrive at a NASA priority. The Why NASA? score for each R&T Challenge is the average of the scores (1, 3, or 9) in the four Why NASA? columns on the right-hand side of the QFD tables. These scores evaluate each R&T Challenge in terms of the following:

• Supporting infrastructure

• Mission alignment

• Lack of alternative sponsors

• Appropriate level of risk

The scores used to assess priorities are based on the current situation, which will change. For example, this study did not attempt to predict how NASA expertise and facilities in various areas might degrade or mature, how NASA’s aeronautics mission might be redefined, how the priorities of other research organizations might change, or how advances in the state of the art might change the risk associated with specific R&T Challenges. Changes in any of these factors will change the Why NASA? scores, which will directly change the NASA priority scores.

Supporting infrastructure refers to whether NASA already possesses the facilities, resources, and expertise to conduct research related to an R&T Challenge. A score of 1 implies that NASA has little or no relevant infrastructure. A score of 3 implies that NASA has infrastructure that is relevant but not unique. That is, industry, academia, or non-NASA federal agencies possess similar infrastructure or could obtain it easily. A score of 9 implies that NASA has infrastructure that is both relevant and unique.

Mission alignment refers to whether research related to the R&T Challenge falls under NASA’s charter, as defined in the National Aeronautics and Space Act of 1958 (As Amended). Relevant portions of the Space Act appear in Box 2-1. A score of 1 implies that the Challenge has little or no relevance to any item in the charter. A score of 3 implies that it has some relevance to and impact on one item in the char-

ter. A score of 9 implies that the Challenge has great relevance to and impact on at least one item in the charter or some relevance to and impact on multiple items.

Lack of alternative sponsors refers to whether other sponsors are able and willing to perform the necessary research. NASA should not be repeating research that is (or should be) done by others. A score of 1 implies that if NASA did not do the research, some other organization would do it, or does already. A score of 3 implies that if NASA did not do the research, it would be done but not be developed to an adequate level of maturity, or it would lack aeronautical focus. A score of 9 implies that if NASA did not do the research, it would not be done.

Appropriate level of risk refers to whether the level of risk associated with an R&T Challenge is appropriate for a NASA research project. For example, NASA should not pursue incremental research that is of such low risk that industry could easily complete the research. Nor should NASA pursue research of great theoretical promise if the scientific and technical hurdles are so high that it has very little chance of success. A score of 1 implies that the Challenge is either very low risk (such that industry could pursue it) or extremely high risk (such that there is only a small chance of seeing any benefit without unforeseen revolutionary breakthroughs). A score of 3 implies that the Challenge either has low risk or very high risk. A score of 9 implies that it has moderate to high risk, which is a good fit to NASA’s level of risk tolerance. All NASA research should be expected to progress toward established goals, but innovation is not possible without tolerance for failure, and the pursuit of moderate-and high-risk technology is appropriate for the nation’s center of excellence for aeronautics.

The top 10 R&T Challenges for each Area, in priority order, are discussed in Chapter 3. All the Challenges are discussed in Appendixes A to E, which also contain specific milestones. The technical discussions and milestones included in this report are intended to be advisory, as it was not feasible to complete a rigorous, comparative assessment of all of the research options that might be associated for each of the 89 Challenges. The committee believes that the best approach for selecting specific research projects to fund would be for NASA to solicit proposals from industry and academia at the level of the individual Challenges.

The QFD process appears to be a rigorous quantitative process, with strict, laid-out criteria for each score. However, while each panel could consistently distinguish between what deserves a 3 and what deserves a 9, for example, some variations from panel to panel were inevitable. Furthermore, QFD is an iterative process. After initially scoring each R&T Challenge, panel members examined their results, assessed the justifications for each score for internal consistency and accuracy, and then adjusted some scores and justifications, as appropriate.

Once each panel completed the QFD process for its R&T Area, the steering committee reviewed the results and raised issues for the panels to reconsider to assure that the results were generally consistent when two panels had similar R&T Challenges. In the end, the panels and the steering committee concurred that (1) the Strategic Objectives were properly defined and weighted and (2) the Challenges were correctly scored and prioritized. Thus, although the steering committee reserved the right to change QFD scores without the concurrence of the panels, it did not find such action necessary.

The steering committee could have attempted to create a single integrated priority list of the R&T Challenges from all five R&T Areas. However, it was not practical for the committee to make extensive pairwise comparisons to assure that the scores for each R&T Challenge from each panel were consistent with the scores for dissimilar R&T Challenges from other panels. The steering committee also considered the value of having a single list of priorities and satisfied itself that (1) the results from each panel were generally consistent and well justified; (2) the high-priority R&T Challenges in each R&T Area were, indeed, high-priority items that should be included in NASA’s aeronautics R&T program; and (3) the ultimate purpose of prioritizing R&T Challenges is presumably to determine which Challenges will be funded, and that determination will depend upon budgetary factors that were beyond the scope of this study (see Appendix G).

Given the above considerations, instead of creating an integrated, prioritized list of R&T Challenges from all five panels, the steering committee decided that the best use of the limited time and resources available to complete the study would be to identify Common Themes and formulate overall findings and recommendations (see Chapters 4 and 5). Given this situation, readers are cautioned against comparing the national and NASA priority scores for R&T Challenges from different panels to determine which is more important. The steering committee firmly believes that NASA should support research in all five R&T Areas, and the priorities identified in this report can be relied on to guide research planning within each of those areas.

National Institute of Aerospace (NIA), National Strategy Team. 2005. Responding to the Call: Aviation Plan for American Leadership. Available online at <www.nianet.org/nianews/AviationPlan.php>.

National Research Council (NRC). 1992. Aeronautical Technologies for the Twenty-first Century. Washington, D.C.: National Academy Press. Available online at <http://books.nap.edu/catalog/2035.html>.