2
Process for Integration and Prioritization

STUDY PROCESS

The study began in September 2005 with a joint kick-off meeting between the steering committee and its supporting panels in order to hear directly from NASA and other federal entities the primary purpose of the study. Committee and panel members were also briefed on a recent report of the National Institute of Aerospace (NIA, 2005) and an earlier NRC study, Aeronautical Technologies for the Twenty-first Century (NRC, 1992).

At the second steering committee meeting, in November 2005, representatives from industry and academia were consulted in a roundtable discussion. The steering committee then developed a framework for the study, Strategic Objectives, and guidelines for the panels. It developed a quality function deployment (QFD) process, described below, for the panels to use. It identified, defined, and weighted the Strategic Objectives as well as the Why NASA? criteria. Finally, it outlined some basic rules and conventions for completing the prioritization process, all of which are summarized below.

From November 2005 through January 2006, each panel held a series of meetings. The panels identified and consulted a broad range of experts with backgrounds in industry, government, and academia. Many of them were able to attend panel meetings (see Appendix I). Working among themselves, the panel members then developed lists of research topics, called R&T Challenges. In some cases, these lists were very long, exceeding 100 items for a single R&T Area. Because it was not feasible to describe and prioritize so many Challenges in detail, the panels winnowed their lists, first by dropping those that seemed to have very little relevance to the Strategic Objectives. The number of Challenges was further reduced (to a total of 89 among all five Areas) by increasing the breadth of many of them, so that several very specific R&T topics could be collected into a single Challenge. Each panel, working under the oversight of the Steering Committee, then used the QFD methodology to relate these Challenges to the Strategic Objectives, generating a list of the 10 highest-priority Challenges within their Area.1 All five panels considered issues related to subsonic, supersonic, and hypersonic flight regimes; infrastructure; transformation of the air transportation system; workforce; and education.

At the final meeting of the steering committee, in February 2006, it compiled inputs from the panels, vetted the prioritized list for each R&T Area, resolved conflicts in scoring among panels that had considered similar technologies, identified common themes among R&T Challenges from more than one R&T Area, and reached consensus on the overall content of the report, including summary findings and recommendations.

PRIORITIZATION

The steering committee directed the panels to use a modified QFD approach to rank the R&T Challenges they identified. QFD is a group decision-making methodology often used in product design. It is very useful for evaluating choices given a specific set of values. Cross-cutting research tends to rank highly, because it helps achieve multiple Objectives. The QFD scores described in this report for each R&T Challenge have no absolute, quantitative value. Rather, the QFD process serves as an organizational system that consistently evaluates each R&T Challenge and clearly conveys the rationale for the priority assigned to it. It is a qualitative process that utilizes the judgments and wisdom of informed experts to achieve a collective ranking of disparate objects.

1

As noted in Chapter 3, the 11 highest-priority Challenges are identified for R&T Area A.



The National Academies | 500 Fifth St. N.W. | Washington, D.C. 20001
Copyright © National Academy of Sciences. All rights reserved.
Terms of Use and Privacy Statement



Below are the first 10 and last 10 pages of uncorrected machine-read text (when available) of this chapter, followed by the top 30 algorithmically extracted key phrases from the chapter as a whole.
Intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text on the opening pages of each chapter. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.

Do not use for reproduction, copying, pasting, or reading; exclusively for search engines.

OCR for page 9
Decadal Survey of Civil Aeronautics: Foundation for the Future 2 Process for Integration and Prioritization STUDY PROCESS The study began in September 2005 with a joint kick-off meeting between the steering committee and its supporting panels in order to hear directly from NASA and other federal entities the primary purpose of the study. Committee and panel members were also briefed on a recent report of the National Institute of Aerospace (NIA, 2005) and an earlier NRC study, Aeronautical Technologies for the Twenty-first Century (NRC, 1992). At the second steering committee meeting, in November 2005, representatives from industry and academia were consulted in a roundtable discussion. The steering committee then developed a framework for the study, Strategic Objectives, and guidelines for the panels. It developed a quality function deployment (QFD) process, described below, for the panels to use. It identified, defined, and weighted the Strategic Objectives as well as the Why NASA? criteria. Finally, it outlined some basic rules and conventions for completing the prioritization process, all of which are summarized below. From November 2005 through January 2006, each panel held a series of meetings. The panels identified and consulted a broad range of experts with backgrounds in industry, government, and academia. Many of them were able to attend panel meetings (see Appendix I). Working among themselves, the panel members then developed lists of research topics, called R&T Challenges. In some cases, these lists were very long, exceeding 100 items for a single R&T Area. Because it was not feasible to describe and prioritize so many Challenges in detail, the panels winnowed their lists, first by dropping those that seemed to have very little relevance to the Strategic Objectives. The number of Challenges was further reduced (to a total of 89 among all five Areas) by increasing the breadth of many of them, so that several very specific R&T topics could be collected into a single Challenge. Each panel, working under the oversight of the Steering Committee, then used the QFD methodology to relate these Challenges to the Strategic Objectives, generating a list of the 10 highest-priority Challenges within their Area.1 All five panels considered issues related to subsonic, supersonic, and hypersonic flight regimes; infrastructure; transformation of the air transportation system; workforce; and education. At the final meeting of the steering committee, in February 2006, it compiled inputs from the panels, vetted the prioritized list for each R&T Area, resolved conflicts in scoring among panels that had considered similar technologies, identified common themes among R&T Challenges from more than one R&T Area, and reached consensus on the overall content of the report, including summary findings and recommendations. PRIORITIZATION The steering committee directed the panels to use a modified QFD approach to rank the R&T Challenges they identified. QFD is a group decision-making methodology often used in product design. It is very useful for evaluating choices given a specific set of values. Cross-cutting research tends to rank highly, because it helps achieve multiple Objectives. The QFD scores described in this report for each R&T Challenge have no absolute, quantitative value. Rather, the QFD process serves as an organizational system that consistently evaluates each R&T Challenge and clearly conveys the rationale for the priority assigned to it. It is a qualitative process that utilizes the judgments and wisdom of informed experts to achieve a collective ranking of disparate objects. 1 As noted in Chapter 3, the 11 highest-priority Challenges are identified for R&T Area A.

OCR for page 9
Decadal Survey of Civil Aeronautics: Foundation for the Future National Priority The QFD process for this study used a matrix like the one shown in Table 2-1. The primary evaluation criteria are the six Strategic Objectives.2 The R&T Challenges to be prioritized appear in the left-hand rows.3 Each panel, as a group, scored each R&T Challenge with respect to the individual Objectives, based on its relevance and impact. Possible scores are limited to 1, 3, or 9. As is often done in QFD exercises, a nonlinear scale is used to magnify the differences in technologies to help delineate the most critical ones. A score of 1 implies that the Challenge has little or no relevance to the Objective. A 3 implies that the Challenge has moderate relevance and impact. A 9 implies that the Challenge has major relevance and impact. The steering committee assigned each of the six Strategic Objectives a weight of 1, 3, or 5 to convey its relative importance to U.S. civil aeronautics research. The committee believes that the first two Objectives, capacity and safety and reliability, are the most critical because of their broad impact on the air transportation system as a whole, the vital importance of safety, and need to meet growing demand, and assigned them a weight of 5. The next two Objectives, efficiency and performance and energy and the environment, directly affect certain stakeholders and indirectly affect the public as a whole through their secondary effects on capacity and safety and reliability. They are considered to be slightly less important overall and are assigned a weight of 3. Finally, synergy with national and homeland security and support to space are assigned a weight of 1. Neither of these Objectives falls directly under the purview of civil aeronautics. Even so, security and the space program are important to the nation, and all other things being equal, civil aeronautics research that also provides benefits for these two Objectives should be of somewhat higher priority than comparable research that does not provide benefits for them. The weight for each Strategic Objective (1, 3, or 5) is multiplied by the relevance and impact score (1, 3, or 9), which describes the impact on that Objective of research in a particular R&T Challenge. The sum of those products for each R&T Challenge then becomes the national priority score for that R&T Challenge. That score is a measure of the relative overall value to the nation of conducting research to overcome that particular R&T Challenge. NASA Priority Every R&T Challenge that has a high national priority does not necessarily become a high priority for NASA’s civil aeronautics research program. To determine the NASA priority scores, each R&T Challenge is given a Why NASA? score, which is multiplied by the national priority score to arrive at a NASA priority. The Why NASA? score for each R&T Challenge is the average of the scores (1, 3, or 9) in the four Why NASA? columns on the right-hand side of the QFD tables. These scores evaluate each R&T Challenge in terms of the following: Supporting infrastructure Mission alignment Lack of alternative sponsors Appropriate level of risk The scores used to assess priorities are based on the current situation, which will change. For example, this study did not attempt to predict how NASA expertise and facilities in various areas might degrade or mature, how NASA’s aeronautics mission might be redefined, how the priorities of other research organizations might change, or how advances in the state of the art might change the risk associated with specific R&T Challenges. Changes in any of these factors will change the Why NASA? scores, which will directly change the NASA priority scores. Supporting infrastructure Supporting infrastructure refers to whether NASA already possesses the facilities, resources, and expertise to conduct research related to an R&T Challenge. A score of 1 implies that NASA has little or no relevant infrastructure. A score of 3 implies that NASA has infrastructure that is relevant but not unique. That is, industry, academia, or non-NASA federal agencies possess similar infrastructure or could obtain it easily. A score of 9 implies that NASA has infrastructure that is both relevant and unique. Mission alignment Mission alignment refers to whether research related to the R&T Challenge falls under NASA’s charter, as defined in the National Aeronautics and Space Act of 1958 (As Amended). Relevant portions of the Space Act appear in Box 2-1. A score of 1 implies that the Challenge has little or no relevance to any item in the charter. A score of 3 implies that it has some relevance to and impact on one item in the char- 2 The QFD matrix used in this study (see Table 2-1 and the QFD matrices in Chapter 3) is a simplified form of the table (sometimes called a house of quality) that is used in a standard QFD assessment. The QFD matrix for this study has also been rotated 90 degrees from the orientation normally used to display a QFD table. The Strategic Objectives in this study take the place of the customer requirements that appear in a standard QFD table, the R&T Challenges take the place of key product and process characteristics, and the Why NASA? composite score takes the place of risk level. The national priority scores are equivalent to the absolute importance rankings in a standard QFD table, and the NASA priority scores are equivalent to risk-weighted importance. 3 Each Challenge is designated by the letter of the Area to which is belongs and by its NASA priority ranking in that Area. Thus, the R&T Challenge with the highest NASA priority in the aerodynamics and aeroacoustics R&T Area is designated A1. If two Challenges in that Area were to tie for second place, they would be listed alphabetically and designated A2a and A2b, and the next highest priority Challenge would be designated A4.

OCR for page 9
Decadal Survey of Civil Aeronautics: Foundation for the Future TABLE 2-1 Sample QFD Prioritization       Strategic Objective National Priority Why NASA? NASA Priority Score       Capacity Safety and Reliability Efficiency and Performance Energy and the Environment Synergies with Security Support to Space Supporting Infrastructure Mission Alignment Lack of Alternative Sponsors Appropriate Level of Risk Why NASA Composite Score R&T Challenge Weight 5 3 1 ¼ each X1 R&T Challenge 1 9 9 3 3 1 1 110 3 9 3 9 6.0 660 X2 R&T Challenge 2 3 9 3 9 1 1 98 1 9 3 9 5.5 539 X3 R&T Challenge 3 1 1 1 3 9 9 40 9 9 9 9 9.0 360 BOX 2-1 NASA’s Mission as Reflected by Selected Items from the National Aeronautics and Space Act of 1958 (As Amended) Section 102. Declaration of policy and purpose (d) The aeronautical and space activities of the United States shall be conducted so as to contribute materially to one or more of the following objectives: The expansion of human knowledge of the Earth and of phenomena in the atmosphere and space; The improvement of the usefulness, performance, speed, safety, and efficiency of aeronautical and space vehicles; The development and operation of vehicles capable of carrying instruments, equipment, supplies, and living organisms through space; The establishment of long-range studies of the potential benefits to be gained from, the opportunities for, and the problems involved in the utilization of aeronautical and space activities for peaceful and scientific purposes; The preservation of the role of the United States as a leader in aeronautical and space science and technology and in the application thereof to the conduct of peaceful activities within and outside the atmosphere; The making available to agencies directly concerned with national defense of discoveries that have military value or significance, and the furnishing by such agencies, to the civilian agency established to direct and control nonmilitary aeronautical and space activities, of information as to discoveries which have value or significance to that agency; Cooperation by the United States with other nations and groups of nations in work done pursuant to this Act and in the peaceful application of the results thereof; The most effective utilization of the scientific and engineering resources of the United States, with close cooperation among all interested agencies of the United States in order to avoid unnecessary duplication of effort, facilities, and equipment; and The preservation of the United States preeminent position in aeronautics and space through research and technology development related to associated manufacturing processes. ter. A score of 9 implies that the Challenge has great relevance to and impact on at least one item in the charter or some relevance to and impact on multiple items. Lack of alternative sponsors Lack of alternative sponsors refers to whether other sponsors are able and willing to perform the necessary research. NASA should not be repeating research that is (or should be) done by others. A score of 1 implies that if NASA did not do the research, some other organization would do it, or does already. A score of 3 implies that if NASA did not do the research, it would be done but not be developed to an adequate level of maturity, or it would lack aeronautical focus. A score of 9 implies that if NASA did not do the research, it would not be done.

OCR for page 9
Decadal Survey of Civil Aeronautics: Foundation for the Future Appropriate level of risk Appropriate level of risk refers to whether the level of risk associated with an R&T Challenge is appropriate for a NASA research project. For example, NASA should not pursue incremental research that is of such low risk that industry could easily complete the research. Nor should NASA pursue research of great theoretical promise if the scientific and technical hurdles are so high that it has very little chance of success. A score of 1 implies that the Challenge is either very low risk (such that industry could pursue it) or extremely high risk (such that there is only a small chance of seeing any benefit without unforeseen revolutionary breakthroughs). A score of 3 implies that the Challenge either has low risk or very high risk. A score of 9 implies that it has moderate to high risk, which is a good fit to NASA’s level of risk tolerance. All NASA research should be expected to progress toward established goals, but innovation is not possible without tolerance for failure, and the pursuit of moderate-and high-risk technology is appropriate for the nation’s center of excellence for aeronautics. NEXT STEPS The top 10 R&T Challenges for each Area, in priority order, are discussed in Chapter 3. All the Challenges are discussed in Appendixes A to E, which also contain specific milestones. The technical discussions and milestones included in this report are intended to be advisory, as it was not feasible to complete a rigorous, comparative assessment of all of the research options that might be associated for each of the 89 Challenges. The committee believes that the best approach for selecting specific research projects to fund would be for NASA to solicit proposals from industry and academia at the level of the individual Challenges. Comparing Priorities Among Different R&T Areas The QFD process appears to be a rigorous quantitative process, with strict, laid-out criteria for each score. However, while each panel could consistently distinguish between what deserves a 3 and what deserves a 9, for example, some variations from panel to panel were inevitable. Furthermore, QFD is an iterative process. After initially scoring each R&T Challenge, panel members examined their results, assessed the justifications for each score for internal consistency and accuracy, and then adjusted some scores and justifications, as appropriate. Once each panel completed the QFD process for its R&T Area, the steering committee reviewed the results and raised issues for the panels to reconsider to assure that the results were generally consistent when two panels had similar R&T Challenges. In the end, the panels and the steering committee concurred that (1) the Strategic Objectives were properly defined and weighted and (2) the Challenges were correctly scored and prioritized. Thus, although the steering committee reserved the right to change QFD scores without the concurrence of the panels, it did not find such action necessary. The steering committee could have attempted to create a single integrated priority list of the R&T Challenges from all five R&T Areas. However, it was not practical for the committee to make extensive pairwise comparisons to assure that the scores for each R&T Challenge from each panel were consistent with the scores for dissimilar R&T Challenges from other panels. The steering committee also considered the value of having a single list of priorities and satisfied itself that (1) the results from each panel were generally consistent and well justified; (2) the high-priority R&T Challenges in each R&T Area were, indeed, high-priority items that should be included in NASA’s aeronautics R&T program; and (3) the ultimate purpose of prioritizing R&T Challenges is presumably to determine which Challenges will be funded, and that determination will depend upon budgetary factors that were beyond the scope of this study (see Appendix G). Given the above considerations, instead of creating an integrated, prioritized list of R&T Challenges from all five panels, the steering committee decided that the best use of the limited time and resources available to complete the study would be to identify Common Themes and formulate overall findings and recommendations (see Chapters 4 and 5). Given this situation, readers are cautioned against comparing the national and NASA priority scores for R&T Challenges from different panels to determine which is more important. The steering committee firmly believes that NASA should support research in all five R&T Areas, and the priorities identified in this report can be relied on to guide research planning within each of those areas. REFERENCES National Institute of Aerospace (NIA), National Strategy Team. 2005. Responding to the Call: Aviation Plan for American Leadership. Available online at <www.nianet.org/nianews/AviationPlan.php>. National Research Council (NRC). 1992. Aeronautical Technologies for the Twenty-first Century. Washington, D.C.: National Academy Press. Available online at <http://books.nap.edu/catalog/2035.html>.