3
Principles and Metrics for Managing Effective Mission-Enabling Portfolios

The committee presents its recommendations for metrics for evaluating mission-enabling activities and criteria for making portfolio allocations in three steps: first, a set of guiding principles that should serve as the foundation for a successful mission-enabling program; second, complementary implementation principles to be considered in translating these principles into specific plans; and third, a series of metrics that could be developed and applied to verify that the balance and distribution of mission-enabling activities is able to satisfy the guiding principles.

GUIDING PRINCIPLES

  1. First and most important, the mission-enabling program, taken in its entirety, should encompass the range and scope of activities needed to support SMD strategic goals, provide the broad knowledge base that is the context necessary to interpreting data from spaceflight missions and to defining new spaceflight missions, and maximize the scientific return from all spaceflight missions.

    Mission-enabling activities are essential to addressing SMD strategic goals, and they are the research foundation on which the success of the spaceflight program rests. They also provide—through theory, modeling, and coordinated research and data analysis—the linkages among the various flight missions, and they independently add to that data to form a cohesive space and Earth science program.

    The scientific value of spaceflight missions depends on actions taken and investments made throughout the lifetime of a mission. Thus, many different components of the mission-enabling program are essential for achieving the maximum scientific return. New technologies have to be made available so that spaceflight missions can make forefront observations and measurements. New theoretical understandings need to be developed so that spaceflight missions can be designed to address the most pressing scientific questions and provide meaningful tests of new scientific ideas. The scientific results of one mission need to be compared with and analyzed in concert with results from other ongoing or past missions studying similar phenomena, to maximize the scientific return from the entire flight program.

  2. There should be a continuous flow of advanced technical capabilities and improved scientific understanding from mission-enabling activities into new spaceflight missions.

    New technologies at many steps in their development ultimately find their way into spaceflight programs. New



The National Academies | 500 Fifth St. N.W. | Washington, D.C. 20001
Copyright © National Academy of Sciences. All rights reserved.
Terms of Use and Privacy Statement



Below are the first 10 and last 10 pages of uncorrected machine-read text (when available) of this chapter, followed by the top 30 algorithmically extracted key phrases from the chapter as a whole.
Intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text on the opening pages of each chapter. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.

Do not use for reproduction, copying, pasting, or reading; exclusively for search engines.

OCR for page 31
3 Principles and Metrics for Managing Effective Mission-Enabling Portfolios The committee presents its recommendations for metrics for evaluating mission-enabling activities and criteria for making portfolio allocations in three steps: first, a set of guiding principles that should serve as the foundation for a successful mission-enabling program; second, complementary implementation principles to be considered in translating these principles into specific plans; and third, a series of metrics that could be developed and applied to verify that the balance and distribution of mission-enabling activities is able to satisfy the guiding principles. GuIDING PRINCIPLES 1. First and most important, the mission-enabling program, taken in its entirety, should encompass the range and scope of activities needed to support SMD strategic goals, provide the broad knowledge base that is the context necessary to interpreting data from spaceflight missions and to defining new spaceflight missions, and maximize the scientific return from all spaceflight missions. Mission-enabling activities are essential to addressing SMD strategic goals, and they are the research foun - dation on which the success of the spaceflight program rests. They also provide through theory, modeling, and coordinated research and data analysisthe linkages among the various flight missions, and they independently add to that data to form a cohesive space and Earth science program. The scientific value of spaceflight missions depends on actions taken and investments made throughout the lifetime of a mission. Thus, many different components of the mission-enabling program are essential for achiev - ing the maximum scientific return. New technologies have to be made available so that spaceflight missions can make forefront observations and measurements. New theoretical understandings need to be developed so that spaceflight missions can be designed to address the most pressing scientific questions and provide meaningful tests of new scientific ideas. The scientific results of one mission need to be compared with and analyzed in concert with results from other ongoing or past missions studying similar phenomena, to maximize the scientific return from the entire flight program. 2. There should be a continuous flow of advanced technical capabilities and improved scientific understanding from mission-enabling activities into new spaceflight missions. New technologies at many steps in their development ultimately find their way into spaceflight programs. New 

OCR for page 31
 AN ENABLING FOUNDATION FOR NASA’S EARTH AND SPACE SCIENCE MISSIONS technologies begin with an idea or concept, at a low technical readiness level (TRL), 1 and if successful undergo continuous refinement and improvement until they can be validated for flight. Ensuring a clear and predictable pathway from early technology development to mission-ready maturity is not always easy or straightforward. There are many opportunities to disrupt this flow and to fail to take advantage of a new capability that could achieve a scientific breakthrough. Active management of mission-enabling programs is thus required to ensure that new technical concepts are being considered and that there are means by which the most promising new technologies can be brought to a sufficiently high TRL to be incorporated into or even serve as the basis for new spaceflight missions. Theory and modeling follow a similar path. New scientific understanding begins with a concept a new theoryoften based on new observations. Such concepts then need to be incorporated into and improve comprehen - sive numerical models, which, when validated by observations, serve to provide increased scientific understanding. This portfolio of incubating ideas also needs to be actively managed, with theoretical research, modeling activities, and data interpretation all adequately supported and linked so that observations drive new theoretical concepts, and new theoretical concepts are tested against observations. 3. The mission-enabling program should enable a healthy scientific and technical workforce capable of con - ducting NASA’s space and Earth science program. Success at NASA and within a scientific discipline requires a skilled, vibrant workforce. In scientific dis - ciplines whose current practitioners entered the field decades ago, there is a need for early-career scientists and engineers. In other scientific disciplines the age distribution may be adequate, but the numbers or skills of scientists and engineers are insufficient to support the planned future program. There may be cases where it is necessary simply to rejuvenate a field with new researchers. It may also be the case that experienced scientists and engineers are being lost in an area due to temporary budget cuts and that the loss needs to be stemmed to be able to meet anticipated future demands. It is in NASA’s best interest to actively assess its internal workforce needs as well as required national capa - bilities in the scientific and engineering disciplines necessary to implement its programs. It is also appropriate that NASA take actions to deploy resources to facilitate and maintain a healthy workforce for those areas. The excit - ing programs of SMD should be widely known among students choosing careers, to encourage them to consider space and Earth science. Opportunities for undergraduate research and an adequate number of graduate fellowships should be available, as well as support for graduate students through NASA grants and contracts. Most important, the universities responsible for training undergraduate and graduate students, particularly experimentalists, should be supported to actively participate in flight programs, including the suborbital program, so that students receive essential hands-on experience that will make them valuable additions to the space and Earth science community. IMPLEMENTATION PRINCIPLES The first of the guiding principles above is applicable to each SMD science division. How the second and third principles are applied, and the priorities established among the elements of the mission-enabling program, will vary from division to division. However, each SMD science division needs a set of metrics that can be used to evaluate the completeness and efficacy of the components of its mission-enabling activities. Without such a set of metrics, it is difficult to know whether a particular resource allocation strategy is indeed appropriate, and if or when adjustments are needed to bring implementation back into alignment with the division’s overall strategic framework. The committee identified the following cross-cutting issues or implementation principles that should be considered in applying a set of metrics. 1 NASA classifies the maturity of a technology according to seven technology readiness levels (TRLs), which start with TRL 1 (basic principles observed and reported), TRL 2 (technology concept and/or application formulated), TRL 3 (analytical and experimental critical function and/or characteristic proof of concept), TRL 4 (component and/or breadboard validation in laboratory environment), and so on, up to TRL 7 (system prototype demonstration in an operational environment).

OCR for page 31
 PRINCIPLES AND METRICS FOR MANAGING EFFECTIVE MISSION-ENABLING PORTFOLIOS 1. Investment needs will be different across SMD divisions. Each SMD science division has distinct strategic goals. Some division science plans require a higher level of integration across multiple spaceflight missions; some require longer-term observations relative to those of other divisions; and the nature and expense of spaceflight missions necessarily differ across divisions. The level of investment in mission-enabling activities will also differ and will depend on factors such as the amount of basic research required to allow cost-effective missions to be defined and the extent to which such research is supported external to NASA. Consequently, there should not be a rigid set of formulas for mission-enabling program funding across SMD, and investments should reflect this reality. 2. Division-level mission statements should clearly articulate strategic priorities. SMD division goals are advanced directly (e.g., by addressing a science objective) as well as indirectly (e.g., by supporting necessary infrastructure) and are supported in both cases by the sum of its mission-enabling activi - ties. Each division’s mission statement for its mission-enabling activities should provide a rational framework for assessing how its portfolio ensures support for the full range of activities. Multiple possible components of a mission statement may include technology-readiness enhancement, development of a junior workforce, sustain - ment of a mid-career non-faculty workforce, and maintenance of critical physical infrastructure. In a constrained budget environment, these multiple components are inevitably in significant tension with one another. Given these tensions and pressures, a mission statement that identifies these components allows for their overt prioritization and active management by each SMD division. There is an important role for the external science community in helping to identify these priorities. 3. Balance between mission-enabling and spaceflight mission portfolios is never rigid. Balancing mission-enabling activities with spaceflight missions presents many challenges, the first of which is the meaning of the term “balance.” The committee does not interpret balance to mean a fixed ratio. Rather, bal - ance refers to finding the appropriate level and profile of funding for each activity within the resources available to NASA. This means that the ratio itself may not be constant over time. In general, mission-enabling activities and spaceflight missions are programmatically very different. Space - flight missions can take 10 years or more to design, develop, launch, and execute, and they have expense profiles that can vary substantially from one year to the next as a project moves through design, development, construction, and post-launch operation phases. Mission-enabling activities have objectives that are as broad and far-reaching as NASA strategic goals. They are executed by a large number of science teams undertaking individual research projects that overlap in time, run at a much lower funding level than spaceflight missions do, and often achieve success within 2 to 3 years. Consequently, the mission-enabling programs cannot sustain significant swings in funding without suffering major disruptions. 4. Programmatic relationships of mission-enabling activities to spaceflight programs should be clearly communicated. Although NASA is a spaceflight-mission-oriented agency and the mission-enabling activities must clearly relate to and support this responsibility, the connections of mission-enabling activities with flight missions are not always clearly appreciated. This relationship is on a general level and does not correspond one-to-one to specific flight missions or even specific flight programs. There may be a fear that some innovative, risky, or long-term mission-enabling activities will be particularly difficult to connect to the flight programs and thus be at financial risk in such a more open analysis. However, the committee believes that senior decision makers and the research community will be receptive to a broad range of mission-enabling activities when given the above context and when the rationale is openly articulated. Mission-enabling programs are actually more at risk when their role within SMD is not openly articulated and argued. 5. Balance within portfolios requires active management. The distribution of resources among the various mission-enabling activities reflects whether such resources are appropriately balanced within schedule and budget constraints to achieve the intended goals and objectives.

OCR for page 31
 AN ENABLING FOUNDATION FOR NASA’S EARTH AND SPACE SCIENCE MISSIONS Too much research at the expense of too little technology could jeopardize the ability to conduct a mission; on the other hand, too little research at the expense of too much technology might result in a mission that is ill-conceived or ill-defined, even if it “flies on-time.” Underinvestment in data analysis (relative to research and/or technology investments) could mean that even should a mission be successful in collecting the appropriate data, there is insuf - ficient ability to analyze that data and turn it into knowledge—the end objective. The characterization of a portfolio of projects from a near-term, mid-term, and far-term perspective determines whether the portfolio will be able to sustain the organizational mission in both the present and the future. Too much near-term investment at the expense of too little far-term investment could dry up the pipeline supporting next- generation missions even if near-term missions are successful. On the other hand, too much far-term investment at the expense of too little near-term investment could jeopardize the success of immediate missions. In a budget-constrained environment (the norm), the probability of strategic success will be significantly enhanced by measuring whether there is an appropriate distribution of resources among mission-enabling elements in the context of supporting a given mission, and by measuring whether the resources within each element have an appropriate time perspective in supporting near-, mid-, and far-term activities. Likewise, traceability of projects linked to objectives, which in turn are linked to strategies, will enhance the probability that such projects are indeed relevant and will answer the question “so what” if a particular project is successful (or unsuccessful). 6. Budget transparency facilitates active management. The current SMD budget structure does not readily lend itself to identifying the total mission-enabling invest - ment within SMD divisions (or programs within those divisions). While some portions of the mission-enabling budget are in readily identifiable budget lines, other portions are embedded in spaceflight programs and other non-mission-enabling budget lines. The lack of transparency of the mission-enabling portion of the SMD budget makes it difficult for managers and outside parties as well to analyze, advocate, and proactively manage this criti - cal activity. Developing the ability to articulate the linkages between mission-enabling activities and the (planned and potential) missions they enable is in the best interests of an organization from a strategic perspective. More specifically, such articulation will result in the identification and continuation of more relevant mission-enabling activities and the reduction or elimination of less relevant ones—the essence of strong and dynamic portfolio management. Stability is a particularly critical aspect of mission-enabling budgets, not only for those programs themselves but also for the long-term viability of the research community and for the stability of spaceflight mission programs. Mission-enabling budgets cannot be treated as fungible sources of money to be tapped to solve problems in other parts of the SMD program. Doing so can have long-enduring impacts on the overall sustainability of space and Earth sciences. METRICS In the case of an SMD division, the metric for each of its mission-enabling activities should provide: 1. A simple statement of what the component of the mission-enabling activity is intended to accomplish and how it supports the strategic or tactical plans of the division. 2. A statement as to how the component is to accomplish its task. 3. An evaluation of the success of the activity relative to the stated mission, unexpected benefits, and lessons learned. 4. A justification for the resource allocation that is being applied to the component vis-à-vis other mission- enabling activities within the division. One of the main purposes of establishing metrics for each component of mission-enabling activities is to inform the administration, the Congress, and the science community of the purpose of the component and the extent to which it is being successful. Such transparency, properly accomplished, provides justification for the essential

OCR for page 31
 PRINCIPLES AND METRICS FOR MANAGING EFFECTIVE MISSION-ENABLING PORTFOLIOS roles of mission-enabling activities in the success of SMD. It also allows the broad national science community to engage with NASA in providing the most effective mission-enabling program. If what NASA intends to accomplish is clear, the science community can participate constructively. The committee presents below a set of metrics for elements of an overall mission-enabling program. They are meant to be examples of what could be used, and not necessarily a rigid prescription. The metrics reflect a common approach to evaluating R&D programs, and often they must be more qualitative than quantitative. The advisory committee process discussed in Chapter 2 will be an essential part of the process of implementing such metrics. It is important to remember that any set of metrics should be utilized in the context of managing a total portfolio of activities and not simply applied in isolation on an activity-by-activity basis. Active management requires that one always consider how applying metrics for one aspect of a program affects the rest of the program. Metrics for Essential Components of a Broad-Based Program to Advance Strategic Goals and Maximize Science Return For Theoretical Research To ensure the continuous flow of new scientific understanding, it is necessary that a component of the mis - sion-enabling program be devoted to basic theoretical research that will serve as the basis for development of new scientific concepts, new interpretations of observations, and improved numerical models. This component is important for each SMD division, particularly those in which many new discoveries are being made. The metric for the theoretical research component of mission-enabling activities should include the following: (1) A statement of the importance of theoretical research to the SMD division. (2) The means by which new theoretical concepts are to be sought, presumably through solicitations for research proposals that are open to all organizations with particular emphasis on universities, nongovernment laboratories, and NASA centers. (3) An evaluation of how many new theoretical concepts are being introduced, and how many are ultimately validated through numerical modeling and data interpretation. (4) An appropriate allocation of resources. Since this research is the foundation on which numerical models are to be built and data interpreted, a percentage, perhaps 5 to 10 percent, of all data analysis and modeling fund - ing within the division seems appropriate. For Numerical Modeling Sophisticated numerical models are becoming increasingly important in some SMD science divisions. In general, the emphasis accorded to numerical modeling depends on whether there are multiple, comprehensive data sets to be assimilated and whether there is or will be an end use (e.g., the prediction of space weather in the case of heliophysics or the prediction of global and regional climate change in the case of Earth science). The metric for the numerical modeling component of mission-enabling activities should include the following: (1) A statement of the importance of numerical modeling to the SMD division. (2) The means by which the numerical modeling is to be supported, e.g., by the establishment of concentrated research centers or consortia at universities or national laboratories. (3) An evaluation of the success of the numerical modeling activities in increasing scientific understanding and ultimately, where appropriate, serving as the basis for end-user predictive models. (4) An appropriate allocation of resources. Since the purpose of the models is to assimilate, understand, and use data collections, the allocation of resources could be a percentage of the total funds expended for data analysis.

OCR for page 31
6 AN ENABLING FOUNDATION FOR NASA’S EARTH AND SPACE SCIENCE MISSIONS For Supporting Observations Many spaceflight missions require supporting observations on the ground or from suborbital flights to be able to plan their observations and interpret their data. Examples include ground-truth or under-flights for calibration and interpretation of Earth remote-sensing data, and ground-based telescope observations for defining and plan - ning planetary spacecraft missions. The metric for the supporting observations component of the mission-enabling program should include the following: (1) A statement of the importance of supporting observations to the SMD division. (2) The means by which the supporting observations are to be obtained, presumably through solicitations that are open to all organizations with the required capabilities. (3) A quantitative measure of the extent to which mission requirements for supporting observations are being met or to which supporting observations are providing the broad context for defining future missions and interpret - ing their data. (4) An appropriate allocation of resources. Supporting observations are intended to satisfy specific mission requirements for ongoing missions as well as serve the much broader requirement to provide the baseline of knowledge needed to understand what missions are required to cost-effectively address open science issues. Thus, the measure of success and the allocation of resources should be determined by both mission requirements and strategic goals. NASA funding for supporting observations should not duplicate funding being provided by other agencies, but neither should funding by other agencies prevent NASA from ensuring that funding for supporting observations is at a net level adequate for it to achieve its goals. For the Suborbital Program The suborbital program, including sounding rockets and balloon and aircraft flights, is important to many different aspects of SMD mission-enabling activities. It provides opportunities for both workforce and technology development, means to collect complementary observations to support spaceflight programs, and cost-effective means to conduct scientific research for which spacecraft observations are not necessary. The suborbital program should be included as a factor in metrics for other activities, such as advanced technol - ogy and workforce development and supporting scientific observations. Despite the duplication, it is appropriate for each division to also establish a metric unique to its suborbital program. The metric for the suborbital program component of mission-enabling activities should include the following: (1) A statement of the importance of the suborbital program to the SMD division, based on the contributions that the suborbital program makes to other mission-enabling activities, as well as the unique scientific capabilities it can offer. (2) The means by which the division intends to conduct its suborbital program, presumably through the solicitation of research proposals, and the establishment of a national infrastructure that supports sounding rocket, balloon, and aircraft-based research. (3) A quantitative measure of the extent to which the suborbital program is improving the workforce, intro - ducing new technologies, or conducting unique science. (4) An appropriate allocation of resources, based on the stated importance of the suborbital program, and the multiple contributions it has to make. For Cross-Mission Research and Data Analysis Activities Research and data analysis activities for a specific spaceflight mission, whether in its prime or extended mission phase, are not classified in this report as a mission-enabling activity. Data analysis activities involving

OCR for page 31
 PRINCIPLES AND METRICS FOR MANAGING EFFECTIVE MISSION-ENABLING PORTFOLIOS multiple missions, and support for researchers who are not directly involved in a particular mission science team, are classified as mission-enabling and are essential for realizing the full scientific return from the portfolio of mis - sions within SMD. This component of mission-enabling activities is particularly important for divisions in which the scientific questions are being addressed by multiple missions or where there is a need to involve researchers other than those selected as part of the mission development. The metric for the research and data analysis component of mission-enabling activities should include the following: (1) A statement of the importance of research and data analysis to the SMD division. (2) The means by which cross-mission research and data analysis activities are to be sought, e.g., through a guest investigator program that is open to all organizations. (3) An evaluation of the scientific questions that are being successfully addressed through multiple-mission data analysis and through involvement of the broader scientific community. (4) An appropriate allocation of resources. The allocation is expected to vary among SMD divisions depending on the portfolio of missions, the scientific questions being addressed, and the need to involve a broader community of researchers. Metric for Advanced Technology Development To ensure the continuous flow of new technical capabilities (the second guiding principle outlined above), it is necessary that a component of the mission-enabling program be devoted to exploring newlow-TRLtechnical concepts, which, if successful, could result in substantial improvements in planned or even newscientific space missions. This component is particularly important in divisions where advances in science will require new techni - cal capabilities, but it is also important even in divisions where adequate technology appears to be available, because it is always possible that a serendipitous discovery will lead to an unanticipated breakthrough in capabilities. The metric for the advanced technology development component of mission-enabling activities should include the following: (1) A statement of the importance of advanced technology development to the long-term success of the SMD division. (2) The means by which new technologies are to be sought, presumably through solicitations for research proposals that are open to all organizations with technical capabilities universities, NASA centers, nonprofit laboratories, and industry. (3) A quantitative measure of how many new technologies are being introduced, and how many of these are sufficiently promising to undergo additional development. (4) An allocation of resources as a percentage of all technology development activities within the division. A typical allocation for this activity for other organizations conducting R&D is 5 to 10 percent of the total technology development budget. (See Chapter 4.) Metric for Workforce Development The development of a capable workforce, both for NASA and for the outside space program community, is the third guiding principle articulated above and a singularly important task for the mission-enabling program. How - ever, it is also one of the hardest for which to establish a metric. The metric should be based on the demographics of the scientific community and on a projection of workforce needs during future decades. Such an assessment will require considerable thought on the part of the science managers of NASA, in coordination with the science community. (See Chapter 4.) The metric for the workforce development component of mission-enabling should include the following:

OCR for page 31
 AN ENABLING FOUNDATION FOR NASA’S EARTH AND SPACE SCIENCE MISSIONS (1) A statement of the importance of workforce development to the SMD division, based on a realistic analysis of the demographics of the community and expectations for future mission opportunities. (2) The means by which the division intends to satisfy its workforce needs, including providing funding for graduate fellowships, ensuring that both undergraduate and graduate education can occur in universities that actively participate in SMD programs, and supporting hardware programs that will provide hands-on opportunities to train experimentalists. (3) A quantitative measure of the extent to which the demographics and the scientific and technical compe - tence of the science and engineering communities, including the relevant NASA workforce, are being improved and maintained. (4) An appropriate allocation of resources, based on the stated workforce need and the means to satisfy the need.