Assessment of NASA's Draft 2003 Earth Science Enterprise Strategy: Letter Report (2003)

1. Does the ESE draft document clearly and compellingly convey the direction of the Enterprise?

At the highest level, the ESE draft document outlines a program to study Earth as a global system, integrating technology-enabled measurements and comprehensive models to yield greatly improved predictive capabilities for climate, weather, and natural hazards. The draft document extends beyond scientific research to include the ESE program's Earth Science Applications theme, which identifies a set of application areas, partner agencies, and activities, including education. The committee finds that the ESE activities are ambitious, exciting, and critically important to the nation. However, the draft document does not clearly or compellingly articulate ESE's rationales, scope, relationships, and programmatic approaches.

Establishing and articulating the draft document's purpose are made difficult by NASA's mandated constraints on format and organization,¹² but doing so is crucial in light of the ambiguities resulting from the subtle differences between a "strategy document" and a "strategic plan." The draft document does not clearly define ESE's aims, nor does it describe its relationship to the higher-level NASA Strategic Plan, or to the lower-level documents that contain important detail on topics that the draft document only summarizes.

The ESE approach to forecasting future needs of the Earth System Science part of its program, including both basic science and activities with eventual operational use, would benefit from greater community involvement.¹³ The decadal surveys performed for the Office of Space Science are examples of such community involvement and contributions.¹⁴ The "ten year outcomes" identified as ends for research in each of the science focus areas (see the table starting at line 435 of the draft document), and in supporting documents, would benefit from greater traceability to the scientific and operational communities, including a record of the establishment of the science focus areas and their associated "prediction questions."

Committee members found the ESE draft document difficult to understand, both in the context of NASA's plans and as a stand-alone document. These difficulties appear to arise from the lack of explanatory text to guide the reader, an over-reliance on bulleted lists and complex figures at the expense of prose, and inconsistent levels of detail in the different sections of the draft document.

The committee understands that ESE must follow a common outline and chapter headings. Unfortunately, the chapter titles are not intuitive, and they provide little guidance to the reader about the content and the objectives of the text. For example, few readers have a common understanding of the differences and relationships embedded in the chapter titles "Strategic Context and Approach" (Chapter II), "Achieving NASA's Objectives" (Chapter III), and "Strategy Implementation" (Chapter IV). Further, Chapter III (the substantive description of ESE programs, objectives, and desired outcomes) begins, after a single two-line sentence, with an extremely complicated two-page table, followed, after six lines of text, by a complex figure, followed after six more lines of text by a cartoon-like figure in an appreciably different style. The chapter is made up primarily of tables, bullet lists, and schematic (but nonetheless exceedingly complex) figures, with little explanatory text.

The ESE briefing to the committee on May 6 included an extremely effective "pyramid" figure illustrating the relationship of ESE activities to overall NASA activities and defining the terminology used throughout the strategic planning document hierarchy. The briefing also included concise introductions to and summaries of each of the draft document's chapters. These briefing materials could be used to improve the ESE draft document itself.

The draft document's treatment of the ESE program's Earth System Science theme should include a clearer textual description of the six science focus areas and of the overall approaches to be pursued for attaining the objectives. The various roadmap figures in the draft cannot be understood without an accompanying explanatory text. The references to supporting documents must include more detail. Specific examples of how new technologies, improved modeling approaches, or new measurement techniques would be applied in pursuit of the science focus areas would also be helpful. These could convey the excitement and sense of intellectual challenge that are now lacking in the ESE draft document.

Establishing consistent levels of detail would both clarify the intended level of emphasis on particular topics and improve the overall readability of the document. Where appropriate, sidebars or similar formatting techniques could be used to present anecdotal examples or additional detail without detracting from the flow, structure, or logic of the main text.

It is essential that the ESE outcomes projected in the six science focus areas be viewed as realizable and attainable within the resources and time specified, and that they be desirable and practical for partner agencies. For instance, although accurate 10-day forecasts of general global weather patterns might be possible, the committee believes that useful predictions 10 days in advance of severe storms or hurricanes are not likely to be achieved within a decade. Similarly,

in most cases, accurate predictions of volcanic eruptions and earthquake probabilities are not likely within the next decade.¹⁵

ESE should carefully evaluate each projected outcome for practicality before including it in future versions of the draft document. In particular, ESE should evaluate the realizability of its "predictive capabilities enabled by an Earth system modeling capability in 2025" (line 920 in the draft document) and/or reword this section to suggest that these are hoped-for capabilities, that is, something to strive for and to use as guideposts.

The ESE draft document does not make clear whether the 10-year outcomes for each of the six science focus areas (articulated in the roadmap charts in Section III) should be construed as ESE's description of executable objectives (one of the required elements of a strategic plan identified below). With the exception of the Weather Forecast Duration Improvement activity in Section III, the 10-year outcomes (objectives) in the roadmap charts are qualitative and measurable only in a subjective sense. (In contrast, much more quantitative objectives and predictive capabilities are offered for the year 2025 [lines 920-960], but these visionary capabilities are beyond the current planning horizon, and it is not clear whether they are attainable.) In addition, a clear and concise description of the Earth Science Applications theme is not found until deep into Section III (lines 589 ff.). These top-level summaries within the ESE draft document should (and could) be inspirational rather than a dry list of goals.

Many activities being pursued and planned by ESE are well described in the draft document. Others, however, are not, even though they have the same levels of scientific and technical challenge, utilization potential, and criticality for achieving ESE objectives. In particular, the committee notes that the sheer volume of raw and processed data associated with the planned observing systems (e.g., the Integrated Observing System alluded to at line 960) is much larger than that currently used by Earth science researchers and other potential users. Moreover, ESE has developed flexible and useful data systems serving several broad communities, and it has ongoing programs aimed at revolutionizing the systems of the future. ESE has demonstrated that it has the knowledge of data set characteristics, scientific and application needs, operating methods, and management processes necessary to develop and operate these large-scale data systems. These capabilities surely fall under the rubric "as only NASA can." The committee believes, therefore, that the amount and variety of data that ESE expects to capture, analyze, archive, and distribute should be more prominently featured in the draft document.

In a recent report,¹⁶ the Committee to Review the U.S. Climate Change Science Program Strategic Plan listed the following elements as being critical to any strategic plan:

• A clear and ambitious guiding vision of the desired outcome;

• A set of unambiguous and executable goals that address the vision and broadly describe what the program is designed to accomplish;

• An assessment of whether existing programs are capable of meeting these goals, thereby identifying required program changes and unmet needs that must be addressed in subsequent implementation planning;

• A clear timetable for accomplishing the goals and criteria for measuring progress;

• A set of explicit prioritization criteria to facilitate program design and resource allocation; and

• A management plan that provides mechanisms for ensuring that the goals are met and for coordinating, integrating, and balancing individual program elements and participating agencies.

The committee finds that the ESE draft document is deficient with regard to several of these elements.

The ESE draft document provides only vague, qualitative timetables for producing the Earth system science outcomes projected between 2003 and 2014. A reader must distill what schedule information exists from an analysis of the roadmap schematics in each of the six science focus areas.¹⁷ The only listed milestones are the 10-year and 2025 deadlines; the document does not provide even approximate dates for completion of intermediate tasks.

The Earth Science Applications component of ESE's program as presented in the draft document lacks roadmaps¹⁸ and thus does not have even schematic timetable information. On the positive side, the program has an explicit benchmarking and assessment component, and it is reasonable to expect that these components might be useful for measuring progress. The 2002 NRC review of the ESE Applications Plan concluded that "the Applications Plan needs to articulate a strategy for translating concepts into more tangible actions." ¹⁹ The committee echoes that conclusion.

The draft document does not provide sufficient high-level information on criteria or processes for establishing priorities. This is a critical problem because of the broad scope and ambitious nature of ESE's activities, the evident gap between available resources and the stated objectives,²⁰ and the reliance on interagency and international partnerships (a laudable approach in the committee's view). Although the first bulleted item (lines 329-332) of the draft document addresses "identifying and prioritizing frontier science questions" as an essential component of the ESE framework, the draft document describes neither the criteria nor the processes for setting relative priorities among the six science focus areas. Similarly, the draft document gives no indication of the relative priorities of the three elements of the ESE's Earth Science Applications theme, nor does it prioritize between the ESE program's Earth System Science and Earth Science Applications themes.

The ESE draft document touches only briefly on prioritization processes in Section IV, "Strategy Implementation." Although an ESE Program Management Council will presumably be the ultimate forum that sets priorities, the document is silent on the composition of the council and the criteria it will apply. What little information is provided (lines 866-873) appears to be specific to space missions and has a strong focus on mission or project initiation. Given the decadal extent of the draft document, re-ranking exercises must be expected in response both to externally imposed perturbations (such as the budget process) and to opportunities and challenges resulting from unanticipated discoveries and new understanding.²¹

The need for prioritization criteria and processes becomes even more urgent in light of ESE's commitment to "align our science priorities with those of major national imperatives," like the U.S. Climate Change Science Program, the U.S. Climate Change Technology Program, the U.S. Weather Research Program, initiatives for a reduction of vulnerability to natural disasters and for homeland security, the National Ocean Partnership Program, and the Study of Environmental Arctic Change (lines 437-443),²² and in the context of the many different interagency partnerships and collaborations identified as required for achieving ESE objectives. In light of these deficiencies, the committee recommends that ESE carefully examine the draft document to ensure that it contains all of the elements to successfully and compellingly describe its strategies.

2. Does the ESE draft document effectively respond to the NASA Strategic Plan?

The draft document embraces and is in many ways responsive to the NASA 2003 Strategic Plan. The draft document states that ESE's role in the NASA vision and mission is "to improve life here" and to "understand and protect our home planet." It also conforms to NASA's overarching principle to engage only in those activities to which NASA can make unique and critical contributions.

However, the draft document is weaker in its discussion of how ESE contributes to NASA's goals. For example, the table at line 399 is presented without any explanatory text, which detracts from the central importance and excitement of ESE's contributions. The draft document is also lacking in a substantive discussion of management approaches and processes and thus is unresponsive to NASA's implementing strategy, to "achieve management and institutional excellence comparable to NASA's technical excellence."²³

The committee strongly endorses NASA's open competition approach, but it is not clear to what extent peer review and open solicitations can be used to manage the complex set of activities and objectives summarized schematically in the roadmaps. Finally, the ESE draft document would be stronger if it summarized the criteria for measuring progress and referred to underlying documents that describe the criteria and processes in greater detail.

The committee notes ESE's contributions to both of the goals under NASA's third mission, "inspire the next generation of explorers," and considers that ESE's record in training

the next generation of researchers is strong and deserves credit. But there are no examples of any such accomplishments in the short section "Earth Science Education" (line 690), which is also incomplete because it focuses primarily on outreach and K-12 education and does not describe how ESE will continue to contribute to the development of the next generation of explorers, a stated goal of NASA's agency-wide Education Enterprise. The committee recommends that the ESE draft document describe a vision for a strong partnership between NASA and universities with regard to long-term education needs and the intellectual development of the next generation of Earth system scientists.

To demonstrate the alignment of the ESE program with the NASA Strategic Plan, the committee recommends that the ESE draft document clearly state the NASA goals to which ESE contributes and that it link ESE objectives clearly to the agency's goals. Once the NASA goals have been clarified, the specific ESE objectives should be introduced, and the document should make clear how each ESE objective supports the NASA goals. While the six ESE science focus areas on predictive capability appear to be introduced in Section II (lines 231-243), it is not clear how each supports one or more of the four NASA goals that the draft document must support. These six focus areas are the centerpiece of ESE's implementation strategy. Their relevance to NASA's goals should be explicitly and unambiguously stated in Section II. Later, in Section III, additional detail should be provided. Finally, in Section IV, the strategies for pursuing each focus area and their interactions should be described.

The bulleted lists "Earth Observation and Data Management" and "Advanced Technology" (lines 524-588) present a combination of objectives and implementing strategies for ESE programs. Explanatory text describing the intent of the bulleted lists would improve the draft document's readability. In some instances, ESE has developed innovative and successful approaches for achieving the stated aims.²⁴ Other areas²⁵ present significant challenges. The draft document would benefit greatly from a more extensive discussion of the past progress and anticipated future challenges associated with each of these implementing strategies.

3. Does the ESE draft document describe an endeavor that stands as an important scientific program and makes needed contributions to broader national priorities?

The ESE draft document describes a program that has two parts, Earth System Science and Earth Science Applications, and that is clearly important both scientifically and in terms of its contribution to broader, societally relevant national priorities. However, because many of ESE's objectives are qualitative their value may be difficult to assess. The draft document would convey a better understanding of future progress if it were to include some descriptive examples of how existing ESE programs have already helped to improve the capability for predicting the effects of Earth system processes. In addition to highlighting the need for new approaches, such

examples would demonstrate clearly the high level of understanding that ESE has of both its present program and its future objectives.

Both the NASA Strategic Plan and the ESE draft document emphasize activities that directly support applications (primarily predictions of climate, weather, and natural hazards). Indeed, the draft document justifies its programs almost exclusively on the basis of potential societal benefit but says little about the exciting fundamental science that will be necessary to achieve its objectives. The committee believes that effective progress can be made only if fundamental and applied research are balanced appropriately.²⁶

Answering specific, societally relevant questions about changes in the Earth system will require investigations of many basic physical, biological, and biogeochemical processes that are not currently understood. Furthermore, understanding how these processes orchestrate the evolution of the Earth system at different time scales is an essential precursor to the development of a scientific framework for the exploration and search for life in the universe-one of the great challenges facing all of science. Progress in this area requires the pursuit of basic knowledge. However, in contrast to some other areas of science, Earth system science has great societal relevance, so it is essential that NASA and the Earth science community pursue parallel and strongly interactive paths, basic and applied, the products of which will add to the body of scientific knowledge and also contribute to the improvement of life on Earth. Considering the unique relationship between basic and applied research in Earth systems science, the committee recommends that the ESE strategy document clearly describe the respective roles of applied and basic science and the appropriate balance between the two.

The six science focus areas are important and challenging. However, the reader is left with questions as to how and why these six were selected. A concise summary of the planning processes that resulted in the six science focus areas and the 12 applications focus areas, along with references to supporting lower-level documentation, would strengthen the draft document by demonstrating ESE's expertise at managing planning processes involving input from broad and diverse communities of implementers and stakeholders.

ESE's strategy for accomplishing its applications goals involves partnerships with other government agencies. The committee supports this strategy but believes it to be incomplete. ESE's ambitious goals for predictive capabilities will also require strong and effective collaborations with entities outside government, including universities, nonprofit organizations, and the private sector. Indeed, new paradigms may be required to achieve ESE's objectives, for example, a paradigm that considers partnerships to be only one element of a larger strategy that would also include, for example, the powerful effects of markets as agents for change.

The NASA applications strategy should emphasize one of the agency's strengths, which is the provision of objective, scientifically derived environmental information to all parties, inside or outside governments, in a neutral, nonpolitical mode. Although the ESE draft document (lines 814-854) addresses the importance of existing external partnerships, it does not describe how the new partnerships required to achieve the stated objectives will be established. In addition, the draft document does not detail how existing partnerships will be exploited. The

committee recommends that this strategic element be added to the ESE draft document in the section on external partnerships (lines 814-854) and that the text in this section be revised accordingly.

4. Does the ESE draft document provide appropriate attention to interdisciplinary aspects, integration of technology development, and overall scientific balance?

The draft document emphasizes many interdisciplinary aspects, beginning most importantly with the compelling need to treat Earth as a single system consisting of continents, atmosphere, oceans, ice, and life and extending to the influence of solar variability on Earth in collaboration with NASA's Space Science Enterprise. Each of the identified six science focus areas is, independently, also fundamentally interdisciplinary. Together, the focus areas constitute the physical, biogeochemical Earth system. However, some important elements of global Earth science appear to be excluded from the ESE program (e.g., the population-driven contributions to land cover change), possibly because they are addressed in the programs of other agencies. The ESE draft document should describe how NASA will work with other organizations, public and private, to ensure that the full spectrum of important topics is addressed and should make clear how ESE will use the results of work done by other organizations.

The ESE draft document is less successful in addressing the integration of technology development and overall scientific balance. Discussion of technology development is limited to a brief list of strategic elements (lines 570-588), lists of technology requirements needed to support future missions in the focus area roadmaps and the 2025 vision (lines 964-1025), and passing mention of the New Millennium Program for technology flight demonstrations. The draft document could be strengthened by including a discussion of the elements and relative roles of ESE's Earth Science Technology Program and the Aerospace Technology Enterprise's Mission and Sensor Measurements Technology Program in the development of new technologies to support future ESE missions.

The committee has pointed out the importance of ensuring an appropriate balance between basic and applied research and notes here several changes that would improve the ESE draft document's scientific balance. The draft document emphasizes missions, measurements, modeling, and prediction, with less attention to the integration of aircraft, laboratory, and field programs. As noted in several places above, although the draft document lists important science questions (lines 1032-1078), it includes very little discussion of how the science will be addressed, how the questions are (or will be) prioritized, and when answers are expected.

The draft document would be improved by a description of how the Earth System Science Pathfinder program will be strategically coupled to ESE's set of defined missions. While such a description might be too detailed for an overview of this kind, including it in the draft document would emphasize the role of technology and hardware in achieving ESE's objectives.

It also would exemplify how NASA will incorporate new concepts rapidly into measurement programs, leading to exciting, cutting-edge programs.

The ESE draft document should demonstrate that important disciplinary science has not been omitted, perhaps by describing each science focus area in more detail. Replacing the bulleted and tabular material between lines 416 and 436 with descriptive text, illustrative of the richness and complexity in each focus area, would be beneficial and more effective. Breadth and inclusiveness are essential in ESE's strategy document, which needs to convey the full range of possible scientific components and interactions.

In the draft document, solid Earth science stands somewhat apart from, and independent of, the other science focus areas. Text describing the solid Earth focus and its connections to the other focus areas would help the reader to understand why Earth must be studied as a system. Connections between seemingly unrelated activities such as the precise measurement of gravitational fields and hydrology/precipitation would help the reader understand the synergies between solid Earth science and the other science focus areas. The importance of ESE's geodetic infrastructural instrumentation, such as its Global Positioning System, Satellite Laser Ranging, and Very Long Baseline Interferometry capabilities, for pursuing such objectives as well as other applications as diverse as monitoring ocean level and atmospheric temperatures and weather forecasting should be emphasized.

It is also important that the description of the science in each focus area reflect the complete range of activities required to perform global Earth science. The existing descriptions do not accomplish this. In particular, the focus area titles (as in the table starting at line 435) are in some cases not well matched to the schematic information included in the roadmaps. Deficiencies identified in the roadmaps for each of the six science focus areas include the following:

• Climate Variability, Change, and Prediction. This focus appears to be strongly oriented toward long-term climate change. A more careful balance including science associated with seasonal variability and other time scales would be valuable.

• Atmospheric Composition. While ozone-related chemistry is critically important, other topics in chemistry and atmospheric composition science, including such things as aerosols and sulfur dioxide, are also extremely important yet are not described in the ESE draft document.

• Ecosystems and Carbon Cycle. While the carbon cycle itself is properly represented as important science, both the role of ecosystems within the carbon cycle and the science of ecosystems external to the carbon cycle (including other biogeochemical cycles) are underrepresented.

• Water and Energy Cycle. The emphasis on the global aspects of the water and energy cycle results in insufficient recognition that regional-scale effects are central to many key water and energy issues, including flooding and drought.

• Weather. This focus area appears to be complete and balanced.

• Earth Surface and Interior. The emphasis on natural hazards, while important, appears to exclude basic understanding of solid Earth physics and processes.

The committee finds that analysis of the measurement data is not emphasized sufficiently to indicate its central importance in ESE's end-to-end strategy. Because the productivity of any particular satellite or satellites depends as much on the readiness of software and data analysis

tools as it does on the readiness of sensor and spacecraft hardware, ESE leadership must "be the advocate for the appropriate balance of investment in data analysis"²⁷ and should "provide strategic planning and oversight concerning the collection, processing, archiving, and dissemination of data and information collected by NASA's space missions."²⁸

During the discussion at the May 6, 2003, committee meeting, ESE officials stated that the allocation of funding between universities and NASA centers is based entirely on open competition. The committee believes that this important strategic principle should be explicitly stated in the ESE draft document. The importance of open competition could be included as part of a general discussion of the principles for the overall ESE strategy.

The relative allocation of resources among various activities represents another important aspect of program balance. While detailed budgets are not a necessary part of the ESE document, a discussion of the relative investments in space observations, data analysis and research, applications, and education would be helpful.

The committee is pleased with the frequent references to international collaborations and partnerships but believes it would be appropriate to mention how NASA develops these strategically. For example, ESE might wish to mention NASA's participation in the Integrated Global Observing System Partnership, the Committee on Earth Observing Systems, and other international forums. Strategic considerations include deliberate redundancy, scientific and technical expertise, and cost-effectiveness.

Earth system science is a complex, highly integrated activity requiring critical interaction among the many agencies and organizations responsible for performing particular elements of the scientific research. ESE recognizes this and has indicated clearly that the Earth "systems" science problem requires a "systems" solution. The committee recognizes that systems approaches may not be as amenable to scientific research as they are to engineering development. Nevertheless, many of the concepts inherent in systems engineering, particularly those related to the coordination of multiple and distinct efforts, can be applied to complex scientific research programs as well.

ESE has appropriately recognized that its contribution is in areas within the rubric "as only NASA can" and includes the role of leading or motivating partnerships to accomplish activities that fall at least partly outside the ESE charter. The committee is concerned that ESE's ability to achieve broad Earth science objectives may be compromised because no single organization is responsible for coordinating the efforts of the many partners. In light of this concern the committee recommends that ESE describe how it will take the steps necessary to ensure that all important scientific components are identified, and that NASA work with other relevant agencies to ensure that the specific roles of partner organizations are defined and activities properly coordinated. The committee believes that it is critical that NASA identify those areas appropriate for NASA leadership, that plans be established to ensure that leadership in other relevant areas has been assigned to and accepted by other organizations, and that mechanisms by which ESE develops or fosters the necessary partnerships be formalized.

Finally, the ESE draft document should address data availability. The draft document is silent on ESE's data policy. An inflexible open data access policy could have a significant impact on the range of partnerships ESE is able to develop, while a substantial shift to limitations on data access could impede interdisciplinary use of many data sets. An expanded discussion of the processes and procedures by which data policies are reviewed and are assessed over time would greatly enhance the utility of the ESE strategy document. The committee recommends that the revised ESE strategy document contain a brief description of ESE data policies and the mechanisms by which they will be regularly reviewed and appropriately revised.

5. What recommendations from recent NRC reports, if any, should be considered in revising the draft?

The committee's review was guided by several recent NRC reports, including Satellite Observations of the Earth's Environment: Accelerating the Transition of Research to Operations, 2003; Review of the U.S. Climate Change Science Program Strategic Plan, 2003; Review of NASA's Earth Science Enterprise Applications Program Plan, 2002; Review of NASA's Earth Science Enterprise Research Strategy for 2000-2010, 2000; and "NASA's Plans for Post-2002 Earth Observing Missions," 1999. These reports are cited elsewhere in this report. Below, the committee calls attention to several additional findings and recommendations of these and other reports that ESE may consider as it refines and improves the draft document.

Section III outlines the approach of the ESE Applications Division in working with other federal agencies to enhance nationally important decision support systems. The NRC review of the ESE Applications Program Plan²⁹ suggested that NASA expand its pool of potential partners, and a recent NRC report on the use of satellite data by local and state governments³⁰ provides valuable advice on how more widespread and beneficial use of satellite data by state and local governments can be encouraged and achieved. The ESE draft document would benefit from an explanation of why regional, state, and local applications no longer form a part of NASA's approach. Some elaboration of why the particular application areas were chosen rather than others would also improve the document.

Recommendation 5 of the NRC's Review of NASA's Earth Science Enterprise Research Strategy for 2000-2010³¹ recommends "matrices that map the detailed science questions to measurement parameters, implementation procedures, models, calibration/validation/verification requirements, and potential partners." The ESE draft document should address these issues in order to demonstrate the links between key science questions and the research missions designed to address them. This suggestion responds to a concern raised above by the committee-namely, that the strategic paths to the achievement of goals should be addressed in more detail. Material of this kind would be best included in Section IV.

Finally, the committee recommends that ESE make reference to its earlier strategic plan³²and describe how the present draft document builds on and continues that strategy,

here new emphases have been added, what they are, and how they were selected. Non-NRC documents, including internal ESE roadmaps, should also be referenced.³³