Review of NASA's Earth Science Enterprise Applications Program Plan (2002)

In reviewing the Applications Plan, the committee sought to carefully evaluate the content of each section, consider the purpose of each section in the context of the whole document, and identify the strengths and weaknesses of each section in that context.

The committee found that the preface would strengthen the Applications Plan overall if this section (or a subsequent introductory section to the document) were revised to better set the stage for the rest of the Applications Plan and effectively summarize the key aspects of the NASA applications strategy. To do this, the committee first recommends the inclusion of the following “big picture” information:

1. NASA should more clearly identify the audience for the Applications Plan. For whom is it being written? Who else might read it? What does NASA expect this audience to gain from reading the Applications Plan?

2. NASA should more clearly articulate both the scope and the boundaries of the NASA applications strategy. This would include a discussion of the following:

   1. How the current strategy relates to previous efforts within the Earth Science Enterprise to direct the Applications Division and how those efforts constituted a prologue to this one.

   2. How the current strategy sets up the framework for the future implementation of specific program elements, or, how the current Plan lays the foundation for a future implementation plan, since this document is not intended to include an implementation plan.

An expanded introductory section could provide a valuable opportunity to address several overarching themes, such as the set of key strategic principles that NASA described to the committee (see page 3 above). For example:

1. The section could discuss in greater detail the meaning and implications of the term “partnerships.”

2. The section could provide a clearer context for its current discussion of specific missions (i.e., GRACE and CALIPSO) in order to make it clear why that discussion is necessary and how those missions, and the data they provide, fit within the strategy.

3. The section might also include a list of drivers for the Applications Division that are similar to the list of questions it includes that drive the science programs in the ESE.

By addressing these issues the expanded section will more clearly articulate the objectives of the strategy and more appropriately set the stage for the details of the Applications Plan as they are laid out in the body of the document. In its current form the Applications Plan contains nothing to introduce some of the critical concepts that must be understood early on if the reader is to follow the Applications Plan from beginning to end. The committee believes that including discussions of the themes and contextual material described above will strengthen the preface or a new introductory section in this regard.

To achieve consistency and clarity throughout the Applications Plan the committee recommends that the following improvements be considered:

The statement in paragraph 3, “Specific elements of the ESE contributions to the visions are…” is ambiguous with respect to whether it refers to the Earth Science Enterprise as a whole or to the ESE Applications Program itself. The bullets appear to neglect an intended role for the private sector.

In references to ESE in general and the ESE Applications Program, it is not clear which is being referred to.

Table 1 refers to NASA ESE plans for predictive capabilities. However, these capabilities cannot be achieved without transitioning results to the partner operational agency. For example, Table 1 states research goals for 2010, but NOAA has the operational responsibility for weather forecasting. It is important that some reference be made to the operational agency’s involvement in implementation as a condition of achieving the stated goals, even though they are NASA ESE goals. If these goals are important to the ESE Applications Program, then they should be put into the context of the program and not just be included as research goals.

The committee noted that the FAA was not included in the list of agencies at the end of Section 1.2, although other NASA material cites FAA partnerships.

Paragraph 3 refers to NASA and its partners but does not define the partners. It would be useful to define who the partners are, such as federal agencies, state and local governments, the private sector, or the academic sector. Execution of the strategy relies heavily on partnerships. Consequently the discussion of partnerships should be expanded for clarity and completeness. When federal agencies are named as examples, if there are

corresponding non-federal partners, the committee recommends listing these examples also to illustrate that NASA’s partnerships reach beyond the federal sector.

A footnote or glossary entry that defines “biogeophysical sciences” would be useful.

The goals do not include any mention of a feedback process for identifying user needs and assessing success. User needs can provide a requirements “pull” that could drive the Earth System Science Research Program. Right now the Applications Program appears to be “pushed” as a science-driven program. In discussions with the committee, NASA representatives described cases where user feedback is being obtained, but the basic utility or importance of such feedback is not especially reflected in the Applications Plan. The committee also noted that the last goal listed in Section 1.3 does not include the private sector and other non-government partners.

The committee felt that the three-stage organization of the program planning strategy seemed practical and that the existing content of the Applications Plan subsections on these topics was useful. However, as is the case elsewhere in the document, the partners’ role in the planning process needs elaboration. Furthermore, the Applications Plan could be strengthened by the addition of specific case studies or examples of how the planning process might proceed. Finally, the committee assumes that the first sentence of Section 2.0 is intended to indicate that identification and selection of needs constitute a first step toward accomplishing the program goals, rather than the sole step, with the implication that subsequent effort will be focused on effectively accomplishing identified tasks.

Section 2.1 is a notable case where the role of the proposed partnerships is not readily apparent. The committee could not discern how input from partners about their needs would be considered. As this section reads, it appears that the planning process is driven more by “technology push” than by “user pull.” It was especially difficult to discern how user levels beyond federal agencies, or even federal agencies for that matter, would have input. The committee felt strongly that, for the proposed linkages with partners to be ultimately successful, the partners would have to be substantively involved in every aspect of planning, implementation, and evaluation of applications. With regard to the 12 national applications identified to date,⁹ the plan should discuss how these were selected (e.g., at the five regional workshops) and the extent to which partners and other stakeholders were involved. The plan should also discuss the process by which the list of

national applications will be revised and to what extent partner and stakeholder input will be utilized.

The committee recommends that ecological impacts be included in NASA’s 12 national application areas where appropriate. This will permit NASA to take advantage of biological program elements of research on climate, weather, and natural hazards.

As part of an effort to address these issues, it might be helpful if some of the details about information gathering and communication activities currently listed in Section 3.4 were introduced in Section 2.1 first. Including a sidebar or “box” detailing an illustrative example for each step in the process of identifying a specific candidate application would also be useful. The wildfire management application cited by NASA in a discussion with the committee¹⁰ would be an effective example.

Section 2.2 includes a reasonable and justified list of selection and prioritization criteria, and it effectively sets the stage for the selection process. What is missing, however, is some discussion of how the information needed to rank candidate applications against these criteria is collected, for example, whether this information was gleaned in part at the regional workshops cited by NASA in discussions with the committee.¹¹ Lastly, it would be instructive if this section contained an example of how an application was selected using the prioritized criteria presented in this section of the Applications Plan.

The committee concluded that it is necessary to clarify whether the process described in Section 2.3 for implementing individual applications applies specifically to ESE Applications or whether, as implied by the first sentence, it describes a role for ESE in general. If the latter is the case, then what is the specific role for the ESE Applications Division?

Once again, in the committee’s view, important details about the role of partnerships are lacking. Establishing “linkages” between NASA capabilities and specific partners and applications seems to be a critical element of the overall NASA applications strategy, but little detail is provided about how this will proceed. For example, will NASA’s partners be involved in the process of solicitation and selection of project performers? And if so, how? Section 2.3 also might benefit from the addition of a box that gives relevant examples.

The committee supports the concept of soliciting for “performers” from the public, academic, and private sectors, and felt that a diagram describing the connections between NASA, a specific partner, and successful respondents to a solicitation might be helpful.

Figure 1. Example of how the Applications Plan might illustrate the relationships between NASA, its formal Applications Program partners, public or private sector “performers” that carry out specific applications projects, and end users.

Figure 1 is the committee’s simplified portrayal of how it understands the process. This figure is meant to indicate that NASA will enter into a two-way communication with a partner agency or other entity that has the resources (relationships with users, decision support system, etc.) to implement a chosen application. As a result of this dialog the needs and products required for implementation will be identified, and then “performers” will be selected (likely through a solicitation process) to provide the link (e.g., data handling and processing, new products, etc.) between NASA resources and the partner’s application requirements. The partner will carry out the final implementation and interaction with ultimate users.

Some reference in the Applications Plan to how performers are solicited and selected via Broad Area Announcements or Request for Proposals, which NASA officials described to the committee, also would be illuminating.

A detail to note is that the multiple uses of the term “project” are somewhat confusing. The word might be removed from the second sentence in Section 2.3 to eliminate ambiguity in relation to those “projects” associated with solicited proposals.

The committee recognized that the purpose of Section 3 is to describe the elements needed to implement the stated NASA applications strategy. These elements are (a) the management structure that includes headquarters Applications Division program managers and personnel at NASA Stennis Space Center; (b) a program framework, which reflects the relationship among the partners’ needs, NASA data, and the use of this data; (c) an implementation element, which describes the readiness of the application (“Application Research”), readiness of the data (“Validation and Verification”), and readiness for transferability to operations (“Applications Demonstration”); and (d) program action, which is a compilation of action items.

Nonetheless the committee found the description of these elements unclear and confusing, which may lead to confusion among users of the Applications Plan. The first three elements (a through c above) appeared to describe actions at a project level, with

Section 3.1 describing project management (within NASA ESE), Section 3.2 describing the project framework, and Section 3.3 covering project development and transfer. From those descriptions, the ESE Applications Program would consist of the collection of these activities, and thus constitute the program’s operations. Thus, it appeared to the committee that the “Concept of Operations” can be described from a project rather than a program perspective, with the appropriate changes to subsection heading and text. In any case clarification of the above interpretation of Section 3 is needed. Further, the committee suggests that the section be titled “Concept of Implementation,” as the section does not describe operational activities.

The committee agrees that the ESE Applications Program should provide the bridge between ESE’s research domain and the public/private sector’s operational activities. It seems that the Applications Plan should better articulate this bridging/transition.

The committee had the following comments about Section 3.2:

• This section seems to describe the life cycle of any particular application project rather than a program framework—perhaps because figure 2 depicts the flow of spatial data at the project level. If the program framework consists of a collection of these project information cycles, then the Applications Plan should explicitly state this.

• In general, the committee found figure 2 confusing and was uncertain how figure 2 related to the program framework, as opposed to a project life cycle, and how figure 2 and figure 1 should interrelate. As an example of this confusion, it is not clear what is meant by “tasking” because of its multiple uses.

• The committee notes that NASA does not address openness of data in the Applications Plan, particularly with respect to data that might be viewed as sensitive in an era of heightened attention to national security. NASA should consider making a statement on this topic in the document. An example of a relevant instance would be data collected on water infrastructure and how such data are made available and used.

The committee had the following comments about Section 3.3:

• This section does not sufficiently describe “Program Implementation” but is consistent with the view that implementation involves project development and applications transfer. The committee had concerns that this element was not sufficiently sensitive to the complex issues of transitioning the applications into operations, including the programmatic and budgetary readiness of the partner, continuity of NASA ESE data used for the application, broad user readiness, and so forth. As written, the Applications Plan reflects the implicit assumption that if the application demonstration is successful, then the partner and its users will quickly adopt the product. Explicit statements of how this transfer will be encouraged would be helpful.

One particularly important aspect has to do with what agency would be responsible for operational satellite programs. Currently NOAA is the only agency with a civilian operational satellite mission. Would other agencies also follow suit, would NOAA broaden its operational mission beyond weather forecasting and climate information, or would NASA take on an operational satellite mission for the other agencies such as EPA or USGS?

Section 3.4 does not seem to fit well into Section 3. While the committee recognizes that a strategic plan should include such action items, it would be more appropriate to include this information as a new Section 5. In addition, the committee encourages NASA to check the list for completeness and to consider adding actions such as evaluation of projects, to include progress metrics, identification of transition issues and success criteria, follow-up evaluations of transitioned projects, and so forth.

The committee further believes that NASA should augment the Applications Plan by adding a discussion of co-funding by federal partners, as such commitments ensure partner buy-in, ownership, and transition as appropriate.

The performance evaluation section appears to rely mainly on concepts and terminology drawn from the Government Performance and Results Act (GPRA). Consistent with GPRA, this section of the Applications Plan identifies “program inputs and outputs, desired outcomes and expected impacts.” In the committee’s view this approach, standing alone, does not provide effective evaluation methodology. In particular, the measures in the current draft are sometimes vague, and they are not quantitative.

However, the committee does not want its recommendations to override any GPRA requirements and, based in part on comments by guest speakers, the committee is somewhat uncertain about expectations for this section of the Applications Plan. For this reason, the committee is offering alternative recommendations.

If the GPRA emphasis and format are required for the Applications Plan, the committee assumes that NASA will follow GPRA requirements regarding performance measures. In this case, the committee recommends using the most effective evaluation methodology consistent with GPRA requirements, especially including specific and measurable outputs and outcomes. (The committee fully understands that NASA will follow GPRA requirements in customary communications with Congress, OMB, OSTP, etc.). Alternatively, if the GPRA emphasis and format are not required for the Applications Plan, the committee recommends replacing the GPRA orientation with NASA-designed evaluation methodology and performance measures appropriate for the process and products set forth in the Applications Plan.

A central role for NASA’s partners and stakeholders is one of the great strengths of the NASA applications strategy and should be one of the Applications Plan’s most prominent features. However, the committee did not find performance measures designed for

partner/stakeholder evaluation of the quality of interactions between NASA and its partners/stakeholders or for assessing the extent to which the ESE Applications Program meets their needs. The committee believes that NASA’s partners and stakeholders should participate in evaluating performance, and recommends that the Applications Plan include specific opportunities and mechanisms for partners and stakeholders to evaluate the extent to which the ESE Applications Program meets their needs.

The three appendices are informative and useful adjuncts to the Applications Plan. At the same time, the committee recommends augmenting the existing appendices with additional information, and moving one appendix to the text and replacing it with related information in a recent NASA publication.

The committee recommends adding three additional terms to the definitions in Appendix A: biogeophysical sciences, partner, and project. The first term generated questions among experts on the committee; the last two words are terms of art that have special meanings in this plan.

Most committee members were familiar with the acronyms used throughout the Applications Plan, and NASA defined many acronyms in the body of the document. However, as a convenience for the general reader, the committee recommends including a list of acronyms, either as part of the existing glossary or as a new, separate appendix.

The introduction to Appendix B states that “[a]greements with Federal, state, local and tribal agencies, international organizations, and the private sector also contribute to the direction of the program” (page 14). However, the ensuing list of “drivers” appears to include federal programs only, with no examples of states, tribal entities, international organizations, or the private sector as drivers. Also, the “response” section refers to “selected academic institutions” without naming them.

Throughout the Applications Plan and in Appendix B, NASA provides solid information about and evidence of its federal partnerships by naming participating agencies and activities. This is one of the strengths of the document. At the same time, although the NASA applications strategy refers often to other organizations, as in the language cited above, specific examples are not given. If agreements with states, tribes, international organizations, or the private sector are, in fact, among the drivers, listing them along with the federal drivers would demonstrate that NASA’s agreements reach beyond the federal sector.

If agreements with non-federal entities are in force or planned, the committee recommends adding two or three examples to the list in Appendix B, along with the names of some of the academic institutions involved. Adding this information would complete and balance the presentation. Alternatively, if NASA intends or expects that federal agencies alone will be drivers, the committee recommends that NASA review the introductory language quoted above to determine if it should be revised. The committee is not recommending expanding the list of drivers beyond the federal sector if NASA intends federal drivers only.

Figure 2 (page 8) stimulated considerable committee discussion, including many questions about the meaning of various elements in the diagram. The committee then observed that the examples in Appendix C mapped to elements in figure 2. In fact, information in the appendix clarified the diagram and represented activities undertaken and results achieved in line with the process outlined in the figure. The committee recommends converting Appendix C to a “table of accomplishments” and incorporating the new table in Section 3.2 (page 8). The table would elaborate on ESE Applications Program participation in the process diagrammed in figure 2.

NASA’s presentation for the committee included numerous informative and clarifying slides. In particular, the committee felt that the 12 areas cited in the article by Birk and Hutchinson¹² (see page 3 of this report) offered information roughly comparable to that in Appendix C, but the article was more comprehensive and descriptive. The committee recommends replacing the information moved from Appendix C to Section 3.2 with the list of representative activities in the article. Lastly, Appendix C should have a note stating that this is an evolving process in which, for example, the list of national applications presented in the aforementioned slide undergoes periodic revision with input such as that from the regional workshops NASA has utilized in the past.