Steps to Facilitate Principal-Investigator-Led Earth Science Missions (2004)

The processes by which a program and its constituent projects are implemented are critical to the success of the program as a whole. Within PI-led programs such as the Earth System Science Pathfinder and Discovery (in NASA’s SSE), individual missions are solicited, selected, and executed as distinct projects whose activities are guided by processes developed over many years with the goal of producing missions characterized by both excellent science and a high likelihood of success. The effectiveness of these processes is a significant factor in achieving success in PI-led missions.

This chapter discusses the life cycle of PI-led projects and provides specific recommendations for enhancing the solicitation, selection, and execution of PI-led missions. Background on existing and previous PI-led programs in both ESE and SSE is provided in Appendix D.

NASA projects, including PI-led missions, are guided by procedures in the NASA project management handbook NPG 7120.5B,¹ which identifies the general process structure to be used in both programs and projects² and establishes the project life cycle. The PI-led mission project life cycle includes the three fundamental processes of solicitation, selection, and execution. Solicitation is used to define the objectives of a program and to establish the guidelines for project selection and execution. Projects are chosen during selection; execution is the activity of accomplishing projects.

NASA has established checks-and-balances mechanisms for each of these processes based on three functions: perform/manage, oversee/evaluate, and approve/select. Figure 5.1 shows how these three functions are applied to solicitation, selection, and execution and who has responsibility for each function. During solicitation, the NASA program office writes the AO solicitation (perform/manage), drafts of the AO are reviewed internally and by the

FIGURE 5.1 The checks-and-balances process used for the three project life-cycle activities of solicitation, selection, and execution.

community of potential proposers (oversee/evaluate), and the ESE associate administrator (AA) approves the AO for release (approve/select). During selection, the PI team advocates its mission through a proposal (perform/ manage), reviewers evaluate the proposal (oversee/evaluate), and the ESE AA assesses the integrity of the proposal/ review process as part of making selections (approve/select). During execution, the PI team develops the mission (perform/manage), the NASA program/project office oversees and ensures the development process (oversee/ evaluate), and the ESE AA confirms that the process has resulted in a mission capable of proceeding to flight and ultimately to launch (approve/select).

All three elements of the checks-and-balances process must work properly to establish an effective solicitation, select viable mission proposals, and carry out the projects to achieve mission success. Furthermore, the three activities must be adequately coupled such that project risk is not introduced at either of the two “handoffs” between activities identified in Figure 5.1. These handoffs are particularly critical as it is at these points that personnel typically leave the activity, introducing possible risks resulting from unrecognized gaps in project knowledge. Failures in any single element of this system are to be expected, but proper implementation of the checks-and-balances approach should produce a robust environment in which single-element failures are caught and rectified.

PI-led projects are particularly susceptible to both weaknesses in the checks-and-balances system and to risks introduced through inadequate handoffs. Like all programs, a PI-led project must balance performance (in this case, science) against cost and schedule, and this can be addressed during the three life-cycle activities. The committee believes that good science can be accomplished within the limited project budget and schedule of a PI-led mission if the candidate PI develops a project plan that can achieve the AO-defined scientific goals within these constrained resources (including appropriate reserves), and the NASA review process properly evaluates the plan.

The solicitation process must provide science objectives that can be accomplished within programmatic limitations, define a process that is capable of selecting missions that provide both high-priority science and can be implemented, and, once a project is selected, establish a programmatic structure that maximizes the likelihood of

mission success. In the preparation of each solicitation, it is incumbent on NASA to consider inputs from the proposers and implementing communities and to build on lessons learned from other PI-led mission solicitations.

The selection process must evaluate all major technical, management, and cost issues; must reflect confidence in the proposers’ ability to achieve the outcome; and must be viewed by reviewers as sufficiently thorough that proposal weaknesses cannot be effectively hidden. The approve/select function of the ESE AA is critical for ensuring that the process identified in the solicitation is robust, that it is followed diligently by the reviewers, and that the selections themselves fit credibly the available cost and schedule constraints.

The execution process is not unlike that for similarly constrained non-PI-led projects. To avoid unforeseen project growth, the PI team must follow the proposal plan as closely as possible, and the NASA project office must support this plan with effective but constrained oversight.

Given the many examples of successful PI-led projects (see Appendix C, Tables C.1 through C.5), it is clear that PI-led missions can be accomplished using available processes. PI-led projects that have been canceled or restructured, or that failed on orbit, can generally trace their problems to a breakdown of the checks-and-balances formalism. For example, the ESSP solicitations have encouraged international partnerships, but the selection and decision process has allowed missions to proceed without firm commitments from partners or with partner relationships that could be foreseen as difficult to manage. The solicitations have also encouraged innovative management approaches, but no process has been provided for evaluating the viability of such approaches. As a result, projects have suffered cost and schedule impacts during execution; the flight project status of one project (VCL) had to be suspended (see Appendix C, Table C.4).

In their preparation of new solicitations, ESE officials are applying the lessons learned from projects in which problems developed. However, the committee encourages ESE to continue to review the checks-and-balances process to ensure that it is sufficiently robust to identify problems before they lead to mission cancellation, restructuring, or on-orbit failure.

Finding: Existing NASA guidelines (e.g., NPG 7120.5B) establish a management system relevant to PI-led missions, including an essential checks-and-balances formalism for the three PI-led mission project life-cycle processes of solicitation, selection, and execution.

Recommendation: NASA’s Earth Science Enterprise should emphasize formal and regular reviews of the life-cycle system of checks and balances as applied to PI-led missions and should continuously strengthen the processes on which the system is based.

The timeline of a PI-led project encompasses the three basic activities of solicitation, selection, and execution. These activities are in turn segmented into elements that correspond to discrete aspects of the project life, as summarized in Figure 5.2, with selection divided into multiple steps³ and execution divided into project cycle phases.⁴ As illustrated in Figure 5.2, both ESE and SSE have historically employed life cycles in which the selection steps have some overlap with the project execution phases.

In general, Step 1 has been used to provide a preliminary competitive selection with a more comprehensive Step 2 proposal used for the final selection. In some programs (e.g., Discovery), Step 2 has corresponded

explicitly to Phase 1 of the project cycle, and the Mission Concept Study Report has functioned as the Step 2 proposal. In other programs (e.g., ESSP), Step 1 has focused primarily on evaluating scientific merit, with Step 2 used primarily to evaluate technical, management, cost, and other (TMCO) merit. The ESSP-3 AO includes a two-part Step 2 selection, with an initial Step 2 selection made by proposal evaluation and a final selection at the end of Phase 2 based on performance during the formulation phase. Typically, of the 20 to 40 proposals that have been received in Step 1, 4 to 10 are selected to proceed to Step 2. At the completion of Step 2, 1 to 3 missions have been selected for flight.

For a project team, the combination of selection and execution activities can be described in terms of two fundamental periods of the project life cycle:

• Competitive formulation: Initial solicitation and competitive selection of proposals (Step 1), followed by initial project formulation (Phase 1 or Phases 1-2) and final competitive selection (Step 2), and

• Noncompetitive formulation/implementation: Completion of project formulation (Phase 2) and implementation (Phases 3-5).

The competitive formulation period is particularly important to mission success. During this period, a PI faces a wide range of challenges that must be addressed and adequately resolved. Many of the issues arising throughout a mission’s lifetime are rooted in decisions made early in the project as the mission concept is developed, team roles and responsibilities are defined, and the management approach is established. Identifying improvements to this portion of the mission process thus provides great leverage for promoting successful implementation of PI-led missions.

Establishing an appropriate balance between competition and formulation during this period is also critical to project success. Both competition and formulation contribute to mission success: competition inspires PI teams to push the limits of missions that can be implemented credibly, and formulation provides teams access to the resources needed to fully demonstrate that the implementation is credible. Effective formulation requires extensive communication with ESE as well as resources tailored to mission needs. The PI team and the ESE project office should be encouraged to jointly identify and resolve all programmatic weaknesses through open dialogue and shared information. Effective competition implies nonpreferential interaction with ESE and resources divided equally among competitors. In contrast to formulation, competition motivates the PI team to emphasize strengths and minimize weaknesses.

Extending the competition period to mission design review (MDR), as was planned for ESSP-3, improves the ability to select the most viable missions among competitors but delays the integration of the selected mission and NASA teams, possibly compromising successful formulation of the selected mission. Projects should include both competition and formulation during the competitive formulation period, with the correct combination tailored to particular program needs. Mission success, however, is enhanced when the competitive formulation period explicitly includes a noncompetitive segment dedicated to formulation, such as when a final Step 2 selection for flight is coincident with the initiation of Phase 2. Furthermore, Phase 2 of the formulation phase must have both adequate funding and an appropriate schedule to achieve proper risk reduction. Resources applied to identifying and resolving problems in the formulation phase are widely recognized as an effective means to avoid significant problems later in the project. Historical guidelines exist for an adequate level of formulation-phase resources, but the particular needs of PI-led missions suggest that such guidelines should be carefully adapted to each AO.

Finding: Many of the issues arising throughout a mission’s lifetime are rooted in decisions made by the PI and project team during the formulation phase—early in the project—as the mission concept is developed, team roles and responsibilities (including NASA’s) are defined, and the management approach is established. Mission success requires that major technical and programmatic issues be identified and jointly addressed by both the PI team and the NASA program office during the formulation phase. While extending competition between PI teams through the entire formulation phase provides NASA with additional insight into the effectiveness of the PI teams and the maturity of the mission designs, it delays the integration of the PI and NASA teams and motivates the PI teams to emphasize strengths and minimize weaknesses.

Recommendation: NASA’s Earth Science Enterprise should avoid extensive overlap between competition and execution activities during the formulation phase of PI-led missions, thus providing an adequate schedule to perform critical formulation tasks after the competitive selection is completed.

PI-led missions have been supported by NASA for more than a decade. Within ESE, the Earth Explorers Program has only included ESSP, because UnESS was canceled before the start of any competitively selected missions. PI-led missions in NASA’s Office of Space Science have included Discovery, Small Explorers (SMEX), Medium-Class Explorers (MIDEX), and University Explorers (UNEX).⁵ This section provides a description of how these programs are structured, including the history of the Earth Explorers programs and how they have evolved.

Throughout its history, ESSP has retained the distinction between a science-focused Step 1 proposal and a TMCO focus in Step 2. Step 1 proposals are prescribed to be relatively short so as to limit time expenditures from a large number of proposers. Step 2 proposals are much more extensive and expensive to prepare. NASA does not fund Step 2 proposal preparation.

Step 1 proposals are prescribed to be about 25 pages, with the greatest emphases on science ideas and proposed investigations (e.g., measurement approach, instrumentation, and technical maturity). Measurements are specifically connected to science questions in a required “Science Traceability Matrix,” and technical maturity is quantified in a required “Instrumentation Technical Maturity Matrix.” Lesser emphases (in order of importance) are placed on technical aspects of the proposal, management strategies, and cost estimations at Step 1.

A limited number of Step 1 proposal teams are then requested to prepare Step 2 proposals (in the case of ESSP-3, only six proposals were promoted to Step 2). Step 2 proposals are much longer (about 120 pages) and involve a reiteration and refinement of the science ideas and proposed investigation, as well as extensive discussions of technical and management issues, costs and cost estimating procedures, impacts on education and opportunities for underrepresented groups, and any other relevant impacts and factors. Step 2 evaluations can also include site visits.

The records to date for ESSP solicitations, cost caps, and down-selects are summarized in Table 2.1. Step 1 proposal numbers have decreased from the initial response for ESSP-1 at 44 proposals in 1996, to 20 and 18 Step 1 proposals in the responses to ESSP-2 (1998) and ESSP-3 (2001), respectively. The GRACE and VCL missions were selected at the end of Step 2a in the competition for ESSP-1. Three missions—CloudSat, CALIPSO, and VOLCAM—were selected at Step 2a in the competition for ESSP-2. The Volcam mission was identified as an alternate should either of the other two missions fail to adequately address a list of requirements for proceeding with mission formulation over a 3-month period following the conclusion of Step 2a. The 3-month period following Step 2a was funded by NASA for all three missions succeeding at Step 2a in ESSP-2.

The GRACE mission was launched in March 2002; the VCL mission experienced implementation difficulties such that it was necessary for NASA to postpone indefinitely further phases of project execution; and CloudSat and CALIPSO are in prelaunch implementation phases.

The declared cost caps at the times of the solicitations have risen over the history of ESSP: in 1996, the ESSP-1 solicitation called for one mission at $90 million (GRACE) and one at $60 million (VCL); in 1998, the ESSP-2 solicitation called for two missions at $120 million each (CloudSat and CALIPSO). In both of these solicitations launch costs were expected to be included under the proposed cost caps (or provided by a non-NASA partner). The ESSP-3 solicitation cost caps are increased to two missions at $125 million each, and launch costs are not included so long as one of three NASA launch options is proposed.

For ESSP-3, six candidate missions were asked to prepare Step 2 proposals. The Aquarius and OCO missions were subsequently selected, with the HYDROS mission as an alternate.⁶ Based on the provisions contained in the AO, it appears that two missions will be selected at MDR to proceed with mission implementation and launch.

ESE currently supports ESSP as its sole PI-led program. However, as explained in Chapter 2, the UnESS program was announced in 1999 and was supported as a second PI-led program with a stated focus on university-led organization. As such, its AO provided substantially more freedom for the PI team to propose management and development processes consistent with university constraints and tailored to the particular needs of each mission. UnESS was funded in ESE through the FY2001 budget, and then canceled.

The 1999 UnESS AO provided for a two-step competitive process, with Step 1 identified as the concept study selection process and Step 2 as the down-select process. Step 2 was a funded (less than $300,000) competitive segment of approximately 9 months’ duration. During Step 2, those missions selected in Step 1 were to refine the mission concept and submit a concept study report to be evaluated as the basis for the final down-selection. Selection criteria for both steps were identical. Selected missions were expected to proceed to Phase 2 (mission definition and preliminary design phase) of the formulation subprocess.

NASA’s experience with PI-led space science missions provides an instructive comparison to the ESE experience. In fact, many of the concerns with respect to PI-led missions in the Earth sciences have already been debated in reviews of PI-led space science programs, for example, the review of the SSE Explorer Program.⁷ The Discovery (2000),⁸ MIDEX (2001),⁹ and SMEX (2000)¹⁰ are the SSE’s PI-led mission programs. These programs all employ a two-step selection process. As with ESSP, Step 1 is focused primarily on evaluation of science merit and Step 2 on TMCO issues. However, Step 2 consists of a 4-month concept study, funded by NASA at about $450,000. The concept study report serves as the Step 2 proposal. Selected missions proceed to Phase B (equivalent to Phase 2; see Figure 5.2) of the formulation process. Approval for flight is made in the mission confirmation review (MCR) at the completion of Phase B.

The quality of the solicitation has an enormous impact on the success or failure of a project. The solicitation establishes both the project expectations and the constraints, and defines the process used to evaluate and select projects. A well-written solicitation provides sufficient time and budget resources to implement missions that satisfy program expectations, and defines a selection process that ensures that this occurs. This section provides a discussion of areas in Earth Explorers solicitations that could be enhanced to increase the likelihood of mission success.

Cost caps are an effective NASA management tool for both managing program costs and controlling Enterprise budgets, as has been demonstrated on many PI-led projects. However, the threat of project cancellation for

exceeding cost caps has proven to be ineffective. Both the ESSP-1 and ESSP-2 missions have exceeded the proposed cost cap, though cost increases have occurred in some cases because of events completely outside the control of the project or NASA.

Though maintaining the credibility of cost caps is important, the best interest of NASA and the science community is often served by continuing to fund projects even though they are projected to exceed the cost cap. Cost caps should therefore be redefined from a threshold for an automatic termination review to a threshold for a remedial review that includes an examination of how the division of responsibility and authority between the PI and ESE might be revised to better control costs. ESE (not the PI) would then be responsible for electing whether to increase funding, descope, or cancel the project based on a variety of ESE considerations. In addition, cost caps should be established only when the project has reached a level of maturity that the budgeted cost is realistic. ESSP-3 establishes firm cost caps only when a project has reached MDR, instead of during the Step 2 proposal as in prior solicitations, and this should improve the ability of projects to stay within cost caps.

One technique that has been used successfully by NASA’s SSE to help implement cost caps is the application of science floors. Defined by the PI in the mission proposal, the science floor represents the minimum science achievement required to justify the mission. It establishes a lower bound to descoping the mission in order to manage cost within the available cost cap.

Setting science floors for ESSP missions could serve two important purposes. First, assuming that analysis as well as production of the data is funded as part of the mission contract, it helps ensure a minimum science achievement sufficient to justify the mission, consistent with the PI’s and NASA’s science objectives. Second, it puts both the PI and NASA in a better position up front to know how to assess and make decisions about potential descoping or termination.

Finding: The threat of project cancellation has not proved effective either in motivating the submission of PI-led proposals with adequate reserves or in constraining costs to meet the cost cap.

Recommendation: NASA’s Earth Science Enterprise should redefine cost caps from a threshold that triggers an automatic termination review to a threshold for a remedial review that includes an examination of how the division of responsibility and authority between the PI and ESE might be revised to better control costs. Cost caps should be established only when the project has reached a sufficient level of maturity that the proposed cost is credible, such as at mission design review. ESE should also consider the use of a science floor, a PI-proposed minimum scientific achievement needed to justify the mission, in setting and managing within cost caps.

Team formation is initiated by the PI and takes place during the earliest stages of the selection process. Recently, ESE has been made aware of the importance of PI team formulation in ESSP PI-led projects.¹¹ The PI team must include experienced individuals in the roles of project and mission managers. The frequent use of international and/or other agency partnerships in ESSP PI-led missions also necessitates the early identification of responsible science and technical parties outside NASA. However, potential science PIs for ESSP PI-led missions are not uniformly well acquainted with the technical communities that will provide many of these key personnel.

PI-led projects require relationships between the PI institution, NASA centers, NASA-sponsored institutions, and industry partners that are generally different from those of non-PI-led projects, and often these relationships are considerably more complex. As the majority of PI-led projects involve a NASA center or NASA-sponsored

institution, selected missions in many cases can involve both the selected center/institution and the particular NASA center associated with the ESSP project office. These overlapping relationships can introduce conflict between management processes, duplication of reviews, and confusion about authority. While the ESSP project office plays a critical role in mission success, avoiding duplication and conflict between the project office role and that of NASA centers/institutions on the PI team is desirable.

In order to augment science content within the ESSP PI-led mission cost cap, it has become common for PIs to establish partnerships outside NASA with domestic and/or international agencies that contribute funding to the mission. While these partnerships provide the tangible benefits of additional funding and multiagency commitment, often they also introduce complexity and risk to mission formulation. Both the PI team and the ESSP project office need to ensure that partnering agreements are completed early in the formulation period, that interface definition is given highest priority, and that a clear management decision chain is understood and accepted by all parties.

Finding: Domestic and international partners have increasingly been included on PI-led mission teams to enhance the quality of science achievable within the available ESE project budget. Despite the many benefits of such collaborations, more complex and diverse teams increase risk and add costs to pay for team interfaces.

Recommendation: NASA’s Earth Science Enterprise should recognize not only the benefits but also the risks of having domestic and international partners in a PI-led mission program. The mission solicitation should identify the need for processes by which both the PI team and the relevant NASA office ensure that partnering agreements are completed early in the formulation phase, that definition of an interface is given high priority, and that the management decision chain is clear and is understood by all parties.

The solicitation process needs to strike a balance between the amount of information required in proposals to enable effective evaluation and the time required to prepare and review proposals.

In each ESSP AO, NASA is challenged to open the mission concept and design process to new and perhaps unexpected ideas from the science community, while defining in precise terms a review system to identify and mitigate risk, and to ensure scientific excellence and engineering, technological, management, and cost feasibility. These challenges imply considerable breadth in the levels of detail necessary in the AO, and the total document is indeed substantial as a result (main text, 42 pages; appendixes, 132 pages). The tendency to micromanage proposal preparation in an effort to ensure adequate information transfer is natural. The AO and materials from a recent preproposal conference illustrate a balance in this regard: Appendix K from the ESSP-3 AO provides a very detailed description of requirements for Step 1 and 2 proposals, while the minimum proposal requirements documentation from the preproposal conference provides a concise practical lower bound for potential investigators.

An AO must not be so burdensome as to discourage PIs from proposing. The two-step AO solicitation is designed to minimize the time and resource commitment required of nonselected PIs, and it has generally achieved that objective. University PIs, however, are particularly affected by funding and time constraints. NASA should therefore continue to refine the two-step process with the objective of obtaining sufficient information for evaluation while minimizing the resources required of proposing PI teams.

Various approaches to funding support for Step 2 have been tried. ESSP currently provides no funding until after the Step 2 selection, while Discovery, SMEX, and MIDEX (and, previously, UnESS) fund Step 2 at amounts under $500,000 per team. Teams that reach Step 2 of the competition have historically spent more on the study report than they received in NASA funding. NASA centers, NASA-sponsored institutions, and industry partners generally have sufficient resources to support Step 2 and are able to make these investments; academic institutions, however, have less access to the needed resources. Step 2 funding thus provides the greatest value to academic institutions and furthers their inclusion in the AO process.

Finding: A properly constructed solicitation balances the need for proposals detailed enough to permit thorough evaluation against the time required both to prepare and to evaluate proposals. The two-step proposal process, in particular the use of short Step 1 proposals within ESSP, has provided a workable balance. However, the lack of NASA-funded support for proposals, particularly during Step 2, is increasingly limiting the ability of smaller organizations and universities to participate.

Recommendation: NASA’s Earth Science Enterprise should maintain the current two-step proposal process for PI-led missions but should provide funding to proposers for Step 2.

PI-led mission proposals are evaluated primarily on the basis of their science objectives, and recent AOs have specifically sought projects that introduced innovative methods to address scientific questions outlined in the ESE strategic plan. However, although the ESSP-3 AO specifies that funding will support “implementation, launch, and science data archival and dissemination,” it is ambiguous about research funding. The AO states that the PI team may carry out “initial scientific/applications evaluation in support of the proposed research objective(s),” but says that further science funding is anticipated through SDAP proposal solicitations released near the launch date. In contrast, the ESSP-3 AO FAQs specify, “The purpose of a mission is to answer science questions relevant to the ESE science goals. If the [postlaunch science] analysis effort is directed at answering the science questions, then this cost must be included in your mission costs.”¹² These statements are likely to be interpreted differently by each of the proposing teams and may lead to difficulties in obtaining objective evaluations of the projects. Since the ESSP-1 and ESSP-2 AO solicitations, NASA has made significant progress in recognizing that support of PI-team postlaunch research is essential both for attracting high-quality PIs and for ensuring achievement of the mission’s science objectives. However, the ESSP-3 AO and its associated FAQs remain ambiguous in this area. The committee recommends that future solicitations make clear that postlaunch research by the PI team toward the mission’s major science goals should be requested as part of the original ESSP proposal.

Although research toward major science goals should be supported through the original ESSP mission funding, the SDAP science exploitation team also plays a role in ensuring that ESSP missions achieve their scientific objectives. Thus far, GRACE is the only ESSP mission to have been launched.¹³ The NASA NRA-01-OES-05 from the Solid Earth and Natural Hazards Research and Applications program was released not long before the launch of GRACE and offered some support for GRACE research, although it was not formally identified as SDAP support. Separate solicitations such as this are valuable for a number of reasons. They can involve investigators with no past history with the mission in order to allow for independent data quality checks. They can also broaden the science team to encourage new data-processing algorithms, improvements in analysis methods, and use of additional sensor data for reasons not anticipated in the original science proposal. Launch-time science solicitations (whether or not they are identified as SDAP solicitations) are important for ensuring that ESSP missions achieve their scientific objectives.

Finding: Scientific results are the primary objective in PI-led missions, but postlaunch science funding commitments are not adequately identified in the mission solicitations.

Recommendation: NASA’s Earth Science Enterprise should clearly specify within the solicitation for a PI-led mission the extent to which scientific investigation and data analysis are expected to be included in the initial mission project budget, as well as the anticipated plans and budget for additional postlaunch science investigations. The science funded for the mission should address a PI-proposed science floor.

Dissemination of lessons learned is an important element of a program to enhance the capacity of potential PIs. Lessons can be communicated through a variety of means, including online libraries, preproposal and postproposal conferences, and mentoring relationships with existing projects. The ESSP office has developed an excellent set of both non-PI-led and PI-led lessons learned¹⁴ and has organized preproposal conference presentations.¹⁵ The preproposal conferences provide a forum for two-way communication so that proposers can learn the nuances of NASA’s intentions with respect to solicitations, and NASA can learn more about the technical sophistication of the PI community. These approaches could potentially be extended to include presentations (online and/or at preproposal conferences) demonstrating successful partnering arrangements in past Earth Explorers PI-led projects and similar SSE programs (e.g., UNEX, SMEX, MIDEX). Mentoring relationships could be formed between key personnel from recently successful and newly selected projects, either case by case or as an intended outcome of NASA-sponsored workshops.

Finding: Effective communication and the transfer of lessons learned between the Earth Explorers Program Office, current flight projects, and potential PI proposers can both increase the number of qualified proposers and reduce the risk associated with proposed projects.

Recommendation: NASA’s Earth Science Enterprise should continue to emphasize and promote communication and the transfer of lessons learned between the Earth Explorers Program Office, current flight projects, and potential PI proposers.

The quality of the selection process determines whether viable projects proceed to execution. A robust selection process approves only those projects that can be executed within the proposed resources, and it identifies risk areas and mitigation recommendations upon initiation of the execution activity. This section provides a discussion of areas in the current Earth Explorers selection process that could be enhanced to increase the likelihood of mission success.

As noted in Chapter 3, the Earth Explorers Program faces the challenge of balancing the scientific potential of proposed missions with the likelihood of achieving a successful mission outcome. The need for an accurate evaluation is heightened by the recognition that most mid-course project remedies involve relaxing essential programmatic constraints that had served to define the scope of, and differentiate between, proposals at the early stages of evaluation. Recent examples of this problem include the selection of a UNEX mission with 12 instruments and another (STEDI) mission with 7 major, complex, x-ray and gamma-ray sensors; in both cases the missions were canceled, but only after the investment of several million dollars and several person-years of effort. The funding lost to these missions would have otherwise been available to missions that had the appropriate balance of scientific return with mission risk. NASA, the PIs, the research community, and the U.S. taxpayers are ill served by the selection of missions that cannot be completed within the programmatic constraints by the identified PI team, regardless of the merits of the proposed science. The evaluation leading to selection must therefore determine whether the proposal demonstrates that the PI team has the staff and infrastructure to properly implement the mission.

Finding: The quality of the selection process determines whether viable projects proceed to execution and thus greatly influences the overall success of PI-led missions. Selection criteria for PI-led missions, particularly those employed in Step 2, must adequately consider the ability of the project team to successfully implement a project; the ESE associate administrator must be provided sufficient information to determine the likely success of a project; and the selection decision must reflect an objective evaluation of the likelihood of success.

Recommendation: NASA’s Earth Science Enterprise should carefully review the selection criteria for PI-led missions to ensure that they adequately identify and promote missions that can succeed.

The selection of an effective evaluation panel is important to the success of the ESSP mission. Proposers must perceive the evaluation process as fair and accurate in order to justify the efforts and resources required to assemble technical and management teams and to prepare proposals for Steps 1 and 2. In addition, there are significant challenges to the identification and participation of suitably qualified reviewers. However, neither the ESSP-3 AO nor the preproposal conference materials describe how evaluation panel members are to be selected.

Within each Earth science subdiscipline (e.g., atmosphere, land surface processes, geophysics, cryosphere, ocean, biogeochemistry), science and technical experts with satellite data and observing systems experience form relatively small subsets of the larger communities. NASA rules regarding conflict of interest further restrict the number of potential evaluation panel members by excluding NASA center personnel and/or university scientists from reviewing proposals originating in their own institutions.

Participation on the science review panel involves considerable effort on the part of peer reviewers.¹⁶ The AO is both detailed and broad, and the proposals submitted span a wide range of scientific disciplines. In the Step 1 review process, panel reviewers can expect to spend one person-day per proposal for approximately 20 proposals in preparation for the panel meeting, which itself involves another week of effort. For continuity and consistency in reviews, it is desirable that NASA retain all or most of the review panel from Step 1 for review of the Step 2 proposals. This time commitment far exceeds peer review efforts for other Earth science programs, in NASA or in other funding agencies (e.g., NSF, NOAA).

Furthermore, most ESSP proposals include a number of the most knowledgeable people in the field on the PI’s science team, and many of the relatively small number of experienced scientists participate formally (as PI or Co-I) on at least one proposal in each solicitation. Otherwise-qualified individuals are therefore often not available to serve as reviewers, given both the level of effort required and conflict-of-interest considerations.

To increase the number of qualified reviewers for ESSP, NASA should consider opening panel review positions to qualified international scientists, and requiring as part of the initial contract that past ESSP (ESE) and Discovery/Explorer series (SSE) PIs serve on subsequent reviews. To the extent that SSE Explorer series and ESE ESSP solicitation and evaluation procedures can be made similar, the reviewers, at least for TMCO considerations, can be shared.

Finding: The number of qualified reviewers for ESE PI-led missions is small, particularly after elimination of scientists with conflicts of interest because of relationships with proposing teams.

Recommendation: NASA’s Earth Science Enterprise should consider enlarging the pool of possible reviewers of PI-led missions by adding qualified international scientists (if feasible, given current International Traffic in Arms Regulations constraints) and scientists from the space science community. ESE should also consider requiring as part of the contract for selected PI-led projects that the PI serve subsequently as a reviewer.

The number of proposals carried forward from Step 1 to Step 2 has always been a controversial issue.¹⁷ On the one hand, NASA would like to have the opportunity for detailed evaluation of the greatest number possible of potentially qualified proposals. On the other hand, carrying forward proposals that have a lower likelihood of winning entails the nonproductive expenditure of resources both among those teams and on the part of NASA and the reviewers. The proposals supported in Step 2 should include only those that have sufficiently high scientific merit and TMCO potential to be fully competitive with all other Step 2 proposals. Informal guidelines for the number of Step 2 awards should be a minimum of two for each flight opportunity to be awarded and a maximum of one-third of the total proposals submitted in Step 1.

Finding: The number of proposals selected for consideration in Step 2 represents a critical compromise between the desire for a large pool of evaluated PI-led mission proposals from which to make the final selection and the need for a pool small enough that available reviewers can perform detailed reviews. Selection for Step 2 of proposals that have a lower probability of final selection results in inefficient use of proposers’ resources.

Recommendation: The proposals supported in Step 2 of the selection process for PI-led missions should include only those that have sufficiently high scientific merit and an acceptable initial evaluation of technical, management, and cost risk so as to be fully competitive with all other Step 2 proposals. As an informal guideline, a minimum of two Step 2 proposals should be selected for evaluation for each flight opportunity to be awarded, and the maximum number considered should be one-third of the total proposals submitted in Step 1.

As discussed previously, competition involves a checks-and-balances process between the proposers, the reviewers, and the selection official. Maintaining and improving this checks-and-balances process and its credibility is the highest priority for enhancing the competition of PI-led mission AOs. As the standard for review quality increases, proposers will increase the quality of their technical concepts, management plans, and cost estimates. The result will be higher-quality proposals recommended to the selection official and a greater likelihood of selecting successful missions.

NASA has commonly included language in its AOs indicating that considerations not related to the merits of the proposals may be included in the selection process.¹⁸ Enterprise objectives clearly go beyond the more focused objectives of review boards convened for a particular AO, and it is widely recognized that AO peer reviews provide only part of the final selection criteria. Nevertheless, every effort should be made to maintain the greatest traceability of the decision to the recommendations of the independent reviews and to established Enterprise priorities.

Finding: Maintaining and improving the credibility of checks and balances is the highest priority for enhancing the selection process for PI-led missions. An effective and credible proposal review process requires a balanced effort among proposers, reviewers, and the selection official. Proposers are motivated to avoid overly optimistic costing if they respect the cost-review process; reviewers are more diligent when their recommendations are likely to be accepted by the selection official; and the selection official relies more readily on reviewer recommendations when the proposal and review process is effective at identifying the best mission candidates.

Recommendation: NASA’s Earth Science Enterprise should strengthen the complementary roles of proposers, reviewers, and the selection official in the selection process for PI-led missions, improving the critical balance between the three roles and focusing on clear traceability of the selection process to independent reviews and established ESE priorities.

Finding: The availability of accurate cost estimates is a very important element of the mission selection process, but establishing accurate estimates of project cost has historically provided one of the largest challenges to both proposers and reviewers of PI-led missions.

Recommendation: NASA’s Earth Science Enterprise should enhance its cost evaluation capabilities to improve the accuracy of mission selection decisions and to motivate improved fidelity of cost proposals.

Project execution encompasses both the formulation and the implementation phases of a mission project. This section discusses several general issues influencing successful project execution and specifically addresses adaptation of the six subelements of mission implementation to a cost- and schedule-constrained PI-led mission.

Many of the problems encountered in recent PI-led missions have root causes in common with non-PI-led missions. In particular, the transition to smaller cost-constrained projects during the 1990s, the aging of the space industry workforce, and other external issues all directly affected project success. These problems should not be attributed to flaws in the PI-led process, but rather applied as general lessons for all small-mission projects.

PI-led projects, however, must be able to address generic issues just as effectively as issues specific to the PI-led process. It is imperative that PI-led missions identify and mitigate mission risks just as rigorously and with as much accountability as non-PI-led missions. Elements of potential failure such as poor team communication, cost and schedule pressure, insufficient reserves, and weak review processes are common to projects both within NASA and in other institutions. Given sufficient time and money, potential failures can often be corrected if they are discovered. The application of cost caps and other program constraints over the last decade, however, has meant that it is more difficult to recover from budget overruns and schedule slips due to unforeseen problems, and mission teams must be adept at adjusting scope to recover. PI-led teams, which tend to be less experienced than NASA-led mission teams, are particularly susceptible to such an experience-driven environment. Within the context of the formulation and implementation phases, it is thus important for ESE to establish processes that emphasize the understanding of generic mission issues and the inclusion of appropriate lessons learned.¹⁹

Finding: Although some of the difficulties with recent PI-led missions are unique, many of the problems encountered have root causes in common with non-PI-led missions. In particular, the transition to smaller cost-constrained projects during the 1990s and the contraction and aging of the space industry workforce have affected project success. These problems should not be attributed to flaws in the PI-mode process, but rather applied as general lessons for all small-mission projects.

Recommendation: NASA’s Earth Science Enterprise should establish management processes for PI-led missions that emphasize understanding all PI-led and non-PI-led mission issues and the inclusion of appropriate lessons learned from both types of missions.

Although the ESSP-3 AO states that “the selected mission team will be totally responsible for the ESSP mission,” the AO process has evolved so that many mission elements, including the review process, are now within the control of the NASA program/project office and not subject to PI authority. NASA should explicitly recognize that mission success is a combined responsibility of the PI-NASA team. Split (not shared) authority is appropriate for achieving mission success and is healthy for the PI community. Lines of authority and responsibility, however, become confused when NASA asserts full PI responsibility but practices extensive management control. The result is that issues arise for which nobody claims responsibility or authority, introducing a significant risk to mission success. Each project plan should explicitly designate the split in authority and responsibility between the PI team and NASA, and both parties should concur prior to MDR. This split should accord with the philosophy that the mission should be defined and developed by the science community itself.²⁰

Finding: Mission success is appropriately viewed as the combined responsibility of the PI-led team and NASA. Split as opposed to shared authority is appropriate for achieving mission success and is healthy for the PI community; split authority and the resulting allocation of responsibility should be explicitly recognized in the project plan and should also reflect the philosophy inherent in PI-led missions that the mission is to be defined and developed by the science community itself.

Recommendation: NASA’s Earth Science Enterprise should explicitly recognize that mission success is a combined responsibility of the PI team and NASA and should establish project management plans, organizations, and processes that reflect an appropriate split, not a sharing, of authority, with the PI taking the lead in defining and maintaining overall mission integrity.

Project controls include budgeting, scheduling, procurement (subcontracting), risk management, technical reviews, requirements management, and technical management. If the mission is to succeed, each of these elements of project control has to be executed in an accurate, timely, and comprehensive manner, but without the resources of a large project management team. Customization and scaling are required to map project control functions to small, cost- and schedule-constrained ESSP missions. The committee fully recognizes the difficulty of staffing and implementing a comprehensive project management function with a very small number of people, many of whom have other duties on or off the project. The purpose of this section is to offer ideas and recommendations to NASA concerning how the function of project management might be scaled to a small ESE mission.

Key Individuals. For this discussion it is assumed that the ESE PI has delegated day-to-day project management responsibilities to a project manager (PM). Daily project management on small missions involves a number of activities, including technical decision making, facilitation of communications among team members, acquisition of resources, coordination with NASA, oversight of the project’s schedule and budgets, and oversight of the mission’s risk management process. Even the smallest of missions requires a full-time PM. An effective PM has been consistently identified as one of the hallmarks of successful missions. The PM in turn has a small number of team leaders who have direct responsibility for the implementation of the various elements of the mission (e.g., camera, spacecraft, integration and test, mission operations).

It is also assumed that NASA has delegated coordination and oversight of the ESE mission to the mission manager (MM) and staff. The MM functions as a liaison for the ESE mission, representing the interests of the mission to senior NASA management and likewise conveying the concerns of NASA back to the PI’s team. The MM often becomes a part of the PI’s management team.

Schedule and Cost Controls. A complete and accurate work breakdown structure (WBS) must be developed and adopted by both the PM and team leaders in order to properly implement a project schedule or cost-reporting system, even on relatively small projects such as ESSP missions. A good WBS should accurately reflect the manner in which work will be performed.

On large projects, cost and schedule controls are implemented with a staff of specialists who may have no other responsibilities. On ESSP missions, there is no such luxury. Project cost and schedule controls are almost certainly the responsibility of the PM with the assistance of perhaps one other person, but NASA must not expect this staff of two people to be able to produce the sorts of comprehensive, detailed cost reports on a monthly basis that are expected from an observatory-class mission.

During the implementation phase of a mission the PM must have a detailed schedule for tracking and reporting progress. Properly used, the schedule can also be resource loaded and used for cost performance evaluation. The PM must have agreement from the team leaders that the schedule accurately portrays the work being done and that they are committed to meeting the milestone dates shown. The team leaders must report their status accurately each month to the PM. In turn, the PM must link the schedules provided by the team leaders in such a way that the mission’s critical path can be clearly identified and reviewed with the MM on a monthly basis. Where progress is not being made at the rate needed to meet milestone dates, the PM must take corrective action as soon as a schedule slippage is identified.

The MM must be willing to help in any way to assist the PM in holding to the key milestone dates, because it is essentially impossible for a mission to maintain cost control if the schedule is not controlled. For cost development and tracking a similar process must be adopted, with the PM and team leaders again working together to develop the implementation phase budgets. It is essential that the team leaders as well as the PM be committed to performing the work for the agreed upon budgets. A variety of tools are commercially available for cost tracking and reporting for at least the major systems and sometimes for lower-level systems as well.

Table 5.1 shows a level of cost and schedule control and reporting frequency that the committee suggests as reasonable to serve the interests of NASA ESE and the PI team.

Risk Management. The identification and tracking of technical and programmatic risks is a vital element of project management. For ESSP missions it is likely that the PM will serve as the owner and operator of the mission risk management system. A tailored risk management system for ESSP missions includes a list of risk items that categorize the risk by type (technical or programmatic), system, criticality (e.g., how severe are the consequences of this risk?), likelihood, mitigation plans for each risk item that can be controlled or retired, and any dates associated with the risk mitigation plans. While NASA has a right to expect that the risk list will be maintained and reported on a monthly basis, ESE should not expect that an ESSP mission will have a staff of specialists monitoring and reporting risks.

Technical Reviews. Technical reviews have always been an important part of the NASA culture. The ESSP AO requires certain critical or “milestone” reviews, including reviews of the system requirements, preliminary design, critical design, preenvironmental test, preshipment, mission readiness, operations readiness, launch readiness, and flight readiness, and the PI team is expected to schedule and budget for these milestone reviews. Most PI teams also schedule peer reviews on all newly designed or extensively modified systems. ESE should expect that minutes and action items will be recorded for both milestone and peer reviews.

Of concern to the committee is the occurrence of unscheduled reviews initiated by ESE to address a specific concern. At issue are the impacts on the cost and schedule as well as on the workload of the very small number of people who will inevitably be assigned the responsibility to respond to any resulting action items. The following steps are advisable relative to unscheduled reviews:

1. The NASA MM makes the decision to hold such a review after discussing its possible impacts with the PI and PM.

2. The PI team is provided funding, above the cost cap, to support the unscheduled review.

3. The review is led by an individual knowledgeable and current in the field.

4. The review report is produced quickly.

Requirements Management. NASA should expect to see in place on PI-led missions a requirements management process tied to the mission’s systems engineering process, which synthesizes science goals and objectives into requirements and specifications for use by the instrument and spacecraft teams in developing their equipment. Missions with a weak or nonexistent systems engineering and requirements management process are not likely to succeed.

A tailored requirements management process will include a single, level 1 science requirements document that provides mission, spacecraft, and instrument requirements. Even a small mission must have a requirements flow-down process that maps these level 1 science requirements down to the individual spacecraft, payload, and ground segment elements of the mission. And for even the smallest of missions some form of verification process must be implemented to ensure that all requirements have been met.

Technical Management. Technical management involves the daily technical oversight and direction of a project. As a rule this is the job of the PM with assistance from a mission systems engineer.

Even ESSP-class missions must have someone functioning in the role of a mission systems engineer, who oversees the operation of the mission’s system engineering process and serves as the chief engineer of the project. The mission systems engineer provides daily technical direction, allocates resources, identifies and documents interfaces, manages the requirements-flow-down process, develops the mission’s environmental design and test guidelines, and manages the mission verification process. A full-time mission systems engineer is required for ESSP missions and may need the assistance of a part-time electrical engineer and mechanical engineer on small missions to work on specific issues. These two engineers may also be needed full-time in a systems engineering role on small missions, depending on the complexity of the mission.

Customer advocacy entails informing the direct customer, the NASA MM, of the status of the mission as well as including the MM in the mission’s decision-making process. One of the lessons learned from successful SSE missions is that good communication between the PI, PM, and MM is an essential element of mission success. The advocacy process ensures that the MM is an integral part of the PI’s management team. Missions of any size benefit from good communications, but with missions as resource constrained as those of the ESSP, communication is vital.

The design and development of a small mission such as an ESSP project require a competent, experienced, and dedicated team of engineers with clearly defined requirements, adequate resources to do the job, and enough freedom to develop the mission with minimum management oversight. When supported by a good systems engineering process and a complementary verification program, the design and development team has the best chance of developing its systems within cost and schedule, assuming that the TRL of the systems under development is sufficiently high that the development team can avoid major technical problems that overrun resources. NASA has been forced to cancel missions that suffered from inadequate technology readiness (e.g., FAME, CATSAT) even though the development teams were talented and dedicated. The low TRLs on these missions created huge cost and schedule risks that could not be overcome with available resources, leaving NASA no choice but to cancel the missions. Tailoring for a small mission, the development team must not be saddled with a low TRL as well as cost and schedule constraints if the mission is to be successful. The higher the TRL the more likely the mission success, especially for small missions. A mission should begin its implementation phase with a sufficiently high TRL, adequate resources and margins, a good systems engineering process, and a comprehensive verification program.

Completion of the development phase is determined by the successful completion of the mission’s verification program. The MSE, working with the team leaders, reviews the acceptance tests of all flight and ground segment elements to ensure that all requirements have been met. The NASA MM must also be an integral part of the acceptance process.

Although overlooked on some small missions, the selection and training of flight controllers are also carried out during the implementation phase. Controllers must be included in mission integration and test activities in order to have enough on-console time to be trusted during initial orbital operations. NASA should work with the PM to ensure proper training of flight controllers. Small missions typically have very small flight operations teams, making it all the more important that the individual controllers be thoroughly trained and experienced through mission simulations.

According to NPG 7120.5B, capturing the knowledge base requires the recording of lessons learned throughout the project and calls for the use of performance metrics to measure how well the project has performed and whether corrective action is necessary and possible. The use of performance metrics is an integral part of ISO 9001 quality management systems, compliance with which is a requirement for new NASA AOs.

On small projects this task involves incorporating any outstanding engineering change orders, closing any nonconformance or waiver requests, updating controlled project documents and flight equipment log books, and closing “fabrication travelers.”²¹ Most small project teams are fighting team exhaustion at this point in the project

and frequently overlook the knowledge-capturing process. NASA should work with the PM to ensure that this step is not overlooked so that if a problem occurs on orbit the engineering team will have accurate documentation to troubleshoot the problem. Troubleshooting will be difficult if the required drawings or software listings are out-of-date.

The implementation of PI-led projects for an ESSP-class mission is difficult at best. Resource constraints force small team size with corresponding high workloads for key team members. To promote success under these circumstances, PI-led projects should have:

1. Completely open communications between the PI team and the NASA MM;

2. A proactive NASA MM who becomes an integral part of the PI’s management team;

3. A good systems engineering process;

4. Stable requirements;

5. Adequate resources and margins;

6. A proactive schedule and cost management process that includes objective performance metrics;

7. A TRL of 6 or above;

8. A flexible and quick decision-making process;

9. A supportive institutional infrastructure at the PI’s home institution;

10. A proactive risk management process; and

11. A comprehensive test program at the observatory level that includes multiple mission simulations for training flight controllers.

Finding: While it may be appropriate for PI-led missions to use management processes that differ from NASA standards, NASA-defined minimum management standards are desirable to reduce programmatic risk to acceptable levels.

Recommendation: NASA’s Earth Science Enterprise should establish and enforce a comprehensive set of minimum standards for program management to be applied to all PI-led missions, while accepting that such missions may employ management processes that differ from those of NASA. These minimum management standards must invoke the rigor that experience has shown is required for success.