3
The Selection Process

Announcements of opportunity (AOs) for the PI-led missions are released on a cycle that ranges between 2 and 4 years, depending on the program and the availability of funding for that mission line.1 The AOs evolve at each opportunity in response to NASA’s experience with the program and also in response to changes in priorities and practices driven by a range of matters—from administration initiatives to congressional concerns to the performance of missions inside and outside the PI-led programs (including human space missions). For example, the AOs issued after the investigations of the Mars failures of the late 1990s2 included requirements for additional reviews. More recent AOs require increases in budget reserves (from approximately 10 percent of mission costs to 25 to 30 percent through mission development)3 in response to the cost growth that occurred in recent Discovery missions and some Explorer missions (see Appendix E). With the exception of New Frontiers missions, the science area of a mission concept must stay within the basic program scopes (e.g., Sun-Earth Connections, astronomy and astrophysics, solar system exploration) mentioned in Chapter 2 but is not otherwise restricted. New Frontiers AOs specify several targeted science goals chosen by NASA Headquarters from community-based and agency prioritization studies—for example, the NRC decadal survey, New Frontiers in the Solar System, and NASA roadmaps.

MISSION PHASES AND MAJOR MILESTONES

For the purposes of the remaining discussion, it is necessary to understand that missions are generally broken up into phases. Figure 3.1 shows the basic stages of virtually all of NASA’s robotic spacecraft missions. Phase A (also known as the concept study) is the first stage after a proposal has been selected and reviewed. In a core mission, for which the PI is usually proposing an instrument or instrument suite, Phase A is typically

1  

Recent AOs can be found on the NASA Web site at <research.hq.nasa.gov/code_s/code_s.cfm>.

2  

NASA, 2000, Mars Program Independent Assessment Team, Summary Report. Available at <appl.nasa.gov/pdf/47305mainFBCspear.pdf>; NASA, 1999, Mars Climate Orbiter Mishap Investigation Board Phase I Report. Available at <ftp.hq.nasa.gov/pub/pao/reports/1999/M

3  

NASA Announcement of Opportunity for Explorer Program, Discovery Program 2004, and Missions of Opportunity, NNH04ZSS002, April 16, 2004.



The National Academies | 500 Fifth St. N.W. | Washington, D.C. 20001
Copyright © National Academy of Sciences. All rights reserved.
Terms of Use and Privacy Statement



Below are the first 10 and last 10 pages of uncorrected machine-read text (when available) of this chapter, followed by the top 30 algorithmically extracted key phrases from the chapter as a whole.
Intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text on the opening pages of each chapter. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.

Do not use for reproduction, copying, pasting, or reading; exclusively for search engines.

OCR for page 24
Principal-Investigator-Led Missions in the Space Sciences 3 The Selection Process Announcements of opportunity (AOs) for the PI-led missions are released on a cycle that ranges between 2 and 4 years, depending on the program and the availability of funding for that mission line.1 The AOs evolve at each opportunity in response to NASA’s experience with the program and also in response to changes in priorities and practices driven by a range of matters—from administration initiatives to congressional concerns to the performance of missions inside and outside the PI-led programs (including human space missions). For example, the AOs issued after the investigations of the Mars failures of the late 1990s2 included requirements for additional reviews. More recent AOs require increases in budget reserves (from approximately 10 percent of mission costs to 25 to 30 percent through mission development)3 in response to the cost growth that occurred in recent Discovery missions and some Explorer missions (see Appendix E). With the exception of New Frontiers missions, the science area of a mission concept must stay within the basic program scopes (e.g., Sun-Earth Connections, astronomy and astrophysics, solar system exploration) mentioned in Chapter 2 but is not otherwise restricted. New Frontiers AOs specify several targeted science goals chosen by NASA Headquarters from community-based and agency prioritization studies—for example, the NRC decadal survey, New Frontiers in the Solar System, and NASA roadmaps. MISSION PHASES AND MAJOR MILESTONES For the purposes of the remaining discussion, it is necessary to understand that missions are generally broken up into phases. Figure 3.1 shows the basic stages of virtually all of NASA’s robotic spacecraft missions. Phase A (also known as the concept study) is the first stage after a proposal has been selected and reviewed. In a core mission, for which the PI is usually proposing an instrument or instrument suite, Phase A is typically 1   Recent AOs can be found on the NASA Web site at <research.hq.nasa.gov/code_s/code_s.cfm>. 2   NASA, 2000, Mars Program Independent Assessment Team, Summary Report. Available at <appl.nasa.gov/pdf/47305mainFBCspear.pdf>; NASA, 1999, Mars Climate Orbiter Mishap Investigation Board Phase I Report. Available at <ftp.hq.nasa.gov/pub/pao/reports/1999/M 3   NASA Announcement of Opportunity for Explorer Program, Discovery Program 2004, and Missions of Opportunity, NNH04ZSS002, April 16, 2004.

OCR for page 24
Principal-Investigator-Led Missions in the Space Sciences FIGURE 3.1 Life cycle of a PI-led mission.

OCR for page 24
Principal-Investigator-Led Missions in the Space Sciences a noncompetitive activity during which the technical designs, costs, schedules, and management details are firmed up. In a PI-led mission, however, Phase A has similar goals but the competition for mission selection is still alive, as described below. Phase B (mission design and definition) is a postselection phase dedicated to making sufficient progress in technical designs and management of the mission concept; it ends with the technical preliminary design review (PDR) and programmatic mission confirmation review (MCR). Successful completion of the MCR allows the project to proceed into Phase C, the final design, which ends with the critical design review (CDR). The CDR is the last major engineering oversight hurdle before the flight hardware is built. Phases C and D then proceed with fabrication, testing, and delivery of flight hardware, ending in integration with the spacecraft and launch vehicle. After launch and a checkout period of 30 days or more, the mission operations phase begins. The operations plus science data analysis constitute Phase E, although the data analysis may not begin immediately if a long cruise is part of the mission. The duration of each and all of these phases is part of the proposed mission concept and is sometimes constrained by the AO requirements for a PI-led program. PROPOSAL PROCESS The selection of PI-led missions is a two-step process. In Step 1, a short list of proposed concepts is selected for a Step 2 competition, the equivalent of a Phase A definition study. (Figure 3.2 shows a generic selection process.) Step 1 proposal deadlines are typically preceded by a program-sponsored preproposal conference at which information is exchanged. Prospective proposers, including PIs and teaming institutions, usually attend these conferences to obtain clarifications on the AO and program requirements. For PI-led programs, except New Frontiers, several tens of Step 1 proposals are typically submitted; two to four usually make the short list of Step 2 concepts. Finally, one or two missions are selected for further support, and the intent is to carry them to completion following a satisfactory final definition phase (Phase B). Thus the probability of having one’s proposal succeed is usually 5 to 10 percent at the outset (see Tables 3.1-3.4; mission names are given in full in Appendix H). The AOs ask for a large amount of organizational, scientific, and technical detail within a strict page limit (typically 40 pages for the main body of the proposals). Proposers must state their science goals, team structure and responsibilities, management approach, including risk management and quality assurance (QA), spacecraft, payload integration plans, launch vehicle, mission operations, and data processing, archiving, and analysis plans. An education and public outreach (EPO) plan of specified minimum cost (presently at least 2 percent of mission cost) must be laid out and sometimes submitted and evaluated through separate NASA channels. PIs must also arrange and budget for the involvement of a small business, a minority institution, and (optional) a guest investigator or similar postlaunch science activity. To complete the proposal with the required detail, PIs must engage accounting and engineering professionals from their own organization or a teaming organization to construct a work breakdown structure (WBS), which is a set of budgetary spreadsheets showing all planned expenditures according to a proposed schedule for mission development, from selection through design definition (Phase B), design finalization, fabrication, integration, and testing (Phases C and D) to launch, mission operations, and science analysis (Phase E). The all-inclusive bottom-line budget for the mission, with launch vehicle and any Deep Space Network (DSN) or other antenna time included, must fit within the cost cap stated in the AO. Budgetary reserves, usually 25 percent of remaining mission costs at confirmation, are now a standard requirement for PI-led missions, as specified in the AOs.4 4   See, for example, the AO for the Discovery PI-led mission issued on April 16, 2004, p. 18.

OCR for page 24
Principal-Investigator-Led Missions in the Space Sciences FIGURE 3.2 Life cycle of a PI mission proposal. The PI’s management job starts before selection. A WBS for team members and consistent, detailed estimated budgets for multiyear efforts must be created and formally approved by all of the participating institutions, requiring additional intensive interaction at administrative levels of the teaming institutions. At the same time, instrument, spacecraft, and subsystem descriptions must be sufficiently comprehensive to allow in-depth technical review by a panel of experts to assess whether the concept can be developed at the proposed cost and on the proposed schedule. Spacecraft contractors or NASA personnel must be engaged to provide detailed technical plans, including interfaces to the launch vehicle, the launch vehicle itself, navigation analysis, and acceptable QA practices, including parts and software standards. In addition to the flight system, all ground systems and data retrieval, processing, and dissemination activities must be

OCR for page 24
Principal-Investigator-Led Missions in the Space Sciences TABLE 3.1 Statistics on Proposals Submitted for Explorer Missions Year Explorer No. of Proposalsa Phase A Studies Selected Full Missions 1997 SMEX 46 total, 40 full n/a RHESSI, GALEX 1998 UNEX 29 total, 23 full n/a CHIPS, IMEX 1998 MIDEX 31 total, 27 full 5 Swift, FAME 1999 SMEX 33 total, 21 full 7 SPIDR, AIM 2001 MIDEX 31 total, 21 full 4 THEMIS, WISE 2003 SMEX 29 total, 22 full 5 IBEX, NuSTAR a “Full” refers to full missions. “Total” number includes proposals for missions of opportunity. SOURCE: NASA, Science Mission Directorate. TABLE 3.2 Statistics on Proposals Submitted for Discovery Missions Year of AO No. of Proposals Phase A Studies Selected Full Missions 1996 34 full 5 CONTOUR, Genesis 1998 30 total, 26 full 6 MESSENGER, Deep Impact 2000 26 total, 23 full 3 Dawn, Kepler 2004 18 total, 16 full 0 None   SOURCE: NASA, Science Mission Directorate. TABLE 3.3 Statistics on Proposals Submitted for One Mars Scout Mission Year of AO No. of Proposals Phase A Studies Selected Missions 2002 18 4 Phoenix TABLE 3.4 Statistics on Proposals Submitted for New Frontiers Year of AO No. of Proposals Phase A Studies Selected Missions 2001 5 2 New Horizons 2004 5 2 Juno   SOURCE: NASA, Science Mission Directorate. fully scheduled and budgeted. Risk management and mitigation strategies must incorporate a credible descope plan5 that preserves a minimum set of prime science goals. Finally, the full proposal package, including subcontractor and teaming agreements, domestic and international, must be approved by the PI’s institution. These requirements can test the limits of some organizations’ capabilities. International contributions require proof of commitment in the form of signed letters from officials of the providing institutions and/or agencies. Issues relevant to the International Traffic in Arms Regulations 5   NASA requires that PIs include a descope plan when proposing for PI-led missions. This plan outlines the systems, instruments, or spacecraft components, mission operations, and schedule elements that will be eliminated from the mission should the mission experience cost or schedule problems. Any eliminations are planned with the goal of preserving the minimum acceptable science for the mission.

OCR for page 24
Principal-Investigator-Led Missions in the Space Sciences (ITAR)6 must also be discussed within the proposal. The total cost estimate must be sufficiently accurate that the mission, if selected at Step 1, will not increase in cost by more than 20 percent of the proposed cost by the end of a Phase A study and that it will remain within the maximum allowed cost cap. PROPOSAL REVIEW AND SELECTION PROCESSES The Step 1 down selection involves two NASA-appointed review panels.7 One, the Science Support Office (SSO) at Langley, was established by the Office of Space Science (now the Science Mission Directorate) in 1996 to provide a neutral organization for the evaluation of Discovery and Explorer proposals. The SSO also assists Headquarters with AO preparation and other functions that support the mission AO and selection process. It now reviews the AOs for all NASA missions, including those for core mission programs and Earth science missions. A standard evaluation process focuses on the technical management, cost, and other (TMCO) parts of the proposed mission concept, leaving the science evaluation task to the second review panel, which comprises 20 to 30 scientists drawn from the nonproposing community and is managed by NASA Headquarters. Typically, 40 to 60 technical evaluators participate in the SSO reviews of PI-led missions at Step 1, including experts from private industry, government laboratories, and NASA centers not participating in the proposals. A primary concern of the TMCO panel is risk assessment on all fronts: leadership; organization; technical readiness (development) level; schedule and cost, with the last two considered in light of the first two.8 For Step 1 down selections, which reduce the competitive field, the goal of the TMCO review panel is to identify high-risk proposals regardless of their proposed science. In particular, independent cost assessments, carried out by experienced cost analysts, assess the proposed missions for their potential to grow in excess of 20 percent of the proposed cost. Such growth is one primary barrier to selection, even if the total cost remains within the program cost cap. Plenary sessions attended by all the TMCO reviewers balance the detailed reviews carried out by the various TMCO expert subpanels. The final rankings are reached in the plenary by consensus. The proposals are evaluated in terms of their major strengths, major weaknesses, minor strengths, and minor weaknesses in the area of technical management, cost, and other. One ground rule is that only major findings may influence the TMCO panel’s consensus ranking of implementation risk. Missions are then categorized by a committee that includes the chairs of the two evaluation panels, the NASA program scientist for the AO, a scientist from the science review panel, and other senior Headquarters representatives. This will be the first time that science and technical merit are considered together. The SSO evaluation for Step 1 proposals typically takes several months and costs the program an estimated 1-2 percent of the mission cap. After the results are announced, proposers whose concepts were not selected are given oral debriefings but not detailed written records of the panel’s technical, management, cost, and science analyses.9 6   ITAR is a regulation pertaining to the export of items on the U.S. Munitions Control List (USMCL). Space science satellites were added to the USMCL in 2000 and are subject to ITAR, which is administered by the Department of State. ITAR concerns hardware for scientific satellites, launch vehicles, and defense articles and services such as the know-how, designs, operations, and other information associated with the hardware. Licenses are required in some cases for the export of ITAR-controlled technical data, hardware, and other defense articles and must be obtained from the Department of State. 7   The committee’s information on the PI-led selection process was acquired, in large part, from R. Wayne Richie, who at the time was Mars Acquisition Manager, NASA Langley Research Center. 8   R. Wayne Richie, Mars Acquisition Manager, NASA Langley Research Center, presentation to the committee on September 1, 2004. 9   R. Wayne Richie, Mars Acquisition Manager, NASA Langley Research Center, presentation to the committee on September 1, 2004.

OCR for page 24
Principal-Investigator-Led Missions in the Space Sciences Following the Step 1 down selection, in which the mission science is evaluated in parallel with the TMCO areas, short-listed winners are funded at <1 percent of mission cost to carry out a Step 2 review (equivalent to a Phase A study) in about 6 months. For the still-competitive Step 2 process, the Step 1 selectees must come up with a more refined and reliable implementation plan, with more accurate cost and schedule estimates for the final TMCO analysis. It is important at this step to have clearly defined the minimum science goals and requirements (Level 1 science) to be used in decision making and risk management. A Step 2 or Phase A concept study report (CSR) of approximately 150 pages is generated for a second comprehensive TMCO review, which is often conducted by a subset of the same panelists who carried out the Step 1 review. The panel, along with program and NASA Headquarters representatives, visits one of the primary teaming institutions as part of Step 2. One purpose of the site visit is to evaluate the adequacy of the project personnel, physical plant, and support infrastructure for carrying out such a major effort. Another purpose is to give the proposing team an opportunity to answer questions posed by the TMCO panel in the wake of its review of the CSR (Phase A report). At the conclusion of this process, one or two missions are finally chosen for development for flight.