National Academies Press: OpenBook

Principal-Investigator-Led Missions in the Space Sciences (2006)

Chapter: 3 The Selection Process

« Previous: 2 PI-Led Programs, Roles, and Relationships
Suggested Citation:"3 The Selection Process." National Research Council. 2006. Principal-Investigator-Led Missions in the Space Sciences. Washington, DC: The National Academies Press. doi: 10.17226/11530.
×

3
The Selection Process

Announcements of opportunity (AOs) for the PI-led missions are released on a cycle that ranges between 2 and 4 years, depending on the program and the availability of funding for that mission line.1 The AOs evolve at each opportunity in response to NASA’s experience with the program and also in response to changes in priorities and practices driven by a range of matters—from administration initiatives to congressional concerns to the performance of missions inside and outside the PI-led programs (including human space missions). For example, the AOs issued after the investigations of the Mars failures of the late 1990s2 included requirements for additional reviews. More recent AOs require increases in budget reserves (from approximately 10 percent of mission costs to 25 to 30 percent through mission development)3 in response to the cost growth that occurred in recent Discovery missions and some Explorer missions (see Appendix E). With the exception of New Frontiers missions, the science area of a mission concept must stay within the basic program scopes (e.g., Sun-Earth Connections, astronomy and astrophysics, solar system exploration) mentioned in Chapter 2 but is not otherwise restricted. New Frontiers AOs specify several targeted science goals chosen by NASA Headquarters from community-based and agency prioritization studies—for example, the NRC decadal survey, New Frontiers in the Solar System, and NASA roadmaps.

MISSION PHASES AND MAJOR MILESTONES

For the purposes of the remaining discussion, it is necessary to understand that missions are generally broken up into phases. Figure 3.1 shows the basic stages of virtually all of NASA’s robotic spacecraft missions. Phase A (also known as the concept study) is the first stage after a proposal has been selected and reviewed. In a core mission, for which the PI is usually proposing an instrument or instrument suite, Phase A is typically

1  

Recent AOs can be found on the NASA Web site at <research.hq.nasa.gov/code_s/code_s.cfm>.

2  

NASA, 2000, Mars Program Independent Assessment Team, Summary Report. Available at <appl.nasa.gov/pdf/47305mainFBCspear.pdf>; NASA, 1999, Mars Climate Orbiter Mishap Investigation Board Phase I Report. Available at <ftp.hq.nasa.gov/pub/pao/reports/1999/M

3  

NASA Announcement of Opportunity for Explorer Program, Discovery Program 2004, and Missions of Opportunity, NNH04ZSS002, April 16, 2004.

Suggested Citation:"3 The Selection Process." National Research Council. 2006. Principal-Investigator-Led Missions in the Space Sciences. Washington, DC: The National Academies Press. doi: 10.17226/11530.
×

FIGURE 3.1 Life cycle of a PI-led mission.

Suggested Citation:"3 The Selection Process." National Research Council. 2006. Principal-Investigator-Led Missions in the Space Sciences. Washington, DC: The National Academies Press. doi: 10.17226/11530.
×

a noncompetitive activity during which the technical designs, costs, schedules, and management details are firmed up. In a PI-led mission, however, Phase A has similar goals but the competition for mission selection is still alive, as described below. Phase B (mission design and definition) is a postselection phase dedicated to making sufficient progress in technical designs and management of the mission concept; it ends with the technical preliminary design review (PDR) and programmatic mission confirmation review (MCR). Successful completion of the MCR allows the project to proceed into Phase C, the final design, which ends with the critical design review (CDR). The CDR is the last major engineering oversight hurdle before the flight hardware is built. Phases C and D then proceed with fabrication, testing, and delivery of flight hardware, ending in integration with the spacecraft and launch vehicle. After launch and a checkout period of 30 days or more, the mission operations phase begins. The operations plus science data analysis constitute Phase E, although the data analysis may not begin immediately if a long cruise is part of the mission. The duration of each and all of these phases is part of the proposed mission concept and is sometimes constrained by the AO requirements for a PI-led program.

PROPOSAL PROCESS

The selection of PI-led missions is a two-step process. In Step 1, a short list of proposed concepts is selected for a Step 2 competition, the equivalent of a Phase A definition study. (Figure 3.2 shows a generic selection process.) Step 1 proposal deadlines are typically preceded by a program-sponsored preproposal conference at which information is exchanged. Prospective proposers, including PIs and teaming institutions, usually attend these conferences to obtain clarifications on the AO and program requirements.

For PI-led programs, except New Frontiers, several tens of Step 1 proposals are typically submitted; two to four usually make the short list of Step 2 concepts. Finally, one or two missions are selected for further support, and the intent is to carry them to completion following a satisfactory final definition phase (Phase B). Thus the probability of having one’s proposal succeed is usually 5 to 10 percent at the outset (see Tables 3.1-3.4; mission names are given in full in Appendix H).

The AOs ask for a large amount of organizational, scientific, and technical detail within a strict page limit (typically 40 pages for the main body of the proposals). Proposers must state their science goals, team structure and responsibilities, management approach, including risk management and quality assurance (QA), spacecraft, payload integration plans, launch vehicle, mission operations, and data processing, archiving, and analysis plans. An education and public outreach (EPO) plan of specified minimum cost (presently at least 2 percent of mission cost) must be laid out and sometimes submitted and evaluated through separate NASA channels. PIs must also arrange and budget for the involvement of a small business, a minority institution, and (optional) a guest investigator or similar postlaunch science activity. To complete the proposal with the required detail, PIs must engage accounting and engineering professionals from their own organization or a teaming organization to construct a work breakdown structure (WBS), which is a set of budgetary spreadsheets showing all planned expenditures according to a proposed schedule for mission development, from selection through design definition (Phase B), design finalization, fabrication, integration, and testing (Phases C and D) to launch, mission operations, and science analysis (Phase E). The all-inclusive bottom-line budget for the mission, with launch vehicle and any Deep Space Network (DSN) or other antenna time included, must fit within the cost cap stated in the AO. Budgetary reserves, usually 25 percent of remaining mission costs at confirmation, are now a standard requirement for PI-led missions, as specified in the AOs.4

4  

See, for example, the AO for the Discovery PI-led mission issued on April 16, 2004, p. 18.

Suggested Citation:"3 The Selection Process." National Research Council. 2006. Principal-Investigator-Led Missions in the Space Sciences. Washington, DC: The National Academies Press. doi: 10.17226/11530.
×

FIGURE 3.2 Life cycle of a PI mission proposal.

The PI’s management job starts before selection. A WBS for team members and consistent, detailed estimated budgets for multiyear efforts must be created and formally approved by all of the participating institutions, requiring additional intensive interaction at administrative levels of the teaming institutions. At the same time, instrument, spacecraft, and subsystem descriptions must be sufficiently comprehensive to allow in-depth technical review by a panel of experts to assess whether the concept can be developed at the proposed cost and on the proposed schedule. Spacecraft contractors or NASA personnel must be engaged to provide detailed technical plans, including interfaces to the launch vehicle, the launch vehicle itself, navigation analysis, and acceptable QA practices, including parts and software standards. In addition to the flight system, all ground systems and data retrieval, processing, and dissemination activities must be

Suggested Citation:"3 The Selection Process." National Research Council. 2006. Principal-Investigator-Led Missions in the Space Sciences. Washington, DC: The National Academies Press. doi: 10.17226/11530.
×

TABLE 3.1 Statistics on Proposals Submitted for Explorer Missions

Year

Explorer

No. of Proposalsa

Phase A Studies

Selected Full Missions

1997

SMEX

46 total, 40 full

n/a

RHESSI, GALEX

1998

UNEX

29 total, 23 full

n/a

CHIPS, IMEX

1998

MIDEX

31 total, 27 full

5

Swift, FAME

1999

SMEX

33 total, 21 full

7

SPIDR, AIM

2001

MIDEX

31 total, 21 full

4

THEMIS, WISE

2003

SMEX

29 total, 22 full

5

IBEX, NuSTAR

a “Full” refers to full missions. “Total” number includes proposals for missions of opportunity.

SOURCE: NASA, Science Mission Directorate.

TABLE 3.2 Statistics on Proposals Submitted for Discovery Missions

Year of AO

No. of Proposals

Phase A Studies

Selected Full Missions

1996

34 full

5

CONTOUR, Genesis

1998

30 total, 26 full

6

MESSENGER, Deep Impact

2000

26 total, 23 full

3

Dawn, Kepler

2004

18 total, 16 full

0

None

 

SOURCE: NASA, Science Mission Directorate.

TABLE 3.3 Statistics on Proposals Submitted for One Mars Scout Mission

Year of AO

No. of Proposals

Phase A Studies

Selected Missions

2002

18

4

Phoenix

TABLE 3.4 Statistics on Proposals Submitted for New Frontiers

Year of AO

No. of Proposals

Phase A Studies

Selected Missions

2001

5

2

New Horizons

2004

5

2

Juno

 

SOURCE: NASA, Science Mission Directorate.

fully scheduled and budgeted. Risk management and mitigation strategies must incorporate a credible descope plan5 that preserves a minimum set of prime science goals.

Finally, the full proposal package, including subcontractor and teaming agreements, domestic and international, must be approved by the PI’s institution. These requirements can test the limits of some organizations’ capabilities.

International contributions require proof of commitment in the form of signed letters from officials of the providing institutions and/or agencies. Issues relevant to the International Traffic in Arms Regulations

5  

NASA requires that PIs include a descope plan when proposing for PI-led missions. This plan outlines the systems, instruments, or spacecraft components, mission operations, and schedule elements that will be eliminated from the mission should the mission experience cost or schedule problems. Any eliminations are planned with the goal of preserving the minimum acceptable science for the mission.

Suggested Citation:"3 The Selection Process." National Research Council. 2006. Principal-Investigator-Led Missions in the Space Sciences. Washington, DC: The National Academies Press. doi: 10.17226/11530.
×

(ITAR)6 must also be discussed within the proposal. The total cost estimate must be sufficiently accurate that the mission, if selected at Step 1, will not increase in cost by more than 20 percent of the proposed cost by the end of a Phase A study and that it will remain within the maximum allowed cost cap.

PROPOSAL REVIEW AND SELECTION PROCESSES

The Step 1 down selection involves two NASA-appointed review panels.7 One, the Science Support Office (SSO) at Langley, was established by the Office of Space Science (now the Science Mission Directorate) in 1996 to provide a neutral organization for the evaluation of Discovery and Explorer proposals. The SSO also assists Headquarters with AO preparation and other functions that support the mission AO and selection process. It now reviews the AOs for all NASA missions, including those for core mission programs and Earth science missions. A standard evaluation process focuses on the technical management, cost, and other (TMCO) parts of the proposed mission concept, leaving the science evaluation task to the second review panel, which comprises 20 to 30 scientists drawn from the nonproposing community and is managed by NASA Headquarters. Typically, 40 to 60 technical evaluators participate in the SSO reviews of PI-led missions at Step 1, including experts from private industry, government laboratories, and NASA centers not participating in the proposals. A primary concern of the TMCO panel is risk assessment on all fronts: leadership; organization; technical readiness (development) level; schedule and cost, with the last two considered in light of the first two.8

For Step 1 down selections, which reduce the competitive field, the goal of the TMCO review panel is to identify high-risk proposals regardless of their proposed science. In particular, independent cost assessments, carried out by experienced cost analysts, assess the proposed missions for their potential to grow in excess of 20 percent of the proposed cost. Such growth is one primary barrier to selection, even if the total cost remains within the program cost cap. Plenary sessions attended by all the TMCO reviewers balance the detailed reviews carried out by the various TMCO expert subpanels. The final rankings are reached in the plenary by consensus. The proposals are evaluated in terms of their major strengths, major weaknesses, minor strengths, and minor weaknesses in the area of technical management, cost, and other. One ground rule is that only major findings may influence the TMCO panel’s consensus ranking of implementation risk. Missions are then categorized by a committee that includes the chairs of the two evaluation panels, the NASA program scientist for the AO, a scientist from the science review panel, and other senior Headquarters representatives. This will be the first time that science and technical merit are considered together. The SSO evaluation for Step 1 proposals typically takes several months and costs the program an estimated 1-2 percent of the mission cap. After the results are announced, proposers whose concepts were not selected are given oral debriefings but not detailed written records of the panel’s technical, management, cost, and science analyses.9

6  

ITAR is a regulation pertaining to the export of items on the U.S. Munitions Control List (USMCL). Space science satellites were added to the USMCL in 2000 and are subject to ITAR, which is administered by the Department of State. ITAR concerns hardware for scientific satellites, launch vehicles, and defense articles and services such as the know-how, designs, operations, and other information associated with the hardware. Licenses are required in some cases for the export of ITAR-controlled technical data, hardware, and other defense articles and must be obtained from the Department of State.

7  

The committee’s information on the PI-led selection process was acquired, in large part, from R. Wayne Richie, who at the time was Mars Acquisition Manager, NASA Langley Research Center.

8  

R. Wayne Richie, Mars Acquisition Manager, NASA Langley Research Center, presentation to the committee on September 1, 2004.

9  

R. Wayne Richie, Mars Acquisition Manager, NASA Langley Research Center, presentation to the committee on September 1, 2004.

Suggested Citation:"3 The Selection Process." National Research Council. 2006. Principal-Investigator-Led Missions in the Space Sciences. Washington, DC: The National Academies Press. doi: 10.17226/11530.
×

Following the Step 1 down selection, in which the mission science is evaluated in parallel with the TMCO areas, short-listed winners are funded at <1 percent of mission cost to carry out a Step 2 review (equivalent to a Phase A study) in about 6 months. For the still-competitive Step 2 process, the Step 1 selectees must come up with a more refined and reliable implementation plan, with more accurate cost and schedule estimates for the final TMCO analysis. It is important at this step to have clearly defined the minimum science goals and requirements (Level 1 science) to be used in decision making and risk management. A Step 2 or Phase A concept study report (CSR) of approximately 150 pages is generated for a second comprehensive TMCO review, which is often conducted by a subset of the same panelists who carried out the Step 1 review. The panel, along with program and NASA Headquarters representatives, visits one of the primary teaming institutions as part of Step 2. One purpose of the site visit is to evaluate the adequacy of the project personnel, physical plant, and support infrastructure for carrying out such a major effort. Another purpose is to give the proposing team an opportunity to answer questions posed by the TMCO panel in the wake of its review of the CSR (Phase A report). At the conclusion of this process, one or two missions are finally chosen for development for flight.

Suggested Citation:"3 The Selection Process." National Research Council. 2006. Principal-Investigator-Led Missions in the Space Sciences. Washington, DC: The National Academies Press. doi: 10.17226/11530.
×
Page 24
Suggested Citation:"3 The Selection Process." National Research Council. 2006. Principal-Investigator-Led Missions in the Space Sciences. Washington, DC: The National Academies Press. doi: 10.17226/11530.
×
Page 25
Suggested Citation:"3 The Selection Process." National Research Council. 2006. Principal-Investigator-Led Missions in the Space Sciences. Washington, DC: The National Academies Press. doi: 10.17226/11530.
×
Page 26
Suggested Citation:"3 The Selection Process." National Research Council. 2006. Principal-Investigator-Led Missions in the Space Sciences. Washington, DC: The National Academies Press. doi: 10.17226/11530.
×
Page 27
Suggested Citation:"3 The Selection Process." National Research Council. 2006. Principal-Investigator-Led Missions in the Space Sciences. Washington, DC: The National Academies Press. doi: 10.17226/11530.
×
Page 28
Suggested Citation:"3 The Selection Process." National Research Council. 2006. Principal-Investigator-Led Missions in the Space Sciences. Washington, DC: The National Academies Press. doi: 10.17226/11530.
×
Page 29
Suggested Citation:"3 The Selection Process." National Research Council. 2006. Principal-Investigator-Led Missions in the Space Sciences. Washington, DC: The National Academies Press. doi: 10.17226/11530.
×
Page 30
Next: 4 Management of PI-Led Missions »
Principal-Investigator-Led Missions in the Space Sciences Get This Book
×
Buy Paperback | $45.00 Buy Ebook | $36.99
MyNAP members save 10% online.
Login or Register to save!
Download Free PDF

Principal Investigator-Led (PI-led) missions are an important element of NASA's space science enterprise. While several NRC studies have considered aspects of PI-led missions in the course of other studies for NASA, issues facing the PI-led missions in general have not been subject to much analysis in those studies. Nevertheless, these issues are raising increasingly important questions for NASA, and it requested the NRC to explore them as they currently affect PI-led missions. Among the issues NASA asked to have examined were those concerning cost and scheduling, the selection process, relationships among PI-led team members, and opportunities for knowledge transfer to new PIs. This report provides a discussion of the evolution and current status of the PIled mission concept, the ways in which certain practices have affected its performance, and the steps that can carry it successfully into the future. The study was done in collaboration with the National Academy of Public Administration.

  1. ×

    Welcome to OpenBook!

    You're looking at OpenBook, NAP.edu's online reading room since 1999. Based on feedback from you, our users, we've made some improvements that make it easier than ever to read thousands of publications on our website.

    Do you want to take a quick tour of the OpenBook's features?

    No Thanks Take a Tour »
  2. ×

    Show this book's table of contents, where you can jump to any chapter by name.

    « Back Next »
  3. ×

    ...or use these buttons to go back to the previous chapter or skip to the next one.

    « Back Next »
  4. ×

    Jump up to the previous page or down to the next one. Also, you can type in a page number and press Enter to go directly to that page in the book.

    « Back Next »
  5. ×

    Switch between the Original Pages, where you can read the report as it appeared in print, and Text Pages for the web version, where you can highlight and search the text.

    « Back Next »
  6. ×

    To search the entire text of this book, type in your search term here and press Enter.

    « Back Next »
  7. ×

    Share a link to this book page on your preferred social network or via email.

    « Back Next »
  8. ×

    View our suggested citation for this chapter.

    « Back Next »
  9. ×

    Ready to take your reading offline? Click here to buy this book in print or download it as a free PDF, if available.

    « Back Next »
Stay Connected!