3
The Selection Process
Announcements of opportunity (AOs) for the PI-led missions are released on a cycle that ranges between 2 and 4 years, depending on the program and the availability of funding for that mission line.1 The AOs evolve at each opportunity in response to NASA’s experience with the program and also in response to changes in priorities and practices driven by a range of matters—from administration initiatives to congressional concerns to the performance of missions inside and outside the PI-led programs (including human space missions). For example, the AOs issued after the investigations of the Mars failures of the late 1990s2 included requirements for additional reviews. More recent AOs require increases in budget reserves (from approximately 10 percent of mission costs to 25 to 30 percent through mission development)3 in response to the cost growth that occurred in recent Discovery missions and some Explorer missions (see Appendix E). With the exception of New Frontiers missions, the science area of a mission concept must stay within the basic program scopes (e.g., Sun-Earth Connections, astronomy and astrophysics, solar system exploration) mentioned in Chapter 2 but is not otherwise restricted. New Frontiers AOs specify several targeted science goals chosen by NASA Headquarters from community-based and agency prioritization studies—for example, the NRC decadal survey, New Frontiers in the Solar System, and NASA roadmaps.
MISSION PHASES AND MAJOR MILESTONES
For the purposes of the remaining discussion, it is necessary to understand that missions are generally broken up into phases. Figure 3.1 shows the basic stages of virtually all of NASA’s robotic spacecraft missions. Phase A (also known as the concept study) is the first stage after a proposal has been selected and reviewed. In a core mission, for which the PI is usually proposing an instrument or instrument suite, Phase A is typically
1 |
Recent AOs can be found on the NASA Web site at <research.hq.nasa.gov/code_s/code_s.cfm>. |
2 |
NASA, 2000, Mars Program Independent Assessment Team, Summary Report. Available at <appl.nasa.gov/pdf/47305mainFBCspear.pdf>; NASA, 1999, Mars Climate Orbiter Mishap Investigation Board Phase I Report. Available at <ftp.hq.nasa.gov/pub/pao/reports/1999/M |
3 |
NASA Announcement of Opportunity for Explorer Program, Discovery Program 2004, and Missions of Opportunity, NNH04ZSS002, April 16, 2004. |
a noncompetitive activity during which the technical designs, costs, schedules, and management details are firmed up. In a PI-led mission, however, Phase A has similar goals but the competition for mission selection is still alive, as described below. Phase B (mission design and definition) is a postselection phase dedicated to making sufficient progress in technical designs and management of the mission concept; it ends with the technical preliminary design review (PDR) and programmatic mission confirmation review (MCR). Successful completion of the MCR allows the project to proceed into Phase C, the final design, which ends with the critical design review (CDR). The CDR is the last major engineering oversight hurdle before the flight hardware is built. Phases C and D then proceed with fabrication, testing, and delivery of flight hardware, ending in integration with the spacecraft and launch vehicle. After launch and a checkout period of 30 days or more, the mission operations phase begins. The operations plus science data analysis constitute Phase E, although the data analysis may not begin immediately if a long cruise is part of the mission. The duration of each and all of these phases is part of the proposed mission concept and is sometimes constrained by the AO requirements for a PI-led program.
PROPOSAL PROCESS
The selection of PI-led missions is a two-step process. In Step 1, a short list of proposed concepts is selected for a Step 2 competition, the equivalent of a Phase A definition study. (Figure 3.2 shows a generic selection process.) Step 1 proposal deadlines are typically preceded by a program-sponsored preproposal conference at which information is exchanged. Prospective proposers, including PIs and teaming institutions, usually attend these conferences to obtain clarifications on the AO and program requirements.
For PI-led programs, except New Frontiers, several tens of Step 1 proposals are typically submitted; two to four usually make the short list of Step 2 concepts. Finally, one or two missions are selected for further support, and the intent is to carry them to completion following a satisfactory final definition phase (Phase B). Thus the probability of having one’s proposal succeed is usually 5 to 10 percent at the outset (see Tables 3.1-3.4; mission names are given in full in Appendix H).
The AOs ask for a large amount of organizational, scientific, and technical detail within a strict page limit (typically 40 pages for the main body of the proposals). Proposers must state their science goals, team structure and responsibilities, management approach, including risk management and quality assurance (QA), spacecraft, payload integration plans, launch vehicle, mission operations, and data processing, archiving, and analysis plans. An education and public outreach (EPO) plan of specified minimum cost (presently at least 2 percent of mission cost) must be laid out and sometimes submitted and evaluated through separate NASA channels. PIs must also arrange and budget for the involvement of a small business, a minority institution, and (optional) a guest investigator or similar postlaunch science activity. To complete the proposal with the required detail, PIs must engage accounting and engineering professionals from their own organization or a teaming organization to construct a work breakdown structure (WBS), which is a set of budgetary spreadsheets showing all planned expenditures according to a proposed schedule for mission development, from selection through design definition (Phase B), design finalization, fabrication, integration, and testing (Phases C and D) to launch, mission operations, and science analysis (Phase E). The all-inclusive bottom-line budget for the mission, with launch vehicle and any Deep Space Network (DSN) or other antenna time included, must fit within the cost cap stated in the AO. Budgetary reserves, usually 25 percent of remaining mission costs at confirmation, are now a standard requirement for PI-led missions, as specified in the AOs.4
The PI’s management job starts before selection. A WBS for team members and consistent, detailed estimated budgets for multiyear efforts must be created and formally approved by all of the participating institutions, requiring additional intensive interaction at administrative levels of the teaming institutions. At the same time, instrument, spacecraft, and subsystem descriptions must be sufficiently comprehensive to allow in-depth technical review by a panel of experts to assess whether the concept can be developed at the proposed cost and on the proposed schedule. Spacecraft contractors or NASA personnel must be engaged to provide detailed technical plans, including interfaces to the launch vehicle, the launch vehicle itself, navigation analysis, and acceptable QA practices, including parts and software standards. In addition to the flight system, all ground systems and data retrieval, processing, and dissemination activities must be
TABLE 3.1 Statistics on Proposals Submitted for Explorer Missions
Year |
Explorer |
No. of Proposalsa |
Phase A Studies |
Selected Full Missions |
1997 |
SMEX |
46 total, 40 full |
n/a |
RHESSI, GALEX |
1998 |
UNEX |
29 total, 23 full |
n/a |
CHIPS, IMEX |
1998 |
MIDEX |
31 total, 27 full |
5 |
Swift, FAME |
1999 |
SMEX |
33 total, 21 full |
7 |
SPIDR, AIM |
2001 |
MIDEX |
31 total, 21 full |
4 |
THEMIS, WISE |
2003 |
SMEX |
29 total, 22 full |
5 |
IBEX, NuSTAR |
a “Full” refers to full missions. “Total” number includes proposals for missions of opportunity. SOURCE: NASA, Science Mission Directorate. |
TABLE 3.2 Statistics on Proposals Submitted for Discovery Missions
Year of AO |
No. of Proposals |
Phase A Studies |
Selected Full Missions |
1996 |
34 full |
5 |
CONTOUR, Genesis |
1998 |
30 total, 26 full |
6 |
MESSENGER, Deep Impact |
2000 |
26 total, 23 full |
3 |
Dawn, Kepler |
2004 |
18 total, 16 full |
0 |
None |
SOURCE: NASA, Science Mission Directorate. |
TABLE 3.3 Statistics on Proposals Submitted for One Mars Scout Mission
Year of AO |
No. of Proposals |
Phase A Studies |
Selected Missions |
2002 |
18 |
4 |
Phoenix |
TABLE 3.4 Statistics on Proposals Submitted for New Frontiers
Year of AO |
No. of Proposals |
Phase A Studies |
Selected Missions |
2001 |
5 |
2 |
New Horizons |
2004 |
5 |
2 |
Juno |
SOURCE: NASA, Science Mission Directorate. |
fully scheduled and budgeted. Risk management and mitigation strategies must incorporate a credible descope plan5 that preserves a minimum set of prime science goals.
Finally, the full proposal package, including subcontractor and teaming agreements, domestic and international, must be approved by the PI’s institution. These requirements can test the limits of some organizations’ capabilities.
International contributions require proof of commitment in the form of signed letters from officials of the providing institutions and/or agencies. Issues relevant to the International Traffic in Arms Regulations
(ITAR)6 must also be discussed within the proposal. The total cost estimate must be sufficiently accurate that the mission, if selected at Step 1, will not increase in cost by more than 20 percent of the proposed cost by the end of a Phase A study and that it will remain within the maximum allowed cost cap.
PROPOSAL REVIEW AND SELECTION PROCESSES
The Step 1 down selection involves two NASA-appointed review panels.7 One, the Science Support Office (SSO) at Langley, was established by the Office of Space Science (now the Science Mission Directorate) in 1996 to provide a neutral organization for the evaluation of Discovery and Explorer proposals. The SSO also assists Headquarters with AO preparation and other functions that support the mission AO and selection process. It now reviews the AOs for all NASA missions, including those for core mission programs and Earth science missions. A standard evaluation process focuses on the technical management, cost, and other (TMCO) parts of the proposed mission concept, leaving the science evaluation task to the second review panel, which comprises 20 to 30 scientists drawn from the nonproposing community and is managed by NASA Headquarters. Typically, 40 to 60 technical evaluators participate in the SSO reviews of PI-led missions at Step 1, including experts from private industry, government laboratories, and NASA centers not participating in the proposals. A primary concern of the TMCO panel is risk assessment on all fronts: leadership; organization; technical readiness (development) level; schedule and cost, with the last two considered in light of the first two.8
For Step 1 down selections, which reduce the competitive field, the goal of the TMCO review panel is to identify high-risk proposals regardless of their proposed science. In particular, independent cost assessments, carried out by experienced cost analysts, assess the proposed missions for their potential to grow in excess of 20 percent of the proposed cost. Such growth is one primary barrier to selection, even if the total cost remains within the program cost cap. Plenary sessions attended by all the TMCO reviewers balance the detailed reviews carried out by the various TMCO expert subpanels. The final rankings are reached in the plenary by consensus. The proposals are evaluated in terms of their major strengths, major weaknesses, minor strengths, and minor weaknesses in the area of technical management, cost, and other. One ground rule is that only major findings may influence the TMCO panel’s consensus ranking of implementation risk. Missions are then categorized by a committee that includes the chairs of the two evaluation panels, the NASA program scientist for the AO, a scientist from the science review panel, and other senior Headquarters representatives. This will be the first time that science and technical merit are considered together. The SSO evaluation for Step 1 proposals typically takes several months and costs the program an estimated 1-2 percent of the mission cap. After the results are announced, proposers whose concepts were not selected are given oral debriefings but not detailed written records of the panel’s technical, management, cost, and science analyses.9
Following the Step 1 down selection, in which the mission science is evaluated in parallel with the TMCO areas, short-listed winners are funded at <1 percent of mission cost to carry out a Step 2 review (equivalent to a Phase A study) in about 6 months. For the still-competitive Step 2 process, the Step 1 selectees must come up with a more refined and reliable implementation plan, with more accurate cost and schedule estimates for the final TMCO analysis. It is important at this step to have clearly defined the minimum science goals and requirements (Level 1 science) to be used in decision making and risk management. A Step 2 or Phase A concept study report (CSR) of approximately 150 pages is generated for a second comprehensive TMCO review, which is often conducted by a subset of the same panelists who carried out the Step 1 review. The panel, along with program and NASA Headquarters representatives, visits one of the primary teaming institutions as part of Step 2. One purpose of the site visit is to evaluate the adequacy of the project personnel, physical plant, and support infrastructure for carrying out such a major effort. Another purpose is to give the proposing team an opportunity to answer questions posed by the TMCO panel in the wake of its review of the CSR (Phase A report). At the conclusion of this process, one or two missions are finally chosen for development for flight.