I
Excerpt from the National Academy of Public Administration Report NASA: Principal Investigator Led Missions in Space Science

SECTION II: FINDINGS, ANALYSIS, AND RECOMMENDATIONS

The history of cost overruns in space flight projects shows that the greatest overruns occur when there is a mismatch between the project’s cost estimate and the scope of the engineering challenge required to achieve a project’s scientific and/or engineering objectives. The mismatch is driven by a number of potential factors. Commonly, the cost estimate has been prepared in an environment influenced by externally dictated “affordability” constraints. Often, the engineering solutions are not well understood for spacecraft systems or instruments that are critical to the mission’s success. Key assumptions used for preparing the estimates prove to be inaccurate. For example: frequently, the resources (funds, talent, facilities, launchers) aren’t available in a timely manner due to another project’s higher priority; occasionally, the project leadership is not up to the management challenge; quite often, the external environment for failure tolerance changes, and new design/test requirements are imposed; and, delivery and quality control problems are encountered that consume the contingencies included in the schedule and cost plans. Finally, the cost estimators simply are handicapped by a lack of relevant prior experience; they lack appropriate cost analogies which could provide a basis of comparison.

NASA program managers have several tools at their disposal to reduce the probability that a new project will be affected by one or more of these factors. First, they can ensure the conditions established in the Announcement of Opportunity do not inadvertently contribute to the creation of a mismatch. The cost caps, schedule caps, funding constraints, and review cycles can reflect the desired science return and the imposed management conditions. Second, they can use unbiased external reviewers with relevant science, engineering, and cost estimating experience to provide the critical insights needed to make informed decisions during the project’s life cycle. Third, they can make allowances for unanticipated but mandated changes, the adverse effects of non-anticipatable events, and anticipatable but unrecognized resource

NOTE: National Academy of Public Administration (NAPA). 2005. NASA: Principal Investigator-Led Missions in Space Science. Washington, D.C., pp. 17-30.



The National Academies | 500 Fifth St. N.W. | Washington, D.C. 20001
Copyright © National Academy of Sciences. All rights reserved.
Terms of Use and Privacy Statement



Below are the first 10 and last 10 pages of uncorrected machine-read text (when available) of this chapter, followed by the top 30 algorithmically extracted key phrases from the chapter as a whole.
Intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text on the opening pages of each chapter. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.

Do not use for reproduction, copying, pasting, or reading; exclusively for search engines.

OCR for page 109
Principal-Investigator-Led Missions in the Space Sciences I Excerpt from the National Academy of Public Administration Report NASA: Principal Investigator Led Missions in Space Science SECTION II: FINDINGS, ANALYSIS, AND RECOMMENDATIONS The history of cost overruns in space flight projects shows that the greatest overruns occur when there is a mismatch between the project’s cost estimate and the scope of the engineering challenge required to achieve a project’s scientific and/or engineering objectives. The mismatch is driven by a number of potential factors. Commonly, the cost estimate has been prepared in an environment influenced by externally dictated “affordability” constraints. Often, the engineering solutions are not well understood for spacecraft systems or instruments that are critical to the mission’s success. Key assumptions used for preparing the estimates prove to be inaccurate. For example: frequently, the resources (funds, talent, facilities, launchers) aren’t available in a timely manner due to another project’s higher priority; occasionally, the project leadership is not up to the management challenge; quite often, the external environment for failure tolerance changes, and new design/test requirements are imposed; and, delivery and quality control problems are encountered that consume the contingencies included in the schedule and cost plans. Finally, the cost estimators simply are handicapped by a lack of relevant prior experience; they lack appropriate cost analogies which could provide a basis of comparison. NASA program managers have several tools at their disposal to reduce the probability that a new project will be affected by one or more of these factors. First, they can ensure the conditions established in the Announcement of Opportunity do not inadvertently contribute to the creation of a mismatch. The cost caps, schedule caps, funding constraints, and review cycles can reflect the desired science return and the imposed management conditions. Second, they can use unbiased external reviewers with relevant science, engineering, and cost estimating experience to provide the critical insights needed to make informed decisions during the project’s life cycle. Third, they can make allowances for unanticipated but mandated changes, the adverse effects of non-anticipatable events, and anticipatable but unrecognized resource NOTE: National Academy of Public Administration (NAPA). 2005. NASA: Principal Investigator-Led Missions in Space Science. Washington, D.C., pp. 17-30.

OCR for page 109
Principal-Investigator-Led Missions in the Space Sciences constraints that lead to inefficient expenditures.21 Fourth, they can ensure a match between the managerial and engineering talent furnished to the project and the challenges inherent in the project. Finally, they can reduce the probability of deliberate underestimating of the “basic cost estimate”—the estimate for the baseline mission elements before the application of reserves—by insisting on a high confidence in that cost estimate. As discussed in the prior section, NASA program managers do avail themselves of these tools, and have made adjustments in their management approach in response to changed circumstances. The findings and recommendations provided below should be viewed as building on and strengthening current NASA practices. The Academy team’s examination of the cost growth of projects from the baseline without reserves (as seen in “the internal cost growth” for Discovery projects) indicates how seriously underestimated the starting estimates for many projects have been. The team’s findings, analysis, and proposed corrective actions are presented below. The analysis is broken down by two principal processes. The first examines how the proposal process for PI-led missions influences the quality of the basic cost estimates. The second focuses on the mission development process. The Proposal Process Finding 1: The proposal and selection process has characteristics and limitations that encourage the submission of optimistic basic mission cost proposals for science missions. The AO proposal process encourages a proposer to adopt strategies that emphasize the prospects for exciting science returns from innovative missions that explore new environments, using a mixture of proven and new technologies. Proposers are astutely packaging their proposals to address NASA’s expectations—as articulated in the AO and NASA program management directives—in terms of accepted project management processes. And proposers are very aware of the need to meet NASA’s stipulation that a specified percentage level of project cost reserves be shown as available at key milestones before the project will be approved for development start. However, the specific rules set up by NASA in the AO work against proposers providing NASA with proposal cost estimates for the baselined mission content that have a higher probability of being successfully executed without reliance on project cost reserves. First, the proposers must satisfy NASA’s emphasis on the amount of reserves included, where higher levels of reserves in the estimates are evaluated as “a strength.” The AO specifies a percentage of “unencumbered reserves” at the time of confirmation for development of at least 25 percent of all development costs in phases C and D; if the unencumbered reserves are lower than 25 percent, the projects “are likely to be judged as having an unacceptably high cost risk and, therefore, not confirmed for further development.”22 As a consequence, the proposers are encouraged to prepare estimates that ensure the required level of reserves can be displayed. The Academy team’s interviews suggest that proposers are achieving this requirement by using more “optimistic” costing assumptions in their basic cost estimates. 21   Obvious examples are delays due to technical facilities not being available, lack of access to the most talented personnel, funding limitations, delays in launch vehicle readiness, and, quality/availability problems with electronic piece parts. 22   See Announcement of Opportunity, New Frontiers Program 2003 and Missions of Opportunity, NASA AO 03-OSS.

OCR for page 109
Principal-Investigator-Led Missions in the Space Sciences Second, the rules of the AO process call for a focus on total mission costs, including the amounts spent during phases A and B. This focus on the end-to-end mission costs at the time of the step 1 proposal establishes a presumption of a level of maturity in the cost estimates that is normally achieved only by the end of the formulation phase. Indeed, the proposer is encouraged to include statements that stress the positives—e.g., a stable design and mature technologies—and minimize the negatives. There are no incentives for a proposal team to propose formulation phase “design trades” to reduce technical risks, even if the trades were to be funded using its own resources. It is better to be quiet about the design risks than to signal to evaluators that there is a design maturity issue. The Academy team also noted that there are even greater disincentives for a proposal team that proposes NASA funds be used during phases A and B for design trades: the expenditures are counted against the cost cap, thereby reducing the total funds available for development, and, in effect, applying funds that otherwise could be counted as “reserves.” Third, proposers are fully aware that a mission plan intended to achieve truly exciting science returns often requires departures from the “experience base.” Employing existing technologies and exploring well-understood spacecraft operating environments are avenues that markedly reduce cost and schedule risks. However, as an example, the exploration of Mercury by the MESSENGER spacecraft clearly necessitated the introduction of advanced materials and technologies and exposure to a poorly understood, stressful spacecraft thermal operating environment; the cost growth experienced during the MESSENGER’s development phase had much to do with the inherent challenges required to achieve the desired science return. The PIs interviewed clearly stated the conflict between achieving exciting science, and achieving lower cost and schedule risks. However, the fierce competition among proposal teams leads to the understanding that step 1 proposals that promise high science return are favored over those with reduced science return but lower cost and schedule risks. This conundrum is an incentive to the proposal teams to take their chances that the step 1 independent reviewers will not fully appreciate the cost and schedule risks inherent in achieving the desired science return. Fourth, NASA has been slow to adjust cost caps to reflect the true impacts of the post-NIAT environment and the progression toward more difficult science exploration missions (the simpler and cheaper missions have already been done or are not as exciting).23 Fifth, the AO restricts the potential cost increase to 20 percent from the time the proposal is received to the concept feasibility report published at the end of phase A, and in no case can it be higher than the cost cap. There is little incentive for proposers to alter the initial cost estimates to reflect their sense of “realism” unless the proposers view implementation of a descoping option as viable. The Academy study team concluded that the only way the project team can achieve, without penalty, a higher level of formulation phase expenditures sufficient to permit design trades to be undertaken to refine the design or to develop engineering testbeds is to do so without NASA funding. The Academy team’s interviews revealed this is the practice the proposal teams have adopted; the proposal teams at least match 23   The PIs and PMs have argued that the cost caps in the AOs should be increased to reflect this. The increase in total mission cost limitations in recent NASA AOs shows that NASA’s Science Mission Directorate has recognized the merits of this argument.

OCR for page 109
Principal-Investigator-Led Missions in the Space Sciences if not exceed the limited amounts of funding NASA provides during phase A.24 In the step 1 proposals, the plans for phase B also have to be provided in some detail; again, there is little incentive for a team to put forth a plan of technology trades or for testbeds that consume funds otherwise available for reserves. In addition, NASA specifies an allowable funding profile that it expects proposers to adhere to in their planning; this militates against a proposer planning a more robust formulation phase than NASA has allowed for in the allowable spending profile. The Academy team also considered the merits of NASA’s strategy of relying on descoping options that could be put into effect should the level of reserves diminish to an unacceptable level. The proposers have to provide a detailed plan for descoping to the “minimum science floor.” The interviewees said this strategy is of limited effectiveness during the formulation phase. Its limitations stem from the incentives for proposers to maintain as much science capability as possible; thus, the proposers are likely to overstate the cost savings achieved by implementing specific descoping options. Because the objective is to be confirmed for development, with a key being NASA’s evaluation of reserve levels, the proposers have every incentive to overstate the savings and little incentive to recognize the additive costs that result from redesigns. The interviewees noted that NASA had the ability to verify the savings from eliminating capabilities, but lacked the detailed insight required to appreciate the costs of redesigns. During implementation, several project managers noted that pre-planned descopes were considered to be of minimal effectiveness in restoring reserves, because post-design review changes in the technical requirements require redesigns that have broader systems engineering and integration impacts. The recognition of the need for changes that will reduce costs is usually six months to a year after the critical design review, too late to implement descopes that could generate appreciable savings. The Academy team concluded that changes to the AO provisions that govern the proposal’s cost estimates could present the best approach to reducing the probability of cost overruns. Instead of relying upon stipulated reserve levels to address uncertainties, NASA could request the proposer to address the uncertainties by attributing a confidence level to the baseline estimate, and showing the magnitude of that uncertainty by providing lower and higher values commensurate with increased and decreased levels of cost estimating uncertainties. By requiring a three point estimating approach, NASA could also gain insight into the areas of greatest technical risk, that is, where there are the greatest dispersions in the low-medium-high confidence estimates. However, the Academy team was concerned about the level of additional investment the proposal teams would have to make to respond to the AO with their step 1 proposals. Although this would be of reduced concern for proposers whose proposals are built on the work expended during previous AO submissions, the new requirement could affect the first-time proposer in assembling the necessary teammates and getting the teammates to invest their resources to generate a proposal. As a consequence, the Academy team concluded the objective intent of the proposed change could be met by requiring the three point estimate as a product of the Concept Feasibility Study. If NASA evaluated the risks as acceptable and approved the proposal for the phase B definition and design studies, NASA could also work with the PI’s team to ensure the schedule and funding resources expended during the preliminary design studies were tailored to reduce the areas of greatest cost estimating uncertainty. The NASA decision official could accordingly focus his Confirmation Review on whether the basic cost estimate had a high level of confidence. 24   The amounts provided by NASA for phase A studies have increased over time. The 1998 Discovery AO provided for $375 thousand; the 2000 Discovery AO provided for $450 thousand; and the 2004 Discovery AO provided for $1 million.

OCR for page 109
Principal-Investigator-Led Missions in the Space Sciences Recommendation 1: NASA should adopt a risk-based estimating focus on the basic elements of cost, and require, at the end of the Concept Feasibility Study (phase A), an analysis of the estimated costs—before reserves—that provides high/medium/low-range estimates for key elements of the work breakdown structure, with the high-to-low-range estimates based on a confidence-level distribution. * * * * * Finding 2: The independent evaluation teams used for NASA’s proposal evaluation process provide the step 1 decision official with an appropriate level of understanding of the science value, management approach, and inherent technical risks. However, the AO-stipulated reserve percentages operate as a disincentive for the proposer to provide more realistic baseline cost estimates and an assessment of the appropriate level of reserves. NASA could gain useful information from proposers by requiring them to address a specific list of “cost risk subfactors.” The NASA evaluation team could use the responses to these subfactors as a consistent basis of comparison among proposals. NASA uses the initial phase of the selection process to eliminate proposals that do not combine an exciting science discovery potential with an appropriate technical, management, and cost approach. NASA carries out the evaluation on the basis of the written proposal, without requiring explanations of unclear elements. It is only after the selection of the best proposals for concept feasibility studies and generation of the study reports that the independent evaluators can meet with the proposal team and complement the written study report with fact-finding discussions. The Academy team’s examination of mission proposal evaluations indicated that the evaluators’ reports were generally effective in identifying and disclosing to the NASA decision official the preponderance of the technical issues associated with the project. However, the cost evaluations of the impact of those technical issues on the pre-reserve basic mission costs were not as effective in providing the decision official with an understanding of the potential costs. The cost evaluations were handicapped by the quality of the information provided in the proposal. Inconsistencies in the proposal’s cost estimates or lack of clarity meant the cost evaluator provided the decision official with step 1 inputs that could only note the inconsistencies/unclear elements. The Academy team considered recommending that NASA allow the evaluators to submit requests for clarification to the proposal teams during the step 1 evaluation process to clear up these ambiguities. However, the number of proposals submitted to NASA in response to the AO is high enough to be concerned about the additional time required for a review and clarification cycle. A better approach appears to be the use of an approach that requires each proposal team to respond to a specific list of “cost risk subfactors.” The JPL cost estimating group published a set of such cost risk subfactors in its December 2003 study of contributing factors to underestimates of baseline cost estimates and insufficient levels of reserves. Both primary and secondary risk subfactors were identified. The primary risk subfactors were: mission with multiple flight elements; operation in harsh environments; mission enabling spacecraft technology with a technology readiness level of less than 5; new design with multiple parameters not meeting the margin requirements specified in the JPL design principles; inadequate team and management experience; contractor inexperienced in mission application; level 1 requirements not well defined in formulation phase; excessive reliability requirements; new system architecture; and late selection of science instruments. The JPL cost evaluation assigned numeric values

OCR for page 109
Principal-Investigator-Led Missions in the Space Sciences against the primary risk subfactors and secondary risk subfactors, and generated an aggregate cost risk evaluation total for each proposal. The total score is indicative of the total cost risk and the commensurate need for reserves. One possibility is to use the JPL list of cost risk subfactors as a basis to generate a set of required information responses that will be included in future AOs. Each proposer would provide information on each cost risk element. While the Academy team believes that the JPL list may not be suitable as the standard for each AO, the NASA cost estimating community could work with the AO science proposal evaluation team to generate a list that could be tailored for a given AO. The responses by the proposers and the evaluations of them by the team would provide a consistent basis of comparison of the cost risks for each step 1 proposal. Recommendation 2: NASA should eliminate the specified reserve levels in the AO and instead specify a list of mission cost risks. Proposers would be required to address each risk in their step 1 proposals. The proposal evaluation teams should participate in the generation of the list included in the AO. The evaluators should consider the proposer’s self-assessment, and present to the NASA decision official their evaluation. The decision official should also receive a comparative analysis of the evaluated cost risks on a consistent basis across the proposals. * * * * * Finding 3: PIs are expected to provide leadership and management, but most lack the requisite skills, which must be based on a fundamental understanding of the elements of project control. Interviews with PIs and PMs revealed that new PIs have a very limited understanding of project control practices—cost estimating, schedule estimating, use of management information systems, and configuration control practices—and how these relate to the science and engineering for their missions. The Academy study team also found that it is the exceptional prospective new PI who has more than a rudimentary understanding of the elements of project control practices. PIs without experience found that NASA did not expect or require them to be knowledgeable partners in project control or engineering issues, but rather expected them to rely on their PMs. Some PIs who had experience on previous NASA space missions were more conversant with project control techniques and could interact effectively with their project management team and contractors. The lack of project control expertise of new PIs has not led to problems with missions in which there has been a stable, close, and trusting PI-PM relationship and a supportive project scientist working on the PM’s staff. It has been problematic in those instances where a PI has had multiple PMs whose different technical and management approaches have contributed to a lack of stability and/or altered the basic requirements established in the proposal. The Academy team’s review of these instances revealed that the PI either did not understand the ramifications of the PM’s decisions, or the PI believed he or she was in a weak position to override the PM’s argument that the decision would not materially impact adherence to the cost and schedule caps. NASA does not treat a PI’s lack of project control knowledge as a weakness in evaluating the merits of proposals. Nor does it offer—or require—elementary training in this skill to prospective PIs. The experience an unsuccessful prospective new PI gains from the initial proposal process, the verbal debriefing by NASA, and the second round of formulating a proposal gives only limited exposure to project control subjects.

OCR for page 109
Principal-Investigator-Led Missions in the Space Sciences Recommendation 3: NASA should provide training and resources in project control for prospective PIs when they indicate an interest in submitting a proposal in response to an upcoming AO. * * * * * Finding 4: The costs incurred by proposers in responding to AOs has been identified as a potential limiting factor by the proposal teams as to whether they will respond to future AOs. However, NASA does not possess good information on the costs incurred by proposers in responding to an AO. There is a risk that participating institutions, agencies, and contractors will limit the number of proposals for which they are willing to support the preparation and submission costs. This could have adverse effects on the scope and variety of science investigations proposed in the future. Two issues are associated with this finding. The first has to do with understanding the level of investment required to submit a proposal that has not only high science merit, but also sufficiently detailed and convincing technical, cost, and schedule information that it will be considered for selection to enter the Concept Feasibility Study stage. The Academy team found that it is unusual for a proposal to be selected the first time it is proposed in response to an AO. By the second—and even third—time a PI submits a mission proposal, the proposal team has amassed much greater detail to buttress its proposal. In addition, the team has benefited from the feedback it received in previous evaluation cycles. After it makes the selections, NASA provides the proposal teams with verbal evaluations of the strengths and weaknesses found in the proposal. The proposer specifically addresses those weaknesses in the next proposal. However, NASA does not ask the proposer in this post-selection period for even a rough order of magnitude estimate of the costs of preparing the proposal and of investments in independent research and development funds to advance the technologies. The Academy team believes the collection of this data could provide NASA officials with feedback on the level of investments made by proposal teams and areas where NASA investments in advancing technology could reduce the time required before an attractive mission concept is mature. Second, the interviews indicated that the number of unsuccessful proposals is sufficiently high that team members—aerospace firms, universities, and government laboratories—find it difficult to justify the cost of the proposal process to their management and consequently are cutting back on the number of proposals they will submit in response to future AOs. If NASA’s objective is to continue a process marked by full and open competition, decisions by these entities to team with PIs on fewer proposals may restrict the competitive process. The Academy team found that NASA has not been interested in the costs of the proposal effort, even though NASA indirectly pays much of the cost. The Academy team’s inquiries revealed that the cost to produce a Discovery-class proposal was about $1.5 million, with the project management center, industry partner, and PI’s institution each accounting for $500,000. This amount did not include precursor investments in technology development or non-recorded expenses. Multiplied by the number of proposals (16) made in the latest Discovery AO, the cost of responding to the AO can be roughly estimated to be at least $20-30 million. The Academy team believes that—without a change in NASA’s approach—the aerospace firms, government laboratories, and universities may respond to the low success rate, high costs, and long lead times by supporting only those proposals they consider to have a higher probability of selection. This would reduce the number of proposals in response to the AOs. Coupled with that could be an increase in the efforts by the institutions, agencies, and aerospace firms to obtain good “market intelligence” on NASA priorities by meeting with NASA officials and scientists. This leads to a concern that the proposers will invest in preparing proposals for only those projects with the scientific objectives, cost and schedule attributes that they assume are favored by NASA’s leadership.

OCR for page 109
Principal-Investigator-Led Missions in the Space Sciences The Academy team believes NASA should be wary of the potential for adverse impacts from the latter, as well as be concerned about the possibility that those parties who invest most in market intelligence efforts will gain a competitive advantage. Instead, the Academy team favors NASA working with the appropriate scientific community to adopt a proactive approach that recognizes that the high costs of proposing could result in negative consequences, and takes actions that focus the investment decisions into areas of greatest scientific interest to NASA for the next series of missions. This will require NASA to be vigilant as to the cost of preparing proposals individually and collectively, and to determine how to maintain full and open competition while continuing to receive exciting science proposals in areas of key interest. In concert with the NRC Committee, the Academy team considered several approaches that NASA could consider to forestall the negative effects of the high costs and long leadtimes for preparing proposals. One approach would be to reduce the investment costs for step 1 proposals by limiting the amount of information the proposers would have to generate in response to the AO. A second approach was for NASA to use the time prior to release of the AO to engage in internal and external discussions regarding scientific priorities, and then advise the program management centers, industry partners, and interested scientists at the earliest opportunity as to the areas of greatest scientific interest. If done sufficiently in advance, this approach could reduce the number of incorrect assumptions by the proposal parties as to what science investigations would be considered high priority. A third approach would be for NASA to play a more active role in the pre-proposal period by underwriting certain costs and providing technical support. Limited support would be provided to scientists with prospective missions, particularly those with innovative but technologically immature science instruments, by providing collaborative support (advanced technology development funding and access to NASA facilities and personnel). This support could be offered to those proposers who had received high evaluation marks during the previous AO but had not been selected for Phase A and/or Phase B. The support provided would be restricted to a specified period (e.g., 12 months) before release of the next AO. As such, it would complement but not supplant technical advice and support resources from industry and institutions. Recommendation 4: NASA should work with the appropriate scientific community to develop specific strategies to identify and proactively address the negative effects of the high preproposal and post-proposal investment costs for proposal team members. * * * * * The Mission Development Process Finding 5: The processes used by NASA to prepare for the confirmation review provide decision officials with a good appreciation of the key technical risks remaining for the development phase. However, the information provided on cost, schedule, and funding risks is more limited. In addition, the confirmation review process does not sufficiently address the constraints that could limit the availability of resources required for the mission development team to succeed. The assessment of the cost estimates focuses too much on the required level of reserves, with insufficient attention given to whether the cost estimate reflects the level of design maturity and the related constraints. The Academy team’s examination of the last decade of PI-led missions showed that NASA has increased both the time spent in formulation and the funds provided in the belief that this will result in a

OCR for page 109
Principal-Investigator-Led Missions in the Space Sciences lower probability of schedule growth and cost overruns. The independent review teams do an excellent job of identifying the technical risks. However, the Academy team believes more emphasis should be placed on the design maturity of the candidate mission and—as emphasized above—mitigation of the negative aspects of the competitive environment on the selection process. The characteristics of the acquisition process necessitate that NASA use independent review teams to evaluate candidate missions. The reviewers attempt to ferret out the uncertainties in both the design approach and the schedule and cost estimates. NASA did not approve several proposed missions for development because of the review teams’ findings. Analyses of the review teams’ reports to the decision official and of the problems experienced with confirmed missions show that the teams identified most of the development risks. Nonetheless, the teams recommended that the missions be approved for development. They did not appreciate the magnitude of the identified risks on the cost and schedule of the missions. The Academy team interviewed key project officials on projects that experienced significant overruns and determined that they understood that the design risks had not been addressed at the start of development. In addition, a number of projects that experienced schedule delays and cost overruns faced “environmental constraints,” that is, constraints on getting the needed engineering expertise or access to equipment and facilities, on having timely hardware and software deliveries, and on getting management to provide timely attention to pressing problems. In many instances, the problems resulted from several factors: a lack of priority assigned to the mission because other higher priority missions had access to the best talent; an over-scheduling of facilities; and more profitable ventures taking precedence when determining launch schedules. The interviews revealed that many of these constraints were predictable but were not taken into account during the evaluation process. Recommendation 5: NASA should focus on design maturity in its evaluation of readiness to proceed into phases C/D, and use that assessment, coupled with an understanding of the inherent environmental constraints, to determine the level of schedule and funding reserves required. This tailored approach would allow decisions to be made on the basis of the specific challenges presented by each mission. * * * * * Finding 6: NASA has adopted quality assurance process controls, risk management practices, and documentation, reporting, and review requirements that have cost consequences that need to be carefully considered in establishing probable cost estimates for future mission proposals. As noted, in the five years following the NIAT report, NASA has emphasized risk reduction practices and de-emphasized lower cost and innovative management approaches for PI-led space science missions. It has increased the cost caps for missions, not only for accounting adjustments and economic factors, but also for the changes in technical management practices. Although NASA made discrete adjustments to the cost caps of missions already in progress during this period, it based the adjustments on inputs from the project teams. The Academy study team found that these inputs were rough order of magnitude estimates based on a limited understanding of the impact of the new requirements. The Academy study team’s examination of the recently completed missions from this time period indicates that, with minor exceptions, the record of project costs does not identify the true cost impacts of the added work. (The exceptions are found in the information supplied by contractors that submitted claims for equitable adjustments to contract target cost baselines.) However, the Academy team was

OCR for page 109
Principal-Investigator-Led Missions in the Space Sciences supplied with an abundance of anecdotal information on the additional hours spent preparing for reviews and the displacement of planned effort caused by these changes. The Academy team concluded that the real costs can only be indirectly inferred by examining the schedule delays experienced by these projects. For future missions, the Academy team believes there is merit in NASA focusing special attention on how these new requirements are factored into the basic cost estimates, starting with the evaluation of the proposal estimates and following through to the confirmation review. Based on the Academy team’s interviews, NASA project officials and PIs question whether an appropriate balance has been found between the benefits of the additional processes and requirements and their cost. However, the absence of good cost data is a critical problem when discussing the need for changes to the added requirements. A specific study is needed to provide decision officials with solid estimates. The findings from such a study would be useful in assessing the benefit vs. cost relationship of these changes and in potentially modifying their implementation. Of particular value, the findings would enable a determination of the reasonableness of the allowance in project cost estimates for these requirements. Recommendation 6: NASA should undertake a detailed fact-finding evaluation of the time and cost incurred by compliance with the additional processes, documentation requirements, review teams, and related risk-reduction practices. The information obtained should be used to evaluate the adequacy of resource estimates for both proposed and approved PI-led missions. A second objective would be to provide NASA’s decision-makers with information that they can use to consider whether risk-reduction practices with marginal benefits and excessive costs should be eliminated. * * * * * Finding 7: In cases where proposed missions have experienced development difficulties sufficient to lead to Termination Reviews, the forecasts given to decision makers of the costs required to complete the missions were consistently understated. Also, the decision official is not provided with high/mid/low confidence engineering estimates of the costs-to-go, and parametric cost-estimating tools are not used to provide comparative cost estimates. The Academy team examined the cost estimates provided to decision makers in the termination review and found the estimates to be poor predictors of the eventual development costs. The project estimates in the annual Program Operating Plan process consistently understated the costs. Table 3 presents a series of estimates to complete a medium class Explorer mission’s development and the dates that the estimates were provided to NASA program management executives. The final development cost was $68 million. Although a number of unexpected and non-anticipatable events occurred during this mission’s development, the preponderance of the cost growth could be anticipated. Interviews revealed there was little incentive early on to abandon the optimistic estimating approach used from the outset. It was not until the available project funding showed signs of being exhausted that a better prediction of the completion costs was made. It is common practice to use parametric cost-estimating tools25 and related models to estimate the costs of projects that lack engineering detail, such as occurs at the outset of projects going through concept feasibility studies. Often—usually during phase B—the amount of design work available to use as a basis 25   Parametric cost estimating tools are computerized cost estimating models using specific technical parameters of space flight systems. The model’s data base of costs and technical parameters is collected from prior space flight missions. Using statistical correlations, equations are generated that relate the technical parameters to costs.

OCR for page 109
Principal-Investigator-Led Missions in the Space Sciences Table 3. Cost Estimates for a Medium Class Explorer Mission Date of Estimate Expenditures as of that date Estimate to Completion Total Estimate September 30, 1999 $15.1 million $30.1 million $45.2 million September 30, 2000 $35.0 million $19.4 million $54.4 million September 30, 2001 $50.1 million $13.7 million $63.8 million September 30, 2002 $62.1 million $ 2.0 million $64.1 million for estimates leads to a reliance on engineering cost estimates—also known as “grassroots” estimates. When sufficiently detailed, the engineering designs for hardware can be used to develop a “bill of materials” that is given to manufacturing cost estimators, who can estimate the labor hours required based on estimating handbooks or specific experience. Individual hardware elements can be estimated with a reasonable level of accuracy (plus/minus 10 percent) if the cost-estimating experience is sufficiently analogous to the current task. Software cost estimating is considerably less precise, a result of the constant change in the hardware available for computing and signal detection. The high level of error in detailed software cost estimates stands in stark contrast to that for hardware. Based on the Academy team’s work, it appears that NASA could benefit from continuing to use parametric cost-estimating tools throughout the development phase for mission elements with estimating analogs at a lower level of confidence. This is particularly the case when the spacecraft systems operate in poorly understood environments, when the scientific instrumentation represents a change in the state of the art, and when the computational systems are similarly advanced. Recommendation 7: NASA should focus on providing decision officials with a range of estimates and should augment its reliance on detailed engineering level estimates with estimates derived from cost modeling and parametric estimates, particularly for mission elements where the accuracy of the cost estimates is uncertain. * * * * * Finding 8: The recorded costs for PI-led science missions understate the true amount of the costs required to execute these missions. Cost estimators rely heavily on the recorded costs of instruments, spacecraft, and other mission elements to estimate future mission costs. Whether the cost estimating is done using parametric tools, detailed engineering estimates, or other approaches, the cost experience of past missions forms the base of knowledge that estimators use as a point of departure for future missions. Therefore, special recognition needs to be given to the finding that the recorded costs for PI-led missions are recognizably understated. Interviewees estimated the understatement for just the amount of unrecorded-uncompensated labor and recorded-uncompensated labor to be at least 20-30 percent. In addition, there are numerous instances of projects taking advantage of available spare hardware, getting special deals on NASA Center overhead charges, obtaining discounted prices from aerospace contractors, and avoiding procurement overhead charges by using other parties to procure instruments from vendors. In several cases, the agreed-to period for mission operations and data analysis was truncated, with the

OCR for page 109
Principal-Investigator-Led Missions in the Space Sciences understanding that an excellent science return would result in the restoration of the mission operations period. NASA would authorize the costs for an extended mission (phase F) in a separate decision, outside the limitations of the approved mission scope. These unrecorded and understated costs reflect the ingenuity of PIs, project managers, and other parties in coping with the constraints imposed by the cost caps, including a perceived threat of cancellation. The Academy team evaluated one mission in which it found major growth in the recorded monthly costs. When NASA decided not to cancel the project, stressing to the project team the need for mission success, the project team, which had been struggling to stave off termination, stopped under-recording costs by ending the “donation” of labor. NASA managers should be aware of the amount of total labor and not assume that contributions of labor will continue indefinitely. The other problem relates to the record of costs that cost estimators and managers rely on to determine the reasonableness of cost estimates for future missions. If the estimators have an appropriate understanding of the “true costs” of the past mission being used as an analog, they can make adjustments. However, if participants know the record but do not publish it, the potential for losing that knowledge when the participants retire or are reassigned is high. NASA’s history is replete with examples of technical mishaps or poor judgments that are the consequence of poor knowledge capture.26 Recommendation 8: NASA’s knowledge capture activities during and after mission development should specifically address discounts, unique situations, and the unrecorded-uncompensated and recorded-uncompensated direct labor hours, and provide appropriate footnotes to the recorded costs. NASA should review approaches that ensure appropriate knowledge transfer, such as collecting and publishing a record of the mission cost and schedule estimates, actual costs and schedules, and adjustments to those actuals, accompanied by information on a project’s management, technical and science development history, and environmental constraints. 26   This finding appears in a number of failure review board reports, including that of the recent Columbia Accident Investigation Board.