Operating in an Uncertain Policy Environment
Because the long-term future of the shuttle is uncertain, and policies related to the Space Shuttle Program are subject to change, shuttle program decision makers do not know how long the space shuttle must continue to operate, the number of flights per year that will ultimately be required, or the missions (if any) beyond supporting the ISS that the shuttle will be expected to perform. Uncertainties about the shuttle's operational lifetime have made it difficult for NASA to decide whether to implement upgrades to combat obsolescence and reduce operating costs. Uncertainties about the shuttle's future roles and flight rates have also made it difficult for NASA to decide whether upgrades to support non-ISS missions should be implemented. The shuttle program's limited budget for upgrades has constrained the program's responses to this environment, which has made it difficult for program managers to prepare adequately for the range of possible future scenarios.
NASA is not the only organization that must decide whether to upgrade an aging fleet and infrastructure in the face of component obsolescence, increasingly stringent environmental regulations, limited budgets, and an uncertain future. The Air Force, for example, has been faced with similar issues about the future of its B-52 long-range bombers, which have been operating for more than 40 years. Other examples can be found in the aerospace, transportation, telecommunications, energy, and defense industries. NASA could learn from the methods that have been used successfully by other large organizations to develop upgrade selection processes and strategies.
Recommendation 1. NASA should benchmark other large organizations' investment processes for technological upgrades and attempt to identify and emulate appropriate processes and investment strategies.
Uncertain Operational Lifetime
Decisions about implementing the more forward-looking and expensive proposed shuttle upgrades—particularly Phase III and IV upgrades—will probably have to be delayed until the time of a national decision on the space shuttle's future. A timely decision would enable NASA to act efficiently by either (1) only implementing the upgrades necessary to keep the program operating until it is phased out or (2) making major investments to reduce long-term program costs and improve long-term reliability. Although the policy decision was originally planned to be made in 2000, there is no guarantee that it will be made that year. The decision could be postponed for a number of reasons, including inconclusive results from other launch vehicle programs, or the unwillingness of the President or Congress to make an election year decision.
The committee supports NASA's approach of using its limited shuttle upgrade budget to fund minor upgrades that have identifiable short-term benefits and to conduct preparatory studies for major upgrades that may be warranted if the shuttle program is called upon to operate after 2012. This approach should help shuttle operations remain relatively safe and efficient for the next few years and enable the program to implement major upgrades if a decision is made to extend the shuttle's lifetime or to close out the upgrade program with minimal waste if the decision is made to phase out the shuttle.
If a national policy decision does not appear to be imminent as the year 2000 approaches, NASA may find it necessary to begin to implement some Phase III or Phase IV upgrades to the shuttle. If so, NASA must balance long-term risks, benefits, and costs, and primarily pursue candidate upgrades that would be valuable even if the shuttle program is later terminated (such as upgrades that provide safety benefits or could be used in other government or commercial programs).
Uncertain Flight Rate
Part of the shuttle program's goal to “improve mission supportability” is to increase the flight rate capability of the shuttle to 10 flights per year by 2002, 12 flights per year by 2004, and 15 flights per year by 2012 (Holloway, 1998). The ability to support increased shuttle flight rates is currently one of the metrics used to prioritize upgrades. However, NASA has not identified a need for more than the seven or eight missions per year planned to support the ISS and conduct research. Unless NASA's own needs for shuttle flights increase drastically, additional customers—most likely from the U.S. Department of Defense (DoD) or
the commercial sector—will have to materialize to support NASA projections. Two barriers would have to be overcome before this could occur.
First, current national space policy states that “the Space Shuttle will be used only for those important missions that require manned presence or other unique Shuttle capabilities, or for which use of the Shuttle is determined to be important for national security, foreign policy, or other compelling purposes” (White House, 1991). This policy has two purposes. It protects the shuttle, which is a unique national resource, from being put at risk in noncritical or nonunique applications, and it protects commercial launch firms from U.S. government-subsidized competition. This policy would have to be revised for the shuttle to be used by virtually any commercial customer for purposes that could be served by other launch vehicles.
Second, the shuttle would have to become a financially attractive launch vehicle for commercial customers. Prior to the Challenger accident, the shuttle was a viable commercial launch vehicle only because launches were heavily subsidized by the government and because competition for commercial payloads was limited. In the current political climate, however, that type of government-subsidized competition against commercial launch vehicles seems unlikely. If the shuttle is to become a viable competitor without government subsidies, a necessary step will be to greatly reduce its cost per pound to deliver payloads to orbit.
The committee believes NASA would be unwise to use an upgrade's ability to support a significantly increased flight rate as a factor (implicit or explicit) in choosing upgrades unless the agency can show through a viable business plan that has been reviewed and approved by financial and technical experts inside and outside the agency, as well as national policy makers that the shuttle could attract sufficient commercial and DoD business to justify the increase in flight rates. The business plan would also be useful for determining which upgrades would be most important for achieving higher flight rates. For example, if the shuttle program intended to launch commercial geostationary communications satellites, an upper stage rocket would be required. However, the shuttle does not currently have an operational upper stage. The inertial upper stage (IUS) is out of production and unavailable for new missions, and the infrastructure for another proven upper stage, the payload assist module (PAM), is virtually nonexistent after years of nonuse. One estimate is that it would take at least $10 million to resurrect the first PAM for shuttle use (Nichols, 1998).
Recommendation 2. The ability of a shuttle-unique upgrade to support an increased flight rate should not be a factor in the prioritization process, unless NASA can show through a viable business plan that has been reviewed and approved by financial and technical experts inside and outside the agency, as well as national policy makers (1) that the shuttle could attract enough business to justify the increased flight rate, and (2) that the shuttle program would not
unfairly compete with commercial launch vehicles or pose unnecessary risks to a national asset.
The $100 million budgeted for new upgrades is not secure because the money comes from shuttle program reserves. Cuts in the overall shuttle budget or problems in a shuttle system that require the use of reserve funds could reduce the amount available for the upgrades program. However, this approach gives the program manager flexibility to shift funds to match immediate priorities and problem areas. If the national decision is eventually made to substantially enlarge the upgrade program, it will be necessary to specifically fund Phase III and Phase IV projects in the NASA budget. Otherwise, the current approach and budget (assuming it is adjusted for inflation and there are no new major technical problems to solve) will probably be adequate for the remainder of the shuttle's operational life.
Refining Program Goals
According to NASA's 1998 strategic plan, the primary goals of the Space Shuttle Program are, in order of priority: (1) fly safely; (2) meet the flight manifest; (3) improve supportability; and (4) reduce costs (NASA, 1997). The shuttle upgrade program considers an upgrade's contributions to meeting these goals in its prioritization process. (Support for other activities in the HEDS enterprise are also considered.) The committee strongly supports NASA's use of program goals to prioritize upgrades. However, the committee also believes that more focused goals would provide better guidance to groups proposing new upgrades and would make the process for prioritizing and selecting upgrades more transparent.
Because goals provide important guidance for teams developing new upgrades, the goals of the upgrade program should accurately reflect the criteria used by program management to select new upgrades. For example, nearly half of the $100 million for new upgrades each year is being spent on obsolescence-related changes. If upgrades that combat obsolescence continue to be given a high priority, this should be reflected in the upgrade program's stated goals. The goals of the $100 million per year upgrade program do not necessarily have to be identical to the goals of the overall Space Shuttle Program. Because a substantial amount of the S&PU budget is already being spent on ongoing safety-related improvements, for example, new upgrades could focus on achieving other elements of the program goals. Table 3-1 provides an example of how goals used to prioritize new upgrades might differ from, but still complement, the goals of the overall program.
Another key to creating clear goals is ensuring that they are feasible. The 1998 NASA strategic plan challenges the shuttle program to pursue “a systems
TABLE 3-1 Sample Goals for the Upgrade Program
Shuttle Program Goals (in order of priority)
Goals for New Upgrades (in order of priority)
• Fly safely
• Fix known safety problems
• Meet the manifest
• Meet requirements of the ISS, the research community, and other known customers
• Improve supportability
• Minimize cost increases and subsystem life problems caused by obsolescence (ensure program viability through ISS era)
• Cut costs
• Reduce predicted flight and ground safety risks
• Help other programs
• Improve efficiencies (reduce the cost of delivering payload to orbit)
• Help other programs
upgrade program that will reduce payload-to-orbit costs by a factor of two by 2002” (NASA, 1997). The current upgrades program cannot meet that goal, not because the goal is technically impossible, but because the upgrades program does not have sufficient funding to meet the goal. A 50 percent cost-per-pound to orbit reduction in this time frame would require that NASA: (1) spend billions of dollars to implement most or all of the known cost-saving upgrades by the year 2002, and/or (2) achieve a flight rate of at least 15 missions per year in addition to implementing significant new cost-cutting (i.e., people reduction) initiatives. The $100 million per year (increased for inflation over time) of program reserves managed by the Space Shuttle Program Development Office is probably sufficient to maintain a reasonable level of obsolescence control through the ISS era, but it is grossly insufficient to meet the cost target stated in the strategic plan.
Recommendation 3. The Space Shuttle Program should reassess the goals used to prioritize candidate upgrades to ensure that they reflect the upgrade program's priorities, are feasible, and are clearly understandable to everyone working in the program.
Recommendation 4. The Human Exploration and Development of Space Enterprise should bring the cost goals for the space shuttle in its strategic plan into line with budget and policy realities.
Prioritizing And Selecting Upgrades
In the past, the Space Shuttle Program Office has used a variety of ad hoc approaches to prioritize the multitude of proposed modifications to the shuttle
and its support systems. In the end, approval for funding has depended on a combination of objective and subjective factors, including the following:
- program requirements
- technical merit
- resource requirements
- life cycle cost
- external political pressures
- agency institutional needs
- relative visibility and vigor of internal advocacy groups (government and contractor)
In the past two years, the program has begun to develop a more formal, less qualitative process to help the program manager make more informed decisions. This improved process will not, and is not meant to make the inevitable political and other subjective decision parameters go away. However, it will, according to the program manager, allow him more visible, apples-to-apples comparison capability for the traditionally objective programmatic and technical decision parameters.
The committee commends NASA for working to develop a more formal process to evaluate and prioritize upgrades. The groups involved in the development of new upgrades appear to appreciate that upgrades are being handled in a relatively proactive and organized manner (compared to past years when upgrades were usually reactive solutions to problems or mishaps). The Decision Support System (DSS) and the QRAS risk assessment system, both of which are still under development, have the potential to improve the selection process, although significant additional modifications (discussed below) will be required before their results can be fully trusted inputs to the process. Additional benefits could be gained by improving cost assessment procedures and modifying the shuttle operations contract to provide stronger incentives for USA corporation or any future prime contractor to develop and implement upgrades.
Quantitative Risk Assessment System.
The NASA administrator initiated the development of QRAS in 1996. QRAS is a risk assessment software tool developed primarily by the University of Maryland and managed by NASA's Office of Safety and Mission Assurance. The system, which builds upon earlier risk analyses of the SSME, the reusable solid rocket booster (RSRB), the APU, and other shuttle components, has primarily focused on risks during the shuttle's launch phase. NASA has spent $1.5 to $2 million on the system to date and is continuing to improve and expand QRAS.
The current version of QRAS has many significant deficiencies that make it
difficult to determine how much faith NASA should place in the validity and utility of its assessments. The primary weakness of QRAS as presently implemented is that it can only consider the impact of one risk at a time. In reality, however, catastrophes often occur when minor problems, which by themselves would not cause a failure, occur in combination with other problems. Other deficiencies of QRAS are listed below:
- It does not consider abort modes.
- It does not consider external dangers, such as meteoroids or orbital debris.
- It does not directly consider human error and crew response.
- It does not consider the effects of software-induced problems.
These are major omissions. Software-induced problems, for example, have caused many recent launch vehicle failures (e.g., Ariane 501 and the initial flight of the Pegasus XL), and human error is the most common cause of aircraft accidents.
To calculate the safety impacts of proposed upgrades, QRAS will also have to be able to incorporate increased risks from implementing each upgrade. New design modifications always involve a risk, however small, of causing new problems from unanticipated interactions with existing subsystems. To get a better picture of the true safety impact of an upgrade, QRAS risk assessments would have to include quantitative assessments of the potential of new hardware or software to increase the risk (including uncertainties) to the shuttle system.
Unless all of these sources of risk are included in the analysis, QRAS will give a skewed picture of the overall risks to the shuttle. Taken out of context, these skewed assessments could lead to inefficient spending to improve shuttle safety. For example, until the risk to the shuttle from meteoroids and orbital debris began to become clear in the mid-1990s, none of the billions of dollars spent on improving shuttle safety had been used to protect the shuttle from these significant external hazards.
The committee believes that this probabilistic modeling tool has the potential to be very helpful in assessing and comparing the impact of shuttle upgrades on shuttle safety. However, it is critical that NASA be aware of the system's limitations. As the scope and capabilities of the QRAS system are increased, the program will be able to rely more on its assessments of safety risks to the shuttle and the ability of various upgrades to reduce that risk.
Recommendation 5. NASA should continue to increase the scope and capabilities of the quantitative risk assessment system by improving its models of failures attributable to combinations of risks, human error, abort modes, on-orbit hazards, reentry and landing, and software. Until these improvements are made, the Space Shuttle Program Development Office should be very cautious in using the quantitative risk assessment system to aid in prioritizing upgrades.
Space Shuttle Upgrades Decision Support System
The DSS (Decision Support System), which is still under development by the Futron Corporation, receives information from every group that proposes an upgrade about the upgrade's cost, technological readiness, contribution to meeting program goals, risks, and ability to satisfy other NASA or federal government requirements. This information is provided to the program manager as qualitative independent assessments of the merits and costs of each candidate. The information is also translated into dollar figures and mathematically manipulated to create quantitative prioritized rankings of upgrades. These rankings (along with many other inputs, including the raw survey data used by the DSS) are used by the Space Shuttle Program Development Office in prioritizing upgrades for implementation.
The committee believes that when this type of support system is more mature it will be a valuable tool in the evaluation and prioritization of candidate upgrades. However, the committee distrusts the accuracy and applicability of the techniques employed by the DSS enough to caution that it not be used as the sole or most influential criterion of the program manager's decision making.
First, some of the means by which the DSS mathematically manipulates data could be improved. For example, the current system assigns dollar values to costs and benefits. It then models the benefit-minus-cost as a Gaussian random variable, whose mean and standard deviation are estimated from survey inputs and historical evidence, and constructs an “S-curve” of the cumulative probability of the upgrade's benefit-minus-cost value. The system then reads the 20th percentile value off the S-curve and uses this to discriminate between upgrades. However, because a Gaussian probability distribution is being used to model the benefit-minus-cost value, the S-curves are superfluous. The 20th percentile value is also unnecessary because, with very little additional work, explicit calculations could be used to compute the full probability that the benefit-minus-cost value of a particular upgrade exceeds the value of a different upgrade.
However, a bigger challenge with the DSS, as currently implemented, is that important information is often lost when survey inputs are transformed into single, numerical “expected upgrade values.” A survey to obtain inputs on upgrades is a valid way to draw on the technical expertise of the NASA and contractor engineering staffs. However, when these essentially nonmathematical inputs are mathematically manipulated, critical information can be lost and results can be perceived as having more credibility than they deserve. For example, in calculating the safety benefit of an upgrade, the DSS divides the cost of the shuttle by the number of items that an upgrade will remove from the shuttle's “critical item list.” The reality that some critical items are orders of magnitude more likely to cause failures than others is lost in the process. A more accurate way to compare the safety merits of two candidate upgrades would be to employ failure probabilities and their associated uncertainty bands.
To address the inherent difficulties associated with quantification of certain characteristics, the case for each upgrade could also be presented in a form that uses more of the available information and, at the same time, results in a much more transparent decision-making process. (A transparent process is critical to convincing upgrade proposers as well as program stakeholders that upgrades are being treated fairly.) One approach that could be used is “expert elicitation” (see Box 3-1). This technique—which is used by the U.S. Department of Energy and the Nuclear Regulatory Commission—can be an extremely effective (and transparent, with proper structuring and documentation) approach for prioritizing a relatively small number of alternatives. Although NASA often consults experts informally, formal expert elicitation would add a structured process that, if performed in accordance with strict rules, would provide what is missing in the existing DSS—a clear rationale for the results.
Recommendation 6. NASA should take care that the Decision Support System's quantitative tools are used as a supplement to, not as a substitute for, formal qualitative evaluations. Expert Elicitation should be considered as an additional formal qualitative tool. Also, NASA should consider modifying the quantification algorithm that the Decision Support System employs for cost-benefit comparisons so that it uses full probability values rather than 20th percentile S-curve values.
BOX 3-1 Eliciting Expert Opinions
Expert elicitation is a process to obtain knowledge from experts on a specific question, issue, or problem (Meyer, 1991). Formal expert elicitation is a process of documenting knowledge (judgments, opinions, parameter distributions, data, etc.) about the outcome of events, physical processes, etc., for which comprehensive observed or actuarial experience is lacking. The systems where expert elicitation has been most actively applied are nuclear power plants (USNRC, 1996) and geological nuclear waste repositories (Kotra et al., 1996).
A number of basic steps appear to be common to all well conceived applications of expert elicitation. These are: (1) properly framing the question or questions to be answered, including the desired form of the results, (2) providing consistent background source material, (3) recruiting and training the experts, and (4) aggregating and documenting the supporting evidence and results. These basic ingredients, together with appropriate leadership (facilitators, process experts, and subject experts), are key to a defensible result.
A key input to prioritizing upgrades is the estimated cost of developing, implementing, and operating each upgrade. For this input to be helpful, cost estimates must be accurate and calculated consistently. The first step in achieving this goal is to ensure that the costs of proposed upgrades are compared using the same definition of a dollar. Calculating the present value of anticipated expenditures is essential for comparing upgrades fairly and making cost-benefit calculations. However, the shuttle upgrades program does not consistently use fixed-year dollars in its assessments of candidate upgrades. The committee found inconsistencies in both the scope and accuracy of upgrade cost data. Although the varying degrees of maturity of the upgrades explains much of this, NASA must strive for consistent cost data in any cost/benefit analysis.
Second, cost estimates must include all costs (including hidden costs) associated with integrating a proposed upgrade into the shuttle system. These should include the following costs:
- integration costs, such as the expense of modifying structure, power, and other shuttle subsystems to comply with the needs of the upgraded components
- potential costs for mitigating the risk of replacing fully developed, tried, and tested hardware and software with newly developed hardware and software
- the cost of ground systems
- the cost of operating and maintaining the upgrade
- the cost of civil service labor
- the costs of transitioning flight and ground systems and personnel to the new upgrade (including the costs of maintenance and operations training, testing, and any costs of operating both old and new systems while the upgrade is being phased in)
- the cost of money
Third, cost estimates for upgrades must be accurate. Inaccurate cost estimates are a particular problem for projects involving a large amount of new software. Government cost estimates for software have been notoriously inaccurate, often underestimating costs by as much as an order of magnitude. This cost increase is typically attributable to an increase in lines of code by a factor of 2 to 3 (see Figure 3-1) and a decrease in the productivity of individual programmers by a factor of 3 to 4 (see Figure 4-1 and associated text).
Recommendation 7. All calculations, comparisons of costs and cost savings, and cost-benefit assessments done by NASA, as well as its DSS independent contractor, should be performed using fixed-year dollars and should include all costs (including hidden costs) associated with the upgrade.
United Space Alliance Selection Process
The SFOC (Space Flight Operations Contract) between NASA and USA corporation covers two fundamental types of work, as shown in Table 3-2. USA corporation is currently performing approximately $75 million in upgrade work (much of it related to obsolescence-driven design changes) under the “operations” part of the contract. USA corporation has also provided technical inputs to many of the upgrades under way under the “program provisioning” part of the contract.
The operations part of the contract gives USA corporation 35 percent of any underrun during the six-year, $7.4 billion contract. USA corporation has invested some of the 35 percent award fee they have earned to date in improvements to processes and training. However, the incentive has not been strong enough to convince USA corporation to invest in major shuttle upgrades because most
TABLE 3-2 Space Flight Operations Contract
Operations (85 percent of contract)
Program Provisioning (15 percent of contract)
• contractor managed
• NASA managed
• primarily covers repetitive operations
• primarily covers nonrepetitive engineering and development work
• performance based
• completion form (level of effort)
• cost plus award fee plus incentive fee
• cost plus award fee
—award fee (fixed percentage of total contract value)
—incentive fee (35 percent of underrun)
development takes years to complete and is not likely to show significant savings before the current contract ends in 2002. (Current procurement rules prohibit the government from compensating the contractor for savings achieved after the end of the contract.)
NASA can, and has, negotiated adjustments to minimize the effects of the weak incentives created by this contractual structure. NASA also has $15 million of discretionary money for award fees that can be given to the contractors for efforts or performance above and beyond the literal requirements of the SFOC. To date, USA corporation has not been awarded any of this discretionary money.
In addition to these “carrots,” NASA also has a “stick” to encourage the contractor to develop shuttle upgrades. The SFOC has provisions for penalties for contractor-caused mishaps and schedule slips, and the contractor must receive a relatively high score in “safety” in order to receive any underrun award fee. This “safety gate” could be considered an indirect incentive for USA corporation to propose safety upgrades.
None of the contractual arrangements covers contractor-financed developments per se. If USA corporation (with NASA's approval) decides to put company resources into a reliability enhancement upgrade that NASA has chosen not to fund, USA corporation could improve its safety grade or otherwise look better to the official determining the award fee. But USA corporation's only direct contractual compensation would be the 35 percent share of operations underruns that result from a cost-saving upgrade. In most cases, these initiatives make no business sense and, not surprisingly, no USA corporation-financed upgrades are in progress.
Although USA corporation appears to be an involved partner in defining and developing shuttle upgrades, the SFOC could be improved to provide stronger incentives for (1) prioritizing shuttle upgrade initiatives more consistently with the program's stated priorities (e.g., safety risk reduction before cost reduction) and (2) developing long-term improvements (both government-and
contractor-financed) to the shuttle system. NASA is currently defining a process that will provide the USA corporation with incentives to undertake upgrades that will result in savings beyond the term of the contract. If permissible under future procurement policies, one approach worth pursuing would be to provide “royalties” or other long-term (post-contract) compensation. Another approach which has been used on other programs is to pay out incentives up-front based on predicted future savings. All modifications to the contract are opportunities to add incentives for USA corporation (and future prime contractors) to initiate upgrades to meet other shuttle program goals.
Recommendation 8. NASA should provide stronger incentives for the shuttle prime contractor to propose, finance, and implement upgrades to meet the shuttle program's goals.
Improving Candidate Upgrades
To ensure that NASA can select the best upgrades for the shuttle program, a pool of high quality potential improvements must be developed. The shuttle program can take five steps to improve the pool of proposed upgrades:
- Broaden the range of proposed upgrades by actively soliciting and supporting proposals from outside of NASA.
- Improve the quality of proposed upgrades by conducting early assessments of their effects on the entire shuttle system.
- Minimize the risk that upgrades will experience problems with software during development or operations.
- Examine alternatives to proposed upgrades and conduct trade-off studies to determine the most cost-effective solutions.
- Modify groupings of upgrades to create sets of upgrades that will contribute most toward meeting particular goals.
Input from Outside NASA
To conserve funds and retain its engineering expertise, NASA is developing many of the shuttle upgrades in house, with minimal contractor participation. Although NASA's efforts appear to be technically excellent, the program runs two risks by taking this approach. First, the transition of an upgrade to industry for production could be difficult if the contractor base is not familiar with the upgrade or the technology involved. Bringing the contractor up to speed could require additional hiring or a slower development process, either of which would increase development costs. Second, NASA could miss out on superior upgrade concepts originated by industry or universities. By requesting upgrade concepts from the outside and, just as important, by taking steps to assure outsiders that
their upgrade candidates will be considered on an equivalent basis with proposals from within the agency, NASA could greatly improve its pool of potential upgrades.
Recommendation 9. Upgrade project managers should involve industry more in the definition and early development of candidate upgrades.
Early Systems Integration
NASA's FY99 budget request states that “the space shuttle upgrade activity will be planned and implemented from a system-wide perspective. Individual upgrades will be integrated and prioritized across all flight and ground systems, ensuring that the upgrade is compatible with the entire program and other improvements” (NASA, 1998). The committee strongly supports this concept but believes that a number of steps can be taken to strengthen the upgrade program.
A concerted effort early on to ensure that upgrades are compatible with other shuttle systems is essential for avoiding more expensive problems later in the development process. The effort might include the following steps:
- Early in the process of defining an upgrade, assess the structural, certification, weight and balance, aerothermal dynamic, and other effects of the upgrade on the entire shuttle system.
- Make detailed cost estimates as early as possible so the program manager can weigh the benefits against total program costs and cancel work on less promising upgrades.
- Analyze upgrades not only to determine potential safety risks to the shuttle design under standard operating conditions but also to determine how the upgrade might perform in a degraded state or under abnormal operating conditions.
In addition to determining the impact of potential upgrades on the rest of the shuttle system, the program could benefit from an early assessment of the effects of the upgrade on achieving program goals. If there is no way to show the connection analytically, the team working on the upgrade could document how the goals were being addressed and how the goals had affected the final design of the upgrade. This would provide an additional incentive for proposers to develop upgrades directed towards achieving program goals.
Obviously, the amount of analysis required early in the process should not overwhelm the actual development of the upgrade concept. The depth of analysis should depend on its relevance to the particular upgrade and the magnitude of the development effort. (For example, a proposed new shuttle wing and a proposed upgrade to the shuttle tires should both be evaluated to identify potential
integration issues, but the depth of analysis should be much greater for the wing upgrade.)
Recommendation 10. The Space Shuttle Program should institute a process early in the development of a candidate upgrade to ensure that the upgrade is compatible with other shuttle systems and relevant to meeting program goals.
Many shuttle modifications are accompanied by software changes. The committee has two major concerns about software changes associated with potential shuttle upgrades. The first is the potential that software changes can dramatically increase an upgrade's development costs and delay its implementation. The second is the potential risk to shuttle operations from the use of commercial off-the-shelf (COTS) software.
As already noted, government estimates of software costs and schedules have been notoriously inaccurate, and problems in producing software can result in large and unexpected cost overruns and delays. Historically, the cost and implementation schedules of shuttle upgrades have often been driven by the software verification process. For example, the modest (60,000 lines of code) software change accompanying the multifunction electronic display systems upgrade to the shuttle cockpit took five years from go-ahead to final qualification. If the changes to software associated with an upgrade could be minimized, NASA could, in many cases, lower the cost, development time, and risk of the upgrade. NASA appears to be taking a wise course with the proposed avionics upgrade, which focuses initially on replacing hardware components, rather than on changing the shuttle software.
The use of COTS software in the shuttle environment is another area of concern. COTS software and the COTS software industry itself often do not meet the requirements for safety, data integrity, robustness, testability, validatability, performance, longevity, and supportability for prolonged use in the shuttle program. NASA currently makes decisions about whether to use particular COTS software on a case-by-case basis, and it also develops rules and guidelines on a case-by-case basis. A strategy for the procurement, verification and validation, maintenance, and other requirements for that particular application is then spelled out in the upgrade's software development plan. No general guidelines for COTS software selection or use are available, partly because of the wide range of applications in which the software is deployed.
Comprehensive guidelines for using COTS software could put an end to the proliferation of potentially unsafe or inefficient ad hoc policies and procedures. The committee recognizes that COTS software will be used in a wide variety of applications and that the guidelines would have to be broad enough to cover all
of them. However, the committee believes that the improvements in efficiency and safety would be worth the effort.
Recommendation 11. NASA should limit the software changes associated with new shuttle upgrades. The agency should consider standardizing its guidelines for using commercial off-the-shelf software in shuttle upgrades.
Alternatives and Trade-off Studies
The committee is concerned that NASA often does not conduct concrete, indepth trade-off studies to determine whether a proposed upgrade is the best approach to solving a particular problem or achieving a particular goal. Upgrade concepts typically originate in the various project elements of the shuttle organization. This allows the people who know the shuttle best to suggest new upgrades but often produces high-technology, expensive upgrade proposals, instead of less radical, more incremental upgrades that could achieve much the same benefit at a much lower cost. The upgrade program manager's role should be to determine whether proposed solutions are cost effective and, if they are not, to implement a more effective alternative.
Recommendation 12. Before embarking on the larger, more costly upgrades, NASA should examine alternative solutions and conduct trade-off studies to determine if the proposed upgrade is the best way to achieve the desired result.
With the exception of avionics, the upgrades were presented to the committee as stand-alone modifications. The most effective way to meet a particular program requirement will often not be through any of the individual upgrades proposed to the Space Shuttle Program Development Office but through a combination of candidate upgrades (or elements of candidate upgrades). For example, the most cost-effective approach to increasing payload capacity at today's flight rate might involve the development of a five-segment solid rocket booster and the extended nose landing gear. A package of upgrades that would enable the shuttle to fly 15 times per year might include a liquid fly-back booster—possibly one less capable than the one currently proposed—electric APUs, and a new high-energy upper stage for the payload bay. The search for efficient groupings could reveal synergies among candidate upgrades, and the results could be useful in optimizing modification schedules and resources and explaining to stakeholders outside of the program the upgrades required to meet specific program goals.
Recommendation 13. The Space Shuttle Program Development Office should not consider proposed upgrades as stand-alone modifications but should look for opportunities to combine upgrades (or features of upgrades) to efficiently meet future requirements.
Aerospace Corporation. 1998. Aerospace Roles in Space Systems Architecting, Acquisition, and Engineering: Cost Management and the Aerospace Role. Los Angeles: The Aerospace Corporation.
Holloway, T. 1998. Space shuttle program goals and objectives. Briefing to the Committee on Space Shuttle Upgrades, Houston, Texas, July 7, 1998.
Kotra, J.P., M.P. Lee, N.A. Eisenberg, and A.R. DeWispelare. 1996. Branch Technical Position on the Use of Expert Elicitation in the High-Level Radioactive Waste Program. NUREG-1563. Prepared for U. S. Nuclear Regulatory Commission.
Meyer, M.A., and J.M. Booker. 1991. Eliciting and Analyzing Expert Judgment—A Practical Guide. San Diego, Calif.: Academic Press, Inc.
NASA (National Aeronautics and Space Administration). 1997. NASA 1988 Strategic Plan. NASA Policy Directive (NPD)-1000.1. Washington, D.C.: National Aeronautics and Space Administration.
NASA. 1998. NASA FY1999 Budget Request. Washington, D.C. National Aeronautics and Space Administration.
Nichols, S. 1998. Telephone communication between Stanley Nichols, NASA headquarters, and Bryan O'Connor, chair of the Committee on Space Shuttle Upgrades. September 30, 1998.
USNRC (U.S. Nuclear Regulatory Commission). 1996. Branch Technical Position on the Use of Expert Elicitation in the High-Level Radioactive Waste Program. NUREG-1563. November, 1996. Washington, D.C.: Nuclear Regulatory Commission.
White House. 1991. National Space Launch Strategy. NSPD-4. July 10, 1991.