PART 1

OVERVIEW AND BACKGROUND



The National Academies | 500 Fifth St. N.W. | Washington, D.C. 20001
Copyright © National Academy of Sciences. All rights reserved.
Terms of Use and Privacy Statement



Below are the first 10 and last 10 pages of uncorrected machine-read text (when available) of this chapter, followed by the top 30 algorithmically extracted key phrases from the chapter as a whole.
Intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text on the opening pages of each chapter. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.

Do not use for reproduction, copying, pasting, or reading; exclusively for search engines.

OCR for page 17
An Assessment of Space Shuttle Flight Software Development Processes PART 1 OVERVIEW AND BACKGROUND

OCR for page 17
An Assessment of Space Shuttle Flight Software Development Processes This page in the original is blank.

OCR for page 17
An Assessment of Space Shuttle Flight Software Development Processes 1 OVERVIEW OF THE STUDY INTRODUCTION The Space Shuttle is one of the most complex engineering projects ever attempted by humans. It is a rocket that is expected to carry humans and large objects into space; an orbiting platform on which detailed scientific investigations are performed; and an aircraft that cannot fly without active control, but which is expected to land at a specified location without power from its engines. None of this would be possible without a very sophisticated system to control the wide variety of aerodynamic actuators and reaction-control system jets that are used to maintain the required atmospheric and on-orbit flight profiles. This highly complicated, interconnected digital control system could not work without the software that is loaded into the on-board computers during the various phases of a Shuttle mission. The Shuttle flight software, and the avionics and control system it operates, was conceived in the early 1970s before modern digital fly-by-wire control systems came into common use on spacecraft, military fighter aircraft, and commercial transport aircraft. The evolution of the design for the software was influenced by a number of factors, including what was, by today's standards, a primitive state of the art in computer and sensor technologies, 1 and the conservatism of the program managers who were reluctant to incorporate unproven technology because of the possible risk to the safety, cost, and schedule of the Shuttle program. This combination of conservatism and a low level of technology led to a premium being placed on efficient use of the on-orbit computation and storage resources. Furthermore, numerous stringent requirements were placed on the capabilities of the software due to the flight characteristics of the vehicle, the types of missions the vehicle was intended to perform, and the flight rate that was envisioned at the time. For example, the Shuttle is an unstable vehicle that cannot fly during ascent or descent without active control from a human pilot or an automatic system. This fact places the control system, and the development and maintenance of the software to run it, squarely on the critical path of safety and mission performance. 1   The state of the art in microcomputers has progressed through several generations since the Shuttle avionics computers were originally chosen in the early 1970s. This was long before the current 386/486 and 68000 series of computer chips were available. At the time, there did not exist a high-level language tailored for digital avionics applications, and structured programming techniques were just beginning to be applied outside the research environment.

OCR for page 17
An Assessment of Space Shuttle Flight Software Development Processes THE COMMITTEE'S TASK In early 1991, NASA's Office of Space Flight commissioned the Aeronautics and Space Engineering Board (ASEB) of the National Research Council (NRC) to investigate the adequacy of the current process by which NASA develops and verifies updates to the Space Shuttle flight software. In January 1992, the ASEB convened the Committee for Review of Oversight Mechanisms for Space Shuttle Flight Software Processes to evaluate the adequacy of the process from initial requirements definition to final machine loading. The Committee's task (see Appendix B) was to: Review the entire flight software development process from the initial requirements definition phase to final implementation, including object code build and final machine loading. Review and critique NASA's independent verification and validation process and mechanisms, including NASA's established software development and testing standards. Determine the acceptability and adequacy of the complete flight software development process, including the embedded validation and verification processes through comparison with (1) generally accepted industry practices and (2) generally accepted Department of Defense and/or other government practices (comparing NASA's program with organizations and projects having similar volumes of software development, software maturity, complexity, criticality, lines of code, and national standards). Consider whether or not independent validation and verification should continue. The first issue the Committee was asked to consider was the Shuttle program's decision to eliminate the independent verification and validation (IV&V) function currently performed on the Shuttle flight software at an annual cost of $3.2 million. The IV&V function had been instituted, in part, as a result of a recommendation of a previous NRC committee evaluating post-Challenger Space Shuttle risk assessment and management. When the Committee began its investigations, the Shuttle Program Office believed that the flight software and the processes that were used to develop and verify updates were sufficiently mature to permit a phase-out of the contractors that perform IV& V. Eliminating this function was primarily a cost-saving move, but one that the Shuttle Program Office believed was justified by the overall quality of the processes and personnel that are in place to maintain the software. In short, the Shuttle Program Office believed that the process was adequate without IV&V and that the money would be better spent in other ways. The IV&V function was scheduled to be eliminated by October 1992. Hence, the Office of Space Flight requested that the Committee first address whether there was a need to continue this function and later address other aspects of the flight software development process. Thus, the Committee initially focused on IV&V and issued an interim report (see Appendix C) that described the Committee's findings and recommendations on the IV&V issue only. This final report expands upon what was discussed in the Committee's Interim Report regarding IV&V and examines other aspects of the flight software development process, such as management and safety issues.

OCR for page 17
An Assessment of Space Shuttle Flight Software Development Processes CONTENTS OF THIS REPORT The sections within this chapter offer a brief description of several previous studies relevant to the Shuttle flight software and a description of the challenges that face those who must maintain and upgrade the current software. Some information is given in the following sections and elsewhere in the report on the pertinent characteristics of the software (e.g., its size and complexity). However, the Committee has not attempted to provide a complete description of the history and evolution of the software nor a complete description of its current state. The reader is referred, instead, to the excellent report by Hanaway and Moorehead (see Bibliography) that was prepared by NASA for those unfamiliar with the Shuttle avionics and software system. In addition, the Committee found it extremely difficult to reach a complete understanding of the process that is used by NASA and its numerous contractors to update and maintain the flight software. The process is partially described in a document, called the roadmap by NASA, that was recently prepared by Intermetrics for the Shuttle Program Office 2 (see Appendix D). However, this document is far from a complete description, and the Committee found it necessary to request many additional documents (see bibliography) and to submit numerous written and verbal questions to NASA and its contractors to obtain complete information. Because of the complexity of the process, the Committee has not provided a complete description. Instead, enough description is included to allow for an understanding of the findings and recommendations. For a complete description, the reader is referred to the documents included in Appendix D and Appendix E and those listed in the bibliography. The remaining chapters of this report outline the Committee's findings, conclusions, and corresponding recommendations regarding the adequacy of the current Space Shuttle flight software development process. Part 1 (Chapter 1, Chapter 2 and Chapter 3) contains the background necessary to understand the processes NASA and its contractors use, and Part 2 (Chapter 4, Chapter 5, Chapter 6 and Chapter 7) contains the details of the Committee's evaluation of those processes. Since the Committee began its investigations in January of 1992, the Shuttle Program Office has agreed, based on recommendations that were made in the Committee's Interim Report (see Appendix C), to maintain the IV&V function in its current form. The Committee applauds NASA for this decision. However, since the Interim Report did not include a complete evaluation of the software development and assurance process, much of what was discussed in the Interim Report is expanded upon in Chapter 2, Chapter 3, Chapter 4, Chapter 5 and Chapter 6, and additional recommendations are made as appropriate. Chapter 2 discusses how verification and validation (V&V) and IV&V are typically accomplished for similar large software systems in industry and other agencies of the government. It includes a definition of the terms used throughout the remainder of the report and a discussion of the advantages and disadvantages generally associated with the various implementations of contractor internal V&V, IV&V, and systems level-V &V. This discussion includes material that relates the NASA process to similar processes in industry and government. Chapter 3 is a very brief discussion of the current maintenance and upgrade process, which 2   The National Aeronautics and Space Administration, Space Shuttle Flight Software Verification and Validation Requirements, NSTS-08271 (Houston, Texas: Johnson Space Center, 1991).

OCR for page 17
An Assessment of Space Shuttle Flight Software Development Processes includes the IV&V function and the embedded process 3 that encompasses those functions that are not part of IV&V. The Committee's findings and recommendations begin in Chapter 4 with ways in which the Committee believes the embedded and IV&V process could be better implemented. Chapter 5 outlines the Committee's concerns regarding the safety program that is currently in place for the Shuttle flight software and other NASA programs, including the need to incorporate techniques to evaluate and track safety issues throughout the process and over the remaining life of the software. Chapter 6 examines organizational issues that relate to the development and assurance of appropriate changes to the Shuttle flight software, and Chapter 7 gives the Committee's thoughts on how the Shuttle flight software process relates to other programs within NASA and the implications for future programs. PREVIOUS STUDIES Following the Challenger accident in 1986, a number of assessments were made of the overall safety of the Shuttle program, many of which addressed verification and validation and the general software process as part of their investigations. These included evaluations by the Rogers Commission; an NRC committee; the House of Representatives ' Committee on Science, Space, and Technology; and the General Accounting Office (GAO). The Rogers Commission 4 concentrated on the direct causes of the Challenger accident, but Appendix E of their report included a statement by Richard Feynman, one of the members of the commission, that pertained specifically to the flight software: . there have been recent suggestions by [NASA] management to curtail elaborate and expensive tests as being unnecessary at this late date in Shuttle history. This must be resisted, for it does not appreciate the mutual subtle influences and sources of error generated by even small changes to one part of a program on another. 5 Among the recommendations of the Rogers Commission was that NASA review certain aspects of its Shuttle risk assessment effort and: “. identify those items that must be improved prior to flight to ensure mission success and flight safety.” The Rogers Commission further recommended that an audit panel be appointed by the NRC to verify the adequacy of the effort and report directly to the Administrator of NASA. This audit panel was convened by the ASEB of the NRC in 1986, and among its conclusions were: 3   The term embedded V&V was coined recently by the Shuttle Program Office in their argument to eliminate IV&V. In the Committee's judgement, it is equivalent to what is commonly referred to by industry as simply verification and validation. 4   Report of the Presidential Commission on the Space Shuttle Challenger Accident, William P. Rogers, Chairman (Washington, D.C.: Government Printing Office, 1986). 5   Feynman, R. P., “Personal Observations on Reliability of Shuttle,” Appendix F of the Report of the Presidential Commission on the Space Shuttle Challenger Accident, William P. Rogers, Chairman (Washington, D.C.: Government Printing Office, 1986).

OCR for page 17
An Assessment of Space Shuttle Flight Software Development Processes In general, hardware certification and verification, and software validation and verification in STS [Space Transportation System] are managed and conducted primarily by the same organizational elements responsible for the design and fabrication of the units. Thus, the independence of the certification, validation, and verification processes is questionable. For example, Independent validation and verification (IV&V) of software is carried out by the same contractor (IBM) that produces the STS software, with some checks being made by the Johnson Space Center. 6 The NRC committee recommended that: Responsibility for approval of hardware certification and software IV&V should be vested in entities separate from the NSTS [National Space Transportation System] Program structure and the centers directly involved in STS development and operation. In March 1988, the House Committee on Science, Space and Technology, echoing the concerns expressed in the NRC report, recommended that NASA establish IV&V to evaluate the development and modification of Shuttle software. Based on these two recommendations, in May 1988 NASA expanded an existing contract with Intermetrics Inc., and instituted the current IV&V function. The original IV&V contract with Intermetrics supported 40 people; recently, the support has been reduced to 24 people, at an approximate annual cost of $3.2 million. Table 1-1 shows the functions that were part of the original 40-person effort and those that are covered under the current IV&V program. In February 1990, the House Committee requested that the GAO determine NASA's progress in improving independent oversight of Shuttle software development. The GAO report, 7 dated February 1991, recommended that NASA: . require independent V&V [Verification and Validation] for Shuttle software, bearing in mind the views of the NRC, the House Committee, the [NASA Space Shuttle] software steering group, 8 and NASA-wide guidance, and ensure that the independent V&V organization is outside the control of the Shuttle Program Office. 6   Post Challenger Evaluation of Space Shuttle Risk Assessment and Management, National Research Council Committee on Shuttle Criticality Review and Hazard Analysis Audit (Washington, D.C.: National Academy Press, 1988). 7   United States General Accounting Office, Space Shuttle: NASA Should Implement Independent Oversight of Software Development (Washington, D.C.: United States General Accounting Office, 1991). 8   The software steering group consisted of officials from the Johnson Space Center, the Kennedy Space Center, the Marshall Space Flight Center, NASA headquarters, the software development contractors, and the Space Transportation System Operations Contractor. The group met once to address the need to bring about changes in NASA's software development and assurance processes but did not produce formal recommendations.

OCR for page 17
An Assessment of Space Shuttle Flight Software Development Processes TABLE 1-1 Functions Covered by IV&V IV&V Functions IV&V Functions at Start of IV&V Contract (40 full-time workers) Current IV&V Functions (24 full-time workers) Ascent guidance, navigation, and control X X Entry guidance, navigation, and control X X On-Orbit guidance, navigation, and control X X Sequencing X X Data processing system X X Main engine controller X X Systems management/payload X   Redundancy management X   Launch processing systems X   Documentation-only Change Requests X   Flight software tools X   Reconfiguration X   Downlist X   I-Load to K-load Change Requests X   “Living” Change Requests X   Source: Intermetrics, Inc. In requesting the current review of the Shuttle flight software development process, the Shuttle Program Office has stated that if funding were not an issue they would continue with a robust IV&V program. However, if it could be shown that the current implementation of IV&V does not appreciably reduce risk, or that its cost could not be justified by the risk it avoids, it could reasonably be eliminated. The Shuttle Program Office did not believe that these issues were adequately addressed by previous studies, which did not have the benefit of recent efforts to document the current V&V process. 9 To investigate the question of whether to continue IV&V, the Committee heard presentations from the Shuttle Program Office, the software development contractors, the current IV&V contractors, and several outside organizations and experts, including the U. S. Air Force and Navy. The Committee also reviewed extensive documentation and data provided by NASA and the contractors describing both the independent and embedded verification and validation processes. The Interim Report (see Appendix C) presented the findings of the Committee along with the following recommendation regarding the continuation of IV&V on the Shuttle software. 9   National Aeronautics and Space Administration, Space Shuttle Flight Software Verification and Validation Requirements, NSTS-08271 (Houston, Texas: Johnson Space Center, 1991). This document was prepared by Intermetrics for NASA to describe the process by which changes to the flight software are agreed upon and implemented. It also describes each organization's role in the verification and validation of those changes.

OCR for page 17
An Assessment of Space Shuttle Flight Software Development Processes . the Committee concluded that the current IV&V process is necessary to maintain NASA's stringent safety and quality requirements for man-rated vehicles. Therefore, the Committee does not support NASA's plan to eliminate funding for the IV&V effort in fiscal year 1993. The Committee believes that the Space Shuttle software development process is not adequate without IV&V and that elimination of IV&V as currently practiced will adversely affect the overall quality and safety of the software, both now and in the future. As mentioned previously, based on this recommendation and the recommendations found in the previous studies described above, NASA has decided to continue IV&V in its current form as a permanent part of the program. It should be noted, however, that the current form of IV&V does not conform to the recommendations set forth by the previous studies described above, in that it does not report to an organization outside the control of the Shuttle Program Office. Instead, the IV &V contractors report to the Shuttle Program Office directly, but at the same level as the software development contractors. THE FLIGHT SOFTWARE CHALLENGE Digital flight control systems of varying sophistication, and the software that ties them together, have existed on aerospace vehicles for decades, including digital flight control on the Apollo spacecraft. However, when it was originally conceived, the Shuttle flight software represented a significantly different set of functions than those that were implemented in earlier launch vehicles or aircraft. At that time, no suitable off-the-shelf microcomputers were available, structured software development techniques were just coming into common use, and no aircraft had been produced with digital fly-by-wire controls. NASA developed the High-order Assembly Language (HAL/S) specifically for the Shuttle flight software and chose a computer (the IBM AP-101) that had been used on several other flight programs, but which required extensive modifications for use on the Shuttle. Because of the unique nature of the programming language and computers used for the flight software, it takes a good deal of time for new employees to develop expertise in this application and environment. This means that it is imperative to retain the appropriate corporate knowledge to avoid losing the expertise once it is obtained. The Shuttle flight software controls most aspects of the ascent, descent, and on-orbit operations of the Shuttle based on assumptions about the physical state of the vehicle and the atmosphere through which it flies. It does so in real-time, often requiring appropriate reactions to the changing environment in fractions of a second. This involves sensing the environment during each phase of flight and coordinating the aerodynamic control surfaces and the reaction control systems jets in order to maintain proper attitudes and flight profiles. Because the computers do not have enough memory for all the software to be resident at any time, multiple software loads are used for various phases of the mission. Recent updates to the computers have alleviated the storage problems somewhat, but the continued growth of the software requires that every machine cycle and bit is used. This complicates the software coding and maintenance

OCR for page 17
An Assessment of Space Shuttle Flight Software Development Processes problems. Furthermore, it seems unlikely that additional upgrades in on-board memory or speed will occur in the remaining lifetime of the Shuttle. The software to accomplish this task consists of approximately 400,000 lines of code in over 1,500 compilable units, 10 while the backup software is approximately 90,000 lines of code. At the time it was developed, this was very large. It also was expensive--the software has evolved over many years of development and operation to require a complex maintenance and upgrade process involving numerous contractor and NASA organizations at a cost of well over $100 million per year. 11 In the ten years in which the software has been operational, it has undergone numerous upgrades (approximately one per year) to provide new functions, to account for errors that have been discovered, and to account for the unique characteristics of new hardware components on the Shuttles and new computers. Table 1-2 shows the number of lines of source code that were changed in each update (called operational increments or OIs) during the ten years of Shuttle operations. The two most recent upgrades (OI-20 and OI-21) included very significant changes to the code (a total of 60,000 lines of code were changed). As discussed in the Committee's Interim Report (see Appendix C), this process of continual upgrade and error resolution, combined with the magnitude of the necessary changes, was one the primary arguments for continuing the IV&V effort. The Shuttle flight software, and the processes used to develop and maintain it, are of very high quality, but they are not as good as the Committee believes they could, and should, be. This report describes several areas where, in the opinion of the Committee, changes are warranted to assure the continued safe and effective operation of the Shuttle. 10   In the Committee's Interim Report, it was stated that the primary software was made up of over 400 compilable units. After publication of the Interim Report the Committee was informed that there are 1522 compilation units, and 2646 distinct software modules in the Primary Avionics Software Systems (PASS) developed and maintained by IBM. 11   The Committee was told that the yearly cost for the flight software development contractors (new development, maintenance, software configuration control, etc.) was approximately $60 million. Operation of the Shuttle Avionics Integration Laboratory, which is used to test the flight software, requires approximately $24 million per year. This total does not include costs for software reconfiguration, development and maintenance of Space Shuttle Main Engine software, and other support contractors.

OCR for page 17
An Assessment of Space Shuttle Flight Software Development Processes TABLE 1-2 Operational Increment Change History Operational Increment Description Year of Incorporation Lines of code changed OI-2 Rendezvous software, Spacelab software 1983 10,600 OI-3 Redesign of main engine controller 1983 8,000 OI-4 Payload re-manifest capabilities 1984 11,400 OI-5 Crew enhancements 1984 5,900 OI-6 Experimental orbit autopilot, Enhanced ground checkout 1985 12,200 OI-7 Western test range, enhanced propellant dumps 1985 8,800 OI-7C Centaur 1985 6,600 OI-8A Post 51-L safety changes 1987 6,300 OI-8B Post 51-L safety changes, Bailout capability 1988 1,100 OI-8C System Improvements 1988 7,200 OI-8D Abort enhancements 1989 12,000 OI-8F Upgrade of general purpose computer (GPC) 1989 1,700 OI-20 Extended landing sites, Trans-Atlantic abort code 1990 28,000 OI-21 Redesign of abort sequencer, 1-engine auto-contingency aborts, hardware changes for new Orbiter 1991 32,000 Source: NASA Office of Space Flight

OCR for page 17
An Assessment of Space Shuttle Flight Software Development Processes This page in the original is blank.