FINAL THOUGHTS AND FUTURE CONSIDERATIONS
There is currently a great deal of discussion within Congress, NASA, and the Federal Executive Branch regarding the future of the Space Shuttle program. There are those who believe the Shuttle is a well-tested and well-understood transportation system and, with appropriate upgrades of new technology and more capable and reliable subsystems, should continue to fly well into the next century. There are also those who believe it is time to begin developing the next generation of manned space transportation systems and that Shuttle operations should be curtailed or eliminated once a new system is operational. In either case, the Committee believes it is imperative that the “lessons learned ” to this point in the current Shuttle program be used as a guide, whether for future operation of the Shuttle or for preparing the development, assurance, and maintenance procedures for other space programs.
GATHERING THE LESSONS LEARNED
It is the Committee's belief that NASA and the current group of development, integration, and IV&V contractors are best positioned to gather all the detailed lessons learned from years of day-to-day operation of the Shuttle. This includes the on-going attempt to document the current processes as fully as possible, as recommended in Chapter 6. It should also include an effort by the current Shuttle program personnel, and as many people who have gone on to other programs as possible, to begin to draw some conclusions based on their experiences and to gather them into a form that future software managers and developers can use to guide their efforts.
As mentioned in Chapter 1, the Committee has found the report written by Hanaway and Moorehead 1 to be valuable because of its discussion of the history and evolution of the Shuttle avionics system. The report includes personal observations and even some candid statements regarding the decisions that were made early in the program. In the opinion of the Committee, this type of document is necessary if useful information is to be passed from one program to another. Unfortunately, this report does not include a detailed discussion of the software development and maintenance process or, more importantly, a discussion of the decisions that
Hanaway, John F., Robert W. Moorehead, Space Shuttle Avionics System, NASA SP-504, National Aeronautics and Space Administration (Washington D.C.: National Aeronautics and Space Administration, 1989).
were made early in the program regarding that process and their eventual impact on the performance, safety, and robustness of the software. The Committee believes that there are numerous areas within the software process, many involving more technical detail about the history of the process and the software, that should be captured in a similar fashion.
The Committee has mentioned several times in this report, and in its Interim Report, the need to capture the knowledge of the current Shuttle program personnel prior to their retirement or transfer to other programs. This same concern applies to programs such as Space Station Freedom (SSF) and the Earth Observing System (EOS). The SSF program, for example, is already far enough advanced that people at all levels of management and technical support are retiring despite the fact that there are still almost four years before the first element of the Space Station reaches orbit. The complete software system for SSF will not be in place until 1999. To avoid the condition that the Committee has found so difficult to deal with in its investigation of the Shuttle processes, the decisions that are being made now about the software that will fly on SSF, EOS, and other potential platforms must be captured before the programs get even farther downstream and more of the original decision-makers retire.
The Committee realizes that it is difficult and tedious to take time to document a process while it is being developed, especially while under pressure to design and build a safe and effective system. But that is precisely when it is most effectively done, because the people who are making the important decisions are still attached to the program on a daily basis. If, instead, these programs are allowed to continue in the same manner as the Shuttle program, there will almost certainly be another committee, similar to this one, convened sometime in the future and asked to investigate the adequacy of the SSF or EOS software development and upgrade processes without adequate documentation. Because the Shuttle flight software is, for a while at least, unique within NASA in its size and years of use, the Committee believes that NASA would do itself, and the nation, a great service if it were to capture these lessons learned and make them available to the SSF program and other planned, or potential, manned programs. A great service would also be performed if these new programs made a concerted effort from their very beginning to fully document all decisions, both formal and informal, that may impact the software or the processes used to develop it.
Recommendation #19: NASA should undertake an effort to capture the lessons learned in the development, maintenance, and assurance of the Shuttle flight software for use by other programs. This not only should take the form of official documentation of the current process but also should include less formal reports, observations, and opinions drawn from current personnel and as many former Shuttle program and contractor management and technical personnel as appropriate. The same type of documentation should be routinely prepared for other programs as well.
Given the above recommendation, the Committee believes that it would be remiss if it did not bring to NASA's attention a few of the most obvious conclusions drawn from its investigations. The following findings and recommendations should be taken in the generic sense, that is, the Committee has found them to be true of the Space Shuttle program, in varying
degrees, and believes the possibility exists that similar problems will occur in the SSF program, EOS, and elsewhere within NASA.
CONTRACT REPORTING REQUIREMENTS
There is the perception, which may or may not be correct, that the development contractors can withhold vital information from the oversight organizations because of proprietary concerns. There were a number of instances during the Committee's investigations when it was told that NASA and its IV&V contractor are unable to routinely obtain sufficient documentation of the development contractor's processes because of disputes over proprietary information. While this committee was not constituted to address this type of dispute and did not have the time to fully investigate the numerous relationships between the contractors and NASA, there is the perception among many who spoke to the Committee that the development contractors can choose to avoid full cooperation with the oversight activities if the contractors determine that it is in their best interest to do so. Unfortunately, it is impossible for a committee of this type to ascertain whether there is any truth to this perception; only the people who are involved in the day-to-day activities of the software development process can know for sure. However, the Committee can offer the recommendation in future contracts, both in the Shuttle program and in future procurements for SSF, EOS, etc., that NASA be more complete in spelling out the type and level of information that must be provided from the developers to the oversight organizations. By further formalizing the information that is transferred from one organization to another, NASA will gain greater confidence that proper information is available to all who need it. Furthermore, the potential for conflict due to corporate competition may be reduced once each company is made aware of precisely what information they are responsible for providing to the rest of the flight software community.
Recommendations #20: In future procurements, NASA should more precisely identify the information that each development and oversight contractor is responsible for making available to each other and to the community as a whole.
The Committee has recommended that NASA better document the current process and try to capture the accumulated wisdom of current and past Shuttle personnel. A related issue is the reluctance shown by the Shuttle program to fully implement the recommendations of the Rogers Commission, the earlier NRC committee, the GAO, and NASA' s own Aerospace Safety Advisory Panel, particularly in regards to the recommendations for fully independent V&V. In the Committee's opinion, NASA has not been as aggressive as it should have been in implementing the recommendations given to it by the various outside panels and committees in the area of software oversight. This is due in large part, the Committee believes, to the lack of
a concerted effort from within NASA to educate the program managers charged with controlling software projects on the benefits of these important oversight functions.
The conclusion drawn by this committee is that the Shuttle program has not fully understood the value of oversight functions such as IV&V and has a limited understanding of the variety of ways they can be implemented to bring about a satisfactory compromise between cost and benefit. The Committee feels the root cause for this limited understanding is that NASA as a whole, and the S&MQ Office in particular, does not make an adequate effort to provide the program with the information needed to make intelligent choices regarding the cost and benefit of software oversight. As mentioned in the previous chapter, if the S&MQ Office were given the resources and manpower necessary to fully educate the program managers, they could help alleviate the problem for the current Shuttle process. This same problem is likely to occur in future programs such as SSF and EOS and may well recur in the Shuttle program as more of the current decision-making personnel at NASA and its contractors retire or move on to other programs. A case in point is the recent report by the GAO 2 that discusses the Space Station programs's software development process. It concludes:
. NASA has not incorporated truly independent V&V into the program for its most critical software. What NASA labels as independent V&V is generally conducted by the same organization that builds the software and does not provide an added level of assurance over basic V&V activities. Program officials believe that little measurable value would be realized from using an independent V& V agent and that such a practice could be costly. For a critical and expensive software undertaking such as that for the Space Station, however, whether to employ independent V&V should not be based solely on the judgement of program officials without data and analysis of additional costs and risks.
The report also says:
Two management techniques key to controlling safety and cost risks associated with developing software are independent V&V and a systematic approach to software risk management. However, NASA has not incorporated these techniques into the [SSF] program. As a result, safety concerns about mission failure or loss of life due to a software failure are increased, as are concerns about higher long-term costs resulting from not implementing these mechanisms.
These are the same concerns expressed by this committee regarding the Shuttle software process, and much the same as were expressed by the Rogers Commission, the earlier NRC Committee, the Aerospace Safety Advisory Panel, and the GAO. While this present committee had access to several documents that were not available to previous investigations, if NASA had had an effective mechanism in place for educating program managers on the benefits of software
The U.S. General Accounting Office, Space Station: NASA's Software Development Approach Increases Safety and Cost Risks, (GAO/IMTEC-92-39) (U.S. Government Printing Office: Washington, D.C., 1992).
oversight, this committee's investigation may not have been necessary. NASA should understand that the recommendations it has been offered in the past are worthy of greater consideration than they appear to have been given.
Recommendation #21: Based on the lessons learned in the Shuttle program, NASA should put in place the mechanisms necessary to ensure that all existing and future programs are given the information needed to make intelligent implementations of software oversight functions such as IV&V.
ESTABLISHING STATE-OF-THE-ART CAPABILITIES WITHIN NASA
NASA has planned some of the most complex software projects ever attempted. The software to support the computers on board the Space Station, for example, is expected to consist of over a million source lines of code. The supporting ground software will likely consist of several times that. Like the Shuttle software, it will be a real-time system controlling numerous life-critical subsystems. Perhaps more importantly, the current plans are to develop the software in a very decentralized manner, with each of the NASA centers that participate in the Space Station program developing different pieces that will later be integrated into a coherent system. Each center has a prime contractor and numerous subcontractors, all of whom will be responsible for designing and building software. The NASA program management at the center will be responsible for managing and overseeing the development. There is no single prime contractor that is responsible for integrating all the software, nor is an IV&V effort planned. This project makes the scope of the Shuttle software seem almost trivial in comparison, and it will stretch the limits of software engineering capabilities. To bring the Space Station software effort, and others such as the EOS Data and Information System (EOSDIS), to a successful completion, NASA will need to design and implement aggressive software development and software-system-safety programs. The software safety programs must take advantage of state-of-the-art technology and leading edge methodologies to build safety into the software and the system while enhancing software development capabilities. This will require upgrading the education and knowledge of the NASA workforce to make it a leader in software engineering and software quality.
The Committee is concerned that the current software engineering and software-systemsafety capabilities within NASA may not be adequate to properly acquire and manage the development of such large, complex, and safety-critical systems. The Committee believes that the importance of software to the success of future NASA programs will only increase; NASA should undertake an effort to keep pace by increasing its in-house expertise both at the working level and among those expected to manage future programs and choose the contractors that will do the work.
Ultimately, the responsibility for the safety and functionality of the software that is put in place on future systems, including future Shuttle flight software upgrades, belongs to NASA. Contractors can be expected to do their best to provide a quality product because not doing so affects the future profit and reputation of their company. If, however, the contractors fail to
provide a quality product, or if the numerous parts of the total system fail to operate as expected, NASA will be the one left to explain to Congress and the nation why the system failed. NASA owes it to itself and to the nation to maintain as much in-house capability as possible to reduce its dependence on contractors and to provide proper assurance that contracted work is done on time and with as much attention to safety as these future systems require and deserve.
Recommendation #22: NASA should upgrade its workforce and management practices to make it a leader in software engineering and software quality. NASA should maintain as much in-house capability as possible to reduce its dependence on contractors and to provide proper assurance that contracted work is done on time and with as much attention to safety and other qualities as future systems require and deserve.