INDEPENDENT VERIFICATION AND VALIDATION OF CRITICAL SOFTWARE
Numerous definitions and perceptions of verification and validation (V&V) and independent verification and validation (IV&V) exist in industry, and the Committee's communication problems were compounded by the coining of new terminology by the National Aeronautics and Space Administration (NASA). In order to provide a frame of reference for the findings and recommendations of the Committee, this chapter attempts to establish definitions for key terms used throughout this report and to provide a sense of the advantages and disadvantages offered by different approaches to software assurance.
The basic objectives in modern software verification and validation are to identify and help resolve software, hardware, and system problems early in a system's development life-cycle. Verification (derived from the Latin veritas, or truth) are those activities associated with proving that the software being built corresponds to what was specified. Validation (derived from the Latin valere or to be worth) are those activities associated with proving that the system meets the operational goals. Today, software practitioners do not try to separate their activities into verification and validation, but rather implement V&V as a single concept aimed at making sure the software will function as required. In general, IV&V has three primary objectives:
Demonstrating the technical correctness, including safety and security, of the system/software;
Assessing the overall quality of the system/software products; and
Ensuring compliance with the development-process standards.
The actual number of discrepancies discovered during the V&V process, although an important indicator, is not the sole measure of how successful the V&V effort has been. The greatest value of V&V lies in the interaction between the developer and V&V organizations. The independent technical activities conducted by the V&V organizations in parallel with the development team's efforts generates a path of constant feedback that ensures that quality and safety are built into the system from the beginning.
IV&V is practiced on most critical Department of Defense projects. In a presentation by representatives from the Air Force Aeronautical Systems Division, the Committee was told that the experience of the Air Force is that IV&V adds significant value to the quality of the developed system. IV&V as applied on Air Force programs has discovered errors and deficiencies that would have been overlooked if IV&V had not existed. Furthermore, the Air Force believes that the mere presence of a capable IV&V team provides a significant incentive to the developer to assure quality development and maintenance processes and products. At the same time, they have found that the timeliness of findings is inversely proportional to the separation of the IV &V team from the development team, that is, the farther removed the IV&V organization is from the day-to-day activities, the longer it takes to get needed information back into the development stream.
Strictly speaking, everyone involved in writing requirements, coding a module, or performing a test is engaged in V&V, including the software developers themselves. For the purposes of this report, however, the Committee has concentrated on the portion of the total V&V effort that is performed by organizations that are in some way independent of the developers. For simplicity, the Committee labels this IV&V. IV&V is defined broadly enough to include everyone involved in the broader V&V effort except the developers themselves.
A specific implementation of V&V or IV&V can be characterized along three dimensions: orientation, scope, and independence.
IV&V activities typically focus on either the software development process or the products produced by that process. Process-oriented IV&V typically involves participation in systems and software requirements reviews, design and code inspections, and test monitoring and audits. Technical review of the development process takes place, most often, within the system and software development environment and ensures that standards and procedures are followed. Product-oriented IV& V involves an independent analysis of the developer's products (system and software requirements, design, code, and test plans and procedures) and independent testing and test planning of the software as a separate item and as part of the entire system. Product-oriented IV&V may take place during development and after delivery.
Most implementations of IV&V perform a combination of both process-oriented and product-oriented assurance. Focusing solely on the development process without a detailed technical review of the product does not guarantee a quality product. For large, complex systems involving many different development organizations and software and system interfaces, it is impossible to know beforehand all of the issues that a review of the development process must address. Focusing only on the development process causes issues to slip through the cracks unless the software products are continually reviewed and integrated. At the same time, maintenance of high-quality products is difficult without a high-quality process. Furthermore, an exhaustive review of the product is too costly to implement and review of the process helps to provide additional confidence in the quality of the product.
Although the scope of IV&V can range along a continuum, it is convenient to identify three levels: comprehensive, focused, and limited. The most effective implementation of IV&V involves in-depth, technical analysis and an integrated view across all areas of software and hardware functions. This comprehensive approach includes a close interaction among all members of the software and system development and review teams that continually provides feedback and recommendations into the development process to improve both the process and the product.
Due to limited resources or other constraints, a comprehensive IV &V may not be feasible. A focused IV&V considers only a small set of software and/or system functions using a process-oriented or product-oriented approach. In-depth technical analysis is performed on those functions that are deemed to be the most critical for safety, reliability, or some other important aspect of the software.
When resources are extremely limited, a cursory monitoring of the process or limited testing that the software meets some minimal standards may be all that is implemented. Such a limited scope does not provide much assurance against errors resulting from the design of the process, nor does it provide assurance that the software will continue to perform correctly in off-nominal situations.
Ideally, the scope of IV&V should be determined by what is needed to ensure a quality product and not based strictly on the available resources. However, in the real world, the scope of coverage is often a function of the available funds, the consequences of missing scheduled launches or program milestones, and the consequences and likelihood of latent errors. If the cost of an undiscovered error is high (as measured by safety, mission effectiveness, or financial considerations), and especially when the magnitude of software changes is large and deadlines are critical, the scope of coverage should be increased, regardless of the cost, to provide adequate assurance that as many potential problems as possible are addressed prior to putting the software into use. Unfortunately, critical deadlines, in combination with budget constraints, can pressure management into reducing, rather than enhancing, IV &V.
Independence is the third, and most misunderstood, distinguishing characteristic of IV&V. Independence concerns the freedom of the IV&V team to operate without interference or restraint and can be evaluated along three dimensions: technical, managerial, and financial.
Technical independence requires that the IV&V team utilize personnel who are not involved in the development of the software and system. An effective IV&V team has members with knowledge of the system or with related experience and engineering background that gives them the ability to quickly learn about it. To maintain technical independence, the IV&V team must, in all instances, formulate their understanding of the problem and their proposed solution without influence from the development team. This technical independence (or fresh viewpoint) is critical to the team's ability to detect the subtle requirements, design, and coding errors that escape detection, for example, by development testing and quality-assurance reviews.
Technical independence also requires that the IV&V team use or develop its own set of test and analysis tools separate from the developer's tools. Sharing of tools, however, is common where it is impractical to build an independent version of the computer support environment (e.g., compilers, assemblers, utilities), system simulations, or test platforms. In this case, the IV&V team should conduct appropriate additional qualification tests on those tools shared with the development team to ensure that common tools do not mask errors in the software being analyzed and tested.
Managerial independence requires that the IV&V responsibility be vested in an organization outside the contractor and program organizations that develop the software systems. Managerial independence also requires that the IV&V team independently decides (1) which areas of the system to analyze and test, (2) the techniques to be used in the IV&V, (3) the schedule of activities to be performed (within the framework of the system schedules), and (4) the technical issues to be acted upon. For maximum effectiveness, a managerially-independent IV&V team should present its findings simultaneously to both the development team and the systems management.
Financial independence requires that control of the IV&V budget be vested in an organization outside the contractor and program teams that develop the system. Financial independence avoids situations where the IV&V team is precluded from completing its duties because funds have been diverted or situations where adverse financial pressures or influences are exerted on the IV&V team that serve to degrade its effectiveness.
Classical IV&V is characterized by technical, managerial, and financial independence. The IV&V team is outside the development contractor's organization and is typically a contractor hired by the customer or, sometimes, a team from within the customer's own organization. Most importantly, the IV&V team reports to a part of the customer 's organization that is not directly involved with the development of the software, although, typically, a close working relationship is formed between the IV&V and development teams to ensure that IV &V results and recommendations are integrated rapidly back into the development process. The U.S. Navy, for example, implements classical IV&V on its Trident submarine program by having one of its Naval laboratories develop software while a separate, independent Naval laboratory performs IV&V. This approach successfully separates management, financial, and technical efforts so as not to compromise the integrity of the IV&V activities. Classical IV&V is typically employed for highly critical, software-intensive systems where the consequences of a software error could cause loss of life, loss of mission, or significant social or financial loss.
Modified IV&V is a tailored form of classical IV&V in use in many large programs today where software is a central element that ties together large, complex systems. In modified IV &V there is an organization called the prime integrator that manages the entire hardware and software system development, including the software IV&V. The prime integrator can be the customer itself or a contractor hired by the customer to manage the development of the system. Usually, one or more contractors are chosen by the prime integrator to do the actual software development, and another contractor is chosen to perform the IV&V. In the modified form, complete technical and managerial independence does not exist because the IV&V team receives
its direction and funding from the same organization within the prime integrator as the development teams. However, since IV&V and development are performed by separate companies, the IV&V effort retains some measure of technical and managerial independence through the contracting mechanisms employed by the prime integrator.
Internal IV&V is performed by the same company that develops the software, which can be the prime integrator or one of its subcontractors. Within the company, however, the IV&V team reports to a different management level than does the development team. In this case, true managerial and financial independence is lost, at least from the customer's perspective, but there remains some degree of technical and managerial independence between the IV&V and development teams, albeit subject to the pressures of corporate profit and expediency. In particular, the technical independence of this form of IV&V is vulnerable to errors of omission because the development and IV&V teams are subject to the same organization, environment, and corporate culture. The internal IV&V team must also contend with direct and indirect peer pressures that may adversely influence the timely reporting of results. The benefit of an internal IV&V team is that there is greater availability of staff familiar with the system, thus minimizing the staff learning curve, gaining efficiency; and reducing cost. This form is often simply called V&V, but for the purposes of this report the Committee prefers the term internal IV&V because it expresses the fact that there can be some independence even when a single company performs both the IV&V and the development.
Embedded IV&V 1 can only barely be thought of as independent because the IV&V team is part of the development contractor and reports directly to the same level of management as the development team. Thus, it does not strictly include any of the three independence parameters. In this form, the IV&V team works alongside the development team, sharing the same checklists and procedures and attending the same walk-throughs, inspections, and reviews as the developers, thus ensuring that the development procedures and standards are followed. Any independence that is provided emanates solely from the diligence and integrity of the IV&V team and not from external management or financial clout or the ability to develop alternative technical solutions. The advantage of this form is that it further enhances the communication between the development and IV&V teams, thereby increasing the timeliness of the feedback obtained from the IV&V team. However, like internal IV&V, the embedded IV&V team is subject to peer pressure and runs the risk of unconsciously approving faulty group decisions when a truly independent solution is required.
NASA uses the term embedded to describe the entire software development process, including the internal activities of the development contractors and the activities of the various NASA organizations that are involved in reviewing and approving changes to the software, but excluding the IV&V activities of Intermetrics and Smith Advanced Technology. NASA's use of this term in its broader sense has proven very confusing to the Committee. Here the term applies strictly to the activities within a development contractor or prime integrator to run a check on its own process or products.
IV&V IN THE SHUTTLE PROGRAM
Details of the approach used by Intermetrics and Smith Advanced Technology to provide software IV&V, as well as the overall NASA approach to flight software V&V, are described in Appendix D and Appendix E, respectively. To summarize, NASA's current practice of software IV&V on the Shuttle program consists of a combination of a modified form of IV&V performed by Intermetrics and Smith Advanced Technology, along with an internal form used by the development contractors.
Each development contractor has a managerially-independent IV&V team that oversees the team that develops the software. For example, the IBM development team and internal IV&V team report to different organizations within the company. The development contractors perform a rigorous internal IV&V to assure that they are following their own established processes correctly and that the delivered product meets the given requirements.
The IV&V contractors, Intermetrics and Smith Advanced Technology, report to NASA at the same level as the development contractors. The IV &V effort by Intermetrics and Smith Advanced Technology is focused and product oriented. For example, Intermetrics concentrates on the ascent and descent phases of the software. Other parts are occasionally addressed, but only after the program identifies them as a pressing issue. In response to written questions from the Committee, the headquarters Safety and Mission Quality (S&MQ) Office described the IV&V process as follows:
IV&V is defined as a process whereby the products of the software development life cycle phases are independently reviewed, verified, and validated by an organization that is neither the developer nor the acquirer of the software. IV&V differs from V&V only in that it is performed by an independent organization. 2
The Safety, Reliability and Quality Assurance (SR&QA) Office at the Johnson Space Center (JSC) reports directly to the center director, not to the Shuttle program or the NASA headquarters S&MQ Office, and so it is managerially independent of the Shuttle program. However, the funds needed for the SR&QA Office to perform its IV&V related activities are obtained in part from the Shuttle Program Office (and the headquarters S&MQ Office) so it is not financially independent from the Shuttle Program Office.
A third level of independence, which is not used by NASA for the Shuttle program but which is sometimes used by the Air Force and Navy, would be provided by having the IV&V contractor report to a group completely outside the Shuttle program (e.g., the NASA headquarters S&MQ Office).
In addition, the Astronaut Office and various contractors and NASA organizations also participate in the evaluation of the process and the product it ultimately produces. Because of the complexity of the process, it is described separately in Chapter 3.
NASA headquarters Safety and Mission Quality Office (Code Q) letter of 13 January 1992: Clarification of NASA's Independent Verification and Validation (IV&V) Perspective.