APPENDIX C
Software Development in Evolutionary Acquisition

Software development is a natural area for evolutionary acquisition; in fact, it has been noted that the concept came from this area. Given the increasing and important role of software in complex defense systems, evolutionary acquisition should take advantage of and be consistent with current best practices in this area. This section provides a brief description of one approach, sometimes referred to as the cleanroom process. It is similar to several other approaches that are representative of best current practices.

CLEANROOM REFERENCE MODEL

The Cleanroom Reference Model developed by the Software Engineering Institute at Carnegie Mellon University defines a set of 14 integrated processes for software management, specification, development, and certification. Prowell et al. (1999) provides a detailed discussion of these individual processes and their role in the software development process. Here we focus on one of the processes: incremental development under statistical quality control, the cleanroom approach to establishing and maintaining management control of a software system that is developed in stages. In the incremental development process (Mills, 1971), at every stage of development one has a workable system to which additional features are added and verified sequentially. The process permits systematic incorporation of changes to requirements throughout the development cycle, which is crucial in an evolutionary context.



The National Academies | 500 Fifth St. N.W. | Washington, D.C. 20001
Copyright © National Academy of Sciences. All rights reserved.
Terms of Use and Privacy Statement



Below are the first 10 and last 10 pages of uncorrected machine-read text (when available) of this chapter, followed by the top 30 algorithmically extracted key phrases from the chapter as a whole.
Intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text on the opening pages of each chapter. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.

Do not use for reproduction, copying, pasting, or reading; exclusively for search engines.

OCR for page 52
Testing of Defense Systems in an Evolutionary Acquisition Environment APPENDIX C Software Development in Evolutionary Acquisition Software development is a natural area for evolutionary acquisition; in fact, it has been noted that the concept came from this area. Given the increasing and important role of software in complex defense systems, evolutionary acquisition should take advantage of and be consistent with current best practices in this area. This section provides a brief description of one approach, sometimes referred to as the cleanroom process. It is similar to several other approaches that are representative of best current practices. CLEANROOM REFERENCE MODEL The Cleanroom Reference Model developed by the Software Engineering Institute at Carnegie Mellon University defines a set of 14 integrated processes for software management, specification, development, and certification. Prowell et al. (1999) provides a detailed discussion of these individual processes and their role in the software development process. Here we focus on one of the processes: incremental development under statistical quality control, the cleanroom approach to establishing and maintaining management control of a software system that is developed in stages. In the incremental development process (Mills, 1971), at every stage of development one has a workable system to which additional features are added and verified sequentially. The process permits systematic incorporation of changes to requirements throughout the development cycle, which is crucial in an evolutionary context.

OCR for page 52
Testing of Defense Systems in an Evolutionary Acquisition Environment INCREMENTAL DEVELOPMENT Large software systems are organized collections of parts. The way a software system is put together from these parts has a critical impact on project success. Incremental development is a top-down approach to development in which a software system is developed and tested as a succession of cumulative subsets of function. A minimal system is developed in the first increment, and function is added in each successive increment until the system is complete. Each increment implements one or more end-user functions. Each increment contains all previously developed capability plus this new capability, so the system is “grown” in cumulative increments. Incremental development enables intellectual control over system development through referential transparency. (Referential transparency essentially implies the use of pluggable components with a fixed interface.) When referential transparency holds, a system part can be implemented from its subspecification with no need for backtracking; there is no rework of previous increments. This strategy enables correctness verification of each increment in a complete system context. This referential transparency clarifies one of the advantages of having system requirements for an evolutionary system known in advance of development. Cleanroom incremental development permits continual integration of referentially transparent user-function increments over the entire development life cycle. Because the design of each increment is based on a verified subspecification and tested interface in a prior increment, deep design and interface errors should be rare. The system evolves in well-defined increments throughout the development. Testing and certification activities begin early in the development cycle. Incremental development as practiced in cleanroom also provides a basis for statistical process control. Each cleanroom increment is a complete cycle of the process, involving specification, development, and verification of new user function plus testing of all work completed to date. This allows for measures of performance in each iteration of the process to be compared with performance targets to determine whether or not the process is “in control” (i.e., performing as expected). If standards are not met, the team can examine performance data from the increment to identify problems, adjust project plans if necessary, and modify the software development process to prevent recurrence of the problems identified. For example, if testing of an increment reveals that the process is “out of control”

OCR for page 52
Testing of Defense Systems in an Evolutionary Acquisition Environment (i.e., quality standards are not met), testing ceases and developers return to the design stage. If the process is in control, work on the next increment can continue. Feedback produced in each increment is also used to improve the process in the next increment. The incremental development process enables early and continual feedback by customers on the executing functionality of an evolving system, to permit changes if necessary. Because the increments execute in a system environment and represent subsets of user function, early increments can be exercised by users for feedback on system functionality and usability. Such feedback helps avoid developing the wrong system and builds user acceptance of the eventual product. (This technique is consistent with some of the early interest in having requirements developed through the interaction of system developers and system users.) Most crucially, in the context of evolutionary acquisition, incremental development allows systematic accommodation of inevitable changes in system requirements and the project environment. At the completion of each increment, the impact of accumulated changes in system requirements can be assessed in terms of current specifications and increment designs. If changes are isolated to future increments, they can often be incorporated into the existing incremental development plan, with possible adjustments to schedules and resources. If changes affect completed increments, a modified system development can be started from the top down, usually with substantial (often total) reuse of code from existing increments, with adjustments to schedules and resources as required. The overall objective is to develop a system with each new increment as an elaboration of the functions implemented in prior increments. That is, new functions in an increment should “plug in” to the previous increment at predefined points in its structure, and they should satisfy the subspecifications associated with the processing requirements at those points. Within the framework of functional dependencies exhibited by a system, incremental planning is also influenced by a wide range of management and technical factors in a project. For example, the user may wish to place certain system functions into operational use prior to system completion. Such functions are likely candidates for early increments. (For more details on incremental development, see Trammell et al., 2005.) An important motivation of the use of incremental development methods is the fact that requirements can rarely be established with certainty at the outset of a project. Under incremental development, customers provide feedback on an evolving system by direct operation of user-executable in-

OCR for page 52
Testing of Defense Systems in an Evolutionary Acquisition Environment crements. The relative clarity of requirements may influence an increment plan in two ways. Volatile requirements may be implemented in an early increment, so they can be clarified. Alternatively, unstable requirements may be planned for later implementation, when questions affecting the requirements have been settled. Increasingly, customers in the commercial sector are specifying formal software reliability requirements. There are methods that can be used to compute the reliability needed for each subsystem to achieve a system-wide reliability. Subsystems with the highest reliability requirements may be candidates for early increments. A functional usage distribution is developed as part of a top-level cleanroom specification. Expected usage probabilities of system functions are established from historical data and estimates provided by customers. System functions with high expected usage probabilities will receive greatest exposure in the field, and they should therefore benefit from the greatest exposure to testing. Since increments are cumulative, the functions developed in early increments will be tested every time a new increment enters the testing process. System functions expected to receive the greatest operational usage by customers are therefore candidates for early increments. Some functions expected to receive low usage may even be regarded as optional and scheduled for development in the final increment if time permits. Systems that involve both hardware and software must be developed as a coordinated effort between hardware and software engineers, and incremental development is an ideal framework for this coordination.