National Academies Press: OpenBook

Testing of Defense Systems in an Evolutionary Acquisition Environment (2006)

Chapter: Appendix C: Software Development in Evolutionary Acquisition

« Previous: Appendix B: Combining Information for Test Design with Staged Development
Suggested Citation:"Appendix C: Software Development in Evolutionary Acquisition." National Research Council. 2006. Testing of Defense Systems in an Evolutionary Acquisition Environment. Washington, DC: The National Academies Press. doi: 10.17226/11575.
×

APPENDIX C
Software Development in Evolutionary Acquisition

Software development is a natural area for evolutionary acquisition; in fact, it has been noted that the concept came from this area. Given the increasing and important role of software in complex defense systems, evolutionary acquisition should take advantage of and be consistent with current best practices in this area. This section provides a brief description of one approach, sometimes referred to as the cleanroom process. It is similar to several other approaches that are representative of best current practices.

CLEANROOM REFERENCE MODEL

The Cleanroom Reference Model developed by the Software Engineering Institute at Carnegie Mellon University defines a set of 14 integrated processes for software management, specification, development, and certification. Prowell et al. (1999) provides a detailed discussion of these individual processes and their role in the software development process. Here we focus on one of the processes: incremental development under statistical quality control, the cleanroom approach to establishing and maintaining management control of a software system that is developed in stages. In the incremental development process (Mills, 1971), at every stage of development one has a workable system to which additional features are added and verified sequentially. The process permits systematic incorporation of changes to requirements throughout the development cycle, which is crucial in an evolutionary context.

Suggested Citation:"Appendix C: Software Development in Evolutionary Acquisition." National Research Council. 2006. Testing of Defense Systems in an Evolutionary Acquisition Environment. Washington, DC: The National Academies Press. doi: 10.17226/11575.
×

INCREMENTAL DEVELOPMENT

Large software systems are organized collections of parts. The way a software system is put together from these parts has a critical impact on project success. Incremental development is a top-down approach to development in which a software system is developed and tested as a succession of cumulative subsets of function. A minimal system is developed in the first increment, and function is added in each successive increment until the system is complete.

Each increment implements one or more end-user functions. Each increment contains all previously developed capability plus this new capability, so the system is “grown” in cumulative increments. Incremental development enables intellectual control over system development through referential transparency. (Referential transparency essentially implies the use of pluggable components with a fixed interface.) When referential transparency holds, a system part can be implemented from its subspecification with no need for backtracking; there is no rework of previous increments. This strategy enables correctness verification of each increment in a complete system context. This referential transparency clarifies one of the advantages of having system requirements for an evolutionary system known in advance of development.

Cleanroom incremental development permits continual integration of referentially transparent user-function increments over the entire development life cycle. Because the design of each increment is based on a verified subspecification and tested interface in a prior increment, deep design and interface errors should be rare. The system evolves in well-defined increments throughout the development. Testing and certification activities begin early in the development cycle.

Incremental development as practiced in cleanroom also provides a basis for statistical process control. Each cleanroom increment is a complete cycle of the process, involving specification, development, and verification of new user function plus testing of all work completed to date. This allows for measures of performance in each iteration of the process to be compared with performance targets to determine whether or not the process is “in control” (i.e., performing as expected). If standards are not met, the team can examine performance data from the increment to identify problems, adjust project plans if necessary, and modify the software development process to prevent recurrence of the problems identified. For example, if testing of an increment reveals that the process is “out of control”

Suggested Citation:"Appendix C: Software Development in Evolutionary Acquisition." National Research Council. 2006. Testing of Defense Systems in an Evolutionary Acquisition Environment. Washington, DC: The National Academies Press. doi: 10.17226/11575.
×

(i.e., quality standards are not met), testing ceases and developers return to the design stage. If the process is in control, work on the next increment can continue. Feedback produced in each increment is also used to improve the process in the next increment.

The incremental development process enables early and continual feedback by customers on the executing functionality of an evolving system, to permit changes if necessary. Because the increments execute in a system environment and represent subsets of user function, early increments can be exercised by users for feedback on system functionality and usability. Such feedback helps avoid developing the wrong system and builds user acceptance of the eventual product. (This technique is consistent with some of the early interest in having requirements developed through the interaction of system developers and system users.)

Most crucially, in the context of evolutionary acquisition, incremental development allows systematic accommodation of inevitable changes in system requirements and the project environment. At the completion of each increment, the impact of accumulated changes in system requirements can be assessed in terms of current specifications and increment designs. If changes are isolated to future increments, they can often be incorporated into the existing incremental development plan, with possible adjustments to schedules and resources. If changes affect completed increments, a modified system development can be started from the top down, usually with substantial (often total) reuse of code from existing increments, with adjustments to schedules and resources as required.

The overall objective is to develop a system with each new increment as an elaboration of the functions implemented in prior increments. That is, new functions in an increment should “plug in” to the previous increment at predefined points in its structure, and they should satisfy the subspecifications associated with the processing requirements at those points. Within the framework of functional dependencies exhibited by a system, incremental planning is also influenced by a wide range of management and technical factors in a project. For example, the user may wish to place certain system functions into operational use prior to system completion. Such functions are likely candidates for early increments. (For more details on incremental development, see Trammell et al., 2005.)

An important motivation of the use of incremental development methods is the fact that requirements can rarely be established with certainty at the outset of a project. Under incremental development, customers provide feedback on an evolving system by direct operation of user-executable in-

Suggested Citation:"Appendix C: Software Development in Evolutionary Acquisition." National Research Council. 2006. Testing of Defense Systems in an Evolutionary Acquisition Environment. Washington, DC: The National Academies Press. doi: 10.17226/11575.
×

crements. The relative clarity of requirements may influence an increment plan in two ways. Volatile requirements may be implemented in an early increment, so they can be clarified. Alternatively, unstable requirements may be planned for later implementation, when questions affecting the requirements have been settled.

Increasingly, customers in the commercial sector are specifying formal software reliability requirements. There are methods that can be used to compute the reliability needed for each subsystem to achieve a system-wide reliability. Subsystems with the highest reliability requirements may be candidates for early increments. A functional usage distribution is developed as part of a top-level cleanroom specification. Expected usage probabilities of system functions are established from historical data and estimates provided by customers. System functions with high expected usage probabilities will receive greatest exposure in the field, and they should therefore benefit from the greatest exposure to testing. Since increments are cumulative, the functions developed in early increments will be tested every time a new increment enters the testing process. System functions expected to receive the greatest operational usage by customers are therefore candidates for early increments. Some functions expected to receive low usage may even be regarded as optional and scheduled for development in the final increment if time permits.

Systems that involve both hardware and software must be developed as a coordinated effort between hardware and software engineers, and incremental development is an ideal framework for this coordination.

Suggested Citation:"Appendix C: Software Development in Evolutionary Acquisition." National Research Council. 2006. Testing of Defense Systems in an Evolutionary Acquisition Environment. Washington, DC: The National Academies Press. doi: 10.17226/11575.
×
Page 52
Suggested Citation:"Appendix C: Software Development in Evolutionary Acquisition." National Research Council. 2006. Testing of Defense Systems in an Evolutionary Acquisition Environment. Washington, DC: The National Academies Press. doi: 10.17226/11575.
×
Page 53
Suggested Citation:"Appendix C: Software Development in Evolutionary Acquisition." National Research Council. 2006. Testing of Defense Systems in an Evolutionary Acquisition Environment. Washington, DC: The National Academies Press. doi: 10.17226/11575.
×
Page 54
Suggested Citation:"Appendix C: Software Development in Evolutionary Acquisition." National Research Council. 2006. Testing of Defense Systems in an Evolutionary Acquisition Environment. Washington, DC: The National Academies Press. doi: 10.17226/11575.
×
Page 55
Next: Appendix D: Workshop Agenda and Attendees »
Testing of Defense Systems in an Evolutionary Acquisition Environment Get This Book
×
Buy Paperback | $29.00 Buy Ebook | $23.99
MyNAP members save 10% online.
Login or Register to save!
Download Free PDF

The Department of Defense (DoD) recently adopted evolutionary acquisition, a dynamic strategy for the development and acquisition of its defense systems. Evolutionary defense systems are planned, in advance, to be developed through several stages in a single procurement program. Each stage is planned to produce a viable system which could be fielded. The system requirements for each stage of development may be specified in advance of a given stage or may be decided at the outset of that stage's development. Due to the different stages that comprise an evolutionary system, there exists a need for careful reexamination of current testing and evaluation policies and processes, which were designed for single-stage developments.

The Office of the Under Secretary of Defense for Acquisition, Technology and Logistics (USD-AT&L) and the Director of Operational Testing and Evaluation (DOT&E) asked the Committee on National Statistics (CNSTAT) of the National Academies to examine the key issues and implications for defense testing from the introduction of evolutionary acquisition. The CNSTAT was charged with planning and conducting a workshop to study test strategies for the evolutionary acquisition. The committee reviewed defense materials defining evolutionary acquisition and interviewed test officials from the three major test service agencies to understand the current approaches used in testing systems procured through evolutionary acquisition. The committee also examined possible alternatives to identify problems in implementation.

At the workshop that took place on December 13-14, 2004, the committee tried to answer many questions including: What are the appropriate roles and objectives for testing in an evolutionary environment?, Can a systematic, disciplined process be developed for testing and evaluation in such a fluid and flexible environment?, and Is there adequate technical expertise within the acquisition community to fully exploit data gathered from previous stages to effectively combine information from various sources for test design and analysis?. Testing of Defense Systems in an Evolutionary Acquisition Environment provides the conclusions and recommendations of the CNSTAT following the workshop and its other investigations.

  1. ×

    Welcome to OpenBook!

    You're looking at OpenBook, NAP.edu's online reading room since 1999. Based on feedback from you, our users, we've made some improvements that make it easier than ever to read thousands of publications on our website.

    Do you want to take a quick tour of the OpenBook's features?

    No Thanks Take a Tour »
  2. ×

    Show this book's table of contents, where you can jump to any chapter by name.

    « Back Next »
  3. ×

    ...or use these buttons to go back to the previous chapter or skip to the next one.

    « Back Next »
  4. ×

    Jump up to the previous page or down to the next one. Also, you can type in a page number and press Enter to go directly to that page in the book.

    « Back Next »
  5. ×

    Switch between the Original Pages, where you can read the report as it appeared in print, and Text Pages for the web version, where you can highlight and search the text.

    « Back Next »
  6. ×

    To search the entire text of this book, type in your search term here and press Enter.

    « Back Next »
  7. ×

    Share a link to this book page on your preferred social network or via email.

    « Back Next »
  8. ×

    View our suggested citation for this chapter.

    « Back Next »
  9. ×

    Ready to take your reading offline? Click here to buy this book in print or download it as a free PDF, if available.

    « Back Next »
Stay Connected!