Summary and Recommendations

Despite the decades of success that the Department of Defense (DOD) has had in leveraging information technology (IT) across the defense enterprise to build the world’s most powerful military force, its acquisition of IT systems continues to be burdened with problems. Briefings to the National Research Council’s (NRC’s) Committee on Improving Processes and Policies for the Acquisition and Test of Information Technologies in the Department of Defense from the staff of the Office of the Secretary of Defense (OSD) show that the acquisition of major automated information systems (MAIS) remains especially slow. This problem has been broadly recognized for years, and there have been many efforts at reform in the past. Nevertheless, today’s processes for the acquisition and testing of the DOD’s IT systems all too often deliver solutions that are too late to satisfy the needs of the user community. Current fielding cycles are, at best, two to three times longer than successful commercial equivalents according to presentations to the committee—representing multiyear delays in delivering improved IT systems to warfighters and the organizations that support them. As a result, the DOD is often unable to keep pace with the rate of IT innovation in the commercial marketplace, cannot fully capitalize on IT-based opportunities, and is unable to deliver IT-based capabilities rapidly to meet urgent requirements. Moreover, to the extent that adversaries have access to state-of-the-art IT and can put it to operational use rapidly, the DOD faces risks across the battlespace.

Regarding the adoption and use of IT, the commercial sector has also experienced delays and failures as well as extraordinary successes. Of



The National Academies | 500 Fifth St. N.W. | Washington, D.C. 20001
Copyright © National Academy of Sciences. All rights reserved.
Terms of Use and Privacy Statement



Below are the first 10 and last 10 pages of uncorrected machine-read text (when available) of this chapter, followed by the top 30 algorithmically extracted key phrases from the chapter as a whole.
Intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text on the opening pages of each chapter. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.

Do not use for reproduction, copying, pasting, or reading; exclusively for search engines.

OCR for page 1
Summary and Recommendations Despite the decades of success that the Department of Defense (DOD) has had in leveraging information technology (IT) across the defense enterprise to build the world’s most powerful military force, its acquisi - tion of IT systems continues to be burdened with problems. Briefings to the National Research Council’s (NRC’s) Committee on Improving Processes and Policies for the Acquisition and Test of Information Tech - nologies in the Department of Defense from the staff of the Office of the Secretary of Defense (OSD) show that the acquisition of major automated information systems (MAIS) remains especially slow. This problem has been broadly recognized for years, and there have been many efforts at reform in the past. Nevertheless, today’s processes for the acquisition and testing of the DOD’s IT systems all too often deliver solutions that are too late to satisfy the needs of the user community. Current fielding cycles are, at best, two to three times longer than successful commercial equivalents according to presentations to the committee—representing multiyear delays in delivering improved IT systems to warfighters and the organizations that support them. As a result, the DOD is often unable to keep pace with the rate of IT innovation in the commercial marketplace, cannot fully capitalize on IT-based opportunities, and is unable to deliver IT-based capabilities rapidly to meet urgent requirements. Moreover, to the extent that adversaries have access to state-of-the-art IT and can put it to operational use rapidly, the DOD faces risks across the battlespace. Regarding the adoption and use of IT, the commercial sector has also experienced delays and failures as well as extraordinary successes. Of 

OCR for page 1
 ACHIEVING EFFECTIVE ACQUISITION OF IT IN THE DOD special interest in the commercial sector, however, is the extent to which new or adaptive corporations using agile approaches for the acquisition and operational deployment of IT capabilities have outstripped indus - trial giants that were beset with their own ponderous, process-bound, industrial-age management structures. As called for in the committee’s statement of task (see Box P.1 in the preface), this report examines the acquisition, culture, practices, pro - cesses, and rules within the DOD as they apply to information tech - nology; assesses whether the DOD could adopt best practices from the commercial sector for IT acquisition, systems engineering, and test and evaluation (T&E); and makes recommendations to improve the speed and effectiveness of IT acquisition programs. Many studies have recommended reforms to the defense acquisition system—that is, the institutions, processes, and rules that govern the development, procurement, testing, and fielding of new capabilities. A number of these, including a recent study by the Defense Science Board, 1 have focused on IT acquisition and concluded that there is a need for a unique acquisition process for IT. This study reaches the same fundamen - tal conclusion but adds another dimension in its elaboration of differing types of IT systems and offers a suggested acquisition process for each. SCOPE AND VOCABULARy FOR INFORMATION TECHNOLOGy ACQUISITION PROGRAMS Information technology is used for a wide variety of purposes in the DOD, a breadth suggested by the definition of “information enterprise” in DOD Directive (DODD) 8000.1.2 However, the issues, challenges, and potential solutions are not the same for all elements of the information enterprise. Moreover, the present defense systems acquisition vocabulary 1 Defense Science Board, Report of the Defense Science Board Task Force on Department of De - fense Policies and Procedures for the Acquisition of Information Technology, Office of the Under Secretary of Defense for Acquisition, Technology and Logistics, Washington, D.C., March 2009. 2 DOD Directive 8000.1 (Management of the Department of Defense Information Enter- prise) defines “information enterprise” as follows: “The DoD information resources, assets, and processes required to achieve an information advantage and share information across the Department of Defense and with mission partners. It includes: (a) the information itself and the Department’s management over the information life cycle; (b) the processes, including risk management, associated with managing information to accomplish the DoD mission and functions; (c) activities related to designing, building, populating, acquiring, managing, operating, protecting, and defending the information enterprise; and (d) related information resources such as personnel, funds, equipment, and IT, including national security systems.”

OCR for page 1
 SUMMARY AND RECOMMENDATIONS embodied in the law and regulations governing IT acquisition3 does not provide a taxonomy of IT program characteristics suitable to allowing program types to be matched with appropriate acquisition approaches. Consequently, the committee created new definitions for a well-defined subset of the information enterprise that has parallels in the commercial sector. The committee decided to consider IT systems to be just those sys - tems that support the DOD information enterprise,” especially those systems expected to run on or interface with existing infrastructure and systems that are user-facing, and limited to those that are deliv - ered through the acquisition process (and not systems “homegrown” in individual commands). (An IT program is defined in this report as the process for acquiring an IT system as defined here.) Excluded from this subset are the IT-based components embedded in weapons systems or DOD-specific hardware. The committee believes that as a general rule such embedded components (for example, firmware that controls a missile’s flight) are so integral to the weapons system that they are best managed using the same processes as those used for managing the other elements of the weapon. The committee considers it likely that at least some of the conclusions in this report would also apply to other systems with significant IT content—but this report and its recommendations are focused on opportunities to improve the acquisition of IT systems as defined here. To better align the acquisition approach with technical characteristics and risk, the committee further subdivides IT systems into two categories: • Software deelopment and commercial off-the-shelf (COTS) integration (SDCI) programs—those focused on the development of new software to provide new functionality or focused on the development of software to integrate COTS components, and • COTS hardware, software, and serices (CHSS) programs—those focused exclusively on COTS (hardware, software, or services) without modification for DOD purposes (that is, the capabilities being purchased are determined solely by the marketplace and not by the DOD). By separating programs that involve software development and/or integration from programs that entail simply adopting COTS technology, these categories facilitate the comparison of defense acquisition processes 3See, for example, DOD Directive 5002.15000 series regulations; Title 10, U.S.C., Chapters 144 (“Major Defense Acquisition” Programs) and 144A (“Major Automated Information Systems Programs”); and DOD Directive 8000.1.

OCR for page 1
 ACHIEVING EFFECTIVE ACQUISITION OF IT IN THE DOD with commercial best practices and the adoption of practices focused on effective and timely leveraging of the commercial IT cycle. FINDINGS AND RECOMMENDATIONS IT Acquisition Process and Culture (Chapter 2) Findings The acquisition framework of the Department of Defense4 prescribes elaborate governance mechanisms and cost thresholds that trigger vary - ing levels of oversight and review. In response to acquisition problems from the past, more oversight and governance, some of it excessively process-centric and adversarial, have been added. Collectively, these well- intended changes have made the timely delivery of IT capabilities more difficult. Today’s IT acquisition process often focuses on review docu - ments and other process artifacts, and the acquisition culture does not reward early and transparent feedback on capabilities and limitations. Success as determined by process metrics in acquisition does not neces - sarily align with success metrics based on the timely delivery of end-user capability. The DOD’s perceived need for caution over speed is understandable. Given the criticality and danger of its mission, its worldwide operations and large workforce, and the frequent need for clear, decisive action, the DOD by its nature is an organization with a classic command-and- control culture. However, if current trends continue, it is likely that processes and systems will become even more top-down and centralized, in spite of the DOD’s desire to move to an integrated, cross-Service environment with empowered decision making at all levels of command. DOD systems acquisition policies, expertise, practice, and culture— including those applied to IT systems—reflect the practices, policies, and cultural norms associated with large weapons systems programs. This report does not address the issue of whether the process for weapons systems is appropriate. Other studies and reports have focused on that question—and many have pointed out shortcomings. With respect to IT, however, there is a long-standing reluctance to deviate from stan- dard weapons system acquisition processes, and acquisition personnel are not trained or led to differentiate the unique aspects of IT systems acquisition. 4 As set forth in Title 10, U.S.C., Chapters 144 (“Major Defense Acquisition Programs”) and 144A (“Major Automated Information System Programs”) and DOD Directive 5002.15000 series regulations.

OCR for page 1
 SUMMARY AND RECOMMENDATIONS The application of current, weapons-system-based acquisition pro - cesses to IT systems has a number of deleterious effects on the DOD’s ability to deliver needed end-user capability, including the following: • With the exception of hardware and licensed software purchased through vehicles such as the DOD Enterprise Software Initiative contracts, applicable COTS technologies are insufficiently leveraged, excessively tailored, inefficiently tested, and excessively delayed. Many programs have experienced acquisition or integration lead times that significantly exceed the life cycles of the underlying COTS technology. The discrepancy between DOD fielding cycles and COTS life cycles is stark, and measured in years. • The oversight process focuses too much on shortcomings of COTS products and services and inhibits the timely delivery of meaningful (albeit imperfect) end-user capabilities. • IT program requirements are often written with overly detailed specifications, take a long time to develop, and are not consistent with the pace of technological change or the rapid delivery of end-user capabilities. • DOD acquisition, budgeting, and requirements processes, which are designed for large weapons systems acquisition programs, are being inappropriately applied to relatively low-dollar IT programs. • Dollar thresholds are used to assign the level of oversight for IT programs. These levels are significantly lower than the dollar levels used for determining oversight levels for weapons system programs. This dis- parity subjects too many IT programs to time-consuming, high-level DOD oversight and prevents the delegation of oversight to lower levels that are more agile. • The DOD’s acquisition training curriculum does not adequately address the special challenges of IT system acquisition or prepare pro- gram managers to run IT programs effectively. This shortfall impedes the DOD’s ability to assess, adapt, and adopt applicable commercial methods, processes, products, and services. Recommendations Recommendation 1. Adopt a new acquisition process tailored for IT systems. For IT systems, the acquisition processes, which are currently defined by the 5000 series of DOD regulations, should be replaced with a new process designed specifically for the timely and effective acquisition of

OCR for page 1
 ACHIEVING EFFECTIVE ACQUISITION OF IT IN THE DOD IT systems. Elements of this process are detailed in Chapter 3 and in Appendixes B and C. The supporting recommendations that follow address key areas where an adjustment of acquisition practice and cultural change are needed. Recommendation 1.1. Emphasize timeliness and end-user mission success in the DOD IT acquisition culture rather than rigid over- sight and process compliance. Cultural aspects of the DOD acquisition process that have an impact on the potential success of IT acquisition efforts include the following: the bias that larger is better, the sense that oversight personnel have no accountability for delaying needed IT capabilities, an emphasis on process risk (executing the acquisition process correctly) rather than on the risk of late delivery of end-user capability, an unwillingness to admit program failure, an emphasis on process over product, a belief that the DOD is genuinely unique, and the belief that what is good for large weapons systems should be good enough for IT systems. In addition to implementing a new acquisition process for IT capa- bilities, the DOD should institute a companion program to address the cultural changes required to make the new acquisition process successful. As examples, the DOD should require that all personnel in the oversight process have accountability for the program (that is, for helping the pro- gram succeed or helping terminate programs that are fatally flawed or no longer required). The focus should be on providing products with value to end users, not the oversight process per se. It is also important that the oversight culture support small, incremental cycles for IT development and fielding. Recommendation 1.2. State IT systems requirements as top-level mission expectations (that is, “big-R” requirements) rather than as detailed processes or technical solutions; develop the details (“small-r” requirements) by iterative refinement with users. Today, there is an extensive requirements-definition period for the typical IT program, resulting in a large volume of requirements docu- mentation that must be formally validated by the Joint Requirements Oversight Council for major joint programs. This requirements-defini - tion and budgeting process encourages the aggregation of requirements into larger programs. In turn, these requirements documents are used to justify program budget requests, and the approval of these requests is a condition for proceeding to successive acquisition program milestones. Because the requirements-definition and approval processes introduce

OCR for page 1
 SUMMARY AND RECOMMENDATIONS significant delays, requirements documents for IT programs often have become inaccurate descriptions of user needs by the time that funding is obtained and the acquisition process is initiated. Instead, IT systems requirements should be defined at the mission capability level. Top-level requirements (big-R) should be rapidly devel - oped and validated, and more detailed requirements (small-r) should be developed as an integral part of the acquisition process.5 Specifically, based on iterative interactions with the actual end users of the IT capa- bility as well as on assessments of available technology, more detailed requirements would be developed for individual programs or projects. As the acquisition effort progresses, feedback from operational users after deployment of increments of capability would be used to feed the con- tinued evolution of (small-r) requirements for the IT acquisition effort. Doing so will permit iterative refinement with users and the best use of commercially available technologies. At the same time, this recom- mendation should not be construed to suggest that big-R requirements will not change. Feedback on those is also important and should be accommodated. The committee’s recommended process is similar to one recently established by a Joint Capabilities Integration Development System pol - icy. The approach is currently being applied to a small set of programs; the committee encourages the expansion of this concept to encompass all IT programs. Recommendation 1.3. Leverage flexibilities within IT acquisi- tion funding to achieve speed and agility in the new acquisition process. The DOD’s process for obtaining funding for new acquisition pro- grams typically takes a number of years. In summary, a DOD capability shortfall is linked to a request to Congress for funding that would be pro- vided in a future year. For solutions that will rely on IT, the time frame for seeking funding can be many times longer than the actual time needed to develop or procure the solution. To achieve rapid delivery of IT solutions, a more responsive process is needed for justifying and allocating funding to address capability shortfalls. In the short term, the DOD should work with Congress to explore how to make use of flexibility consistent with current legal requirements. 5 “Big-R” requirements convey a widely recognized purpose, mission, and expected out - come. “Small-r” requirements provide a set of more detailed requirements associated with specific user interfaces and utilities that will evolve within the broader specified architecture as articulated in the initial big-R requirements.

OCR for page 1
 ACHIEVING EFFECTIVE ACQUISITION OF IT IN THE DOD (Even where legislative changes are not needed, flexibility must nonethe - less be negotiated with the congressional defense oversight committees.) For example, the DOD has authority to allocate funds for urgent war- fighter needs and to reallocate funding after congressional appropriation as a result of changing needs. This authority could be used to begin the development of a needed capability in weeks or months. Also, acquisition funds are sometimes allocated by Congress to a larger mission or program area, or in some cases to a portfolio of projects identified with an area of mission need—for example, to fund software upgrades in a particu- lar mission area or system. Funding could rapidly be allocated to meet demands for IT capabilities within these areas. In both cases, transparent reporting to Congress will be essential to ensure proper oversight and to demonstrate that the flexibility yields a more rapid delivery of capability to the field. In the longer term, the DOD should work with Congress to estab- lish a new set of funding mechanisms for meeting IT requirements that would align congressional funding with mission or capability areas rather than with individual acquisition programs. Under this concept, Congress would allocate funding to a mission area that would be governed in the DOD with a portfolio-management-like process. In implementing this concept, DOD officials would be responsible for setting priorities and allocating the funding to individual projects following appropriations for a portfolio of mission requirements. This approach would ensure appro - priate justification of funding needs tied to mission requirements during budget submission as well as the rapid allocation of appropriated funding consistent with the pace of evolving mission requirements and technology advancements. Currently, the DOD uses a process similar to this concept for funding maintenance upgrades to aircraft avionics software. Likewise, a somewhat similar process is also used for managing IT projects funded through working capital funding processes. Recommendation 1.4. Provide IT systems acquisition profession- als with education in modern IT systems and establish minimum competency standards. Training a professional acquisition workforce in modern IT program management is critical. IT systems-acquisition professionals must under- stand key discriminants of the process, be able to evaluate proposals, and make trade-offs. Their training should include requirements specification; development, integration, and test processes; listening to and incorporat - ing “the voice of the user”; and the use of COTS.

OCR for page 1
 SUMMARY AND RECOMMENDATIONS Recommendation 1.5. Use pilot programs to institutionalize the new IT acquisition process recommended in this report. Pilot programs would provide the DOD with the means to apply new acquisition approaches and to capture valuable lessons learned and develop the necessary guidance for applying the knowledge acquired to larger and future programs. The committee believes that the establish- ment of 10 pilot programs for the rapid acquisition of IT systems capabili- ties under the alternative IT acquisition process described in this report would greatly help in moving the acquisition process forward. These pilot programs could be the catalyst for institutionalizing a new and updated IT systems acquisition process and provide an opportunity for the DOD to deepen the IT systems acquisition experience of its workforce. Recommendation 1.6. Propose legislative and regulatory changes (1) to codify a new agile process for acquiring IT systems and (2) to revise dollar thresholds for the oversight of IT systems acquisition in order to foster decentralization. The committee proposes that two major initiatives be undertaken to update legislation and regulations governing IT acquisition in the DOD. The first initiative would involve the adoption of a new agile process as the default for the acquisition of IT systems. Central to the new process would be the concept of the iterative, incremental acquisition of capa- bilities that are delivered to end users as successive packages that are aggregated over time into comprehensive capabilities—a concept that is consistent with today’s best commercial practices. This concept would apply to the acquiring of custom IT capabilities that are developed or integrated and are deployed on existing IT infrastructures (SDCI systems) as described above. In addition, a companion process would be tailored to the acquisition of off-the-shelf IT capabilities (CHSS systems). Processes for both types of IT acquisition are described in detail in this report. The second initiative would be to restructure and decentralize the IT acquisition oversight process in order to align it with the fast-paced cycles of agile and rapid acquisition. Today’s acquisition oversight process in the DOD is designed for the disciplined management of large, expen - sive, and complex weapons systems. However, the dollar thresholds for designating oversight levels for IT programs are significantly lower than those used for weapons systems (by a factor of five). Moreover, the cur- rent legislation has no provision for MAIS programs to be designated as acquisition category (ACAT) II, which would provide for oversight at the Service or agency level. One approach to solving the problem of highly centralized oversight with its attendant delays would be to use the same

OCR for page 1
0 ACHIEVING EFFECTIVE ACQUISITION OF IT IN THE DOD dollar thresholds in effect for major defense acquisition programs for the designation of ACAT levels to MAIS programs. This change in thresholds for IT programs would foster decentralization and would better align authority for IT program oversight to the appropriate levels—at OSD, the Services and agencies, and lower echelons. IT Systems and Software Engineering Processes and Practices (Chapter 3) Findings Information technology programs are profoundly affected by the rapid and relentless pace of change in the underlying technologies. Hard - ware capability per unit expenditure doubles roughly every 18 months. Software capability is driven by the even-faster pace of technology change in the Internet environment and by the elevated level of end-user expecta- tions that this causes. DOD IT systems acquisition programs progress at a markedly slower pace. As a result, the DOD is unable to keep pace with the rate of IT innovation in the commercial marketplace, cannot fully capi- talize on IT-based opportunities, is unable to deliver IT-based capabilities rapidly, and, accordingly, will not have the requisite agility. Historically, system development has followed a “waterfall” process calling for formally documented specification, followed by a request for bids, followed by contracting, delivery, installation, and maintenance. Major elements of the waterfall method, which is document-intensive, attempt to satisfy management’s goals for ensuring that projects will suc - cessfully meet their objectives, but they end up emphasizing the acquisi- tion process rather than the capability being delivered. To deliver software capability more rapidly, the commercial world has widely embraced the iterative, incremental development (IID) approach, which addresses two issues of central importance for IT systems—(1) the need for user interaction in setting requirements and (2) complexity. 6 Key attributes of the IID approach include (1) the prominence of the end user’s voice, (2) a focus on big-R requirements during early planning, (3) the close integration of developmental and operational T&E into the development cycle, and (4) the breaking down of a project into incremen- 6 There is another, alternative strategy based on incremental development strategies—evo - lutionary acquisition, which addresses a concern somewhat different from that addressed by IID, namely, the issue of technology maturation. Under evolutionary acquisition, early increments provide end-user capabilities based on mature technology, and work on later increments is deferred until needed technology has been matured.

OCR for page 1
 SUMMARY AND RECOMMENDATIONS tally deliverable parts. The commercial world has widely embraced the IID approach. Although the DOD’s current governance and oversight structure per- mits tailoring and provides the flexibility needed for a milestone decision authority and program manager to adjust how the process is applied to specific programs, there is no established best practice or accepted template for tailoring. Instead, each decision maker must independently address on an ad hoc basis the misalignment between an oversight system designed for hardware and weapons system development and the very different types of issues that contribute to risk in an IT system program. With the existence of multiple oversight bodies and large numbers of participants in the program oversight and review process, the current sys- tem gives undue leverage to groups that often are not true stakeholders in the process. This can have negative effects, including too many detailed and ad hoc small-r requirements placed on the program, an inability to pri- oritize requirements effectively, and “corner case” requirements (focused on rare situations that only occur outside normal operating parameters) that can be contradictory or extremely difficult to implement. Recommendations Recommendation 2. Adopt an iterative, incremental approach for acquiring information technology systems. The following supporting recommendations are aimed at enabling more rapid and nimble IT systems acquisition processes and fostering the institutionalizing of IID practices. Recommendation 2.1. Establish iterative, incremental development (IID) processes based on agile software development and related approaches as the default for IT system development. Software engineering, although still a young discipline, has advanced substantially over the past four decades, and much is understood about how to structure development efforts to better manage the unique risks inherent in IT system programs and to improve the probability of success. In particular, it is now widely appreciated that the waterfall development process is not appropriate for most software development projects. One emerging approach to IID that has many attractive attributes is agile software development (ASD),7 which has at its core ongoing, integrated developmental and operational testing and a close coupling between 7 A. Cockburn, Agile Software Deelopment, Addison-Wesley, Boston, Mass., 2001.

OCR for page 1
 ACHIEVING EFFECTIVE ACQUISITION OF IT IN THE DOD those responsible for requirements, for development, and for testing. It emphasizes frequent testing and interaction over formal, up-front docu - mentation. The adoption and implementation of an agile-inspired IID approach do not equate simply to compressing traditional waterfall mod- els into shorter time periods, nor does appending current document-cen - tric oversight processes to a series of release phases equate to the use of ASD. IID and ASD are proven approaches for building capabilities in line with end-user expectations and needs and for rapidly iterating require - ments and solutions based on end-user feedback. Such approaches have been embraced in the commercial world as being highly effective ways to deliver incremental improvements to the field rapidly. Their adoption not only would conform to widely accepted commercial best practices but also would implement more than 20 years of recommendations from forums such as the Defense Science Board. The adoption of IID approaches coupled with a focus on the end-user experience does not mean, however, that other stakeholders and nonfunctional requirements (such as informa- tion assurance, reliability, and so on) are unimportant. Historically, other stakeholder voices have dominated the process to the exclusion of the end user. The committee urges a rebalancing and a focus on end-user mission success, because it is the end user who is in the best position to judge whether a capability is useful and should be fielded. Recommendation 2.2. Allocate top-level DOD mission expecta- tions (i.e., big-R requirements) across increments and use each increment to define and satisfy detailed requirements (i.e., small-r requirements). Beyond the highest set of (big-R) requirements, there is a more detailed set of (small-r) requirements for such things as specific user interfaces and utilities that will be developed and will evolve within the context of the big-R requirements. Stated another way, users cannot effectively articulate their requirements without interacting with (even partially) fielded sys- tems—in IT systems, one cannot establish what is needed without a sense of what is possible. Requirements such as the expected user interface and user paradigms and integration with other concurrently evolving systems and security practices all dictate that the initial specification be limited to broad system goals and physical operating conditions and that more detailed requirements be evolved in concert with user feedback garnered during incremental development cycles. Recommendation 2.3. Establish separate and distinct strategies and processes for acquiring custom versus off-the-shelf IT systems.

OCR for page 1
13 SUMMARY AND RECOMMENDATIONS There are fundamentally different classes of issues involved when dealing with software development and commercial off-the-shelf integra - tion, or SDCI, versus the commercial off-the-shelf hardware, software, and services, or CHSS, components of IT system programs; therefore, different strategies are appropriate when addressing them. In both cases, rapid change in virtually all aspects of the technology—requirements, capabilities, user expectations, usage and development environments, and so on—is a fundamental factor that must be addressed, and IID acqui- sition strategies are appropriate. Where the requirement includes COTS hardware or licensed software that can be procured at volume discounts through DOD enterprise contract vehicles (such as Enterprise Software Initiative contracts), decentralized procurement should be encouraged. When the DOD awards contracts requiring that such components are to be integrated into an IT system solution, those COTS products should be acquired by the most cost-effective means available. Options should include authorizing the contractor to procure COTS components from established government vehicles such as Enterprise Software Initiative contracts and providing the COTS components as government-furnished equipment to avoid unnecessary handling or procurement fees. This is particularly true for licensed software when existing enterprise contracts permit direct downloading. Recommendation 2.4. Establish, employ, and report measures of success that emphasize the end-user experience, including timeli- ness to field. The committee fully anticipates that any new IT systems acquisition process that is defined and adopted will need to evolve over time. The committee has identified shortcomings in the present system, and recom - mends a new direction. An appropriate evaluative framework should consider end-user capability, end-user satisfaction, timeliness, quality, operating costs, and acquisition (that is, improvements in the process and overall competitive strategy). Recommendation 2.5. Provide a stable budget profile across mul- tiple increments for iterative, incremental development of IT programs. Stable budget profiles for the IT program acquisition cycle are neces - sary to ensure that end users’ requirements can and will be addressed, some at the outset and others in future increments. This will avoid the unintended but real consequence of users attempting to overload require-

OCR for page 1
14 ACHIEVING EFFECTIVE ACQUISITION OF IT IN THE DOD ments into the first capability increment. Multiple time-boxed capability increments will fit within each Planning, Programming, Budgeting, and Execution budget cycle. The confidence of key stakeholders, including users, will increase, as will the transparency of overall program execu- tion status. IT System Testing (Chapter 4) Findings Today, testing discipline is integrated too late and serially into DOD IT systems acquisition practices. Indeed, programs generally defer the testing of IT systems in realistic operational environments until the man- dated operational test. Without regular feedback from a user perspective on IT system development, program managers and milestone decision authorities lack critical information for managing and supporting pro- grams, and other key stakeholders lack the knowledge that can build their confidence. This approach stands in contrast to best commercial practice, whereby user testing is performed early and often to ensure that user feedback guides all stages of system development. Because DOD end- user engagement is nonexistent or limited, the test community is unfairly tasked with representing the perspective and needs of end users and is often engaged too late to have a substantive impact on requirements, system architecture, or system functionality. Finally, the acquisition com- munity has been reluctant to embrace virtualized testing and has been overtly precluded from reusing or accessing operationally relevant test data and environments. Recommendations Recommendation 3. Perform continuous testing, with early involve- ment from end users, in acquiring DOD information technology systems. The following supporting recommendations would establish testing and evaluation policies and practices in support of the committee’s pro- posed DOD IT systems acquisition process.  Time boxing refers to a deadline-driven approach to system development in which work items may slip from one iteration to the next, but iterations are completed according to schedule, thus affording the opportunity to quickly identify erroneous estimates of the time required to complete deliverables and ensuring continuous user input regarding priorities.

OCR for page 1
15 SUMMARY AND RECOMMENDATIONS Recommendation 3.1. Adopt continuous testing in DOD IT systems development, and insist on the use of metrics, especially emphasiz- ing measures of end-user satisfaction. Continuous user testing is an integral part of successful IT systems development and deployment. An acceptance team emphasizing the per- spective of the end user and composed of operational end users, develop- ment testing and operational testing stakeholders, security certification and accreditation stakeholders, and interoperability stakeholders should be continuously integrated into the development process. Such integra- tion is critical to ensuring both that the big-R requirements are the right ones and that the system is meeting both big-R and small-r requirements. To facilitate measuring success, a process involving a robust set of metrics should be in place through an IT system’s life cycle, from development to ultimate decommissioning. A process that collects, aggregates, and analyzes metrics on end-user service consumption and experiences will provide visibility into what features end users are actually using. Success- ful commercial suppliers of IT systems place considerable emphasis on metrics that capture actual end-user behavior measured by online interac- tions with deployed IT systems. Recommendation 3.2. Emphasize the needs of end users by having the acceptance team play a lead role in recommending deployment decisions. Continuous user input is critical to the successful development of new IT systems. There are numerous examples of new IT systems meeting every formal aspect of their specifications but not delivering the expected value to their users; conversely, a system that may not meet all aspects of a specification may already be an improvement over existing systems. With these situations in mind, the acceptance team should play a leading role in recommending deployment decisions. This would prevent the deploy - ment of systems that are not adding value (independent of their readi- ness on paper per the specification), and it would also ensure more rapid deployment of systems that can improve the effectiveness of DOD users (especially the warfighter). Deployments may take the form of small trials, large-scale “beta” programs, or even full production deployments. Recommendation 3.3. Test with users in their actual work or field environment (sometimes referred to as a beta deployment). It is essential to put capabilities into the hands of users early and to measure performance and otherwise obtain feedback. Engaging expected

OCR for page 1
16 ACHIEVING EFFECTIVE ACQUISITION OF IT IN THE DOD users of the system early enough makes it possible to provide meaningful feedback to developers and influence the small-r requirements. One espe- cially useful approach is to engage users through pilot projects deployed in the field, possibly operating in parallel with production systems. Recommendation 3.4. Accept certification and functional IT system component test results across organizational boundaries. The DOD has broadly adopted a set of networking capabilities that are integral to every IT system. As a matter of DOD policy, such capabilities should not be required to undergo separate revalidation or recertification during operational testing. Examples include such capabilities as Internet Protocol and Domain Name System software modules. As more of the technology stack becomes commoditized or provided as a service, the set of associated capabilities not requiring revalidation should likewise grow. Current examples include public key infrastructure and on-demand com - puting and storage. The same policy should apply to DOD-deployed and DOD-certified software modules that are reused across DOD programs. The testing of the composite system product that is evaluated in a realistic environment for operational effectiveness and suitability is sufficient. Recommendation 3.5. Use virtual test environments to support both continuous feedback and certification of IT program increments. A variety of opportunities to establish virtual test environments already exist, as identified in Chapter 4. The definition of test environ - ment could, over time, be significantly extended to include the use of beta testing in actual operational environments, the use of commensurate commercial data and/or operating environments, and the extension of virtual test environments to include operations monitoring as a source of continuous test feedback. The DOD should codify this approach and encourage the use of virtual environments. Operational realism is impor- tant but must be balanced against pragmatic considerations. COnClUSIOn The current DOD approach to IT acquisition has not been broadly successful in delivering needed capability in a timely manner. To lever- age the potential of IT fully, it is essential that the DOD not simply alter a process that has repeatedly failed. Instead, it should adopt a new process tailored specifically to IT system acquisition. Full commitment to this change will touch every aspect of the DOD culture, its processes, and its workforce.