Achieving Effective Acquisition of Information Technology in the Department of Defense

Below are the first 10 and last 10 pages of uncorrected machine-read text (when available) of this chapter, followed by the top 30 algorithmically extracted key phrases from the chapter as a whole.
Intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text on the opening pages of each chapter. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.

Do not use for reproduction, copying, pasting, or reading; exclusively for search engines.

OCR for page R1
Committee on Improving Processes and Policies for the Acquisition and Test of Information Technologies in the Department of Defense Computer Science and Telecommunications Board Division on Engineering and Physical Sciences

OCR for page R1
THE NATIONAL ACADEMIES PRESS 500 Fifth Street, N.W. Washington, DC 20001 NOTICE: The project that is the subject of this report was approved by the Gov- erning Board of the National Research Council, whose members are drawn from the councils of the National Academy of Sciences, the National Academy of Engi- neering, and the Institute of Medicine. The members of the committee responsible for the report were chosen for their special competences and with regard for appropriate balance. This study was supported by Contract No. W911NF-07-C-0115 between the National Academy of Sciences and the Defense Information Systems Agency. Any opinions, findings, conclusions, or recommendations expressed in this publication are those of the author(s) and do not necessarily reflect the views of the organiza- tions or agencies that provided support for the project. International Standard Book Number-13: 978-0-309-14828-3 International Standard Book Number-10: 0-309-14828-6 Copies of this report are available from the National Academies Press, 500 Fifth Street, N.W., Lockbox 285, Washington, DC 20055; (800) 624-6242 or (202) 334-3313 (in the Washington metropolitan area); Internet, http://www.nap.edu. Copyright 2010 by the National Academy of Sciences. All rights reserved. Printed in the United States of America

OCR for page R1
The National Academy of Sciences is a private, nonprofit, self-perpetuating society of distinguished scholars engaged in scientific and engineering research, dedicated to the furtherance of science and technology and to their use for the general welfare. Upon the authority of the charter granted to it by the Congress in 1863, the Academy has a mandate that requires it to advise the federal govern - ment on scientific and technical matters. Dr. Ralph J. Cicerone is president of the National Academy of Sciences. The National Academy of Engineering was established in 1964, under the charter of the National Academy of Sciences, as a parallel organization of outstanding engineers. It is autonomous in its administration and in the selection of its mem - bers, sharing with the National Academy of Sciences the responsibility for advis - ing the federal government. The National Academy of Engineering also sponsors engineering programs aimed at meeting national needs, encourages education and research, and recognizes the superior achievements of engineers. Dr. Charles M. Vest is president of the National Academy of Engineering. The Institute of Medicine was established in 1970 by the National Academy of Sciences to secure the services of eminent members of appropriate professions in the examination of policy matters pertaining to the health of the public. The Institute acts under the responsibility given to the National Academy of Sciences by its congressional charter to be an adviser to the federal government and, upon its own initiative, to identify issues of medical care, research, and education. Dr. Harvey V. Fineberg is president of the Institute of Medicine. The National Research Council was organized by the National Academy of Sciences in 1916 to associate the broad community of science and technology with the Academy’s purposes of furthering knowledge and advising the federal government. Functioning in accordance with general policies determined by the Academy, the Council has become the principal operating agency of both the National Academy of Sciences and the National Academy of Engineering in pro - viding services to the government, the public, and the scientific and engineering communities. The Council is administered jointly by both Academies and the Institute of Medicine. Dr. Ralph J. Cicerone and Dr. Charles M. Vest are chair and vice chair, respectively, of the National Research Council. www.national-academies.org

OCR for page R1

OCR for page R1
COMMITTEE ON IMPROVING PROCESSES AND POLICIES FOR THE ACQUISITION AND TEST OF INFORMATION TECHNOLOGIES IN THE DEPARTMENT OF DEFENSE WILLIAM H. CAMPBELL, BAE Systems, Inc., Co-Chair DAWN C. MEYERRIECKS,1 Dawn Meyerriecks, LLC, Co-Chair ROBERT F. BEHLER, MITRE Corporation PHILIP E. COYLE III, World Security Institute RENATO A. DiPENTIMA, SRA International (retired) JOHN M. GILLIGAN, Gilligan Group, Inc. JOHN GOODENOUGH, Carnegie Mellon University PAUL J. KERN (NAE),2 The Cohen Group H. STEVEN KIMMEL, Alion Science and Technology DEIDRE A. LEE, Professional Services Council JOSHUA S. LEVINE, ESP Technologies Corporation NACHIAPPAN NAGAPPAN, Microsoft Research FRANK A. PERRY, Science Applications International Corporation VAHO REBASSOO, The Boeing Company DANIEL C. STURMAN, Google, Inc. Staff JON EISENBERG, Director, Computer Science and Telecommunications Board KEVIN LEWIS, Senior Program Officer LYNETTE I. MILLETT, Senior Program Officer RENEE HAWKINS, Financial and Administrative Manager VIRGINIA BACON TALATI, Program Associate MORGAN MOTTO, Program Associate (through April 2009) 1 Dawn Meyerriecks resigned from the committee in September 2009 upon her appoint - ment as Deputy Director of National Intelligence for Acquisition and Technology. 2 National Academy of Engineering. 

OCR for page R1
COMPUTER SCIENCE AND TELECOMMUNICATIONS BOARD ROBERT F. SPROULL, Sun Microsystems, Inc., Chair PRITHVIRAJ BANERJEE, Hewlett Packard Company WILLIAM J. DALLY, NVIDIA Corporation and Stanford University DEBORAH ESTRIN, University of California, Los Angeles KEVIN C. KAHN, Intel Corporation JAMES KAJIYA, Microsoft Corporation JOHN E. KELLY III, IBM Research JON M. KLEINBERG, Cornell University WILLIAM H. PRESS, University of Texas, Austin PRABHAKAR RAGHAVAN, Yahoo! Research DAVID E. SHAW, Columbia University ALFRED Z. SPECTOR, Google, Inc. PETER SZOLOVITS, Massachusetts Institute of Technology PETER J. WEINBERGER, Google, Inc. JON EISENBERG, Director VIRGINIA BACON TALATI, Program Associate SHENAE BRADLEY, Senior Program Assistant RENEE HAWKINS, Financial and Administrative Manager HERBERT S. LIN, Chief Scientist LYNETTE I. MILLETT, Senior Program Officer ERIC WHITAKER, Senior Program Assistant ENITA A. WILLIAMS, Associate Program Officer For more information on CSTB, see its website at http://www.cstb.org, write to CSTB, National Research Council, 500 Fifth Street, N.W., Washington, DC 20001, call (202) 334-2605, or e-mail the CSTB at cstb@nas.edu. i

OCR for page R1
Preface The information technology (IT) revolution of the past several decades has dramatically changed the world. The Internet, Web 2.0 technologies, social networking tools, online search engines, text messaging, video teleconferencing, and multimedia-enabled smart-phones with embedded cameras are but a sample of IT-based capabilities that have altered the ways in which people communicate and work. In the military, IT has enabled profound advances in weapons sys- tems and the management and operation of the defense enterprise. A significant portion of the Department of Defense (DOD) budget is spent on capabilities acquired as commercial IT commodities, developmental IT systems that support a broad range of warfighting and functional applications, and IT components embedded in weapons systems. The ability of the DOD and its industrial partners to harness and apply IT for warfighting, command and control and communications, logistics, and transportation has contributed enormously to fielding the world’s best defense force. But despite the DOD’s decades of success in leveraging IT across the defense enterprise, the acquisition of IT systems continues to be burdened with serious problems. Accordingly, the Defense Information Systems Agency (DISA) asked the National Research Council (NRC) to assess the efficacy of the DOD’s acquisition and test and evaluation (T&E) pro - cesses as applied to IT. In response, the NRC formed the Committee on Improving Processes and Policies for the Acquisition and Test of Infor- mation Technologies in the Department of Defense—a group of IT sys- ii

OCR for page R1
iii PREFACE BOX P.1 Statement of Task This study will bring together defense and defense industry experts in ac- quisition and test and evaluation (T&E); commercial software developers; and software engineers, computer scientists, and other academic researchers to assess the efficacy of the DOD acquisition and T&E processes as specifically applied to information technology. Through briefings, site visits, and committee deliberations, the study committee will: 1. Evaluate legislative requirements for acquisition and T&E and the cur- rent DOD acquisition process (as defined in the “DOD 5000 series”) to determine whether the law and the defined processes permit enough flexibility to rapidly bring capabilities to users; 2. Examine the processes and capabilities of the commercial IT sector to determine whether industry best practices can be adopted by DOD to improve the acquisition, systems engineering, and T&E process; 3. Examine the Department’s various concepts for systems engineering and testing in virtual environments, and make recommendations for how to integrate them into a cohesive, efficient, and robust capability; 4. Examine the DOD acquisition environment, including its institutional and cultural dimensions, for barriers that inhibit program managers/ac- quisition executives from taking advantage of existing flexibility in law and defined processes and recommend solutions; and 5. Make recommendations to responsible agency, executive branch, and legislative officials about how to improve the acquisition, systems en- gineering, and T&E processes to achieve the Department’s net-centric goals. tems acquisition and T&E experts, commercial software developers; and software engineers, computer scientists, and other academic researchers. The committee was tasked with the following: (1) an evaluation of appli - cable legislative requirements, (2) an examination of the processes and capabilities of the commercial IT sector, (3) an examination of the DOD’s concepts for systems engineering and testing in virtual environments, (4) an examination of the DOD acquisition environment, and (5) the for- mulation of recommendations on how to improve the acquisition, systems engineering, and T&E processes to achieve the DOD’s network-centric goals. (The full statement of task appears in Box P.1.) The tasks were com- pleted in November 2009. This report provides the committee’s findings and recommendations, which are based on document reviews, briefings from commercial and military experts in IT systems acquisition, internal deliberations, and the committee members’ personal expertise. Briefings to the committee from staff of the Office of the Secretary

OCR for page R1
ix PREFACE of Defense showed that the acquisition of major automated information systems (MAIS) is especially troublesome. This problem has been broadly recognized for years, and there have been many attempts at reform. Nonetheless, today’s processes for the acquisition and testing of DOD IT systems often last 5 or more years before delivering solutions to the end users. Given the rapid pace of change in the IT world, it is no wonder that solutions ultimately delivered by DOD IT programs are often considered by end users to be inadequate. Much the same could be said about the historical adoption of IT in the commercial sector, where there have been extraordinary successes and colossal failures. Fortunately, the commercial sector has enjoyed some great successes in recent years by employing agile IT acquisition approaches that can also be leveraged by the DOD. In examining the current DOD processes for acquiring IT systems and comparing them with the processes adopted by leading-edge firms in the commercial sector, the committee found stark differences. The DOD is hampered by a culture and acquisition-related practices that favor large programs, high-level oversight, and a very deliberate, serial approach to development and testing (the waterfall model). Programs that are expected to deliver complete, nearly perfect solutions and that take years to develop are the norm in the DOD. In contrast, leading-edge commer- cial firms have adopted agile approaches that focus on delivering smaller increments rapidly and aggregating them over time to meet capabil - ity objectives. Moreover, the DOD’s process-bound, high-level oversight seems to make demands that cause developers to focus more on process than on product, and end-user participation often is too little and too late. These approaches run counter to agile acquisition practices in which the product is the primary focus, end users are engaged early and often, the oversight of incremental product development is delegated to the lowest practical level, and the program management team has the flexibility to adjust the content of the increments in order to meet delivery schedules. The committee concluded that the key to resolving the chronic prob- lems with the DOD acquisition of IT systems is for the DOD to adopt a fundamentally different process—one based on the lessons learned in the employment of agile management techniques in the commercial sector. Agile approaches have allowed their adopters to outstrip estab - lished industrial giants that were beset with ponderous, process-bound, industrial-age management structures. Agile approaches have succeeded because their adopters recognized the issues that contribute to risks in an IT program and changed their management structures and processes to mitigate the risks. There are clear parallels in the DOD that support mak - ing this process change the centerpiece of improving IT acquisition. For the DOD to succeed in adopting new approaches to IT acquisi- tion, the first step is to acknowledge that simply tailoring the existing

OCR for page R1
x PREFACE processes is not sufficient. DOD acquisition regulations do permit tailor- ing, but the committee found few examples of the successful application of the current acquisition regulations to IT programs, and those that were successful required herculean efforts or unique circumstances. Changes broader than tailoring are necessary; they must encompass changes to culture, redefinition of the categories of IT systems, and restructured procurement, development, and testing processes as identified in this report. In the aggregate, these changes must realign processes that today are dominated by deliberate approaches designed for the development of large, complex, hardware-dominated weapons systems to processes adapted to the very different world of software-dominated IT systems. The specific, actionable recommendations made by the committee address the four dimensions of its task discussed above. The body of the report and the appendixes include detailed discussions, rationale, and two proposed new process models for acquiring IT within the DOD. One model is structured for programs focused on the development of new software to provide new functionality or to integrate commercial off-the- shelf (COTS) components (e.g., MAIS programs). The second model is designed for the acquisition of COTS IT hardware, software, or services. Both have parallels in the commercial sector and are especially relevant for acquiring systems that support DOD information enterprise require- ments and operate using the DOD IT infrastructure. The changes are not recommended for adoption in acquiring IT components embedded in weapons systems at this time, but the committee believes that as these changes are refined and institutionalized, many will be applicable to IT components of weapons systems as well. The committee believes that there is an imperative for change, and it strongly urges the DOD to adopt the recommendations offered in this report. Strong support from the highest levels of the DOD will be required to implement changes of the magnitude recommended. The committee extends its thanks to the individuals listed in Appen- dix E who briefed the committee. It also thanks Steven Hutchison, DISA Test and Evaluation Executive, for helping to make this study possible, and Dr. Hutchison and Judith Hill for their assistance throughout the course of the study. Finally, the committee extends its thanks and appre - ciation to Jon Eisenberg, Kevin Lewis, Lynette Millett, and Virginia Bacon Talati of the NRC’s Computer Science and Telecommunications Board whose dedicated support made this report possible. William H. Campbell, Co-Chair Committee on Improving Processes and Policies for the Acquisition and Test of Information Technologies in the Department of Defense

OCR for page R1
Acknowledgment of Reviewers This report has been reviewed in draft form by individuals chosen for their diverse perspectives and technical expertise, in accordance with procedures approved by the National Research Council’s Report Review Committee. The purpose of this independent review is to provide candid and critical comments that will assist the institution in making its pub - lished report as sound as possible and to ensure that the report meets institutional standards for objectivity, evidence, and responsiveness to the study charge. The review comments and draft manuscript remain confidential to protect the integrity of the deliberative process. We wish to thank the following individuals for their review of this report: Eddie Bair, E. Bair Associates, LLC, Calvin Carerra, The Carrera Group, Inc., Felix Dupré, The Durango Group, LLC, Bruce A. Finlayson, University of Washington, Jacques S. Gansler, University of Maryland, Michael F. Goodchild, University of California, Santa Barbara, Richard F. Hilliard II, Independent Consultant, Bar Harbor, Maine, Steven B. Lipner, Microsoft Corporation, Charles E. McQueary, Independent Consultant, Arlington, Virginia, Frank Ostroff, Ostroff Consultants Group, LLC, Stuart H. Starr, National Defense University, John P. Stenbit, TRW, Inc. (retired), Kevin J. Sullivan, University of Virginia, xi

OCR for page R1
xii ACKNOWLEDGMENT OF REVIEWERS Anthony M. Valletta, SRA International, George Wauer, Independent Consultant, Centreville, Virginia, and Peter J. Weinberger, Google, Inc. Although the reviewers listed above have provided many construc- tive comments and suggestions, they were not asked to endorse the con- clusions or recommendations, nor did they see the final draft of the report before its release. The review of this report was overseen by Butler W. Lampson, Microsoft Corporation. Appointed by the National Research Council, he was responsible for making certain that an independent examination of this report was carried out in accordance with institu - tional procedures and that all review comments were carefully consid- ered. Responsibility for the final content of this report rests entirely with the authoring committee and the institution.

OCR for page R1
Contents SUMMARY AND RECOMMENDATIONS 1 1 INTRODUCTION 17 Definitions of the Term “IT System,”17 Effective Approaches to Information Technology in the Commercial Sector, 19 The Defense Acquisition System, 22 Results of Current Acquisition Processes and Practices for Information Technology Systems, 23 Scope and Context of This Report, 27 2 THE ACQUISITION PROCESS AND CULTURE 28 Introduction, 28 Differences Between Information Technology Systems and Weapons Systems Are Not Reflected in Current Process, 30 Requirements Process Impedes Use of Commercial Off-the-Shelf Solutions, 33 Overly Large Information Technology Programs Increase Risk, 34 Funding Process Impedes Flexibility, 35 Excessive Oversight, Yet Insufficient Program Accountability, 36 Cultural Impediments Take Precedence over Rapid Development, 40 Inadequate Information Technology Acquisition Workforce, 42 Legislative Impediments, 44 Measures of Success, 44 xiii

OCR for page R1
xi CONTENTS 3 SYSTEMS AND SOFTWARE ENGINEERING IN DEFENSE 47 INFORMATION TECHNOLOGY ACQUISITION PROGRAMS The Evolution of Department of Defense Policy and Practice for Software Development, 47 Iterative, Incremental Development, 51 Platforms and Virtualization: Key Underpinnings for Information Technology Systems, 60 A Recommended Acquisition Management Approach for Information Technology Programs, 63 Proposed Acquisition Management for SDCI Programs, 66 Proposed Acquisition Management for CHSS Programs, 74 4 ACCEPTANCE AND TESTING 79 Introduction, 79 Shortcomings of Present Defense Test and Evaluation, 80 “Big-R” Requirements and “Small-r” Requirements, 85 Incorporating the Voice of the User, 86 Toward Continuous Operational Assessment, 86 Acceptance Teams, 88 Evaluation Through Operational Use Metrics, 89 Incorporating Common Services Definitions, 90 Virtual Information Technology Test Environments, 92 BIBLIOGRAPHY 97 APPENDIXES A Brief Overview of the Defense Acquisition System for Information Technology 103 B Program Phases and Decision Milestones for SDCI Programs 116 C Program Phases and Decision Milestones for CHSS Programs 123 D Programs That Succeeded with Nontraditional Oversight 127 E Briefings to the Committee 131 F Biosketches of Committee and Staff 134 G Acronyms 147