Achieving Effective Acquisition of Information Technology in the Department of Defense
NATIONAL RESEARCH COUNCIL
OF THE NATIONAL ACADEMIES
THE NATIONAL ACADEMIES PRESS
THE NATIONAL ACADEMIES PRESS
500 Fifth Street, N.W.
Washington, DC 20001
NOTICE: The project that is the subject of this report was approved by the Governing Board of the National Research Council, whose members are drawn from the councils of the National Academy of Sciences, the National Academy of Engineering, and the Institute of Medicine. The members of the committee responsible for the report were chosen for their special competences and with regard for appropriate balance.
This study was supported by Contract No. W911NF-07-C-0115 between the National Academy of Sciences and the Defense Information Systems Agency. Any opinions, findings, conclusions, or recommendations expressed in this publication are those of the author(s) and do not necessarily reflect the views of the organizations or agencies that provided support for the project.
International Standard Book Number-13: 978-0-309-14828-3
International Standard Book Number-10: 0-309-14828-6
Copies of this report are available from the
National Academies Press,
500 Fifth Street, N.W., Lockbox 285, Washington, DC 20055; (800) 624-6242 or (202) 334-3313 (in the Washington metropolitan area); Internet, http://www.nap.edu.
Copyright 2010 by the National Academy of Sciences. All rights reserved.
Printed in the United States of America
THE NATIONAL ACADEMIES
Advisers to the Nation on Science, Engineering, and Medicine
The National Academy of Sciences is a private, nonprofit, self-perpetuating society of distinguished scholars engaged in scientific and engineering research, dedicated to the furtherance of science and technology and to their use for the general welfare. Upon the authority of the charter granted to it by the Congress in 1863, the Academy has a mandate that requires it to advise the federal government on scientific and technical matters. Dr. Ralph J. Cicerone is president of the National Academy of Sciences.
The National Academy of Engineering was established in 1964, under the charter of the National Academy of Sciences, as a parallel organization of outstanding engineers. It is autonomous in its administration and in the selection of its members, sharing with the National Academy of Sciences the responsibility for advising the federal government. The National Academy of Engineering also sponsors engineering programs aimed at meeting national needs, encourages education and research, and recognizes the superior achievements of engineers. Dr. Charles M. Vest is president of the National Academy of Engineering.
The Institute of Medicine was established in 1970 by the National Academy of Sciences to secure the services of eminent members of appropriate professions in the examination of policy matters pertaining to the health of the public. The Institute acts under the responsibility given to the National Academy of Sciences by its congressional charter to be an adviser to the federal government and, upon its own initiative, to identify issues of medical care, research, and education. Dr. Harvey V. Fineberg is president of the Institute of Medicine.
The National Research Council was organized by the National Academy of Sciences in 1916 to associate the broad community of science and technology with the Academy’s purposes of furthering knowledge and advising the federal government. Functioning in accordance with general policies determined by the Academy, the Council has become the principal operating agency of both the National Academy of Sciences and the National Academy of Engineering in providing services to the government, the public, and the scientific and engineering communities. The Council is administered jointly by both Academies and the Institute of Medicine. Dr. Ralph J. Cicerone and Dr. Charles M. Vest are chair and vice chair, respectively, of the National Research Council.
COMMITTEE ON IMPROVING PROCESSES AND POLICIES FOR THE ACQUISITION AND TEST OF INFORMATION TECHNOLOGIES IN THE DEPARTMENT OF DEFENSE
WILLIAM H. CAMPBELL,
BAE Systems, Inc.,
DAWN C. MEYERRIECKS,1
Dawn Meyerriecks, LLC,
ROBERT F. BEHLER,
PHILIP E. COYLE III,
World Security Institute
RENATO A. DiPENTIMA,
SRA International (retired)
JOHN M. GILLIGAN,
Gilligan Group, Inc.
Carnegie Mellon University
PAUL J. KERN (NAE),2
The Cohen Group
H. STEVEN KIMMEL,
Alion Science and Technology
DEIDRE A. LEE,
Professional Services Council
JOSHUA S. LEVINE,
ESP Technologies Corporation
FRANK A. PERRY,
Science Applications International Corporation
The Boeing Company
DANIEL C. STURMAN,
JON EISENBERG, Director,
Computer Science and Telecommunications Board
KEVIN LEWIS, Senior Program Officer
LYNETTE I. MILLETT, Senior Program Officer
RENEE HAWKINS, Financial and Administrative Manager
VIRGINIA BACON TALATI, Program Associate
MORGAN MOTTO, Program Associate (through April 2009)
COMPUTER SCIENCE AND TELECOMMUNICATIONS BOARD
ROBERT F. SPROULL,
Sun Microsystems, Inc.,
Hewlett Packard Company
WILLIAM J. DALLY,
NVIDIA Corporation and Stanford University
University of California, Los Angeles
KEVIN C. KAHN,
JOHN E. KELLY III,
JON M. KLEINBERG,
WILLIAM H. PRESS,
University of Texas, Austin
DAVID E. SHAW,
ALFRED Z. SPECTOR,
Massachusetts Institute of Technology
PETER J. WEINBERGER,
JON EISENBERG, Director
VIRGINIA BACON TALATI, Program Associate
SHENAE BRADLEY, Senior Program Assistant
RENEE HAWKINS, Financial and Administrative Manager
HERBERT S. LIN, Chief Scientist
LYNETTE I. MILLETT, Senior Program Officer
ERIC WHITAKER, Senior Program Assistant
ENITA A. WILLIAMS, Associate Program Officer
For more information on CSTB, see its website at http://www.cstb.org, write to CSTB, National Research Council, 500 Fifth Street, N.W., Washington, DC 20001, call (202) 334-2605, or e-mail the CSTB at email@example.com.
The information technology (IT) revolution of the past several decades has dramatically changed the world. The Internet, Web 2.0 technologies, social networking tools, online search engines, text messaging, video teleconferencing, and multimedia-enabled smart-phones with embedded cameras are but a sample of IT-based capabilities that have altered the ways in which people communicate and work.
In the military, IT has enabled profound advances in weapons systems and the management and operation of the defense enterprise. A significant portion of the Department of Defense (DOD) budget is spent on capabilities acquired as commercial IT commodities, developmental IT systems that support a broad range of warfighting and functional applications, and IT components embedded in weapons systems. The ability of the DOD and its industrial partners to harness and apply IT for warfighting, command and control and communications, logistics, and transportation has contributed enormously to fielding the world’s best defense force.
But despite the DOD’s decades of success in leveraging IT across the defense enterprise, the acquisition of IT systems continues to be burdened with serious problems. Accordingly, the Defense Information Systems Agency (DISA) asked the National Research Council (NRC) to assess the efficacy of the DOD’s acquisition and test and evaluation (T&E) processes as applied to IT. In response, the NRC formed the Committee on Improving Processes and Policies for the Acquisition and Test of Information Technologies in the Department of Defense—a group of IT sys-
Statement of Task
This study will bring together defense and defense industry experts in acquisition and test and evaluation (T&E); commercial software developers; and software engineers, computer scientists, and other academic researchers to assess the efficacy of the DOD acquisition and T&E processes as specifically applied to information technology. Through briefings, site visits, and committee deliberations, the study committee will:
tems acquisition and T&E experts, commercial software developers; and software engineers, computer scientists, and other academic researchers. The committee was tasked with the following: (1) an evaluation of applicable legislative requirements, (2) an examination of the processes and capabilities of the commercial IT sector, (3) an examination of the DOD’s concepts for systems engineering and testing in virtual environments, (4) an examination of the DOD acquisition environment, and (5) the formulation of recommendations on how to improve the acquisition, systems engineering, and T&E processes to achieve the DOD’s network-centric goals. (The full statement of task appears in Box P.1.) The tasks were completed in November 2009. This report provides the committee’s findings and recommendations, which are based on document reviews, briefings from commercial and military experts in IT systems acquisition, internal deliberations, and the committee members’ personal expertise.
Briefings to the committee from staff of the Office of the Secretary
of Defense showed that the acquisition of major automated information systems (MAIS) is especially troublesome. This problem has been broadly recognized for years, and there have been many attempts at reform. Nonetheless, today’s processes for the acquisition and testing of DOD IT systems often last 5 or more years before delivering solutions to the end users. Given the rapid pace of change in the IT world, it is no wonder that solutions ultimately delivered by DOD IT programs are often considered by end users to be inadequate. Much the same could be said about the historical adoption of IT in the commercial sector, where there have been extraordinary successes and colossal failures. Fortunately, the commercial sector has enjoyed some great successes in recent years by employing agile IT acquisition approaches that can also be leveraged by the DOD.
In examining the current DOD processes for acquiring IT systems and comparing them with the processes adopted by leading-edge firms in the commercial sector, the committee found stark differences. The DOD is hampered by a culture and acquisition-related practices that favor large programs, high-level oversight, and a very deliberate, serial approach to development and testing (the waterfall model). Programs that are expected to deliver complete, nearly perfect solutions and that take years to develop are the norm in the DOD. In contrast, leading-edge commercial firms have adopted agile approaches that focus on delivering smaller increments rapidly and aggregating them over time to meet capability objectives. Moreover, the DOD’s process-bound, high-level oversight seems to make demands that cause developers to focus more on process than on product, and end-user participation often is too little and too late. These approaches run counter to agile acquisition practices in which the product is the primary focus, end users are engaged early and often, the oversight of incremental product development is delegated to the lowest practical level, and the program management team has the flexibility to adjust the content of the increments in order to meet delivery schedules.
The committee concluded that the key to resolving the chronic problems with the DOD acquisition of IT systems is for the DOD to adopt a fundamentally different process—one based on the lessons learned in the employment of agile management techniques in the commercial sector. Agile approaches have allowed their adopters to outstrip established industrial giants that were beset with ponderous, process-bound, industrial-age management structures. Agile approaches have succeeded because their adopters recognized the issues that contribute to risks in an IT program and changed their management structures and processes to mitigate the risks. There are clear parallels in the DOD that support making this process change the centerpiece of improving IT acquisition.
For the DOD to succeed in adopting new approaches to IT acquisition, the first step is to acknowledge that simply tailoring the existing
processes is not sufficient. DOD acquisition regulations do permit tailoring, but the committee found few examples of the successful application of the current acquisition regulations to IT programs, and those that were successful required herculean efforts or unique circumstances. Changes broader than tailoring are necessary; they must encompass changes to culture, redefinition of the categories of IT systems, and restructured procurement, development, and testing processes as identified in this report. In the aggregate, these changes must realign processes that today are dominated by deliberate approaches designed for the development of large, complex, hardware-dominated weapons systems to processes adapted to the very different world of software-dominated IT systems.
The specific, actionable recommendations made by the committee address the four dimensions of its task discussed above. The body of the report and the appendixes include detailed discussions, rationale, and two proposed new process models for acquiring IT within the DOD. One model is structured for programs focused on the development of new software to provide new functionality or to integrate commercial off-the-shelf (COTS) components (e.g., MAIS programs). The second model is designed for the acquisition of COTS IT hardware, software, or services. Both have parallels in the commercial sector and are especially relevant for acquiring systems that support DOD information enterprise requirements and operate using the DOD IT infrastructure. The changes are not recommended for adoption in acquiring IT components embedded in weapons systems at this time, but the committee believes that as these changes are refined and institutionalized, many will be applicable to IT components of weapons systems as well.
The committee believes that there is an imperative for change, and it strongly urges the DOD to adopt the recommendations offered in this report. Strong support from the highest levels of the DOD will be required to implement changes of the magnitude recommended.
The committee extends its thanks to the individuals listed in Appendix E who briefed the committee. It also thanks Steven Hutchison, DISA Test and Evaluation Executive, for helping to make this study possible, and Dr. Hutchison and Judith Hill for their assistance throughout the course of the study. Finally, the committee extends its thanks and appreciation to Jon Eisenberg, Kevin Lewis, Lynette Millett, and Virginia Bacon Talati of the NRC’s Computer Science and Telecommunications Board whose dedicated support made this report possible.
William H. Campbell, Co-Chair
Committee on Improving Processes and Policies for the Acquisition and Test of Information Technologies in the Department of Defense
Acknowledgment of Reviewers
This report has been reviewed in draft form by individuals chosen for their diverse perspectives and technical expertise, in accordance with procedures approved by the National Research Council’s Report Review Committee. The purpose of this independent review is to provide candid and critical comments that will assist the institution in making its published report as sound as possible and to ensure that the report meets institutional standards for objectivity, evidence, and responsiveness to the study charge. The review comments and draft manuscript remain confidential to protect the integrity of the deliberative process. We wish to thank the following individuals for their review of this report:
Eddie Bair, E. Bair Associates, LLC,
Calvin Carerra, The Carrera Group, Inc.,
Felix Dupré, The Durango Group, LLC,
Bruce A. Finlayson, University of Washington,
Jacques S. Gansler, University of Maryland,
Michael F. Goodchild, University of California, Santa Barbara,
Richard F. Hilliard II, Independent Consultant, Bar Harbor, Maine,
Steven B. Lipner, Microsoft Corporation,
Charles E. McQueary, Independent Consultant, Arlington, Virginia,
Frank Ostroff, Ostroff Consultants Group, LLC,
Stuart H. Starr, National Defense University,
John P. Stenbit, TRW, Inc. (retired),
Kevin J. Sullivan, University of Virginia,
Anthony M. Valletta, SRA International,
George Wauer, Independent Consultant, Centreville, Virginia, and
Peter J. Weinberger, Google, Inc.
Although the reviewers listed above have provided many constructive comments and suggestions, they were not asked to endorse the conclusions or recommendations, nor did they see the final draft of the report before its release. The review of this report was overseen by Butler W. Lampson, Microsoft Corporation. Appointed by the National Research Council, he was responsible for making certain that an independent examination of this report was carried out in accordance with institutional procedures and that all review comments were carefully considered. Responsibility for the final content of this report rests entirely with the authoring committee and the institution.