EMPOWERING THE
Defense Acquisition Workforce
TO IMPROVE MISSION OUTCOMES
USING DATA SCIENCE
Committee on Improving Defense Acquisition
Workforce Capability in Data Use
Board on Mathematical Sciences and Analytics
Committee on Applied and Theoretical Statistics
Air Force Studies Board
Computer Science and Telecommunications Board
Division on Engineering and Physical Sciences
Board on Higher Education and Workforce
Policy and Global Affairs
Committee on National Statistics
Division of Behavioral and Social Sciences and Education
A Consensus Study Report of
THE NATIONAL ACADEMIES PRESS
Washington, DC
www.nap.edu
THE NATIONAL ACADEMIES PRESS 500 Fifth Street, NW Washington, DC 20001
This activity was supported by contract FA8650-19-F-9301 between the National Academy of Sciences and the Department of Defense. Any opinions, findings, conclusions, or recommendations expressed in this publication do not necessarily reflect the views of any organization or agency that provided support for the project.
International Standard Book Number-13: 978-0-309-68493-4
International Standard Book Number-10: 0-309-68493-5
Digital Object Identifier: https://doi.org/10.17226/25979
This publication is available from the National Academies Press, 500 Fifth Street, NW, Keck 360, Washington, DC 20001; (800) 624-6242 or (202) 334-3313; http://www.nap.edu.
Copyright 2021 by the National Academy of Sciences. All rights reserved.
Printed in the United States of America
Suggested citation: National Academies of Sciences, Engineering, and Medicine. 2021. Empowering the Defense Acquisition Workforce to Improve Mission Outcomes Using Data Science. Washington, DC: The National Academies Press. https://doi.org/10.17226/25979.
The National Academy of Sciences was established in 1863 by an Act of Congress, signed by President Lincoln, as a private, nongovernmental institution to advise the nation on issues related to science and technology. Members are elected by their peers for outstanding contributions to research. Dr. Marcia McNutt is president.
The National Academy of Engineering was established in 1964 under the charter of the National Academy of Sciences to bring the practices of engineering to advising the nation. Members are elected by their peers for extraordinary contributions to engineering. Dr. John L. Anderson is president.
The National Academy of Medicine (formerly the Institute of Medicine) was established in 1970 under the charter of the National Academy of Sciences to advise the nation on medical and health issues. Members are elected by their peers for distinguished contributions to medicine and health. Dr. Victor J. Dzau is president.
The three Academies work together as the National Academies of Sciences, Engineering, and Medicine to provide independent, objective analysis and advice to the nation and conduct other activities to solve complex problems and inform public policy decisions. The National Academies also encourage education and research, recognize outstanding contributions to knowledge, and increase public understanding in matters of science, engineering, and medicine.
Learn more about the National Academies of Sciences, Engineering, and Medicine at www.nationalacademies.org.
Consensus Study Reports published by the National Academies of Sciences, Engineering, and Medicine document the evidence-based consensus on the study’s statement of task by an authoring committee of experts. Reports typically include findings, conclusions, and recommendations based on information gathered by the committee and the committee’s deliberations. Each report has been subjected to a rigorous and independent peer-review process and it represents the position of the National Academies on the statement of task.
Proceedings published by the National Academies of Sciences, Engineering, and Medicine chronicle the presentations and discussions at a workshop, symposium, or other event convened by the National Academies. The statements and opinions contained in proceedings are those of the participants and are not endorsed by other participants, the planning committee, or the National Academies.
For information about other products and activities of the National Academies, please visit www.nationalacademies.org/about/whatwedo.
COMMITTEE ON IMPROVING DEFENSE ACQUISITION WORKFORCE CAPABILITY IN DATA USE
WENDY M. MASIELLO, U.S. Air Force (retired), Co-Chair
REBECCA NUGENT, Carnegie Mellon University, Co-Chair
PHILIP S. ANTON, Stevens Institute of Technology
TRILCE ESTRADA, University of New Mexico
MILLARD FIREBAUGH, University of Maryland (until December 2019)
STEPHEN FORREST, NAE,1 University of Michigan
CHRISTINE FOX, Johns Hopkins University Applied Physics Laboratory
MELVIN GREER, Intel Corporation
CHARLES ISBELL, Georgia Institute of Technology
PETER LEVINE, Institute for Defense Analyses
ANN McKENNA, Arizona State University
ALYSON WILSON, North Carolina State University
JUN ZHUANG, University at Buffalo
Staff
LIDA BENINSON, Senior Program Officer, Study Co-Director
TYLER KLOEFKORN, Senior Program Officer, Study Co-Director
CARL-GUSTAV ANDERSON, Program Officer
STEVEN DARBES, Program Officer
SELAM ARAIA, Senior Program Assistant
ADRIANNA HARGROVE, Finance Business Partner (until December 2020)
HEATHER LOZOWSKI, Senior Finance Business Partner
BEN WATZAK, Research Assistant
MICHELLE K. SCHWALBE, Director, Board on Mathematical Sciences and Analytics
___________________
1Member, National Academy of Engineering.
BOARD ON MATHEMATICAL SCIENCES AND ANALYTICS
Division on Engineering and Physical Sciences
MARK L. GREEN, University of California, Los Angeles, Chair
HÉLÈNE BARCELO, Mathematical Sciences Research Institute
RUSSEL E. CAFLISCH, NAS,1 New York University
DAVID S.C. CHU, Institute for Defense Analyses
RONALD R. COIFMAN, NAS, Yale University
JAMES (JIM) CURRY, University of Colorado, Boulder
RONALD D. FRICKER, JR., Virginia Polytechnic Institute and State University
SHAWNDRA HILL, Facebook
TRACHETTE JACKSON, University of Michigan
LYDIA KAVRAKI, NAM,2 Rice University
TAMARA KOLDA, NAE,3 Sandia National Laboratories
PETER KOUMOUTSAKOS, ETH Zurich
RACHEL KUSKE, Georgia Institute of Technology
JILL PIPHER, Brown University
YORAM SINGER, WorldQuant
TATIANA TORO, University of Washington
LANCE WALLER, Emory University
KAREN E. WILLCOX, University of Texas, Austin
Staff
MICHELLE K. SCHWALBE, Director
CARL-GUSTAV ANDERSON, Program Officer
SELAM ARAIA, Senior Program Assistant
LINDA CASOLA, Associate Program Officer (until December 2019)
ADRIANNA HARGROVE, Finance Business Partner (until December 2020)
TYLER KLOEFKORN, Senior Program Officer
HEATHER LOZOWSKI, Senior Finance Business Partner
BEN WATZAK, Research Assistant (until September 2020)
___________________
1Member, National Academy of Sciences.
2Member, National Academy of Medicine.
3Member, National Academy of Engineering.
COMMITTEE ON APPLIED AND THEORETICAL STATISTICS
Division on Engineering and Physical Sciences
NICHOLAS J. HORTON, Amherst College, Co-Chair
LANCE WALLER, Emory University, Co-Chair
F. DUBOIS BOWMAN, University of Michigan
ALICIA CARRIQUIRY, NAM,1 Iowa State University
RONG CHEN, Rutgers University, The State University of New Jersey
WEI CHEN, NAE,2 Northwestern University
MICHAEL J. DANIELS, University of Florida
AMY H. HERRING, Duke University
TIM HESTERBERG, Google, Inc.
REBECCA A. HUBBARD, University of Pennsylvania
KRISTIN LAUTER, Microsoft Research
DAVID MADIGAN, Columbia University
XIAO-LI MENG, Harvard University
RAQUEL PRADO, University of California, Santa Cruz
NANCY M. REID, NAS,3 University of Toronto
CYNTHIA RUDIN, Duke University
AARTI SINGH, Carnegie Mellon University
ELIZABETH A. STUART, Johns Hopkins University
JONATHAN WAKEFIELD, University of Washington
ALYSON G. WILSON, North Carolina State University
Staff
TYLER KLOEFKORN, Director
CARL-GUSTAV ANDERSON, Program Officer
SELAM ARAIA, Senior Program Assistant
LINDA CASOLA, Associate Program Officer (until December 2019)
ADRIANNA HARGROVE, Finance Business Partner (until December 2020)
HEATHER LOZOWSKI, Senior Finance Business Partner
BEN WATZAK, Research Assistant (until September 2020)
MICHELLE K. SCHWALBE, Director, Board on Mathematical Sciences and Analytics
___________________
1Member, National Academy of Medicine.
2Member, National Academy of Engineering.
3Member, National Academy of Sciences.
AIR FORCE STUDIES BOARD
Division on Engineering and Physical Sciences
ELLEN M. PAWLIKOWSKI, NAE,1 U.S. Air Force (retired), Chair
KEVIN G. BOWCUTT, NAE, The Boeing Company
CLAUDE R. CANIZARES, NAS,2 Massachusetts Institute of Technology
MARK COSTELLO, Georgia Institute of Technology
WESLEY HARRIS, NAE, Massachusetts Institute of Technology
JAMES E. HUBBARD JR., NAE, Texas A&M University
LESTER L. LYLES, U.S. Air Force (retired)
WENDY M. MASIELLO, U.S. Air Force (retired)
LESLIE ANN MOMODA, HRL Laboratories, LLC
OZDEN OCHOA, Texas A&M University
HON F. WHITTEN PETERS, Williams & Connolly, LLP
HENDRICK RUCK, Edaptive Computing, Inc.
JULIE J.C.H. RYAN, Wyndrose Technical Group
MICHAEL D. SCHNEIDER, Lawrence Livermore National Laboratory
GRANT H. STOKES, MIT Lincoln Laboratory
Staff
ELLEN CHOU, Director
GEORGE COYLE, Senior Program Officer
RYAN MURPHY, Program Officer
ADRIANNA HARGROVE, Finance Business Partner
MARGUERITE SCHNEIDER, Administrative Coordinator
EVAN ELWELL, Research Assistant
___________________
1Member, National Academy of Engineering.
2Member, National Academy of Sciences.
COMPUTER SCIENCE AND TELECOMMUNICATIONS BOARD
Division on Engineering and Physical Sciences
LAURA HAAS, NAE,1 University of Massachusetts, Amherst, Chair
DAVID E. CULLER, NAE, University of California, Berkeley
ERIC HORVITZ, NAE, Microsoft Research
CHARLES ISBELL, Georgia Institute of Technology
ELIZABETH MYNATT, Georgia Institute of Technology
CRAIG PARTRIDGE, Colorado State University
DANIELA RUS, NAE, Massachusetts Institute of Technology
FRED B. SCHNEIDER, NAE, Cornell University
MARGO I. SELTZER, NAE, University of British Columbia
NAMBIRAJAN SESHADRI, NAE, University of California, San Diego
MOSHE Y. VARDI, NAS2/NAE, Rice University
Staff
JON EISENBERG, Senior Board Director
SHENAE BRADLEY, Administrative Assistant
RENEE HAWKINS, Financial and Administrative Manager
LYNETTE I. MILLETT, Associate Director
KATIRIA ORTIZ, Associate Program Officer
BRENDAN ROACH, Program Officer
___________________
1Member, National Academy of Engineering.
2Member, National Academy of Sciences.
BOARD ON HIGHER EDUCATION AND WORKFORCE
Policy and Global Affairs
KUMBLE R. SUBBASWAMY, University of Massachusetts, Amherst, Chair
ANGELA BYARS-WINSTON, University of Wisconsin, Madison
JAIME CURTIS-FISK, The Dow Chemical Company
MARIELENA DESANCTIS, Broward College
APRILLE J. ERICSSON, NASA Goddard Space Flight Center
JOAN FERRINI-MUNDY, University of Maine
GABRIELA GONZALEZ, NAS,1 Louisiana State University
TASHA R. INNISS, Spelman College
SALLY K. MASON, University of Iowa
DOUGLAS S. MASSEY, NAS, Princeton University
RICHARD K. MILLER, NAE,2 Olin College of Engineering
PATRICIA SILVEYRA, University of North Carolina at Chapel Hill
KATE STOLL, Massachusetts Institute of Technology Washington Office
MEGHNA TARE, University of Texas
MARY WOOLLEY, NAM,3 President and CEO, Research! America
Staff
LEIGH MILES JACKSON, Acting Director
THOMAS RUDIN, Director (until November 2020)
AUSTEN APPLEGATE, Senior Program Assistant
ARIELLE BAKER, Associate Program Officer
ASHLEY BEAR, Senior Program Officer
LIDA BENINSON, Senior Program Officer
FRAZIER BENYA, Senior Program Officer
IMANI BRAXTON-ALLEN, Senior Program Assistant (until April 2021)
MARIA LUND DAHLBERG, Senior Program Officer
ALEX HELMAN, Program Officer
REBEKAH HUTTON, Program Officer
PRIYANKA NALAMADA, Associate Program Officer
LAYNE A. SCHERER, Senior Program Officer
JOHN VERAS, Senior Program Assistant
MARQUITA WHITING, Senior Program Assistant
___________________
1Member, National Academy of Sciences.
2Member, National Academy of Engineering.
3Member, National Academy of Medicine.
COMMITTEE ON NATIONAL STATISTICS
Division of Behavioral and Social Sciences and Education
ROBERT M. GROVES, NAS1/NAM,2 Georgetown University, Chair
LAWRENCE D. BOBO, NAS, Harvard University
ANNE C. CASE, NAS/NAM, Princeton University
MICK P. COUPER, University of Michigan
JANET CURRIE, NAS/NAM, Princeton University
DIANA FARRELL, JPMorgan Chase Institute, Washington, DC
ROBERT GOERGE, University of Chicago
ERICA L. GROSHEN, Cornell University
HILARY HOYNES, University of California, Berkeley
DANIEL KIFER, Pennsylvania State University
SHARON LOHR, Arizona State University
JEROME P. REITER, Duke University
JUDITH A. SELTZER, University of California, Los Angeles
C. MATTHEW SNIPP, Stanford University
ELIZABETH A. STEWART, Johns Hopkins Bloomberg School of Public Health
JEANETTE M. WING, Columbia University
Staff
BRIAN HARRIS-KOJETIN, Director
TARA BECKER, Program Officer
CONSTANCE F. CITRO, Senior Scholar
MICHAEL L. COHEN, Senior Program Officer
DANIEL L. CORK, Senior Program Officer
STUART ELLIOTT, Scholar
MARY GHITELMAN, Senior Program Assistant
NANCY J. KIRKENDALL, Senior Program Officer
REBECCA KRONE, Program Coordinator
CHRISTOPHER D. MACKIE, Senior Program Officer
MALAY MAJMUNDAR, Senior Program Officer
ANTHONY S. MANN, Program Associate
KRISZTINA MARTON, Senior Program Officer
MICHAEL J. SIRI, Associate Program Officer
JORDYN WHITE, Program Officer
___________________
1Member, National Academy of Sciences.
2Member, National Academy of Medicine.
This page intentionally left blank.
Preface
The ability to use analytical tools to collect and analyze data is rapidly expanding and bringing along new challenges and opportunities. Following the National Academies of Sciences, Engineering, and Medicine report on Data Science for Undergraduates: Opportunities and Options and Roundtable on Data Science Postsecondary Education—and in recognition of the rapidly evolving premium on data and its power—the Department of Defense (DoD) requested this study to determine how it could prepare its acquisition workforce to embrace and exploit the rapidly evolving promise of data science.
To carry out the task, the National Academies appointed a committee of 13 members. The committee was extraordinary in its makeup, with experts in defense acquisition processes, policies, analyses, and workforce management; leading industry data scientists; and academic experts in engineering, mathematics, statistics, computer science, data science, and education.
Over the course of the study, the committee came to appreciate the complexity and diversity of the DoD acquisition and personnel systems and processes, as well as the multitude of ways industry and government counterparts have addressed the upskilling of their workforces. In the end, the committee concluded that embracing data science and the possibilities for insight at the acquisition level must be driven by leadership. It will take time to upskill the entire acquisition workforce, but the tools, training, and supplemental skills are available today. Deliberate commitment, resources, and clear priorities across DoD will be required to make it happen.
We thank all of the members of the committee for their intensive effort and collaborative spirit in crafting this report. We were aided by remark-
ably talented co-study directors, Lida Beninson and Tyler Kloefkorn, and an able group of staff, including Carl-Gustav Anderson, Selam Araia, Steven Darbes, Adrianna Hargrove, Heather Lozokowski, Katiria Ortiz, Michelle Schwalbe, and Ben Watzak. It was our pleasure and honor to be a part of this study.
We look forward to seeing the DoD acquisition community harness data to address today’s challenges and opportunities as well those on the horizon, both in the acquisition process and in the products that system delivers.
Wendy M. Masiello and Rebecca Nugent, Co-Chairs
Committee on Improving Defense Acquisition
Workforce Capability in Data Use
Acknowledgments
The committee would like to thank the following individuals for providing input to this study.
Bess Dopkeen, U.S. House Committee on Armed Services
Eric Eckstrand, Naval Postgraduate School
Tammy Foster, Lockheed Martin
Michael Gilmore, Institute for Defense Analyses
Vance Gilstrap, Defense Acquisition University
James Greene, Naval Postgraduate School
Mary Hauber, Air Force Contracting Center
Pete Herrmann, Air Force Contracting Center
Hans Jerrell, Defense Acquisition University
Sallie Ann Keller, University of Virginia
Mark Krzysko, Department of Defense
Bruce Litchfield, Lockheed Martin
Megan McKernan, RAND Corporation
Sears Merritt, MassMutual
Robert Mortlock, Naval Postgraduate School
William Muir, Naval Postgraduate School
Paul Nielsen, Optum Technologies
Walter Owen, Naval Postgraduate School
Sezin Palmer, Johns Hopkins University Applied Physics Laboratory
William Parker, Defense Acquisition University
Renee Pasman, Lockheed Martin
Matthew Rattigan, University of Massachusetts Amherst
Rene Rendon, Naval Postgraduate School
David Robinson, Defense Acquisition University
Thomas Sasala, U.S. Navy
Clyde Scandrett, Naval Postgraduate School
James Scrofani, Naval Postgraduate School
Garry Shafovaloff, Human Capitals Initiative
Heidi Shyu, VK Integrated Systems
Jaeki Song, Texas Tech University
Nancy Spruill, Department of Defense (ret.)
Luis Stevens, Target Corporation
Carol Tisone, Defense Acquisition University
Darlene Urquhart, Defense Acquisition University
Eileen Vidrine, U.S. Air Force
Maryann Watson, Defense Acquisition University
Rebecca Weirick, Office of the Deputy Assistant Secretary of the Army
Roger Westermeyer, Air Force Contracting Center
Acknowledgment of Reviewers
This Consensus Study Report was reviewed in draft form by individuals chosen for their diverse perspectives and technical expertise. The purpose of this independent review is to provide candid and critical comments that will assist the National Academies of Sciences, Engineering, and Medicine in making each published report as sound as possible and to ensure that it meets the institutional standards for quality, objectivity, evidence, and responsiveness to the study charge. The review comments and draft manuscript remain confidential to protect the integrity of the deliberative process.
We thank the following individuals for their review of this report:
Although the reviewers listed above provided many constructive comments and suggestions, they were not asked to endorse the conclusions or recommendations of this report nor did they see the final draft before its release. The review of this report was overseen by Stephen M. Robinson, NAE, University of Wisconsin, Madison. He was responsible for making certain that an independent examination of this report was carried out in accordance with the standards of the National Academies and that all review comments were carefully considered. Responsibility for the final content rests entirely with the authoring committee and the National Academies.
Contents
Harnessing Data for Defense Acquisition
Leadership Driving Change in a Data-Centric Organization
Study Origin, Objectives, and Approach
2 DEFENSE ACQUISITION PROCESS, DATA, AND WORKFORCE: THE SHORT VERSION
Acquisition Data Use and Context
3 DATA SCIENCE AND THE DATA LIFE CYCLE: THE SHORT VERSION
What Is Data Science and Who Does It?
Data Science Is Collaborative and Cyclical
The Data Life Cycle and Its Phases
4 DATA SCIENCE IN DOD ACQUISITION
Opportunities for Improved Data Use in Defense Acquisition
Defense Acquisition Functions and the Data Life Cycle
5 DATA LIFE CYCLE MINDSET, SKILLSET, TOOLSET: ROLES AND TEAMS
Team Structures for Data Science
6 PREPARING AND SUSTAINING A DATA-CAPABLE DEFENSE ACQUISITION WORKFORCE
Overview of Training Approaches in Data Science and Analytics
Data Use and Analysis Training Opportunities for Defense Acquisition Personnel—Current Programs
Industry and Government Training Approaches in Data Science and Analytics
Envisioned Future for Training in Data Use Capabilities for Defense Acquisition Personnel
7 FINDINGS, CONCLUSIONS, AND RECOMMENDATIONS
Data Science in DoD Acquisition
Data Life Cycle Mindset, Skillset, Toolset: Roles and Teams
Preparing and Sustaining a Data-Capable Workforce
A Meeting and Workshop Agendas
C Data Science Case Studies in Defense Acquisition
D Skills for Data Science Mastery
