Proceedings of a Workshop

Improving Defense Acquisition Workforce Capability in Data Use

Proceedings of a Workshop—in Brief

Data science (the science and technology of extracting value from data) has the potential to improve defense acquisition processes, which includes the full range of activities related to development, procurement, test and evaluation, deployment, and sustainment of materiel to serve military missions and needs. The Department of Defense (DoD) seeks to capitalize on innovations in data science and analytics to increase the efficiency of acquisition programs to meet rapidly evolving mission needs, identify alternative solutions to long-standing acquisition challenges, enable timely deployment of new systems, and ensure cost containment. To move toward more data-driven decision-making within the defense acquisition workforce, DoD seeks to identify necessary data science skills, options for training, and models for building teams with enhanced data capabilities.

The National Academies of Sciences, Engineering, and Medicine’s Board on Mathematical Sciences and Analytics (in cooperation with the Board on Higher Education and Workforce, Board on Army Research and Development, Air Force Studies Board, and Computer Science and Telecommunications Board) was charged by DoD to conduct a consensus study to investigate and recommend ways in which the department can “upskill” its workforce to leverage data science and analytics to conduct materiel acquisition.¹ Specifically, the study committee was tasked with identifying relevant data science skills and capabilities necessary for the acquisitions workforce and developing a framework for training and educating acquisition professionals (see Box 1 for the committee’s full statement of task). To gather insight and information from the community on these issues, the committee convened a virtual workshop on April 14, 2020,² with approximately 300 participants. This document gives a brief overview of the day’s activities, panel specific observations or suggestions from individual speakers in Boxes 2 through 4, and highlights overarching themes summarized by committee member Christine Fox, Johns Hopkins University Applied Physics Laboratory.

___________________

¹ More information on the consensus study can be found at National Academies of Sciences, Engineering, and Medicine, (NASEM), “Improving Defense Acquisition Workforce Capability in Data Use,” https://www.nationalacademies.org/our-work/improving-defense-acquisition-workforce-capability-in-data-use.

² The workshop agenda, presentations, member biographies, and speaker biographies are available at NASEM, “Workshop on Improving Defense Acquisition Workforce Capability in Data Use,” https://www.nationalacademies.org/event/04-14-2020/improving-defense-acquisition-workforce-capability-in-data-use-meeting-5.

LANDSCAPE OF DATA USE IN DEFENSE ACQUISITION AND PATHS FORWARD

Study Co-Chair Lieutenant General Wendy Masiello (USAF, ret.) opened the workshop with remarks to frame the discussions by emphasizing DoD’s long-standing interest in supporting data-driven acquisition decisions. Key capabilities sought by DoD include using data to inform vendor selection, supply-chain analysis, and cost estimation. Masiello and Study CoChair Rebecca Nugent, Carnegie Mellon University, then introduced speakers from DoD who further articulated a vision for data use in defense acquisition.

Mark Krzysko, Principal Deputy Director, Acquisition Policy and Analytics, DoD, described the genesis of the consensus study in the 2018 National Defense Authorization Act (NDAA). Section 913 of the act calls on DoD to “establish a set of activities that use data analysis, measurement, and other evaluation-related methods to improve acquisition program outcomes.” DoD, Krzysko said, has been working over the past few years to address the issue of how to increase access to data across technical, acquisition, and cost-assessment programs and activities, and across the whole of the department’s workforce, including program managers, program executive officers, and senior leaders.

David S. Cadman, Director for Acquisition, Analytics and Policy, Office of the Secretary of Defense (OSD), DoD, expressed DoD’s desire to leverage developments in the commercial and academic sectors while acknowledging the unique challenges posed by DoD regulations. One of DoD’s objectives, Cadman said, is to use data on the performance of past acquisition programs to inform rapid improvements

in acquisition pathways. To achieve this objective, Cadman noted the increasing need to collect and analyze relevant data.

Michael Conlin, Chief Data Officer, DoD, provided the workshop’s keynote address. He stressed that implementing evidence-based acquisition will require a major cultural shift across the department. Conlin noted that the long-standing culture of experience-based decision-making, combined with reliance on data that is months or years old, has served the department well for decades, but observed that senior DoD leaders increasingly expect decisions (e.g., costing and contracting in acquisition) to be based on current and real-time program data⁴ backed by statistically sound analysis. The Secretary and Deputy Secretary of Defense have called for decision-making based on real-time data. This challenge, Conlin continued, involves bringing everyone in DoD to the point where they are capable of and comfortable with using data as a fundamental element of their decision-making. Conlin cited the Foundations for Evidence-Based Policymaking Act, the federal government’s new data strategy,⁵ and the President’s Management Agenda⁶—which emphasizes better use of federal data—as the drivers for this cultural shift.

The Services have also worked to enhance data-based decision-making in acquisition and beyond. Following Conlin’s perspective from DoD, the Chief Data Officers of the Services were invited to share their data strategies.

Thomas Sasala, Chief Data Officer, U.S. Department of the Navy (DON), shared a “placemat” chart (Figure 1) that highlights the lines of effort and the structures of their data strategy, noting that

___________________

³ National Defense Authorization Act for Fiscal Year 2018, Sec. 913, Establishment of Set of Activities that Use Data Analysis, Measurement, and Other Evaluation-Related Methods to Improve Acquisition Program Outcomes. (a) ESTABLISHMENT REQUIRED. David S. Cadmin noted that not later than one year after the date of the enactment of this Act, the Secretary of Defense shall establish a set of activities that use data analysis, measurement, and other evaluation-related methods to improve the acquisition outcomes of the Department of Defense and enhance organizational learning.

⁴ Program data can include, but is not limited to, performance, timeline, and costs.

⁵ “Federal Data Strategy 2020 Action Plan,” released December 2019, https://strategy.data.gov/assets/docs/2020-federal-datastrategy-action-plan.pdf.

⁶ Executive Office of the President, “President’s Management Agenda,” released March 2018, https://www.whitehouse.gov/wp-content/uploads/2018/03/Presidents-Management-Agenda.pdf.

every aspect relates to a business or mission function. The DON data initiative aims to engage in lines of effort to create the “unity of action” in data. Data initiatives will be further detailed in a forthcoming DON Implementation Plan (I-PLAN), which will link to objectives from the 2020 DoD Data Strategy.

Regarding efforts to upskill the DON workforce, Sasala noted that the DON has created partnerships with commercial companies to offer micro-degrees to build up data acumen, and public universities, like Old Dominion University, to offer onsite training. DON is also working to identify strategies to educate senior leadership in data use.

Eileen Vidrine, Chief Data Officer, U.S. Department of the Air Force, noted that the Air Force aims to make data a part of every Airman’s “core DNA.” Ideally, the Air Force wants its data to be visible, accessible, understandable, linked, trustworthy, interoperable, and secure, as seen in Figure 1. In collaboration with the Chief Information Officer (CIO), Deputy CIO, and Deputy Under Secretary, the Air Force has moved to streamline its technical, data management, and business processes. Vidrine commented that the Air Force workforce needs the right data platforms, tools, and skills to advance its missions. She acknowledged that they are addressing training needs around data use, including the establishment of the Air Force Digital University. The Digital University will use what she described as building block-style short courses on various topics, including data use and data science. The Digital University will focus on online content for Airmen who want 24/7 access to training. Additionally, the Air Force is engaging in efforts to offer digital training to senior executive leaders, such as through an Artificial Intelligence Accelerator program with the Massachusetts Institute of Technology to provide background on artificial intelligence (AI) and machine learning.

As data assets increase in the Air Force, Vidrine noted the importance of subject-matter expertise on project teams. She stated that the Data Lab was established to focus on use cases with the potential for enterprise-level impact. Thus far, the laboratory’s successful efforts are attributed to populating case teams with a data scientist, data architect, and subject-matter expert.

Both Sasala and Vidrine agreed that making data available to the highest extent possible based on rules, regulations, and permission provides the workforce with tools to innovate. Vidrine stated that in the future, ease of access to certain data sets and the accuracy of those data sets will support mission-critical use cases and must be prioritized.

USING DATA TO ADVANCE ACQUISITION POLICY AND PRACTICE

Gary Bliss, Institute for Defense Analyses and DoD (retired), described DoD as a collection of functional fields and organizations that have their own charter and value streams. He stated that three necessary conditions for useful data in these systems are (1) human capital capability, (2) the analytic tools available to them, and (3) the availability of data. While there are early- and mid-career professionals with sufficient capabilities to use and analyze data, he continued, the lack of available data—as well as data that is inaccessible outside of individual program offices—is a critical impediment to the use of data science in DoD acquisition. Bliss suggested that the primary challenge with DoD’s data access is that many of DoD’s functional fields have statutory authorities⁷ and institutional prerogatives⁸ that are fixed in a way that makes it difficult for the information technology and analytic community to overcome. Further, these functional fields frequently rely on legacy data systems that prevent data access and interoperability.

___________________

⁷ Title 10 of the U.S. Code authorizes the military services (Army, Navy, Air Force) to organize, train, and equip military forces to include acquisition of military systems.

⁸ These can vary not only across services but within them as different parts of the acquisition systems function with greater and lesser degrees of autonomy. For instance, the Operational Test and Evaluation community is granted functional authority from the Department of Defense while the program offices that oversee acquisition of specific systems fall under the authority of individual services.

Bliss noted that data governance is an important aspect of data availability and transparency. For effective data governance, involved parties should have an understanding of what is being discussed—the meaning of the data made available, agreement on authoritative sources for data, and the interface through which the data are accessed or analyzed.

The Honorable Lisa Disbrow Johns Hopkins University Applied Physics Laboratory and former Under Secretary of the Air Force, spoke to the challenges and opportunities for data use in operational requirements and cost estimation. She noted that too frequently the data used in those activities are insufficient and received in a multitude of diverse formats. According to Disbrow, the evaluation and estimation of total ownership costs of a program from beginning to end⁹—as well as surveying new manufacturing techniques and other comparable capabilities—are ideal opportunities for the use of advanced data capabilities. Advanced data capabilities would also improve the development of combat effectiveness requirements for weapons programs. Data and data analysis, she emphasized, that is visible, adaptable, credible, and repeatable will add speed and certainty to these processes.

Lieutenant General Bruce Litchfield (USAF, ret.), Lockheed Martin Corporation, reflecting on his career in the Air Force, focused on the use and application of data in sustainment.¹⁰ Litchfield discussed how culture change was key to enhancing data use at the Air Force Sustainment Center (AFSC). AFSC was established with the goal of collaborative execution, where all work was aligned to the same data set, responsibilities, objectives, and goals. By gathering information on what data-driven processes were

___________________

⁹ Including development, procurement, military construction (MILCON), operations and sustainment (O&S), training, deployment, sustainment (when possible), and disposal.

¹⁰ The provision of logistics and personnel services required to maintain and prolong operations until successful mission accomplishment.

available or needed, AFSC identified goals, set expectations, and developed strategies to meet those goals. To drive culture change from the top, AFSC developed a senior leader course that includes knowledge on how data-driven processes work and provides the training and skills necessary to support those processes. After implementing changes, AFSC identified how processes were impacted in order to identify where further improvements were needed. In 9 months, AFSC surpassed their cost-savings goal of $1 billion.

Jennifer Bowles, OSD Cost Assessment and Program Evaluation (CAPE), explained that CAPE’s role is to prescribe policies and procedures for reporting and collecting actual cost data for acquisition programs. CAPE hires civilian and military operation research analysts (ORAs) who represent a broad spectrum of analytical disciplines.^11,12 Within the Land and Naval Warfare Cost Analysis Division, she continued, they view ORAs as data scientists, which she believes has contributed to their success. In partnership with military department cost agencies, they created the Cost Assessment Data Enterprise (CADE), which is the primary cost data source for acquisition programs, including contractor and subcontractor information.¹³ That data serves as a reference point for cost estimates, program estimates, and various milestones, which informs negotiations for follow-on contracts. However, Bowles noted that while OSD collects a broad array of data, there is no common understanding or methodology within

___________________

¹¹ Such as engineering, applied math, physical sciences, and who have 24 credits in math/science, with at least 3 credits of calculus.

¹² In light of the COVID-19 pandemic, she noted that they have been called to help with the analysis of all the data being pushed out in order to get the right information to officials.

¹³ This includes labor and material costs and some acquisition costs like SARS and technical/programmatic/capability information, more than 600 active and legacy programs, more than 4,500 prime contracts and 1,800 subcontracts, and 3,000 users (primarily government but some industry).

DoD or industry on how that data should be formatted, what data are important, and how the data should be acquired.

TRAINING A WORKFORCE IN DATA CAPABILITIES

To better understand education and training from defense institutes and universities, Maryann P. Watson, Defense Acquisition University (DAU), and Darryl K. Ahner, Air Force Institute of Technology (AFIT), discussed data science curricula and programs.

Watson indicated that feedback from stakeholders and the thrust of the national defense strategy had prompted efforts to move away from one-size-fits-all training. DAU is now piloting micro-credentials focusing on critical skills that could lead to better acquisition outcomes and include topics such as AI, digital engineering, data science, and data analytics. Watson indicated that DAU recognized a need for acquisition program managers and other individuals to understand where data science and analytics can improve program outcomes, but they do not necessarily need an understanding of the technical details of data science. Realizing that skills necessary for specific roles varies, Watson stated that DAU is considering the adoption of a “chiclet model” for career learning and development (Figure 2), which tailors particular learning programs to the career requirements and the learners’ needs. This process is facilitated by online learning. To avoid creating training from scratch, DAU will incorporate Coursera courses, existing postgraduate school programs, and the Army Logistics University as part of its model.

Ahner explained that his background of supporting major acquisition programs has given him the insight to look into different types of training and education for the workforce. AFIT currently offers training programs based on the national defense strategy and suggestions from leadership (see Figure 3). One of those programs is a continuing education effort that includes short courses offering opportunities to a broad audience to learn the skills needed for their current job. AFIT also has longer-term

programs such as those offering a data analytics certificate or master’s degree. Ahner noted that increasing hiring efforts for data analytics in the Air Force is difficult. Optimally, he suggested, the Air Force can focus efforts to build the data acumen of its current workforce, even among those who are not data analysts.

Seeking to build on these insights from defense agencies and institutions, the committee invited academic and industry professionals to describe other notable data science education and training opportunities, including partnerships between the two sectors.

Paul Nielsen, Optum Technologies, explained that his organization works with enormous amounts of data, which presents challenges in terms of access and workforce capabilities. To tackle this challenge, Optum established a College of AI, which builds their workforce’s AI capabilities, and the UHG Academy of Technology,¹⁴ which recognizes advanced professional skills. Optum has also established a technology leader career path in which engineers are prepared to enter the Academy of Technology. Together, these changes positively impact work culture and how Optum approaches its work.

Sears Merritt, MassMutual, indicated that MassMutual views data science as an interdisciplinary field that uses the scientific method, alongside processes and systems to develop and extract knowledge from data. By building up its data capabilities, MassMutual has been able to deploy data analytics in all major aspects of the company, including policy, revenue, and investments. In educating its workforce, MassMutual identified a dichotomy of information producers and information consumers. The former needs to be well versed in scientific methods and be able to clearly communicate its results, and the latter needs to understand those results, as it relates to their current context, to make better data-driven decisions. MassMutual established partnerships with liberal arts institutions to teach individuals data science methods relevant to the company, which has allowed these institutions to serve as talent pools. Merritt shared that they also partnered with the University of Massachusetts to establish a college of data science and create a master’s degree program. By the end of the process, students hired out of these programs by MassMutual have solid data science fundamentals that they can apply in the context of the business.

Luis Stevens, Target Corporation, emphasized that a large, data-intensive organization like his, which acquires and analyzes roughly a petadata of data every few days, needs to make sure its data workforce is knowledgeable about the full suite of technologies used to manage and analyze data. Such alignment entails ensuring that data scientists and analysts are familiar not only with the higher-level algorithms and programming languages used for analysis but also the lower-level (in the sense of a software stack) tools, such as those used to move data from one system to another—and vice-versa for those who develop and maintain the lower-level tools.

Melvin Greer, Intel Corporation Americas, shared that Intel shifted their focus from a company built around developing physical, silicon architecture to becoming a data focused company. In support of this goal, Intel invested approximately $13 billion in funding research at over 100 universities globally and $4 billion in the acquisition and merger of 27 AI companies to increase its workforce capabilities. Engineering capabilities have been refocused on how to develop and deliver world-class data science capabilities. Now, every employee at Intel needs a basic understanding of how data and data science works, regardless of their department or role. In fact, he continued, those who need data acumen at Intel are those whose primary job is something other than data science. Every person on the sales team, for example, received a 2- to 3-day training about how data drives decision-making for the development

___________________

¹⁴ Nielsen stated the UHG Academy of Technology was modeled after the IBM Academy of Technology.

of revenue. Intel has also created a new curriculum for those on the sales team who want to grow in their capabilities and learn about algorithmic manipulation or algorithmic warfare and how data science applies to particular domains.

Jaeki Song, Texas Technology University, noted that in 2015, the university started offering an online and in-person master’s degree in data science. It is a 1-year intensive program, but students who work full-time can extend to 2 years online. The program requirements include five courses on databases, data science project management, scripting language, big data management, and decision theory. While the students take classes in the data science program, they participate in various activities, including a capstone symposium, industry office hours, and corporate trips. Song stated that they are in the process of creating a capstone symposium with some companies who will be their strategic partners. The companies provide students access to real data to develop a project and present them to those companies. Song noted that adopting a data science culture should be initiated by industry supervisors by creating integrated and diverse teams where members are comfortable sharing their best ideas.

Matthew Rattigan, University of Massachusetts, Amherst (UMass Amherst), noted that many industries are adopting data-driven decision-making, and some more quickly than others. As a result, UMass Amherst offers a summer program that matches data science undergraduate and graduate students to organizations in order to work on external projects, such as the Appalachian Mountain Club to track the carbon footprint of different lodges. Rattigan stressed that different depths of data capabilities are needed across the workforce. However, utilizing the scientific method and knowing how to ask questions is key. Specific expertise will allow people to interpret the data and identify any areas that could have potential biases or identify where there may be undesirable feedback loops.

REFLECTIONS

Giving final remarks for the workshop, Fox reiterated Krzysko’s statement that DoD is not currently a leader in using data science and analytics, but it needs to be. This need, she noted, stems partly from Congress and legislation, but also reflects a world-wide trend: “Organizations globally recognize the importance and value of data” to their success. To become a leader, Fox summarized, DoD will require new policies, changes in culture, and an upskilled workforce.

Fox identified several areas where speakers had noted positive policy developments, such as “legislation, requiring evidence-based decision for acquisitions . . . a pretty clear policy signal.” Inside DoD, Fox noted, senior leaders are “encouraging clear and open conversations about data quality and statistical approaches . . . at the highest level” and new positions for chief data officers have been created—both signs of positive policy movement, she suggested.

She reiterated that several speakers had noted that they were working to set the right culture, as “we heard from many speakers that the acceptance of the value and the use of data analytics is not a given.” According to Fox, there remains much to learn in how best to do this. Fox highlighted insights from speakers representing industry who had shared experiences in transforming culture in large corporate environments, but noted that the lessons of industry may require some “translation” to be applicable to the unique cultures, policies, and needs of DoD.

Fox recognized that speakers were not aligned in their view of who needs to be trained, but all speakers agreed that senior leaders “need a deep appreciation of the capabilities of and limitations of data and data analytics.” She noted that “DoD leadership expressed the view that everyone needs a basic understanding of statistics and data visualization; some industry approaches agreed with this as well.” Fox noted that speakers had expressed the view that young people entering the Services will ex-

pect access to data analytics tools and decision-making grounded in data, as well as training in how best to use those tools in the DoD environment. Fox recalled a comment from Vidrine on this point: “Young people in the military are doing amazing things with Excel—it is like giving a plumber a hammer and seeing them doing amazing things with the hammer—just think what they would be able to do with a wrench.”

Finally, Fox highlighted that although speakers’ perspectives may vary, they shared the view that better use of data and data analytics could significantly enhance DoD’s acquisition programs. However, Fox noted several important challenges, including a broad need to establish and support “authoritative data sets—those that are correct . . . and that are useful for decisions and that are accepted to underlie the decisions at issue.” Ultimately, Fox noted that “everyone agreed, throughout the day, that data and data analytics are needed and powerful. No one said that incorporating data analytics into DoD acquisitions processes was not worth pursuing so, even though it is hard, our speakers . . . recommended pushing forward.”

For additional information about the study, visit https://www.nationalacademies.org/our-work/improving-defenseacquisition-workforce-capability-in-data-use.

DISCLAIMER: This Proceedings of a Workshop—in Brief was prepared by Katiria Ortiz as a factual summary of what occurred at the workshop. The statements made are those of the rapporteurs or individual workshop participants and do not necessarily represent the views of all workshop participants; the planning committee; or the National Academies of Sciences, Engineering, and Medicine.

PLANNING COMMITTEE: The National Academies of Sciences, Engineering, and Medicine’s planning committees are solely responsible for organizing the workshop, identifying topics, and choosing speakers. The responsibility for the published Proceedings of a Workshop—in Brief rests with the rapporteurs and the institution. The planning committee comprises Wendy Masiello, Co-Chair, Wendy Mas Consulting, LLC; Rebecca Nugent, Co-Chair, Carnegie Mellon University; Philip S. Anton, RAND Corporation; Trilce Estrada, University of New Mexico; Stephen R. Forrest, University of Michigan; Christine H. Fox, Johns Hopkins University Applied Physics Laboratory; Melvin Greer, Intel Corporation; Charles Isbell, Georgia Institute of Technology; Peter Levine, Institute for Defense Analyses; Ann F. McKenna, Arizona State University; Alyson G. Wilson, North Carolina State University; and Jun Zhuang, University at Buffalo.

REVIEWERS: To ensure that it meets institutional standards for quality and objectivity, this Proceedings of a Workshop—in Brief was reviewed by Fred Oswald, Rice University, and Rebecca Weirick, U.S. Army Contracting Command—Redstone.

STAFF: Tyler Kloefkorn, Carl-Gustav Anderson, Michelle Schwalbe, and Selam Araia, Board on Mathematical Sciences and Analytics, Division on Engineering and Physical Sciences (DEPS); Lida Beninson, Policy and Global Affairs; Steven Darbes, Board on Army Research and Development, DEPS; Ellen Chou, Air Force Studies Board, DEPS; Jon Eisenberg and Katiria Ortiz, Computer Science and Telecommunications Board, DEPS.

SPONSORS: This workshop was sponsored by the U.S. Department of Defense.

SUGGESTED CITATION: National Academies of Sciences, Engineering, and Medicine. 2020. Improving Defense Acquisition Workforce Capability: Proceedings of a Workshop—in Brief. Washington, DC: The National Academies Press. https://doi.org/10.17226/25922.

Division on Engineering and Physical Sciences

Copyright 2020 by the National Academy of Sciences. All rights reserved.