4

Data Science in DoD Acquisition

The use of data to support decision making is not new to the acquisition community and has facilitated critical program support and even informed acquisition strategies for decades—see Appendix C. However, an increasing emphasis on data collection and management, together with use of data tools and associated visualization and analysis techniques, offer the Department of Defense (DoD) an opportunity to create, tap into existing, enhance, or even institutionalize what might have been an ad hoc process into a cohesive, consistent data strategy across acquisition.

This chapter discusses several instances where the application of improved data use could augment the success of the acquisition programs, followed by a brief reminder of the breadth of acquisition functions engaged in data-informed decision-making. Finally, for several of these acquisition functions, this chapter explores data-informed decision opportunities and identifies accompanying phases in the data life cycle needed to enable or enhance those decisions.

OPPORTUNITIES FOR IMPROVED DATA USE IN DEFENSE ACQUISITION

Frank Kendall, the Under Secretary of Defense for Acquisition, Technology and Logistics from 2012 to 2017, was an avid data-informed decision maker. In fact, one of his Better Buying Power principles was that “Data should drive policy”; emphasizing this point was a sign posted outside his office door that read, “In God We Trust; All Others Must Bring Data” (Kendall 2016).

While defense acquisition data is collected in repositories such as the Office of the Secretary of Defense (OSD) Cost Assessment and Program Evaluation’s (CAPE’s) Cost Assessment Data Enterprise (CADE) system and reports such as Selected Acquisition Reports and the Defense Contract Management Agency’s Program Assessment Reports have been available as data analytics resources for many years, the committee sought insights regarding access to defense acquisition data in testimony from previous leaders in the DoD acquisition community. In each instance, missed opportunities were identified for improved use arising from a lack of enterprise-wide data collecting, managing, and sharing practices.

Heidi Shyu, a former Assistant Secretary of the Army for Acquisition, Logistics, and Technology (ASA(ALT)), noted several areas where improved data science and data analytics might have been applied to enhance DoD acquisition programs and processes. Ms. Shyu acknowledged that data and information were available to inform her decisions, but she often found that the data were dispersed across the organization such that collection and analysis would be a tedious months-long process when data-informed decisions were needed daily. Her assessment was that DoD lacks the capability to easily share data across program offices, decreasing visibility across programs and enabling perpetuation of “siloed” programs.

She noted that a common data sharing infrastructure and common dashboard for accessing said data is an opportunity for enhancing DoD acquisition programs. Ms. Shyu cited a number of examples where centralized defense data would have enhanced her ability to make better data-informed decisions for the Army. These opportunities include:

• Budget management by managing multi-year budgets across program offices;
• Program management by enhancing cost, schedule, and performance management, risk assessments, milestone delivery, and program reviews;
• Supply-chain analysis and maintenance by identifying critical companies and potential vulnerabilities;
• Business intelligence by monitoring companies progress and financials;
• Personnel management by tracking personnel roles and skillsets, including past programs of personnel and performance metrics and expertise management; and
• Data management by interfacing across multiple disparate databases with substantial legacy data to maintain and process.

Dr. J. Michael Gilmore, a former Director of Operational Test and Evaluation (DOT&E) and a former Assistant Director for National Security

at the Congressional Budget Office (CBO), spoke with the committee on the challenges of managing data in acquisition, incentivizing the acquisition community to make better use of data science, and the need for leadership. Dr. Gilmore reinforced the committee’s findings regarding common data challenges with the following observations: Data are not monolithic, and there are challenges facing collection and collation of data, including costs, access, and lack of a central archive to house the data.

Even when leadership has created a culture where data are well collected, well curated, centrally available, shared, and analyzed, the committee found that decisions need to be data-informed, not blindly data-driven. For example, a report on Assessing Department of Defense Use of Data Analytics and Enabling Data Management to Improve Acquisition Outcomes by Anton et al. (2019) assessed several large acquisitions programs that had overrun their cost targets. In some cases, DoD had determined that decision makers had failed to act on available data information (DoD 2016b, p. 28). Further analysis, however, identified other considerations (such as pressing military needs, risks, budgets, and political considerations) that played into the acquisition decision process. While acquisition decisions involve contextual considerations, the quality of the data and associated analyses that reach the decision maker is still an important input to acquisition decision making. The better the data and the more competent the workforce, the better the analyses and the better informed the decision maker will be.

DEFENSE ACQUISITION FUNCTIONS AND THE DATA LIFE CYCLE

One of the challenges in identifying opportunities for applying data science to defense acquisition is that the applications must be deeply embedded in the acquisition functions and processes—and rightly so (Anton et al. 2019). Thus, it is useful to have at least a top-level understanding of the range of acquisition functions. Box 4.1 provides a list of those functions, showing the multi-faceted activities involved in defense acquisition (Anton et al. 2019). Further details on how the range of acquisition functions and processes align with analyses in support of decisions are illustrated in Figure B.5 of Appendix B.

By combining the list in Box 4.1 with the data life cycle depicted in Figure 3.5, one can begin to see potential for use of data science on specific acquisition functions; below, the committee describes several opportunities for improvement.

Contracting and Supply-Chain Cost

Government contracting officers are obligated to ensure that the government is paying a fair and reasonable price, often using cost insights to establish the final contract price. Supply chain costs are built into the prime contractor’s price, and the supply chain feeds the development, production and sustainment pipeline for a system’s lifetime. Often, each contract is siloed within each military service. The contracting officer (in conjunction

with the program manager, if there is one) is responsible for determining the contractor’s past cost, schedule, and performance history to understand past performance, independently estimate costs for the current solicitation, and assess the reputation and reliability of suppliers. Contracting officers also monitor the status of current contracts, ensure deliverables are received, authorize payments, and report on contractor performance.

In the future, the data life cycle, as applied to this contracting function, suggests that DoD contracting systems could collect, curate, and

share aggregated historical contract data for vendors across programs, even across the military services. This could allow, for example, the identification of subcontract management issues with problematic supply chain vendors less likely to deliver on time due to internal or external factors and use data analysis to predict which contracts or programs are more likely to be affected.

Sustainment of Acquired Systems

Roughly 70–80 percent of a DoD acquisition program’s total cost is in the sustainment of the acquired system. As systems age, the costly and time-consuming effort of replacing obsolete parts becomes an increased concern, particularly as today’s systems increasingly rely on electronics.

In the past decade, data science—together with computer science and information technology—has been used to create “digital twin” parts to model the lifetimes and replacements of the thousands of parts comprising each system (Raghunathan 2010). More recently, evolving digital engineering capabilities demonstrated in the Air Force’s e-T7A Red Hawk (USAF 2021) will likely improve predictability in maintenance and parts service life of acquired systems. Investing and increasing in the generation, collection, storage, and subsequent modeling of data for designs, parts, and processes will enable DoD to rapidly respond to changing threats with real-time testing of new components in any system, and ultimately make better informed decisions regarding the sustainment, improvement, and retirement of military systems over time (Mizokami 2020).

Business Case and Economic Analysis

For any new acquisition program, DoD needs requirements, a business case, and economic analysis to justify the expenditure and acquisition approach. This includes an assessment of the return-on-investment and economic trends underlying the capability provided by the program. An investment in a new technology that is expensive but that should provide a significant military advantage would provide a justifiable return-on-investment. If a capability was projected to be at the early height of its cost-maturity curve and it was not a vital capability for military operations, the economic analysis would allow decision makers to decide whether or not to delay the acquisition to take advantage of declining prices.

These tradeoffs are currently conducted using whatever information is readily available about threat capabilities, technology advancements, and the cost of new technologies. The data life cycle suggests that these data could be defined more broadly, collected and curated across all of DoD, and made available to the acquisition workforce in the military departments

and OSD. With better data, more appropriate data analytics could be applied and the results visualized in a way that would show trends or tipping points that would be helpful in designing acquisition programs to maximize benefits and minimize costs.

Cybersecurity

Recently, DoD has created a Cyber Maturity Model Certification¹ to assess and ensure the cybersecurity posture and readiness of contractors in the defense industrial base. As DoD attempts to increase cybersecurity and prevent the loss of intellectual property and weapon system designs, tracking and sharing the cybersecurity maturity of different contractors will be very helpful to program managers in selecting among bidders on source selections. By collecting, curating, and sharing the cybersecurity maturity level of different contractors over time, data analyses could be designed to identify cybersecurity weaknesses. Trends and strengths assessments could be conducted, and the results could be used to inform acquisition and contract decisions.

Cost Estimating

The ability to accurately estimate the cost of a new program is necessary to support investment decisions and tradeoffs within a budget and to set reasonable baselines to assess cost performance over time. As discussed earlier, CADE contains an established historical database that provides detailed collected cost and schedule data for a variety of prior systems. This database, which is shared across all of DoD, supports data analytics by helping cost analysts estimate the costs of new programs based on extrapolations from historical data. CADE is an existing widely used example of the value of using the data life cycle.

Acquisition and Contract-Management Strategies

As discussed earlier, Heidi Shyu identified multiple areas where improved data collection, sharing, and analyses would have been helpful to her as she worked to develop successful acquisition strategies. She suggested many areas where she and other decision makers would have benefited from better data. By aligning more fully with the data life cycle the information she needed would be available to future acquisition leaders. For example, DoD could collect, curate, and share information about:

• The personnel roles and skillsets resident in various contractors,

___________________

¹See https://www.acq.osd.mil/cmmc/.

• Companies that provide critical capabilities but that may be experiencing financial stress, and
• General business intelligence about companies that provide needed capabilities procurement.

With this information, acquisition and contract-management strategies could be better designed by the acquisition leadership. By making these data accessible across DoD, use of the data life cycle would allow DoD to avoid poor performers and ensure critical capabilities were available. It could also identify duplication and opportunities for creating larger, cross-DoD contracts that might lead to cost savings. For example, earlier in the data life cycle, DoD could search for already existing data sources, internal or external, that could be used with minimal effort.

In addition to these case studies above, Table D.2 provides some additional but brief examples of acquisition-related functions and decisions using multiple phases of the data life cycle.

Recommendation 4.1: The Department of Defense should continue to seek improvements in defense acquisition through the increased application of data science, including addressing shortfalls in data collection, curation, management, and sharing.

REFERENCES