2

Background

DOD MANUFACTURING INSTITUTES

DoD has a strong tradition of innovation development and implementation in support of next-generation defense systems. Within DoD’s technology development enterprise, there is constant attention to the maturation of technologies that support the challenging performance goals of future military systems. It is recognized that, in order to effectively mature and transition DoD science and technology advances into production, access to a robust and responsive U.S. industrial base is essential. Ideally, the industrial base would be equipped with advanced manufacturing technologies, which could deliver critical products and systems affordably and rapidly.

To better support the need for timely, effective manufacturing technology development and transition, DoD has established nine Manufacturing Innovation Institutes through its Defense-wide Manufacturing Science and Technology (DMS&T) program element within the DoD Manufacturing Technology (ManTech) program. The institutes are considered by DoD to be crucial facilitators that bring together innovative often non-traditional ecosystems in key technology and market sectors in the United States, enabling their vibrancy and robustness.¹ The DoD Manufacturing Innovation Institutes are industry-led, with a dual public–private benefit, providing large commercial market potential while meeting key U.S. defense needs. The mission of the nine DoD-established institutes addresses both commercial and defense manufacturing needs within specific, defense-relevant technology areas.

___________________

¹ See Department of Defense (DoD), 2017, Department of Defense Manufacturing USA Strategy, Version Date September 8, 2017, Director DoD Manufacturing Technology Program, OUSD(R&E) Strategic Technology Protection and Exploitation.

As is shown in Figure 2.1, DoD has committed approximately $1.12 billion to date for these nine institutes,² and non-DoD partners have contributed approximately $1.93 billion. As noted in Fiscal Year 2020 Industrial Capabilities Report to Congress,³ DoD views the MII public–private partnerships as a means to achieve significant advances in important manufacturing technologies in a cost-constrained environment. However, budgeted core funding for the institutes is planned to decrease from the FY 2020 appropriated level of $167 million to about $64 million in FY 2022-2025,⁴ which will require difficult decisions by DoD and the institutes on how to accommodate the drop in future funding.

The mission of the DoD MIIs is consistent with the responsibilities for manufacturing technology assigned to the USD(R&E) and overseen by the OSD ManTech office.⁵ That office recognized the need for a coherent and effective strategy to guide the establishment and sustainment of MIIs, and also to ensure their value to DoD and the nation, by setting five strategic goals for its Manufacturing USA institutes:

• Goal 1: Drive impactful advanced manufacturing research and development.
• Goal 2: Encourage the creation of viable and sustainable institute business plans.
• Goal 3: Maintain an optimal program design to maximize value delivery.
• Goal 4: Maximize stakeholder understanding of DoD’s Manufacturing USA institutes.
• Goal 5: Effectively support a capable workforce.⁶

The OSD ManTech office provides oversight of the DoD institutes and has performed annual reviews of all of the MIIs since the inception of the institutes.

EVALUATION CRITERIA AND METRICS FOR THE DOD MIIS

The federal government has long recognized the need for periodic program assessments to assure the performance, cost, and effectiveness of taxpayer-funded programs. Government agencies, such as the Office of Management and Budget (OMB) and the Government Accountability Office (GAO), and professional societies, such as the American Evaluation Association and the National Academies, have performed a number of studies on best practices and strategies for assessment of a wide variety of federal programs.^7,8,9 DoD was an early adopter and strong proponent of formal program evaluation processes. In 1961, DoD established the Office of Systems Analysis to perform the program evaluation function. Since then, requirements for program evaluation have been included in DoD policies and guidebooks for a wide range of defense programs and organizations. One example is the Comprehensive Review Guidelines for DoD federally funded research and development centers (FFRDCs).¹⁰

___________________

² DoD, 2021, Fiscal Year 2020 Industrial Capabilities Report to Congress, p. 157, https://media.defense.gov/2021/Jan/14/2002565311/-1/-1/0/FY20-INDUSTRIAL-CAPABILITIES-REPORT.PDF.

³ Ibid.

⁴ Budget Exhibit R-2 for PE 060368D8Z: Defense Wide Manufacturing Science and Technology, Feb 2020

⁵ DoD, 2018, “DoD Directive 5200.14: Manufacturing Technology (ManTech) Program,” Change 1, October.

⁶ Ibid.

⁷ Phase 4 Implementation of the Foundations for Evidence-Based Policymaking Act of 2018: Program Evaluation Standards and Practices, https://www.whitehouse.gov/wp-content/uploads/2020/03/m-20-12.pdf Accessed on 11/23/2020.

⁸ An Evaluation Roadmap for a More Effective Government, https://www.eval.org/evaluationroadmap, accessed on 11/23/2020.

⁹ NASEM, 2017, Principles and Practices for Federal Program Evaluation: Proceedings of a Workshop, Washington, DC: The National Academies Press, https://doi.org/10.17226/24831.

¹⁰ USD(AT&L) Policy Memo, Subject: FFRDC Management Plan and Associated “How To” Guides, 2 May 2011.

The evaluation framework that has been developed by DoD for the MIIs is based on one that DoD has used successfully in review of FFRDCs and University Affiliated Research Centers (UARCs). The success of the evaluation framework is demonstrated by the fact that the DoD has used the framework to complete 5-year, strategic evaluations of 24 DoD FFRDC and UARC organizations over a 20+ year period. The evaluations performed, based on this framework, are used by senior DoD officials as the basis for decisions to renew the FFRDC or UARC, to reopen the competition in the core competency areas, or to end the activity. This framework is consistent with the models of evaluation processes that have been proposed and used by many federal agencies as well. For example, a National Academies study that examined major National Science Foundation (NSF) program recompetitions and suggested that the two key questions for reviewers to assess were the following:

1. Is this still a worthwhile activity or has it become obsolete in the face of scientific and technological advances?; and
2. If the activity is deemed worthwhile, is the current grantee doing a good job that merits continuation or should the award be opened to competition?¹¹

The OUSD R&E evaluation process takes a similar approach but has additional considerations which are relevant for the more mission-focused DoD MIIs. The DoD will assess the MIIs in four key question areas, as seen in Figure 2.2, which is taken from the MII Strategy and Assessment document.¹²

___________________

¹¹ National Academy of Sciences, National Academy of Engineering, and Institute of Medicine, 1994, Major Award Decisionmaking at the National Science Foundation, Washington, DC: The National Academies Press, https://doi.org/10.17226/2268. See Chapter 5, “Recompetition of Awards,” p. 110.

¹² Table from the MII Strategy and Assessment_ManTech_Oct2020_cleared PowerPoint, received on November 9, 2020, from ManTech.

OUSD R&E plans to use the Joint Manufacturing Defense Council (JDMC) as an independent source to perform the 5-year strategic evaluations of the MIIs, as seen in Figure 2.3.¹³ The JDMC, chartered in 2020, is an OSD-led organization with 18 organizational members within DoD; it is tasked with coordination and strategic guidance of advanced manufacturing activities across DoD. The mission of the organization is:

Important goals of the JDMC include the following: alignment of defense research, manufacturing, maintenance, and supply capabilities; increased collaboration and efficiencies in DoD manufacturing activities; development of strategies for a robust organic and commercial manufacturing base; and development of DoD’s human capital plans in coordination with other national programs. Since the MIIs address these goals, it is appropriate that the JDMC serve as the evaluator for this substantial DoD investment in advanced manufacturing. In addition, the JDMC is well positioned to not only assess the performance of the MIIs but also to facilitate increased visibility of the MIIs within DoD and enhance the impact of the institutes for DoD. The JDMC has been formally charged with organization and execution of the periodic, 5-year reviews of the MIIs. The JDMC will put together an evaluation team to

___________________

¹³ JDMC Charter, https://www.DoDmantech.com/JDMTP/Files/JDMTP_SubPanels_Charter_2011.pdf, accessed January 15, 2021.

perform the review of an MII. The team will be responsible for the assessment of the value and performance of the MII in support of Department mission goals. The team membership will include all of the expertise required to complete the assessment of performance in the critical elements of the MII (e.g., manufacturing technology, education and workforce development, and ecosystem). The team will primarily consist of personnel from the JDMC organizations but may be supplemented with additional experts as needed. The deliverable from the JDMC evaluation team will be a recommendation to OSD ManTech and OSD R&E on the continuation of core funding to the MII. Senior DoD leadership will then incorporate JDMC evaluation team’s recommendations into broader strategic considerations within the Department and make the decision on the future of the MII. OSD ManTech will manage the evaluation process for all of the MIIs. These reviews will be performed on a rotating basis and are scheduled to start in 2021.

In addition to the development of a strategy and approach for periodic independent evaluation of the institutes, DoD has made significant progress in the development of metrics for assessing MII performance. DoD has produced the “2020 DoD Metrics List” and the “Final Metrics List for the DoD MIIs.”¹⁴ These documents, representing the output from the latest activity on metrics development for the DoD MIIs, began with activities sponsored by the Advanced Manufacturing National Program Office

___________________

¹⁴ Excel books: “Final DoD MII Performance Metrics_cleared” and “2020 MII Complete Metrics List_cleared,” both received from ManTech on December 11, 2020.

(AMNPO),¹⁵ which examined metrics for all of the manufacturing institutes sponsored by federal agencies. As the MIIs have matured, the metrics used to assess performance have grown from the initial, organizational development-focused metrics to the output focused-metrics and impact-focused metrics Based on these metrics lists, OSD ManTech currently collects a considerable amount of data on the MIIs on a weekly, quarterly, semi-annual, or annual basis. The information topics include: Ecosystem (e.g., patents filed and licensed, successful technology transitions, extent of member partnering, state involvement in MII), EWD (e.g., EWD projects underway and completed, number of manufacturing workforce participants in EWD activities, number of participants in Institute-aligned training, apprenticeship, or certification programs), Financial (e.g., core and non-core funding levels (from DoD, federal agencies, state agencies, industry, Foundations, etc.)) and Operations (e.g., projects completed, projects meeting objectives, number of small and large companies involved in projects, number of Government directed projects), An example of the information gathered via the metrics lists is shown in Figure 2.4. The JDMC will benefit from all of the data gathered by OSD ManTech in the completion of their independent 5-year assessment of the MIIs.

___________________

¹⁵ National Network of Manufacturing Institutes, 2015, Guidance on Institute Performance Metrics, https://www.manufacturingusa.com/reports/guidance-institute-performance-metrics-national-networkmanufacturing-innovation.