Based on the information gathered from all sources, the committee has developed the following list of findings regarding the DoD MMIs. In this interim report, the committee confines its comments to the first task of the study’s statement of task: evaluation criteria and metrics (the statement of task is reprinted in Appendix A).
Finding 1: As noted in Fiscal Year 2020 Industrial Capabilities Report to Congress,1 support for the DoD MIIs remains high among senior leadership at DoD, but the declining budget request for core funding will require difficult decisions. At the 5-year review point for each institute, DoD will need to decide whether to continue core funding support, possibly at a reduced level or at a higher level, or decline to renew the agreement in order to apply available budget to higher priorities.
Finding 2: DoD has a sound framework for conducting 5-year assessments of each institute to support such decisions. The four issue areas in the Joint Defense Manufacturing Council (JDMC) evaluation plan2 are based on an assessment process which is familiar to DoD and will facilitate the assessment of the MIIs within the context of DoD’s mission needs. The planned evaluation process is similar to good practices the committee has seen in other agencies. The JDMC, based on its charter, is well suited to conduct the 5-year assessments. Over time, the committee believes the 5-year reviews can improve the coupling between public and private interests in each MII and move all MIIs toward collective implementation of best practices. Equally important, the JDMC can play a key role in the integration of the MIIs into DoD’s technology delivery pipeline, which will be further explored in this committee’s work as described by the third task in the statement of task for this study, and presented in the final report.
Finding 3: The metrics needed for strategic 5-year assessments are not limited to the metrics used for ongoing management and oversight of the MIIs. DoD has a robust set of metrics for quarterly reviews that are important to assessing progress toward annual performance objectives. The 5-year reviews are more strategic in nature. Additional topics that are appropriate in the 5-year assessment would include the continued need, appropriateness of the public–private partnership, and long-term performance trends of the MII. Strategic metrics would also be appropriate for the 5-year reviews.
Finding 4: The unique public–private partnership aspect of the MIIs requires strategic consideration of the health of the “private” component of the institutes. To the extent that DoD views these cost-shared
1 Department of Defense, 2021, Fiscal Year 2020 Industrial Capabilities Report to Congress, https://media.defense.gov/2021/Jan/14/2002565311/-1/-1/0/FY20-INDUSTRIAL-CAPABILITIES-REPORT.PDF.
2 The MII Strategy and Assessment_ManTech_Oct2020_cleared PowerPoint, received on November 9, 2020, from ManTech.
private–public partnerships as essential, an additional consideration would be input from MII members and non-members from the ecosystem.
Finding 5: Greater involvement of the DoD acquisition and sustainment communities are needed for DoD to fully benefit from the technology development, EWD, and manufacturing ecosystem of DoD MIIs. The JDMC has an opportunity to use the 5-year assessments and the growth of this engagement over time as a key measure for the impact of the MIIs on DoD advanced technology development and implementation.
Recommendation: The Department of Defense should phase in formal 5-year assessments of the Manufacturing Innovation Institutes (MIIs) to support decisions on renewal of agreements and to provide budget justification for future years commensurate with the value of the MII program.
Recommendation: The Office of the Undersecretary of Defense Manufacturing Technology (OSD ManTech) and the Joint Defense Manufacturing Council should expand the evaluation framework within its four topic areas to consider additional strategic considerations focused on the MII key objectives of advancing research, EWD, and ecosystem. Design of the evaluation team and details of the evaluation tasks should consider the best practices of other agencies discussed in section 3, including provisions for access to outside technical experts as needed. The process should be tailored to the unique characteristics of the Manufacturing Innovation Institute (MII) public–private partnerships and should be made available to the MIIs and the rest of the manufacturing community.
Recommendation: The Office of the Undersecretary of Defense Manufacturing Technology (OSD ManTech) and the Joint Defense Manufacturing Council should amend the planned evaluation process to make provisions for additional strategic metrics and trend analysis, and for additional sources of input beyond those currently planned, such as the following: other stakeholder survey results, assessment of the evaluation criteria, site visit results, and terms of reference planning agreements. In assessing effectiveness of governance and management, specific attention should be given to the effectiveness of government and industry leadership in achieving desired outcomes.
Recommendation: The Joint Defense Manufacturing Council 5-year assessment should address strengths, weaknesses, and opportunities for improvement of the Manufacturing Innovation Institutes (MIIs) relative to non-Department of Defense (DoD) components of the public–private partnership to inform decisions on whether a public–private partnership is the best way to meet DoD needs. The Office of the Undersecretary of Defense Manufacturing Technology (OSD ManTech) should also perform a stakeholder survey as a component of the 5-year MII assessment. The survey should include all stakeholders such as large, medium, and small businesses, academia, state governments, and foundations that are within the MII ecosystem.
Recommendation: As the Department of Defense starts the 5-year Manufacturing Innovation Institute evaluation process in 2021, it is recommended that a process be established to catalog lessons learned and assess the effectiveness of the Joint Defense Manufacturing Council evaluation process to facilitate continuous improvements. A meta-evaluation can be accomplished by appropriate groups in professional societies or within suitable not-for-profit organizations.