3

Ensuring That PHEMCE Decisions and Recommendations Are Defensible

During the course of its work, the committee heard numerous comments about problems with operations in the Public Health Emergency Medical Countermeasures Enterprise (PHEMCE). For example, PHEMCE processes for reviewing, assessing, and procuring medical countermeasures (MCMs) for the U.S. Strategic National Stockpile (SNS) were described as not fully scientific, justifiable, transparent, adaptive, or accountable. Political transitions were described as limiting PHEMCE’s effectiveness, with changes in the presidency leading to inappropriate neglect of established practices and mandates, often without repercussions.¹

SOUND BUSINESS PRACTICES

Formal and Informal Processes

Organizations depend on both formal and informal management processes for their success. The former provide predictability and accountability; the latter provide the detailed local knowledge and personal ties needed for collaborative efforts, especially for organizations that are tested in emergencies. The formal processes must be clearly specified and supported by policies for recruiting, rewarding, and retaining personnel—they provide predictability and operationalize the principle of accountability (see Chapter 2). The informal processes must be flexible enough to adapt

___________________

¹ This section draws on remarks presented during the committee’s third open-session meeting (see Appendix A).

to unexpected conditions and have that knowledge incorporated in formal processes (NRC, 2011a,b).

As not all knowledge is codified in formal procedures, organizations depend on people who know “how to get things done” and “who knows what.” Effective organizations support those vital individuals. Ineffective organizations push them out of the way, losing their institutional knowledge (Hansen, 1999; Kaufman, 1960). The combination of stable formal procedures and informal interpersonal networks is especially important when leadership or conditions change (Argote et al., 2018; Fuchs, 2010; Ren and Argote, 2011).

PHEMCE faces distinct challenges in creating the mutually supportive formal and informal processes needed for effective, predictable business practices:

• Coordinate the work of multiple organizations.
• Accommodate personnel changes associated with political events or public health emergencies (PHEs), including hiring subject-matter experts for any and all aspects of emergency operations.
• Shift priorities in response to emerging threats.
• Balance competing interests, including political, efficiency, and effectiveness concerns and the considerations of PHEMCE stakeholders.

Quality Management Systems (QMSs) are a recognized way to create and sustain such organizations.

Quality Management Systems

The U.S. Government Accountability Office (GAO) recommended that the Assistant Secretary for Preparedness and Response (ASPR) implement records management practices within PHEMCE after identifying discrepancies in PHEMCE’s document management practices (GAO, 2021)—a recommendation that could be readily operationalized within a mission-appropriate QMS. A QMS creates stable and predictable business practices, structures change control and documentation practices, and supports a culture of continuous learning. Widely used and globally accepted QMS frameworks include the international standards for QMSs and development of Quality Culture articulated in ISO 9001² and ISO 10018.³ The U.S. Food and Drug Administration (FDA) has already adopted ISO 9001 in its 2020 Staff Manual Guide, as has the Pharmaceutical Inspection Cooperation Scheme, a cooperative arrangement of 54 regulatory bodies aligning global pharmaceutical inspection prac-

___________________

² See https://www.iso.org/iso-9001-quality-management.html (accessed September 17, 2021).

³ See https://www.iso.org/standard/69979.html (accessed September 17, 2021).

tices. The International Council for Harmonization of Technical Requirements for Pharmaceuticals for Human Use (ICH) published the Q10 Pharmaceutical Quality System,⁴ which is considered guidance in the United States but a rule elsewhere. The International Society for Pharmaceutical Engineering published its Baseline Guide 5⁵ on Commissioning and Qualification of pharmaceutical manufacturing facilities and Good Automation Manufacturing Practices⁶ for ensuring quality management in manufacturing facilities. These systems have been used by global concerns that routinely face demanding timelines and high failure costs. In addition to renewing systemic oversight of PHEMCE procedures, adopting a QMS using one of them would stabilize operations via terms and practices compatible with its national and international partners, increasing trust and efficiency in interactions with them.

Addressing Conflicts of Interest

Similar to government agencies, PHEMCE must avoid conflicts of interest (COIs) to the extent possible. These may arise when setting priorities and fulfilling them. PHEMCE’s complex partner networks (e.g., developers, manufacturers, consultants), spread across agencies with different disclosure requirements, can make it difficult to observe conflicts and violations. The pressures of PHEs can hamper normal vetting procedures. QMS can reduce these risks, by anticipating and addressing potential problems, while increasing transparency.

The committee suggests that PHEMCE and ASPR consider two specific options:

• A firewall between PHEMCE and the Biomedical Advanced Research and Development Authority (BARDA), so that PHEMCE decisions on developing MCMs are independent of BARDA’s execution of those decisions.
• A COI policy that is as robust as the COI policies of PHEMCE partner agencies, such as FDA’s prohibited financial interests guidance⁷ and the Defense Advanced Research Projects Agency’s (DARPA’s) contract management procedures.⁸

___________________

⁴ See https://www.ema.europa.eu/en/documents/scientific-guideline/international-conferenceharmonisation-technical-requirements-registration-pharmaceuticals-human_en.pdf (accessed September 17, 2021).

⁵ See https://ispe.org/publications/guidance-documents/baseline-guide-vol-5-commissioning-qualification-2nd-edition (accessed September 17, 2021).

⁶ See https://ispe.org/product-types/gamp-good-practice-guides (accessed September 17, 2021).

⁷ See https://www.fda.gov/about-fda/ethics/fact-sheet-prohibited-financial-interests-fda-employees (accessed September 3, 2021).

⁸ See https://www.darpa.mil/work-with-us/contract-management (accessed September 3, 2021).

PRIORITY SETTING

While a current description of the PHEMCE priority-setting process was not provided to the committee, it is known that PHEMCE’s priority setting is built around threats to national security identified by DHS. PHEMCE needs a transparent, publicly acceptable process for defining its priorities, recognizing these as ethical decisions, based on scientific evidence, not technical, bureaucratic ones. Clear definitions of priorities would be critical to focus on the enterprise’s needs and give partners greater confidence in engaging with clarity of timeline and expected investment. This would provide a stronger environment for strengthening the industrial base.

Faced with a related priority-setting dilemma, the National Academies Committee on Equitable Allocation of Vaccine for the Novel Coronavirus adopted a process that might guide PHEMCE priority setting (NASEM, 2020):

• adopting widely acceptable ethical principles (maximize benefits, equal regard, mitigate health inequities, fairness, evidence based, transparent);
• translating those principles into easily understood quantitative measures (probability of getting the disease, probability of severe consequences, probability of transmitting to others, impact on others if unavailable);
• using readily available (or produced) data to estimate those four measures for various population groups (e.g., first responders, K–12 teachers and staff and child care workers, children);
• setting priorities using those estimates;
• conducting sensitivity analyses to identify priorities that would change with changes in the estimates or external circumstances; and
• establishing two-way communications with stakeholders to understand their concerns, ensure their understanding, and increase the legitimacy of its priorities.

Since the committee’s review was limited to the scarce vaccine supply, PHEMCE would have to consider how it can embrace such an approach within the constraints of its own scope (e.g., a broad range of MCM addressing a large threat space, often with sparse data).

Box 3-1 shows the criteria used in the 2014 PHEMCE SIP (ASPR, 2015), which can inform the work of a revitalized PHEMCE and its broad mission space while also adopting a principled approach akin to the Framework for Equitable Allocation of COVID-19 Vaccine. Details on the committee’s analytical and consultation process can be found in its report (NASEM, 2020), which draws on the extensive research litera-

tures regarding consultative processes, risk communication, and priority setting (e.g., Fischhoff, 2013; NASEM 2017a,b; NRC, 2008; Pidgeon et al., 2014).

Threat and Needs Assessment

Risk reflects both the probability of threats and their impact on valued outcomes (e.g., morbidity, mortality, equity, social, economic, national morale). The value of an MCM depends on risks with and without that countermeasure. Establishing the priority for MCM investments requires estimating those risks and this requires a threat and needs assessment.

For existing MCMs, PHEMCE has used a Preparedness Assessment Framework to identify gaps and options for closing them (ASPR, n.d.). When applied comprehensively, it considers the entire MCM life cycle, including threats to supply chains, storage and maintenance, deployment, and usability, given recipients’ training, legal constraints, and competing duties. Such an analytical framework is a natural component of a QMS. By identifying vulnerabilities, these analyses of threats and needs identify ways

to ensure the usability of existing MCMs, while accepting the priorities that guided the investment in them.

To identify future MCM needs, the analytical approach must suit the problem. Probabilistic risk analysis is appropriate for problems where risk factors can be quantified, by reliable observation, modeling, or expert judgment (Fischhoff, 2015; Morgan, 2017). Two National Academies workshops on gain-of-function research aired many concerns about extending such analyses beyond PHEMCE’s domain (NASEM, 2016; NRC and IOM, 2015). Simulation and tabletop exercises can stress test a supply chain for robustness and resilience (Gao et al., 2019) while building informal organizational ties. Additionally, machine learning and artificial intelligence may provide unique insights, when training sets are available to calibrate the quality of predictions, though such tools would require enhanced transparency to maintain their credibility.

Creating a shared understanding of such potentially complex, uncertain, and changing analyses will require a concerted effort. Without it, effective collaboration and trust may be impossible. In its threats and needs assessments, PHEMCE leadership may wish to consider the Benefit-Risk Framework adopted by FDA’s Center for Drug Evaluation and Research and Center for Biologics Evaluation and Research (FDA, 2018). Designed to navigate uncertain and sparse data, the framework creates a shared top-level summary table, pooling estimates and interpretations from diverse experts. FDA uses it to both coordinate the analytical process and communicate with internal and external stakeholders. Although it may have certain limitations (NASEM, 2017c), the standard format reflects FDA’s priorities, along with the supporting evidence, with links to additional documentation.

EVALUATION AND ACCOUNTABILITY

Performance Metrics and Quality Assurance

Measurement drives continuous quality improvement. The ultimate measure of PHEMCE’s effectiveness is whether it delivers needed MCM at the right time in the right place to the right people. PHEMCE’s leadership needs to know not only how well the system is performing overall but also where it can, and must, be improved (Frazelle, 2002).

PHEMCE needs KPIs that assess its efficiency, effectiveness, and equity for each major threat type. KPIs should provide the information needed by external stakeholders (e.g., companies stockpiling product, entities administering it); address the whole of PHEMCE and the life cycle of its MCM, reflect its overall goals, and inform its budgetary process; and be as stable as possible, to allow comparisons across time, while being flexible enough to reflect changes in PHEMCE and the world, and both backward looking (asking

how the system has performed, and why) and forward looking (asking how possible actions will affect future performance). As with threat assessments, these measures should use the best-available methods (e.g., artificial intelligence, decision analytics) while also recognizing their potential limitations.

Evaluation

As part of PHEMCE’s QMS, these summary measures should be complemented by after-action reports, providing narrative root-cause analyses of PHEMCE successes and failures, and plans to address the failures and build on the successes. PHEMCE and its nonfederal and private-sector partners and stakeholders should meet periodically to review its processes and preparedness. One approach that PHEMCE leadership may consider is a joint evaluation that it designs and conducts, in collaboration with independent parties, knowledgeable about PHEMCE but not directly involved in its current operations. These reports could be timed to inform periodic revisions of its SIP. Another useful evaluation would be a series of PHEMCE portfolio reviews.

Another form of joint evaluations that could address PHEMCE operational assumptions is preparedness “sprints”⁹: rapid simulation exercises, involving the people and organizations that will use MCM systems, in realistic scenarios, at appropriate scales. They are designed to test processes and systems for specific scenarios, to reveal gaps and challenges, and should simulate both likely and emerging threat scenarios, which range in size but should also dovetail with real-life need and deliver a public good, including progress toward an MCM for public health.

PHEMCE should make these evaluations as widely available as possible, resorting to classification or For Official Use Only, only when essential.

DATA SYSTEMS AND TECHNOLOGY

To fulfill its mission, PHEMCE needs reliable, redundant data infrastructure and supporting quality management of MCM life cycles. PHEMCE must know, and be able to share, the status of its operations in real time. That infrastructure should have the properties of robust data systems (Rudolph, 1989). The committee understands that the current system lacks these properties, in part due to the lack of a robust digital infrastructure throughout the federal government limiting PHEMCE’s ability to protect the American people, but believes that aspirational goals for updating the data infrastructure are key to a re-envisioned PHEMCE.

___________________

⁹ This section draws on remarks presented to the committee by Matthew Hepburn during the public workshop of the committee’s third meeting (see Appendix A).

• Reliability and Availability. The data infrastructure and the measures it provides must allow real-time analyses meeting PHEMCE’s needs. The system must be resilient to natural disasters and cybersecurity threats.
• Serviceability. The data infrastructure must require little maintenance that interrupts its service and be usable by operators.
• Interoperability. The data system must support pooling data from multiple sources, so that it is interoperable with federal agencies and relevant external entities.
• Scalability. The data system must be adaptable to preparedness and response activities of the different types and scales, within PHEMCE’s responsibility.

A cross-HHS data system that provides information from raw materials to “last-mile” capability is ideal. Its utility is first and foremost to identify areas of vulnerability that need to be addressed nimbly and strategically. It should also be useful for regulators and response and logistics planning. While classification concerns pose challenges for cross-agency data systems, too much classification limits sharing of and access to information that encourages innovation and participation.

CONCLUDING REMARKS

The committee heard many reports of failures in PHEMCE’s operations. While it could not investigate these, it could observe that PHEMCE needs strengthening in several areas essential to its mission, including sound business practices, support for priority setting, and tools for evaluation and accountability.

RECOMMENDATIONS

REFERENCES

NASEM. 2017a. Building communication capacity to counter infectious disease threats: Proceedings of a workshop. Washington, DC: The National Academies Press.

NASEM. 2017b. Communicating science effectively: A research agenda. Washington, DC: The National Academies Press.

NASEM. 2017c. Pain management and the opioid epidemic: Balancing societal and individual benefits and risks of prescription opioid use. Washington, DC: The National Academies Press.

NASEM. 2020. Framework for equitable allocation of COVID-19 vaccine. Washington, DC: The National Academies Press.

NRC (National Research Council). 2008. Public participation in environmental assessment and decision making. Washington, DC: The National Academies Press.

NRC. 2011a. Intelligence analysis for tomorrow: Advances from the behavioral and social sciences. Washington, DC: The National Academies Press.

NRC. 2011b. Intelligence analysis: Behavioral and social scientific foundations. Washington, DC: The National Academies Press.

NRC and IOM (National Research Council and Institute of Medicine). 2015. Potential risks and benefits of gain-of-function research: Summary of a workshop. Washington, DC: The National Academies Press.

Pidgeon, N., C. Demski, C. Butler, K. Parkhill, and A. Spence. 2014. Creating a national citizen engagement process for energy policy. Proceedings of the National Academy of Sciences of the United States of America 111(Suppl 4):13606–13613.

Ren, Y., and L. Argote. 2011. Transactive memory systems 1985–2010: An integrative framework of key dimensions, antecedents, and consequences. The Academy of Management Annals 5(1):189–229.

Rudolph, P. 1989. Reliability, availability, serviceability. In The design of a microprocessor. Berlin/Heidelberg: Springer. Pp. 132–143.