8

Improving and Expanding the Evidence Base for Public Health Emergency Preparedness and Response

As described in preceding chapters, despite investments in public health emergency preparedness and response (PHEPR) research in the past two decades, the corresponding PHEPR evidence base remains limited (Carbone and Thomas, 2018). Figure 8-1 shows the study design distribution of all articles included in a committee-commissioned scoping review focused on the 15 PHEPR Capabilities (see Chapter 2 and Appendix D for additional detail on the commissioned scoping review). Among the 1,692 articles included, approximately 35 percent were classified as opinion, concept, or position papers or literature reviews, while 65 percent reflected some form of systematic data collection and analysis that could potentially provide evidence regarding 1 of the 15 PHEPR Capabilities. The most common study design category was quantitative nonimpact studies, which accounted for 33 percent of all articles and 51 percent of evidentiary studies. The quantitative nonimpact category includes studies that describe and identify the magnitude, severity, and preventability of a PHEPR problem and could potentially be used to inform the development of PHEPR practices aimed at addressing that problem. The quantitative impact category, accounting for only about 6 percent of all articles, includes studies that evaluate specific PHEPR practices.

This distribution of study designs presents challenges for identifying evidence-based practices, and in Chapter 2, the committee presents the following conclusion:

Conclusion: With the increasing complexity of both public health emergencies and the PHEPR system, policy makers and practitioners have a crucial need for access to guidance based on robust evidence to support their decisions on practices, policies, and programs for saving lives during future public health emergencies. Therefore, a coordinated and comprehensive approach to prioritizing and aligning research efforts and ensuring that research is relevant and consistently connected to practice, along with investments in research infrastructure, is necessary to strengthen the PHEPR evidence base, thereby ensuring that PHEPR practitioners have the scientific evidence they need to guide and inform their actions. At the same time, PHEPR practitioners will require incentives to base their practices, policies, and programs on evidence.

In this chapter, the committee describes a framework to support the systematic development of knowledge in the PHEPR field and sets forth the aspirations for high-quality, rigorous PHEPR research and evaluation.

A NATIONAL PHEPR SCIENCE FRAMEWORK

The generation of a PHEPR evidence base has been hindered by the inherent challenges of conducting research on public health emergencies, which can limit the opportunities to observe and proactively study the effects of practices used to mitigate harm before, during, and after such an event (see Chapter 2 for additional details). In addition to those challenges, federal partners and other stakeholders repeatedly have attempted to enhance PHEPR research capacity. However, many of these efforts have not been sustained or adequately resourced, and the result has been a limited infrastructure to support the generation and dissemination of PHEPR research.

Conclusion: Funding for and prioritization of research before, during, and following public health emergencies are currently fragmented and disorganized, spread across multiple funding agencies, inconsistent, and do not encourage the progression of quality research or the sustainable development of research expertise. This situation has contributed to a field based on long-standing rather than evidence-based practice.

Despite the implications of public health emergencies for the nation’s health and economic security, there is currently no mechanism for ensuring the coordinated resourcing, monitoring, and execution of public- and private-sector PHEPR research.

Key Components of a National PHEPR Science Framework

The importance of developing a disaster research strategy has previously been underscored (Keim et al., 2019). The committee proposes that a comprehensive National PHEPR Science Framework could move the PHEPR field beyond the near-term goal of a research agenda toward a more coherent vision of coordinating and aligning efforts effectively to advance evidence-based practice in PHEPR (see Figure 8-2). The key components of this framework are discussed in the following sections.

System Leadership to Transform the PHEPR Research Enterprise

The foundations of scientific progress in PHEPR lie in building and sustaining a research enterprise. Strong leadership at all levels, but especially at the federal level, is central to the framework and essential to support systems-level change and mobilize agencies to transform the way PHEPR research is coordinated, funded, and conducted. An interagency and multidisciplinary effort led by the Centers for Disease Control and Prevention (CDC) will be necessary to develop and implement a National PHEPR Science Framework; establish an authority and process for supporting research before, during, and following public health emergencies; and ensure that adequate research funding, capacities, and infrastructure are in place. CDC is the funding agency with the primary mission responsibility in PHEPR, and it is important that the agency responsible for supporting PHEPR planning and implementation also leads efforts to increase the scientific evidence base that supports the execution of that responsibility. However, the committee acknowledges that no one agency can accomplish this transformation of the PHEPR research enterprise, and it will be necessary to leverage the strengths of different partners, including funding partners, in these efforts. Given the complexity of the PHEPR system and the fact that it is nested within many integrated, larger systems (e.g., supply chain, transportation), it will be imperative for this effort to be carried out in cooperation with other fields, industries, and organizations. It will also be essential to provide CDC with adequate resources and expertise to support its lead role in these efforts and to encourage CDC to become a learning organization. Appropriation language will need to be explicit and clear that funding can be used for these research efforts.

An enterprise for coordinating medical countermeasure (MCM) efforts in the federal government could serve as a model for coordinating and funding PHEPR research (HHS, 2017a). The Public Health Emergency Medical Countermeasures Enterprise (PHEMCE), established by the U.S. Department of Health and Human Services (HHS) in 2006 and led by the Assistant Secretary for Preparedness and Response (ASPR), has the following core members: the director of CDC, the director of the National Institute of Allergy and Infectious Diseases within the National Institutes of Health (NIH), and the commissioner of the U.S. Food and Drug Administration. Key PHEMCE partners include senior leadership from other federal agencies and numerous nonfederal partners. Recognizing that the development of MCMs requires significant resources in terms of time and cost, the PHEMCE helps coordinate funding across portfolios and prioritize efforts through a coherent plan that spans several years (HHS, 2017b). PHEMCE members are motivated and guided by the need to develop responses and cost-efficient methods to protect the nation against novel threats. This enterprise has been successful in bringing attention to the issue of MCM development, generating additional research investment, and prompting government agencies and departments to coordinate their research efforts. Such a coordinated effort can bring common terminology and infrastructure to a fragmented research enterprise.

Recognition of PHEPR Science as a Unique Academic Discipline Within Public Health

PHEPR research is transdisciplinary and draws on the knowledge base from many different fields, from behavioral and social sciences and political science to systems science and operations research, among others. However, recognition that PHEPR science is a unique academic discipline within the broader public health field represents the first step in addressing the substantial need for research; qualified, well-trained researchers; and preexisting, durable, and reliable engagement and partnerships with PHEPR practitioners.

The nuances and complexity of the PHEPR field pose unique challenges for designing research studies that differ from those in the broader public health field (e.g., population health research), and the usual approach for clinical research—randomized controlled trials (RCTs)—is not always feasible. Researching public health emergencies requires specific knowledge and understanding of how the systems involved work, and public health scientists may lack the knowledge, experience, and credibility to conduct research before, during, or following a public health emergency. Given the nuances of PHEPR research, it can be especially important for researchers to be familiar with the strengths and limitations of various approaches to and methods for evaluation. These may include, for example, observational studies designed in real time with strong data collection, predesigned pragmatic trials ready for deployment, simulations and exercises that use RCT approaches, side-by-side qualitative research studies to supplement quantitative methods, and methods for capturing data from after action reports (AARs) in a way that supports knowledge generation.

Recognizing PHEPR science as a unique academic discipline within the broader public health field could generate the resources and efforts needed to better support current and new academic departments or centers focused on PHEPR; establish degree or certification programs; and support career mechanisms that would enhance the conduct of high-quality, rigorous PHEPR research. Training and career development for PHEPR researchers are discussed in greater depth in the section on workforce capacity development for PHEPR research and practitioners later in this chapter. With recognition of PHEPR as a distinct field of study, the PHEPR research community could develop its own unique culture with the support of scientific societies and associations, which currently is lacking.

A Forward-Looking PHEPR Research Agenda and Common Evidence Guidelines

A research agenda is necessary to galvanize the PHEPR research field to meet the needs and respond to the concerns of PHEPR practitioners and society at large (IOM, 2008). This agenda must be more than a simple inventory of research needs (see Box 8-1). It will require leadership, and an organizing entity will need to be identified and made responsible for aggregating research conducted in alignment with the agenda and for tracking progress on and updating the agenda. A component of the agenda could highlight what is and is not known or what PHEPR research is currently under way. PHEPR evidence gaps could, for example, be identified and communicated by the PHEPR evidence-based guidelines group proposed in Chapter 3. Another essential feature of the research agenda will be to describe a process for rapidly identifying and prioritizing research needs during a public health emergency and to establish a minimum set of data elements that would be sought by anyone collecting data during such an event (NBSB, 2011).

A formal research prioritization process will be necessary, including a top-down, bottom-up approach to setting the research agenda. An important consideration is for the process to be inclusive of governmental, nongovernmental, private, and academic organizations, as well as broad public input from practitioners, policy makers, researchers, and the community. Given that outcomes of interest in PHEPR also include process and system improvements (Carbone and Thomas, 2018), it will be valuable to engage in the research agenda development process experts in the fields of health services research; social science; implementation and improvement science; operations research; complex interventions; quality improvement; cost-effectiveness; and systems, policy, and organizational research. The National Institute of Environmental Health Sciences (NIEHS) has conducted a successful town hall model in communities across the country for bringing practitioners, researchers, and community members

together to participate in setting a research agenda, enhance practitioner–researcher partnerships, and foster greater awareness of community and public health needs (O’Fallon et al., 2003). O’Fallon and colleagues (2003) note several best practices for successful town halls:

• the meeting is held in a location that is convenient and comfortable for the participants;
• controversial topics are encouraged, and when such a topic is selected, it is important to ensure that both sides of the issue are presented;
• lecturing is minimized and audience participation is maximized; and
• the final agenda is determined by the host organization.

In addition to town hall models, workshops and leadership retreats for practitioners and researchers can be useful to identify new developments and research topics in PHEPR (O’Fallon et al., 2003).

It will be important for the research agenda to encompass a range of research questions that would be addressed through different methods of inquiry (see the section on common evidence guidelines for PHEPR later in this chapter), including and going beyond the topics covered within CDC’s 15 PHEPR Capabilities. A good starting point for identifying research questions will be to examine previously developed research agendas for PHEPR (Acosta et al., 2009; IOM, 2008). Additional research priorities may be informed by the evidence gaps identified in the committee’s commissioned scoping review (see Appendix D) and the committee’s evidence reviews for practices within the Community Preparedness, Emergency Operations Coordination, Information Sharing, and Non-Pharmaceutical Interventions Capabilities (see the sections on future research priorities in Chapters 4–7).

Meaningful Partnerships Between PHEPR Practitioners and Researchers to Promote Knowledge

Crucial to a National PHEPR Science Framework will be ensuring a strong connection between PHEPR practitioners and researchers, as well as strong community partnerships. Understanding how to promote, improve, and sustain the engagement of PHEPR practitioners and communities in a thoughtful and inclusive process for generating research will be an essential element of a robust and effective PHEPR research field (Miller et al., 2016). With the proper incentives in place, PHEPR practitioners and researchers can be encouraged to engage in more meaningful partnerships to promote knowledge. Academic partners can help public health agencies by providing data and findings to inform practice (especially during a real public health emergency, when public health agencies may be challenged in conducting research), executing studies, facilitating stakeholder meetings, assessing training needs, providing technical assistance, and collaborating on publications (IOM, 2015). Specifying expectations related to the conduct, and subsequent publication, of research and evaluation in practitioner position descriptions, or even designating a dedicated “science” position within the agency, could provide an incentive for collaborating and partnering with researchers. Furthermore, partnerships between PHEPR practitioners and researchers could be strengthened by integrating expectations into Project Public Health Ready (PPHR), a criteria-based training and recognition program that assesses local health departments’ PHEPR capacity, or the Public Health Accreditation Board (PHAB) accreditation process. Additionally, establishing the trust of the community before a public health emergency occurs is critical to ensuring that research can be conducted effectively and equitably. The community also has many resources to offer the PHEPR research enterprise, including its experiences and knowledge of its needs and existing networks that can be leveraged. Overall, PHEPR research requires the collaboration, insight, and trust of professionals from public health, other response agencies, academia, private entities, and members of the community prior to a public health emergency, and an effective National PHEPR Science Framework will support strategies for strengthening and maintaining these partnerships to promote successful PHEPR research. Collaboration and engagement among practitioners, researchers, and the community is a critical element that is highlighted throughout this chapter—from the development of a research agenda and funding of research programs to participation in the design and conduct of the research and translation, dissemination, and implementation of evidence-based practices.

PHEPR Evidence-Based Guidelines Group and Other Efforts to Facilitate the Translation, Dissemination, and Implementation of PHEPR Research

If the PHEPR field is to be grounded in evidence, the translation, dissemination, and implementation of research findings represent a crucial component of a National PHEPR Science Framework. The research and other evidence driven by this framework will need to be translated into clear evidence-based practices for public health agencies through an ongoing evidence review process. Accordingly, it will be important for the PHEPR evidence-based guidelines group proposed in Recommendation 1 in Chapter 3 to be integrated into the activities of the National PHEPR Science Framework, and to review relevant research and distill it into guidelines for the benefit of practitioners.

The use of sustainable strategies and mechanisms, such as training specialists in translation and implementation science, particularly for the PHEPR field, can help bridge the often daunting gap between practice and research (Carbone and Thomas, 2018). Researchers need to engage with potential users of their research, involving them in the research design

and implementation process to increase the likelihood that results will be translatable and practiced (Jillson et al., 2019). At the same time, while practitioners’ perspectives and needs are an important consideration in developing research projects, practitioners also need to be accepting of innovations that emerge from the research community (Carbone and Thomas, 2018). Workforce capacity development programs are necessary to improve the implementation capacity of public health agencies. These issues are explored in depth in the sections on workforce capacity development for PHEPR research and practitioners and on translation of research into practice and dissemination and implementation of evidence-based PHEPR practices later in this chapter.

Ensuring Adequate Infrastructure and Supporting Mechanisms to Facilitate the Conduct of PHEPR Research

To conduct research effectively before, during, and following a public health emergency, additional capacities, infrastructure, policies, and other elements must be strengthened or created. Adequate and sustained federal funding for PHEPR research is necessary to ensure the continual flow of scientific discoveries to mitigate the health impacts of public health emergencies.

Recognizing that this is an applied research field, there needs to be an emphasis on funding mechanisms that facilitate practice-based approaches and support the collection of experiential evidence from real-world practice and public health emergencies. It is imperative to ensure that funded research programs are relevant to practice, and in Chapter 2, the committee concludes that, in addition to investments in research infrastructure, a coordinated and comprehensive approach to prioritizing and aligning research efforts and ensuring that research is relevant and consistently connected to practice is necessary to strengthen the PHEPR evidence base.

The best practitioner-centered evaluations will be achieved through trusting and durable partnerships between practitioners and researchers. Some example practice-based approaches that could be supported include a researcher residency model, a practice-based research network (PBRN) model, and a research-oriented tabletop exercise model. A researcher residency model (i.e., researchers embedded in the PHEPR system) could enable researchers to attend and observe exercises, have a seat in the emergency operations center during a public health emergency, and participate in the after action reporting. In learning health care systems, it has increasingly been recognized that embedding researchers in the system offers multiple benefits, including the identification of practitioner- and systems-relevant research questions and the ability to close the research and practice gap (both of which are persistent challenges in PHEPR) (Forrest et al., 2018). CDC could also consider requiring the inclusion of PHEPR practitioners in research proposals or implementing a PBRN model within PHEPR that might enable important advances in these areas. PBRNs—groups of practitioners and researchers working together to answer community-based questions and translate research findings into practice—are a result of the increasing need for research conducted in real-world settings (AHRQ, 2019; DeVoe et al., 2012; Mays et al., 2013; RWJF and University of Kentucky College of Public Health, 2020). PBRNs have previously been supported in public health (such a program was supported by the Robert Wood Johnson Foundation [RWJF] from approximately 2007 to 2015) (RWJF, 2013), and some of this work focused on PHEPR (CTEC, 2016; Wimsatt, 2017). Additionally, research-oriented tabletop exercises can aid in developing practice-driven research. For example, Chandra and colleagues (2015) developed a community resilience tabletop exercise and administered it to stakeholders from multiple disciplines to assess progress in community resilience and provide an opportunity for quality

improvement and capacity building. The NIEHS Disaster Research Response (DR2) program has previously conducted regional tabletop exercises for researchers and practitioners to facilitate the development of these partnerships and to educate practitioners on ways of incorporating data collection and research into disaster response (NIH, 2019b). Attention to emerging practice-based approaches is needed as well.

An equally important aspect of funding is mechanisms that allow for investigator-driven research, facilitate engagement and collaboration with researchers from different disciplines, and encourage the progression from development to intervention to secondary analysis to center grants—something that is currently lacking in the PHEPR field but is fundamental to any research enterprise. Several models for sustained funding programs for research series and multidisciplinary and collaborative research centers, as well research education and training projects (i.e., career development grants), currently exist at the National Science Foundation (NSF) and NIH and could be replicated for the PHEPR field (NIH, 2019c, 2020a; NSF, 2020b). CDC currently funds research through the use of grants and cooperative agreements (CDC, 2020b). Foundations, such as RWJF, the Alfred P. Sloan Foundation, the Bill & Melinda Gates Foundation, and the Open Philanthropy Project, among others, can support research programs when federal support is limited (Alfred P. Sloan Foundation, 2020; Bill & Melinda Gates Foundation, 2020; Open Philanthropy, 2020; RWJF, 2020a).

Addressing PHEPR knowledge gaps will require sustained lines of research, with multiple studies addressing similar research questions in different contexts and populations in nonemergency times and with the ability to refocus efforts or activate additional protocols, if warranted, in the event of a future public health emergency. Given the inherent challenges of conducting research during public health emergencies, it is important to give careful consideration to opportunities for advancing PHEPR science during nonemergency times and to the pre-event planning needed to enable research during and following future public health emergencies. Types of PHEPR research that could be supported in nondisaster times include modeling, simulations and exercises, and research on public health implementation issues that would likely translate in the event of a public health emergency. One of the challenging aspects of all PHEPR activities, including research, is the natural aversion to creating unused capacity. A solution to this challenge could be providing funding and resources for individuals conducting research in nonemergency times with the expectation—and support—that they will turn their attention to researching a public health emergency at hand. Research funders could consider requiring researchers to account for this possibility by describing such contingency processes in their proposals.

As mentioned above, deliberative planning during nonemergency times is necessary so that the resources and supporting mechanisms needed to rapidly conduct scientific research in the context of a public health emergency will be in place. Past efforts have made strides toward developing mechanisms to support scientific research in the context of a public health emergency, some of which have indeed been developed and even tested (e.g., rapid identification and prioritization of research needs and funding after Hurricane Sandy) (Lurie et al., 2013). In Table 8-1, Lurie and colleagues (2013) describe key components of an integrated approach to research before, during, and after an emergency and explicitly lay out actions that could be taken before as well as during the emergency. One consideration not included in this table is the action of developing metrics and outcome measures, an important component of the conduct of research and evaluation of practices. Future efforts to ensure adequate infrastructure and supporting mechanisms to facilitate the conduct of research during public health emergencies could be guided and informed by these past efforts.

To support research during a public health emergency, sustainable, rapid, and nimble funding mechanisms, together with award criteria and preapproved PHEPR research study

TABLE 8-1 Key Components of Research Response in the Context of Public Health Emergencies

SOURCE: Lurie et al., 2013. Reprinted with permission from Massachusetts Medical Society.

protocols, are needed. Several such mechanisms currently exist, but they are uncoordinated and focused disproportionately on infrastructure, engineering, and environmental health. An example is NSF’s Rapid Response Researich funding mechanism, used when quick-response research on disasters is needed (NSF, 2018). Additionally, with the support of NSF, the National Hazards Center at the University of Colorado administers a Quick Response Research Grant Program that provides small grants to help researchers collect data following an event (National Hazards Center, 2019). This program can be mobilized quickly to put some of the necessary research infrastructure in place in the immediate aftermath of a disaster and could serve as a model for funders, although it is at a smaller scale than that needed for the PHEPR research enterprise. NIEHS’s Time-Sensitive Research Grants Program is another example of a rapid funding mechanism for public health emergencies. Its aim is to receive, review, and fund research applications within 90–120 days, and it supports research to characterize initial exposures, collect human biological samples, and collect human health and exposure data (Miller et al., 2016). Other agencies could replicate these rapid response funding models specifically for PHEPR research. Additionally, partnerships with foundations that are interested in addressing the needs of communities and health-related research could help fill gaps in funding.

Rapid funding to support research in the event of a public health emergency is not enough, however; efforts are also needed to enhance capacities to conduct the research and improve data collection capabilities. Such efforts might include establishing formalized academic–public health agency research partnerships and a cadre of researchers and preassembled teams embedded in the response system—specifically within the incident management structure—and available to respond rapidly to public health emergencies (Lurie et al., 2013). In 2012, for example, the U.S. Department of the Interior’s Strategic Sciences group was created to meet the immediate need for scientific information during an environmental disaster (DOI, 2020). In the case of PHEPR, CDC could build research expertise into the training for Epidemic Intelligence Service (EIS) officers, a subset of whom could then be deployed for the sole purpose of conducting PHEPR research during public health emergencies (CDC, 2020a). A benefit of doing so would be that EIS officers are often most knowledgeable about needs and opportunities in communities and have the relationships to carry out the necessary research and evaluations in real time. They can be mobilized quickly and are

positioned to have their results incorporated rapidly into guidance, funding, and translation efforts. With respect to data collection, the NIEHS DR2 program has a central repository¹ of publicly available data collection tools, such as surveys, questionnaires, and protocols, that could be used to establish early baselines and cohorts for research.

Also important is to synergize and catalog past, ongoing, and future PHEPR-related research to avoid duplication (unless warranted) and reduce participant burden when research is conducted during or following a public health emergency. A database of PHEPR research studies categorized to facilitate analysis could be created, which would help foster research that would progressively improve PHEPR. Establishing and incentivizing partnerships and networks across research teams and institutions, other disciplines, and other organizations could also help coordinate research efforts.

Research involving human subjects during or following public health emergencies may pose ethical and data sharing challenges (Packenham et al., 2017). The Public Health Emergency Research Review Board was established in 2012 to provide centralized, rigorous, and expeditious reviews of human subject protections for HHS-conducted, -supported, or -regulated research studies addressing public health emergencies (HHS and NIH, 2020). This entity is currently under the auspices of the NIH network of institutional review boards (IRBs), and an IRB Authorization Agreement between NIH and the institutions conducting research is required (IOM, 2015). The NIEHS Best Practices Working Group for Special IRB Considerations in the Review of Disaster-Related Research has also been making progress in this area. The DR2 program has developed a pre-event generic protocol, the Rapid Acquisition of Pre- and Post-Incident Disaster Data (RAPIDD) protocol, for provisional approval by IRBs, which has been used by several universities. The objective of the RAPIDD protocol is to facilitate the collection of epidemiologic information and laboratory test results and the collection and storage of human biospecimens (Miller et al., 2016).

When it comes to research, public health agencies have different sets of concerns from researchers and institutions. A key issue is the security of confidential data and the privacy of subjects. Research during or following public health emergencies can also raise ethical challenges, including the burden on the population, potential harms, and the potential for therapeutic misconception (IOM, 2015). Further guidance and support are needed for academic entities and public health agencies to develop effective and efficient means for reviewing and addressing unique ethical issues in the conduct of PHEPR research, such as through pre-emergency review of standard protocols, training of IRB members on unique aspects of PHEPR research, and the establishment of specific review mechanisms for this research.

Conclusion and Recommendations

___________________

¹ See https://dr2.nlm.nih.gov/tools-resources (accessed March 10, 2020).

holders to develop a National PHEPR Science Framework so as to ensure resourcing, coordination, monitoring, and execution of public- and private-sector PHEPR research. The National PHEPR Science Framework should do the following:

• Build on and improve coordination, integration, and alignment among existing PHEPR research efforts (e.g., the National Institute of Environmental Health Sciences’ Disaster Research Response Program), and ensure integration of these efforts with the activities of the PHEPR evidence-based guidelines group proposed in Recommendation 1.
• Recognize and support PHEPR science as a unique academic discipline within the broader public health field to address the substantial need for research and diverse and qualified researchers.
• Create a common, robust, and forward-looking PHEPR research agenda that supports advancement beyond traditional epidemiological research to include research in the fields of social science, implementation science, complex interventions, and quality improvement, as well as intervention, operations, systems, and cost-effectiveness research.
• Support meaningful partnerships between PHEPR practitioners and researchers, and develop strategies to better ensure that PHEPR research is relevant to practice.
• Prioritize sustainable strategies and mechanisms for the translation, dissemination, and implementation of PHEPR research.

RECOMMENDATION 4: Ensure Infrastructure and Funding to Support Public Health Emergency Preparedness and Response (PHEPR) Research

The Centers for Disease Control and Prevention (CDC), in collaboration with other relevant funding agencies, should ensure adequate and sustained oversight, coordination, and funding to support a National PHEPR Science Framework and to further develop the infrastructure necessary to support more efficient production of and better-quality PHEPR research. Such infrastructure should include

• sustained funding for practice-based and investigator-driven research that allows for the progression from exploratory to effectiveness to scale-up research and encourages researcher diversity;
• support for partnerships (e.g., with academic institutions, hospital systems, and state, local, tribal, and territorial public health agencies) to facilitate collaboration in research on the preparedness, response, and recovery phases of a public health emergency;
• development of a rapid research funding mechanism and interdisciplinary rapid response teams with applied research expertise (similar to CDC’s Epidemic Intelligence Service) for deployment to conduct just-in-time studies related to the implementation of PHEPR practices at the time of events; and
• enhanced mechanisms to enable routine, standardized, efficient data collection with minimal disruption to delivery of services (including preapproved, adaptable research and institutional review board protocols and a research arm within the response structure).

SUPPORTING METHODOLOGICAL IMPROVEMENTS FOR PHEPR RESEARCH

The results of the committee’s four evidence reviews highlight the paucity of research that has generated credible evidence to inform PHEPR practice (see Chapters 4–7). In some cases, the committee found few to no studies with which to address the questions of interest (e.g., no quantitative studies were included in the review for emergency operations coordination). In other cases, a sizable body of research exists, but the committee noted limitations in study designs (e.g., lack of baseline measurements or comparison groups, use of unvalidated or subjective and self-reported measures), execution (e.g., underpowered studies, high loss to follow-up), analysis (e.g., failure to conduct statistical tests or lack of statistical adjustment), or reporting of information (e.g., lack of details on the methodology, the PHEPR practice, population characteristics, other contextual factors, and outcomes). Some research methods were also poorly matched to the research questions they were intended to address. To help ensure that future studies yield results from which stronger conclusions can be drawn about the effectiveness of PHEPR practices, future investments in PHEPR research will need to remedy these common methodological shortcomings. Standards, guidance, and incentives can help raise the quality and evidentiary value of research in the PHEPR field.

Common Evidence Guidelines

Federal agencies can have a significant influence on the generation of the evidence base for practice (Maynard, 2018). In fields other than PHEPR setting priorities and standards for research and using them to guide funding decisions has improved the quality and usefulness of the evidence base. An example is the response of the education research field to shifts in funding priorities to align with agency evidence agendas and guidelines, which began in 2002 with the creation of the Institute of Education Sciences (IES) within the U.S. Department of Education.² Early on, IES established evidence guidelines for causal inference studies; a system for sourcing, grading, and synthesizing evidence; a web-based clearinghouse for evidence reviews; and an active program of funding for professional development (pre- and postdoctoral training grants; professional meetings and association development). These efforts resulted in a dramatic shift in the methodological rigor of education evaluations (Whitehurst, 2018).

Similar improvements could be achieved in PHEPR by drawing on these experiences in other fields to enact policies and practices that can improve how PHEPR research is conducted, disseminated, and used (IES and NSF, 2013). The goal is to ensure that scarce evaluation dollars are used most productively to advance the evidence available to inform policy and practice. Achieving this goal necessitates careful balancing of several factors: the importance of the questions studied, the rigor with which the questions can and will be studied, the timeliness of the research findings, and the accessibility and usability of the findings. Tiered evidence standards for grantmaking can be a useful mechanism to guide funding decisions, as they allow federal agencies to award smaller amounts to promising concepts and larger amounts to practices grounded in strong evidence of success, encouraging innovation while still rewarding programs with robust research backing (GAO, 2016).

Going forward, the PHEPR research field will need to have clear guidelines and standards for evaluation methods and study designs that will produce credible answers to various types of questions of importance to the field. The objective is to encourage a balance

___________________

² Education Sciences Reform Act. HR 3801, Section 116. 107th Cong. (January 23, 2002).

of research throughout the knowledge-building continuum, from basic science through effectiveness trials and modeling studies, and to foster rather than stifle research innovation.

Guiding the Use of Different Types of Research Methods and Approaches

Well-crafted guidance will incorporate the full spectrum of research methods, which may range from exploratory case studies to RCTs and modeling studies for evaluating PHEPR practices. The PHEPR research field would be strengthened by creating a unified taxonomy of research methods, accompanied by guidelines for judging the credibility of study findings intended to address various types of questions. A first step in developing this guidance will be to identify the various genres of PHEPR research, and for each genre describe its purpose (i.e., how that type of research contributes to the evidence base) (see Annex 8-1). It will also be important for each genre of research to be supported by theoretical and empirical justifications when possible, and to adhere to established expectations for research design, methods, and products of the research. Expectations will need to be established as well for review of the products of each type of research (i.e., what information is required to judge the credibility and applicability of the findings and how that information can be judged).

While acknowledging the value of randomization for demonstrating a causal link between interventions and outcomes, the committee recognizes that it is difficult, if not infeasible or inappropriate, to implement RCTs for some PHEPR practices, particularly in the context of a real public health emergency. As discussed in Chapter 3, other study types (e.g., quasi-experimental study designs) may provide credible estimates of a causal impact (or lack thereof) when PHEPR practices are evaluated. Table 8-2 describes the strengths and limitations of common study designs for quantitative impact evaluation with applicability to PHEPR.

Other experimental study designs have begun to emerge that may also present opportunities for PHEPR research. Adaptive platform trial designs, such as the vaccine trial proposed during the 2014–2015 Ebola outbreak, allow for flexibility along the timeline of an event, with interim analysis of data to enable investigators to determine whether to continue moving forward, change course, or divest more rapidly from interventions that are not showing promise (Berry et al., 2016). Pragmatic trials evaluate the effectiveness of interventions in real-life routine practice rather than the highly controlled conditions typical of experimental research studies (Ford and Norrie, 2016; Patsopoulos, 2011). This design incorporates more real-world evidence into controlled trials, allowing for a broader environment for testing, as well as greater generalizability of findings. A stepped wedge cluster design is a type of pragmatic trial used to evaluate the efficacy of service delivery interventions. This type of design may be a good option when operating within logistical or political constraints and has been used in a variety of areas, ranging from vaccine development to social policy and criminal justice (Hemming et al., 2015).

On the other hand, the committee encourages the PHEPR field to move beyond experimental study designs and consider a broader range of methods for exploring what works (and when, why, and for whom). Many PHEPR practices are designed to improve outcomes, particularly systems-level outcomes, in complex settings in response to unpredictable events. PHEPR practitioners and researchers are often interested in whether a practice made a difference or what would have happened had it not been implemented (e.g., what would have happened had the public health emergency operations center not been activated). Qualitative research methodologies (e.g., ethnographic observations, interviews, and focus group discussions) can inform why and how PHEPR practices may or may not be effective (Teti et

TABLE 8-2 A Brief Overview of Strengths and Limitations of Study Designs for Quantitative Impact Evaluation for PHEPR

al., 2020). A range of approaches are gaining recognition, such as realist evaluations³ and qualitative comparative analyses,⁴ that acknowledge the complexity of causality (Blanchet et al., 2018). There is also defined guidance for evaluating complex interventions (e.g., the UK Medical Research Council, the Patient-Centered Outcomes Research Institute), but these concepts have yet to be fully adopted by the PHEPR research field (AcademyHealth, 2017; MRC, 2019; PCORI, 2019). Furthermore, as PHEPR research is transdisciplinary, design methodologies used in such fields as public health services and systems research, operations research, behavioral and social sciences, organizational research, and quality improvement can also provide evidence for understanding PHEPR practices. In particular, simulation-based research methods (e.g., tabletop exercises), systematic expert opinion methodologies (e.g., Delphi), and systems science approaches (e.g., social network analyses, causal process diagrams, adaptive systems theories, modeling, machine learning, and big data analyses) can provide insight on systems-level outcomes and the interdependent relationships among the many components of the PHEPR system. Overall, there are many rigorous methodologies from diverse fields that could be used to evaluate PHEPR practices, and the key takeaway is to match the study design appropriately to the research question to produce credible answers. Annex 8-1 provides a brief summary of genres of research, example research questions, and some appropriate methods.

Comprehensive guidance would include suggestions for strategically mixing methods to improve both the design of intervention studies (e.g., through baseline studies conducted before a PHEPR practice is implemented) and understanding of the findings, including their breadth and limitations (postintervention). An example of such a strategic mixed-method approach in PHEPR is the Los Angeles County Community Disaster Resilience (LACCDR) project⁵ (see Figure 8-3). The PHEPR field could benefit from conducting sequential or parallel studies focused on particular aspects of PHEPR practices. It is also important to note that the LACCDR project used community participatory methods, and the committee’s evidence review on engaging with and training community-based partners (see Chapter 4) found that community and stakeholder involvement in research and programmatic efforts from conceptualization to implementation may correspond with more effective engagement and training through enhanced inclusion, cultural acceptability, shared ownership, and capacity building of community members. Comprehensive guidance would incorporate such participatory methods, and also refer to their use in such emerging fields as engagement science (Dungan et al., 2019).

Standards for Reporting of Study Information

It is essential for all intervention studies to have well-articulated research plans that, when possible, are published before the analysis itself begins (Burlig, 2018; Lupia and Alter, 2014; Moravcsik, 2014). Such plans describe the study design, identify the primary and supplemental research questions to be addressed, provide background on the study setting,

___________________

³ “Realist evaluations are based on an assumption that projects and programs work under certain conditions and are influenced by the way that different stakeholders respond to them. Realist evaluations attempt to answer questions such as what works, for whom, in which circumstances, and why. They are designed to improve understanding about how development interventions work in different contexts” (INTRAC, 2017b).

⁵ See http://www.laresilience.org (accessed May 11, 2020).

⁴ “Qualitative Comparative Analysis (QCA) is a methodology that enables the analysis of multiple cases in complex situations. It can help explain why change happens in some cases but not others. QCA is designed for use with an intermediate number of cases, typically between 10 and 50. It can be used in situations where there are too few cases to apply conventional statistical analysis” (INTRAC, 2017a).

define the target population(s), and explain why the proposed program may change practice and improve decision making and outcomes for PHEPR practitioners or the community. They also detail the data collection plan, including measures to be used, and describe the analysis and reporting plans.

Given that PHEPR research funding and prioritization efforts are currently fragmented, disorganized, and inconsistent, there is no standardized peer-reviewed grant process, and as a result there are currently no specific standard guidelines or benchmarks for reporting the results of evaluations of the effectiveness of PHEPR practices. Reporting guidelines for health-related research have been developed for RCTs (Begg et al., 1996), observational studies in epidemiology (von Elm et al., 2007), systematic reviews of complex interventions (Guise et al., 2017a,b), studies of diagnostic accuracy (Bossuyt et al., 2015), qualitative research (O’Brien et al., 2014; Tong et al., 2007), implementation studies (Pinnock et al., 2015), and quality improvement studies (Ogrinc et al., 2016), among others (Hoffmann et al., 2014; Simera et al., 2008). Given the experience with a wide range of other types of research, it appears likely that developing, publishing, and disseminating tailored guidelines for PHEPR evaluations might well improve the reporting of such studies.

Federal agencies can support standardized reporting (e.g., through the development of guidance and standards and requirements linked to grants), which improves the usability of results and may over time result in efficiencies and cost savings (Maynard, 2018). Professional associations and journals also have important roles in the adoption of and commitment to reporting standards. PHEPR professional associations could establish the need and advocate for well-defined reporting standards, gather and review standards developed by other fields, draft standards for use by journals, and ensure that standards are shared and understood by the PHEPR research and practice fields. Journals play a vital role in communicating research findings to practitioners, as well as making information available to those in other sectors. By requiring the use of reporting standards, they can also promote the transparency and reproducibility of scientific research.

Conclusion and Recommendation

IMPROVING SYSTEMS TO GENERATE HIGH-QUALITY EXPERIENTIAL EVIDENCE FOR PHEPR

Public health agencies typically conduct after action reviews following real or simulated (exercise) public health emergencies in an effort to identify lessons learned and strengths and weaknesses of the response, and ultimately to improve emergency preparedness and response capabilities (Davies et al., 2019). The after action review process is an important source of experiential evidence in PHEPR and is the primary approach used by public health agencies to evaluate public health emergency response. In evaluation of the effectiveness of PHEPR practices, AARs offer the potential for improved understanding of context and implementation considerations that could be difficult to obtain through research. AARs can also be used to develop theories and logic models to inform future research. However, because they are not designed to be research, they are not without their methodological limitations. To help ensure that future AARs result in more useful and meaningful information for the evaluation of PHEPR practices (including the establishment of credible baselines for evaluation), it will be necessary to focus on strengthening methodological approaches, establishing mechanisms for analysis and dissemination of lessons learned from the reviews, and fostering a culture of improvement.

In the United States, several agencies and organizations that fund or oversee aspects of PHEPR, including CDC and ASPR, formally require after action reviews. The Federal Emergency Management Agency’s (FEMA’s) Homeland Security Exercise and Evaluation Program (HSEEP) developed a framework for agencies, including public health agencies, to use when developing, executing, and evaluating exercises. This program provides AAR templates and guidance to inform agencies in documenting strengths, areas for improvement, and corrective actions (FEMA, 2013). Because the framework is not organized according to the CDC PHEPR Capabilities, however, public health agencies have had to make major adaptations, such that the potential advantages of a standardized AAR have not been realized (Barnett et al., 2020). Thus, current reporting requirements and methodological standards for AARs lack clarity and uniformity. Moreover, evaluations are rarely conducted by independent evaluators with appropriate expertise (Davies et al., 2019; Gossip et al., 2017). AARs are typically reviewed and vetted throughout the agency or agencies that produced them before being submitted or shared with partners or the public, which may limit the candor of the information they contain (Gossip et al., 2017). The result is significant variability in the quality and reliability of AARs. The lack of consistent reporting requirements and variable report structures, together with limited CDC and public health agency resources, impedes the aggregation of AAR data and thus their use as a potential source of evidence for evaluating the effectiveness of PHEPR practices.

Limitations of AARs as a Source of Experiential Evidence for Mixed-Method Evidence Reviews

In recognition of the potential of AARs to inform the effectiveness and implementation of PHEPR practices in different contexts, the committee considered evidence from AARs in two of its evidence reviews. Davies and colleagues (2019) recently developed an appraisal tool with which to compare methodological reporting and document validity for AARs (see Box 8-2). To inform efforts focused on improvements needed to enhance the evidentiary

value of AARs for future use, the committee commissioned a quality assessment of the 38 AARs included in its evidence reviews using this 11-item appraisal tool (Patel, 2019).^6,7

Overall, the application of the tool to 38 AARs yielded low scores. Notably, consultants wrote two of the three highest-quality AARs, and AARs based on real events were of better quality on average than those based on exercises. The vast majority of AARs failed to provide a rationale for data selection, and more than half provided no detail on information sampling or multiple data sources, making it difficult to ascertain the appropriateness of the sample or sources used to inform the AAR findings. Practitioners were often surveyed for comments and observations, but the sample size of those practitioners, the timeline for data collection (immediate postevent versus after a reflection period), the information collected, and the format for collection varied widely and were rarely documented. Limiting samples to response leadership potentially skewed findings toward a leadership perspective at the cost of including feedback from staff engaged more directly in response operations. The grouping of leadership with general staff in feedback sessions could have discouraged staff from fully expressing any critiques they may have had regarding how leadership handled a response. Excluding communities from the after action review process also represented a missed opportunity to hear from diverse voices that might not have been reflected in the demographics of the leadership or staff. None of the AARs mentioned negative case analysis or respondent validation. Only three described peer debriefing and support; two of these were written by consultants, and one validated regional findings at the state level.

Overall, findings from applying the AAR quality assessment tool indicate a significant need to improve both after action review processes and the level of detail included in the reports themselves. It is unclear whether AAR authors omitted basic methodological information in a process that was otherwise rigorous, or if the reports would have scored low even if the requisite categories had been included.

In addition to these methodological shortcomings, the committee noted several other gaps and biases in its review of AARs and the AAR generation process that will need to be addressed moving forward:

• local political pressures and fear of judgment or retribution for reporting errors or negative outcomes;
• retrospective, subjective reporting based on the recall of participants, which may be influenced by the experience itself, pressures to “move on” and resume usual workflow, and limited roles in and siloed views of the activities;
• lack of methodological standards and tools for collecting, aggregating, analyzing, and disseminating information and reports;
• limited access to and the variable quality of data, information, and reports; and
• limited formal training, infrastructure, and resources to develop specialized personnel and/or programs to critically analyze AAR data and information in a culture of quality improvement.

___________________

⁶ This section draws heavily on a report commissioned by the committee on “Quality Assessment of After Action Reports: Findings and Recommendations,” by Sneha Patel.

⁷ The committee included 38 AARs in its evidence reviews. Approximately 61 percent of those reports were based on real events, and full-scale and functional exercises accounted for 16 percent and 21 percent of the reports, respectively. Hazards and threats included infectious diseases (e.g., H1N1, Ebola, hepatitis A), natural disasters, and human-made disasters (e.g., oil spills, explosions). Incident years ranged from 2009 to 2017 in 20 U.S. states. The AARs were all published either in the same year or the year following the real event or exercise.

Strengthening Methodological Approaches

Given the shortcomings discussed above, there is a clear need for CDC, in collaboration with ASPR and FEMA, to develop after action review policies and guidance that will ensure the capture of data relevant to the evolving response to a public health emergency and allow for in-depth analysis of the response. After action reviews can serve multiple purposes, including continuous quality improvement and, in some cases, accountability as part of grant requirements (Savoia et al., 2012; Stoto et al., 2013), and these purposes often require different methodological approaches to data collection, aggregation, and analysis. After action reviews are frequently completed by public health agencies themselves, and the methods used to collect information and data for the reports vary widely from agency to agency. In general, though, after action reviews use a wide variety of fairly common qualitative and quantitative methods, including surveys, interviews, focus groups or hotwashes, workshops, public forums, document reviews, and site visits (ECDC, 2018).

There is broad agreement that an after action review should seek to establish more than the immediate cause of response and recovery issues, and should analyze the factors behind the immediate causes, aiming to get to the root causes (Barnett et al., 2020; Davies et al., 2019; ECDC, 2018; Piltch-Loeb et al., 2014b; Singleton et al., 2014; Stoto et al., 2015, 2019) (see Figure 8-4 for an example depicting the steps of root-cause analysis). This systematic approach to root-cause analysis forms the basis of most approaches, such as peer assessment approaches⁸ and facilitated look-backs.⁹Gossip and colleagues (2017) note that public health agencies frequently utilize partner agencies and academic centers in the evaluation of exercises, but only rarely utilize this same expertise during or after a real-life response. Furthermore, while there is consensus that after action reviews should identify root causes, gaps remain in public health agencies’ use of this approach. An analysis of AARs conducted by Barnett and colleagues (2020) found several cases in which stated recommendations did not identify an underlying problem; an earlier analysis by Singleton and colleagues (2014) also highlighted the frequent failure of recommendations and corrective actions to include root-cause analysis.

AARs are at risk of the same biases as the qualitative and quantitative methods on which they rely, and findings from the committee’s commissioned quality assessment of AARs indicate that the reports typically omit the majority of important validity categories that could foster greater confidence in after action findings. Guidance aimed at improving after action review methods and the level of detail included in AAR methods sections is needed for both transparency and quality purposes, and AARs need to meet some minimum criteria concerning methods and reporting (Davies et al., 2019). The PHEPR field could benefit from drawing on the broader public health field to apply more rigorous evaluation processes when assessing lessons learned from public health emergencies. Training for evaluation participants, including academic programs in HSEEP certification and evaluation design, need to be encouraged and supported, if not required (Stoto et al., 2019). Standards and expectations regarding AARs could be strengthened by being integrated into PPHR (Summers and Ferraro, 2017). Similarly, PHAB could shape the evaluation of PHEPR by modifying its standards and measures to specifically include those that relate to PHEPR, thereby fostering efforts at quality improvement and evaluation (Brownson et al., 2018). Most important is for CDC and state,

___________________

⁸ The peer assessment approach employs an evaluation conducted by peers in similar jurisdictions. This approach offers the potential for objective analyses by PHEPR professionals and knowledge of the particularities of the system being assessed (Piltch-Loeb et al., 2014b).

⁹ The facilitated look-back approach uses a neutral facilitator and a no-fault approach to probe the nuances of decision making through moderated discussions (Piltch-Loeb et al., 2014b).

local, tribal, and territorial (SLTT) public health agencies, in addition to approaching AARs as an administrative requirement, to begin viewing them as a source of experiential evidence that could inform the development of evidence-based PHEPR practices and providing the necessary training, infrastructure, and resources to improve the quality of AARs produced.

Independent After Action Review Panel

According to Gossip and colleagues (2017), partnering with external organizations (e.g., peer agencies, consultants, academic centers) improves the depth and quality of documentation and assessments, as external organizations often have the requisite expertise and skills and capacity (e.g., time and personnel) to conduct or guide more rigorous evaluations. The United States has a strong history of creating objective, independent review panels (e.g., the National Transportation Safety Board and the National Institute of Standards and Technology’s National Construction Safety Team Advisory Committee), consisting of collaborative partners to develop guidelines, evaluate data and findings, and investigate incidents. These objective bodies have been proposed as a model for the PHEPR field (Barnett et al., 2020; Keim et al., 2019; Kirsch et al., 2018). These types of processes ensure objective expertise by eliminating the inherent biases of self-assessment and make use of consistent methods so that findings are comparable over time. Such a panel could review all events reaching the threshold of a Stafford Act or Public Health Service Act event. This process could be conducted by a newly established group, through existing professional associations, accrediting bodies, or regional academic partnerships and networks.

Essential Core Elements of a PHEPR AAR

To enable aggregation and analysis of AAR data for use as a potential source of evidence on the effectiveness of PHEPR practices, it is essential to define the core elements of a PHEPR AAR that builds on the existing HSEEP format but embraces more of a public health perspective. These elements would include a standardized core dataset and root-cause analysis framework that ultimately could be used not only by one jurisdiction, but also across jurisdictions for purposes of aggregation, trend analysis, and systemwide comparison (see Box 8-3 for the committee’s suggested elements for such an AAR template). Focusing on system-level root causes rather than specific problems would help make the experience more broadly applicable (i.e., enhance generalizability) (Stoto et al., 2019). An executive summary with high-level findings for each AAR would aid further in developing the empirical evidence base. An online platform prepopulating evaluation forms for practitioner-specific reporting objectives could also be developed (Agboola et al., 2015). Such a platform, with standardized questions at the national level, would assist practitioners in completing evaluations, integrate consistent measures into the review process, and aid in conducting trend analyses over time.

Establishing Mechanisms for Analysis and Dissemination of Lessons Learned from AARs

Proper conditions for establishing the utility and credibility of AARs as a less biased source are needed to improve the utility of these reports in evidence reviews and guidance for practical decision making. Improved mechanisms for public sharing of lessons learned from AARs, such as an enhanced national AAR repository, that specifically preclude the use of the reports for punitive purposes would foster more accurate and reliable information.

Currently, finding relevant and publicly available AARs that can serve as reliable sources of information and identifying useful observations within them is a notably labor-intensive process with variable yield. With the proper incentives and standards in place to encourage submission of AARs (e.g., formal recognition from the user community or appropriate agencies or qualification for additional exercise funding), an enhanced national AAR repository could be a useful tool for improving PHEPR (Kearns, 2010; Piltch-Loeb et al., 2014a; Turner et al., 2018).

FEMA’s Lessons Learned Information Sharing (LLIS) platform was developed to serve as a protected repository of AARs, reports of best practices, and lessons learned for all emergency response communities (HSDL, 2020). In 2015, LLIS was consolidated with the Homeland Security Digital Library (HSDL). Presently, the HSDL serves as the only repository for AARs, but submission of AARs to this database is voluntary, AARs include those submitted by sectors other than public health, other documents are also submitted, and the database is not easily searchable. These factors make it difficult to locate public health AARs to include in analyses.

A national AAR repository would ideally include indexed and searchable executive summaries and searchable high-level findings derived from AARs of federal- and state-supported exercises and tabletops, as well as responses to real events. This information would be organized and referenced by thematic content, findings, and type of event. To ensure a national AAR repository that is both feasible and useful, it is important to consider the following issues:

• the required resources and barriers to overcome for the creation, establishment, and curating of such a repository;
• the necessary infrastructure and protocols, which may include the creation and the adoption and assurance of submission standards, technology-based capture of key information from submitted AARs, and automated data aggregation and analysis;

• the incorporation of wide-ranging and user-friendly capabilities, which at a minimum would need to include systematic searchable parameters enabling identification of specific examples and details likely to emerge as themes across AARs;
• the ability to support analysis by verified users by providing channels for sharing anonymous or deidentified data;
• the ability to routinely audit and review entries, coupled with technical support to ensure accuracy and relevance; and
• mechanisms for response to user feedback and other continuous quality improvement measures.

Fostering a Culture of Quality Improvement

PHEPR programs reside in SLTT public health agencies, and because they are government entities, their reports are open to public access through local and state freedom of information laws. Reports filed with CDC are open to access through the Freedom of Information Act. Sensitivity to the release of security issues captured in reports, as well as concern about the exposure of vulnerabilities or weaknesses, has resulted in a reluctance to publish or even complete these reports. PHEPR practitioners are hesitant to report problems or weaknesses observed during a response because of the potential political ramifications and fear of punitive action (Gossip et al., 2017). Therefore, current agency culture leads to overlooking AARs as critical for informing evidence-based decision making. This situation stands in contrast to data used for health care quality assurance and protected in most states by “safe harbor” laws, data that have been cited as pivotal in participation in quality improvement initiatives by health care entities and practitioners (Gliklich et al., 2014). The PHEPR field would similarly benefit from a more open, structured, and meaningful culture of quality improvement that would enable the ongoing reporting and analysis of successes and challenges.

Conclusions and Recommendation

WORKFORCE CAPACITY DEVELOPMENT FOR RESEARCHERS AND PRACTITIONERS IN PHEPR

A robust and sustained commitment to workforce capacity development for both PHEPR researchers and practitioners is essential to advancing evidence-based practice in PHEPR. Investigators will have to produce research that is relevant to practitioners, and practitioners will have to turn routinely to research when making important decisions about implementing practices. Although there exists a network of researchers capable of conducting high-quality research on public health emergencies, these researchers require additional and ongoing training to design and implement studies and analyze and report study findings in a rigorous manner. It will also be important for practitioners to participate in or conduct evaluations. New skills, some going beyond traditional public health training, are often needed for practitioners to successfully conduct evaluations and identify and implement evidence-based practices (Brownson et al., 2018). Currently, the PHEPR workforce comprises both researchers and practitioners who often are educated and trained in schools of public health, but many other disciplines are represented as well, including physicians, nurses, social workers, veterinarians, and social scientists. Yet, there is little evidence that the graduate education and training these individuals receive in their respective domains is designed or tailored to address the unique challenges of conducting research in disaster-affected communities or during public health emergencies.

Researchers

Academic disaster research centers proliferated after the events of September 11, 2001, and funding for training for disaster researchers was readily available (see Chapter 2). Given the ongoing reduction in funding over the past several years, however, workforce attrition is likely. Academic researchers who moved into PHEPR research from public health and other related fields when funding was readily available may have left the PHEPR field when funding became more limited.

There exists today virtually no investment in the developmental pipeline for PHEPR researchers, a gap that corresponds to the relative dearth of funding opportunities for related research. The scarcity of PHEPR research and training grants inhibits the training of a new generation of researchers in the PHEPR field. In addition to funding for research awards, it is important to consider funding for training grants and career development awards. In the past, CDC has funded Mentored Research Scientist Development Awards (K01) to provide support for intensive research career development under the guidance of a mentor in areas addressing bioterrorism, other infectious disease outbreaks, and other public health threats and emergencies, among other areas (CDC, 2007). To ensure diverse, adequately trained, and sufficiently available disaster researchers, it will be necessary to invest in improved and sustained researcher training programs and grants (e.g., NIH career development K awards, RWJF leadership development programs), particularly those aimed at increasing PHEPR research capacity in the areas of implementation science and evaluation of complex interventions (NIH, 2020b; RWJF, 2020b). Researcher training programs can help direct training toward specific important areas; provide support for training in other, emerging areas; and establish research standards.

Additionally, special attention is essential to recruiting and supporting underrepresented minority researchers, including those who understand critical race theory, indigenous research methods, and other relevant frameworks, so that PHEPR science can advance equitably. Efforts are needed to connect doctoral and postdoctoral students early on in their careers so as to build relationships, connections, and teams to carry out ongoing PHEPR research projects. In a similar vein, mentoring programs involving senior and early-career researchers can support training for future generations. Many fields have recognized that providing means of supporting a steady infusion of highly trained researchers with new ideas is necessary to advance the quality of research (DHS, 2020; NCER, 2020; NIH, 2019a; NSF, 2020a).

Practitioners

As previously mentioned, PHEPR practitioners require the knowledge and skills not only to identify and implement evidence-based practices but also to conduct and participate in research and evaluation. In general, the public health workforce varies greatly in terms of job descriptions, education requirements, and experience, which are not standardized across public health agencies. A recent study found that having a public health degree was significantly associated with reduced odds of reporting a skill gap in identifying and applying evidence-based approaches to address a public health issue (Maddock, 2018; Taylor and Yeager, 2019).

A barrier to PHEPR practitioners working effectively with researchers may be that practitioners do not have the training or experience to identify programmatic and practice challenges that could be studied by researchers (Carbone and Thomas, 2018). Renewable certification programs, similar to the certified emergency manager program (IAEM, 2020), and a standardized training program for CDC project officers and state preparedness directors, as well as practitioners working in PHEPR more broadly, could ensure familiarity with evidence-based practices and promote the consistent creation and evaluation of real-world evidence as captured in AARs. CDC’s TRAIN Learning Network; ASPR’s Technical Resources, Assistance Center, and Information Exchange; and FEMA’s Emergency Management Institute provide a wealth of trainings and resources for PHEPR practitioners that could be leveraged and enhanced to provide practitioners with the knowledge and skills needed to identify and implement evidence-based practices and to conduct and participate in research and evaluation (ASPR, 2020; FEMA, 2020; PHF, 2020).

A number of other policy and operational barriers, including workload and responsibilities and priorities during a response, impede PHEPR practitioners’ abilities to conduct and participate in evaluations. Another model for addressing these barriers is to give practitioners access to external contractors with evaluation expertise that can provide tailored evaluation tools, appropriate training, and feedback on program evaluation plans and products (Maynard, 2018).

Recognizing that not everyone is equally suited or professionally able to be both a practitioner and researcher, it is necessary to develop stronger systems, infrastructure, and norms around the notion of an integrated PHEPR research and practice system that includes both those who are focused on advancing the science and those applying this knowledge. Platforms to provide ongoing technical assistance, peer networking, and collaboration across disciplines for both practitioners and researchers could ensure an interconnected workforce. Individuals who perceived themselves as part of this new interconnected PHEPR undertaking would share the mission of improving population health outcomes during public health emergencies and would be better positioned to interact with their colleagues across the traditional divides of practice and research.

Conclusion and Recommendation

TRANSLATION, DISSEMINATION, AND IMPLEMENTATION OF PHEPR RESEARCH TO PRACTICE

While there is a clear need to strengthen the evidence base for PHEPR practices through improvements in research and quality improvement processes, an equally pressing challenge is the translation, dissemination, and implementation of the evidence to practice (Carbone and Thomas, 2018). Numerous barriers impede the uptake of evidence-based practice, including varying awareness of the existing evidence base and lack of coordination between researchers and practitioners, lack of time, inadequate funding, inability to analyze and interpret evidence, and absence of cultural and managerial support (Brownson et al., 2018; Siegfried et al., 2017). Nonetheless, the absence of incentives has been found to be the greatest barrier, compounded by the disincentive of the time required (Jacobs et al., 2010). Such challenges are not unique to PHEPR. For example, the cultural shift toward evidence-based practice in health care evolved over decades, driven in part by financial and reputational liabilities for institutions and practitioners and increased accountability for clinical outcomes (Carbone and Thomas, 2018). While the adoption of evidence-based practices has been an ongoing challenge in the PHEPR field, significant investments made in implementation science more broadly across public health, health care, and other disciplines could guide the translation and implementation of research to practice for PHEPR (Brown et al., 2017; Brownson et al., 2012).

Recognizing the importance of this issue, CDC in 2015 initiated the Translation, Dissemination, and Implementation of Public Health Preparedness and Response Research and Training Initiative (TDI Initiative), a 2-year effort to consolidate and synthesize the research produced by the Preparedness and Emergency Response Research Centers and Preparedness and Emergency Response Learning Centers and translate that research into practice (Qari et al., 2018) (see Chapter 2). Table 8-3 describes best practices for the translation, dissemination, and implementation of PHEPR tools and resources that emerged from this initiative. Despite these efforts, however, barriers to the adoption of evidence-based PHEPR practices persist, related largely to capacity issues.

Building Implementation Capacity

Successful implementation requires that public health agencies have sufficient capacity to identify and translate scientific knowledge into practice (Brownson et al., 2018). Uneven infrastructure and capacity represent a long-standing challenge for public health systems (Baker and Koplan, 2002), and many SLTT public health agencies lack sufficient capacity (i.e., resources, infrastructure, and workforce) to identify and translate research and implement evidence-based practices. Fluctuations in public health funding are common and often result in smaller public health entities having to prioritize “keeping the doors open” and ensuring that critical public health services remain functional, leaving them with little to no financial means for investing in PHEPR training and implementation efforts (NORC at the University of Chicago, 2017). Furthermore, best practices currently are often not developed

TABLE 8-3 Best Practices for the Translation, Dissemination, and Implementation of Evidence-Based PHEPR Practices

SOURCES: Adapted from NORC at the University of Chicago, 2017; Qari et al., 2018.

with smaller or tribal or territorial public health agencies in mind, which frequently leads to implementation issues.

Addressing these barriers will require a multipronged and coordinated approach involving a combination of training, technical assistance, tools, and incentives (e.g., grant and funding requirements). The Interactive Systems Framework for Dissemination and Implementation offers a model for building public health agencies’ capacity to use best practices (Wandersman et al., 2008). The TDI Initiative utilized this framework, incorporating prevention support systems, which “provide information, training, technical assistance, or other support for practitioners,” to enhance program implementation efforts (Qari et al., 2018, p. S359).

Creating and Disseminating Practice-Ready Resources and Tools

Researchers are often ill equipped to translate research and resources for practitioners’ implementation. There is a clear need for tools and resources that are ready for “on the ground” use and disseminated in accordance with practitioner demand. Engaging translation and implementation specialists in the development of such tools and resources could help ensure that they are practice ready and can be implemented by practitioners at all levels

(Carbone and Thomas, 2018). For example, understanding that a translation gap persists between the availability of PHEPR tools and their adoption within public health agencies, Revere and colleagues (2018) explored with practitioners a process for identifying communication products to be considered as the highest implementation priority to improve communications in public health agencies. To increase awareness of existing resources and knowledge, CDC could provide dissemination support by deploying field agents to champion and disseminate evidence-based practices and tools across the PHEPR practice community (Carbone and Thomas, 2018).

Facilitating Tailoring of Evidence-Based Practices to Fit Local Contexts

Given differences among public health jurisdictions, including variability in contexts and capacity, it is essential for any implementation effort to recognize the importance of local needs and allow for the tailoring of practices to those needs (Arora et al., 2018; Baseman et al., 2018; Eisenman et al., 2018; Revere et al., 2018; Testa et al., 2018). Practitioners need appropriate training and a robust public health knowledge base to make appropriate and effective modifications to a practice so that it suits new settings or contexts (Eisenman et al., 2018). Eisenman and colleagues (2018) recently implemented a two-component intervention involving training and technical assistance and the creation and use of a guidebook to enhance the capacity of local public health agencies to translate and implement evidence-based practices in emergency preparedness. Evaluation of this program showed that after it was implemented, practitioners’ skills had significantly increased, indicating that such a program could be useful to other public health agencies. Existing tools are also available to decision makers to facilitate the implementation of evidence-based policies. For example, the SUPporting POlicy relevant Reviews and Trials tools are designed to help decision makers consider the applicability of evidence-based practices to their local conditions and tailor implementation strategies accordingly to address identified barriers and facilitators (Lavis et al., 2009; Lewin et al., 2009).

When a PHEPR practice is being considered for implementation in different contexts, identifying its core components can help determine what should remain intact and what can be modified without jeopardizing outcomes. An intervention’s core components are the basic principles and activities deemed necessary to generate desired outcomes, and they are rooted in its theoretical drivers of change (ASPE, 2013). Identifying specific core components can thus help practitioners tailor existing evidence-based practices to the local context. This tailoring is based on answers to such operational questions as: What activities are we going to conduct? Who is going to carry out the activity? What must people learn? and What resources are required? Additional research into the identification of core components of PHEPR practices would improve understanding of which components are essential for evidence-based practices to produce desired outcomes and how this is moderated by differences in population, setting, and other contextual factors, thus enabling practitioners to better operationalize interventions in their local setting. Encouraging the inclusion of core components in descriptions of evidence-based practices could also help ensure ongoing fidelity over time (ASPE, 2013). Funding for the translation, dissemination, and implementation of these identified core components would promote the replication and scalability of practices and the associated tools and resources across a variety of settings.

Incentivizing Adoption of Evidence-Based PHEPR Practices

Other changes to federal programming and policy could facilitate the implementation of evidence-based PHEPR practices. As discussed in Chapter 2, CDC’s Public Health Emergency Preparedness (PHEP) Cooperative Agreement represents the primary mechanism for funding by the federal government for SLTT public health agencies, accounting for the overwhelming majority of any preparedness funds at the SLTT level. However, current programmatic grant requirements do not sufficiently emphasize or support the generation of PHEPR evidence or the use of evidence-based practices, and PHEPR practitioners generally have no incentive to use evidence-based practices in their planning (CDC, 2019). To begin to bridge the gap between research and practice, CDC could incorporate the use of evidence-based practices (e.g., those identified by the PHEPR evidence-based guidelines group proposed in Recommendation 1 in Chapter 3) into the PHEP Cooperative Agreement, with linkage to CDC’s PHEPR Capabilities, asking grantees to use evidence-based practices where available and to justify why they are not doing so when they are not. It will be important to engage with state health officials as well as other key stakeholders with regard to this proposed strategy. Accreditation bodies such as PHAB and recognition programs such as PPHR are also positioned to act as drivers for embedding evidence-based practices into PHEPR programs, and could be leveraged to facilitate the translation, dissemination, and implementation of research to practice (Brownson et al., 2018; Summers and Ferraro, 2017).

Conclusion and Recommendation

• publishing evidence-based practices in CDC communication platforms (e.g., the Morbidity and Mortality Weekly Report, blogs) and partnering with public health professional organizations, such as the Association of State and Territorial Health Officials and the National Association of County & City Health Officials (NACCHO), to disseminate evidence-based practices;
• incorporating the requirement of utilizing evidence-based PHEPR practices into such processes as the Public Health Accreditation Board accreditation and such recognition programs as NACCHO’s Project Public Health Ready; and
• incorporating implementation science principles, such as the conduct of research to understand core components required for intervention effectiveness, into PHEPR research.

ANNEX 8-1
GENRES OF RESEARCH TO INFORM PUBLIC HEALTH EMERGENCY PREPAREDNESS AND RESPONSE PRACTICES

TABLE ANNEX 8-1 Genres of Research to Inform PHEPR Practices: Purpose, Example Research Questions, and Appropriate Methods

* For all genres of research, engaging end users and other stakeholders is advised when feasible.

REFERENCES

Carbone, E. G., and E. V. Thomas. 2018. Science as the basis of public health emergency preparedness and response practice: The slow but crucial evolution. American Journal of Public Health 108(S5):S383–S386.

CDC (Centers for Disease Control and Prevention). 2007. CDC mentored public health research scientist development award (K01). https://grants.nih.gov/grants/guide/rfa-files/RFA-CD-07-003.html#PartII (accessed February 12, 2020).

CDC. 2019. Public health emergency preparedness (PHEP) cooperative agreement. CDC-RFA-TP19-1901. https://www.grants.gov/web/grants/view-opportunity.html?oppId=310318 (accessed May 14, 2020).

CDC. 2020a. Epidemic intelligence service. https://www.cdc.gov/eis/index.html (accessed March 13, 2020).

CDC. 2020b. Grants. https://www.cdc.gov/grants/index.html (accessed May 13, 2020).

Cha, B. S., R. I. Lawrence, J. C. Bliss, K. B. Wells, A. Chandra, and D. P. Eisenman. 2016. The road to resilience: Insights on training community coalitions in the Los Angeles County Community Disaster Resilience Project. Disaster Medicine and Public Health Preparedness 10(6):812–821.

Chandra, A., M. Williams, A. Plough, A. Stayton, K. B. Wells, M. Horta, and J. Tang. 2013. Getting actionable about community resilience: The Los Angeles County Community Disaster Resilience Project. American Journal of Public Health 103(7):1181–1189.

Chandra, A., M. V. Williams, C. Lopez, J. Tang, D. Eisenman, and A. Magana. 2015. Developing a tabletop exercise to test community resilience: Lessons from the Los Angeles County Community Disaster Resilience Project. Disaster Medicine and Public Health Preparedness 9(5):484–488.

Chi, G. C., M. Williams, A. Chandra, A. Plough, and D. Eisenman. 2015. Partnerships for community resilience: Perspectives from the Los Angeles County Community Disaster Resilience Project. Public Health. http://www.escholarship.org/uc/item/30f4f1kf (accessed March 10, 2020).

CTEC (California Tribal Epidemiology Center). 2016. Health policy brief—Barriers to collaboration between tribal and county governments: Planning for major disasters and other emergencies. Sacramento, CA: California Tribal Epidemiology Center. http://www.publichealthsystems.org/sites/default/files/PHS4/72458GPreport_06.pdf (accessed May 14, 2020).

Davies, R., E. Vaughan, G. Fraser, R. Cook, M. Ciotti, and J. E. Suk. 2019. Enhancing reporting of after action reviews of public health emergencies to strengthen preparedness: A literature review and methodology appraisal. Disaster Medicine and Public Health Preparedness 13(3):618–625.

DeVoe, J. E., S. Likumahuwa, P. Eiff, C. A. Nelson, J. E. Carroll, C. N. Hill, R. Gold, and P. A. Kullberg. 2012. Developing a new practice-based research network (PBRN): Lessons learned and challenges ahead. Journal of the American Board of Family Medicine 25(5).

DHS (U.S. Department of Homeland Security). 2020. Welcome to the Centers of Excellence. https://www.dhs.gov/science-and-technology/centers-excellence (accessed February 21, 2020).

DOI (U.S. Department of the Interior). 2020. Strategic sciences group. https://www.doi.gov/strategicsciences (accessed May 22, 2020).

Dungan, R., R. Angove, E. Cope, and H. Peay. 2019. Engagement science: Introducing inclusive research practices and potential impacts. AcademyHealth. https://www.academyhealth.org/blog/2019-01/engagement-science-introducing-inclusive-research-practices-potential-impacts (accessed May 13, 2020).

ECDC (European Centre for Disease Prevention and Control). 2018. Best practice recommendations for conducting after action reviews to enhance public health preparedness. Solna, Sweden: European Centre for Disease Prevention and Control. https://www.ecdc.europa.eu/sites/default/files/documents/public-health-preparednessbest-practice-recommendations.pdf (accessed March 10, 2020).

Eisenman, D. P., R. M. Adams, C. M. Lang, M. Prelip, A. Dorian, J. Acosta, D. Glik, and M. Chinman. 2018. A program for local health departments to adapt and implement evidence-based emergency preparedness programs. American Journal of Public Health 108(S5):S396–S398.

FEMA (Federal Emergency Management Agency). 2013. Homeland security exercise and evaluation program. https://www.fema.gov/media-library-data/20130726-1914-25045-8890/hseep_apr13_.pdf (accessed April 1, 2019).

FEMA. 2020. Emergency management institute: Distance learning. https://training.fema.gov/is (accessed May 13, 2020).

Ford, I., and J. Norrie. 2016. Pragmatic trials. New England Journal of Medicine 375(5):454–463.

Forrest, C. B., F. D. Chesley, Jr., M. L. Tregear, and K. B. Mistry. 2018. Development of the learning health system researcher core competencies. Health Services Research 53(4):2615–2632.

GAO (U.S. Government Accountability Office). 2016. Tiered evidence grants: Opportunities exist to share lessons from early implementation and inform future federal efforts. Washington, DC: GAO. https://www.gao.gov/assets/680/679917.pdf (accessed March 10, 2020).

Gliklich, R., N. Dreyer, and M. Leavy. 2014. Protecting data: Confidentiality and legal concerns of providers, manufacturers, and health plans. In Registries for evaluating patient outcomes: A user’s guide, 3rd ed. Rockville, MD: Agency for Healthcare Research and Quality.

Gossip, K., H. Gouda, Y. Y. Lee, S. Firth, R. Bermejo, 3rd, W. Zeck, and E. Jimenez Soto. 2017. Monitoring and evaluation of disaster response efforts undertaken by local health departments: A rapid realist review. BMC Health Services Research 17(1):450.

Guise, J.-M., M. E. Butler, C. Chang, M. Viswanathan, T. Pigott, and P. Tugwell. 2017a. AHRQ series on complex intervention systematic reviews—Paper 6: PRISMA-CI extension statement and checklist. Journal of Clinical Epidemiology 90:43–50.

Guise, J.-M., M. Butler, C. Chang, M. Viswanathan, T. Pigott, and P. Tugwell. 2017b. AHRQ series on complex intervention systematic reviews—Paper 7: PRISMA-CI elaboration and explanation. Journal of Clinical Epidemiology 90:51–58.

Hemming, K., T. P. Haines, P. J. Chilton, A. J. Girling, and R. J. Lilford. 2015. The stepped wedge cluster randomised trial: Rationale, design, analysis, and reporting. BMJ 350:h391.

HHS (U.S. Department of Health and Human Services). 2017a. Public health emergency medical countermeasures enterprise (PHEMCE) strategy and implementation plan. https://www.phe.gov/Preparedness/mcm/phemce/Pages/strategy.aspx (accessed March 10, 2020).

HHS. 2017b. Public health emergency medical countermeasures enterprise multiyear budget: Fiscal years 2016–2020. https://www.phe.gov/Preparedness/mcm/phemce/phemce-myb/Pages/default.aspx (accessed March 10, 2020).

HHS and NIH (National Institutes of Health). 2020. Public health emergency research review board (PHERRB). https://ohsr.od.nih.gov/pherrb.php (accessed February 21, 2020).

Hoffmann, T. C., P. P. Glasziou, I. Boutron, R. Milne, R. Perera, D. Moher, D. G. Altman, V. Barbour, H. Macdonald, M. Johnston, S. E. Lamb, M. Dixon-Woods, P. McCulloch, J. C. Wyatt, A. W. Chan, and S. Michie. 2014. Better reporting of interventions: Template for intervention description and replication (TIDieR) checklist and guide. BMJ 348:g1687.

HSDL (Homeland Security Digital Library). 2020. Lessons learned information sharing. https://www.hsdl.org/c/llis-in-the-hsdl (accessed May 13, 2020).

IAEM (International Association of Emergency Managers). 2020. Certification introduction. https://www.iaem.org/certification/intro (accessed May 13, 2020).

IES and NSF (Institute of Education Sciences and National Science Foundation). 2013. Common guidelines for education research and development. https://www.nsf.gov/pubs/2013/nsf13126/nsf13126.pdf?WT.mc_id=USNSF_124 (accessed February 21, 2020).

INTRAC (International NGO Training and Research Centre). 2017a. Qualitative comparative analaysis (QCA). https://www.intrac.org/wpcms/wp-content/uploads/2017/01/Qualitative-comparative-analysis.pdf (accessed March 10, 2020).

INTRAC. 2017b. Realist evaluation. https://www.intrac.org/wpcms/wp-content/uploads/2017/01/Realist-evaluation.pdf (accessed March 10, 2020).

IOM (Institute of Medicine). 2008. Research priorities in emergency preparedness and response for public health systems: A letter report. Washington, DC: The National Academies Press.

IOM. 2015. Enabling rapid and sustainable public health research during disasters: Summary of a joint workshop by the Institute of Medicine and the U.S. Department of Health and Human Services. Washington, DC: The National Academies Press.

Jacobs, J. A., E. A. Dodson, E. A. Baker, A. D. Deshpande, and R. C. Brownson. 2010. Barriers to evidence-based decision making in public health: A national survey of chronic disease practitioners. Public Health Reports 125(5):736–742.

Jillson, I. A., M. Clarke, C. Allen, S. Waller, T. Koehlmoos, W. Mumford, J. Jansen, K. McKay, and A. Trant. 2019. Improving the science and evidence base of disaster response: A policy research study. BMC Health Services Research 19(1):274.

Kearns, S. 2010. Good Catch program encourages reporting near-miss medical errors. https://www.healthleadersmedia.com/clinical-care/good-catch-program-encourages-reporting-near-miss-medical-errors (accessed March 10, 2020).

Keim, M. E., T. D. Kirsch, O. Alleyne, G. Benjamin, L. DeGutis, D. Dyjack, and F. M. Burkle, Jr. 2019. The need for a national strategy to assess and reduce disaster-related mortality in the United States. American Journal of Public Health 109(4):539–540.

Kirsch, T. D., G. Benjamin, P. D. Kivela, D. T. Dyjack, and M. Keim. 2018. National Disaster Safety Board: Objective measures, improved outcomes. https://www.healthaffairs.org/do/10.1377/hblog20180720.168527/full (accessed March 10, 2020).

Lavis, J. N., A. D. Oxman, S. Lewin, and A. Fretheim. 2009. SUPPORT tools for evidence-informed health policymaking (STP). Health Research Policy and Systems 7(Suppl 1):I1. https://health-policy-systems.biomedcentral.com/articles/supplements/volume-7-supplement-1 (accessed March 10, 2020).

Lewin, S., A. D. Oxman, J. N. Lavis, A. Fretheim, S. Garcia Marti, and S. Munabi-Babigumira. 2009. SUPPORT tools for evidence-informed policymaking in health 11: Finding and using evidence about local conditions. Health Research Policy and Systems 7(Suppl 1):S11. https://health-policy-systems.biomedcentral.com/articles/10.1186/1478-4505-7-S1-S11 (accessed March 10, 2020).

Lupia, A., and G. Alter. 2014. Data access and research transparency in the quantitative tradition. Political Science and Politics 47(1):54–59.

Lurie, N., T. Manolio, A. P. Patterson, F. Collins, and T. Frieden. 2013. Research as a part of public health emergency response. New England Journal of Medicine 368(13):1251–1255.

Maddock, J. E. 2018. Preparing public health for the unexpected. American Journal of Public Health 108(S5):S348.

Maynard, R. A. 2018. The role of federal agencies in creating and administering evidence-based policies. The Annals of the American Academy of Political and Social Science 678(1):134–144.

Mays, G. P., R. A. Hogg, D. M. Castellanos-Cruz, A. G. Goover, and L. C. Fowler. 2013. Engaging public health settings in research implementation and translation activities: Evidence from practice-based research networks. American Journal of Preventive Medicine 45(6):752–762.

Miller, A., K. Yeskey, S. Garantziotis, S. Arnesen, A. Bennett, L. O’Fallon, C. Thompson, L. Reinlib, S. Masten, J. Remington, C. Love, S. Ramsey, R. Rosselli, B. Galluzzo, J. Lee, R. Kwok, and J. Hughes. 2016. Integrating health research into disaster response: The new NIH disaster research response program. International Journal of Environmental Research and Public Health 13(7).

Moravcsik, A. 2014. Transparency: The revolution in qualitative research. Political Science and Politics 47(1):48–53.

MRC (Medical Research Council). 2019. Developing and evaluating complex interventions. https://mrc.ukri.org/documents/pdf/complex-interventions-guidance (accessed February 21, 2020).

National Hazards Center. 2019. Quick Response research grant program. https://hazards.colorado.edu/research/quick-response (accessed January 9, 2020).

NBSB (National Biodefense Science Board). 2011. Call to action: Include scientific investigations as an integral component of disaster planning and response. https://www.phe.gov/Preparedness/legal/boards/nbsb/Documents/nbsbrec14.pdf (accessed March 10, 2020).

NCER (National Center for Education Research). 2020. Predoctoral interdisciplinary research training programs in the education sciences. https://ies.ed.gov/ncer/projects/program.asp?ProgID=16 (accessed March 10, 2020).

NIH (National Institutes of Health). 2019a. About the Division of Biomedical Research Workforce (DBRW). https://researchtraining.nih.gov/dbrw (accessed February 21, 2020).

NIH. 2019b. NIH disaster research response training and exercises. https://dr2.nlm.nih.gov/training-exercises#NIEHS_Disaster_Research_Response_Tabletop_Exercise_Assessment (accessed February 21, 2020).

NIH. 2019c. Types of grant programs. https://grants.nih.gov/grants/funding/funding_program.htm#RSeries (accessed May 13, 2020).

NIH. 2020a. P30 center core grant. https://www.niddk.nih.gov/research-funding/process/apply/funding-mechanisms/p30 (accessed May 13, 2020).

NIH. 2020b. Research career development awards. https://researchtraining.nih.gov/programs/career-development (accessed May 13, 2020).

NORC at the University of Chicago. 2017. Evaluation of the translation, dissemination, and implementation of public health preparedness response research and training project. Bethesda, MD: NORC at the University of Chicago. https://s3.amazonaws.com/ASPPH_Media_Files/Docs/CDC_Preparedness_Evaluation_Report (accessed March 10, 2020).

NSF (National Science Foundation). 2018. Proposal preparation instructions: Rapid research (RAPID) proposal. https://www.nsf.gov/pubs/policydocs/pappg18_1/pappg_2.jsp#IIE1 (accessed January 9, 2020).

NSF. 2020a. Accelerating discovery: Educating the future STEM workforce. https://nsf.gov/funding/pgm_summ.jsp?pims_id=505552&org=EHR&sel_org=EHR&from=fund (accessed February 21, 2020).

NSF. 2020b. About funding. https://www.nsf.gov/funding/aboutfunding.jsp (accessed May 13, 2020).

O’Brien, B. C., I. B. Harris, T. J. Beckman, D. A. Reed, and D. A. Cook. 2014. Standards for reporting qualitative research: A synthesis of recommendations. Academic Medicine 89(9):1245–1251.

O’Fallon, L. R., G. M. Wolfle, D. Brown, A. Dearry, and K. Olden. 2003. Strategies for setting a national research agenda that is responsive to community needs. Environmental Health Perspectives 111(16):1855–1860.

Ogrinc, G., L. Davies, D. Goodman, P. Batalden, F. Davidoff, and D. Stevens. 2016. Squire 2.0 (Standards for QUality Improvement Reporting Excellence): Revised publication guidelines from a detailed consensus process. BMJ Quality and Safety 25(12):986–992.

Open Philanthropy. 2020. Grants database. https://www.openphilanthropy.org/giving/grants (accessed May 13, 2020).

Packenham, J. P., R. T. Rosselli, S. K. Ramsey, H. A. Taylor, A. Fothergill, J. Slutsman, and A. Miller. 2017. Conducting science in disasters: Recommendations from the NIEHS working group for special IRB considerations in the review of disaster related research. Environmental Health Perspectives 125(9):094503.

Patel, S. 2019. Quality assessment of after action reports: Findings and recommendations. Paper commissioned by the Committee on Evidence-Based Practices for Public Health Emergency Preparedness and Response.

Patsopoulos, N. A. 2011. A pragmatic view on pragmatic trials. Dialogues in Clinical Neuroscience 13(2):217–224.

PCORI (Patient-Centered Outcomes Research Institute). 2019. PCORI methodology standards. https://www.pcori.org/research-results/about-our-research/research-methodology/pcori-methodology-standards#Complex (accessed February 21, 2020).

PHF (Public Health Foundation). 2020. Welcome to the TRAIN learning network. https://www.train.org/main/welcome (accessed May 13, 2020).

Piltch-Loeb, R., J. D. Kraemer, C. Nelson, and M. A. Stoto. 2014a. A public health emergency preparedness critical incident registry. Biosecurity and Bioterrorism: Biodefense Strategy, Practice, and Science 12(3).

Piltch-Loeb, R. N., C. D. Nelson, J. D. Kraemer, E. Savoia, and M. A. Stoto. 2014b. A peer assessment approach for learning from public health emergencies. Public Health Reports 129(Suppl 4):28–34.

Pinnock, H., E. Epiphaniou, A. Sheikh, C. Griffiths, S. Eldridge, P. Craig, and S. J. Taylor. 2015. Developing standards for reporting implementation studies of complex interventions (STaRI): A systematic review and e-Delphi. Implementation Science 10(42).

Qari, S. H., M. R. Leinhos, T. N. Thomas, and E. G. Carbone. 2018. Overview of the translation, dissemination, and implementation of public health preparedness and response research and training initiative. American Journal of Public Health 108(S5):S355–S362.

Revere, D., S. Allan, H. Karasz, and J. Baseman. 2018. Expanding methodologies to identify high-priority emergency preparedness tools for implementation in public health agencies. American Journal of Public Health 108(S5):S372–S374.

RWJF (Robert Wood Johnson Foundation). 2013. Robert Wood Johnson Foundation practice-based research networks in public health. https://www.rwjf.org/en/library/research/2013/06/robert-wood-johnson-foundation-practice-based-research-networks-.html (accessed May 13, 2020).

RWJF. 2020a. Grants and grant programs. https://www.rwjf.org/en/how-we-work/grants-and-grant-programs.html (accessed May 13, 2020).

RWJF. 2020b. Leadership for better health. https://www.rwjf.org/en/our-focus-areas/focus-areas/health-leadership.html (accessed May 13, 2020).

RWJF and University of Kentucky College of Public Health. 2020. The national coordinating center for public health services and systems research and public health practice-based research networks. http://www.publichealthsystems.org (accessed May 13, 2020).

Savoia, E., F. Agboola, and P. D. Biddinger. 2012. Use of after action reports (AARs) to promote organizational and systems learning in emergency preparedness. International Journal of Environmental Research and Public Health 9(8):2949–2963.

Siegfried, A. L., E. G. Carbone, M. B. Meit, M. J. Kennedy, H. Yusuf, and E. B. Kahn. 2017. Identifying and prioritizing information needs and research priorities of public health emergency preparedness and response practitioners. Disaster Medicine and Public Health Preparedness 11(5):552–561.

Simera, I., D. G. Altman, D. Moher, K. F. Schulz, and J. Hoey. 2008. Guidelines for reporting health research: The Equator Network’s survey of guideline authors. PLOS Medicine 5(6):e139.

Singleton, C. M., S. Debastiani, D. Rose, and E. B. Kahn. 2014. An analysis of root cause identification and continuous quality improvement in public health H1N1 after-action reports. Journal of Public Health Management and Practice 20(2):197.

Stoto, M. A., C. Nelson, M. A. Higdon, J. Kraemer, L. Hites, and C. M. Singleton. 2013. Lessons about the state and local public health system response to the 2009 H1N1 pandemic: A workshop summary. Journal of Public Health Management and Practice 19(5):428–435.

Stoto, M., A., R. Piltch-Loeb, and E. Savoia. 2015. The public health system response to the 2014 West Virginia water crisis. https://cdn1.sph.harvard.edu/wp-content/uploads/sites/1609/2015/01/WV-water-crisis-report-final.pdf (accessed March 10, 2020).

Stoto, M. A., C. Nelson, R. Piltch-Loeb, L. N. Mayigane, F. Copper, and S. Chungong. 2019. Getting the most from after action reviews to improve global health security. Globalization and Health 15(1):58. http://creativecommons.org/licenses/by/4.0 (accessed June 17, 2020).

Summers, S. K., and M. J. Ferraro. 2017. Project Public Health Ready: History and evolution of a best practice for public health preparedness planning. American Journal of Public Health 107(S2):S138–S141.

Taylor, H. L., and V. A. Yeager. 2019. Core competency gaps among governmental public health employees with and without a formal public health degree. Journal of Public Health Management Practice. [Epub ahead of print.]

Testa, M. A., E. Savoia, M. Su, and P. D. Biddinger. 2018. Social media learning collaborative for public health preparedness. American Journal of Public Health 108(S5):S375–S377.

Teti, M., E. Schatz, and L. Liebenberg. 2020. Methods in the time of COVID-19: The vital role of qualitative inquiries. International Journal of Qualitative Methods 19(1–5).

Tong, A., P. Sainsbury, and J. Craig. 2007. Consolidated Criteria for Reporting Qualitative Research (COREQ): A 32-item checklist for interviews and focus groups. International Journal for Quality in Health Care 19(6):349–357.

Turner, D. A., J. Bae, G. Cheely, J. Milne, T. A. Owens, and C. M. Kuhn. 2018. Improving resident and fellow engagement in patient safety through a graduate medical education incentive program. Journal of Graduate Medicine Education 10(6):671–675.

von Elm, E., D. G. Altman, M. Egger, S. J. Pocock, P. C. Gotzsche, and J. P. Vandenbroucke. 2007. The Strengthening the Reporting of OBservational studies in Epidemiology (STROBE) statement: Guidelines for reporting observational studies. Annals of Internal Medicine 147(8):573–577.

Wandersman, A., J. Duffy, P. Flaspohler, R. Noonan, K. Lubell, L. Stillman, M. Blachman, R. Dunville, and J. Saul. 2008. Bridging the gap between prevention research and practice: The interactive systems framework for dissemination and implementation. American Journal of Community Psychology 41(3–4):171–181.

Whitehurst, G. J. 2018. The Institute of Education Sciences: A model for federal research offices. The Annals of the American Academy of Political and Social Science 678(1):124–133.

Williams, M. V., A. Chandra, A. Spears, D. Varda, K. B. Wells, A. L. Plough, and D. P. Eisenman. 2018. Evaluating community partnerships addressing community resilience in Los Angeles, California. International Journal of Environmental Research and Public Health 15(4).

Wimsatt, M. A. 2017. Cross-jurisdictional sharing for emergency management-related public health: Exploring the experiences of tribes and counties in California. Frontiers in Public Health 5(254). https://doi.org/10.3389/fpubh.2017.00254.

This page intentionally left blank.