National Academies Press: OpenBook

Evidence-Based Practice for Public Health Emergency Preparedness and Response (2020)

Chapter: 8 Improving and Expanding the Evidence Base for Public Health Emergency Preparedness and Response

« Previous: 7 Implementing Quarantine to Reduce or Stop the Spread of a Contagious Disease
Suggested Citation:"8 Improving and Expanding the Evidence Base for Public Health Emergency Preparedness and Response." National Academies of Sciences, Engineering, and Medicine. 2020. Evidence-Based Practice for Public Health Emergency Preparedness and Response. Washington, DC: The National Academies Press. doi: 10.17226/25650.
×
Page 233
Suggested Citation:"8 Improving and Expanding the Evidence Base for Public Health Emergency Preparedness and Response." National Academies of Sciences, Engineering, and Medicine. 2020. Evidence-Based Practice for Public Health Emergency Preparedness and Response. Washington, DC: The National Academies Press. doi: 10.17226/25650.
×
Page 234
Suggested Citation:"8 Improving and Expanding the Evidence Base for Public Health Emergency Preparedness and Response." National Academies of Sciences, Engineering, and Medicine. 2020. Evidence-Based Practice for Public Health Emergency Preparedness and Response. Washington, DC: The National Academies Press. doi: 10.17226/25650.
×
Page 235
Suggested Citation:"8 Improving and Expanding the Evidence Base for Public Health Emergency Preparedness and Response." National Academies of Sciences, Engineering, and Medicine. 2020. Evidence-Based Practice for Public Health Emergency Preparedness and Response. Washington, DC: The National Academies Press. doi: 10.17226/25650.
×
Page 236
Suggested Citation:"8 Improving and Expanding the Evidence Base for Public Health Emergency Preparedness and Response." National Academies of Sciences, Engineering, and Medicine. 2020. Evidence-Based Practice for Public Health Emergency Preparedness and Response. Washington, DC: The National Academies Press. doi: 10.17226/25650.
×
Page 237
Suggested Citation:"8 Improving and Expanding the Evidence Base for Public Health Emergency Preparedness and Response." National Academies of Sciences, Engineering, and Medicine. 2020. Evidence-Based Practice for Public Health Emergency Preparedness and Response. Washington, DC: The National Academies Press. doi: 10.17226/25650.
×
Page 238
Suggested Citation:"8 Improving and Expanding the Evidence Base for Public Health Emergency Preparedness and Response." National Academies of Sciences, Engineering, and Medicine. 2020. Evidence-Based Practice for Public Health Emergency Preparedness and Response. Washington, DC: The National Academies Press. doi: 10.17226/25650.
×
Page 239
Suggested Citation:"8 Improving and Expanding the Evidence Base for Public Health Emergency Preparedness and Response." National Academies of Sciences, Engineering, and Medicine. 2020. Evidence-Based Practice for Public Health Emergency Preparedness and Response. Washington, DC: The National Academies Press. doi: 10.17226/25650.
×
Page 240
Suggested Citation:"8 Improving and Expanding the Evidence Base for Public Health Emergency Preparedness and Response." National Academies of Sciences, Engineering, and Medicine. 2020. Evidence-Based Practice for Public Health Emergency Preparedness and Response. Washington, DC: The National Academies Press. doi: 10.17226/25650.
×
Page 241
Suggested Citation:"8 Improving and Expanding the Evidence Base for Public Health Emergency Preparedness and Response." National Academies of Sciences, Engineering, and Medicine. 2020. Evidence-Based Practice for Public Health Emergency Preparedness and Response. Washington, DC: The National Academies Press. doi: 10.17226/25650.
×
Page 242
Suggested Citation:"8 Improving and Expanding the Evidence Base for Public Health Emergency Preparedness and Response." National Academies of Sciences, Engineering, and Medicine. 2020. Evidence-Based Practice for Public Health Emergency Preparedness and Response. Washington, DC: The National Academies Press. doi: 10.17226/25650.
×
Page 243
Suggested Citation:"8 Improving and Expanding the Evidence Base for Public Health Emergency Preparedness and Response." National Academies of Sciences, Engineering, and Medicine. 2020. Evidence-Based Practice for Public Health Emergency Preparedness and Response. Washington, DC: The National Academies Press. doi: 10.17226/25650.
×
Page 244
Suggested Citation:"8 Improving and Expanding the Evidence Base for Public Health Emergency Preparedness and Response." National Academies of Sciences, Engineering, and Medicine. 2020. Evidence-Based Practice for Public Health Emergency Preparedness and Response. Washington, DC: The National Academies Press. doi: 10.17226/25650.
×
Page 245
Suggested Citation:"8 Improving and Expanding the Evidence Base for Public Health Emergency Preparedness and Response." National Academies of Sciences, Engineering, and Medicine. 2020. Evidence-Based Practice for Public Health Emergency Preparedness and Response. Washington, DC: The National Academies Press. doi: 10.17226/25650.
×
Page 246
Suggested Citation:"8 Improving and Expanding the Evidence Base for Public Health Emergency Preparedness and Response." National Academies of Sciences, Engineering, and Medicine. 2020. Evidence-Based Practice for Public Health Emergency Preparedness and Response. Washington, DC: The National Academies Press. doi: 10.17226/25650.
×
Page 247
Suggested Citation:"8 Improving and Expanding the Evidence Base for Public Health Emergency Preparedness and Response." National Academies of Sciences, Engineering, and Medicine. 2020. Evidence-Based Practice for Public Health Emergency Preparedness and Response. Washington, DC: The National Academies Press. doi: 10.17226/25650.
×
Page 248
Suggested Citation:"8 Improving and Expanding the Evidence Base for Public Health Emergency Preparedness and Response." National Academies of Sciences, Engineering, and Medicine. 2020. Evidence-Based Practice for Public Health Emergency Preparedness and Response. Washington, DC: The National Academies Press. doi: 10.17226/25650.
×
Page 249
Suggested Citation:"8 Improving and Expanding the Evidence Base for Public Health Emergency Preparedness and Response." National Academies of Sciences, Engineering, and Medicine. 2020. Evidence-Based Practice for Public Health Emergency Preparedness and Response. Washington, DC: The National Academies Press. doi: 10.17226/25650.
×
Page 250
Suggested Citation:"8 Improving and Expanding the Evidence Base for Public Health Emergency Preparedness and Response." National Academies of Sciences, Engineering, and Medicine. 2020. Evidence-Based Practice for Public Health Emergency Preparedness and Response. Washington, DC: The National Academies Press. doi: 10.17226/25650.
×
Page 251
Suggested Citation:"8 Improving and Expanding the Evidence Base for Public Health Emergency Preparedness and Response." National Academies of Sciences, Engineering, and Medicine. 2020. Evidence-Based Practice for Public Health Emergency Preparedness and Response. Washington, DC: The National Academies Press. doi: 10.17226/25650.
×
Page 252
Suggested Citation:"8 Improving and Expanding the Evidence Base for Public Health Emergency Preparedness and Response." National Academies of Sciences, Engineering, and Medicine. 2020. Evidence-Based Practice for Public Health Emergency Preparedness and Response. Washington, DC: The National Academies Press. doi: 10.17226/25650.
×
Page 253
Suggested Citation:"8 Improving and Expanding the Evidence Base for Public Health Emergency Preparedness and Response." National Academies of Sciences, Engineering, and Medicine. 2020. Evidence-Based Practice for Public Health Emergency Preparedness and Response. Washington, DC: The National Academies Press. doi: 10.17226/25650.
×
Page 254
Suggested Citation:"8 Improving and Expanding the Evidence Base for Public Health Emergency Preparedness and Response." National Academies of Sciences, Engineering, and Medicine. 2020. Evidence-Based Practice for Public Health Emergency Preparedness and Response. Washington, DC: The National Academies Press. doi: 10.17226/25650.
×
Page 255
Suggested Citation:"8 Improving and Expanding the Evidence Base for Public Health Emergency Preparedness and Response." National Academies of Sciences, Engineering, and Medicine. 2020. Evidence-Based Practice for Public Health Emergency Preparedness and Response. Washington, DC: The National Academies Press. doi: 10.17226/25650.
×
Page 256
Suggested Citation:"8 Improving and Expanding the Evidence Base for Public Health Emergency Preparedness and Response." National Academies of Sciences, Engineering, and Medicine. 2020. Evidence-Based Practice for Public Health Emergency Preparedness and Response. Washington, DC: The National Academies Press. doi: 10.17226/25650.
×
Page 257
Suggested Citation:"8 Improving and Expanding the Evidence Base for Public Health Emergency Preparedness and Response." National Academies of Sciences, Engineering, and Medicine. 2020. Evidence-Based Practice for Public Health Emergency Preparedness and Response. Washington, DC: The National Academies Press. doi: 10.17226/25650.
×
Page 258
Suggested Citation:"8 Improving and Expanding the Evidence Base for Public Health Emergency Preparedness and Response." National Academies of Sciences, Engineering, and Medicine. 2020. Evidence-Based Practice for Public Health Emergency Preparedness and Response. Washington, DC: The National Academies Press. doi: 10.17226/25650.
×
Page 259
Suggested Citation:"8 Improving and Expanding the Evidence Base for Public Health Emergency Preparedness and Response." National Academies of Sciences, Engineering, and Medicine. 2020. Evidence-Based Practice for Public Health Emergency Preparedness and Response. Washington, DC: The National Academies Press. doi: 10.17226/25650.
×
Page 260
Suggested Citation:"8 Improving and Expanding the Evidence Base for Public Health Emergency Preparedness and Response." National Academies of Sciences, Engineering, and Medicine. 2020. Evidence-Based Practice for Public Health Emergency Preparedness and Response. Washington, DC: The National Academies Press. doi: 10.17226/25650.
×
Page 261
Suggested Citation:"8 Improving and Expanding the Evidence Base for Public Health Emergency Preparedness and Response." National Academies of Sciences, Engineering, and Medicine. 2020. Evidence-Based Practice for Public Health Emergency Preparedness and Response. Washington, DC: The National Academies Press. doi: 10.17226/25650.
×
Page 262
Suggested Citation:"8 Improving and Expanding the Evidence Base for Public Health Emergency Preparedness and Response." National Academies of Sciences, Engineering, and Medicine. 2020. Evidence-Based Practice for Public Health Emergency Preparedness and Response. Washington, DC: The National Academies Press. doi: 10.17226/25650.
×
Page 263
Suggested Citation:"8 Improving and Expanding the Evidence Base for Public Health Emergency Preparedness and Response." National Academies of Sciences, Engineering, and Medicine. 2020. Evidence-Based Practice for Public Health Emergency Preparedness and Response. Washington, DC: The National Academies Press. doi: 10.17226/25650.
×
Page 264
Suggested Citation:"8 Improving and Expanding the Evidence Base for Public Health Emergency Preparedness and Response." National Academies of Sciences, Engineering, and Medicine. 2020. Evidence-Based Practice for Public Health Emergency Preparedness and Response. Washington, DC: The National Academies Press. doi: 10.17226/25650.
×
Page 265
Suggested Citation:"8 Improving and Expanding the Evidence Base for Public Health Emergency Preparedness and Response." National Academies of Sciences, Engineering, and Medicine. 2020. Evidence-Based Practice for Public Health Emergency Preparedness and Response. Washington, DC: The National Academies Press. doi: 10.17226/25650.
×
Page 266
Suggested Citation:"8 Improving and Expanding the Evidence Base for Public Health Emergency Preparedness and Response." National Academies of Sciences, Engineering, and Medicine. 2020. Evidence-Based Practice for Public Health Emergency Preparedness and Response. Washington, DC: The National Academies Press. doi: 10.17226/25650.
×
Page 267
Suggested Citation:"8 Improving and Expanding the Evidence Base for Public Health Emergency Preparedness and Response." National Academies of Sciences, Engineering, and Medicine. 2020. Evidence-Based Practice for Public Health Emergency Preparedness and Response. Washington, DC: The National Academies Press. doi: 10.17226/25650.
×
Page 268
Suggested Citation:"8 Improving and Expanding the Evidence Base for Public Health Emergency Preparedness and Response." National Academies of Sciences, Engineering, and Medicine. 2020. Evidence-Based Practice for Public Health Emergency Preparedness and Response. Washington, DC: The National Academies Press. doi: 10.17226/25650.
×
Page 269
Suggested Citation:"8 Improving and Expanding the Evidence Base for Public Health Emergency Preparedness and Response." National Academies of Sciences, Engineering, and Medicine. 2020. Evidence-Based Practice for Public Health Emergency Preparedness and Response. Washington, DC: The National Academies Press. doi: 10.17226/25650.
×
Page 270
Suggested Citation:"8 Improving and Expanding the Evidence Base for Public Health Emergency Preparedness and Response." National Academies of Sciences, Engineering, and Medicine. 2020. Evidence-Based Practice for Public Health Emergency Preparedness and Response. Washington, DC: The National Academies Press. doi: 10.17226/25650.
×
Page 271
Suggested Citation:"8 Improving and Expanding the Evidence Base for Public Health Emergency Preparedness and Response." National Academies of Sciences, Engineering, and Medicine. 2020. Evidence-Based Practice for Public Health Emergency Preparedness and Response. Washington, DC: The National Academies Press. doi: 10.17226/25650.
×
Page 272
Suggested Citation:"8 Improving and Expanding the Evidence Base for Public Health Emergency Preparedness and Response." National Academies of Sciences, Engineering, and Medicine. 2020. Evidence-Based Practice for Public Health Emergency Preparedness and Response. Washington, DC: The National Academies Press. doi: 10.17226/25650.
×
Page 273
Suggested Citation:"8 Improving and Expanding the Evidence Base for Public Health Emergency Preparedness and Response." National Academies of Sciences, Engineering, and Medicine. 2020. Evidence-Based Practice for Public Health Emergency Preparedness and Response. Washington, DC: The National Academies Press. doi: 10.17226/25650.
×
Page 274
Suggested Citation:"8 Improving and Expanding the Evidence Base for Public Health Emergency Preparedness and Response." National Academies of Sciences, Engineering, and Medicine. 2020. Evidence-Based Practice for Public Health Emergency Preparedness and Response. Washington, DC: The National Academies Press. doi: 10.17226/25650.
×
Page 275
Suggested Citation:"8 Improving and Expanding the Evidence Base for Public Health Emergency Preparedness and Response." National Academies of Sciences, Engineering, and Medicine. 2020. Evidence-Based Practice for Public Health Emergency Preparedness and Response. Washington, DC: The National Academies Press. doi: 10.17226/25650.
×
Page 276
Suggested Citation:"8 Improving and Expanding the Evidence Base for Public Health Emergency Preparedness and Response." National Academies of Sciences, Engineering, and Medicine. 2020. Evidence-Based Practice for Public Health Emergency Preparedness and Response. Washington, DC: The National Academies Press. doi: 10.17226/25650.
×
Page 277
Suggested Citation:"8 Improving and Expanding the Evidence Base for Public Health Emergency Preparedness and Response." National Academies of Sciences, Engineering, and Medicine. 2020. Evidence-Based Practice for Public Health Emergency Preparedness and Response. Washington, DC: The National Academies Press. doi: 10.17226/25650.
×
Page 278
Suggested Citation:"8 Improving and Expanding the Evidence Base for Public Health Emergency Preparedness and Response." National Academies of Sciences, Engineering, and Medicine. 2020. Evidence-Based Practice for Public Health Emergency Preparedness and Response. Washington, DC: The National Academies Press. doi: 10.17226/25650.
×
Page 279
Suggested Citation:"8 Improving and Expanding the Evidence Base for Public Health Emergency Preparedness and Response." National Academies of Sciences, Engineering, and Medicine. 2020. Evidence-Based Practice for Public Health Emergency Preparedness and Response. Washington, DC: The National Academies Press. doi: 10.17226/25650.
×
Page 280
Suggested Citation:"8 Improving and Expanding the Evidence Base for Public Health Emergency Preparedness and Response." National Academies of Sciences, Engineering, and Medicine. 2020. Evidence-Based Practice for Public Health Emergency Preparedness and Response. Washington, DC: The National Academies Press. doi: 10.17226/25650.
×
Page 281
Suggested Citation:"8 Improving and Expanding the Evidence Base for Public Health Emergency Preparedness and Response." National Academies of Sciences, Engineering, and Medicine. 2020. Evidence-Based Practice for Public Health Emergency Preparedness and Response. Washington, DC: The National Academies Press. doi: 10.17226/25650.
×
Page 282

Below is the uncorrected machine-read text of this chapter, intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text of each book. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.

8 Improving and Expanding the Evidence Base for Public Health Emergency Preparedness and Response As described in preceding chapters, despite investments in public health emergency preparedness and response (PHEPR) research in the last two decades, the corresponding PHEPR evidence base remains limited (Carbone and Thomas, 2018). Figure 8-1 shows the study design distribution of all articles included in a committee-commissioned scoping review focused on the 15 PHEPR Capabilities (see Chapter 2 and Appendix D for additional detail on the commissioned scoping review). Among the 1,692 articles included, approximately 35 percent were classified as opinion, concept, or position papers or literature reviews, while 65 percent reflected some form of systematic data collection and analysis that could potentially provide evidence regarding 1 of the 15 PHEPR Capabilities. The most common study design category was quantitative nonimpact studies, which accounted for 33 percent of all articles and 51 percent of evidentiary studies. The quantitative nonimpact category includes studies that describe and identify the magnitude, severity, and preventability of a PHEPR problem and could potentially be used to inform the development of PHEPR practices aimed at addressing that problem. The quantitative impact category, accounting for only about 6 percent of all articles, includes studies that evaluate specific PHEPR practices. PREPUBLICATION COPY: UNCORRECTED PROOFS 8-1

8-2 EVIDENCE-BASED PRACTICE FOR PHEPR FIGURE 8-1 Distribution of all scoping review articles by study design (N = 1,692). This distribution of study designs presents challenges for identifying evidence-based practices, and in Chapter 2, the committee presents the following conclusion: Conclusion: With the increasing complexity of both public health emergencies and the PHEPR system, policy makers and practitioners have a crucial need for access to guidance based on robust evidence to support their decisions on practices, policies, and programs for saving lives during future public health emergencies. Therefore, a coordinated and comprehensive approach to prioritizing and aligning research efforts and ensuring that research is relevant and consistently connected to practice, along with investments in research infrastructure, is necessary to strengthen the PHEPR evidence base, thereby ensuring that PHEPR practitioners have the scientific evidence they need to guide and inform their actions. At the same time, PHEPR practitioners will require incentives to base their practices, policies, and programs on evidence. In this chapter, the committee describes a framework to support the systematic development of knowledge in the PHEPR field and sets forth the aspirations for high-quality, rigorous PHEPR research and evaluation. PREPUBLICATION COPY: UNCORRECTED PROOFS

IMPROVING AND EXPANDING THE PHEPR EVIDENCE BASE 8-3 A NATIONAL PUBLIC HEALTH EMERGENCY PREPAREDNESS AND RESPONSE SCIENCE FRAMEWORK The generation of a PHEPR evidence base has been hindered by the inherent challenges of conducting research on public health emergencies, which can limit the opportunities to observe and proactively study the effects of practices used to mitigate harm before, during, and after such an event (see Chapter 2 for additional detail). In addition to those challenges, federal partners and other stakeholders repeatedly have attempted to enhance PHEPR research capacity. However, many of these efforts have not been sustained or adequately resourced, and the result has been a limited infrastructure to support the generation and dissemination of PHEPR research. Conclusion: Funding for and prioritization of research before, during, and following public health emergencies are currently fragmented and disorganized, spread across multiple funding agencies; are inconsistent; and do not encourage the progression of quality research and the sustainable development of research expertise. This situation has contributed to a field based on long-standing rather than evidence-based practice. Despite the implications of public health emergencies for the nation’s health and economic security, there is currently no mechanism for ensuring the coordinated resourcing, monitoring, and execution of public- and private-sector PHEPR research. Key Components of a National PHEPR Science Framework The importance of developing a disaster research strategy has previously been underscored (Keim et al., 2019). The committee proposes that a comprehensive National PHEPR Science Framework could move the PHEPR field beyond the near-term goal of a research agenda toward a more coherent vision of coordinating and aligning efforts effectively to advance evidence-based practice in PHEPR (see Figure 8-2). The key components of this framework are discussed in the following sections. PREPUBLICATION COPY: UNCORRECTED PROOFS

8-4 EVIDENCE-BASED PRACTICE FOR PHEPR FIGURE 8-2 Key components of a National PHEPR Science Framework. System Leadership to Transform the PHEPR Research Enterprise The foundations of scientific progress in PHEPR lie in building and sustaining a research enterprise. Strong leadership at all levels, but especially at the federal level, is central to the framework and essential to support systems-level change and mobilize agencies to transform the way PHEPR research is coordinated, funded, and conducted. An interagency and multidisciplinary effort led by the Centers for Disease Control and Prevention (CDC) will be necessary to develop and implement a National PHEPR Science Framework; establish an authority and process for supporting research before, during, and following public health emergencies; and ensure that adequate research funding, capacities, and infrastructure are in place. CDC is the funding agency with the primary mission responsibility in PHEPR, and it is important that the agency responsible for supporting PHEPR planning and implementation also lead efforts to increase the scientific evidence base that supports the execution of that PREPUBLICATION COPY: UNCORRECTED PROOFS

IMPROVING AND EXPANDING THE PHEPR EVIDENCE BASE 8-5 responsibility. However, the committee acknowledges that no one agency can accomplish this transformation of the PHEPR research enterprise, and it will be necessary to leverage the strengths of different partners, including funding partners, in these efforts. Given the complexity of the PHEPR system and the fact that it is nested within many integrated, larger systems (e.g., the supply chain, transportation), it will be imperative for this effort to be carried out in cooperation with other fields, industries, and organizations. It will also be essential to provide CDC with adequate resources and expertise to support its lead role in these efforts and to encourage CDC to become a learning organization. Appropriation language will need to be explicit and clear that funding can be used for these research efforts. An enterprise for coordinating medical countermeasure (MCM) efforts in the federal government could serve as a model for coordinating and funding PHEPR research (HHS, 2017a). The Public Health Emergency Medical Countermeasures Enterprise (PHEMCE), established by the U.S. Department of Health and Human Services (HHS) in 2006 and led by the Assistant Secretary for Preparedness and Response (ASPR), has the following core members: the director of CDC, the director of the National Institute of Allergy and Infectious Diseases (NIAID) within the National Institutes of Health (NIH), and the commissioner of the U.S. Food and Drug Administration (FDA). Key PHEMCE partners include senior leadership from other federal agencies and numerous nonfederal partners. Recognizing that the development of MCMs requires significant resources in terms of time and cost, the PHEMCE helps coordinate funding across portfolios and prioritize efforts through a coherent plan that spans several years (HHS, 2017b). PHEMCE members are motivated and guided by the need to develop responses and cost-efficient methods to protect the nation against novel threats. This enterprise has been successful in bringing attention to the issue of MCM development, generating additional research investment, and prompting government agencies and departments to coordinate their research efforts. Such a coordinated effort can bring common terminology and infrastructure to a fragmented research enterprise. Recognition of PHEPR Science as a Unique Academic Discipline Within Public Health PHEPR research is transdisciplinary and draws on the knowledge base from many different fields, from behavioral and social sciences and political science to systems science operations research, among others. However, recognition that PHEPR science is a unique academic discipline within the broader public health field represents the first step in addressing the substantial need for research; qualified well-trained researchers; and preexisting, durable, and reliable engagement and partnerships with PHEPR practitioners. The nuances and complexity of the PHEPR field pose unique challenges for designing research studies that differ from those in the broader public health field (e.g., population health research), and the usual approach for clinical research—randomized controlled trials (RCTs)—is not always feasible. Researching public health emergencies requires specific knowledge and understanding of how the systems involved work, and public health scientists may lack the knowledge, experience, and credibility to conduct research before, during, or following a public health emergency. Given the nuances of PHEPR research, it can be especially important for researchers to be familiar with the strengths and limitations of various approaches to and methods for evaluation. These may include, for example, observational studies designed in real time with strong data collection, predesigned pragmatic trials ready for deployment, simulations and exercises that use RCT approaches, side-by-side qualitative research studies to supplement PREPUBLICATION COPY: UNCORRECTED PROOFS

8-6 EVIDENCE-BASED PRACTICE FOR PHEPR quantitative methods, and methods for capturing data from after action reports (AARs) in a way that supports knowledge generation. Recognizing PHEPR science as a unique academic discipline within the broader public health field could generate the resources and efforts needed to better support current and new academic departments or centers focused on PHEPR; establish degree or certification programs; and support career mechanisms that would enhance the conduct of high-quality, rigorous PHEPR research. Training and career development for PHEPR researchers are discussed in greater depth in the section on workforce capacity development for PHEPR research and practitioners later in this chapter. With recognition of PHEPR as a distinct field of study, the PHEPR research community could develop its own unique culture with the support of scientific societies and associations, which currently is lacking. A Forward-Looking PHEPR Research Agenda and Common Evidence Guidelines A research agenda is necessary to galvanize the PHEPR research field to meet the needs and respond to the concerns of PHEPR practitioners and society at large (IOM, 2008). This agenda must be more than a simple inventory of research needs (see Box 8-1). It will require leadership, and an organizing entity will need to be identified and made responsible for aggregating research conducted in alignment with the agenda and for tracking progress on and updating the agenda. A component of the agenda could highlight what is and is not known or what PHEPR research is currently under way. PHEPR evidence gaps could, for example, be identified and communicated by the PHEPR evidence-based guidelines group proposed in Chapter 3. Another essential feature of the research agenda will be to describe a process for rapidly identifying and prioritizing research needs during a public health emergency and to establish a minimum set of data elements that would be sought by anyone collecting data during such an event (NBSB, 2011). A formal research prioritization process will be necessary, including a top-down, bottom- up approach to setting the research agenda. An important consideration is for the process to be inclusive of governmental, nongovernmental, private, and academic organizations, as well as broad public input from practitioners, policy makers, researchers, and the community. Given that outcomes of interest in PHEPR also include process and system improvements (Carbone and Thomas, 2018), it will be valuable to engage in the research agenda development process experts in the fields of health services research; social science; implementation and improvement science; operations research; complex interventions; quality improvement; cost-effectiveness; and systems, policy, and organizational research. The National Institute of Environmental Health Sciences (NIEHS) has conducted a successful town hall model in communities across the country for bringing practitioners, researchers, and community members together to participate in setting a research agenda, enhance practitioner–researcher partnerships, and foster greater awareness of community and public health needs (O’Fallon et al., 2003). O’Fallon and colleagues (2003) note several best practices for successful town halls:  the meeting is held in a location that is convenient and comfortable for the participants;  controversial topics are encouraged, and when such a topic is selected, it is important to ensure that both sides of the issue are presented;  lecturing is minimized and audience participation is maximized; and  the final agenda is determined by the host organization. PREPUBLICATION COPY: UNCORRECTED PROOFS

IMPROVING AND EXPANDING THE PHEPR EVIDENCE BASE 8-7 In addition to town hall models, workshops and leadership retreats for practitioners and researchers can be useful to identify new developments and research topics in PHEPR (O’Fallon et al., 2003). It will be important for the research agenda to encompass a range of research questions that would be addressed through different methods of inquiry (see the section on common evidence guidelines for PHEPR later in this chapter), including and going beyond the topics covered within CDC’s 15 PHEPR Capabilities. A good starting point for identifying research questions will be to examine previously developed research agendas for PHEPR (Acosta et al., 2009; IOM, 2008). Additional research priorities may be informed by the evidence gaps identified in the committee’s commissioned scoping review (see Appendix D) and the committee’s evidence reviews for practices within the Community Preparedness, Emergency Operations Coordination, Information Sharing, and Non-Pharmaceutical Interventions Capabilities (see the sections on future research priorities in Chapters 4–7). BOX 8-1 Components of a PHEPR Research Agenda The committee identified the following essential components of a PHEPR research agenda:  a convening or organizing entity with the authority to develop and update the research agenda;  funding for development of the agenda, ongoing technical assistance for researchers, and aggregation of research conducted in alignment with the agenda;  guidance for informing decisions about federal and nonfederal funding for research;  a development process that employs various input strategies and is inclusive of governmental, nongovernmental, private, and academic organizations and based on broad public input from practitioners, researchers, and the community;  a method for determining research needs and gaps and research questions based on an inclusive and rigorous process for identifying and categorizing the available literature and data;  a specified, logical timeline that supports the progression from building an evidence base through the advancement of quality standards and methodological improvements, now and into the future; and  a process for tracking research and evaluating progress toward meeting the priorities set forth in the research agenda. Meaningful Partnerships Between PHEPR Practitioners and Researchers to Promote Knowledge Crucial to a National PHEPR Science Framework will be ensuring a strong connection between PHEPR practitioners and researchers, as well as strong community partnerships. Understanding how to promote, improve, and sustain the engagement of PHEPR practitioners and communities in a thoughtful and inclusive process for generating research will be an PREPUBLICATION COPY: UNCORRECTED PROOFS

8-8 EVIDENCE-BASED PRACTICE FOR PHEPR essential element of a robust and effective PHEPR research field (Miller et al., 2016). With the proper incentives in place, PHEPR practitioners and researchers can be encouraged to engage in more meaningful partnerships to promote knowledge. Academic partners can help public health agencies by providing data and findings to inform practice (especially during a real public health emergency, when public health agencies may be challenged in conducting research), executing studies, facilitating stakeholder meetings, assessing training needs, providing technical assistance, and collaborating on publications (IOM, 2015). Specifying expectations related to the conduct, and subsequent publication, of research and evaluation in practitioner position descriptions, or even designating a dedicated “science” position within the agency, could provide an incentive for collaborating and partnering with researchers. Furthermore, partnerships between PHEPR practitioners and researchers could be strengthened by integrating expectations into Project Public Health Ready (PPHR), a criteria-based training and recognition program that assesses local health departments’ PHEPR capacity, or the Public Health Accreditation Board (PHAB) accreditation process. Additionally, establishing the trust of the community before a public health emergency occurs is critical to ensuring that research can be conducted effectively and equitably. The community also has many resources to offer the PHEPR research enterprise, including its experiences and knowledge of its needs and existing networks that can be leveraged. Overall, PHEPR research requires the collaboration, insight, and trust of professionals from public health, other response agencies, academia, private entities, and members of the community prior to a public health emergency, and an effective National PHEPR Science Framework will support strategies for strengthening and maintaining these partnerships to promote successful PHEPR research. Collaboration and engagement among practitioners, researchers, and the community is a critical element that is highlighted throughout this chapter— from the development of a research agenda and funding of research programs to participation in the design and conduct of the research and translation, dissemination, and implementation of evidence-based practices. PHEPR Evidence-Based Guidelines Group and Other Efforts to Facilitate the Translation, Dissemination, and Implementation of PHEPR Research If the PHEPR field is to be grounded in evidence, the translation, dissemination, and implementation of research findings represent a crucial component of a National PHEPR Science Framework. The research and other evidence driven by this framework will need to be translated into clear evidence-based practices for public health agencies through an ongoing evidence review process. Accordingly, it will be important for the PHEPR evidence-based guidelines group proposed in Recommendation 1 in Chapter 3 to be integrated into the activities of the National PHEPR Science Framework, and to review relevant research and distill it into guidelines for the benefit of practitioners. The use of sustainable strategies and mechanisms, such as training specialists in translation and implementation science, particularly for the PHEPR field, can help bridge the often daunting gap between practice and research (Carbone and Thomas, 2018). Researchers need to engage with potential users of their research, involving them in the research design and implementation process to increase the likelihood that results will be translatable and practiced (Jillson et al., 2019). At the same time, while practitioners’ perspectives and needs are an important consideration in developing research projects, practitioners also need to be accepting of innovations that emerge from the research community (Carbone and Thomas, 2018). Workforce capacity development programs are necessary to improve the implementation PREPUBLICATION COPY: UNCORRECTED PROOFS

IMPROVING AND EXPANDING THE PHEPR EVIDENCE BASE 8-9 capacity of public health agencies. These issues are explored in depth in the sections on workforce capacity development for PHEPR research and practitioners and on translation of research into practice and dissemination and implementation of evidence-based PHEPR practices later in this chapter. Ensuring Adequate Infrastructure and Supporting Mechanisms to Facilitate the Conduct of PHEPR Research To conduct research effectively before, during, and following a public health emergency, additional capacities, infrastructure, policies, and other elements must be strengthened or created. Adequate and sustained federal funding for PHEPR research is necessary to ensure the continual flow of scientific discoveries to mitigate the health impacts of public health emergencies. Recognizing that this is an applied research field, there needs to be an emphasis on funding mechanisms that facilitate practice-based approaches and support the collection of experiential evidence from real-world practice and public health emergencies. It is imperative to ensure that funded research programs are relevant to practice, and in Chapter 2, the committee concludes that, in addition to investments in research infrastructure, a coordinated and comprehensive approach to prioritizing and aligning research efforts and ensuring that research is relevant and consistently connected to practice is necessary to strengthen the PHEPR evidence base. The best practitioner-centered evaluations will be achieved through trusting and durable partnerships between practitioners and researchers. Some example practice-based approaches that could be supported include a researcher residency model, a practice-based research network (PBRN) model, and a research-oriented tabletop exercise model. A researcher residency model (i.e., researchers embedded in the PHEPR system) could enable researchers to attend and observe exercises, have a seat in the emergency operations center during a public health emergency, and participate in the after action reporting. In learning health care systems, it has increasingly been recognized that embedding researchers in the system offers multiple benefits, including the identification of practitioner- and systems-relevant research questions and the ability to close the research and practice gap (both of which are persistent challenges in PHEPR) (Forrest et al., 2018). CDC could also consider requiring the inclusion of PHEPR practitioners in research proposals or implementing a PBRN model within PHEPR that might enable important advances in these areas. PBRNs—groups of practitioners and researchers working together to answer community-based questions and translate research findings into practice—are a result of the increasing need for research conducted in real-world settings (AHRQ, 2019; DeVoe et al., 2012; Mays et al., 2013; RWJF and University of Kentucky College of Public Health, 2020). PBRNs have previously been supported in public health (such a program was supported by the Robert Wood Johnson Foundation [RWJF] from approximately 2007 to 2015) (RWJF, 2013), and some of this work focused on PHEPR (CTEC et al., 2016; Wimsatt, 2017). Additionally, research- oriented tabletop exercises can aid in developing practice-driven research. For example, Chandra and colleagues (2015) developed a community resilience tabletop exercise and administered it to stakeholders from multiple disciplines to assess progress in community resilience and provide an opportunity for quality improvement and capacity building. The NIEHS Disaster Research Response (DR2) program has previously conducted regional tabletop exercises for researchers and practitioners to facilitate the development of these partnerships and to educate practitioners on ways of incorporating data collection and research into disaster response (NIH, 2019b). Attention to emerging practice-based approaches is needed as well. PREPUBLICATION COPY: UNCORRECTED PROOFS

8-10 EVIDENCE-BASED PRACTICE FOR PHEPR An equally important aspect of funding is mechanisms that allow for investigator-driven research, facilitate engagement and collaboration with researchers from different disciplines, and encourage the progression from development to intervention to secondary analysis to center grants—something that is currently lacking in the PHEPR field but is fundamental to any research enterprise. Several models for sustained funding programs for research series and multidisciplinary and collaborative research centers, as well research education and training projects (i.e., career development grants) currently exist at the National Science Foundation (NSF) and NIH and could be replicated for the PHEPR field (NIH, 2019c, 2020a; NSF, 2020b). CDC currently funds research through the use of grants and cooperative agreements (CDC, 2020b). Foundations, such as RWJF, the Alfred P. Sloan Foundation, the Bill & Melinda Gates Foundation, and the Open Philanthropy Project, among others, can support research programs when federal support is limited (Alfred P. Sloan Foundation, 2020; Bill & Melinda Gates Foundation, 2020; Open Philanthropy, 2020; RWJF, 2020a). Addressing PHEPR knowledge gaps will require sustained lines of research, with multiple studies addressing similar research questions in different contexts and populations in nonemergency times and with the ability to refocus efforts or activate additional protocols, if warranted, in the event of a future public health emergency. Given the inherent challenges of conducting research during public health emergencies, it is important to give careful consideration to opportunities for advancing PHEPR science during nonemergency times and to the pre-event planning needed to enable research during and following future public health emergencies. Types of PHEPR research that could be supported in nondisaster times include modeling, simulations and exercises, and research on public health implementation issues that would likely translate in the event of a public health emergency. One of the challenging aspects of all PHEPR activities, including research, is the natural aversion to creating unused capacity. A solution to this challenge could be providing funding and resources for individuals conducting research in nonemergency times with the expectation—and support—that they will turn their attention to researching a public health emergency at hand. Research funders could consider requiring researchers to account for this possibility by describing such contingency processes in their proposals. As mentioned above, deliberative planning during nonemergency times is necessary so that the resources and supporting mechanisms needed to rapidly conduct scientific research in the context of a public health emergency will be in place. Past efforts have made strides toward developing mechanisms to support scientific research in the context of a public health emergency, some of which have indeed been developed and even tested (e.g., rapid identification and prioritization of research needs and funding after Hurricane Sandy) (Lurie et al., 2013). In Table 8-1, Lurie and colleagues (2013) describe key components of an integrated approach to research before, during, and after an emergency and explicitly lay out actions that could be taken before as well as during the emergency. One consideration not included in this table is the action of developing metrics and outcome measures, an important component of the conduct of research and evaluation of practices. Future efforts to ensure adequate infrastructure and supporting mechanisms to facilitate the conduct of research during public health emergencies could be guided and informed by these past efforts. PREPUBLICATION COPY: UNCORRECTED PROOFS

IMPROVING AND EXPANDING THE PHEPR EVIDENCE BASE 8-11 TABLE 8-1 Key Components of Research Response in the Context of Public Health Emergencies SOURCE: Lurie et al., 2013. Reprinted with permission from Massachusetts Medical Society. To support research during a public health emergency, sustainable, rapid, and nimble funding mechanisms, together with award criteria and preapproved PHEPR research study protocols, are needed. Several such mechanisms currently exist, but they are uncoordinated and focused disproportionately on infrastructure, engineering, and environmental health. An example PREPUBLICATION COPY: UNCORRECTED PROOFS

8-12 EVIDENCE-BASED PRACTICE FOR PHEPR is NSF’s Rapid Response Research (RAPID) funding mechanism, used when quick-response research on disasters is needed (NSF, 2018). Additionally, with the support of NSF, the National Hazards Center at the University of Colorado administers a Quick Response Research Grant Program that provides small grants to help researchers collect data following an event (National Hazards Center, 2019). This program can be mobilized quickly to put some of the necessary research infrastructure in place in the immediate aftermath of a disaster and could serve as a model for funders, although it is at a smaller scale than that needed for the PHEPR research enterprise. NIEHS’s Time-Sensitive Research Grants Program is another example of a rapid funding mechanism for public health emergencies. Its aim is to receive, review, and fund research applications within 90–120 days, and it supports research to characterize initial exposures, collect human biological samples, and collect human health and exposure data (Miller et al., 2016). Other agencies could replicate these rapid response funding models specifically for PHEPR research. Additionally, partnerships with foundations that are interested in addressing the needs of communities and health-related research could help fill gaps in funding. Rapid funding to support research in the event of a public health emergency is not enough, however; efforts are also needed to enhance capacities to conduct the research and improve data collection capabilities. Such efforts might include establishing formalized academic–public health agency research partnerships and a cadre of researchers and preassembled teams embedded in the response system—specifically within the incident management structure—and available to respond rapidly to public health emergencies (Lurie et al., 2013). In 2012, for example, the U.S. Department of the Interior’s Strategic Sciences group was created to meet the immediate need for scientific information during an environmental disaster (DOI, 2020). In the case of PHEPR, CDC could build research expertise into the training for Epidemic Intelligence Service (EIS) officers, a subset of whom could then be deployed for the sole purpose of conducting PHEPR research during public health emergencies (CDC, 2020a). A benefit of doing so would be that EIS officers are often most knowledgeable about needs and opportunities in communities and have the relationships to carry out the necessary research and evaluations in real time. They can be mobilized quickly and are positioned to have their results incorporated rapidly into guidance, funding, and translation efforts. With respect to data collection, the NIEHS DR2 program has a central repository1 of publicly available data collection tools, such as surveys, questionnaires, and protocols, that could be used to establish early baselines and cohorts for research. Also important is to synergize and catalog past, ongoing, and future PHEPR-related research to avoid duplication (unless warranted) and reduce participant burden when research is conducted during or following a public health emergency. A database of PHEPR research studies categorized to facilitate analysis could be created, which would help foster research that would progressively improve PHEPR. Establishing and incentivizing partnerships and networks across research teams and institutions, other disciplines, and other organizations could also help coordinate research efforts. Research involving human subjects during or following public health emergencies may pose ethical and data sharing challenges (Packenham et al., 2017). The Public Health Emergency Research Review Board was established in 2012 to provide centralized, rigorous, and expeditious reviews of human subject protections for HHS-conducted, -supported, or -regulated research studies addressing public health emergencies (HHS and NIH, 2020). This entity is currently under the auspices of the NIH network of institutional review boards (IRBs), and an 1 See https://dr2.nlm.nih.gov/tools-resources (accessed March 10, 2020). PREPUBLICATION COPY: UNCORRECTED PROOFS

IMPROVING AND EXPANDING THE PHEPR EVIDENCE BASE 8-13 IRB Authorization Agreement between NIH and the institutions conducting research is required (IOM, 2015). The NIEHS Best Practices Working Group for Special IRB Considerations in the Review of Disaster-Related Research has also been making progress in this area. The DR2 program has developed a pre-event generic protocol, the Rapid Acquisition of Pre- and Post- Incident Disaster Data (RAPIDD) Protocol, for provisional approval by IRBs, which has been used by several universities. The objective of the RAPIDD protocol is to facilitate the collection of epidemiologic information and laboratory test results and the collection and storage of human biospecimens (Miller et al., 2016). When it comes to research, public health agencies have different sets of concerns from researchers and institutions. A key issue is the security of confidential data and the privacy of subjects. Research during or following public health emergencies can also raise ethical challenges, including the burden on the population, potential harms, and the potential for therapeutic misconception (IOM, 2015). Further guidance and support are needed for academic entities and public health agencies to develop effective and efficient means for reviewing and addressing unique ethical issues in the conduct of PHEPR research, such as through pre- emergency review of standard protocols, training of IRB members on unique aspects of PHEPR research, and the establishment of specific review mechanisms for this research. Conclusion and Recommendations Conclusion: A National PHEPR Science Framework can establish the goals and objectives necessary to improve coordination, integration, and alignment among existing PHEPR research efforts, but will require adequate resourcing and oversight. RECOMMENDATION 3: Develop a National Public Health Emergency Preparedness and Response (PHEPR) Science Framework To enhance and expand the evidence base for PHEPR practices and translation of the science to the practice community, the Centers for Disease Control and Prevention should work with other relevant funding agencies; state, local, tribal, and territorial public health agencies; academic researchers; professional associations; and other stakeholders to develop a National PHEPR Science Framework so as to ensure resourcing, coordination, monitoring, and execution of public- and private-sector PHEPR research. The National PHEPR Science Framework should do the following:  Build on and improve coordination, integration, and alignment among existing PHEPR research efforts (e.g., the National Institute of Environmental Health Sciences Disaster Research Response Program), and ensure integration of these efforts with the activities of the PHEPR evidence-based guidelines group proposed in Recommendation 1.  Recognize and support PHEPR science as a unique academic discipline within the broader public health field to address the substantial need for research and diverse and qualified researchers.  Create a common, robust, and forward-looking PHEPR research agenda that supports advancement beyond traditional epidemiological research to include research in the fields of social science, implementation science, complex PREPUBLICATION COPY: UNCORRECTED PROOFS

8-14 EVIDENCE-BASED PRACTICE FOR PHEPR interventions, and quality improvement, as well as intervention, operations, systems, and cost-effectiveness research.  Support meaningful partnerships between PHEPR practitioners and researchers, and develop strategies to better ensure that PHEPR research is relevant to practice.  Prioritize sustainable strategies and mechanisms for the translation, dissemination, and implementation of PHEPR research. RECOMMENDATION 4: Ensure Infrastructure and Funding to Support Public Health Emergency Preparedness and Response (PHEPR) Research The Centers for Disease Control and Prevention (CDC), in collaboration with other relevant funding agencies, should ensure adequate and sustained oversight, coordination, and funding to support a National PHEPR Science Framework and to further develop the infrastructure necessary to support more efficient production of and better-quality PHEPR research. Such infrastructure should include  sustained funding for practice-based and investigator-driven research that allows for the progression from exploratory to effectiveness to scale-up research and encourages researcher diversity;  support for partnerships (e.g., with academic institutions, hospital systems, and state, local, tribal, and territorial public health agencies) to facilitate collaboration in research on the preparedness, response, and recovery phases of a public health emergency;  development of a rapid research funding mechanism and interdisciplinary rapid response teams with applied research expertise (similar to CDC’s Epidemic Intelligence Service) for deployment to conduct just-in-time studies related to the implementation of PHEPR practices at the time of events; and  enhanced mechanisms to enable routine, standardized, efficient data collection with minimal disruption to delivery of services (including preapproved, adaptable research and institutional review board protocols and a research arm within the response structure). SUPPORTING METHODOLOGICAL IMPROVEMENTS FOR PUBLIC HEALTH EMERGENCY PREPAREDNESS AND RESPONSE RESEARCH The results of the committee’s four evidence reviews highlight the paucity of research that has generated credible evidence to inform PHEPR practice (see Chapters 4–7). In some cases, the committee could find very few to no studies with which to address the questions of interest (e.g., no quantitative studies were included in the review for emergency operations coordination). In other cases, a sizable body of research exists, but the committee noted limitations in study designs (e.g., lack of baseline measurements or comparison groups, use of unvalidated or subjective/self-reported measures), execution (e.g., underpowered studies, high loss to follow-up), analysis (e.g., failure to conduct statistical tests or lack of statistical adjustment), or reporting of information (e.g., lack of details on the methodology, the PHEPR practice, population characteristics, other contextual factors, and outcomes). Some research methods were also poorly matched to the research questions they were intended to address. To PREPUBLICATION COPY: UNCORRECTED PROOFS

IMPROVING AND EXPANDING THE PHEPR EVIDENCE BASE 8-15 help ensure that future studies yield results from which stronger conclusions can be drawn about the effectiveness of PHEPR practices, future investments in PHEPR research will need to remedy these common methodological shortcomings. Standards, guidance, and incentives can help raise the quality and evidentiary value of research in the PHEPR field. Common Evidence Guidelines Federal agencies can have a significant influence on the generation of the evidence base for practice (Maynard, 2018). In fields other than PHEPR (e.g., education) setting priorities and standards for research and using them to guide funding decisions has improved the quality and usefulness of the evidence base. An example is the response of the education research field to shifts in funding priorities to align with agency evidence agendas and guidelines, which began in 2002 with the creation of the Institute of Education Sciences (IES) within the U.S. Department of Education.2 Early on, IES established evidence guidelines for causal inference studies; a system for sourcing, grading, and synthesizing evidence; a Web-based clearinghouse for evidence reviews; and an active program of funding for professional development (pre- and postdoctoral training grants; professional meetings and association development). These efforts resulted in a dramatic shift in the methodological rigor of education evaluations (Whitehurst, 2018). Similar improvements could be achieved in PHEPR by drawing on these experiences in other fields to enact policies and practices that can improve how PHEPR research is conducted, disseminated, and used (IES and NSF, 2013). The goal is to ensure that scarce evaluation dollars are used most productively to advance the evidence available to inform policy and practice. Achieving this goal necessitates careful balancing of several factors: the importance of the questions studied, the rigor with which the questions can and will be studied, the timeliness of the research findings, and the accessibility and usability of the findings. Tiered evidence standards for grantmaking can be a useful mechanism to guide funding decisions, as they allow federal agencies to award smaller amounts to promising concepts and larger amounts to practices grounded in strong evidence of success, encouraging innovation while still rewarding programs with robust research backing (GAO, 2016). Going forward, the PHEPR research field will need to have clear guidelines and standards for evaluation methods and study designs that will produce credible answers to various types of questions of importance to the field. The objective is to encourage a balance of research throughout the knowledge-building continuum, from basic science through effectiveness trials and modeling studies, and to foster rather than stifle research innovation. Guiding the Use of Different Types of Research Methods and Approaches Well-crafted guidance will incorporate the full spectrum of research methods, which may range from exploratory case studies to RCTs and modeling studies for evaluating PHEPR practices. The PHEPR research field would be strengthened by creating a unified taxonomy of research methods, accompanied by guidelines for judging the credibility of study findings intended to address various types of questions. A first step in developing this guidance will be to identify the various genres of PHEPR research, and for each genre describe its purpose (i.e., how that type of research contributes to the evidence base) (see Annex 8-1). It will also be important for each genre of research to be supported by theoretical and empirical justifications when possible, and to adhere to established expectations for research design, methods, and products of 2 Education Sciences Reform Act. H.R. 3801, Section 116. 107th Session of Congress (January 23, 2002). PREPUBLICATION COPY: UNCORRECTED PROOFS

8-16 EVIDENCE-BASED PRACTICE FOR PHEPR the research. Expectations will need to be established as well for review of the products of each type of research (i.e., what information is required to judge the credibility and applicability of the findings and how that information can be judged). While acknowledging the value of randomization for demonstrating a causal link between interventions and outcomes, the committee recognizes that it is difficult, if not infeasible or inappropriate, to implement RCTs for some PHEPR practices, particularly in the context of a real public health emergency. As discussed in Chapter 3, other study types (e.g., quasi-experimental study designs) may provide credible estimates of a causal impact (or lack thereof) when PHEPR practices are evaluated. Table 8-2 describes the strengths and limitations of common study designs for quantitative impact evaluation with applicability to PHEPR. PREPUBLICATION COPY: UNCORRECTED PROOFS

IMPROVING AND EXPANDING THE PHEPR EVIDENCE BASE 8-17 TABLE 8-2 A Brief Overview of Strengths and Limitations of Study Designs for Quantitative Impact Evaluation for PHEPR Study Design Strengths Limitations Randomized Provides the most unbiased, robust, and reliable estimates It is often difficult to conduct an RCT at the community or controlled trial of the effectiveness of a PHEPR practice, which gives national level, which is often the target of PHEPR practices. (RCT) confidence that any measured differences between groups are due to the intervention. Depending on the Results from a simulated trial may not mirror those in a true sample size and diversity, it may be possible to conduct emergency, and it may not be feasible to conduct an RCT subgroup analysis to determine whether impacts vary by during a public health emergency. If there is a desire to conditions in the implementing sites (e.g., urban/rural, conduct an RCT during an emergency, completing the study diversity of languages spoken by residents) that influence requires waiting for emergencies to occur. Testing the the effectiveness of the PHEPR practice. differential effectiveness of strategies in real emergencies introduces uncertainties in the timeframe, cost, and context for the study. It may be costly to recruit the sample for the study, and it may be difficult to persuade decision makers of the benefits of this design given political and ethical issues concerning randomization. Quasi-experimental Provides reasonably strong evidence of the relationship There could be systematic differences between the study (matched between the PHEPR practice and outcomes measured. It jurisdictions implementing the PHEPR practice that are not comparison group is a powerful method for exploring the impact of a captured in the data, and therefore that cannot be controlled study, interrupted PHEPR practice when randomization is not possible. It for in the analysis. This could result in less reliable findings. time series, can be applied to large communities, and launching such regression a study may be more feasible than an RCT close to the Matching techniques require a great deal of data, and the discontinuity design, time of an emergency, which would improve the ability study could require considerable resources (time and cost) to multivariate to collect reliable data prior to the emergency. identify jurisdictions that had implemented the PHEPR analysis) practices of interest and collect the data. These designs require complex analytical work and specialized knowledge. Pre-post comparison For studies based on simulated emergency situations, There are significant threats to internal validity, but a study design comparison of outcomes pre and post provides plausible of this type could provide preliminary evidence of indications of whether the PHEPR practice was effectiveness. Changes may be occurring in the study sites implemented and whether outcomes changed as a result. between the pre and post periods, such as the adoption of In cases in which it is possible to measure outcomes for other PHEPR practices or staff turnover. For some outcomes, PREPUBLICATION COPY: UNCORRECTED PROOFS

8-18 EVIDENCE-BASED PRACTICE FOR PHEPR multiple time periods prior to and after implementation there is likely to be considerable variation that cannot be of the PHEPR practice, it is possible to compare not only explained by contextual factors, and it may be difficult or differences in outcomes immediately before and after the impossible to obtain reliable and consistent measures of the event but also differences in trends before and after. outcomes of interest through existing records or recall and reconstruction. This design may be the most feasible option given that it requires relatively little time and money, depending on the outcomes of interest and the cooperation of practitioners in the jurisdictions selected for study. Some outcomes of interest may be sufficiently predictable over time that observed shifts after implementation of the PHEPR practice will have high credibility. PREPUBLICATION COPY: UNCORRECTED PROOFS

IMPROVING AND EXPANDING THE PHEPR EVIDENCE BASE 8-19 Other experimental study designs have begun to emerge that may also present opportunities for PHEPR research. Adaptive platform trial designs, such as the vaccine trial proposed during the 2014–2015 Ebola outbreak, allow for flexibility along the timeline of an event, with interim analysis of data to enable investigators to determine whether to continue moving forward, change course, or divest more rapidly from interventions that are not showing promise (Berry et al., 2016). Pragmatic trials evaluate the effectiveness of interventions in real- life routine practice rather than the highly controlled conditions typical of experimental research studies (Ford and Norrie, 2016; Patsopoulos, 2011). This design incorporates more real-world evidence into controlled trials, allowing for a broader environment for testing, as well as greater generalizability of findings. A stepped wedge cluster design is a type of pragmatic trial used to evaluate the efficacy of service delivery interventions. This type of design may be a good option when operating within logistical or political constraints and has been used in a variety of areas, ranging from vaccine development to social policy and criminal justice (Hemming et al., 2015). On the other hand, the committee encourages the PHEPR field to move beyond experimental study designs and consider a broader range of methods for exploring what works (and when, why, and for whom). Many PHEPR practices are designed to improve outcomes, particularly systems-level outcomes, in complex settings in response to unpredictable events. PHEPR practitioners and researchers are often interested in whether a practice made a difference or what would have happened had it not been implemented (e.g., what would have happened had the public health emergency operations center not been activated). Qualitative research methodologies (e.g., ethnographic observations, interviews, and focus group discussions) can inform why and how PHEPR practices may or may not be effective (Teti et al., 2020). A range of approaches are gaining recognition, such as realist evaluations3 and qualitative comparative analyses,4 approaches that acknowledge the complexity of causality (Blanchet et al., 2018). There is also defined guidance for evaluating complex interventions (e.g., the UK Medical Research Council, the Patient-Centered Outcomes Research Institute [PCORI]), but these concepts have yet to be fully adopted by the PHEPR research field (AcademyHealth, 2017; PCORI, 2019; MRC, 2019). Furthermore, as PHEPR research is transdisciplinary, design methodologies used in such fields as public health services and systems research, operations research, behavioral and social sciences, organizational research, and quality improvement can also provide evidence for understanding PHEPR practices. In particular, simulation-based research methods (e.g., tabletop exercises), systematic expert opinion methodologies (e.g., Delphi’s), and systems science approaches (e.g., social network analyses, causal process diagrams, adaptive systems theories, modeling, machine learning, and big data analyses) can provide insight on systems-level outcomes and the interdependent relationships among the many components of the PHEPR system. Overall, there are many rigorous methodologies from diverse fields that could be used to evaluate PHEPR practices, and the key takeaway is to match the study design appropriately to the research question to produce credible answers. Annex 8-1 3 “Realist evaluations are based on an assumption that projects and programs work under certain conditions and are influenced by the way that different stakeholders respond to them. Realist evaluations attempt to answer questions such as what works, for whom, in which circumstances, and why. They are designed to improve understanding about how development interventions work in different contexts” (INTRAC, 2017b). 4 “Qualitative Comparative Analysis (QCA) is a methodology that enables the analysis of multiple cases in complex situations. It can help explain why change happens in some cases but not others. QCA is designed for use with an intermediate number of cases, typically between 10 and 50. It can be used in situations where there are too few cases to apply conventional statistical analysis” (INTRAC, 2017a). PREPUBLICATION COPY: UNCORRECTED PROOFS

8-20 EVIDENCE-BASED PRACTICE FOR PHEPR provides a brief summary of genres of research, example research questions, and some appropriate methods. Comprehensive guidance would include suggestions for strategically mixing methods to improve both the design of intervention studies (e.g., through baseline studies conducted before a PHEPR practice is implemented) and understanding of the findings, including their breadth and limitations (postintervention). An example of such a strategic mixed-method approach in PHEPR is the Los Angeles County Community Disaster Resilience (LACCDR) project5 (see Figure 8-3). The PHEPR field could benefit from conducting sequential or parallel studies focused on particular aspects of PHEPR practices. It is also important to note that the LACCDR project used community participatory methods, and the committee’s evidence review on engaging with and training community-based partners (see Chapter 4) found that community and stakeholder involvement in research and programmatic efforts from conceptualization to implementation may correspond with more effective engagement and training through enhanced inclusion, cultural acceptability, shared ownership, and capacity building of community members. Comprehensive guidance would incorporate such participatory methods, and also refer to their use in such emerging fields as engagement science (Dungan et al., 2019). 5 See http://www.laresilience.org (accessed May 11, 2020). PREPUBLICATION COPY: UNCORRECTED PROOFS

IMPROVING AND EXPANDING THE PHEPR EVIDENCE BASE 8-21 FIGURE 8-3 Los Angeles County Community Disaster Resilience (LACCDR) mixed-method research study. NOTES: CBO = community-based organization; CBP = community-based partner; SES = socioeconomic status. The numbers shown on the figure denote the following sources: 1Adams et al., 2017; 2Bromley et al., 2017; 3Cha et al., 2016; 4Chandra et al., 2013; 5Chi et al., 2015; 6Williams et al., 2018. PREPUBLICATION COPY: UNCORRECTED PROOFS

8-22 EVIDENCE-BASED PRACTICE FOR PHEPR Standards for Reporting of Study Information It is essential for all intervention studies to have well-articulated research plans that, when possible, are published before the analysis itself begins (Burlig, 2018; Lupia and Alter, 2014; Moravcsik, 2014). Such plans describe the study design, identify the primary and supplemental research questions to be addressed, provide background on the study setting, define the target population(s), and explain why the proposed program may change practice and improve decision making and outcomes for PHEPR practitioners or the community. They also detail the data collection plan, including measures to be used, and describe the analysis and reporting plans. Given that PHEPR research funding and prioritization efforts are currently fragmented, disorganized, and inconsistent, there is no standardized peer-reviewed grant process, and as a result there are currently no specific standard guidelines or benchmarks for reporting the results of evaluations of the effectiveness of PHEPR practices. Reporting guidelines for health-related research have been developed for RCTs (Begg et al., 1996), observational studies in epidemiology (von Elm et al., 2007), systematic reviews of complex interventions (Guise et al., 2017a, b), studies of diagnostic accuracy (Bossuyt et al., 2015), qualitative research (O’Brien et al., 2014; Tong et al., 2007), implementation studies (Pinnock et al., 2015), and quality improvement studies (Ogrinc et al., 2016), among others (Hoffmann et al., 2014; Simera et al., 2008). Given the experience with a wide range of other types of research, it appears likely that developing, publishing, and disseminating tailored guidelines for PHEPR evaluations might well improve the reporting of such studies. Federal agencies can support standardized reporting (e.g., through the development of guidance and standards and requirements linked to grants), which improves the usability of results and may over time result in efficiencies and cost savings (Maynard, 2018). Professional associations and journals also have important roles in the adoption of and commitment to reporting standards. PHEPR professional associations could establish the need and advocate for well-defined reporting standards, gather and review standards developed by other fields, draft standards for use by journals, and ensure that standards are shared and understood by the PHEPR research and practice fields. Journals play a vital role in communicating research findings to practitioners, as well as making information available to those in other sectors. By requiring the use of reporting standards, they can also promote the transparency and reproducibility of scientific research. Conclusion and Recommendation Conclusion: The lack of formal guidance and expectations regarding the various genres of PHEPR research has led to variable levels of credibility of the evidence produced. Given that evidence-based practices are dependent upon existing research, efforts to delineate common expectations for PHEPR research need to be a priority to enhance the conduct of high-quality research and evaluation and help organize research investments. PREPUBLICATION COPY: UNCORRECTED PROOFS

IMPROVING AND EXPANDING THE PHEPR EVIDENCE BASE 8-23 RECOMMENDATION 5: Improve the Conduct and Reporting of Public Health Emergency Preparedness and Response (PHEPR) Research The Centers for Disease Control and Prevention, the Office of the Assistant Secretary for Preparedness and Response, the National Institutes of Health, the U.S. Department of Homeland Security, the National Science Foundation, and other relevant PHEPR research funders should use funding requirements to drive needed improvements in the conduct and reporting of research on the effectiveness and implementation of PHEPR practices. Such efforts should include  developing guidance on and incorporating into funding decisions the use of appropriate research methods as determined by the level of research (e.g., exploratory, effectiveness, scale-up) and type of research question(s) being addressed, including but not limited to encouraging the use of concurrent comparison groups when feasible and assessment of baseline measures;  establishing guidelines for evaluations using different designs and evidence streams and concepts from emerging evaluation approaches, such as complex intervention evaluations; and  developing reporting guidelines, including essential reporting elements (e.g., addressing contextual factors, confounding factors, and negative results), in partnership with professional associations, journal editors, researchers, and methodologists for PHEPR intervention studies. IMPROVING SYSTEMS TO GENERATE HIGH-QUALITY EXPERIENTIAL EVIDENCE FOR PUBLIC HEALTH EMERGENCY PREPAREDNESS AND RESPONSE Public health agencies typically conduct after action reviews following real or simulated (exercise) public health emergencies in an effort to identify lessons learned and strengths and weaknesses of the response, and ultimately to improve emergency preparedness and response capabilities (Davies et al., 2019). The after action review process is an important source of experiential evidence in PHEPR and is the primary approach used by public health agencies to evaluate public health emergency response. In evaluation of the effectiveness of PHEPR practices, AARs offer the potential for improved understanding of context and implementation considerations that could be difficult to obtain through research. AARs can also be used to develop theories and logic models to inform future research. However, because they are not designed to be research, they are not without their methodological limitations. To help ensure that future AARs result in more useful and meaningful information for the evaluation of PHEPR practices (including the establishment of credible baselines for evaluation), it will be necessary to focus on strengthening methodological approaches, establishing mechanisms for analysis and dissemination of lessons learned from the reviews, and fostering a culture of improvement. In the United States, several agencies and organizations that fund or oversee aspects of PHEPR, including CDC and ASPR, formally require after action reviews. The Federal Emergency Management Agency’s (FEMA’s) Homeland Security Exercise and Evaluation Program (HSEEP) developed a framework for agencies, including public health agencies, to use when developing, executing, and evaluating exercises. This program provides AAR templates PREPUBLICATION COPY: UNCORRECTED PROOFS

8-24 EVIDENCE-BASED PRACTICE FOR PHEPR and guidance to inform agencies in documenting strengths, areas for improvement, and corrective actions (FEMA, 2013). Because the framework is not organized according to the CDC PHEPR Capabilities, however, public health agencies have had to make major adaptations, such that the potential advantages of a standardized AAR have not been realized (Barnett et al., 2020). Thus current reporting requirements and methodological standards for AARs lack clarity and uniformity. Moreover, evaluations are rarely conducted by independent evaluators with appropriate expertise (Davies et al., 2019; Gossip et al., 2017). AARs are typically reviewed and vetted throughout the agency or agencies that produced them before being submitted or shared with partners or the public, which may limit the candor of the information they contain (Gossip et al., 2017). The result is significant variability in the quality and reliability of AARs. The lack of consistent reporting requirements and variable report structures, together with limited CDC and public health agency resources, impedes the aggregation of AAR data and thus their use as a potential source of evidence for evaluating the effectiveness of PHEPR practices. Limitations of AARs as a Source of Experiential Evidence for Mixed-Method Evidence Reviews In recognition of the potential of AARs to inform the effectiveness and implementation of PHEPR practices in different contexts, the committee considered evidence from AARs in two of its evidence reviews. Davies and colleagues (2019) recently developed an appraisal tool with which to compare methodological reporting and document validity for AARs (see Box 8-2). To inform efforts focused on improvements needed to enhance the evidentiary value of AARs for future use, the committee commissioned a quality assessment of the 38 AARs included in its evidence reviews using this 11-item appraisal tool (Patel, 2019).6,7 BOX 8-2 Eleven-Item Tool for Assessing the Methodological Rigor of After Action Reports  Prolonged engagement with the subject of inquiry—Has the review included lengthy and perhaps repeated interviews with respondents, and/or days and weeks of engagement within a case study site or group?  Use of theory—Has theory been used to guide sample selection, data collection, and analysis?  Data selection—Has purposive selection been used to allow prior theory and initial assumptions to be tested or to examine “average” or unusual experience?  Information sampling—Has the review gathered views from a wide range of perspectives and respondents rather than letting one viewpoint (person, organization, or specialty) dominate? Has it sampled enough people, places, times, etc., to ensure that the influence of these factors on the behavior and views of those people providing 6 This section draws heavily on a report commissioned by the committee on “Quality Assessment of After Action Reports: Findings and Recommendations,” by Sneha Patel. 7 The committee included 38 AARs in its evidence reviews. Approximately 61 percent of those reports were based on real events, and full-scale and functional exercises accounted for 16 percent and 21 percent of the reports, respectively. Hazards and threats included infectious diseases (e.g., H1N1, Ebola, hepatitis A), natural disasters, and man-made disasters (e.g., oil spills, explosions). Incident years ranged from 2009 to 2017 in 20 U.S. states. The AARs were all published either in the same year or the year following the real event or exercise. PREPUBLICATION COPY: UNCORRECTED PROOFS

IMPROVING AND EXPANDING THE PHEPR EVIDENCE BASE 8-25 the information is minimized? Has sampling been expanded in the light of early findings?  Multiple data sources—Has the review sought multiple information sources (documents, personal testimony, site visits) and collated multiple examples of each? For example, have duplicate formal interviews with all sampled staff been undertaken? Has the review used researcher observation and informal discussion, and have interviews been conducted with people in different roles and levels of seniority?  Triangulation—Has the review looked for patterns of convergence and divergence by comparing results across multiple sources of evidence (e.g., across interviewees, and between interview and other data), among researchers, and across different methodological approaches? Has it also made comparisons within data, e.g., comparing different interview accounts?  Negative case analysis—Has the review looked for evidence that contradicts its initial findings, explanations, and theory, and refined them accordingly?  Peer debriefing and support—Has the review included a step whereby the findings and reports have been reviewed by other researchers or investigators?  Respondent validation—Have findings and reports been reviewed by respondents to check investigators’ interpretation of their input?  Clear report of methods of data collection and analysis (audit trail)—Has the review kept and reported a full record of activities that is available to others and presented a full account of how methods evolved and were applied?  Depth and insight—Has the review established the direct/indirect root causes and underlying contributory factors linked to errors, inaction, or latent failures? SOURCES: Davies et al., 2019; ECDC, 2018. Overall, the application of the tool to 38 AARs yielded low scores. Notably, consultants wrote two of the three highest-quality AARs, and AARs based on real events were of better quality on average than those based on exercises. The vast majority of AARs failed to provide a rationale for data selection, and more than half provided no detail on information sampling or multiple data sources, making it difficult to ascertain the appropriateness of the sample or sources used to inform the AAR findings. Practitioners were often surveyed for comments and observations, but the sample size of those practitioners, the timeline for data collection (immediate postevent versus after a reflection period), the information collected, and the format for collection varied widely and were rarely documented. Limiting samples to response leadership potentially skewed findings toward a leadership perspective at the cost of including feedback from staff engaged more directly in response operations. The grouping of leadership with general staff in feedback sessions could have discouraged staff from fully expressing any critiques they may have had regarding how leadership handled a response. Excluding communities from the after action review process also represented missed opportunity to hear from diverse voices that might not have been reflected in the demographics of leadership or staff. None of the AARs mentioned negative case analysis or respondent validation. Only three described peer debriefing and support; two of these were written by consultants, and one validated regional findings at the state level. Overall, findings from applying the AAR quality assessment tool indicate a significant need to improve both after action review processes and the level of detail included in the reports themselves. It is unclear whether AAR authors omitted basic methodological information in a PREPUBLICATION COPY: UNCORRECTED PROOFS

8-26 EVIDENCE-BASED PRACTICE FOR PHEPR process that was otherwise rigorous, or the reports would have scored low even if the requisite categories had been included. In addition to these methodological shortcomings, the committee noted several other gaps and biases in its review of AARs and the AAR generation process that will need to be addressed moving forward:  local political pressures and fear of judgment or retribution for reporting errors or negative outcomes;  retrospective, subjective reporting based on the recall of participants, which may be influenced by the experience itself, pressures to “move on” and resume usual workflow, and limited roles in and siloed views of the activities;  lack of methodological standards and tools for collecting, aggregating, analyzing, and disseminating information and reports;  limited access to and the variable quality of data, information, and reports; and  limited formal training, infrastructure, and resources to develop specialized personnel and/or programs to critically analyze AAR data and information in a culture of quality improvement. Strengthening Methodological Approaches Given the shortcomings discussed above, there is a clear need for CDC, in collaboration with ASPR and FEMA, to develop after action review policies and guidance that will ensure the capture of data relevant to the evolving response to a public health emergency and allow for in- depth analysis of the response. After action reviews can serve multiple purposes, including continuous quality improvement and, in some cases, accountability as part of grant requirements (Savoia et al., 2012; Stoto et al., 2013), and these purposes often require different methodological approaches to data collection, aggregation, and analysis. After action reviews are frequently completed by public health agencies themselves, and the methods used to collect information and data for the reports vary widely from agency to agency. In general, though, after action reviews use a wide variety of fairly common qualitative and quantitative methods, including surveys, interviews, focus groups or hotwashes, workshops, public forums, document reviews, and site visits (ECDC, 2018). There is broad agreement that an after action review should seek to establish more than the immediate cause of response and recovery issues, and should analyze the factors behind the immediate causes, aiming to get to the root causes (Barnett et al., 2020; Davies et al., 2019; ECDC, 2018; Piltch-Loeb et al., 2014b; Singleton et al., 2014; Stoto et al., 2015, 2019) (see Figure 8-4 for an example depicting the steps of root-cause analysis). This systematic approach to root-cause analysis forms the basis of most approaches, such as peer assessment approaches8 and facilitated look-backs.9 Gossip and colleagues (2017) note that public health agencies frequently utilize partner agencies and academic centers in the evaluation of exercises, but only rarely utilize this same expertise during or after a real-life response. Furthermore, while there is 8 The peer assessment approach employs an evaluation conducted by peers in similar jurisdictions. This approach offers the potential for objective analyses by PHEPR professionals and knowledge of the particularities of the system being assessed (Piltch-Loeb et al., 2014b). 9 The facilitated look-back approach uses a neutral facilitator and a no-fault approach to probe the nuances of decision making through moderated discussions (Piltch-Loeb et al., 2014b). PREPUBLICATION COPY: UNCORRECTED PROOFS

IMPROVING AND EXPANDING THE PHEPR EVIDENCE BASE 8-27 consensus that after action reviews should identify root causes, gaps remain in public health agencies’ use of this approach. An analysis of AARs conducted by Barnett and colleagues (2020) found several cases in which stated recommendations did not identify an underlying problem; an earlier analysis by Singleton and colleagues (2014) also highlighted the frequent failure of recommendations and corrective actions to include root-cause analysis. FIGURE 8-4 Root-cause analysis: steps and examples. NOTE: AAR = after action report; IHR = international health regulations; PHEP = public health emergency preparedness; RCA = root cause analysis SOURCE: Reprinted with permission from Stoto et al., 2019. AARs are at risk of the same biases as the qualitative and quantitative methods on which they rely, and findings from the committee’s commissioned quality assessment of AARs indicate that the reports typically omit the majority of important validity categories that could foster greater confidence in after action findings. Guidance aimed at improving after action review methods and the level of detail included in AAR methods sections is needed for both transparency and quality purposes, and AARs need to meet some minimum criteria concerning methods and reporting (Davies et al., 2019) The PHEPR field could benefit from drawing on the PREPUBLICATION COPY: UNCORRECTED PROOFS

8-28 EVIDENCE-BASED PRACTICE FOR PHEPR broader public health field to apply more rigorous evaluation processes when assessing lessons learned from public health emergencies. Training for evaluation participants, including academic programs in HSEEP certification and evaluation design, need to be encouraged and supported, if not required (Stoto et al., 2019). Standards and expectations regarding AARs could be strengthened by being integrated into PPHR (Summers and Ferraro, 2017). Similarly, PHAB could shape the evaluation of PHEPR by modifying its standards and measures to specifically include those that relate to PHEPR, thereby fostering efforts at quality improvement and evaluation (Brownson et al., 2018). Most important is for CDC and state, local, tribal, and territorial (SLTT) public health agencies, in addition to approaching AARs as an administrative requirement, to begin viewing them as a source of experiential evidence that could inform the development of evidence-based PHEPR practices and providing the necessary training, infrastructure, and resources to improve the quality of AARs produced. Independent After Action Review Panel According to Gossip and colleagues (2017), partnering with external organizations (e.g., peer agencies, consultants, academic centers) improves the depth and quality of documentation and assessments, as external organizations often have the requisite expertise and skills and capacity (e.g., time and personnel) to conduct or guide more rigorous evaluations. The United States has a strong history of creating objective, independent review boards (e.g., the National Transportation Safety Board and the National Institute of Standards and Technology’s National Construction Safety Team Advisory Committee) consisting of collaborative partners to develop guidelines, evaluate data and findings, and investigate incidents. These objective bodies have been proposed as a model for the PHEPR field (Barnett et al., 2020; Keim et al., 2019; Kirsch et al., 2018). These types of processes ensure objective expertise by eliminating the inherent biases of self-assessment and make use of consistent methods so that findings are comparable over time. Such a panel could review all events reaching the threshold of a Stafford Act or Public Health Service Act event. This process could be conducted by a newly established group, through existing professional associations, accrediting bodies, or regional academic partnerships and networks. Essential Core Elements of a PHEPR AAR To enable aggregation and analysis of AAR data for use as a potential source of evidence on the effectiveness of PHEPR practices, it is essential to define the core elements of a PHEPR AAR that builds on the existing HSEEP format but embraces more of a public health perspective. These elements would include a standardized core dataset and root-cause analysis framework that ultimately could be used not only by one jurisdiction, but also across jurisdictions for purposes of aggregation, trend analysis, and systemwide comparison (see Box 8- 3 for the committee’s suggested elements for such an AAR template). Focusing on system-level root causes rather than specific problems would help make the experience more broadly applicable (i.e., enhance generalizability) (Stoto et al., 2019). An executive summary with high- level findings for each AAR would aid further in developing the empirical evidence base. An online platform prepopulating evaluation forms for practitioner-specific reporting objectives could also be developed (Agboola et al., 2015). Such a platform, with standardized questions at the national level, would assist practitioners in completing evaluations, integrate consistent measures into the review process, and aid in conducting trend analyses over time. PREPUBLICATION COPY: UNCORRECTED PROOFS

IMPROVING AND EXPANDING THE PHEPR EVIDENCE BASE 8-29 BOX 8-3 The Committee’s Suggested Elements for a PHEPR After Action Report (AAR) Template  a structured executive summary focused on what agency leadership would find most useful for decision making to inform current and future responses (and designed to be database searchable with a maximum word count);  a word-limited abstract to provide an overview of the incident, important contextual factors, the tested PHEPR Capabilities, and key findings;  an acronym and definition list, as jurisdictions often use terms differently;  a background section that provides enough information for someone with an outsider’s perspective to understand the context of the situation and event without being overburdened by unnecessary details;  a methods section that describes the sampling, data sources, and other elements from the European Centre for Disease Prevention and Control (ECDC) 11-item validity tool;  a section for root-cause analysis of the system response, with specific assessments of the PHEPR Capabilities and explicit interventions used in the event;  specific fields for the reporting of standardized information, such as a defined list of outcomes and corrective actions; and  inclusion of the tools used to conduct the review for the AAR (e.g., surveys, hotwash guides, focus group guides) as an appendix, so jurisdictions need not reinvent the wheel. Establishing Mechanisms for Analysis and Dissemination of Lessons Learned from AARs Proper conditions for establishing the utility and credibility of AARs as a less biased source are needed to improve the utility of these reports in evidence reviews and guidance for practical decision making. Improved mechanisms for public sharing of lessons learned from AARs, such as an enhanced national AAR repository, that specifically preclude the use of the reports for punitive purposes would foster more accurate and reliable information. Currently, finding relevant and publicly available AARs that can serve as a reliable source of information and identifying useful observations within them is a notably labor- intensive process with variable yield. With the proper incentives and standards in place to encourage submission of AARs (e.g., formal recognition from the user community or appropriate agencies or qualification for additional exercise funding), an enhanced national AAR repository could be a useful tool for improving PHEPR (Kearns, 2010; Piltch-Loeb et al., 2014a; Turner et al., 2018). FEMA’s Lessons Learned Information Sharing (LLIS) platform was developed to serve as a protected repository of AARs, reports of best practices, and lessons learned for all emergency response communities (HSDL, 2020). In 2015, LLIS was consolidated with the Homeland Security Digital Library (HSDL). Presently, the HSDL serves as the only repository for AARs, but submission of AARs to this database is voluntary, the AARs include those submitted by sectors other than public health, other documents are also submitted, and the database is not easily searchable. These factors make it difficult to locate public health AARs to include in analyses. PREPUBLICATION COPY: UNCORRECTED PROOFS

8-30 EVIDENCE-BASED PRACTICE FOR PHEPR A national AAR repository would ideally include indexed and searchable executive summaries and searchable high-level findings derived from AARs of federal- and state- supported exercises and tabletops, as well as responses to real events. This information would be organized and referenced by thematic content, findings, and type of event. To ensure a national AAR repository that is both feasible and useful, it is important to consider the following issues:  the required resources and barriers to overcome for the creation, establishment, and curating of such a repository;  the necessary infrastructure and protocols, which may include the creation and the adoption and assurance of submission standards, technology-based capture of key information from submitted AARs, and automated data aggregation and analysis;  the incorporation of wide-ranging and user-friendly capabilities, which at a minimum would need to include systematic searchable parameters enabling identification of specific examples and details likely to emerge as themes across AARs;  the ability to support analysis by verified users by providing channels for sharing anonymous or deidentified data;  the ability to routinely audit and review entries, coupled with technical support to ensure accuracy and relevance; and  mechanisms for response to user feedback and other continuous quality improvement measures. Fostering a Culture of Quality Improvement PHEPR programs reside in SLTT public health agencies, and because they are government entities, their reports are open to public access through local and state freedom of information laws. Reports filed with CDC are open to access through the Federal Freedom of Information Act. Sensitivity to the release of security issues captured in reports, as well as concern about the exposure of vulnerabilities or weaknesses, has resulted in a reluctance to publish or even complete these reports. PHEPR practitioners are hesitant to report problems or weaknesses observed during a response because of the potential political ramifications and fear of punitive action (Gossip et al., 2017). Therefore, current agency culture leads to overlooking AARs as critical for informing evidence-based decision making. This situation stands in contrast to data used for health care quality assurance and protected in most states by “safe harbor” laws, data that have been cited as pivotal in participation in quality improvement initiatives by health care entities and practitioners (Gliklich et al., 2014). The PHEPR field would similarly benefit from a more open, structured, and meaningful culture of quality improvement that would enable the ongoing reporting and analysis of successes and challenges. Conclusions and Recommendation Conclusion: Unclear reporting requirements, a lack of methodological standards, and disincentives for accurate reporting of problems experienced during response operations limit the evidentiary value of AARs for the development of evidence-based guidelines. These issues represent a significant lost opportunity to learn from practice and to leverage real-world experience to develop such guidelines. PREPUBLICATION COPY: UNCORRECTED PROOFS

IMPROVING AND EXPANDING THE PHEPR EVIDENCE BASE 8-31 Conclusion: Unless protection is assured for sensitive data collected in AARs, information on shortcomings during public health emergency responses will remain hidden, and similar errors will be repeated. Improved systems for public sharing of AARs that specifically preclude their use for punitive purposes would foster access to more accurate and reliable information, supporting the utility of the reports in evidence reviews and guidance for practical decision making. RECOMMENDATION 6: Pursue Efforts to Further a Process of Quality Improvement to Enhance the Quality and Utility of After Action Reports (AARs) The Centers for Disease Control and Prevention, in collaboration with the Office of the Assistant Secretary for Preparedness and Response and the Federal Emergency Management Agency, should convene an expert panel of relevant federal agencies; state, local, tribal, and territorial public health agencies; and professional associations to advance a process for quality improvement at the local, regional, state, and national levels to enhance the quality and utility of AARs and support their use as sources of evidence for evaluating the effectiveness of public health emergency preparedness and response (PHEPR) practices. This process should foster a culture of improvement in public health emergency response and include, but not be limited to, discussions aimed at  raising standards and expectations regarding the quality of information reported in AARs by defining the essential core elements of a PHEPR AAR;  establishing an independent review panel with a standardized after action reporting process, with the aims of reducing bias and increasing the utility of AARs produced following public health emergency responses;  establishing and maintaining a national repository of AARs or of reports based on analysis of AARs that is readily accessible to support the dissemination of key findings, lessons learned, and best practices for public health emergency response; and  exploring the relevant privacy issues and the protection of information in AARs from use in legal proceedings or other punitive actions against practitioners and organizations, as has been done for “peer-review” data in other fields (medicine, aviation, and occupational health). WORKFORCE CAPACITY DEVELOPMENT FOR RESEARCHERS AND PRACTITIONERS IN PUBLIC HEALTH EMERGENCY PREPAREDNESS AND RESPONSE A robust and sustained commitment to workforce capacity development for both PHEPR researchers and practitioners is essential to advancing evidence-based practice in PHEPR. Investigators will have to produce research that is relevant to practitioners, and practitioners will have to turn routinely to research when making important decisions about implementing practices. Although there exists a network of researchers capable of conducting high-quality research on public health emergencies, these researchers require additional and ongoing training to design and implement studies and analyze and report study findings in a rigorous manner. It will also be important for practitioners to participate in or conduct evaluations. New skills, some PREPUBLICATION COPY: UNCORRECTED PROOFS

8-32 EVIDENCE-BASED PRACTICE FOR PHEPR going beyond traditional public health training, are often needed for practitioners to successfully conduct evaluations and identify and implement evidence-based practices (Brownson et al., 2018). Currently, the PHEPR workforce comprises both researchers and practitioners who often are educated and trained in schools of public health, but many other disciplines are represented as well, including physicians, nurses, social workers, veterinarians, and social scientists. Yet, there is little evidence that the graduate education and training these individuals receive in their respective domains is designed or tailored to address the unique challenges of conducting research in disaster-affected communities or during public health emergencies. Researchers Academic disaster research centers proliferated after the events of September 11, 2001, and funding for training for disaster researchers was readily available (see Chapter 2). Given the ongoing reduction in funding over the last several years, however, workforce attrition is likely. Academic researchers who moved into PHEPR research from public health and other related fields when funding was readily available may have left the PHEPR field when funding became more limited. There exists today virtually no investment in the developmental pipeline for PHEPR researchers, a gap that corresponds to the relative dearth of funding opportunities for related research. The scarcity of PHEPR research and training grants inhibits the training of a new generation of researchers in the PHEPR field. In addition to funding for research awards, it is important to consider funding for training grants and career development awards. In the past, CDC has funded Mentored Research Scientist Development Awards (K01) to provide support for intensive research career development under the guidance of a mentor in areas addressing bioterrorism, other infectious disease outbreaks, and other public health threats and emergencies, among other areas (CDC, 2007). To ensure diverse, adequately trained, and sufficiently available disaster researchers, it will be necessary to invest in improved and sustained researcher training programs and grants (e.g., NIH career development K awards, RWJF leadership development programs), particularly those aimed at increasing PHEPR research capacity in the areas of implementation science and evaluation of complex interventions (NIH, 2020b; RWJF, 2020b). Researcher training programs can help direct training toward specific important areas; provide support for training in other, emerging areas; and establish research standards. Additionally, special attention is essential to recruiting and supporting underrepresented minority researchers, including those who understand critical race theory, indigenous research methods, and other relevant frameworks, so that PHEPR science can advance equitably. Efforts are needed to connect doctoral and postdoctoral students early on in their careers so as to build relationships, connections, and teams to carry out ongoing PHEPR research projects. In a similar vein, mentoring programs involving senior and early-career researchers can support training for future generations. Many fields have recognized that providing means of supporting a steady infusion of highly trained researchers with new ideas is necessary to advance the quality of research (DHS, 2020; NCER, 2020; NIH, 2019a; NSF, 2020a). Practitioners As previously mentioned, PHEPR practitioners require the knowledge and skills not only to identify and implement evidence-based practices but also to conduct and participate in research and evaluation. In general, the public health workforce varies greatly in terms of job PREPUBLICATION COPY: UNCORRECTED PROOFS

IMPROVING AND EXPANDING THE PHEPR EVIDENCE BASE 8-33 descriptions, education requirements, and experience, which are not standardized across public health agencies. A recent study found that having a public health degree was significantly associated with reduced odds of reporting a skill gap in identifying and applying evidence-based approaches to address a public health issue (Maddock, 2018; Taylor and Yeager, 2019). A barrier to PHEPR practitioners working effectively with researchers may be that practitioners do not have the training or experience to identify programmatic and practice challenges that could be studied by researchers (Carbone and Thomas, 2018). Renewable certification programs, similar to the certified emergency manager program (IAEM, 2020), and a standardized training program for CDC project officers and state preparedness directors, as well as practitioners working in PHEPR more broadly, could ensure familiarity with evidence-based practices and promote the consistent creation and evaluation of real-world evidence as captured in AARs. CDC’s TRAIN Learning Network; ASPR’s Technical Resources, Assistance Center, and Information Exchange; and FEMA’s Emergency Management Institute provide a wealth of trainings and resources for PHEPR practitioners that could be leveraged and enhanced to provide practitioners with the knowledge and skills needed to identify and implement evidence-based practices and to conduct and participate in research and evaluation (ASPR, 2020; FEMA, 2020; PHF, 2020). A number of other policy and operational barriers, including workload and responsibilities and priorities during a response, impede PHEPR practitioners’ abilities to conduct and participate in evaluations. Another model for addressing these barriers is to give practitioners access to external contractors with evaluation expertise that can provide tailored evaluation tools, appropriate training, and feedback on program evaluation plans and products (Maynard, 2018). Recognizing that not everyone is equally suited or professionally able to be both a practitioner and researcher, it is necessary to develop stronger systems, infrastructure, and norms around the notion of an integrated PHEPR research and practice system that includes both those who are focused on advancing the science and those applying this knowledge. Platforms to provide ongoing technical assistance, peer networking, and collaboration across disciplines for both practitioners and researchers could ensure an interconnected workforce. Individuals who perceived themselves as part of this new interconnected PHEPR undertaking would share the mission of improving population health outcomes during public health emergencies and would be better positioned to interact with their colleagues across the traditional divides of practice and research. Conclusion and Recommendation Conclusion: Training, education, and technical assistance programs to provide PHEPR researchers and practitioners with the knowledge, skills, and competencies to conduct rigorous and relevant PHEPR research and evaluation and implement evidence-based practices are seriously deficient. Comprehensive efforts to support workforce capacity development are necessary to ensure that a qualified and competent PHEPR researcher and practitioner workforce is developed and sustained to address increasingly complex public health emergencies. PREPUBLICATION COPY: UNCORRECTED PROOFS

8-34 EVIDENCE-BASED PRACTICE FOR PHEPR RECOMMENDATION 7: Support Workforce Capacity Development and Technical Assistance Programs for Public Health Emergency Preparedness and Response (PHEPR) Researchers and Practitioners The Centers for Disease Control and Prevention (CDC) and the Office of the Assistant Secretary for Preparedness and Response should work with professional and academic organizations that represent multiple disciplines to guide and support the creation of the workforce capacity development and technical assistance programs necessary to ensure the conduct of quality PHEPR research and evaluation and improve the implementation capacity of state, local, tribal, and territorial public health agencies. Such efforts should include  developing a research training infrastructure and career development grants— institutional and individual predoctoral, postdoctoral, loan repayment, and career awards—to develop and support researchers in PHEPR in order to address research gaps in the field;  providing training grants so that PHEPR researcher and practitioner teams can learn how to develop PHEPR practices that are grounded in science and theory and to evaluate the effectiveness and implementation of PHEPR practices using rigorous and appropriate designs;  providing ongoing technical assistance and peer networking for both PHEPR researchers and practitioners; and  creating a training and certification program for CDC project officers and state preparedness directors to ensure their familiarity with evidence-based practices and promote consistent creation and evaluation of real-world evidence as captured in after action reports. TRANSLATION, DISSEMINATION, AND IMPLEMENTATION OF PUBLIC HEALTH EMERGENCY PREPAREDNESS AND RESPONSE RESEARCH TO PRACTICE While there is a clear need to strengthen the evidence base for PHEPR practices through improvements in research and quality improvement processes, an equally pressing challenge is the translation, dissemination, and implementation of the evidence to practice (Carbone and Thomas, 2018). Numerous barriers impede the uptake of evidence-based practice, including varying awareness of the existing evidence base and lack of coordination between researchers and practitioners, lack of time, inadequate funding, inability to analyze and interpret evidence, and absence of cultural and managerial support (Brownson et al., 2018; Siegfried et al., 2017). Nonetheless, the absence of incentives has been found to be the greatest barrier, compounded by the disincentive of the time required (Jacobs et al., 2010). Such challenges are not unique to PHEPR. For example, the cultural shift toward evidence-based practice in health care evolved over decades, driven in part by financial and reputational liabilities for institutions and practitioners and increased accountability for clinical outcomes (Carbone and Thomas, 2018). While the adoption of evidence-based practices has been an ongoing challenge in the PHEPR field, significant investments made in implementation science more broadly across public health, health care, and other disciplines could guide the translation and implementation of research to practice for PHEPR (Brown et al., 2017; Brownson et al., 2012). PREPUBLICATION COPY: UNCORRECTED PROOFS

IMPROVING AND EXPANDING THE PHEPR EVIDENCE BASE 8-35 Recognizing the importance of this issue, CDC in 2015 initiated the Translation, Dissemination, and Implementation of Public Health Preparedness and Response Research and Training Initiative (TDI Initiative), a 2-year effort to consolidate and synthesize the research produced by the Preparedness and Emergency Response Research Centers (PERRCs) and Preparedness and Emergency Response Learning Centers (PERLCs) and translate that research into practice (Qari et al., 2018) (see Chapter 2). Table 8-3 describes best practices for the translation, dissemination, and implementation of PHEPR tools and resources that emerged from this initiative. Despite these efforts, however, barriers to the adoption of evidence-based PHEPR practices persist, related largely to capacity issues. TABLE 8-3 Best Practices for the Translation, Dissemination, and Implementation of Evidence- Based PHEPR Practices Area Best Practices Translation  Engage the end user (i.e., public health practice community) from the start to ensure that evidence-based practices are appropriate and relevant.  Create training webinars to acquaint PHEPR practitioners with new skills or processes to improve performance.  Include real-world and local community scenarios and examples to demonstrate how to use practices, processes, tools, and resources. Dissemination  Word of mouth and personal recommendation.  Conferences and national meetings.  Radio stations, to present and translate important information (especially for tribes).  In-person meetings, which provide an opportunity for direct communication.  The Internet, including websites, blog posts, email newsletters, gaming techniques, distribution lists, and social media.  Trusted public health agencies and public health professional organizations, such as the Association of State and Territorial Health Officials (ASTHO) and the National Association of County & City Health Officials (NACCHO).  Publication of evidence-based practices in the Centers for Disease Control and Prevention (CDC) communication platforms (e.g., Morbidity and Mortality Weekly Report, blogs).  Peer-to-peer learning through existing networks.  State health departments, which receive information and resources from CDC and share them with local health departments.  Health care coalitions that exist within various states and localities. Implementation  Ensure that products and tools can be implemented by different agency and organization types (health care, public health, emergency management agencies, others).  Provide implementation support for a new tool or resource, as it can be challenging for public health agencies to gain momentum for implementation.  Identify an advocate within the agency who will ensure that the tool remains a priority during planning and implementation.  Provide support to bring staff together to review and practice implementing evidence-based practices.  Create products that are ready to use, because practitioners have limited time to review, modify, and adapt resources for local use. PREPUBLICATION COPY: UNCORRECTED PROOFS

8-36 EVIDENCE-BASED PRACTICE FOR PHEPR  Build in-person trainings into existing grant structures. SOURCES: Adapted from NORC at the University of Chicago, 2017; Qari et al., 2018. Building Implementation Capacity Successful implementation requires that public health agencies have sufficient capacity to identify and translate scientific knowledge into practice (Brownson et al., 2018). Uneven infrastructure and capacity represents a long-standing challenge for public health systems (Baker and Koplan, 2002), and many SLTT public health agencies lack sufficient capacity (i.e., resources, infrastructure, and workforce) to identify and translate research and implement evidence-based practices. Fluctuations in public health funding are common and often result in smaller public health entities having to prioritize “keeping the doors open” and ensuring that critical public health services remain functional, leaving them with little to no financial means for investing in PHEPR training and implementation efforts (NORC at the University of Chicago, 2017). Furthermore, best practices currently are often not developed with smaller or tribal or territorial public health agencies in mind, which frequently leads to implementation issues. Addressing these barriers will require a multipronged and coordinated approach involving a combination of training, technical assistance, tools, and incentives (e.g., grant and funding requirements). The Interactive Systems Framework for Dissemination and Implementation offers a model for building public health agencies’ capacity to use best practices (Wandersman et al., 2008). The TDI Initiative utilized this framework, incorporating prevention support systems, which “provide information, training, technical assistance, or other support for practitioners,” to enhance program implementation efforts (Qari et al., 2018, p. S359). Creating and Disseminating Practice-Ready Resources and Tools Researchers are often ill equipped to translate research and resources for practitioners’ implementation. There is a clear need for tools and resources that are ready for “on the ground” use and disseminated in accordance with practitioner demand. Engaging translation and implementation specialists in the development of such tools and resources could help ensure that they are practice-ready and can be implemented by practitioners at all levels (Carbone and Thomas, 2018). For example, understanding that a translation gap persists between the availability of PHEPR tools and their adoption within public health agencies, Revere and colleagues (2018) explored with practitioners a process for identifying communication products to be considered as the highest implementation priority to improve communications in public health agencies. To increase awareness of existing resources and knowledge, CDC could provide dissemination support by deploying field agents to champion and disseminate evidence-based practices and tools across the PHEPR practice community (Carbone and Thomas, 2018). Facilitating Tailoring of Evidence-Based Practices to Fit Local Contexts Given differences among public health jurisdictions, including variability in contexts and capacity, it is essential for any implementation effort to recognize the importance of local needs and allow for the tailoring of practices to those needs (Arora et al., 2018; Baseman et al., 2018; Eisenman et al., 2018; Revere et al., 2018; Testa et al., 2018). Practitioners need appropriate training and a robust public health knowledge base to make appropriate and effective modifications to a practice so that it suits new settings or contexts (Eisenman et al., 2018). Eisenman and colleagues (2018) recently implemented a two-component intervention involving PREPUBLICATION COPY: UNCORRECTED PROOFS

IMPROVING AND EXPANDING THE PHEPR EVIDENCE BASE 8-37 training and technical assistance and the creation and use of a guidebook to enhance the capacity of local public health agencies to translate and implement evidence-based practices in emergency preparedness. Evaluation of this program showed that after it was implemented, practitioners’ skills had significantly increased, indicating that such a program could be useful to other public health agencies. Existing tools are also available to decision makers to facilitate the implementation of evidence-based policies. For example, the SUPporting POlicy relevant Reviews and Trials (SUPPORT) tools are designed to help decision makers consider the applicability of evidence-based practices to their local conditions and tailor implementation strategies accordingly to address identified barriers and facilitators (Lavis et al., 2009; Lewin et al., 2009). When a PHEPR practice is being considered for implementation in different contexts, identifying its core components can help determine what should remain intact and what can be modified without jeopardizing outcomes. An intervention’s core components are the basic principles and activities deemed necessary to generate desired outcomes, and they are rooted in its theoretical drivers of change (ASPE, 2013). Identifying specific core components can thus help practitioners tailor existing evidence-based practices to the local context. This tailoring is based on answers to such operational questions as: What activities are we going to conduct? Who is going to carry out the activity? What must people learn? and What resources are required? Additional research into the identification of core components of PHEPR practices would improve understanding of which components are essential for evidence-based practices to produce desired outcomes and how this is moderated by differences in population, setting, and other contextual factors, thus enabling practitioners to better operationalize interventions in their local setting. Encouraging the inclusion of core components in descriptions of evidence-based practices could also help ensure ongoing fidelity over time (ASPE, 2013). Funding for the translation, dissemination, and implementation of these identified core components would promote the replication and scalability of practices and the associated tools and resources across a variety of settings. Incentivizing Adoption of Evidence-Based PHEPR Practices Other changes to federal programming and policy could facilitate the implementation of evidence-based PHEPR practices. As discussed in Chapter 2, CDC’s Public Health Emergency Preparedness (PHEP) Cooperative Agreement represents the primary mechanism for funding by the federal government for SLTT public health agencies, accounting for the overwhelming majority of any preparedness funds at the SLTT level. However, current programmatic grant requirements do not sufficiently emphasize or support the generation of PHEPR evidence or the use of evidence-based practices, and PHEPR practitioners generally have no incentive to use evidence-based practices in their planning (CDC, 2019). To begin to bridge the gap between research and practice, CDC could incorporate the use of evidence-based practices (e.g., those identified by the PHEPR evidence-based guidelines group proposed in Recommendation 1 in Chapter 3) into the PHEP Cooperative Agreement, with linkage to CDC’s PHEPR Capabilities, asking grantees to use evidence-based practices where available and to justify why they are not doing so when they are not. It will be important to engage with state health officials as well as other key stakeholders with regard to this proposed strategy. Accreditation bodies such as PHAB and recognition programs such as PPHR are also positioned to act as drivers for embedding evidence-based practices into PHEPR programs, and could be leveraged to facilitate the PREPUBLICATION COPY: UNCORRECTED PROOFS

8-38 EVIDENCE-BASED PRACTICE FOR PHEPR translation, dissemination, and implementation of research to practice (Brownson et al., 2018; Summers and Ferraro, 2017). Conclusion and Recommendation Conclusion: Evidence derived from PHEPR research and evaluation efforts will improve response outcomes in the event of a public health emergency only if it is translated to and implemented by PHEPR practitioners. Ensuring the translation, dissemination, and implementation of evidence-based PHEPR practices will require the infrastructure not only to produce evidence but also to summarize, synthesize, and disseminate it and ensure its effective use. RECOMMENDATION 8: Ensure the Translation, Dissemination, and Implementation of Public Health Emergency Preparedness and Response (PHEPR) Research to Practice The Centers for Disease Control and Prevention (CDC) should use a coordinated implementation science approach to ensure that the evidence-based practice recommendations resulting from the PHEPR evidence-based guidelines group proposed in Recommendation 1 achieve broad reach and become the standard of practice of the target audience. Strategies to this end include  incorporating evidence-based practices into the Public Health Emergency Preparedness and Response Capabilities: National Standards for State, Local, Tribal, and Territorial Public Health guidance document;  building evidence-based practices into the design of and funding decisions for the Public Health Emergency Preparedness (PHEP) Cooperative Agreement program;  incentivizing and requiring state, local, tribal, and territorial public health agencies to test and evaluate new or adapted practices and embed program evaluations into routine operations to help better understand whether evidence-based practices worked, under what conditions, with what impacts and consequences, and at what cost;  publishing evidence-based practices in CDC communication platforms (e.g., the Morbidity and Mortality Weekly Report, blogs) and partnering with public health professional organizations, such as the Association of State and Territorial Health Officials and the National Association of County & City Health Officials (NACCHO), to disseminate evidence-based practices;  incorporating the requirement of utilizing evidence-based PHEPR practices into such processes as the Public Health Accreditation Board accreditation and such recognition programs as NACCHO’s Project Public Health Ready; and  incorporating implementation science principles, such as the conduct of research to understand core components required for intervention effectiveness, into PHEPR research. PREPUBLICATION COPY: UNCORRECTED PROOFS

IMPROVING AND EXPANDING THE PHEPR EVIDENCE BASE 8-39 Annex 8-1 Genres of Research to Inform Public Health Emergency Preparedness and Response Practices ANNEX TABLE 8-1 Genres of Research to Inform PHEPR Practices: Purpose, Example Research Questions, and Appropriate Methods Genre of Appropriate Methods and Research* Purpose Example Research Question(s) Theoretical Perspectives Priority Setting To understand the cause and What are the nature, magnitude, and Epidemiological studies (cohort and Clarifying magnitude of the problem and to appropriate framing of the problem, as well as and case control), community the Problem and determine that something should be contributing factors, that together indicate the surveys, and analyses of data Causes done. These studies can also provide need for intervention? systems evidence of relationships between emergency events and consequences in What PHEPR practices are needed, and what Qualitative studies (focus groups, a particular context and/or for would they look like? interviews, observations, particular population groups. documentary analysis) What outcomes are important, and for whom? To understand the PHEPR system Quantitative, qualitative, and challenges and stakeholders’ views and mixed-method synthesis methods experiences, and to provide contextual information relevant to other forms of Rapid review methods, research research. priority-setting methods PREPUBLICATION COPY: UNCORRECTED PROOFS

8-40 EVIDENCE-BASED PRACTICE FOR PHEPR Design and To inform the design and development What is the PHEPR practice, how is it Qualitative studies (focus groups, Development, of new or improved PHEPR practices intended to work, and what outcomes is it interviews, observations) Including that will address particular problems in designed to achieve and with whom? Feasibility and PHEPR. These studies can also help Community-based participatory Pilot Testing develop the intervention program What outcome measures are appropriate? research, mixed methods, case theory (theory of change), core studies components, and selected outcomes of Which stakeholders need to be involved in the interest. implementation of a preparedness training Qualitative and mixed-method program for community-based partners? process evaluations, realist To test the feasibility of implementing synthesis methods the intervention in specific contexts, How feasible is the PHEPR practice to recruitment of participants, and the implement? Feasibility studies, health availability and feasibility of collecting economic methods, pilot trials data. To establish the sample size What aspects of complexity appear important calculation if needed; clarify the when implementing this PHEPR practice Implementation science methods program theory; and determine (e.g., system adaptivity and feedback loops)? primary outcomes, whether a Other considerations: Consider the complexity perspective would be Does the PHEPR practice need further use of existing intervention useful, and what cost data are required development or adaptation? development frameworks for and available. complex intervention development Does the pilot trial work as intended in this (e.g., UK Medical Research The information from such studies is setting with this PHEPR practice? Council Framework). Also, often used to identify evidence that sociobehavioral research is needed suggests potential causal relationships What implementation strategy works best to lay the theoretical and empirical (or the lack thereof) for a PHEPR with whom and in which context? foundation for PHEPR practices practice. (e.g., it is challenging to have an effective program for delivering To determine how best to implement vaccines or countermeasures the PHEPR practice, such as what without knowing about such strategies (e.g., training and resources) sociobehavioral domains as are required and what approaches trust/mistrust; risk perceptions and could be used. health behaviors; and the effects of racism, both historical and everyday/ordinary on health care access). PREPUBLICATION COPY: UNCORRECTED PROOFS

IMPROVING AND EXPANDING THE PHEPR EVIDENCE BASE 8-41 Quantitative To determine whether PHEPR Is activating public health emergency Randomized controlled trials, Impact practices are effective or work as operations centers [specify components] more quasi-experimental studies Evaluation intended to achieve the desired effective than the status quo approach (matched comparison group outcomes; to understand the benefits (commonly referred to as “business as usual”) studies, interrupted time series, and harms of PHEPR practices. in achieving Y [specified outcomes]? regression discontinuity design, multivariate analyses), pre-post Impact evaluations are typically Is strategy A [specify] for communicating comparison design conducted in cases where there exist a public health guidance and alerts with well-developed theory of change and a technical audiences during a public health well-defined PHEPR practice that is emergency more effective than strategy B being tested. In some cases, the impact [specify] in achieving Y [specified evaluation will be testing the outcomes]? intervention in a well-controlled setting and with high-fidelity What core components of public health implementation, while in other cases, it emergency operations centers are critical to may be tested under routine conditions their effectiveness? of practice. What are the comparative costs of strategy A and strategy B? Process To determine what happened when a What worked for whom in which contexts? Realist trials and process Evaluation PHEPR practice was implemented evaluation methods (e.g., what worked for whom in which What would have happened if the practice had contexts). not been implemented? Qualitative and mixed-method process evaluations To learn about the relationships Is the PHEPR practice acceptable and feasible between PHEPR-related practices and for recipients and key stakeholders? Qualitative comparative analysis outcomes in various contexts and gather implementation knowledge What are the barriers to and facilitators of Equity frameworks and methods (e.g., adoption, fidelity, barriers, effective engagement and training of facilitators, scale-up costs, equity). community-based partners? Case studies, after action reviews To identify variation among impacts across implementation contexts and explore potential mediators and moderators of the impacts. PREPUBLICATION COPY: UNCORRECTED PROOFS

8-42 EVIDENCE-BASED PRACTICE FOR PHEPR Systems and To use quantitative approaches to In what circumstances (e.g., based on biologic Statistical and mathematical Operations analyze strategic, tactical, and factors, risks, resource availability, legal modeling, simulation modeling, Research operational aspects of public health authorities, social context) is quarantine queueing theory, optimization emergency response to improve effective at reducing or stopping the spread of preparedness policies or practices. a contagious disease? Models typically use historical or simulated data to estimate expected responses to emergency conditions under varying preparedness conditions, controlling for or varying potential confounding or complicating factors. Implementation, To determine how best to adapt and What needs adapting and how? Randomized controlled trials, Adaptation, and tailor PHEPR practices to specific quasi-experimental studies Scale-Up populations and contexts. Is the scaled-up version as effective as the (matched comparison group original version? studies, interrupted time series, To determine how best to scale up regression discontinuity design, interventions for widespread adoption. What does it cost to scale up an intervention? multivariate analyses), pre-post comparison design Realist trials and process evaluation methods Qualitative and mixed-method process evaluations Implementation science methods Health economic methods * For all genres of research, engaging end users and other stakeholders is advised when feasible. PREPUBLICATION COPY: UNCORRECTED PROOFS

8-43 EVIDENCE-BASED PRACTICE FOR PHEPR REFERENCES AcademyHealth. 2017. Evaluating complex health interventions: A guide to rigorous research designs. https://www.academyhealth.org/sites/default/files/AH_Evaluation_Guide_FINAL.pdf (accessed March 10, 2020). Acosta, J. D., C. Nelson, E. B. Beckjord, S. R. Shelton, E. Murphy, K. L. Leuschner, and J. Wasserman. 2009. A national agenda for public health systems research on emergency preparedness. Santa Monica, CA: RAND Health. Adams, R. M., H. Rivard, and D. P. Eisenman. 2017. Who participates in building disaster resilient communities: A cluster-analytic approach. Journal of Public Health Management & Practice 23(1):37–46. Agboola, F., D. Bernard, E. Savoia, and P. D. Biddinger. 2015. Development of an online toolkit for measuring performance in health emergency response exercises. Prehospital and Disaster Medicine 30(5):503–508. AHRQ (Agency for Healthcare Research and Quality). 2019. Pratice-based research networks: Research in everyday practice. https://pbrn.ahrq.gov (accessed February 21, 2020). Alfred P. Sloan Foundation. 2020. Biosecurity. https://sloan.org/programs/completed- programs/biosecurity (accessed May 13, 2020). Arora, M., B. Granillo, T. K. Zepeda, and J. L. Burgess. 2018. Experiential adult learning: A pathway to enhancing medical countermeasures capabilities. American Journal of Public Health 108(S5):S378–S380. ASPE (Office of the Assistant Secretary for Planning and Evaluation). 2013. Core intervention components: Identifying and operationalizing what makes programs work. https://aspe.hhs.gov/report/core-intervention-components-identifying-and-operationalizing-what- makes-programs-work (accessed February 21, 2020). ASPR (Assistant Secretary for Preparedness and Response). 2020. Welcome to ASPR TRACIE. https://asprtracie.hhs.gov (accessed May 13, 2020). Baker, E. L., Jr., and J. P. Koplan. 2002. Strengthening the nation’s public health infrastructure: Historic challenge, unprecedented opportunity. Health Affairs (Project Hope) 21(6). Barnett, D. J., K. Strauss-Riggs, V. L. Klimczak, A. J. Rosenblum, and T. D. Kirsch. 2020. An analysis of after action reports from Texas hurricanes in 2005 and 2017. Journal of Public Health Management and Practice (prepublication). Baseman, J., D. Revere, H. Karasz, and S. Allan. 2018. Implementing innovations in public health agency preparedness and response programs. American Journal of Public Health 108(S5):S369–S371. Begg, C., M. Cho, S. Eastwood, R. Horton, D. Moher, I. Olkin, R. Pitkin, D. Rennie, K. F. Schulz, D. Simel, and D. F. Stroup. 1996. Improving the quality of reporting of randomized controlled trials: The consort statement. Journal of the American Medical Association 276(8):637–639. Berry, S. M., E. A. Petzold, P. Dull, N. M. Thielman, C. K. Cunningham, G. R. Corey, M. T. McClain, D. L. Hoover, J. Russell, J. M. Griffiss, and C. W. Woods. 2016. A response adaptive randomization platform trial for efficient evaluation of ebola virus treatments: A model for pandemic response. Clinical Trials 13(1):22–30. Bill & Melinda Gates Foundation. 2020. How we work—Grant opportunities. https://www.gatesfoundation.org/how-we-work/general-information/grant-opportunities (accessed May 13, 2020). Blanchet, K., C. Allen, J. Breckon, P. Davies, D. Duclos, J. Jansen, H. Mthiyane, and M. Clarke. 2018. Using research evidence in the humanitarian sector: A practice guide. London: Evidence Aid, London School of Hygiene and Tropicla Medicine and Nesta (Alliance for Useful Evidence). https://www.evidenceaid.org/wp- PREPUBLICATION COPY: UNCORRECTED PROOFS

8-44 EVIDENCE-BASED PRACTICE FOR PHEPR content/uploads/2018/10/Evidence_Aid_Practice_Guide_52pp_DIGITAL-FINAL-VERSION- 2018-10-22.pdf (accessed March 10, 2020). Bossuyt, P. M., J. B. Reitsma, D. E. Bruns, C. A. Gatsonis, P. P. Glasziou, L. Irwig, J. G. Lijmer, D. Moher, D. Rennie, H. C. W. de Vet, H. Y. Kressel, N. Rifai, R. M. Golub, D. G. Altman, L. Hooft, D. A. Korevaar, and J. F. Cohen. 2015. STARD 2015: An updated list of essential items for reporting diagnostic accuracy studies. Radiology 277(3):826–832. Bromley, E., D. P. Eisenman, A. Magana, M. Williams, B. Kim, M. McCreary, A. Chandra, and K. B. Wells. 2017. How do communities use a participatory public health approach to build resilience? The Los Angeles County Community Disaster Resilience Project. International Journal of Environmental Research and Public Health 14(10). Brown, C. H., G. Curran, L. A. Palinkas, G. A. Aarons, K. B. Wells, L. Jones, L. M. Collins, N. Duan, B. S. Mittman, A. Wallace, R. G. Tabak, L. Ducharme, D. A. Chambers, G. Neta, T. Wiley, J. Landsverk, K. Cheung, and G. Cruden. 2017. An overview of research and evaluation designs for dissemination and implementation. Annual Review of Public Health 38(1):1–22. Brownson, R. C., G. A. Colditz, and E. K. Proctor. 2012. Dissemination and implementation research in health: Translating science to practice. Oxford; New York: Oxford University Press. Brownson, R. C., J. E. Fielding, and L. W. Green. 2018. Building capacity for evidence-based public health: Reconciling the pulls of practice and the push of research. Annual Review of Public Health 39(1):27–53. Burlig, F. 2018. Improving transparency in observational social science research: A pre-analysis plan approach. Economics Letters 168:56–60. Carbone, E. G., and E. V. Thomas. 2018. Science as the basis of public health emergency preparedness and response practice: The slow but crucial evolution. American Journal of Public Health 108(S5):S383–S386. CTEC (California Tribal Epidemiology Center). 2016. Health policy brief—Barriers to collaboration between tribal and county governments: Planning for major disasters and other emergencies. Sacramento, CA. http://www.publichealthsystems.org/sites/default/files/PHS4/72458GPreport_06.pdf (accessed May 14, 2020). CDC (Centers for Disease Control and Prevention). 2007. CDC mentored public health research scientist development award (K01). https://grants.nih.gov/grants/guide/rfa-files/RFA-CD-07- 003.html#PartII (accessed February 12, 2020). CDC. 2019. Public health emergency preparedness (PHEP) cooperative agreement. CDC-RFA-TP19- 1901. https://www.grants.gov/web/grants/view-opportunity.html?oppId=310318 (accessed May 14, 2020). CDC. 2020a. Epidemic intelligence service. https://www.cdc.gov/eis/index.html (accessed March 13, 2020). CDC. 2020b. Grants. https://www.cdc.gov/grants/index.html (accessed May 13, 2020). Cha, B. S., R. I. Lawrence, J. C. Bliss, K. B. Wells, A. Chandra, and D. P. Eisenman. 2016. The road to resilience: Insights on training community coalitions in the Los Angeles County Community Disaster Resilience Project. Disaster Medicine and Public Health Preparedness 10(6):812–821. Challen, K., A. C. Lee, A. Booth, P. Gardois, H. B. Woods, and S. W. Goodacre. 2012. Where is the evidence for emergency planning? A scoping review. BMC Public Health 12(542). Chandra, A., M. Williams, A. Plough, A. Stayton, K. B. Wells, M. Horta, and J. Tang. 2013. Getting actionable about community resilience: The Los Angeles County Community Disaster Resilience Project. American Journal of Public Health 103(7):1181–1189. Chandra, A., M. V. Williams, C. Lopez, J. Tang, D. Eisenman, and A. Magana. 2015. Developing a tabletop exercise to test community resilience: Lessons from the Los Angeles County Community Disaster Resilience Project. Disaster Medicine and Public Health Preparedness 9(5):484–488. PREPUBLICATION COPY: UNCORRECTED PROOFS

IMPROVING AND EXPANDING THE PHEPR EVIDENCE BASE 8-45 Chi, G. C., M. Williams, A. Chandra, A. Plough, and D. Eisenman. 2015. Partnerships for community resilience: Perspectives from the Los Angeles County Community Disaster Resilience Project. Public Health. http://www.escholarship.org/uc/item/30f4f1kf (accessed March 10, 2020). Davies, R., E. Vaughan, G. Fraser, R. Cook, M. Ciotti, and J. E. Suk. 2019. Enhancing reporting of after action reviews of public health emergencies to strengthen preparedness: A literature review and methodology appraisal. Disaster Medicine and Public Health Preparedness 13(3):618–625. DeVoe, J. E., S. Likumahuwa, P. Eiff, C. A. Nelson, J. E. Carroll, C. N. Hill, R. Gold, and P. A. Kullberg. 2012. Developing a new practice-based research network (PBRN): Lessons learned and challenges ahead. Journal of the American Board of Family Medicine 25(5). DHS (U.S. Department of Homeland Security). 2020. Welcome to the Centers of Excellence. https://www.dhs.gov/science-and-technology/centers-excellence (accessed February 21, 2020). DOI (U.S. Department of the Interior). 2020. Strategic sciences group. https://www.doi.gov/strategicsciences (accessed May 22, 2020). Dungan, R., R. Angove, E. Cope, and H. Peay. 2019. Engagement science: Introducing inclusive research practices and potential impacts. AcademyHealth. https://www.academyhealth.org/blog/2019-01/engagement-science-introducing-inclusive- research-practices-potential-impacts (accessed May 13, 2020). ECDC (European Centre for Disease Prevention and Control). 2018. Best practice recommendations for conducting after action reviews to enhance public health preparedness. Solna, Sweden. https://www.ecdc.europa.eu/sites/default/files/documents/public-health-preparedness-best- practice-recommendations.pdf (accessed March 10, 2020). Eisenman, D. P., R. M. Adams, C. M. Lang, M. Prelip, A. Dorian, J. Acosta, D. Glik, and M. Chinman. 2018. A program for local health departments to adapt and implement evidence-based emergency preparedness programs. American Journal of Public Health 108(S5):S396–S398. FEMA (Federal Emergency Management Agency). 2013. Homeland security exercise and evaluation program. https://www.fema.gov/media-library-data/20130726-1914-25045- 8890/hseep_apr13_.pdf (accessed April 1, 2019). FEMA. 2020. Emergency management institute: Distance learning. https://training.fema.gov/is (accessed May 13, 2020). Ford, I., and J. Norrie. 2016. Pragmatic trials. New England Journal of Medicine 375(5):454–463. Forrest, C. B., F. D. Chesley Jr, M. L. Tregear, and K. B. Mistry. 2018. Development of the learning health system researcher core competencies. Health Services Research 53(4):2615–2632. GAO (U.S. Government Accountability Office). 2016. Tiered evidence grants: Opportunities exist to share lessons from early implementation and inform future federal efforts. Washington, DC. https://www.gao.gov/assets/680/679917.pdf (accessed March 10, 2020). Gliklich, R., N. Dreyer, and M. Leavy. 2014. Protecting data: Confidentiality and legal concerns of providers, manufacturers, and health plans. In Registries for evaluating patient outcomes: A user’s guide, 3rd ed. Rockville, MD: Agency for Healthcare Research and Quality. Gossip, K., H. Gouda, Y. Y. Lee, S. Firth, R. Bermejo, 3rd, W. Zeck, and E. Jimenez Soto. 2017. Monitoring and evaluation of disaster response efforts undertaken by local health departments: A rapid realist review. BMC Health Services Research 17(1):450. Guise, J.-M., M. E. Butler, C. Chang, M. Viswanathan, T. Pigott, and P. Tugwell. 2017a. AHRQ series on complex intervention systematic reviews—Paper 6: PRISMA-CI extension statement and checklist. Journal of Clinical Epidemiology 90:43–50. Guise, J.-M., M. Butler, C. Chang, M. Viswanathan, T. Pigott, and P. Tugwell. 2017b. AHRQ series on complex intervention systematic reviews—Paper 7: PRISMA-CI elaboration and explanation. Journal of Clinical Epidemiology 90:51–58. Hemming, K., T. P. Haines, P. J. Chilton, A. J. Girling, and R. J. Lilford. 2015. The stepped wedge cluster randomised trial: Rationale, design, analysis, and reporting. BMJ 350:h391. Hoffmann, T. C., P. P. Glasziou, I. Boutron, R. Milne, R. Perera, D. Moher, D. G. Altman, V. Barbour, H. Macdonald, M. Johnston, S. E. Lamb, M. Dixon-Woods, P. McCulloch, J. C. Wyatt, A. W. Chan, PREPUBLICATION COPY: UNCORRECTED PROOFS

8-46 EVIDENCE-BASED PRACTICE FOR PHEPR and S. Michie. 2014. Better reporting of interventions: Template for intervention description and replication (TIDieR) checklist and guide. BMJ 348:g1687. HHS (U.S. Department of Health and Human Services). 2010. Evidence-based clinical and public health: Generating and applying the evidence. https://www.healthypeople.gov/sites/default/files/EvidenceBasedClinicalPH2010.pdf (accessed March 10, 2020). HHS. 2017a. Public health emergency medical countermeasures enterprise (PHEMCE) strategy and implementation plan. https://www.phe.gov/Preparedness/mcm/phemce/Pages/strategy.aspx (accessed March 10, 2020). HHS. 2017b. Public health emergency medical countermeasures enterprise multiyear budget: Fiscal years 2016–2020. https://www.phe.gov/Preparedness/mcm/phemce/phemce- myb/Pages/default.aspx (accessed March 10, 2020). HHS and NIH (National Institutes of Health). 2020. Public health emergency research review board (PHERRB). https://ohsr.od.nih.gov/pherrb.php (accessed February 21, 2020). HSDL (Homeland Security Digital Library). 2020. Lessons learned information sharing. https://www.hsdl.org/c/llis-in-the-hsdl (accessed May 13, 2020). IAEM (International Association of Emergency Managers). 2020. Certification introduction. https://www.iaem.org/certification/intro (accessed May 13, 2020). IES and NSF (Institute of Education Sciences and National Science Foundation). 2013. Common guidelines for education research and development. https://www.nsf.gov/pubs/2013/nsf13126/nsf13126.pdf?WT.mc_id=USNSF_124 (accessed February 21, 2020). INTRAC (International NGO Training and Research Centre). 2017a. Qualitative comparative analaysis (QCA). https://www.intrac.org/wpcms/wp-content/uploads/2017/01/Qualitative-comparative- analysis.pdf (accessed March 10, 2020). INTRAC. 2017b. Realist evaluation. https://www.intrac.org/wpcms/wp-content/uploads/2017/01/Realist- evaluation.pdf (accessed March 10, 2020). IOM (Institute of Medicine). 2008. Research priorities in emergency preparedness and response for public health systems: A letter report. Washington, DC: The National Academies Press. IOM. 2015. Enabling rapid and sustainable public health research during disasters: Summary of a joint workshop by the Institute of Medicine and the U.S. Department of Health and Human Services. Washington, DC: The National Academies Press. Jacobs, J. A., E. A. Dodson, E. A. Baker, A. D. Deshpande, and R. C. Brownson. 2010. Barriers to evidence-based decision making in public health: A national survey of chronic disease practitioners. Public Health Reports 125(5):736–742. Jillson, I. A., M. Clarke, C. Allen, S. Waller, T. Koehlmoos, W. Mumford, J. Jansen, K. McKay, and A. Trant. 2019. Improving the science and evidence base of disaster response: A policy research study. BMC Health Services Research 19(1):274. Kearns, S. 2010. Good Catch program encourages reporting near-miss medical errors. https://www.healthleadersmedia.com/clinical-care/good-catch-program-encourages-reporting- near-miss-medical-errors (accessed March 10, 2020). Keim, M. E., T. D. Kirsch, O. Alleyne, G. Benjamin, L. DeGutis, D. Dyjack, and F. M. Burkle, Jr. 2019. The need for a national strategy to assess and reduce disaster-related mortality in the United States. American Journal of Public Health 109(4):539–540. Kirsch, T. D., G. Benjamin, P. D. Kivela, D. T. Dyjack, and M. Keim. 2018. National Disaster Safety Board: Objective measures, improved outcomes. https://www.healthaffairs.org/do/10.1377/hblog20180720.168527/full (accessed March 10, 2020). Lavis, J. N., A. D. Oxman, S. Lewin, and A. Fretheim. 2009. SUPPORT tools for evidence-informed health policymaking (STP). Health Research Policy and Systems 7(Suppl 1):I1. https://health- PREPUBLICATION COPY: UNCORRECTED PROOFS

IMPROVING AND EXPANDING THE PHEPR EVIDENCE BASE 8-47 policy-systems.biomedcentral.com/articles/supplements/volume-7-supplement-1 (accessed March 10, 2020). Lewin, S., A. D. Oxman, J. N. Lavis, A. Fretheim, S. Garcia Marti, and S. Munabi-Babigumira. 2009. SUPPORT tools for evidence-informed policymaking in health 11: Finding and using evidence about local conditions. Health Research Policy and Systems 7(Suppl 1):S11. https://health-policy- systems.biomedcentral.com/articles/10.1186/1478-4505-7-S1-S11 (accessed March 10, 2020). Lupia, A., and G. Alter. 2014. Data access and research transparency in the quantitative tradition. Political Science and Politics 47(1):54–59. Lurie, N., T. Manolio, A. P. Patterson, F. Collins, and T. Frieden. 2013. Research as a part of public health emergency response. New England Journal of Medicine 368(13):1251–1255. Maddock, J. E. 2018. Preparing public health for the unexpected. American Journal of Public Health 108(S5):S348. Maynard, R. A. 2018. The role of federal agencies in creating and administering evidence-based policies. The ANNALS of the American Academy of Political and Social Science 678(1):134–144. Mays, G. P., R. A. Hogg, D. M. Castellanos-Cruz, A. G. Goover, and L. C. Fowler. 2013. Engaging public health settings in research implementation and translation activities: Evidence from practice-based research networks. American Journal of Preventive Medicine 45(6):752–762. Miller, A., K. Yeskey, S. Garantziotis, S. Arnesen, A. Bennett, L. O’Fallon, C. Thompson, L. Reinlib, S. Masten, J. Remington, C. Love, S. Ramsey, R. Rosselli, B. Galluzzo, J. Lee, R. Kwok, and J. Hughes. 2016. Integrating health research into disaster response: The new NIH disaster research response program. International Journal of Environmental Research and Public Health 13(7). Moher, D., A. Liberati, J. Tetzlaff, and D. G. Altman. 2009. Preferred reporting items for systematic reviews and meta-analyses: The prisma statement. PLOS Medicine 6(7):e1000097. Moravcsik, A. 2014. Transparency: The revolution in qualitative research. Political Science and Politics 47(1):48–53. MRC (Medical Research Council). 2019. Developing and evaluating complex interventions. https://mrc.ukri.org/documents/pdf/complex-interventions-guidance (accessed February 21, 2020). National Hazards Center. 2019. Quick Response research grant program. https://hazards.colorado.edu/research/quick-response (accessed January 9, 2020). NBSB (National Biodefense Science Board). 2011. Call to action: Include scientific investigations as an integral component of disaster planning and response. https://www.phe.gov/Preparedness/legal/boards/nbsb/Documents/nbsbrec14.pdf (accessed March 10, 2020). NCER (National Center for Education Research). 2020. Predoctoral interdisciplinary research training programs in the education sciences. https://ies.ed.gov/ncer/projects/program.asp?ProgID=16 (accessed March 10, 2020). NIH (National Institutes of Health). 2019a. About the Division of Biomedical Research Workforce (DBRW). https://researchtraining.nih.gov/dbrw (accessed February 21, 2020). NIH. 2019b. NIH disaster research response training and exercises. https://dr2.nlm.nih.gov/training- exercises#NIEHS_Disaster_Research_Response_Tabletop_Exercise_Assessment (accessed February 21, 2020). NIH. 2019c. Types of grant programs. https://grants.nih.gov/grants/funding/funding_program.htm#RSeries (accessed May 13, 2020). NIH. 2020a. P30 center core grant. https://www.niddk.nih.gov/research-funding/process/apply/funding- mechanisms/p30 (accessed May 13, 2020). NIH. 2020b. Research career development awards. https://researchtraining.nih.gov/programs/career- development (accessed May 13, 2020). NORC at the University of Chicago. 2017. Evaluation of the translation, dissemination, and implementation of public health preparedness response research and training project. Bethessda, MD. PREPUBLICATION COPY: UNCORRECTED PROOFS

8-48 EVIDENCE-BASED PRACTICE FOR PHEPR https://s3.amazonaws.com/ASPPH_Media_Files/Docs/CDC_Preparedness_Evaluation_Report (accessed March 10, 2020). NSF (National Science Foundation). 2018. Proposal preparation instructions: Rapid research (RAPID) proposal. https://www.nsf.gov/pubs/policydocs/pappg18_1/pappg_2.jsp#IIE1 (accessed January 9 2020). NSF. 2020a. Accelerating discovery: Educating the future STEM workforce. https://nsf.gov/funding/pgm_summ.jsp?pims_id=505552&org=EHR&sel_org=EHR&from=fund (accessed February 21, 2020). NSF. 2020b. About funding. https://www.nsf.gov/funding/aboutfunding.jsp (accessed May 13, 2020). O’Brien, B. C., I. B. Harris, T. J. Beckman, D. A. Reed, and D. A. Cook. 2014. Standards for reporting qualitative research: A synthesis of recommendations. Academic Medicine 89(9):1245–1251. O’Fallon, L. R., G. M. Wolfle, D. Brown, A. Dearry, and K. Olden. 2003. Strategies for setting a national research agenda that is responsive to community needs. Environmental Health Perspectives 111(16):1855–1860. Ogrinc, G., L. Davies, D. Goodman, P. Batalden, F. Davidoff, and D. Stevens. 2016. Squire 2.0 (Standards for QUality Improvement Reporting Excellence): Revised publication guidelines from a detailed consensus process. BMJ Quality and Safety 25(12):986–992. Open Philanthropy. 2020. Grants database. https://www.openphilanthropy.org/giving/grants (accessed May 13, 2020). Packenham, J. P., R. T. Rosselli, S. K. Ramsey, H. A. Taylor, A. Fothergill, J. Slutsman, and A. Miller. 2017. Conducting science in disasters: Recommendations from the NIEHS working group for special IRB considerations in the review of disaster related research. Environmental Health Perspectives 125(9):094503. Patel, S. 2019. Quality assessment of after action reports: Findings and recommendations. Paper commissioned by the Committee on Evidence-Based Practices for Public Health Emergency Preparedness and Response. Patsopoulos, N. A. 2011. A pragmatic view on pragmatic trials. Dialogues in Clinical Neuroscience 13(2):217–224. PCORI (Patient-Centered Outcomes Research Institute). 2019. PCORI methodology standards. https://www.pcori.org/research-results/about-our-research/research-methodology/pcori- methodology-standards#Complex (accessed February 21, 2020). PHF (Public Health Foundation). 2020. Welcome to the TRAIN learning network. https://www.train.org/main/welcome (accessed May 13, 2020). Piltch-Loeb, R., J. D. Kraemer, C. Nelson, and M. A. Stoto. 2014a. A public health emergency preparedness critical incident registry. Biosecurity and Bioterrorism: Biodefense Strategy, Practice, and Science 12(3). Piltch-Loeb, R. N., C. D. Nelson, J. D. Kraemer, E. Savoia, and M. A. Stoto. 2014b. A peer assessment approach for learning from public health emergencies. Public Health Reports 129(Suppl 4):28– 34. Pinnock, H., E. Epiphaniou, A. Sheikh, C. Griffiths, S. Eldridge, P. Craig, and S. J. Taylor. 2015. Developing standards for reporting implementation studies of complex interventions (STaRI): A systematic review and e-Delphi. Implementation Science 10(42). Qari, S. H., M. R. Leinhos, T. N. Thomas, and E. G. Carbone. 2018. Overview of the translation, dissemination, and implementation of public health preparedness and response research and training initiative. American Journal of Public Health 108(S5):S355–S362. Revere, D., S. Allan, H. Karasz, and J. Baseman. 2018. Expanding methodologies to identify high- priority emergency preparedness tools for implementation in public health agencies. American Journal of Public Health 108(S5):S372–S374. RWJF (Robert Wood Johnson Foundation). 2013. Robert Wood Johnson Foundation practice-based research networks in public health. https://www.rwjf.org/en/library/research/2013/06/robert- wood-johnson-foundation-practice-based-research-networks-.html (accessed May 13, 2020). PREPUBLICATION COPY: UNCORRECTED PROOFS

IMPROVING AND EXPANDING THE PHEPR EVIDENCE BASE 8-49 RWJF. 2020a. Grants and grant programs. https://www.rwjf.org/en/how-we-work/grants-and-grant- programs.html (accessed May 13, 2020). RWJF. 2020b. Leadership for better health. https://www.rwjf.org/en/our-focus-areas/focus-areas/health- leadership.html (accessed May 13, 2020). RWJF and University of Kentucky College of Public Health. 2020. The national coordinating center for public health services and systems research and public health practice-based research networks. http://www.publichealthsystems.org (accessed May 13, 2020). Savoia, E., F. Agboola, and P. D. Biddinger. 2012. Use of after action reports (AARs) to promote organizational and systems learning in emergency preparedness. International Journal of Environmental Research and Public Health 9(8):2949–2963. Siegfried, A. L., E. G. Carbone, M. B. Meit, M. J. Kennedy, H. Yusuf, and E. B. Kahn. 2017. Identifying and prioritizing information needs and research priorities of public health emergency preparedness and response practitioners. Disaster Medicine and Public Health Preparedness 11(5):552–561. Simera, I., D. G. Altman, D. Moher, K. F. Schulz, and J. Hoey. 2008. Guidelines for reporting health research: The Equator Network’s survey of guideline authors. PLOS Medicine 5(6):e139. Singleton, C. M., S. Debastiani, D. Rose, and E. B. Kahn. 2014. An analysis of root cause identification and continuous quality improvement in public health H1N1 after-action reports. Journal of Public Health Management and Practice 20(2):197. Stoto, M. A., C. Nelson, M. A. Higdon, J. Kraemer, L. Hites, and C. M. Singleton. 2013. Lessons about the state and local public health system response to the 2009 H1N1 pandemic: A workshop summary. Journal of Public Health Management and Practice 19(5):428–435. Stoto, M., A., R. Piltch-Loeb, and E. Savoia. 2015. The public health system response to the 2014 West Virginia water crisis. https://cdn1.sph.harvard.edu/wp-content/uploads/sites/1609/2015/01/WV- water-crisis-report-final.pdf (accessed March 10, 2020). Stoto, M. A., C. Nelson, R. Piltch-Loeb, L. N. Mayigane, F. Copper, and S. Chungong. 2019. Getting the most from after action reviews to improve global health security. Globalization and Health 15(1):58. http://creativecommons.org/licenses/by/4.0 (accessed June 17, 2020). Summers, S. K., and M. J. Ferraro. 2017. Project Public Health Ready: History and evolution of a best practice for public health preparedness planning. American Journal of Public Health 107(S2):S138–S141. Taylor, H. L., and V. A. Yeager. 2019. Core competency gaps among governmental public health employees with and without a formal public health degree. Journal of Public Health Management Practice. Testa, M. A., E. Savoia, M. Su, and P. D. Biddinger. 2018. Social media learning collaborative for public health preparedness. American Journal of Public Health 108(S5):S375–S377. Teti, M., E. Schatz, and L. Liebenberg. 2020. Methods in the time of COVID-19: The vital role of qualitative inquiries. International Journal of Qualitative Methods 19(1–5). Tong, A., P. Sainsbury, and J. Craig. 2007. Consolidated Criteria for Reporting Qualitative Research (COREQ): A 32-item checklist for interviews and focus groups. International Journal for Quality in Health Care 19(6):349–357. Turner, D. A., J. Bae, G. Cheely, J. Milne, T. A. Owens, and C. M. Kuhn. 2018. Improving resident and fellow engagement in patient safety through a graduate medical education incentive program. Journal of Graduate Medicine Education 10(6):671–675. von Elm, E., D. G. Altman, M. Egger, S. J. Pocock, P. C. Gotzsche, and J. P. Vandenbroucke. 2007. The Strengthening the Reporting of OBservational studies in Epidemiology (STROBE) statement: Guidelines for reporting observational studies. Annals of Internal Medicine 147(8):573–577. Wandersman, A., J. Duffy, P. Flaspohler, R. Noonan, K. Lubell, L. Stillman, M. Blachman, R. Dunville, and J. Saul. 2008. Bridging the gap between prevention research and practice: The interactive systems framework for dissemination and implementation. American Journal of Community Psychology 41(3–4):171–181. PREPUBLICATION COPY: UNCORRECTED PROOFS

8-50 EVIDENCE-BASED PRACTICE FOR PHEPR Whitehurst, G. J. 2018. The Institute of Education Sciences: A model for federal research offices. The ANNALS of the American Academy of Political and Social Science 678(1):124–133. Williams, M. V., A. Chandra, A. Spears, D. Varda, K. B. Wells, A. L. Plough, and D. P. Eisenman. 2018. Evaluating community partnerships addressing community resilience in Los Angeles, California. International Journal of Environmental Research and Public Health 15(4). Wimsatt, M. A. 2017. Cross-jurisdictional sharing for emergency management-related public health: Exploring the experiences of tribes and counties in California. Frontiers in Public Health 5(254). https://doi.org/10.3389/fpubh.2017.00254. PREPUBLICATION COPY: UNCORRECTED PROOFS

Next: Appendix A: Detailed Description of the Committee's Methods for Formulating the Scope of the Reviews and Capturing the Evidence »
Evidence-Based Practice for Public Health Emergency Preparedness and Response Get This Book
×
Buy Prepub | $135.00 Buy Paperback | $125.00
MyNAP members save 10% online.
Login or Register to save!
Download Free PDF

When communities face complex public health emergencies, state local, tribal, and territorial public health agencies must make difficult decisions regarding how to effectively respond. The public health emergency preparedness and response (PHEPR) system, with its multifaceted mission to prevent, protect against, quickly respond to, and recover from public health emergencies, is inherently complex and encompasses policies, organizations, and programs. Since the events of September 11, 2001, the United States has invested billions of dollars and immeasurable amounts of human capital to develop and enhance public health emergency preparedness and infrastructure to respond to a wide range of public health threats, including infectious diseases, natural disasters, and chemical, biological, radiological, and nuclear events. Despite the investments in research and the growing body of empirical literature on a range of preparedness and response capabilities and functions, there has been no national-level, comprehensive review and grading of evidence for public health emergency preparedness and response practices comparable to those utilized in medicine and other public health fields.

Evidence-Based Practice for Public Health Emergency Preparedness and Response reviews the state of the evidence on PHEPR practices and the improvements necessary to move the field forward and to strengthen the PHEPR system. This publication evaluates PHEPR evidence to understand the balance of benefits and harms of PHEPR practices, with a focus on four main areas of PHEPR: engagement with and training of community-based partners to improve the outcomes of at-risk populations after public health emergencies; activation of a public health emergency operations center; communication of public health alerts and guidance to technical audiences during a public health emergency; and implementation of quarantine to reduce the spread of contagious illness.

  1. ×

    Welcome to OpenBook!

    You're looking at OpenBook, NAP.edu's online reading room since 1999. Based on feedback from you, our users, we've made some improvements that make it easier than ever to read thousands of publications on our website.

    Do you want to take a quick tour of the OpenBook's features?

    No Thanks Take a Tour »
  2. ×

    Show this book's table of contents, where you can jump to any chapter by name.

    « Back Next »
  3. ×

    ...or use these buttons to go back to the previous chapter or skip to the next one.

    « Back Next »
  4. ×

    Jump up to the previous page or down to the next one. Also, you can type in a page number and press Enter to go directly to that page in the book.

    « Back Next »
  5. ×

    To search the entire text of this book, type in your search term here and press Enter.

    « Back Next »
  6. ×

    Share a link to this book page on your preferred social network or via email.

    « Back Next »
  7. ×

    View our suggested citation for this chapter.

    « Back Next »
  8. ×

    Ready to take your reading offline? Click here to buy this book in print or download it as a free PDF, if available.

    « Back Next »
Stay Connected!