A

Detailed Description of the Committee’s Methods for Formulating the Scope of the Reviews and Capturing the Evidence

The committee was charged with conducting a systematic review of the evidence base for selected public health emergency preparedness and response (PHEPR) practices from the 15 PHEPR Capabilities defined in the Centers for Disease Control and Prevention’s (CDC’s) Public Health Emergency Preparedness and Response Capabilities: National Standards for State, Local, Tribal, and Territorial Public Health (CDC, 2018). This appendix provides additional detail on some aspects of the committee’s evidence review methods, which are briefly described in Chapter 3. Specifically, this appendix details the committee’s review methods only to the point of data extraction and quality assessment of individual studies. The committee’s evidence synthesis and grading methodology is described in detail in Chapter 3. For examples of how the methods were applied to the review topics, see Chapters 4–7.

To develop its evidence review and evaluation methodology, the committee reviewed the standards, protocols, and best practices from several prominent guideline groups and their relevant publications (see Box A-1), although other sources were also consulted. Recognizing the complexity of PHEPR practices, the committee focused substantial effort up front on formulating the scope of the reviews and adapting review methods that would take into account practice and system complexity. Additional information regarding the complexity perspective that guided the committee’s approach to its task can be found in Chapter 3.

FORMULATING THE SCOPE OF THE REVIEWS

Selecting the Review Topics

In response to its charge, the committee developed a process for identifying which of the CDC PHEPR Capabilities would be the focus of its systematic literature reviews. Recognizing that the PHEPR Capabilities are broad, overarching topics, the committee gradually focused the scope of its reviews to specific PHEPR practices within the selected PHEPR Capabilities.

The evidence review topics were scoped through iterative stages with input from the literature, key stakeholders, and the committee’s expertise.

Preliminary Literature Review

To better understand the current evidence base for PHEPR practices, the committee conducted a preliminary literature review. This preliminary review helped formulate the scope of the final reviews and frame the key review questions that governed the committee’s systematic searches of the evidence. The committee focused its efforts on reviewing published scoping and literature reviews on the PHEPR literature (Abramson et al., 2007; Acosta et al., 2009; Challen et al., 2012; Khan et al., 2015; Savoia et al., 2017; Yeager et al., 2010) to get a sense of the research gaps in the field, as well as PHEPR practitioner and program assessments, when relevant, to understand the type of research that would be relevant and important to practitioners (CDC, 2016; Center for Public Health Systems and Services Research, 2018; Horney et al., 2017; Murthy et al., 2017; Siegfried et al., 2017).

Engaging with Stakeholders to Meet Practitioner Needs

To ensure that its report would be relevant and useful to key stakeholders, the committee appointed nine diverse state, local, tribal, and territorial (SLTT) PHEPR practitioners as consultants to advise on the systematic literature review process. The members of this group were suggested by practitioner associations, such as the National Association of County & City Health Officials (NACCHO) and the Association of State and Territorial Health Officials (ASTHO), and selected based on such criteria as jurisdiction type and demographics, size

of public health agency, individual tenure and experience, number of emergencies during tenure, the Federal Emergency Management Agency (FEMA) region, and state governance, as well as meeting conflict-of-interest requirements. The committee engaged with these PHEPR practitioner consultants throughout the review process on the refinement of its conceptual approach and the selection, development, and refinement of review topics, and solicited their feedback on the review findings and the committee’s recommendations. The PHEPR practitioner consultants provided real-world input and assisted in focusing the review topics on issues relevant to decision makers.

Selection Criteria and Selected Review Topics

In making decisions about the scope of its reviews, the committee, which included individuals with diverse expertise in evidence review and evaluation methodologies and the PHEPR subject matter (see Appendix F for biosketches of the committee members), considered criteria adapted from The Community Guide (Zaza et al., 2000) and the U.S. Preventive Services Task Force (USPSTF, 2015). These criteria are described in Box A-2.

Selected review topics

The committee determined that, to develop an evidence review and evaluation methodology and demonstrate its feasibility, the task of selecting review topics needed to be approached from a “proof-of-concept” perspective. Therefore, for the initial development of its methodology for generating recommendations for evidence-based PHEPR practices, the committee applied the selection criteria discussed in Box A-2 and narrowed the scope of its reviews to four PHEPR Capabilities: Community Preparedness, Emergency Operations Coordination, Information Sharing, and Non-Pharmaceutical Interventions. In consultation with the PHEPR practitioner consultant group, the committee further focused its reviews on four topic areas within each of these four PHEPR Capabilities:

• engaging with and training community-based partners to improve the outcomes of at-risk populations after public health emergencies (falls under Capability 1, Community Preparedness);
• activating a public health emergency operations center (Capability 3, Emergency Operations Coordination);
• communicating public health alerts and guidance with technical audiences during a public health emergency (Capability 6, Information Sharing); and
• implementing quarantine to reduce or stop the spread of a contagious disease (Capability 11, Non-Pharmaceutical Interventions).

The process of focusing the selected broad PHEPR Capabilities into more precise topics was a considerable challenge because the Capabilities are standards, not practices; thus, to achieve each Capability requires many different practices, which are not always evident. To guide the selection process, the committee developed a comprehensive list of potential PHEPR practices by breaking down the functions and tasks within the PHEPR Capabilities into more manageable topics.

Selection justification

It is important to note that the committee’s process for selecting review topics was systematic and based on classifying the topics on a number of key dimensions and criteria. The committee acknowledges that a different group might have chosen a different set of topics. Given that the committee was able to conduct reviews for only a very small subset of PHEPR practices, it sought to inform future priorities for review topics by commissioning a scoping review and evidence map to examine and describe the extent and the nature of research conducted on practices within all of the CDC PHEPR Capabilities (see Chapter 2 and Appendix D). The discussion below describes how the committee applied the criteria listed in Box A-2 to select review topics.

Relevance and importance to national health security

In thinking more broadly about the strategies, priorities, strengths, and weaknesses in PHEPR for the nation, the committee reviewed the 2018 National Health Security Preparedness Index;¹ Trust for America’s Health 2017 Ready or Not?: Protecting the Public’s Health from Diseases, Disasters, and Bioterrorism Report;² CDC’s Public Health Preparedness and Response 2018 National Snapshot;³ CDC’s Public Health Emergency Preparedness (PHEP) Cooperative Agreement funding announcement;⁴ and the Assistant Secretary for Preparedness and Response’s

___________________

¹ See https://nhspi.org (accessed June 18, 2020).

² See https://www.tfah.org/report-details/ready-or-not-2017 (accessed June 18, 2020).

³ See https://www.cdc.gov/cpr/pubs-links/2018/documents/2018_Preparedness_Report.pdf (accessed June 18, 2020).

⁴ See https://www.cdc.gov/cpr/readiness/phep.htm (accessed June 18, 2020).

(ASPR’s) National Health Security Strategy.⁵ The committee deliberated as to where a review and evaluation of the evidence might advance the capacity of the nation to prepare for and respond to disasters and public health emergencies.

Current needs among key stakeholders

In considering the current needs for research to support practices among key stakeholders, including practitioners and policy makers, the committee reviewed the research domains and questions deemed a priority by practitioners in Siegfried et al. (2017) and the findings from the 2016 practitioner assessment of the CDC PHEPR Capabilities (CDC, 2016). Siegfried and colleagues’ findings provided crucial insight into where PHEPR practitioners perceived knowledge gaps and where future research studies would provide the information they need to enhance their capacity. One key finding was that the research question within the community resilience domain related to appropriate methods and procedures for identifying and mapping at-risk populations received the highest importance rating from the practitioners surveyed. As described later in this appendix, the committee leveraged this important work in conducting a structured priority-setting activity to identify priority topics for future systematic evidence reviews (refer to the section below on “Prioritization of Future Systematic Evidence Review Topics”).

The committee also considered the input offered by CDC in its remarks to the committee at the outset of this study, as well as input from the PHEPR practitioner consultants. In the 2016 practitioner assessment of the CDC PHEPR Capabilities, it was noted that such Capabilities as Mass Care, Fatality Management, Community Recovery, and Medical Surge are often the responsibilities of agencies other than public health (CDC, 2016). In its remarks, CDC also noted that Volunteer Management is less critical for public health than other PHEPR Capabilities and that it should be of lower priority in the allocation of resources for review (Carbone, 2018). Public Health Laboratory Testing and Public Health Surveillance and Epidemiological Investigation are considered well-established public health functions (Carbone, 2018; CDC, 2016). Furthermore, the evidence supporting Emergency Public Information and Warning, Responder Safety and Health, and Medical Materiel Management and Distribution is derived primarily from disciplines other than public health. Therefore, the committee decided not to focus its efforts and resources on reviewing these Capabilities because an aim of the study was to develop PHEPR-specific methods.

Potential to affect PHEPR practice

Siegfried and colleagues (2017) note that knowledge gaps identified by the practice community may result from insufficient dissemination, ineffective implementation, or a lack of existing research. Abramson and colleagues (2007) state that it is fundamental for potential research questions to question assumptions (e.g., asking whether the way in which response systems have been organized using an incident command system is effective). The committee deliberated about those areas in which a review and evaluation of the evidence would have the highest potential to increase the implementation of effective practices and phase out widely used but less effective practices.

Methodological diversity

The committee approached its reviews as a proof of concept for its methodology for generating recommendations for evidence-based PHEPR practices. Therefore, the committee assigned this criterion more weight than the others to ensure that the methodology it developed would be flexible enough to accommodate the range of PHEPR practices that may be reviewed in the future. Through an initial review of the literature

___________________

⁵ See https://www.phe.gov/Preparedness/planning/authority/nhss/Documents/NHSS-Strategy-508.pdf (accessed June 18, 2020).

and its members’ expertise, the committee considered how the type of research required to support evidence-based practice recommendations might vary across the CDC PHEPR Capabilities. The type of research that falls within the Community Preparedness and Information Sharing Capabilities aligns more with traditional research designs, such as randomized controlled trials (RCTs) and quasi-experimental studies, whereas the type of research that falls within Emergency Operations Coordination aligns more with noncomparative studies and experience-based evidence, including case reports and after action reports (AARs), as well as organizational theory, systems, and processes. The committee was also interested in developing a methodology that would accommodate modeling studies and qualitative evidence, and the research within the Non-Pharmaceutical Interventions Capability includes a considerable number of such studies.

Additionally, the committee considered characteristics of the PHEPR practices within these four Capabilities that might differ in ways important to interpreting the evidence. For example, the Emergency Operations Coordination Capability is inward facing, with the aim of supporting the ongoing response effort, while the Non-Pharmaceutical Interventions Capability is outward facing, with the aim of helping the public. Likewise, practices within Information Sharing may need to be implemented more quickly than those within Community Preparedness in order to be effective. These characteristics, defined by the committee as classification dimensions (listed in Figure 3-1 in Chapter 3) were examined for each PHEPR Capability to aid in the selection of practices that were diverse with respect to those characteristics.

Developing the Analytic Frameworks and Key Review Questions

Once the four review topics had been selected, the committee focused on further describing and constraining the scope of its reviews through the development of a detailed analytic framework and set of key review questions for each topic. The analytic frameworks and key review questions for each of the four review topics can be found in Chapters 4–7 and Appendixes B1–B4.

Analytic Frameworks

The purpose of developing analytic frameworks is to present clearly in visual format the causal pathway and interactions between a practice and its components, populations, and outcomes of interest (i.e., a logic model). As described in Chapter 3, the analytic frameworks also facilitated a mixed-method approach to the committee’s systematic reviews by serving as a construct that enabled integration of the findings from separate syntheses of quantitative, qualitative, and case report and AAR evidence.

The committee developed analytic frameworks using its members’ experiential knowledge, feedback from the PHEPR practitioner consultants, and the available research. The primary results of this process are focused on the identification of the outcomes of a practice, its postulated harms, and key factors to examine for potential effect modification and applicability of the results to other contexts (e.g., populations, settings).

For the purposes of this review, the committee adapted The Community Guide approach for developing analytic frameworks (Briss et al., 2000). The committee used the analytic frameworks for the four practices to conceptualize their relationship to outcomes of interest. In addition to health outcomes (e.g., reduced morbidity and mortality), other outcomes of interest included intermediate outcomes (e.g., knowledge, behaviors), as well as system- and process-level outcomes (e.g., accelerated recovery and cohesive and effective operations,

respectively). These frameworks were not simple, linear models, and represented both direct pathways to outcomes (e.g., a study that directly assessed the impact of an intervention on the outcomes of interest), and indirect pathways (e.g., the intermediate steps or intermediate outcomes that led to an effect on the final outcomes of interest).

The committee had to make explicit judgments about the extent to which PHEPR practices are grouped together and considered in the same body of evidence because PHEPR practices are carried out in different settings and populations, and often implemented in different ways. If practices are defined very narrowly, there may not be sufficient evidence to evaluate effectiveness. However, when practices are grouped together, the available evidence may be heterogeneous. While that possibility poses a different kind of challenge for analyzing the evidence, it also makes it possible to assess generalizability and consistency across different contexts. Thus, the committee discussed the degree to which the grouping of PHEPR practices was needed to achieve a balance between sufficiency and heterogeneity of evidence. Similarly, the committee had to make judgments about how to define outcomes for its evidence synthesis, given the significant variation in the measures used in the studies included in its review. For example, rather than evaluating preparedness behaviors related to stockpiling supplies and developing a family communication plan separately, these outcome measures were grouped together into an outcome category of preparedness behavior.

Key Review Questions

As the committee generated an initial list of review questions for each review topic, it became clear that beyond questions about a practice’s effectiveness (i.e., what works), it was also important to consider questions about how and why it works, for whom, and under what circumstances. The initial question lists were then reviewed and prioritized in collaboration with the PHEPR practitioner consultants. The committee drafted an overarching question for each of the four topics, which was further broken down into several sub-questions that were addressed in the review as well (see Chapters 4–7 and Appendixes B1–B4). These sets of key review questions specified the logic and scope of the review of each topic and were critical in guiding the literature searches, data extraction, and evidence analyses.

The committee then applied the PICOTS (population, intervention/phenomena of interest, comparators, outcomes, timing, and setting) question formulation framework (Butler et al., 2017). The committee added timing and setting to the traditional PICO in an effort to capture important contextual factors. The PICOTS framework helped specify the committee’s search parameters and define the inclusion and exclusion criteria.

Throughout the process of framing questions, the committee returned to the issues of complexity and the need to answer questions that go beyond the effectiveness of a practice. These questions also relate to practitioners’ needs for recommendations that are practicable and evidence that can help them make informed decisions about response activities.

Topic Refinement: Iterative Process

When addressing such complex topics, finalizing analytic frameworks and key review questions a priori is often not a suitable approach. Therefore, the committee refined the analytic frameworks and key review questions iteratively as it explored the evidence and engaged with stakeholders.

LITERATURE SEARCH

Multicomponent Approach to Searching the Evidence

Recognizing that evaluations of PHEPR practices may be published not only as intervention research studies but also as organizational reports, white papers, or program evaluations, including AARs, the committee adopted a multicomponent approach to searching the evidence. This approach included searching bibliographic databases and gray literature sources, as well as issuing a call for reports.

A professional librarian worked closely with the committee to plan the literature search strategies, ensure the appropriate translation of the key review questions into relevant and accurate terms, and conduct the searches so as to identify relevant research. Specific details regarding the search strategies and article selection process for each of the four review topics can be found at the end of this appendix.

Bibliographic Database Search

The committee conducted a series of searches in four databases—PubMed, Scopus, Medline (Ovid), and Embase (Ovid)—between December 2017 and January 2019 to identify peer-reviewed literature for the four selected PHEPR practices. The committee applied a date limit of 2001 to the present, limited the literature to the English language, and excluded editorials from the search results. The first search, conducted in December 2017, captured 14 of the 15 CDC PHEPR Capabilities (excluding Community Recovery, which is out of scope) and leveraged search strategies from existing scoping reviews. The searches conducted in July 2018 and December 2018 (and updated in June 2019) focused on the selected four review topics. In January 2019, the committee conducted two expanded searches on Information Sharing and Emergency Operations Coordination in subject-specific databases. The complete search syntax and search terms for each topic can be found at the end of this appendix.

Gray Literature Search

The committee identified gray literature published by relevant domestic and international organizations and agencies. These entities included the Association of Public Health Laboratories, ASPR, ASTHO, CDC, the Center for Health Security, the Council of State and Territorial Epidemiologists, the European Centre for Disease Prevention and Control, the Disaster Information Management Research Center at the National Library of Medicine in the National Institutes of Health (NIH), the U.S. Department of Homeland Security, FEMA, the U.S. Government Accountability Office, NACCHO, the National Center for Disaster Medicine and Public Health (NCDMPH), Preparedness and Emergency Response Research Centers (PERRCs), Public Health Canada, Public Health England, RAND Corporation, and the World Health Organization. In addition, the committee obtained 370 AARs published from 2009 to 2019 from the Homeland Security Digital Library.⁶

Call for Reports

In addition to online searching, the committee proactively solicited reports, both published and unpublished, through a request for documents via internal listservs at the National Academies and external mechanisms. An online request was published on the webpage for

___________________

⁶ See https://www.hsdl.org/c (accessed May 26, 2020).

this study,⁷ and the National Academies’ Board on Health Sciences Policy distributed the call for reports through the Forum on Medical and Public Health Preparedness for Disasters and Emergencies and the Disaster Science Action Collaborative. Staff contacted CDC, the study sponsor, for document suggestions, and also asked the agency to disseminate the announcement to its networks, particularly the former PERRC and Preparedness and Emergency Response Learning Centers networks. Additionally, staff sent targeted emails to PHEPR practitioner associations (e.g., NACCHO and ASTHO) and disaster science organizations (e.g., NIH’s Disaster Research Response system, NCDMPH, and the Association of Schools and Programs of Public Health). Submissions were accepted through March 8, 2019. This proved to be an effective way of collecting AARs, theses, and white papers.

Article Selection Method

Selecting which articles to include in the committee’s reviews was a multistep process that involved developing inclusion and exclusion criteria, conducting an initial screening of titles and abstracts, and retrieving and reviewing selected full-text articles. Deciding which articles were relevant to the analytic frameworks and key review questions required significant judgment and thorough documentation. Preferred Reporting Items for Systematic Reviews and Meta-Analyses flowcharts for each review topic can be found in Chapters 4–7.

Inclusion and Exclusion Criteria

The committee developed specific inclusion and exclusion criteria based on the PICOTS framework for each of the four review topics. Generally, the committee did not exclude based on study design, lack of comparison groups, or lack of explicit outcomes (i.e., it included articles that describe lessons learned or present conclusions). Articles were excluded if they were editorials, opinion pieces, or commentaries with no indication of empirical evidence. A common set of inclusion and exclusion criteria was used for screening articles (see Box A-3).

Initial Screening and Full-Text Review

One staff member conservatively screened titles and abstracts for relevance to the four topic areas. Additional articles were identified for inclusion in the review through reference mining.

The next step was to review the full-text articles that had passed the first screen. Prior to that review, the process was pilot tested on a sample of articles, with screeners participating in a calibration training call. Two individuals, one committee member and one staff, then worked independently to review all selected full-text articles against the inclusion and exclusion criteria (see Box A-3). Discrepant articles were adjudicated primarily by the committee chair and in some instances by staff, and committee members were given the opportunity to review and object if necessary to the decisions made during the adjudication process. Articles on studies that used a clear research method were categorized as Tier 1, and articles that did not, including case reports, as Tier 2; both tiers were selected for extraction of key findings. Articles that were excluded but contained information that could provide background information for the review were categorized as background and kept for potential use at a later date. The committee used EndNote to manage its references and maintained in real time a detailed account of study selections and decisions.

___________________

⁷ See https://www.nationalacademies.org/our-work/evidence-based-practices-for-public-health-emergencypreparedness-and-response-assessment-of-and-recommendations-for-the-field (accessed May 26, 2020).

DATA EXTRACTION AND STUDY QUALITY ASSESSMENT

Studies captured in the literature searches for each review topic were placed in six categories:

• quantitative comparative studies,
• quantitative noncomparative (single-group) studies of specific interventions,
• surveys (descriptive only),

• simulation (and related) models,
• qualitative research studies, and
• AARs and case reports.

Mixed-method studies (having both quantitative and qualitative components) were included in both the quantitative and qualitative study categories, as appropriate. Data extraction and the quality assessment process were tailored as necessary to these categories of study types.

Data extraction and quality assessment for individual quantitative studies were performed by the Center for Evidence Synthesis in Health at Brown University, an evidence-based practice center (EPC) that conducts reviews for the Agency for Healthcare Research and Quality and others. Data extraction and quality assessment for the qualitative studies were performed by a commissioned team at Wayne State University. Prioritization of case reports and AARs for inclusion in the systematic reviews and data extraction for these reports were conducted by a PHEPR expert in evaluation at Columbia University. The evaluation and extraction of findings from selected modeling studies were performed by a modeling expert at Stanford University, as described further in Chapter 3.

Data Extraction

Data extracted from each study included the description of the practice being reviewed, elements needed to make determinations about the effect of the practice, and contextual

elements that would contribute to an assessment of the applicability of its results to other contexts (e.g., populations, settings). Extracted data elements for quantitative studies are listed in Box A-4.

Qualitative studies, AARs, and case reports were coded in accordance with the framework synthesis method (Barnett-Page and Thomas, 2009; Pope et al., 2000), as described in Chapter 3. The codebook for extracting study characteristics and findings was developed in consultation with the committee and National Academies staff. Training sessions for the use of the codebook were conducted with the research team, and a pilot test of the codebook portion on extracting study characteristics and findings was conducted to refine the process.

Individual Study Quality Assessment

The quality assessment methodology was determined based on study design. Many standardized tools are available for assessing quality or risk of bias (RoB), each with its own

merits and shortcomings, and new tools continue to be developed. Described here is the approach taken by the committee and the groups commissioned to assess study quality and RoB; however, different tools and methods could reasonably be applied in future PHEPR evidence reviews.

Quantitative Studies

For quantitative comparative studies, an assessment tool was developed by the Brown University EPC⁸ by drawing selected RoB domains from existing tools, including the Cochrane Risk of Bias version 2.0 tool (Higgins et al., 2019), Cochrane’s suggested risk-of-bias criteria for Effective Practice and Organisation of Care reviews (Cochrane, 2017), and the Cochrane Risk of Bias in Non-Randomized Studies of Interventions (ROBINS-I) tool (Sterne et al., 2016). The selection of RoB domains reflected the dual goals of adequately addressing important potential methodological concerns and being mindful of the resources and time available for assessing the methodological quality of studies with a wide range of potential study designs. The final set of domains and their assessment criteria are as follows:

• Study population (eligibility criteria): Was the included sample prespecified, clearly specified, defined, and uniformly applied? Low RoB if yes; High RoB if no. This domain is consistent across outcomes.
• Allocation concealment (and randomization method): For RCTs, was there a problem with the randomization method or allocation concealment? High RoB if yes; Low RoB if explicitly no problem; Unclear RoB if insufficient reporting to judge. For nonrandomized comparative studies (NRCSs) of different interventions, High RoB unless analytic methods were used to account adequately for inherent baseline differences in compared groups or if it is otherwise reasonable to assume that compared groups are sufficiently similar. If pre-post study (of a single group), then “None.” This domain is consistent across outcomes.
• Comparator group: Was the comparator group chosen from the same population, with the same general eligibility criteria, as the intervention group? For RCTs, Low RoB. For NRCSs, there is overlap between this assessment and the assessment of “Allocation” (see above). If pre-post study (of a single group), Low RoB (unless there is an indication that groups differed pre- and postintervention). This domain is consistent across outcomes.
• Sample size: Is there a justification of the sample size or power/analysis per outcome? High RoB if no; Low RoB if yes (and the sample size was reached) or if the analysis was statistically significant. This domain may differ for each outcome.
• Loss to follow-up: Was there high loss to follow-up, arbitrarily set at 20 percent, or was there unequal loss to follow-up between groups? This criterion is largely based on comparisons between enrolled (or randomized) individuals and the numbers analyzed. High RoB if yes; Low RoB if no. This domain may differ for each outcome.
• Outcome measurement or ascertainment bias: Was there a problem with how each outcome was measured? High RoB if unvalidated subjective outcome. For studies comparing different interventions, includes whether an outcome was measured differently in the different intervention groups. This domain may differ for each outcome.
• Group similarity at baseline: Were the groups (intervention and comparator) similar at baseline? If similar, Low RoB. If there was a (nonminor) difference, was the difference

___________________

⁸ See Appendix C.

• statistically accounted for for each outcome? Judgment of whether a difference was “nonminor” depended on both statistical and clinical significance. Unclear RoB only if baseline descriptions were omitted or were too sparse to evaluate for possible differences. If pre-post study (of a single group), Low RoB (unless there is an indication that groups differed pre- and postintervention). This domain may differ for each outcome (based primarily on whether adequate statistical adjustment was conducted).
• Outcome assessor blinding: Regardless of study design, was the outcome assessor blinded, or were methods used to minimize biased outcome assessment? “Hard” outcomes (unambiguous, potentially like death) or outcomes based on objective measurements (e.g., laboratory measurements or government records, such as number quarantined) generally qualified as Low RoB, as did outcomes that were explicitly blinded. Other outcomes from observational studies were assumed to have High RoB unless otherwise indicated. Self-reported outcomes were typically High RoB unless the participants were blinded to their intervention. This domain may differ for each outcome.
• Group differences and confounders: Did the analyses account for potential group differences or confounders, for example, by multivariable adjustment or propensity score analysis? For RCTs, Low RoB was assumed absent a suggestion of a lack of similarity between groups (despite randomization). For NRCSs, regardless of whether groups were similar at baseline, High RoB if there was no adjustment for potential differences or if adjustment was made only for something minor or insufficient (e.g., only sex across disparate populations). For pre-post studies, Low RoB (unless there was an indication that groups differed pre- and postintervention). This domain may differ for each outcome.
• Other important limitations per data extractor or as reported by study authors. This domain may differ for each outcome.

Each outcome of each quantitative comparative study was evaluated for all of the above domains. Then an overall assessment of the study (or outcome) methodology (rated as good, moderate, or poor) was made based on the judgment of the evaluators after considering the various bias domains, which were weighted differently for different study designs. Each study (and outcome) was assessed for methodological quality by the Brown team’s senior researcher and was reviewed by at least one other experienced team member and was altered in discussion as needed.

The Brown University EPC developed and applied a separate tool for the assessment of descriptive surveys, drawing on published methods (Bennett et al., 2010; Davids and Roman, 2014). Descriptive surveys were assessed using the following domains and assessment criteria:

• Adequacy of survey tool development: Low RoB: A priori methodology with group development and pretesting reported that survey has been validated and/or found reliable. High RoB: Lack of structured methodology for developing questions, single person or group developed, and/or no outside input or pilot, field, or pretesting of questions (or prior use). Unclear RoB: No or incomplete description of development process.
• Study population eligibility criteria prespecified and uniformly applied: Low RoB: Explicitly reported, clear, and no major deviations from protocol. High RoB: Not prespecified or major deviation from protocol. Unclear RoB: Not reported whether prespecified or deviation.

• Adequacy and appropriateness of polling/sampling methodology: Low RoB: Everyone who met criteria (universe, census); probability sampling (e.g., random selection of telephone, email, and text of population with high access to these technologies); other unbiased sampling of population of interest. High RoB: Problems such as that sampling was likely biased (e.g., texting may miss individuals of low socioeconomic status or those difficult to reach), nonprobability sample (e.g., for focus group, convenience sample); if sample of general population, no attempt to capture difficult-to-reach individuals (e.g., those with no phone, email). Unclear RoB: Not adequately described.
• Respondents nonrepresentative of the target population: Low RoB: Respondents representative of target population and not different from nonrespondents. High RoB: Explicitly nonrepresentative; respondents differed from nonrespondents or target population. Unclear RoB: No description of target population or nonrespondents (and not High RoB).
• Percentage who responded: The actual response rate, without a judgment of its adequacy.
• Information on margin of error reported: Low RoB: If margin-of-error calculations made and reported, the reported values were extracted. Unclear RoB: No information on margin-of-error calculations. (While margin of error is a concept related to precision and not bias, the same terminology [High, Low, Unclear] was used for clarity and consistency.)

Qualitative Studies

Quality assessment for qualitative studies was undertaken as a component of a qualitative evidence synthesis commissioned to Wayne State University. The qualitative studies meeting the inclusion criteria for each review topic were appraised individually using the Critical Appraisal Skills Programme (CASP, 2018) checklist, which is applicable to assessing qualitative research. Areas of appraisal by CASP include appropriateness of qualitative methodology, data collection, relationship between research and participants, ethics, rigor of data analysis, clarity of findings, and value of research. Each area was assessed using “yes,” “no,” or “can’t tell.” In line with the Grading of Recommendations Assessment, Development and Evaluation Confidence in the Evidence from Reviews of Qualitative Research (GRADE-CERQual) approach for assessing methodological limitations (Lewin et al., 2018), each study received a final overall quality rating of “no or very minor concerns” (no significant methodological limitations), “minor concerns” (minor methodological limitations not impacting the credibility and validity of findings), “moderate concerns” (some methodological limitations likely to impact the credibility and validity of findings), or “serious concerns” (serious methodological limitation impacting the credibility and validity of findings).

After Action Reports

An appraisal tool for evaluating the methodological rigor of AARs was published in 2018 (ECDC, 2018) and was applied by a commissioned PHEPR expert to the AARs included in the committee’s analyses. The 11-item tool is designed to assist with the systematic documentation of methods used in AARs, compare validity, and potentially inform best practices for a standard template. The tool includes the following criteria:

• prolonged engagement with the subject of inquiry,
• use of theory,

• data selection,
• information sampling,
• multiple data sources,
• triangulation,
• negative case analysis,
• peer debriefing and support,
• respondent validation,
• clear report of methods of data collection and analysis (audit trail), and
• depth and insight.

PRIORITIZATION OF FUTURE SYSTEMATIC EVIDENCE REVIEW TOPICS

The committee engaged with a second diverse group of 10 PHEPR practitioners, also representing SLTT agencies, in an open session to identify knowledge gaps that matter to practitioners and assess the relative priority, from their perspective, of potential evidence review topics encompassed within the CDC PHEPR Capabilities. This practitioner engagement activity was conducted after the committee’s four evidence review topics had been selected and therefore did not inform the selection process. The activity was intended to inform the identification of priorities for future PHEPR evidence reviews, as well as future research needs, to address critical knowledge gaps in PHEPR practice. For this prioritization activity, the committee modified an existing group judgment process originally developed to assess the appropriateness of medical procedures, but since modified for many other uses (i.e., modified Delphi-like process) (Shekelle, 2004; Sloss et al., 2000). This process involved the selection of this group of SLTT PHEPR practitioners, a review of the literature, and multiple rounds of group voting with feedback between rounds. This process was led by an experienced moderator.

This second group of PHEPR practitioners was identified through a process similar to that used to select the PHEPR practitioner consultants. The list of PHEPR topics included in this activity was adapted from an existing list of 44 research questions deemed by PHEPR practitioners to be priority areas (Siegfried et al., 2017). Building from that list, the committee combined several research questions into one topic area, removed several research questions that did not lend themselves to a review of the evidence, and added several questions that the committee derived from a review of the literature. This process resulted in 39 topic areas across the six PHEPR domains identified by Siegfried and colleagues: community resilience, incident management, information management, countermeasures and mitigation, surge management, and biosurveillance.

In January 2019, the committee engaged with these PHEPR practitioners in an open session in a virtual premeeting for the first round of voting and at an in-person meeting for the second round of voting. All 10 PHEPR practitioners participated in both rounds. The committee used Sli. Do, an online polling software, to conduct this activity.

The PHEPR practitioners were asked to rank the importance of the 39 topic areas on the committee’s list by rating each on a 5-point Likert rating scale—highest priority, high priority, moderate priority, low priority, and not a priority—as well as to provide any other comments or suggestions. They were provided with the following guidance with regard to rating the priority of topics:

Results from the first round (virtual premeeting) were compiled for the second round (in-person meeting), during which the PHEPR practitioners received feedback on the results from the first round, discussed those results, and revoted if doing so was warranted. The outcome of this process was a set of PHEPR review topics classified into the five Likert scale categories: highest priority, high priority, moderate priority, low priority, and not a priority (see Box A-5). This information, along with the published literature, aided the committee in identifying priorities for future evidence reviews, as well as future research needs, to address critical knowledge gaps in PHEPR practice.

DATABASE SEARCH STRATEGIES

Initial searches were run on December 3, 2018, covering 2001–December 2018. Final update searches were performed on June 5, 2019, covering December 2018–June 2019.

Community Preparedness

Search Parameters:

Date: 2001–Present

Language: English

Document Type: Exclude commentaries, editorials, letters, and notes

Databases: Medline (Ovid), Embase (Ovid), Scopus

Search Syntax:

Medline (Ovid):

Embase (Ovid):

Scopus:

TITLE-ABS-KEY(((natural W/1 (disaster* or hazard*)) or hurricane* or flood* or typhoon* or earthquake* or fire* or cyclon* or heatwave* or freezing or ((ice or snow or lightning) W/1 storm*) or blizzard* or “heat wave” or (extreme W/1 (temperature* or heat or cold)) or tsunami* or drought* or “tidal wave” or epidemic* or pandemic* or terrorism or bioterrorism or “mass casualt*” or (firesetting or arson or explosion? or bomb* or (explo* W/1 device*) or blackout* or tornado* or brownout* or ((power or equipment) W/1 (loss or failure)) or radioactive or radiation or (nuclear W/1 (disaster or meltdown or catastrophe or fail*)) or (refugee* or “mass migration*” or SARS or ebola or smallpox or plague or measles or riot* or influenza or “communicable disease*” or ((chemical or biological) W/1 warefare) or (civil W/1 (disorder* or defense or unrest)))) AND ((“public health” W/1 (practice or administration)) or “preventive medicine” or “health department” or ((public or local or state or tribal or territorial or multi) W/1 health W/1 (department* or agenc* or jurisdiction)))) OR ((emergency W/1 (preparedness or response or management)) or (disaster W/1 (plan* or preparedness or mitigation or recovery or cycle or medicine or resilience or readiness or ready))) AND (“vulnerable population*” or “minority group*” or {sexual and gender minorities} or “disabled person*” or immigrant* or emigrant* or transient* or refugee* or migrant* or “african ameri-

can*” or black or “hispanic American*” or hispanic or latin* OR indian* or “native american*” or “asian american*” or Asian* or poverty or “socioeconomic factor*” or (“at risk” w/1 (individual* or population*)) or “functional need*” or (vulnerable w/1 (individual* or population*)) or vulnerability or elderly or disabled or ((racial or ethnic or sexual or gender) w/1 minorit*) or “older adult” or pediatric* or child* or (limited w/1 (English or language)) or migrant* or immigrant* or “low income” or “under resourced” or tribal or “dialysis patient*” or “electricity dependent” or “medically vulnerable” or infant* or newborn* or {aged 80 and over} or “pregnant wom*n” or “pre existing condition*” or “chronic condition*” or diabetes or “heart disease*” or blind* or deaf* or “assistive device*” or “service animal*” or “personal assistance service provider*” or autism or dementia or anxiety or “transportation need*” or homeless* or “lift equipped” or “oxygen tank*” or “language barrier*” or ((low or limited) W/1 “health literacy”) or “mental health disorder*” or “group home patient*” or “substance abuse disorder*” or alcoholism or “substance disorder*” or “mental disorder*” or “stress disorder*” or PTSD or depression or “depressive disorder*” or schizophrenia or “psychotic disorder*”) AND (“community preparedness” OR “community partners” OR “community partnerships” OR “community engagement” OR “Community Health Planning” OR “Community Networks” OR “Community-Institutional Relations” or “cooperative behavior” or “risk assessment*” or “social capital” or “social support” or {community based participatory research} or (community w/2 (engagement or outreach or preparedness or partner* or partnership* or train* or toolkit or education or plan*)) or toolkit* or (“at risk” w/1 (database* or registr* or map*)) or “social capital” or CMIST OR “communication medical independence supervision transportation” or sovi or “social vulnerability index” or (“health measure*” w/1 data) or “empower initiative” or (community W/2 (stakeholder* or spokesperson* or spokespeople or gathering* or venue*)) or “town hall meeting*” or “faith based organization*” or “animal service agenc*” or “childcare organization*” or “chronic disease program*” or “communicable disease program*” or “community coalition*” or “emergency management agenc*” or “emergency medical service*” or “environmental health agenc*” or “fire department*” or {fire and rescue} or “health care coalition*” or “health care organization*” or “health care system*” or “health care provider*” or “infection control program*” or “housing authorit*” or “shelter* authorit*” or “human service provider*” or “immunization program*” or “jurisdictional strategic advisory council*” or “law enforcement” or “media organization*” or “local media” or “metal health provider*” or “behavioral health provider*” or “public health preparedness program*” or “school agenc*” or “education agenc*” or “social service*” or “state office of aging” or “surveillance program*” or “volunteer organization*” or “emergency shelter*” or NGO* or “red cross” or “local business*” or “home care” or “hospice care” or hospice* or “home hospice” or “assisted living facilit*” or “skilled care facilit*”)) AND PUBYEAR AFT 2000

Exclude: Note, editorials, letters, and notes

Results: 3,406

Emergency Operations Coordination

Search Parameters:

Date: 2001–Present

Language: English

Document Type: Exclude commentaries, editorials, letters, and notes

Databases: Medline (Ovid), Embase (Ovid), Scopus

Search Syntax:

Medline (Ovid):

Embase (Ovid):

Scopus:

TITLE-ABS-KEY(((natural W/1 (disaster* or hazard*)) or hurricane* or flood* or typhoon* or earthquake* or fire* or cyclon* or heatwave* or freezing or ((ice or snow or lightning) W/1 storm*) or blizzard* or “heat wave” or (extreme W/1 (temperature* or heat or cold)) or tsunami* or drought* or “tidal wave” or epidemic* or pandemic* or terrorism or bioterrorism or “mass casualt*” or (firesetting or arson or explosion? or bomb* or (explo* W/1 device*) or blackout* or tornado* or brownout* or ((power or equipment) W/1 (loss or failure)) or radioactive or radiation or (nuclear W/1 (disaster or meltdown or catastrophe or fail*)) or (refugee* or “mass migration*” or SARS or ebola or smallpox or plague or measles or riot* or influenza or “communicable disease*” or ((chemical or biological) W/1 warefare) or (civil W/1 (disorder* or defense or unrest)))) AND ((“public health” W/1 (practice or administration)) or “preventive medicine” or “health department” or ((public or local or state or tribal or territorial or multi) W/1 health W/1 (department* or agenc* or jurisdiction)))) OR ((emergency W/1 (preparedness or response or management)) or (disaster W/1 (plan* or preparedness or mitigation or recovery or cycle or medicine or resilience or readiness or ready))) AND PUBYEAR AFT 2000

Exclude: Editorials, letters, and notes

Results: 2,813

Information Sharing

Search Parameters:

Date: 2001–Present

Language: English

Document Type: Exclude commentaries, editorials, letters, and notes

Databases: Medline (Ovid), Embase (Ovid), Scopus

Search Syntax:

Medline (Ovid):

Embase (Ovid):

Scopus:

TITLE-ABS-KEY(((natural W/1 (disaster* or hazard*)) or hurricane* or flood* or typhoon* or earthquake* or fire* or cyclon* or heatwave* or freezing or ((ice or snow or lightning) W/1 storm*) or blizzard* or “heat wave” or (extreme W/1 (temperature* or heat or cold)) or tsunami* or drought* or “tidal wave” or epidemic* or pandemic* or terrorism or bioterrorism or “mass casualt*” or (firesetting or arson or explosion? or bomb* or (explo* W/1 device*) or blackout* or tornado* or brownout* or ((power or equipment) W/1 (loss or failure)) or radioactive or radiation or (nuclear W/1 (disaster or meltdown or catastrophe or fail*)) or (refugee* or “mass migration*” or SARS or ebola or smallpox or plague or measles or riot* or influenza or “communicable disease*” or ((chemical or biological) W/1 warefare) or (civil W/1 (disorder* or defense or unrest)))) AND ((“public health” W/1 (practice or administration)) or “preventive medicine” or “health department” or ((public or local or state or tribal or territorial or multi) W/1 health W/1 (department* or agenc* or jurisdiction)))) OR ((emergency W/1 (preparedness or response or management)) or (disaster W/1 (plan* or preparedness or mitigation or recovery or cycle or medicine or resilience or readiness or ready))) AND (physician* or doctor* or nurse* or responder* or pharmacist* or “health worker*” or “technical audience*” or “health department*” or “public health agenc*” or provider* or “homeless service provider*” or veterinarian* or “environmental health provider*” or “Hazardous Materials Response Teams” or (“hazardous material*” W/4 (responder* or team*)) or clinician* or coroner* or “medical examiner*” or ((federal or state* or local or tribal or territorial) W/1 government*) or “private sector” or “first responder*” or “Program for Monitoring Emerging Diseases” or “Top Officials Three Exercise”) AND (“health alert network*” or “public health alert*” or “public health messag*” or “emergency alert*” or (“joint information” W/1 (center* or system*)) or “communication system*” or warning* or notification* or “community health information exchange” or messag* or dissemination or ((crisis or risk) W/1 communication*) or “communication channel*” or email* or “text messag*” or “conference call*” or “provider access line*” or website* or “guidance document*” or “threshold criteria” or webinar* or webex or webcast* or “new technolog*” or “proprietary technolog*” or “bi directional information” or “data exchange*” or “communication plan*” or “communication protocol*” or (alert* W/1 (activation or trigger*)) or ((activation or trigger*) W/1 alert*) or telefacsimile)) AND PUBYEAR > 2000

Exclude: Editorials, letters, and notes

Results: 1,198

Non-Pharmaceutical Interventions: Quarantine

Search Parameters:

Date: 2001–Present

Language: English

Document Type: Exclude commentaries, editorials, letters, and notes

Databases: Medline (Ovid), Embase (Ovid), Scopus

Search Syntax:

Medline (Ovid):

Embase (Ovid):

Scopus:

TITLE-ABS-KEY(((natural W/1 (disaster* or hazard*)) or hurricane* or flood* or typhoon* or earthquake* or fire* or cyclon* or heatwave* or freezing or ((ice or snow or lightning) W/1 storm*) or blizzard* or “heat wave” or (extreme W/1 (temperature* or heat or cold)) or tsunami* or drought* or “tidal wave” or epidemic* or pandemic* or terrorism or bioterrorism or “mass casualt*” or (firesetting or arson or explosion? or bomb* or (explo* W/1 device*) or blackout* or tornado* or brownout* or ((power or equipment) W/1 (loss or failure)) or radioactive or radiation or (nuclear W/1 (disaster or meltdown or catastrophe or fail*)) or (refugee* or “mass migration*” or SARS or ebola or smallpox or plague or measles or riot* or influenza or “communicable disease*” or ((chemical or biological) W/1 warefare) or (civil

W/1 (disorder* or defense or unrest)))) AND ((“public health” W/1 (practice or administration)) or “preventive medicine” or “health department” or ((public or local or state or tribal or territorial or multi) W/1 health W/1 (department* or agenc* or jurisdiction)))) OR ((emergency W/1 (preparedness or response or management)) or (disaster W/1 (plan* or preparedness or mitigation or recovery or cycle or medicine or resilience or readiness or ready))) AND (“nonpharmaceutical intervention*” or “non-pharmaceutical intervention*” or quarantine* or “social distance*” or “self isolation*” or “voluntary quarantine*” or “involuntary quarantine*” or “home quarantine*” or “hospital quarantine*” or ((adherence or compliance) W/4 (quarantine* or “control measure*”)) or “restricted movement*” or (separation W/4 expos*))) AND PUBYEAR AFT 2000

Exclude: Editorials, letters, and notes

Results: 245

REFERENCES

ECDC (European Centre for Disease Prevention and Control). 2018. Best practice recommendations for conducting after-action reviews to enhance public health preparedness. Solna, Sweden. https://www.ecdc.europa.eu/sites/default/files/documents/public-health-preparedness-best-practice-recommendations.pdf (accessed March 26, 2020).

Higgins, J. P. T., J. Savovic, M. J. Page, J. Sterne, and ROB2 Development Group. 2019. Revised Cochrane risk-of-bias tool for randomized trials (ROB2). https://sites.google.com/site/riskofbiastool/welcome/rob-2-0-tool/current-version-of-rob-2 (accessed March 26, 2020).

Horney, J. A., E. G. Carbone, M. Lynch, Z. J. Wang, T. Jones, and D. A. Rose. 2017. How health department contextual factors affect public health preparedness (PHP) and perceptions of the 15 PHP Capabilities. American Journal of Public Health 107(S2):S153–S160.

Khan, Y., G. Fazli, B. Henry, E. de Villa, C. Tsamis, M. Grant, and B. Schwartz. 2015. The evidence base of primary research in public health emergency preparedness: A scoping review and stakeholder consultation. BMC Public Health 15(432).

Lewin, S., A. Booth, C. Glenton, H. Munthe-Kaas, A. Rashidian, M. Wainwright, M. A. Bohren, O. Tuncalp, C. J. Colvin, R. Garside, B. Carlsen, E. V. Langlois, and J. Noyes. 2018. Applying GRADE-CERQual to qualitative evidence synthesis findings: Introduction to the series. Implementation Science 13(Suppl 1):2.

Murthy, B. P., N. M. Molinari, T. T. LeBlanc, S. J. Vagi, and R. N. Avchen. 2017. Progress in public health emergency preparedness: United States, 2001–2016. American Journal of Public Health 107(S2):S180–S185.

Pope, C., S. Ziebland, and N. Mays. 2000. Qualitative research in health care: Analysing qualitative data. BMJ (Clinical Research) 320(7227):114–116.

Savoia, E., L. Lin, D. Bernard, N. Klein, L. P. James, and S. Guicciardi. 2017. Public health system research in public health emergency preparedness in the United States (2009–2015): Actionable knowledge base. American Journal of Public Health 107(S2):e1–e6.

Shekelle, P. 2004. The appropriateness method. Medical Decision Making 24(2):228–231.

Siegfried, A. L., E. G. Carbone, M. B. Meit, M. J. Kennedy, H. Yusuf, and E. B. Kahn. 2017. Identifying and prioritizing information needs and research priorities of public health emergency preparedness and response practitioners. Disaster Medicine and Public Health Preparedness 11(5):552–561.

Sloss, E. M., D. H. Solomon, P. G. Shekelle, R. T. Young, D. Saliba, C. H. MacLean, L. Z. Rubenstein, J. F. Schnelle, C. J. Kamberg, and N. S. Wenger. 2000. Selecting target conditions for quality of care improvement in vulnerable older adults. Journal of the American Geriatrics Society 48(4):363–369.

Sterne, J. A., M. A. Hernan, B. C. Reeves, J. Savovic, N. D. Berkman, M. Viswanathan, D. Henry, D. G. Altman, M. T. Ansari, I. Boutron, J. R. Carpenter, A. W. Chan, R. Churchill, J. J. Deeks, A. Hrobjartsson, J. Kirkham, P. Juni, Y. K. Loke, T. D. Pigott, C. R. Ramsay, D. Regidor, H. R. Rothstein, L. Sandhu, P. L. Santaguida, H. J. Schunemann, B. Shea, I. Shrier, P. Tugwell, L. Turner, J. C. Valentine, H. Waddington, E. Waters, G. A. Wells, P. F. Whiting, and J. P. Higgins. 2016. ROBINS-I: A tool for assessing risk of bias in non-randomised studies of interventions. BMJ 355:i4919.

USPSTF (U.S. Preventive Services Task Force). 2015. U.S. Preventive Services Task Force procedure manual. https://www.uspreventiveservicestaskforce.org/Page/Name/procedure-manual (accessed March 4, 2020).

Yeager, V. A., N. Menachemi, L. C. McCormick, and P. M. Ginter. 2010. The nature of the public health emergency preparedness literature 2000–2008: A quantitative analysis. Journal of Public Health Management and Practice 16(5):441–449.

Zaza, S., R. S. Lawrence, C. S. Mahan, M. Fullilove, D. Fleming, G. J. Isham, and M. Pappaioanou. 2000. Scope and organization of the Guide to Community Preventive Services: The task force on community preventive services. American Journal of Preventive Medicine 18(1 Suppl):27–34.