Engaging the Public in Critical Disaster Planning and Decision Making: Workshop Summary (2013)

Below is the uncorrected machine-read text of this chapter, intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text of each book. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.

Engaging the Public in Critical Disaster Planning and Decision Making INTRODUCTION1 Building on the recommendations and guidance in the 2012 Institute of Medicine (IOM) report Crisis Standards of Care: A Systems Framework for Catastrophic Disaster Response, the IOM Forum on Medical and Public Health Preparedness for Catastrophic Events sponsored an interactive workshop session at the National Association of County and City Health Officials (NACCHO) Public Health Prep- aredness Summit, held March 12-15, 2013, in Atlanta, Georgia. The goals of the session were to provide practitioners with practical guidance and encourage participants to strategize their efforts and leverage work already being done around the country. Specifically, presentation and discussion objectives were to2  Introduce the key principles of public engagement  Provide practical guidance on how to plan and implement a public engagement activity  Provide attendees with sample tools to facilitate planning 1 The role of the ad hoc planning committee of the Institute of Medicine (IOM) Forum on Medical and Public Health Preparedness for Catastrophic Events was limited to developing this session (i.e., workshop) for the National Association of County and City Health Officials Summit. This summary has been prepared by the rapporteurs as a factual overview of the presentations and discussions at the session. Statements, recommend- ations, and opinions expressed are those of individual presenters and participants, and are not necessarily endorsed or verified by the IOM or the Forum, and they should not be construed as reflecting any group consensus. 2 The complete statement of task can be found in Appendix B. 1 

2 ENGAGING THE PUBLIC IN CRITICAL DISASTER PLANNING  Introduce and simulate different methods of engagement exercises The workshop examined theories and practices of public engagement, explored challenges and lessons learned, and included sample public engagement exercises. The workshop was organized in two parts. First, invited panelists provided background on crisis stand- ards of care (CSC) and public engagement and discussed specific examples of recent public engagement exercises and lessons learned in their communities. In the second part, attendees participated in two simulated, interactive public engagement exercises using two different methods. These abbreviated exercises were intended to give attendees a better sense of the engagement process and a sampling of tools available to them for working with their own communities. Public engagement is a useful approach for obtaining public input about pending policy decisions that require difficult choices among competing values (IOM, 2012a). Although average citizens may lack the expertise to comment on technical issues (e.g., the use of Sequential Organ Failure Assessment scores to allocate ventilators), they are very capable of deliberating on the values underlying public policy decisions (e.g., whether to withhold or withdraw life-preserving care, such as ventilators, in crisis situations where resources become scarce). Some of the benefits of public engagement are that it can help inform members of the community, include their input in disaster planning to increase legitimacy and acceptance, and reveal public misunderstandings, biases, and areas of deep disagreement. Policy makers can then work to address these matters during the development of disaster plans and during the plan dissemination phase by having community members at the table in the beginning stages of the process. About This Summary The report that follows summarizes the presentations by the expert panelists and the open panel discussions that took place during the workshop. Beginning by framing and defining public engagement outreach during the planning stages, it moves to describing the specific case studies speakers presented during the workshop. These include Seattle and King County, Washington; Harris County, Texas; the State of Michigan; and an example from the IOM and the Centers for Disease 

WORKSHOP SUMMARY 3 Control and Prevention (CDC). Following these case studies, challenges and lessons learned are discussed, and finally a description of the two simulated public engagement exercises performed during the workshop: a community conversation simulation and a Q-sort activity based on a severe influenza pandemic scenario. A list of references, the statement of task, the workshop agenda, and biographical sketches of the panelists are available in the appendixes. FRAMING PUBLIC ENGAGEMENT Unexpected incidents occur every day, and generally people find a way to work around them and continue on, said Dr. Umair Shah, deputy director, Harris County Public Health and Environmental Services (HCPHES), and director of the HCPHES Division of Disease Control and Clinical Prevention. When such incidents are life-threatening or cause significant harm, they are referred to as emergencies. When emergencies completely overwhelm our abilities to cope or respond, they are called disasters. Disaster incidents are categorized as catastrophic or pervasive and can be manmade or natural in occurrence, Shah explained. Catastrophic incidents occur suddenly, with little or no notice (e.g., intentional acts of violence, mass casualty incidents, earthquakes, tsunamis, hurricanes). In contrast, Shah continued, pervasive incidents can have a slower onset and often progress gradually (e.g., floods, wildfires, pandemic influenza). Defining Public Engagement The process of disaster preparedness, response, and recovery is a multifaceted partnership, he said, and communities are key partners in making sure that a disaster response is successful. Citing the 2009 IOM report, Shah said that government at all levels should “partner with and work to ensure strong public engagement of community and provider stakeholders, with particular attention given to the needs of vulnerable populations including those with special needs,” who often are left out of the decision-making process. To help facilitate public involvement, the 2012 IOM Crisis Standards of Care report defines the essential principles of public engagement and includes a user-friendly toolkit for use by state and local agencies. 

4 ENGAGING THE PUBLIC IN CRITICAL DISASTER PLANNING The overall goal of public engagement is to ensure that there is public input about policy decisions that may require difficult choices among competing values (IOM, 2012a). This involves two-way com- munication: both informing community members of sensitive policy decisions and receiving community input on difficult matters. Shah highlighted five essential tenets of public engagement discussed in the IOM CSC report (2012a): 1. Policy makers may seek public engagement for a variety of reasons. 2. Adequate support and resources are needed to allow for a high- quality process. 3. Participants should represent the diversity of the community, especially underrepresented populations. 4. The process should offer participants a meaningful opportunity for deliberation (but not necessarily consensus). 5. Policy makers should ensure transparency around how com- munity input will be used in policy development and share final policy decisions. Many of the benefits of public engagement are obvious, Shah said. In the short term, public engagement provides greater visibility and public awareness about the need for local disaster preparedness plans and initiatives and the importance of community and individual preparedness in general. In the long term, policies that reflect community values and priorities will be met with greater public acceptance and adherence, should they ever need to be implemented. One of the main challenges to public engagement is ensuring the credibility of the process, for example, convening participants who reflect the diversity of the community and facilitating meaningful conversations. There are also challenges in applying the outputs of community engagement to policy making, for example, collecting actionable data and managing expectations around how those data will be used. Initiating and sustaining the public engagement process amid ever- increasing competition for resources is also a challenge. 

WORKSHOP SUMMARY 5 Expanding Practical Experience Shah provided a brief overview of the toolkit from the 2012 IOM Crisis Standards of Care report and referred participants to the full report for further details. The development of the toolkit by the IOM committee was informed by earlier practical experiences of Seattle and King County, Washington; Harris County, Texas; the State of Minnesota; and two pilot programs in Boston and Lawrence, Massachusetts. The result is a set of resources for “community conversations” that state and local jurisdictions can tailor and adapt to their specific needs. The toolkit includes a sample agenda, content slides, facilitator scripts and strategies, surveys, scenarios (e.g., a major earthquake), data collection templates, and other resources. These can be used to engage the lay public in discussion about the values that should underlie the allocation of scarce medical resources in a disaster. The toolkit is designed for agencies to address a list of important questions; Shah continued—for example, how to engage community partners, how to reflect a community’s diversity, and how to make the materials understandable for the intended audience. What is the appropriate length of a public engagement meeting and what skills and backgrounds should facilitators have? What should be done with the data, and is this research that should be reviewed by an institutional review board (IRB)? Further, the toolkit is designed to provide a framework beyond simply CSC engagement activities. For example, it also was used during a CDC-sponsored IOM workshop that examined the public perception of alternative strategies for facilitating antiviral medication during pandemic flu (IOM, 2012b). THEORY TO PRACTICE: PLANNING AND IMPLEMENTING PUBLIC ENGAGEMENT ACTIVITIES Sharing lessons learned across communities provides valuable information as more states, regions, and counties take on the responsibility of engaging the public in disaster preparedness planning. Learning from the successes and challenges of others can help to shorten the timeframe of a project, make the most of limited resources, and ensure that activities are effective in reaching their set goals. To assist jurisdictions in planning public engagement activities, presenters offered four examples of recent public engagement activities they had been a 

6 ENGAGING THE PUBLIC IN CRITICAL DISASTER PLANNING part of at the local, state, and national levels. Seattle and King County, Washington, and Harris County, Texas, used different approaches for operating similar public meetings on allocation; Michigan leveraged the Internet for outreach on ethical guidelines; and the CDC partnered with the IOM to execute targeted meetings in geographically diverse locations around the country. Seattle and King County, Washington Meredith Li-Vollmer, risk communication specialist for Public Health–Seattle and King County, Washington, pointed out that public health departments already use different forms of community engage- ment in their day-to-day work. There are various forms of engagement with the public, spanning the spectrum from educational outreach to consultative information-gathering activities to community-led initiatives (see Figure 1). For the purposes of engaging the public in disaster planning, Li-Vollmer focused on a deliberative meeting model of public eng- agement. This is a consultation process in which public health officials gather information and exchange ideas and opinions on policy options with the public. This model of engagement is most effective when used to gather community input that will shape the outcome of a specific policy question, Li-Vollmer said, and should not simply be a presentation of a predetermined policy for which officials are seeking buy-in. She noted that holding a deliberative meeting in which policy options are weighed sets the expectation that input provided by the community will be considered in policy development. As noted by Shah earlier, trans- parency about this process is essential to promote trust. Medical Service Prioritization During an Influenza Pandemic Li-Vollmer shared the experiences of Seattle and King County Public Health from engaging the public on CSC for the allocation of scarce medical resources during an influenza pandemic. The meetings were funded using CDC grants to address pandemic influenza. She stressed the importance of having a clear understanding from the beginning of exactly what input is being sought from the community. For example, CSC can encompass many issues, so a steering committee of 

WORKSHOP SUMMARY 7 FIGURE 1 Spectrum of public engagement. SOURCE: Li-Vollmer presentation, adapted from R. Bernier, CDC Immun- ization Program. stakeholders was convened to help narrow down the topics and define key questions for the public engagement session. In this particular public engagement session, the questions posed to the community were  Should we change how medical treatment decisions are made during a severe influenza pandemic?  What should be the goals when decisions are made about medical treatment during a severe influenza pandemic?  How should decisions be made about the rationing of limited, lifesaving resources (e.g., intensive care unit beds, ventilators)?  Should these decisions about medical treatment be consistent across hospitals in the region, across the state, or across the nation? Recruitment Four community engagement meetings were held at four distinct sites in different parts of King County. Participants were selected for broad diversity based on gender, race, ethnicity, age, income, and education (Seattle & King County Public Health, 2009). Participants across all four meetings included 30 stakeholders, 57 residents of North King County, 49 residents of South King County, and 17 Spanish- speaking residents (each distinct group was part of a separate meeting). 

8 ENGAGING THE PUBLIC IN CRITICAL DISASTER PLANNING Li-Vollmer explained that the Spanish-speaking residents attended a meeting conducted only in Spanish, with relevant Spanish language materials that were transcribed and then translated back to English for ease of understanding by public health staff. For public recruitment, the public health department used input from the steering committee and the Vulnerable Populations Actions Team (VPAT) to recruit residents-at- large (“public participants”). VPAT identified different local org- anizations that served as project partners to recruit participants, assist with the development of culturally appropriate materials and methods, and review any translated materials. Additional recruitment was conducted through schools, volunteer emergency response organizations, parent groups and associations, faith-based organizations, housing organizations, student organizations, community centers, libraries, online through Craigslist, and other public places near the meeting sites. Participants were compensated for their time in the amount of $100 (Seattle & King County Public Health, 2009). Similarly, for key stakeholders, participants were recruited from a wide range of community organizations and agencies, including  hospitals, health care providers, and administration staff, including community health clinics, home health care agencies, and nursing and adult living homes;  emergency response and management;  businesses;  faith-based organizations;  social service and advocacy organizations serving diverse populations, including immigrant and refugee service providers, sensory and physical disability providers, and housing service providers; and  schools (Seattle & King County Public Health, 2009). Meetings Li-Vollmer explained that the meetings were generally between 4 and 8 hours in length, which allowed ample time to gather in-depth input. A variety of methods were used to gauge community opinion on the issues. Pre- and post-session surveys were administered to collect information on public attitudes. Participants were split into small groups to discuss decision making in hypothetical situations. Q-sort opinion ranking exercises were also very successful at engaging people, she said 

WORKSHOP SUMMARY 9 (the specifics of Q-sort methodology are described in Box 1). Two interactive public engagement exercises, including a Q-sort activity, were demonstrated at the workshop by Shah and Li-Vollmer and are described in further detail in the final section on exercises. BOX 1 Q-Sort Methodology Preparation In Q method participants are asked to sort a set of statements represent- ing a broad diversity of opinions and perspectives on the phenomenon being investigated. Items for the Q set can be gathered from a variety of sources; for example, direct quotes and themes from interviews with participants . . . and statements originating from academic literature and popular media in addition to interviews. . . . A complete set of scale items (from previous research) can be used to create a ready-made Q set. A set of between 40 and 80 statements is considered satisfactory. Be- tween 40 and 60 participants are recommended, but effective studies with far fewer participants have been carried out Pilot studies require a small number, perhaps selected strategically to include participants who can provide a wide range of viewpoints, helpful comments, and additional statements from a varie- ty of perspectives. In preparation for the sorting task, each item is numbered and written on a separate card. Sorting Participants sort the cards according to the instructions given by the re- searcher. For example, an instruction could be to sort the cards initially into three piles according to whether the person “agrees,” “disagrees,” or “neither agrees, nor disagrees (neutral)” with the statement. Participants continue to sort the cards within each broad pile, according to the number of possible positions in the sorting template. For example, working with the “agree” pile, participants select the two items they agree with most (+6 column in the template), then the three items with a slightly lower degree of agreement (+5), and proceed until all the items in the agree pile have been allocated. The process is repeated with the “disagree” pile and continues with the participant distributing the cards in the neutral pile into the remaining positions until all cards have been sorted. Participants then write all of the statement numbers in the appropriate boxes in the template provided. In a post-sorting interview, each participant is asked to comment on the statements, to suggest additional items that might be includ- ed, and to point out items that are not clear, and so on. Such open-ended ques- tions aid the interpretations of the sorting configuration. Processing Each completed template is entered as data. A general statistical package such as SPSS or a dedicated Q package can be used.a The program corre- 

10 ENGAGING THE PUBLIC IN CRITICAL DISASTER PLANNING lates each Q sort (i.e., a completed template) with each other Q sort to identify a small number of factors that can represent shared forms of understandings among participants. Various techniques of factor rotation and statistical proce- dures are used to safeguard factor reliability.b The Q sorts of all participants who loaded significantly on a factor are merged to produce a single configura- tion, which serves as a factor array, or factor exemplar. A table of all factors and the ranking assigned to each statement in each factor is constructed to serve as a basis for factor interpretation. _____________________________ a PCQ . . . or freeware PQMethod-2.11d, available at http://schmolck.userweb.mwn. de/qmethod (accessed September 20, 2013). b These can be performed automatically by the program . . . For a factor to be in- terpretable, one requirement is an eigenvalue greater than 1.0 (an eigenvalue is the sum of squared loadings for a factor; it conceptually represents the amount of variance accounted for by a factor). A second requirement is that a factor must have at least two sorts that load significantly on it alone. SOURCE: Shinebourne, 2009. Public engagement meetings were held with both vested stakeholders and diverse members of the community. Li-Vollmer clarified that, because everyone is really a stakeholder in a crisis situation, the term “vested stakeholders” was used to describe those who were already involved in planning and response in some fashion. For example, vested stakeholders in a public engagement exercise on the allocation of scarce medical resources would include health care providers, health care administrators, emergency management, large employers, schools, and advocates from community- and faith-based organizations. For community meetings, she said, it is important to try to engage people across the spectrum of socioeconomics, education, race, ethnicity, and age who are representative of those living in the community. In some cases, interpreters may be needed, and for the vested stakeholder group, an American Sign Language Interpreter was present for that meeting. Li- Vollmer reported that lay members of the community were engaged, active, and respectful during the sessions, and were very capable of deliberating on the complex health and ethical issues under discussion. Keys to Success Key to success, Li-Vollmer said, were the partnerships with the community-based organizations and human service providers on the steering committee. Input from these community partners played an 

WORKSHOP SUMMARY 11 important role in project design. For example, one proposed question for the session was “should the federal government be ethically obligated to develop a data collection infrastructure to help with decision making about medical triage?” However, steering committee members from community-based organizations felt that this question would not engage people. Instead, they urged more focus on issues of access, and the question used for the public session was “what concerns do you have about equitable access to lifesaving medical resources in an emergency?” Partners helped to identify barriers to participation (e.g., transportation to the site), advised on meeting structure (e.g., people would more likely attend a 1-day, 7- to 8-hour meeting than shorter sessions over 2 days), and reviewed the meeting materials for cultural relevance, literacy level, and quality of translations. Community partners were also crucial to recruiting participants for the session. With regard to resources, Li-Vollmer explained that the CDC grant funding was used to cover staff time (and overtime), facility rentals, meals, incentives for participation, translators, and bilingual facilitators. The greatest expenditure, she said, was staff time. A full-time project manager was hired because, in addition to all of the planning and setup, Q-sort is a particularly labor-intensive method involving numerous interviews and extensive data collection. Seattle and King County was among the first to perform this type of public engagement, and had to learn and develop processes and materials along the way. But Li-Vollmer advised participants that there are now many resources available, and she and others around the country who have done such activities are willing to share their materials. For useful resources, she referred participants to the public engagement toolkit from the Seattle and King County Advanced Practice Center in association with the NACCHO Advanced Practice Center,3 as well as to the IOM Crisis Standards of Care public engagement toolkit (2012a). Results of Seattle and King County Public Engagement Q-Sort Activity Li-Vollmer shared some of the findings from the Seattle and King County public engagement on scarce medical resource allocation.4 The most striking finding, she said, was how similarly people felt. She had 3 See http://www.apctoolkits.com/documents/vp/Public_Engagement_Project_Final_ Report_for_web.pdf (accessed April 26, 2013). 4 Full results are available in the final report. See http://www.apctoolkits.com/documents /vp/Public_Engagement_Project_Final_Report_for_web.pdf (accessed April 26, 2013). 

12 ENGAGING THE PUBLIC IN CRITICAL DISASTER PLANNING expected that there would be strong differences of opinion between vested stakeholders and the general public, but that was not the case. The majority of participants felt strongly that saving the greatest number of people was by far the most important consideration, even if it meant that some people would not survive. At the same time, factor analysis of the data showed that participants fell into two main groups, one group for which maintaining response capabilities was most important, and another for which maximizing resources was the primary concern. The response capabilities group favored prioritizing treatment for health care workers and first responders because of the need for them to return to their jobs in the midst of a pandemic. This response group was made up primarily of community members. The group ranked survivability as slightly less important, although still important, and also gave more priority to pregnant women and children. For the maximizing resources group, survivability was the highest- ranked criterion for receiving treatment. This group tended to be made up of the vested stakeholders (e.g., health care providers, hospital administrators, emergency managers). Interestingly, health care workers argued that they would not get well fast enough to be useful, so treating them first would not really be the best use of resources. Harris County, Texas As another example of a local public engagement project, Shah described the mass care/mass fatality planning activity undertaken by HCPHES in 2011. Harris County, Texas, is the third most populous county in the nation, with 4.1 million residents spread over 1,700 square miles (an area roughly equivalent to the size of Rhode Island). The daytime population (i.e., including workers and visitors) is estimated to be 5 to 6 million. From May to July 2011, HCPHES sponsored nine meetings (eight citizen sessions and one stakeholder session) to obtain input on a draft plan for prioritizing vaccines, antivirals, and ventilators in the event of an influenza pandemic, and a 2.5-day partner workshop to review current plans and identify strategies for overcoming any gaps identified. Key to completing this process in just more than 2 months, Shah said, was leveraging internal and external resources and strengths. Internally, there was an already-established Houston–Harris County Committee on Medical Standards of Care for Pandemic Influenza and 

WORKSHOP SUMMARY 13 Highly Infectious Respiratory Diseases. In addition, Shah said, they made use of the 2009 report from the Seattle and King County public engagement project on Medical Service Prioritization During an Influenza Pandemic (discussed by Li-Vollmer above), adapting it to the needs of Harris County. Pandemic Influenza Public Engagement Project The basic meeting process included a pre-session survey, an overview and background presentation on pandemics, a question-and- answer period, presentation of scenarios (for vaccines, antivirals, and ventilators) followed by small group discussions (including Q-sort activities), reports from the small groups and discussion of results, electronic polling (using wireless handheld response pads or “clickers,”5 which instantly tabulate participant votes and can display them on a computer or projection screen for all to see), and a post-session survey. A logic model was developed to evaluate the public engagement process, and the University of Nebraska Public Policy Center was contracted to address five key evaluation questions: 1. How successful was the process in attracting citizens and stakeholders to participate? 2. How successful was the process in attracting citizens with diverse perspectives? 3. Was the process successful in providing sufficient knowledge for informed decisions? 4. How did the process affect citizen perceptions about vaccine goals and values? 5. Did the process result in a balanced, honest, and reasoned discussion of the issues? Methods of data collection included, for example, comparison of public session participant demographics with community characteristics, pre- and post-surveys to assess changes in knowledge and opinions, and post-process surveys and participant focus groups to assess the quality of the process. Shah shared some of the data from the process evaluation. 5 Used response technology from http://www.turningtechnologies.com (accessed Sep- tember 20, 2013). 

14 ENGAGING THE PUBLIC IN CRITICAL DISASTER PLANNING Recruitment To recruit members of the community for the eight public engagement sessions, HCPHES used a variety of methods, including mainstream media, social media, flyers,6 and a video7 on its website. Breakfast and lunch were provided during the sessions, and a $75 stipend was offered. The goal was to attract 50 participants to each of the full- day meetings, or 400 participants in total. In fact, there were more than 50 participants in each session, and several sessions had more than 100 participants. Shah noted that the majority of community members learned of the meetings through word of mouth, whereas most stakeholders got their information from e-mails (see Table 1). The HCPHES project was generally successful in attracting participants from diverse communities, backgrounds, and perspectives, Shah said. There is no majority community in Harris County. The two largest groups are Hispanic (39 percent) and Caucasian (35 percent); however, the majority of community participants were African American (68 percent) (see Table 2). Shah also pointed out that the larger-than- expected representation by lower-income groups (relative to census data) was important, as this is a key demographic to include in the engagement process (see Table 3). TABLE 1 Methods for Learning About the Meeting Percent of Participants Citizens Stakeholders Medium (n=584) (n=29) Newspaper 1.7 0 Internet (Not E-mail) 5.5 0 Television 1.0 0 Radio 0.5 0 E-mail 15.2 44.8a Flyer 10.8 0 Community or Religious 11.0 27.6a Organization 6 See http://www.hcphes.org/PublicEngagementPandemic/PDF/HCPHES-Public- Engagement-Flyer-Eng.pdf (accessed April 26, 2013). 7 See http://www.hcphes.org/PublicEngagementPandemic/Video%20Invitations%20to%20 Participate.htm (accessed April 26, 2013). 

WORKSHOP SUMMARY 15 Percent of Participants Citizens Stakeholders Medium (n=584) (n=29) Professional Colleague or 9.1 41.4a Professional Organization Government Agency 5.8 24.1a Facebook 1.5 0 Word of Mouth (Friend/Family 51.5 3.4a Member) Other 6.2 3.4 a Numbers with asterisks show statistical significance between citizen and stakeholder reporting for that category SOURCE: Shah presentation, slide 8 (March 14, 2013). TABLE 2 Race/Ethnicity for Harris County Adults, Citizens, and Stakeholders Citizen Stakeholder Census Data Sample Sample Hispanic 39.8% 11.3% 12.0% (n=63) (n=3) White/Caucasian 35.1% 11.7% 48.0% (n=65) (n=12) Black/African American 17.9% 68.2% 24.0% (n=379) (n=6) Asian 5.7% 5.0% 16.0% (n=28) (n=4) Native Hawaiian/Other 0.1% 0.4% 0% Pacific Islander (n=2) (n=0) Native American/Alaskan 0.2% 0.7% 0% Native (n=4) (n=0) Other 1.2% 2.7% 0% (n=15) (n=0) No Answer (n=48) (n=5) SOURCE: Shah presentation, slide 9 (March 14, 2013). 

16 ENGAGING THE PUBLIC IN CRITICAL DISASTER PLANNING TABLE 3 Income for Harris County Adults, Citizens, and Stakeholders Citizen Stakeholder Census Data Sample Sample Less than $15,000 12.8% 31.5% 3.8% (n=162) (n=1) $15,000-$34,999 22.3% 27.0% 7.7% (n=139) (n=2) $35,000-$49,999 14.3% 17.7% 19.2% (n=91) (n=5) $50,000-$74,999 17.6% 14.2% 34.6% (n=73) (n=9) $75,000-$99,999 11.3% 5.0% 19.2% (n=26) (n=5) $100,000 or more 21.6% 4.7% 15.4% (n=24) (n=4) No Answer (n=89) (n=4) SOURCE: Shah presentation, slide 9 (March 14, 2013). Meetings The post-session evaluation revealed that the process was also successful in improving the knowledge of participants so that they could engage in informed discussions about national vaccine policy and CSC, and citizens changed their perspectives and opinions as a result of the deliberative process. In addition, the public engagement process was perceived to be of high quality by citizens and stakeholders, and participants were pleased that they had a voice in the policy-making process. The 2.5-day partner workshop in July 2011 brought together representatives from a diverse array of sectors, including public health, fire departments, emergency medical services (EMS), law enforcement, mortuary/forensics, emergency management, medical societies, hos- pitals, primary care, ambulatory surgical care, home health care, long- term care, pharmacies, laboratories, nursing, veterinarians, businesses, faith-based organizations, and others from the community. Within-sector and cross-sector small group breakout discussions were convened to 

WORKSHOP SUMMARY 17 consider different planning scenarios, identify gaps, and develop a steering committee for mass care/mass fatality planning for Harris County. Shah referred workshop participants to the HCPHES website for further information and resources, including the partner workshop summary and the evaluation tool for the public engagement project.8 This activity met its goals of recruiting participants from diverse communities and providing participants with sufficient knowledge for informed decisions that affect citizen perceptions about vaccine goals and values. This activity also resulted in a balanced, honest, and reasoned discussion of the issue among participants. State of Michigan Linda Scott, manager of the Healthcare Preparedness Program at the Michigan Department of Community Health, described the State of Michigan’s experience in developing ethical guidelines and other support materials for the allocation of scarce medical resources during emergencies. The collaborative process has spanned 5 years thus far, from the initial research, which began in 2008 (before the release of the 2009 IOM letter report), to the public engagement and education efforts taking place in 2013. The first phase involved research into what was already being done elsewhere, particularly in Minnesota. One of the early lessons learned from others was that people are very sensitive about the government making decisions relating to their medical care. Project leaders then decided to contract with Lance Gable, a professor at Wayne State University with a strong background in public health law, to lead the initiative. The next challenge was to establish a statewide ethics advisory committee that was small enough to be functional and effective, and large enough to represent many different disciplines. The advisory committee had representation from the state EMS coordinating committee, professional organizations (e.g., Michigan State Medical Society, Michigan Osteopathic Association), universities, physicians (including pediatricians), health care coalitions (e.g., Michigan Health and Hospital Association), and the Michigan Department of Health, as well as experts in preparedness, an IRB coordinator, and a public health legal advisor. 8 See http://www.hcphes.org/PandemicMassCare.htm (accessed April 30, 2013). 

18 ENGAGING THE PUBLIC IN CRITICAL DISASTER PLANNING Ethical Guidelines for Allocation of Scarce Medical Resources Scott said that after assembling the advisory committee it took about 1 year to develop the framework for the guidelines and define the most relevant areas to address. There were consultations with subject matter experts to inform development of the scope and content so that the guidelines would be broad and flexible enough to support numerous situations. Scott pointed out that, although other guidelines focus on pandemic influenza or other specific issues, the Michigan guidelines were designed to cover all hazards. Once the framework was established, numerous meetings and discussions were held to draft the core ethical guideline. Scott added that around this time, the 2009 IOM report was released, which provided validation for the preparedness activities going on in Michigan. The Ethics Advisory Committee finalized the guidelines in 2010 and began outreach to other relevant stakeholders for additional feedback. The draft was provided to select stakeholders as a hard copy, Scott said, to help prevent the uncontrolled dissemination that can occur with electronic versions. Reviewers were given a survey and other documents that could be used to provide feedback, and some of the large health care facilities met with Gable for discussions. After systematic revisions and improvements to the draft guidelines, a broader public engagement strategy was begun. Outreach was made to all Michigan hospitals, all health care coalitions, state-level medical and health care associations (e.g., pediatrics, gerontology, respiratory therapy), and local and state public health offices. There was also focused outreach to other advocacy organizations, such as those for the hearing- and vision- impaired, and community rights advocacy organizations, such as the American Civil Liberties Union. The initial outreach was done in the form of a letter that provided a Web link to the draft guideline document. The letter was mailed in August in preparation for a mid-September meeting, but Scott said that there was not a good response, and the first public engagement session had to be delayed until November. The focus now is on completion of ancillary materials to supplement the guidelines and planning for widespread dissemination of the guidelines and materials. A public website was also created to obtain input on the guidelines document and the annexes.9 The website provides a survey for individual 9 See http://mimedicalethics.org/default.aspx (accessed April 26, 2013). 

WORKSHOP SUMMARY 19 feedback and a blog by Gable. The website was launched on February 11, 2012, and as of March 2013, Scott said there had been about 3,200 visits to the site, or less than 10 per day. The Department of Community Health has issued press releases, sent messages via Twitter, and modified key words to be more recognizable in search engines, but the website is still not getting the level of feedback that was hoped for, Scott said. Still, reaching several people per day is better than not having the website at all, and they will continue to look for ways to improve their use of technology to engage the public. Providing Alternative Strategies for Increasing Access to Antiviral Medications and Treatment Advice In February and March 2012, at the request of the CDC, the IOM Forum on Medical and Public Health Preparedness for Catastrophic Events designed and convened a series of workshops that explored the public’s view of strategies for access to antiviral medication and treatment during an influenza pandemic (IOM, 2012b). Shah explained that community meetings were held in geographically and demo- graphically diverse locations across the United States, including Fort Benton, Montana (a frontier rural community); Chattanooga, Tennessee (a midsize urban community); and Los Angeles, California (a large metropolis). Sessions lasted 4 hours, and 232 community members participated across the 3 sites. Community participants were asked to consider several proposed strategies to provide quicker access to antiviral drugs, including nurse triage lines,10 collaborative practice agreements, and a text message system. Presentations, audience response surveys, scenarios, and discussions were designed to elicit participants’ thoughts on the advantages and disadvantages, ethics, and acceptability of the proposals for use in an emergency. Recruitment Shah highlighted the broad diversity of the communities sampled. For example, participants in Fort Benton, Montana, were generally older, 10 Per the report, nurse triage lines (NTLs) would be “phone hotlines that people with flu-like symptoms could call for advice, discuss whether an antiviral medication is indi- cated and, if so, possibly receive a prescription that the NTL would transmit to callers’ local pharmacies for dispensing (if feasible)” (IOM, 2012b, p. 4). 

20 ENGAGING THE PUBLIC IN CRITICAL DISASTER PLANNING 100 percent were white (7 percent were from the Hutterite community), and 84 percent had some college education. Those from Los Angeles were generally middle-aged, more than 90 percent were African American or Hispanic, and 55 percent had some college education. Participants in Chattanooga were young-adult to middle-aged, 41 percent were African American and 4 percent were Hispanic, and around 80 percent had some college education. Interestingly, the closest pharmacy for nearly one-third of Fort Benton participants was more than 25 miles from their home, while almost all participants in Los Angeles (98 percent) and Chattanooga (92 percent) had a pharmacy within 5 miles. Meetings A set of program materials was developed for the 4-hour sessions at all three venues. These materials provided participants with information needed for the discussion. The workshops were structured as follows (see Box 2). BOX 2 Major Structural Elements of Each Public Engagement Workshop Introductory Exercise: Small-group table discussions to break the ice, estab- lish ground rules, and introduce participants to the facilitated discussion and report-out method. All participants were invited to state briefly why they decid- ed to attend the community conversation and to identify their greatest hope and greatest fear should a flu pandemic strike. Audience Response System (ARS) Pre-Survey: A series of questions about participant demographics and other relevant characteristics, followed by a se- ries of opinion statements related to the proposed alternative strategies for delivering antivirals during a severe pandemic. Participants used electronic response devices (handheld keypads) to respond to questions and statements that were both projected as slides and read aloud to accommodate partici- pants with low vision or literacy. The main purpose of the pre-survey opinion statements was to quickly immerse participants in the topic and to focus their attention on the issues that would be considered during the session. Expert Presentation: A slide presentation providing background on pandemic influenza, antiviral medications, anticipated challenges in delivery of antivirals during a pandemic, and proposed alternative strategies for discussion during the session. The information was targeted to general public audiences. The issues were framed as follows: 

WORKSHOP SUMMARY 21  In a severe pandemic, people seeking antiviral drugs could meet many roadblocks: o Many people sick at the same time. o Crowded hospitals, medical offices, and clinics. o Long waits to see a health care provider for a prescription. o Sick, contagious people occupying waiting rooms and other public spaces. o People not sure where to turn for information.  In response, possible new strategies are now being developed o to provide quick, safe, fair access to antiviral drugs (nurse triage lines, pick-up and delivery by community contacts, pharmacist pre- scribing). o to offer advice to sick people and the worried well (pandemic flu website, text messaging tool).  We want to hear your views on these potential strategies, and other ideas you might have. Small-Group Scenario Discussions: Two 30-minute scenario discussions, led by table facilitators, were at the core of the agenda. Participants were asked to consider specific questions about the acceptability and feasibility of possible alternate strategies for delivering medications under two sets of circumstanc- es, described in the scenarios as  a severe flu pandemic that has led to long delays to see health care providers who can write prescriptions for antiviral medications.  a point in time during the pandemic when there are no longer enough antiviral medications for everyone in need. Report-Outs to the Large Group: Following each small-group discussion, brief presentations by table representatives of three key points that table members elected to share with the full group. Local Partner Presentation: Short presentations by local partners or their de- signees on public health preparedness planning and resources available to the local community, and an opportunity for participants to receive answers to questions about local preparedness initiatives. ARS Post-Survey: After the last group activity, a repeat of the same opinion statements to which participants responded in the pre-survey, immediately followed by display of the comparative responses from the pre- and post- surveys for participant reaction and discussion. Evaluation: A series of statements delivered by ARS to elicit participants’ opin- ions about their experiences and perceived value of the community conversa- tions, followed by written responses to several open-ended questions. SOURCE: IOM, 2012b. 

22 ENGAGING THE PUBLIC IN CRITICAL DISASTER PLANNING Overall, Shah said, participant surveys in all three locations showed a better understanding of the issues and increased acceptance of the proposed strategies at the end of the session. During the meetings, many participants stressed the need for transparency and communication, but also felt that the scenario discussions were productive and allowed them to express their views and hear the opinions of others. In addition, many said they would recommend that family and friends participate in a similar session if offered the chance. CHALLENGES AND LESSONS LEARNED Situational experience has revealed challenges to using public engagement to inform policy, including the view that public engagement is public opinion, not research. Support from political leadership is also difficult to obtain, particularly if the topic is controversial. Community leaders in cities already undertaking public engagement activities have developed guidelines and best practices for tackling these challenges as well as other insights on experiences that can result in successful efforts in their own regions as well as other communities. Some of the challenges and relevant experiences shared include regulations around IRBs and the Common Rule, garnering leadership buy-in, and consequently managing changes in organizational leadership. Regulatory Considerations: Public Engagement and the Common Rule Donna Levin, general counsel for the Massachusetts Department of Public Health, discussed public engagement from a human subject research perspective, particularly the applicability of the Common Rule.11 The purpose of the Common Rule, she explained, is to protect human subjects who participate in research from unwarranted risks, ensure that any risks are adequately explained, and ensure that human 11 The Federal Policy for the Protection of Human Subjects (45 CFR part 46, subpart A), known as the Common Rule, outlines the basic provisions for human subject research conducted or supported by a federal department or agency, including ethical review of proposed research by IRBs, informed consent from participants, and assurances of compliance from research institutions. See http://www.hhs.gov/ohrp/humsnsubjects/ commonrule/index.html (accessed May 1, 2013). 

WORKSHOP SUMMARY 23 subjects have consented to those risks. Levin noted that although the Common Rule technically only applies to federal agencies and other entities that receive federal funding, it is widely followed. As defined in the Common Rule, research is “a systematic investigation, including research, development, testing, and evaluation, designed to develop or contribute to generalizable knowledge.” A key challenge in applying the Common Rule is interpreting terms such as “systemic investigation,” “testing,” and “generalizable knowledge,” and Levin noted that there is some ongoing discussion on revising these regulations to achieve better understanding. Levin suggested that when participants are planning a public engagement initiative, they should first contact their institutional attorneys to determine if the Common Rule applies. If an attorney is not sure, or is not willing to give a definitive answer, then the IRB should be provided with a short description of the initiative.12 The IRB will determine whether the initiative is research or not, and if it is research, whether it is exempt from review (most public engagement activities would be considered low risk, as participant anonymity is preserved). Levin reminded workshop participants to inquire about the need for IRB approval early in the process to allow enough time for any needed approvals before recruitment. Levin opined that the kind of public engagement activity being discussed at the workshop, in which participants are being asked for their opinions on developing policy, is much more akin to “deliberative democracy” than to research. Public engagement is not a “systematic investigation,” she argued. Rather, the process is similar to public hearings on draft regulations, or town meeting listening sessions on decisions that will affect residents. Levin stressed that her statements were her own opinion, and that workshop participants should always check with their institutional lawyers before proceeding in their own communities. The Massachusetts Experience Levin shared an example of a public engagement session considered to be research subject to the Common Rule. In this case, the Massachu- 12 If a state agency does not have its own IRB, some options include using another state agency’s or state university’s IRB, or a university IRB if that partnership already exists, or hiring an independent IRB. State agencies should consult an attorney about these options. 

24 ENGAGING THE PUBLIC IN CRITICAL DISASTER PLANNING setts Department of Public Health assisted Harvard Medical School in testing public engagement tools and methodology for inclusion in the IOM report on CSC (IOM, 2012a). This was considered research because the investigators were using human subjects to test the best way to en- gage the community in discussions on CSC issues; seeking generalizable knowledge; and using the data obtained to determine best practices and for publication. The end goal was not public engagement to discuss poli- cy issues, but rather testing the engagement process itself (e.g., recruit- ment methods). In this case, the Harvard Medical School IRB reviewed and approved the protocol and informed consent documentation. Leadership Buy-In Another challenge for the public engagement process is securing support from leadership. This can be made even more difficult when a topic is controversial and may be reducible to a sensational (but incorrect) sound bite. For example, Levin noted that the term “death panel” was used by some to describe the process of health care providers engaging their patients on end-of-life care options. But often, public engagement is needed most when the subject is difficult. Levin offered several suggestions for securing leadership buy-in for public engagement:  Brief decision makers as early as possible on the topic, and continue to update them as more is learned.  Make a clear case for why developing and implementing the policy is necessary, and why public engagement is needed. In some cases, public engagement may even be a condition of grant funding.  If needed, obtain assistance from communications staff to best convey the difficult issues the public engagement will raise. Difficult issues can be discussed in a public forum with the right approach and risk communication strategy, Levin said.  If possible, show leadership examples from your jurisdiction or other jurisdictions where this type of public engagement was successful and resulted in better informed and accepted policy.  Have key stakeholders and opinion leaders on the issue available to talk with leadership and support the public engagement proposal. 

WORKSHOP SUMMARY 25  If possible, partner with a professional society, health care organization, or academic institution for the public engagement. A well-planned and well-executed public engagement will result in better policy, with greater public buy-in, and will reflect well on leadership, Levin concluded. Managing Changes in Leadership Once buy-in has been secured from leadership at a particular department or agency, Shah said, a change in leadership can lead to additional challenges (e.g., following an election). Scott agreed, noting that she has worked with two governors and three department directors during the course of the past 5 years on the Michigan ethical guidelines project. When the project launched, the administration was supportive and the department director was very engaged in preparedness. When the administration changed, there was a new department director and other changes within the department. Scott and her staff ran educational sessions for individuals who supported the director (e.g., public information officer, deputy director) so that the administration could get up to speed quickly. She also set up a meeting with the new director, the point of contact for the project, and one of the physicians from the advisory committee. These external stakeholders were advocates for the project and could discuss the progress of the Michigan project relative to other states’ efforts. The Value of Outside Expertise and Sharing Best Practices Bringing outside expertise or benchmarking to best practices can help jurisdictions get a public engagement project up and running more quickly and often uses fewer resources than if the engagement exercise were designed from scratch. For example, the Michigan ethical guidelines project started out by adapting the model that the CDC used for its deliberative meetings, Scott said. They also tapped Roger Bernier, a former CDC program officer, to help guide the steering committee. Scott also pointed out that, during the course of a multiyear project, people will cycle on and off advisory committees, and there can be a steep learning curve even for those with prior knowledge of the issues. 

26 ENGAGING THE PUBLIC IN CRITICAL DISASTER PLANNING To help bring new members of the statewide Ethics Advisory Committee up to speed quickly, Scott and her colleagues are developing a primer on the issues that will help keep the group moving forward, regardless of inevitable changes. Recruitment Strategies Li-Vollmer elaborated on the processes used to ensure a diversity of perspectives that were reflective of the Seattle and King County community. For one of the early public engagements, the group was approached by the CDC to help with recruitment for an influenza pandemic planning meeting 3 weeks before the meeting date, and there was no funding available for travel reimbursement, meals, or other incentives to attend. Li-Vollmer said that most of those who attended were already interested in the issue of influenza, and were eager for the chance to meet with the CDC. Many came with materials they had printed themselves from websites and had very specific questions. Although their input was valuable, the people you really want to have in the room, she said, are those who have never considered these kinds of issues before. Input from these people can provide a better sense of how the general public may react in a crisis situation. The lesson learned was that there should be some kind of incentive to get members of the general public to attend. Li-Vollmer agreed with Levin about the importance of leadership buy-in and added that securing leadership buy-in can be especially difficult when funding for incentives will be needed in a time of fiscal austerity. Stipend For a subsequent public engagement session on allocation of scarce medical resources, meals were provided and participants received a $100 stipend for attending. Incentives can make a huge difference in levels of recruitment, Li-Vollmer said, and can reduce the time spent on recruitment. If giving out cash incentives is problematic, she suggested offering gift cards instead. Shah concurred, reiterating that Harris County offered a $75 stipend for the pandemic influenza public engagement meeting, as well as a $50 gift card for the nurse triage line project. Even so, it was still challenging to gather a diversity of perspectives. Shah underscored the point that it is often the underserved, more vulnerable, or 

WORKSHOP SUMMARY 27 harder-to-reach members of the community who are left out of the decision-making process and who should be incentivized to attend. Meeting Location It is also important to consider ways to make community participation easier. The location of the meeting, for example, should be reachable by public transportation, Li-Vollmer said. She added that transportation is a particular issue for people with disabilities; however, people with disabilities may be those most in need of scarce resources in a crisis (e.g., generators for people requiring oxygen). Holding meetings at locations familiar and comfortable to the community can assist in greater participation. These might include community centers or civic or religious centers community members already frequent. In cases where it is not feasible for members of the community from a sensory or functional needs group to participate, input can be obtained from community organizations that advocate on behalf of these groups. Additionally, considering ways to address childcare issues can help promote attendance and involvement in meetings so that interested attendees do not have to find and pay for childcare. In retrospect, Scott noted that sending written invitations was not particularly effective for the Michigan public engagement project. Of 355 organizations invited, only 15 agreed to attend. The meeting was rescheduled for 2 months later, and 40 organizations attended. Moving forward, Scott said, they are refocusing their engagement strategies for the next session. Outreach and Dissemination To facilitate future recruitment, Scott said that participants in public engagement meetings were asked if they were willing to provide contact information, and if they would be willing to attend future meetings or would be interested in hearing about future opportunities. Those who provided their contact information were sent copies of the final report, electronically or through the mail, so that they could see the findings from the project in which they participated. Li-Vollmer recommended that organizers ask participants to sign up if they would like to hear about future opportunities to participate. She noted that, in her experience, nearly every participant signed up when offered the chance, which she felt was a testament to how positive they felt about the experience. 

28 ENGAGING THE PUBLIC IN CRITICAL DISASTER PLANNING Li-Vollmer said that Seattle and King County worked with an agency for recruitment and outreach, and the agency chose to use a mass media campaign. The results were similar to Shah’s (see Table 1), in that very few people came to the engagement meetings based on television or radio ads. It is much better to do targeted outreach if you are trying to reach a diverse audience, she said. One targeted approach was to contract with community organizations for recruitment. Li-Vollmer recalled that the organizations were paid between $300 and $500, and, in turn, they committed to bringing 20 of their clientele to the meeting. Although there was a lot of attrition of individuals who signed up in advance to attend, there were much better results when participants came through their association with the organizations. Community blogs were also used for outreach. Shah added that the recruitment videos posted to the Harris County webpage were available in English, Mandarin Chinese, Spanish, and Vietnamese, the four most-spoken languages in the county. Links to the videos were sent to the partners and stakeholder organizations so that they could direct people to the website to view the videos. PUBLIC ENGAGEMENT SAMPLE EXERCISES To give attendees a better sense of the process, two simulated interactive public engagement exercises were facilitated by Shah and Li- Vollmer: a community conversation and a Q-sort activity. The community conversation allowed workshop participants to discuss their concerns about catastrophic disasters and included breakout groups, brief report-outs, real-time pre- and post-surveys, and an expert presentation. Given the scenario of a severe influenza pandemic, the Q-sort exercise engaged workshop participants in a hands-on activity for ranking opinion statements from least to most important. These exercises gave workshop participants experience with tools they can utilize in public engagement activities for their own communities. Community Conversation Simulation The first exercise was a community conversation taken from the IOM Crisis Standards of Care toolkit discussed earlier by Shah (IOM, 

WORKSHOP SUMMARY 29 2012a).13 Attendees were divided into two groups, and Li-Vollmer and Scott served as group facilitators, while Shah served as the lead facilitator. Shah reviewed the agenda that participants would go through in a real engagement exercise (see Table 4). Following opening remarks and an overview of the topic, an introductory exercise would be conducted, asking participants what worries them the most about a catastrophic disaster (i.e., what prompted them to attend the session). This would be followed by a brief report-out from the groups, and then a pre-survey using clicker audience response technology would be conducted to establish baseline opinions (see Box 3). TABLE 4 Sample Agenda for Interactive Public Engagement Exercise Activity Purpose 1. Opening Remarks  Welcome participants.  Introduce leaders/facilitators.  Briefly describe topic.  Explain charge of the day. 2. Introductory Exercise  Break the ice.  Ground rules.  Take temperature—why did they attend? What is on their minds? 3. Brief Report-Out  Introduce participants to report-out method.  Segue into crisis standards of care (CSC). 4. Pre-Survey  Quick immersion into CSC: context, scope, and complexity of the issues.  Help maintain focus on CSC vs. other preparedness or health care issues.  Establish baseline opinions. 5. Expert Presentation  Educate participants about CSC: what, when, why, how?  Opportunity for brief Q&A. 6. Earthquake Scenario  Elicit views on age and likelihood of survival as criteria for resource allocation. Also, withdrawals of treatment and whether 13 See Volume 6 of the IOM Crisis Standards of Care report: http://books.nap.edu/ openbook.php?record_id=13351&page=347 (accessed April 26, 2013). 

30 ENGAGING THE PUBLIC IN CRITICAL DISASTER PLANNING Activity Purpose all of the region’s health care providers should follow the same rules. Other features: sudden onset, geographically contained event.  Participants asked to do forced ranking of hypothetical patients and share their perceptions of the associated challenges. Participants asked to share views on what underlying values and goals should drive decisions. 7. Deadly Virus Scenario  Elicit views on key worker status as criterion for resource allocation. Also, address the role of government agencies in promoting consistent application of CSC rules within the affected area, and withdrawal of treatment.  Participants asked to share views on what underlying values and goals should drive decisions. 8. Report-Out  Give table groups the chance to share key thoughts and identify points of difference and intersection within the larger group.  Spark larger group discussion and input.  Further inform and influence thinking of participants.  Identify changes in participant views pre- 9. Post-Survey and post-session.  Give participants a chance to consider the range of opinions in the room. 10. Discussion of Survey  Satisfy participant interest in survey results. -----Results  Final opportunity to elicit more information on participant views around CSC issues. 11. Final Question  Big-picture question to wrap up discussion. 12. Wrap-Up  Words from sponsor about local disaster preparedness planning initiatives and resources and the importance of individual and community preparedness.  Thank participants. 

WORKSHOP SUMMARY 31 Activity Purpose 13. Evaluation  How did participants experience the session? Do they feel better informed, that they had a chance to share their views, and that it was a valuable exercise? SOURCE: Shah presentation (March 14, 2013), adapted from IOM, 2012a. BOX 3 Sample Introductory Exercise to Establish Baseline Opinions (Abbreviated) Imagine a major disaster strikes (earthquake, hurricane, terrorist attack, pandemic flu, etc.). Suddenly, there are not enough medical resources to provide the normal level of health care to everyone in need. Do you (1) strongly agree, (2) agree, (3) disagree, or (4) strongly disagree with the following statements? 1. It is better to save the most lives—even if it means that some people will not get all of the medical care they would get under normal conditions. 2. More medical care should go to save younger patients because they have the most years to live. 3. The best way to decide who should be treated is to do a lottery or draw straws 4. Health care providers should be at the front of the line for care if they will be able to help save others when they recover. 5. Health care providers should be allowed to perform services different from their usual duties if that might save more patients. 6. Every hospital in the disaster area should follow the same rules when deciding how to use limited medical resources. SOURCE: Shah presentation (March 14, 2013) adapted from IOM, 2012a. After the participants and facilitators have completed the intro- ductory exercises and become familiar with the format of the meeting (e.g., break-out group structure and reporting out, clicker technology for surveys), there would be an expert presentation about CSC so that everyone in the room would have the same minimum level of background on the issue. Crisis scenarios would then be presented and participants would again break into small groups to deliberate and then report out. 

32 ENGAGING THE PUBLIC IN CRITICAL DISASTER PLANNING After all breakout groups have reported on their deliberations, participants are given a post-survey consisting of the same questions they were asked in the pre-survey. The intent is to identify changes in opinions after the presentations and discussions. There are two ways to conduct the pre- and post-surveys, Shah said. The facilitators could show the data from the pre-survey “live” as it is collected with the audience response technology, or they could wait and show the pre- and post- responses side by side during the discussion of the post-survey results. One concern raised during the workshop about showing pre-survey results as they are collected is that participants whose response was in the minority for a particular question may be hesitant to express that opinion again (i.e., some post-survey responses could be more influenced by the initial popular opinion than by the deliberations). Finally, after wrap-up comments from the sponsor of the public engagement exercise, participants are asked to evaluate their experience at the session. Attendee Discussion of the Simulated Community Conversation Although workshop attendees discussed actual policy questions about CSC as part of these simulated exercises, the purpose was not to collect public opinion data on the issues but rather for attendees to experience the process. As such, this summary of the discussion of the issues is included to illustrate the process and some of the challenges participants might face during the activities, and is not a presentation of valid public opinion data on the issues. For the simulated public engagement exercise in this workshop session, Shah presented the deadly virus scenario and questions from the IOM toolkit for workshop attendees to discuss in the breakout groups (see Box 4). The attendees divided themselves into two groups that were similar in number, but they did not give any attention to balance of background, age, or occupation. During this simulated public engage- ment exercise, the first working group considered question 1 from the deadly virus scenario and the second group considered question 2 (see Box 4). A note-taker and a spokesperson were chosen by each group, and after 10 minutes of discussion among members, the spokesperson reported to the larger group the basis of their discussion. This report-out included conclusions that individual group members came to, as well as potential issues of concern members had with definitions and details of mentioned roles. With regard to the process, the spokesperson also reported that the group quickly realized how important it is to frame the 

WORKSHOP SUMMARY 33 question clearly so that a useful answer is obtained. The spokesperson acknowledged that they had differences in opinion based on different priorities among group members, but many eventually came to a general agreement for the question. The group also noted that it would be important to have a mechanism for each participant’s voice to be heard, and the importance of strong facilitation was stressed. BOX 4 Deadly Virus Scenario A new deadly virus has killed many thousands of people around the world. It is highly contagious and is now spreading quickly in the United States. There is only one known treatment for people who fall ill with this virus—an antiviral drug that is in short supply worldwide. It will take several months to make enough of this drug to treat most of the people in the country who need it. A vaccine to protect against this virus will not be ready for even longer, so the numbers of people sickened by the virus will continue to rise. The virus reached your community last week. People of all walks of life have started to fall ill, including health care and other emergency workers. Of the people who catch the virus, 20 percent will die if they are not treated with the drug. Although everyone is at risk of dying, experts have determined that among those who catch the virus, healthy young adults and pregnant women face the highest risk of death by far. Your community’s public health agency will receive its first supply of the antiviral drug in a few days, but only enough to treat fewer than half of the seriously ill patients who currently need this treatment. The agency will divide the drugs between the local hospitals and temporary clinics that have been set up to handle the overflow of patients. The hospitals and clinic will have to make do with whatever supply of the antiviral drug they receive—there is no other source from which they can get more at the present time. Patients who are chosen for treatment will need to take the drug for 7 days. The agency is now preparing guidelines to help hospitals and clinics decide which patients to treat with this limited supply of the antiviral drug. Discussion Questions 1. Should the agency’s guidelines give health care workers priority for treatment? What about other workers on whom society relies for saving lives (e.g., electrical power and water supply workers, police, firefighters, and other key workers)?  What are the best reasons for and against such a policy?  Does a health care worker’s specific role or contact with patients matter (e.g., patient care providers such as doctors and nurses; custodians or food service workers; managers or administrative staff)? 2. In order to save the most lives, the agency is planning to direct all hospitals and clinics to restrict use of the antiviral drug to sick patients 

34 ENGAGING THE PUBLIC IN CRITICAL DISASTER PLANNING in the two high-risk groups (pregnant women and previously healthy young adults) until more of the drug becomes available  Do you agree with this policy? Why or why not?  Is it important for all hospitals and clinics in the city to follow the same rules when deciding which patients to treat? Why or why not? 3. It is now day 3 of treatment for patients in the first group to receive the drug. Two of those patients have not responded to treatment, and their doctors now believe that they will almost certainly die. If treatment is stopped now, there will still be enough of the drug left over to treat one more patient who might be saved. The families of the two dying patients will not agree to end the treatment.  Should the hospital go against their wishes and use the remaining doses to try to save another patient?  Why or why not? SOURCE: Shah presentation (March 14 2013) reprinted from IOM, 2012a. Shah and Li-Vollmer emphasized that when conducting a public engagement activity, it is very important to communicate clearly at the beginning of the discussion that everyone’s opinions are valued, but this does not mean that agreement among individual group members from any particular session will decide the final policy. For example, a workshop participant said that if the final policy did not seem to take into account the concerns expressed in the breakout session, attendees might be disappointed and feel that in the end, “the government is going to do what they want to do, no matter what I say.” Li-Vollmer noted that longer public engagement sessions give people the opportunity to hear a wider variety of perspectives and to realize that there are many ways of thinking about the particular issue, as was experienced during the workshop exercise. Breakout group members commented that it was interesting to hear about different perspectives, such as that of a professional policy maker, or a young adult without any children, and learn how their experiences shape their opinions. Li-Vollmer encouraged the inclusion of health care workers in engagement sessions that consider whether health care workers should be prioritized. She reiterated that, in her experience, although the public wanted to give priority to health care workers, many health care workers did not feel that they should have priority access to treatments, and they offered a variety of reasons why. 

WORKSHOP SUMMARY 35 Q-Sort Simulation The second simulated public engagement exercise presented at the workshop was a Q-sort activity (see methodology described in Box 1), and Li-Vollmer presented a severe influenza pandemic scenario to set up the exercise (see Box 5). For the Q-sort, each workshop attendee was given a laminated board (see Figure 2) and a plastic bag full of cards printed with statements about how decisions should be made regarding allocation of lifesaving resources. For example, statement cards included “priority for medical care should be given to patients expected to live the longest,” and “it is important to give priority to certain groups. Otherwise, the hospital will fill up with people who get there first and those who get sick later may not get medical care.” Li-Vollmer explained that these statements reflect actual opinions given by both members of the general public and health care experts. BOX 5 Pandemic Influenza Scenario During a severe influenza pandemic, hospitals become extremely overcrowded and must attend to numerous people who are very ill with flu while still caring for the other incoming patients they see every day (e.g., accident victims, heart attack victims, pregnant women in labor). In addition to being beyond capacity, they are understaffed, as many health care workers are home sick themselves, and short on supplies. Not everyone can get treatment, and tough decisions must be made about who among the most sick will get access to lifesaving care. Those who cannot get lifesaving care will need to be given some kind of “comfort care” to ease their symptoms. Hospitals around the region, state, and country are all dealing with the same catastrophic situation. Assumptions for Q-sort exercise:  The focus of discussion is limited to access to very scarce lifesaving medical resources (e.g., intensive care unit beds, ventilators).  There are not enough of the lifesaving medical resources to treat everyone who needs them.  People who do not get lifesaving care will most likely die.  The majority of the population will survive the flu pandemic. SOURCE: Li-Vollmer presentation (March 14, 2013). 

36 ENGAGING THE PUBLIC IN CRITICAL DISASTER PLANNING FIGURE 2 Q-sort board for ranking opinion statements printed on cards. SOURCE: Li-Vollmer presentation (March 14, 2013). With the scenario in mind, participants were instructed to sort the different statements into two roughly equal piles according to whether the stated opinions were least important or most important to them personally. Next, Li-Vollmer instructed participants to pick out the one statement from the “most important” pile that is the most important to them, and place it in column 9 (the brown column) on the board. Similarly, participants then picked from their “least important” pile the one statement that was the least important of all, and placed it in column 1 (the red column). Participants then worked through the rest of their cards, placing the next two most important statements in column 8 (grey) and the next two least important statements in column 2 (yellow), and so forth until their boards were filled. Cards within a column did not need to be in any particular order. Developing the Q-Sort Opinion Statement Cards Creating the opinion statement cards can be labor-intensive, Li- Vollmer said. According to the methodology, the opinion statements are developed by conducting a broad spectrum of interviews with people from all walks of life. The formulation of the opinions is also informed by a literature search, including articles in the popular press, about CSC and ethical principles. The opinion statements must then be carefully worded so that they are easily understandable and fit on a small card. 

WORKSHOP SUMMARY 37 Statements are then tested and refined. Li-Vollmer referred participants to the Seattle and King County opinion toolkit website for the opinion statements used on the cards.14 The Value of the Q-Sort Approach for Public Engagement Exercises Traditionally (e.g., in psychiatric and psychological research), Q-sort is done as a one-on-one interaction where people sort cards and an interviewer talks to them about their choices as they sort. Li-Vollmer explained that Seattle and King County adapted the Q-sort methodology for group use. The Q-sort exercise itself is a methodical process of sorting through opinion statements, and the placement of the cards on each participant’s board is documented, both initially and later, if any changes are made after discussion. A critical part of the process is the discussion, Li-Vollmer said, and a lot of rich qualitative data are obtained from hearing people explain their choice of rankings. In public engagement exercises in Seattle and King County, the Q-sort is used as the first activity (after the pre-survey, overview of the issue, and the subject matter expert presentation). In many cases, Li- Vollmer said, this is the first time many participants have been exposed to these issues, and they have not had any prior chance to process their thoughts. Ranking the different opinions presented to them using the Q-sort method pushes them to really consider what is important to them and helps them crystallize their own opinions before they engage in group discussion. She added that participants are given the opportunity to modify their Q-sort board after the group discussion. The ranking process also stimulates conversation as participants discuss why they ranked a particular statement the way they did. This qualitative data allows for more subjective nuance than a survey that prompts participants to answer yes, no, or maybe, Li-Vollmer said. If desired, Q-sort can also be used for factor analysis, and she noted that there is free software available online that can analyze responses and identify groups with similar perspectives. Attendee Discussion of the Simulated Q-Sort Exercise Again, although workshop attendees discussed actual policy ques- tions about CSC as part of these simulated exercises, the purpose was not 14 See http://www.apctoolkits.com/vulnerablepopulation/knowing/public_engagement (accessed April 26, 2013). 

38 ENGAGING THE PUBLIC IN CRITICAL DISASTER PLANNING to collect public opinion data on the issues, but rather for attendees to experience the process. As such, this summary of the discussion of the issues is included to illustrate the process of the act-ivities, and is not a presentation of valid public opinion data on the issues. After all boards were completed in the simulated exercise, Li- Vollmer asked for volunteers to share what card they put in the column 9 box (the statement they felt was the most important consideration when deciding who will get priority access to lifesaving resources [see Figure 3 for a sample Q-sort board completed at the workshop]). Demonstrating how to facilitate discussion, Li-Vollmer then asked, “Do you want to expound about why you chose that as your single most important consideration?” and “Did anybody else choose that as their most important consideration?” Participants explained the reasoning for their answers, and others chimed in with their own highest-priority rankings. Some participants cited the ethical principle of justice for their choices, as well as the need to have a rational process instead of an arbitrary distribution for their choices. FIGURE 3 Example of a Q-sort board completed during the simulated public engagement exercise at the workshop. 

WORKSHOP SUMMARY 39 As facilitator, Li-Vollmer continued by asking if there were other opinions that should be considered but were not included on the cards. Attendees gave suggestions to add specific details to the opinion cards regarding the scenario to give participants a better idea of the operating picture. A participant also raised concerns about certain statements being highly subjective and potentially leading to discrimination against certain demographics. During discussion, many participants voiced their ap- preciation for being a subject in this type of activity and having an opportunity to see how their opinions matched up with those of their peers in the scenario and to hear needs and opinions different from their own. FINAL REMARKS The above exercises gave workshop participants hands-on knowledge of just a few examples of public engagement methodologies and pro- cesses that can be successfully used in their own communities. Through practical application, these exercises built upon the workshop’s goal of providing attendees with theories and practices, lessons learned, and the overall challenges of public engagement. Public engagement is an important aspect of disaster preparedness, especially with important decisions and policy matters that may be controversial, and gives local health and community leaders essential public input on complex problems. Participants appreciate the opportunity to be heard, as well as to hear what is happening in their own community. When done well, Shah concluded, public engagement can yield tremendously useful information for all parties. Different methodologies (e.g., scenario-based group discussion, individual Q-sort activity15) can be used in the same public engagement session, Shah said. For example, Harris County used Q-sort to assess opinions on allocation of vaccines and antivirals and facilitated group discussions for allocation of ventilators. There are advantages to each methodology, depending on the situation. There are also different techniques that can be used for the different methodologies. Voting during a community conversation, for example, can be done using the clicker audience response system or by simply having participants put stickers in different columns on poster boards. Conducting these activities, various speakers pointed out, can help to inform policy 15 See Box 1 for details on Q-sort methodology. 

40 ENGAGING THE PUBLIC IN CRITICAL DISASTER PLANNING decisions and educate the community, as well as increase the legitimacy and public acceptance of local disaster plans. Solutions might not be identified in just one or two meetings, but starting the conversations and building strong stakeholder relationships in the planning stage are vital to efficient and successful execution of disaster plans and communication when a disaster occurs.