The 2009 H1N1 Influenza Vaccination Campaign: Summary of a Workshop Series (2010)

Data collection and analysis have critical roles in shaping programs, tracking progress, evaluating results, demonstrating accountability, and informing policies that will improve plans for future events. However, participants noted that it is extremely challenging to develop feasible methods for collecting accurate, reliable, and meaningful data during a public health emergency. Several participants noted that data collection is always secondary to the on-the-ground provision of care. During the 2009 H1N1 pandemic, “In terms of vaccination, we were not really there to collect data,” explained Megan Davies, state epidemiologist in North Carolina. “Data were very secondary to me in the vaccination effort. The one thing we don’t want to do is come out with some recommendation that puts big obstacles in the way of getting vaccine into human beings.” For this reason, it is important to consider data collection and analysis needs during a pandemic as preparedness plans are developed and revised so they can be fully integrated into the plans and included in exercises and drills.

In general, data tracking of vaccine administration was considered poor in most jurisdictions, making it difficult for public health authorities and healthcare providers to determine in real time, or near real time, whether their efforts were successful.

Many workshop participants discussed how their organizations struggled with the issue of mandatory data entry to track vaccine administration because of fears that this would become an obstacle to vaccination. West Virginia’s Slemp noted, “We knew that if our private providers were going to get engaged in this system, we had to minimize the amount of data we wanted.” There was concern that healthcare providers would not order vaccine if data entry into a registry was required.

“As much as from a data [collection] standpoint I hated that it wasn’t required … we needed everybody administering as much as possible,” said Angie Hagy, infectious disease epidemiologist for the City of Milwaukee Health Department’s Division of Disease Control and Environmental Health.

At a national level, the CDC developed a project with the University of Michigan to conduct regular surveys with immunization program managers via phone calls or e-mail (Clark et al., 2010). The surveys were designed to provide the CDC with real-time information about how states were implementing their vaccine distribution and administration plans. They also gave the CDC feedback on their communications as well as information about needs that could be addressed. In the early weeks, the surveys looked at which target groups the states were prioritizing for vaccine. In the later weeks, the surveys asked which states were doing school vaccinations, whether vaccine had been distributed to retail pharmacies, and what plans were made for coming weeks. Pascale Wortley, chief of the Health Services Research and Evaluation Branch of the CDC’s Immunization Services Division, noted that careful measures were needed to set up the survey project successfully and ensure that busy immunization program managers would be willing to invest the significant time needed to participate. She said the CDC worked through the Association of Immunization Managers (AIM) and ASTHO to develop buy-in, and also noted that the surveys were done by a team whom the immunization program managers already knew and had worked with before. The information was shared through AIM and ASTHO so states could learn what other states were doing. Although it was clear that data collection requirements should not become a burden to the public health and healthcare provider community, workshop participants also discussed how the absence of certain kinds of data had a negative impact on the vaccination campaign. Such data would have enabled them to adjust plans throughout the campaign and improve plans for the next public health emergency. Participants identified a number of datasets that were not available but that would have been valuable in informing their campaigns (Box 7-1). Therefore, as West Virginia’s Slemp highlighted, there is a need to develop and integrate standardized data collection systems into all pandemic plans and exercise these plans to ensure they will not be a burden during a response. Furthermore, several participants noted, more resources are needed to facilitate the collection and analysis of the data.

Each jurisdiction made different decisions based on their data collection systems and provider feedback. Jurisdictions required varying levels of data collection. Many data collection methods were used, including electronic data collection via batch downloads, direct interfaces, data entry, and paper and pen.

Some of the data collection models used by state, city, and county public health authorities are described below. They give an overview of the variety of approaches used and highlight aspects of approaches that participants identified as being especially successful. This section is not intended to be a comprehensive review of state and local data collection models, but rather to highlight some models presented during the workshop series.

In Boston, a patient tracking system designed to rapidly collect information using personal digital assistant (PDA) devices scanned each patient’s driver’s license upon entry into a mass clinic, if the patient was willing to provide one. The license was scanned again upon exit, and the type of medication received was entered into the system. This provided real-time information on what was happening in the clinics. Laura Williams, deputy chief of staff with Boston EMS, said approximately 60 percent of

people going through the clinics provided their license. For the administration sites that did not have electronic data collection capabilities, either an Excel spreadsheet or paper forms were used and the data were incorporated later.

This was the first time that Boston Public Health had fully used electronic data collection for a public health emergency response, and they found it to be a success. The data collected included the patient’s name, address, phone number, age, and gender; clinic location; date; medication information; and relevant information regarding a second dose, as applicable. “The benefit of having an electronic, condensed spreadsheet was that we could provide real-time reports to the executive director—real-time flexibility with rapid feedback,” Williams said. “If there was a reaction, [we could] quickly search and provide the person with specific medication they received, when, and any other information they requested.”

Using a similar strategy, Indiana was able to work with its Bureau of Motor Vehicles and download all driver’s licenses into their registry. Although there was a lot of duplication, the patient’s name, address, and age were already present at the mass clinics and could be cross-correlated. These data are likely to provide a wealth of information to Indiana as the state assesses and updates its pandemic plan.

The use of immunization information systems—commonly known as immunization registries—varied widely by state. There were anecdotes of healthcare providers not vaccinating because of a registry reporting requirement. For example, Slemp heard that one major medical center in West Virginia did not vaccinate its in-patient population because it could not readily export the data. In Texas, Jennifer Jackson, R.N., of the Williamson County and Cities Health District in Georgetown, Texas, noted that some healthcare providers decided not to order 2009 H1N1 vaccine when the state required doctors to use the ImmTrac registration system. Once the state relaxed the requirement, more providers ordered vaccine. Although co-occurring factors may have also led to increased provider orders, this highlights the need to take into account the needs of practitioners, especially those with smaller practices, when developing collection and reporting requirements.

Because use of registries can potentially decrease participation by healthcare providers, the benefits of getting data and tracking vaccine had to be weighed against a reduction in administration of vaccine, the context of the actual event, the amount of vaccine available, the speed of vaccine production, the severity of the disease, and the importance of vaccinating target groups. “Ultimately, only a quarter of the states required patient-level reporting through immunization registries,” said Beth Rowe-West, head of the Immunization Branch of the North Carolina Department of Health and Human Services.

In states that did require reporting through registries, if electronic data transfer was not available, the states had to rely on manual data entry, which was arduous and costly, especially if healthcare providers saw no direct benefit to entering the information. Furthermore, training new practitioners on data collection for a registry takes time and resources, which could strain already busy private practices.

Virginia provided a $1,000 grant to new registry users to encourage use of the registry. This grant helped offset healthcare providers’ costs in training staff on the system or connecting their existing electronic medical system to the registry.

Washington State used billing data and an interface with the CHILD profile, its health promotion and immunization registry, to obtain data from private healthcare providers, noting that few private providers such as pediatricians or family practice doctors have the staff, time, and money required to keep up with the data entry requirements. “It [takes] basically an FTE [a full-time equivalent employee] to do that when you are doing immunizations on a daily basis,” noted Kaneshiro, past president of the state’s chapter of the American Academy of Pediatrics.

Using registries during the 2009 H1N1 vaccination campaign had short- and long-term benefits. A short-term benefit of having more data available earlier was the ability to modify plans and improve efficiency and effectiveness during implementation of the vaccination campaign. In the long term, participants noted, requiring healthcare providers to become familiar with using the registry built up awareness and infrastructure, which could be leveraged in future routine activities and public health emergencies. The challenge now is working with clinicians to ensure they continue to use the registry, not only to enter the information, but to be able to use that information and see the value in the system.

Registries have the potential to increase the ability to track countermeasure distribution, making them a more holistic point-to-point response management system, although such an expansion would require

significant investments. One issue raised in making such changes to registries is the legal implication of going beyond a registry’s stated purpose. But this legal change would also arise if registries were used to target vaccine administration to certain groups based on health status. Registries are good at telling who needs vaccinations and doing basic reporting, noted Milwaukee’s Hagy. “But it is definitely not set up as an electronic health record. You can’t go into the Wisconsin Immunization Registry and find out who is pregnant, who has underlying medical conditions, [and] who is a healthcare worker, so that we could answer the questions that everybody wanted to know: What percentage of shots were going into people of different risk groups? Were the shots equally distributed? Were we impacting racial and ethnic minorities the same?” Without that type of information, this type of query had to be answered based on zip codes and limited racial and ethnic data. Additional information regarding states’ experience with using registries is described in Box 7-2.

The information management systems showed a high level of variability in tracking administration of vaccine across state and local public health systems. This caused problems particularly for partners who had to interact with multiple public health jurisdictions and/or also report to their own internal systems, such as the Department of Veterans Affairs and large pharmacy chains. Several participants suggested it would be useful to explore standardization of information management systems and data reporting requirements and to analyze current immunization registries and other systems to assess where variability is not warranted.

In LA County, the service planning areas with the highest number of flu outbreaks also had the highest number of vaccine doses administered. Mascola, of the county’s Department of Public Health, hypothesized that this was because if people see sick people around them, they are more likely to get a vaccine. However, the PODs did not have good situational awareness because of the lack of access to real-time data about disease outbreak or vaccine administration. Thus, like most jurisdictions, it was not possible to modify plans and respond to what was happening out in the field. LA County did use Scantron optical forms, which allowed it to look at the hourly flow of people in each POD. This helped inform staffing ratios for PODs used later in the campaign.

Mascola commented that the analysis of the datasets showed that “it was the ethnic minority groups, the Latinos and African Americans, who had the highest [2009 H1N1] flu death rates in LA County per 100,000.” Unfortunately, the communication strategies they used were not developed in a manner that would allow for real-time updates based on the changing situation. As new plans are developed based on lessons learned during the 2009 H1N1 vaccination campaign, it will be important to develop communication campaigns with greater dynamic flexibility.

Without access to real-time data, it was difficult to determine during the event if particular strategies were effective or if particular areas within a jurisdiction were more or less crowded. However, even with improved situational awareness, metrics still need to be established to determine what a success is.

Even now, many community health centers and pharmacies lack information technology, such as computers or Internet access for direct data entry, or Internet connectivity or computers at the point of sale or clinic location. During the 2009 H1N1 pandemic, this meant that alternative data entry systems had to be created, such as Excel spreadsheets for those with computers but no Internet, and paper forms for those without computers. This resulted in an even greater lack of conformity in the data collected, making large-scale analysis more difficult. Furthermore, these alternative systems also created additional staffing demands because the data had to be entered and integrated into the primary data collection system.

Many participants noted that data collection required significant staff time and associated costs in public health departments, pharmacies, private practices, and other locations used to administer vaccine. For example, the Boston Public Health Commission hired a part-time intern and three temporary employees to compile and enter into the database all the information from clinics and other locations that used Excel spreadsheets and paper forms.

Jurisdictions that were not able to hire additional staff or receive forms on a daily basis experienced huge backlogs in data entry—even if

optical-scan forms were used. Once backlogged, catching up became difficult and much staff time and effort went toward data entry. Hagy of Milwaukee noted that if the vaccine supply had been as large during the initial months as was originally expected, the time lag for data entry would have grown exponentially. Therefore, although there is a tremendous need for real-time data collection to improve situational awareness, strategies need to be developed and used minimize staff and cost burdens.

In some areas, few data were received from private healthcare providers. In LA County, where 80 percent of the vaccine was administered by private providers, few data were returned. Public health, which administered the other 20 percent of vaccine, collected data via forms that could be scanned. However, not all locations had good experiences with such forms; the City of Milwaukee Health Department ran into many problems with the optical character recognition software due to variations in handwriting. This highlights the need to establish policies that create push-or-pull incentives to ensure that data are collected and returned to the appropriate authorities.

Numerous individual suggestions were made about opportunities to improve data collection, monitoring, evaluation, and use. These suggestions are compiled here as part of the factual summary of the workshops and should not be construed as reflecting consensus or endorsement by the workshops, the Preparedness Forum, or The National Academies. They are as follows:

• Develop technologies that facilitate real-time data collection and reporting to improve situational awareness and guide program implementation during a public health emergency.

• Explore standardization of information management systems and data reporting requirements. Analyze current immunization registries and other systems to assess where variability is not warranted.

• Develop and enhance systems that automatically share information from electronic medical records and practice management systems with systems that track vaccine administration.

• Simplify data collection and reporting requirements. Determine the most important data elements to collect during a public health emergency. Some information may be critical to the mission and associated data should be collected in all types of events, but other information may be able to be prioritized according to severity of incident and availability of resources.

• Establish data collection forms for use in everyday practice that can also be used during public health emergencies.

• Use seasonal influenza vaccination data from previous years to inform vaccination plans and help determine where additional outreach may be necessary. For example, LA County’s data showed that one area with a large African American population had the lowest number of seasonal flu immunizations, and this low rate was also found in the 2009 H1N1 vaccination campaign.

• Establish common performance metrics. “We need to know what we are striving for and how to measure that and how to know whether we are being successful,” said Jack Herrmann, senior advisor, Public Health Preparedness with the National Association of County and City Health Officials (NACCHO). Consider nonempirical measures of success if prior data measures are unavailable.

• Direct grantees (e.g., Chicago) should have access to data on their own area of control, rather than being aggregated with state data.

• Bar coded and color code vaccine to reduce time and potential for errors in vaccine administration and data entry.