THE NATIONAL ACADEMIES

Advisers to the Nation on Science, Engineering, and Medicine

National Research Council

Division on Earth and Life Studies

Board on Environmental Studies and Toxicology

500 Fifth Street, NW Washington, DC 20001 Phone: 202 334 2347 Fax: 202 334 2752

April 20, 2009

Dr. Carol Maczka

Assistant Administrator

Office of Food Defense and Emergency Response

Food Safety Inspection Service

United States Department of Agriculture

South Agriculture Building 1400 Independence Avenue SW, Room 3130 Washington, DC 20201

Dear Dr. Maczka:

At the request of the Food Safety and Inspection Service (FSIS), the National Academies Division on Earth and Life Studies established the ad hoc Committee for Review of the Food Safety and Inspection Service (FSIS) Risk-Based Approach to Public-Health Attribution. This ad hoc National Research Council committee is one of three independent committees charged to examine the use of public-health data in FSIS food-safety programs. The committee was selected to include expertise in food science, food safety, meat processing, microbiology, biostatistical sampling, microbial-risk assessment, foodborne-disease epidemiology, and disease attribution.

The committee was tasked with evaluating the proposed FSIS methodology and the adequacy of the data used to estimate foodborne-disease attribution for the purpose of ranking slaughtering and processing establishments according to public-health risk. The statement of task is discussed further below.

The committee held two meetings, each with a public session in which FSIS staff presented their proposed approach to public-health risk-based inspection, including the calculation and use of attribution estimates to link foodborne illnesses to specific FSIS-regulated products and application of the model by Hald et al. (2004) to calculate Salmonella attribution estimates. In addition, the committee heard from stakeholders regarding their concerns with the proposed FSIS risk-based approach.

This letter report first provides background information on the FSIS approach to attribution and its method of estimating attribution. The strengths and limitations of approaches used for estimating attribution are discussed next. The proposed FSIS approach to estimating attribution for risk ranking and determination of performance objectives is then assessed, and the report closes with conclusions and recommendations regarding future steps that FSIS should take to ensure that attribution estimates and related public-health goals make use of the best available science and enhance protection of public health. Several attachments are included: Attachment A, a verbatim statement of the task; Attachment B, the committee roster and biographies; Attachment C, a bibliography; Attachment D, acknowledgment of



The National Academies | 500 Fifth St. N.W. | Washington, D.C. 20001
Copyright © National Academy of Sciences. All rights reserved.
Terms of Use and Privacy Statement



Below are the first 10 and last 10 pages of uncorrected machine-read text (when available) of this chapter, followed by the top 30 algorithmically extracted key phrases from the chapter as a whole.
Intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text on the opening pages of each chapter. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.

Do not use for reproduction, copying, pasting, or reading; exclusively for search engines.

OCR for page 1
National Research Council 500 Fifth Street, NW Division on Earth and Life Studies Washington, DC 20001 Board on Environmental Studies and Toxicology Phone: 202 334 2347 Fax: 202 334 2752 April 20, 2009 Dr. Carol Maczka Assistant Administrator Office of Food Defense and Emergency Response Food Safety Inspection Service United States Department of Agriculture South Agriculture Building 1400 Independence Avenue SW, Room 3130 Washington, DC 20201 Dear Dr. Maczka: At the request of the Food Safety and Inspection Service (FSIS), the National Academies Division on Earth and Life Studies established the ad hoc Committee for Review of the Food Safety and Inspection Service (FSIS) Risk-Based Approach to Public-Health Attribution. This ad hoc National Research Council committee is one of three independent committees charged to examine the use of public-health data in FSIS food-safety programs. The committee was selected to include expertise in food science, food safety, meat processing, microbiology, biostatistical sampling, microbial-risk assessment, foodborne-disease epidemiology, and disease attribution. The committee was tasked with evaluating the proposed FSIS methodology and the adequacy of the data used to estimate foodborne-disease attribution for the purpose of ranking slaughtering and processing establishments according to public-health risk. The statement of task is discussed further below. The committee held two meetings, each with a public session in which FSIS staff presented their proposed approach to public-health risk-based inspection, including the calculation and use of attribution estimates to link foodborne illnesses to specific FSIS-regulated products and application of the model by Hald et al. (2004) to calculate Salmonella attribution estimates. In addition, the committee heard from stakeholders regarding their concerns with the proposed FSIS risk-based approach. This letter report first provides background information on the FSIS approach to attribution and its method of estimating attribution. The strengths and limitations of approaches used for estimating attribution are discussed next. The proposed FSIS approach to estimating attribution for risk ranking and determination of performance objectives is then assessed, and the report closes with conclusions and recommendations regarding future steps that FSIS should take to ensure that attribution estimates and related public-health goals make use of the best available science and enhance protection of public health. Several attachments are included: Attachment A, a verbatim statement of the task; Attachment B, the committee roster and biographies; Attachment C, a bibliography; Attachment D, acknowledgment of 1

OCR for page 1
reviewers; Attachment E, Agendas for Public Meetings, Attachment F, the form from the Electronic Foodborne-Outbreak Reporting System, and Attachment G, a paper prepared by FSIS, Foodborne Disease Attribution. Sincerely, John C. Bailar III, Chair Committee for Review of the Food Safety and Inspection Service Risk-Based Approach to Public-Health Attribution 2

OCR for page 1
LETTER REPORT ON THE REVIEW OF THE FOOD SAFETY AND INSPECTION SERVICE PROPOSED RISK-BASED APPROACH TO AND APPLICATION OF PUBLIC-HEALTH ATTRIBUTION SUMMARY The committee applauds FSIS’s efforts to develop a Public Health Risk-Based Inspection System (PHRBIS) for the purpose of improving and facilitating priority-setting and resource allocation for FSIS- inspected products that are the major contributors to human-foodborne illness. However, the committee concludes that the algorithm for ranking slaughtering and processing establishments needs to be made more transparent (that is clear and understandable) to clearly present the rationale behind the agency’s approach and to ensure that the system prioritizes resources according to public-health risk. In addition, uncertainty characterization should be included in the inputs into the risk-ranking algorithm, including the attribution estimates. The committee considers that FSIS should use additional data beyond the Centers for Disease Control and Prevention (CDC) outbreak data and expert elicitations to develop better- informed attribution estimates. For example, the CDC outbreak database can be mined further to obtain additional information on sources of contamination. However, the committee acknowledges that it struggled with evaluating the attribution estimates alone, independent of their use in risk ranking, since FSIS’s primary use for these attribution data are for ranking the slaughtering and processing establishments. Because of the limitations of the attribution estimates, the committee considers that it is premature for FSIS to develop performance measures to evaluate the PHRBIS system. The committee concludes that Salmonella serotyping will be critical for improved subtype-based attribution efforts but the Salmonella serotype-based and subtype-based attribution models are not currently ready for policy decision-making. Recognizing the difficulty in estimating foodborne-disease attribution, the committee recommends that FSIS should consider alternative prioritization methods to allocate and prioritize inspection resources, including ranking methods that do not rely on attribution data per se or risk-ranking models that approach the attribution issue differently. Once FSIS selects a risk-ranking approach, it should provide clear and transparent documentation for how it conducted its analyses, including characterization of uncertainty. To the extent practicable, the risk ranking should consider the importance of differences in disease severity associated with different pathogens. Recognizing the value of food- attribution data to many agencies, FSIS should work collaboratively with CDC, FDA, and other federal and state agencies to develop a common set of definitions for microbial foodborne-disease attribution and a coordinated approach to improve the quality and consistency of data used among agencies in determining food-attribution estimates. FSIS should continue to support the collection of serotype and molecular subtype data for Salmonella and perhaps other relevant pathogens, and the development of mathematical models that use these serotype and subtype data for understanding food (and source) attribution of human Salmonella infections. BACKGROUND The Food Safety and Inspection Service (FSIS) is the regulatory arm of the U.S. Department of Agriculture (USDA) responsible for ensuring the safety of the nation’s commercial supply of meat, poultry, and egg products. FSIS is proposing a PHRBIS for all slaughtering and processing establishments. The PHRBIS has been designed to improve the ability of FSIS to protect public health by using data-driven approaches to target inspection resources toward establishments that pose the greatest health risk (Hurd 2008). The PHRBIS is intended to ensure that the basis of proposed decisions is “clearly delineated, transparent, and scientifically-driven” (FSIS 2008a, p. 1). 3

OCR for page 1
In 2003, the Institute of Medicine and the National Research Council published Scientific Criteria to Ensure Safe Food, which provided guidance to FSIS and other relevant U.S. regulatory agencies on the development of science-based performance standards and food-safety criteria to protect public health (IOM/NRC 2003). In 2004, FSIS began to develop a risk-based inspection program for identifying slaughtering and processing establishments that pose the greatest risk to food safety and for developing rankings that enable efficient, targeted use of FSIS inspection resources. In 2007, the USDA Office of Inspector General (OIG) completed an audit of the FSIS risk-based inspection program (OIG 2007). The OIG found that FSIS lacked the infrastructure needed to oversee and support a timely and reliable risk- based inspection program. Among the OIG’s concerns was the lack of FSIS procedures to set priorities for and schedule food-safety assessments, the “fundamental building block for assessing establishment risk” (OIG 2007, p. 32). The OIG stated that FSIS, as it moves forward with the development and implementation of a risk-based inspection program, “should ensure that components of the selected algorithm are thoroughly documented and evaluated with limitations mitigated and are transparent (i.e., clear and understandable) to all stakeholders” (p. vii). In response to the OIG’s report, comments from industry and consumer groups, and previous work by FSIS to develop a PHRBIS to focus inspection resources on the greatest food-safety risks, FSIS is again seeking ways to improve its ability to protect public health. Food may be contaminated by disease-causing pathogens at many points between production (farm) and consumption (fork), so epidemiologic and microbiologic data are used to identify the points of contamination along the farm-to- fork continuum. In many cases, however, it is still difficult to attribute a specific pathogen to a specific food commodity definitively. For example, it is widely known that Salmonella is associated with poultry products, but we do not know what proportion of human salmonellosis cases are caused by chicken, turkey, eggs, or nonpoultry sources, so we do not know how much disease FSIS actions could theoretically prevent. Similarly, we do not know what proportion of cases are associated with undercooking, cross-contamination, or other means of transmission. Without such information, it is difficult to design the most effective strategies to control Salmonella contamination in poultry to reduce the pathogen level in the end product or to evaluate the efficacy of FSIS control strategies after they have been implemented. FSIS asked the National Academies to evaluate the strengths and limitations of the methods that FSIS intends to use to attribute microbial disease to food contamination and to investigate whether the limitations and assumptions in FSIS’s methods hinder their use in policy development. FSIS also asked for comments on the adequacy of Salmonella serotype data for use in informing FSIS attribution estimates. The task undertaken by this committee is part of a broader effort by the National Academies to evaluate data and methods used by FSIS in risk assessment and in planning for risk-based inspection. The charge to the committee is presented in Appendix A. This committee’s evaluation was undertaken in parallel with two companion efforts to evaluate aspects of FSIS’s risk-based inspection system. The Institute of Medicine’s Food and Nutrition Board evaluated the use of process indicators for ranking slaughtering and processing establishments (IOM 2009), and the National Research Council Board on Agriculture and Natural Resources reviewed FSIS’s proposed methods for risk-based regulation of in-commerce activities (NRC 2009). OVERVIEW OF ATTRIBUTION The findings and recommendations of IOM/NRC (2003) stated, “the Committee concludes that science-based food safety criteria must be clearly linked to the public health problem they are designed to address. To accomplish this, a cause/effect relationship needs to be established between contaminants in foods and human disease, that is, to allocate the burden of foodborne disease among foods and food groups” (p. 250). That statement underlines the need for foodborne-disease attribution. The first work on the development of food-attribution estimates was undertaken by Mead et al. (1999), who attempted to attribute the proportions of pathogen-associated illnesses, hospitalizations, and 4

OCR for page 1
deaths in the United States transmitted by food. They addressed transmission routes but not specific food vehicles. Batz et al. (2005) provided a more specific definition for food attribution: “the capacity to attribute cases of foodborne disease to the food vehicle or other source responsible for illness” (p. 993). They emphasized that identifying the food vehicles and other sources responsible for illness is critical for designing and setting priorities for “effective food safety interventions” (p. 993). Similarly, the European Food Safety Authority (EFSA) discussed “efforts to quantify the (relative) importance of specific food sources and animal reservoirs for human cases of food-borne illness” (EFSA 2008, p. 5). Both Batz et al. and EFSA clearly focused on attributing specific pathogens to specific foods, with both addressing the issue of attribution at the point of production and at the point of consumption. On the basis of FSIS-provided documentation, the committee finds that the focus of FSIS efforts is on the proportion of foodborne illnesses or cases attributable to FSIS-regulated foods—meat, poultry, and processed egg products. For example, FSIS estimates that about 60% of all Salmonella illnesses transmitted via FSIS products in 2007 are attributable to poultry products (FSIS 2008b). Because of the heavy reliance on epidemiologic data associated with outbreaks to determine its estimates, FSIS is inherently assessing attribution at the point of consumption. It is important to emphasize that estimating attribution is extraordinarily difficult and ideally takes into account all phases of the farm-to-fork continuum. For example, imagine that a passing starling drops feces on a farm that results in Salmonella infection in a chicken. The microorganism is then transferred to another chicken after slaughter during the chilling process. Cross-contamination in the kitchen results in contamination of lettuce used to make a salad; ultimately, consumption of the salad results in human illness. To what is the illness attributed? If “point-of-consumption” attribution is the intent, the illness is attributed to the lettuce, if it is “point-of-processing,” the illness is attributed to the chicken, and if it is the “reservoir” the illness might be attributed to the starling. The point is that the answer differs depending on at what level the question is being asked, and this underscores the need to clearly define attribution. The only steps under clear FSIS control are the slaughtering and chilling processes, but other circumstances contributed to the occurrence of disease. DEVELOPMENT AND USE OF ATTRIBUTION ESTIMATES BY THE FOOD SAFETY AND INSPECTION SERVICE FSIS has developed food-attribution estimates for Salmonella, Escherichia coli O157:H7, and Listeria monocytogenes by combining two types of data: information from CDC on reported outbreaks of food-related illness, including the products reported to have caused the illnesses; and two separate expert elicitations, one produced by FSIS (Karns et al. 2007) and the other by Resources for the Future (Hoffmann et al. 2007a). FSIS also intends to develop attribution estimates for Campylobacter, although this was not considered in the committee’s deliberations (Dreyling 2008). FSIS considers that those three data sources “are the most comprehensively available datasets for use in estimating foodborne disease attribution” (FSIS 2008c, p. A-4). The committee notes that FSIS conducted an earlier expert elicitation (Karns et al. 2005), which produced results similar to those presented in Karns et al. (2007). FSIS (2008c) and Dreyling (E. Dreyling, FSIS, unpublished material,1 January 6, 2009) provide detailed discussions of how the three data sources are combined to estimate attribution. The committee notes that CDC outbreak data used as an input to estimate attribution by FSIS (2008c) and Dreyling (E. Dreyling, FSIS, unpublished material, January 6, 2009) are not consistent, so attribution estimates based on the two documents differ.2 FSIS presented the attribution estimates derived from the two expert elicitations and the CDC outbreak data and then compared them; finding excellent agreement among the three estimates. In fact, 1 Food Disease Attribution. 2 All unpublished materials cited in this report are available through the National Academies Public Access Records Office, paro@nas.edu. 5

OCR for page 1
when attribution estimates derived from Karns et al. (2007) and Hoffmann et al. (2007a) were compared, correlation coefficients were 0.989, 0.998, and 0.998 for Salmonella, E. coli O157:H7, and L. monocytogenes, respectively (FSIS 2008c). FSIS acknowledges that part of this agreement among the estimates may result from the fact that the members of the expert elicitation may have relied on common sources of information in developing their estimates (FSIS 2008c), in particular the CDC outbreak data. FSIS combined the three data sets to develop their attribution estimates. Karns et al. (2007) provided detailed information on specific meat and poultry categories, while Hoffmann et al. (2007a) included attribution data on both FSIS- and FDA-inspected food products. The attribution estimates from the three data sets were first normalized for the same four product categories (meat, poultry, pork, and deli meats), and the average attribution estimates were calculated for Salmonella, E. coli O157:H7, and Listeria monocytogenes. These average attribution estimates were adjusted by the percent contribution of FSIS-inspected foods, and then further adjusted for the percent contribution to each of the individual FSIS food-product categories (see E. Dreyling, FSIS, unpublished material, January 6, 2009.) Attribution Estimates for Developing a Public-Health Risk-Ranking Algorithm FSIS intends to use the food-attribution estimates for two purposes: to develop a public-health risk-ranking algorithm and to develop performance objectives. (The use of attribution estimates for developing performance objectives is not in the committee’s formal charge but is discussed briefly because FSIS requested the committee to address this at the committee’s first public session [Travis 2008]). The algorithm would be used to rank establishments according to public-health risk for focusing “FSIS inspection resources on those establishments and points within slaughter and processing that can have the greatest impact on the microbial growth and contamination of products” (FSIS 2008b, p. 2). The risk-ranking algorithm consists of two factors, “public-health impact” and “indicators of process control,” that together produce a “public-health risk ranking” (FSIS 2008b, p. 11). (See Box 1 for the Conceptual Approach for the Risk-Ranking Algorithm.) FSIS considers that “public-health impact” is a measure of the effect if the hazard occurs and that “indicators of process control” are measures of the probability of the hazard. With regard to “indicators of process control,” FSIS first places a slaughtering or processing establishment into one of three “levels of inspection” (LOI 1, LOI 2, or LOI 3) on the basis of the establishment’s ability to satisfy a list of process-control indicators, such as recalls, enforcements, and BOX 1 Conceptual Approach for Risk-Ranking Algorithm Public Health Impact Indicators of Process Control Measures over time Establishment Episodic Measures (for example, verification Establishment Volume / Public Health Public Health (for example, FSAs, = testing, health based National Volume Attribution Risk Ranking recalls, enforcements) noncompliance reports) Potential number of human illnesses an Indicator of how well establishment is maintaining establishment might cause if a contamination process control event were to occur Source: Dreyling 2008. 6

OCR for page 1
verification testing. (The appropriateness of placing an establishment into LOI 1, LOI 2, or LOI 3 was examined by the Institute of Medicine Committee on Review of Use of Process Indicators in FSIS Public Health Risk-Based Inspection System and was not within the present committee’s charge.) In theory, establishments in LOI 1 are maintaining the greatest level of process control and thus pose the lowest probability of a hazard, whereas establishments in LOI 3 have the lowest level of process control, have the highest probability of a hazardous contaminant, and are subject to immediate in-depth inspection. With regard to “public-health impact,” FSIS intends to rank establishments within LOI 1 and LOI 2 to set priorities for food-safety assessments and to determine the frequency of particular inspection procedures (such as comprehensive verification). (Box 2 provides an overview of the risk-ranking algorithm and illustrates ranking of establishments in LOI1 or LOI2 based on public-health impact.) To rank the potential public-health impact of each establishment in LOI 1 or LOI 2, FSIS multiplies the attribution estimate for each pathogen-product pair by the product fractional volume (see Box 1). The product fractional volume is the volume of a particular product (such as raw ground chicken) generated by an establishment divided by the national volume of the product (that is across all establishments). FSIS (2008a) states, “if the establishment produces more than one product with the same pathogen of concern, select the maximum potential public impact” (p. 29) and goes on to state, “sort the ranked establishments into one of four pathogen categories—Salmonella, L. monocytogenes, E. coli O157:H7, Campylobacter—or place in fifth category of establishments not susceptible to any of those pathogens” (p. 29). However, data analysis provided by FSIS (E. Dreyling, FSIS, unpublished material,3 January 7, 2009) shows that FSIS sums the fractional attribution-volume estimates for an establishment across all of the pathogen-product pairs to obtain a ranking in a given LOI that is designated “public- health risk ranking.” Therefore it appears that the methods presented by FSIS in FSIS 2008a and in Dreyling (E. Dreyling, FSIS, unpublished material, January 7, 2009) may be inconsistent. Attribution Estimates for Developing Performance Objectives FSIS also intends to use public-health attribution data to develop performance objectives on the basis of CDC Healthy People 2010 goals for E. coli O157:H7 in ground beef, L. monocytogenes in ready- to-eat (RTE) products, and Salmonella in broiler chickens. A performance objective, which is calculated BOX 2 Overview of Risk-Ranking Algorithum Separate Establishments Based on Indicators of Process Control Rank LOI 1 & 2 on Potential Public Health Impact ___ ___ ___ ___ Routine Inspection ___ LOI 1 ___ ___ ___ ___ ___ Focused Inspection Establishments LOI 2 ___ ___ In-Depth Inspection LOI 3 Source: Dreyling 2008. 3 Data analysis to calculate risk ranking. 7

OCR for page 1
as the product of the FSIS foodborne-illness attribution estimate (fraction attributable to an FSIS- regulated food product) and the CDC Healthy People 2010 goal for illness (number of cases of illness per 100,000 people), estimates the proportion of the Healthy People 2010 goal for which FSIS is responsible. FSIS intends to evaluate the success of the PHRBIS system based on the performance objectives (FSIS 2008a, p. 35). STRENGTHS AND LIMITATIONS OF ATTRIBUTION METHODS Overview There are five generally recognized approaches to attributing foodborne disease (EFSA 2008):  Foodborne-disease surveillance, including pathogen-specific surveillance and outbreak investigations, from which population-level aggregated data on foodborne-disease outbreaks are analyzed to identify relationships between agents and sources.  The case-control study, an epidemiologic method designed to identify relevant risk factors, including sources of infection, for sporadic cases of disease identified through pathogen-specific surveillance. The relative importance of risk factors can be estimated by calculating population- attributable fractions.  Microbiologic subtyping, which applies strain-typing methods to microorganisms (usually pathogens) isolated from animal and food sources and from human clinical specimens. The subtyping information is often collated with results of epidemiologic surveillance programs and mathematical modeling to estimate attribution.  Quantitative microbial risk assessment (QMRA), which uses a combination of data, expert opinion, and mathematical modeling to estimate the overall probability of illness associated with a food product along the entire farm-to-fork continuum. QMRA can be used to identify the steps in production, processing, and handling of a food product that affect the foodborne-disease risk associated with consumption of the product. One component of QMRA is a comparative exposure assessment in which mathematical modeling is used to estimate exposure to all relevant foods.  Expert elicitation (also termed expert judgment), which poses a series of attribution-related questions to a set of experts. The experts are usually provided with extensive briefing material, training activities, and calibration exercises to assist in estimating attribution, along with their degree of confidence or certainty in those estimates. Each of those five approaches has strengths and limitations, which are summarized in Table 1 and discussed in more detail below in relation to FSIS’s proposed approach to attribution. Some of the important overriding themes are considered briefly here. First, foodborne-disease surveillance and microbial subtyping are supported by evidence obtained from foodborne-disease outbreak investigations. The CDC defines a foodborne-disease outbreak as an incident in which two or more persons experience a similar illness resulting from the ingestion of a common food (Lynch et al. 2006). Outbreak investigations are inherently biased toward point sources of contamination or to continuous outbreaks confined to well-defined populations, so they usually do not account for multisource or diffuse outbreaks and sporadic cases. But it is well documented that the vast majority of cases of foodborne disease are sporadic and that outbreaks caused by many infectious agents, particularly ones that cause milder illness, are not investigated (Samuel et al. 2001; Widdowson et al. 2005; CDC 2008). In addition, there are many instances in which the etiologic agent of an outbreak is never definitively identified or in which several foods (possibly multicomponent foods) are implicated. Case-control studies are useful in the identification of sources of sporadic disease, but they, too, are subject to limitations, including bias and 8

OCR for page 1
confounding. Consequently, reliance solely on outbreak data for estimating foodborne-disease attribution associated with one or more agent-vehicle combinations results in an incomplete picture. Second, foodborne-disease attribution can be estimated at various degrees of resolution with respect to food source and microbial strain. Ideally, one would attribute specific pathogens (species and serotypes or strains) to specific foods (for example, a specific package of ground beef, steak, or pot roast). In reality, because of the complex nature of food contamination, microbiologic subtyping, outbreak investigation, case-control studies, and expert elicitation rarely achieve that degree of resolution but instead categorize food into general groups (for example, beef and chicken). In some cases, the species or strain of the pathogen is not delineated. The only method of foodborne-disease attribution that is amenable to a greater degree of resolution with respect to food source is QMRA, because it can take into account detailed information from human-consumption databases. However, these methods can be subject to substantial uncertainty, and no single approach to attribution is ideal. The choice of attribution method or methods for disease attribution will depend on the specific questions to be addressed and the available data and resources. Foodborne-Disease Surveillance Background Public-health disease surveillance, an application of epidemiology, is the continuing systematic collection, analysis, interpretation, and dissemination of data on public-health events for use in reducing morbidity and mortality and hence improving public health. The “health event” in foodborne-disease surveillance is typically an individual case of disease, which may appear as an isolated event or may be aggregated as part of a foodborne-disease “outbreak.” Surveillance of foodborne diseases is conducted to control and prevent outbreaks, determine the causes of foodborne disease, monitor trends in occurrence of disease, and measure the magnitude of disease. Acquisition of foodborne-disease surveillance data can be “passive” or “active.” Passive surveillance is typically characterized by state or local medical reporting on diseased people. Active surveillance generally involves widespread screening of a target population to identify cases of illness or disease. In the United States, much of the foodborne-disease surveillance is conducted under the authority of individual states’ communicable-disease reporting laws and regulations. Responsibility for disease reporting and investigation varies by state. In the 50 states, more than 3,000 local health departments act with various degrees of autonomy. A 2007 survey of the states regarding the structure and practices of foodborne-disease surveillance programs found that gastrointestinal-disease surveillance was the responsibility of local agencies in about half the states, was centralized in a single state office in about one-fourth of the states, and was distributed among regional state offices in most of the remainder (Keene and Kanwat 2007). Individual cases of illness caused by E. coli O157:H7, Listeria, and Salmonella are reported to CDC by state epidemiologists through the National Notifiable Disease Surveillance System (NNDSS) and are summarized and analyzed from time to time in Morbidity and Mortality Weekly Report. CDC aggregates surveillance data on a national level in the National Outbreak Reporting System and provides consultation and coordination for multistate investigations of outbreaks. CDC also supports the Foodborne Diseases Active Surveillance Network (FoodNet), which, in addition to active surveillance, conducts case-control studies of sporadic illnesses and population surveys to determine background rates of diarrheal illness and exposure to various food items. Trends in the occurrence of foodborne diseases under active surveillance by FoodNet since 1996 have been important in establishing consistent methods for capturing data on foodborne illness. Those data, with the results of various passive surveillance systems, provided the basis for estimating the total number of cases of foodborne disease attributable to different microbiologic agents, as reported by Mead et al. (1999). Theoretically, such information can be used to evaluate the effects of regulatory changes made by FSIS. 9

OCR for page 1
10 TABLE 1 Summary of Approaches to Food Attribution Compiled by the Committee Approach Data Needs Advantages Limitations Examples Foodborne-disease Coordinated surveillance Usually applied to multiple foods Assumes equivalence in pathogen- Adak et al. 2005; Batz et al. 2004; surveillance systems with similar data- and pathogens; outbreak data are specific contributions of each food DeWaal et al. 2006 Based on the use of aggregate collection efforts throughout a measure of attribution at the type to disease; a challenge to data from epidemiologic specified location, region, point of consumption; able to take adequately classify investigation of outbreaks country. Takes many years to into account other routes of multicomponent foods; serious accumulate sufficient data. transmission (such as travel, gaps in databases Must contain food vehicle contact with animals); addresses a information. broad range of microbes and foods; data are collected routinely on a national basis for a large number of pathogens over many years. Case-control study Systematic review of Identification of sources of Classic weaknesses of case-control Multiple examples, but specific Analytic epidemiologic study published case-control studies, sporadic infections; classic studies, including applications to food attribution are that compares diseased (cases) case series to identify relevant strengths of case-control study misclassification, recall bias, sparse (see EFSA 2008) with nondiseased (controls) risk factors for disease; design, including ability to limited resolution for commonly persons with respect to calculation of population- explore multiple exposures consumed foods, establishing previous exposures; relative attributable to estimate (specific foods, food-preparation temporality; many cases required role of exposure is determined relative importance of practices, cross-contamination, for adequate statistical power; by comparing frequencies in different exposures travel, other risk factors) usually limited to single cases and controls microorganism rather than multiple agents; expensive Microbial subtyping Integrated, active surveillance Best suited for pathogens that are Usually applied to single Hald et al. 2004, 2007; van Pelt et Based on combination of strain of most major sources (food, clonally disseminated through pathogen; in current use, not well al. 2003 typing collated with animals); reliance on food chain; useful in suited for organisms that have epidemiologic-surveillance extensive collection of identification of original relatively unstable DNA; does not data and mathematical representative strains for reservoir, setting priorities for take into account effects of other modeling; underlying comparison; information on interventions when contamination contamination risk factors (human hypothesis is that control of amount of animal or food occurs during production handling, cross-contamination); ultimate reservoir (usually product available for assumes equivalence in relative occurring before harvest) will consumption pathogen-specific contribution of prevent human exposure each food type to disease; expensive; requires extensive libraries of isolates from a range of foods or reservoirs.

OCR for page 1
Quantitative microbial risk Information on estimates for Allows high level of detail with Serious data gaps; substantial CFSAN/FSIS/CDC 2003 (Listeria assessment parameters to use in model respect to specific food uncertainty (should be monocytogenes); Evers et al. 2008 Yields mathematically derived and uncertainty distributions commodities; theoretically, can accompanied by uncertainty estimates of risk for estimates; logic model of integrate data obtained from analysis); resource-intensive how parameters are related to national surveillance programs each other Expert elicitation More explicit, structured, Best suited for filling in data Subjective in nature; potential for Hoffmann et al. 2007a,b; Havelaar et Experts combine and weigh quantitative methods— gaps; able to combine information bias (based on respondent al. 2008 data from different sources to including well-calibrated from multiple sources (different background, personal bias) estimate attribution methods, performance-based experts with different knowledge although use of structured weighting—are being used bases) approaches decreases bias; best and are increasing resolution methodologic approaches and transparency (calibrated, highly structured) require high degree of expertise and are resource-intensive 11

OCR for page 1
empirical datasets are frequently incomplete, unreliable, or inappropriately analyzed, so it can be difficult to use the datasets alone in predicting outcomes with a given level of confidence. In such cases, expert elicitation can be used to help in quantifying risks. Expert elicitation is a formal synthesis of expert judgments. It is based on the premise that an expert or small group of experts in a discipline can provide valuable insight into interpretation and quantification of available data. Given that most decisions are complex and involve multiple aspects, obtaining input from a large, heterogeneous group of multidisciplinary experts reduces the influence of individual biases (van der Fels-Klerx et al. 2002). The results of expert elicitation must be interpreted cautiously. Havelaar et al. (2008) state that “an expert opinion may not be expected to provide an unbiased estimate of the relative importance of different transmission routes. Rather it should be regarded as a structured way to obtain consensus opinion, based on available data” (p. 9). Considerable care must be taken in collecting expert opinion. Well-defined procedures— including the identification of target, query, and performance variables; appropriate selection of experts; and a systematic combination or synthesis of expert judgment—can yield valuable information on the questions posed (Cooke and Goosens 1999). Formal (and sometimes complex) methods should be used to determine overall probabilities with classical or Bayesian approaches to data aggregation (Clemen and Winkler 1999). Whatever method is used to assess overall probabilities, it is critical that expert elicitation contains some estimate of uncertainty. Expert elicitation has been used in combination with other tools to estimate the relative importance of particular routes of exposure (for example, food, water, environmental, and human-to- human) for infectious-disease transmission in different countries (VanDuynhoven et al. 2002; Cressey and Lake 2004; Hall et al. 2005; Vaillant et al. 2005; Havelaar et al. 2007). The estimated attributable proportions of infections caused by specific pathogens in specific food sources (or other transmission vehicles) may vary considerably from country to country, depending on such factors as geographic differences in contamination frequency or magnitude, food choices and cooking habits, hygiene, environment and climate, population susceptibility, and agricultural and food production and processing practices. Therefore, although estimates obtained from other regions may demonstrate similar trends or patterns of disease transmission, it is imperative that population-specific estimates be obtained. Recently, Hoffmann (Hoffmann et al. 2007a) used structured expert elicitation to estimate the association of pathogens with specific food commodities, and Karns et al. (2007) described the results of an expert elicitation that focused on the role of meat and poultry products in foodborne illness. Those two studies present the only publicly available data on the application of expert elicitation to food-source attribution and public health in the United States. Strengths and Limitations of Expert Elicitation Using a written survey instrument, Hoffmann et al. (2007a,b, 2008) asked 42 expert panelists to provide source-attribution estimates for 11 pathogens commonly transmitted in food—Campylobacter spp., nontyphoidal Salmonella, Shigella spp., Yersinia enterocolitica, E. coli O157:H7, Vibrio spp., L. monocytogenes, Cyclospora cayetanensis, Cryptosporidium parvum, Toxoplasma gondii, and “Norwalk- like” viruses—in association with 12 foods, including one “other” category. Participants were asked to include confidence bounds around their estimates. Karns et al. (2007) explored the perceived role of meat and poultry in the transmission of four bacterial pathogens—Salmonella (nontyphoidal and multiple-drug- resistant), E. coli O157, L. monocytogenes, and thermotolerant Campylobacter spp. (C. jejuni and C. coli)—associated with foodborne disease by asking 17 panelists to rank the likelihood of disease in healthy adult consumers and vulnerable consumers (defined as “very young, the elderly, pregnant women and their fetuses, and those with compromised immune systems”) from consumption or handling of 25 processed meat and poultry products. Experts were asked to provide their estimates about the percentages of illness that would be attributed to each food category, including their degree of confidence in their 17

OCR for page 1
estimates—1 (little or no confidence) to 3 (very confident)—for each pathogen. However, different information was elicited for different pathogen-food combinations. Although these two studies are among the first to provide important information on food attribution in the U.S., there are some notable limitations. Neither Hoffmann et al. nor Karns et al. used control or “seed” variables with known distributions, which are increasingly included in expert elicitations to assess how closely the expert judgment corresponds with known parameters (Cooke and Goossnes 1999). Self-reported measures of certainty were recorded in both studies, and preliminary reports from Hoffmann et al. (2007b, 2008) describe how these data may be qualitatively analyzed to understand the magnitude of uncertainty in expert elicitation estimates; however, a full analysis of the data was not provided. Some differences between the findings of Hoffmann et al. (2007a) and Karns et al. (2007) were noted. For example, poultry was identified by both studies as responsible for the largest fraction of cases of foodborne illness, but Hoffmann et al. found that “luncheon and other meats” were considered to be a more significant source of illness than eggs, beef, or pork, whereas Karns et al. (2007) found that the experts considered the same RTE products to be the smallest contributor to foodborne illness. That may be due to differences in RTE product definitions used in the two elicitations, specifically, the presence or absence of preservatives and postlethality treatment. (Postlethality treatment includes high-pressure processing, post pasteurization, or bactericidal surface treatments to reduce the presence or limit the growth of microorganisms after processing.) Karns et al. asked participants to assume that no additives were included in RTE products to inhibit the growth of L. monocytogenes and that no postlethality treatment was applied to products—two assumptions that do not reflect current practices of many commercial processors. Although only two people served on both panels, the two studies may not constitute independent observations and are likely to be subject to common cognitive biases introduced through heuristics related to members of each group. For example, the most readily available data on foodborne-disease attribution are those reported by Mead et al. (1999), and it is likely that all panelists relied on them; this illustrates the problem of availability bias. Other potential biases may also be of concern. Anchoring bias, or focalism, is the tendency to rely on a single point or reference from which other estimates are made; participants in the Hoffmann et al. study were instructed to start their responses with a food category that they knew best, which potentially introduced anchoring bias. Representativeness bias can be introduced if people rank two products similarly because the two products themselves are somewhat similar; in the Karns et al. study, the inclusion of two similar product categories—“raw ground, comminuted, or otherwise nonintact pork” and “raw ground, or otherwise nonintact meat other than beef or pork”—may have introduced representativeness bias. In both expert-elicitation studies, the experts were asked to attribute illness to specific food commodities consumed and not to consider sources of cross-contamination. For example, someone may become ill by eating a salad that was contaminated on a cutting board that had been used to cut raw beef, but the experts were expected to consider such an illness as arising from produce, not beef (Hoffmann et al. 2007a, p. 1223). However, raw-meat products can contribute to the contamination of a variety of other foods in many ways. Interventions that reduce the likelihood and magnitude of contamination in raw meats would also reduce cross-contamination risks, but these were not considered in either expert- elicitation study. Comparisons of Expert Elicitation and Outbreak Data Descriptions of how well expert opinion correlated with outbreak data are provided by Hoffmann et al. (2008). When the attributable fractions are multiplied by correction factors to account for underreporting and the expected number of illnesses, the resulting estimates of numbers of cases of illness, hospitalizations, and deaths predicted by expert elicitation and the outbreak data sometimes agree and sometimes do not. For example, the correlation of expert elicitation and outbreak data for attribution 18

OCR for page 1
of illness to beef, poultry, luncheon meats, and pork was 0.81, 0.71, 0.55, and 0.53, respectively. However, the estimates of cases attributable to eggs did not correlate (r = 0.00). This may result from the fact that eggs are often used as an ingredient in complex dishes, and may be typically undercounted in outbreaks (Adak et al. 2005). EVALUATION OF THE PROPOSED FOOD SAFETY AND INSPECTION SERVICE APPROACH TO ATTRIBUTION AND USE OF ATTRIBUTION ESTIMATES Attribution Estimates Foodborne pathogens have different biologic and environmental niches in animals and in the environment, and they may enter or change their level in the food chain (farm-to-fork) at different points. Thus, pathogens have many potential source pathways, and this makes the attribution of human illness to foods challenging. Developing accurate estimates of attribution of foodborne illness to specific foods requires a comprehensive program that combines many methods and datasets. Harmonization and structured categorization of food items based on both the foods themselves and the combined effects of their production, processing (preservation), and handling methods are crucial for accurate source attribution, particularly when data are collected from various sources and when results from more than one source are to be compared or integrated. Scientific Criteria to Ensure Safe Food (IOM/NRC 2003) stated that “depending on the quality of available data, food safety regulatory agencies could use controlled studies, expert opinion, or a combination thereof to develop science-based food safety criteria. Because of common gaps in available data and scientific knowledge, the combination strategy is the optimal science-based procedure to develop food safety criteria” (IOM/NRC 2003, p. 7). FSIS (2008c), likewise, acknowledges that no single source of information can provide a comprehensive picture of food-source attribution and has identified several datasets that could be used to establish improved attribution estimates, including CDC outbreak data; results of FoodNet case-control studies; FSIS and FDA risk assessments; CDC, FDA, and FSIS Salmonella serotype data; and expert elicitation. It also stated that “FSIS will use these and other advances to improve foodborne disease attribution estimates as better information becomes available” (FSIS 2008c, p. A-1). Nonetheless, the proposed approach was limited to the combination of CDC outbreak data and the two expert elicitations. The committee finds that additional data are available and can be exploited. Examples are described below. However, the committee acknowledges that they struggled with evaluating attribution alone, independent of its use for public-health-risk ranking of processing and slaughtering establishments. The CDC outbreak database (the Electronic Foodborne Outbreak Reporting System; see Attachment F) can be mined further to obtain additional information on sources of contamination. This database provides information on contributing factors (contamination, survival, and amplification) that may be helpful in refining attribution estimates. For example, a produce-associated outbreak with documented cross-contamination by raw meat or poultry might be considered an outbreak attributed to an FSIS-regulated product. It may be possible to investigate contributing factors by using the additional data in the CDC outbreak database and to prevent misattribution while improving the accuracy of current attribution estimates. To that end, the committee finds that additional use of the CDC outbreak data for food-attribution modeling may provide further incentive to state and local agencies to report their outbreak data more accurately and quickly for use by CDC. Currently foodborne-illness information available to local and state public-health agencies might not be included in the national CDC outbreak database, because reports from state health departments on some illnesses might not be received or because the illnesses do not meet CDC criteria for foodborne outbreaks due to specific pathogens (Batz et al. 2005). CDC outbreak data do not include sporadic cases of foodborne illness. Consequently, pathogens that rarely cause outbreaks are underrepresented. A possible solution for that limitation is to use data 19

OCR for page 1
from population-based FoodNet case-control studies to refine food-attribution estimates so that they reflect the effects of sporadic illness better. FoodNet has conducted at least 15 population-based case- control studies of a variety of pathogens, including Salmonella, E. coli O157:H7, Campylobacter, and L. monocytogenes (CDC 2006b). Although the studies were not designed specifically for source attribution, but rather to examine risk factors, they should be relevant in refining source-attribution estimates. In addition to individual case-control studies, systematic review of published studies of sporadic foodborne illness could provide information on population-attributable fractions for each exposure. Such information would be valuable in estimating the relative burden of illness caused by a pathogen attributed to multiple exposures. Quantitative data on exposure to pathogens from a multitude of sources may be obtained by application of risk-assessment methods, and FSIS and FDA have conducted several QMRAs. Those studies have focused on single food products or processes and do not necessarily provide attribution estimates, but some of the resulting data—for example, related to E. coli O157:H7 in ground beef (FSIS 2001) or L. monocytogenes in delicatessen meat (CFSAN/FSIS/CDC 2003)—may be relevant to food attribution inasmuch as the QMRAs provided data on how many human cases were linked to these foods. Outbreak data yield information for attribution at “point of consumption” because outbreak investigations link illness to the food that was consumed. Attribution at that point is necessarily the sum of the effects of contamination at pre-harvest, processing, and final preparation in the kitchen. By relying on outbreak data, FSIS includes all of the chain, even though they are only responsible for part of it. Therefore, the use of outbreak data in this capacity does not translate directly into the attribution at the point of processing. Another major consideration is that, unlike the work of Evers et al. (2008), who estimated that foodborne sources accounted for less than 33% of all Campylobacter exposures, non- foodborne sources of infections (such as travel-related cases, and “unknown” sources) are not always fully considered in the disease attribution estimates. Taken together, the current FSIS approach to attribution may under or over-estimate the proportion of illnesses attributable to specific FSIS-inspected foods. Algorithm for Public-Health Risk Ranking The committee acknowledges FSIS’s efforts to develop a public-health risk-ranking method for meat and poultry establishments. However, FSIS should provide greater justification and clear documentation to explain fully the rationale for the ranking method and to enable an independent reviewer to duplicate the risk-ranking calculations. Although the committee was tasked with examining only the “public-health impact” calculation that feeds into the public-health risk ranking, it considers that because the “indicators of process control” and the LOI categorization are intimately linked with the “public-health impact” (the product of attribution and fractional volume), it is difficult to evaluate the effectiveness of ranking by attribution and fractional volume without also evaluating the LOI categorization. The product of attribution and fractional volume does not incorporate the totality of the public-health impact of the equation; rather, much of the impact is captured in the LOI categorization. The documentation provided to the committee by FSIS did not explain why and how the proposed approach for ranking establishments was selected from among other potential methods for risk ranking. The committee is aware of other risk-ranking approaches, such as those of Ross and Sumner (2002) and Batz et al. (2004). For example, one alternative means of conducting risk-based inspection would be to rank all facilities according to public-health risk without prior categorization by LOI. The committee is concerned that FSIS provided no estimates of uncertainty in the public-health attribution estimates and the public-health risk ranking. Rather, the attribution fractional-volume estimates were calculated to five decimal places; they inappropriately reflected substantially greater precision than the data warrant. For example, FSIS reported that the total fractional attribution of the 20

OCR for page 1
plant in LOI 2 with the highest rank is 0.02595; however, the uncertainty associated with this estimate was never given (E. Dreyling, FSIS, unpublished material,4 January 7, 2009). Box 3 illustrates the importance of conducting a sensitivity analysis of the attribution estimates to explore the effect of the uncertainty in the attribution estimates on the stability and reliability of the public-health rankings. As described in Box 3, a relatively small change in an attribution estimate can exert widespread change on the public-health rankings of establishments. An understanding of the uncertainty can help inform the level of “comfort” with the results and provide information on the plausible alternative values that fall within the bounds of the uncertainty. BOX 3 Illustration of Change in Public-Health Risk Ranking Based on Change in Attribution Estimate of E. coli O157:H7 from 35.9% to 39.2% for Ground Beef and from 8.7% to 9.5% for Other Raw Ground Meat, for Top 50 Plants in Each Category Change in All Establishments LOI 2 Public-Health Risk Ranking Number Percent Number Percent +4 20 40 1 2 +3 3 6 3 6 +2 1 2 5 10 +1 0 0 9 18 No change 18 36 12 24 -1 3 6 13 26 -2 1 2 3 6 -3 0 0 2 4 -5 0 0 1 2 -6 0 0 1 2 > -20 4 8 0 0 Total 50 100 50 100 Source: E. Dreyling, FSIS, Data Analysis to Calculate Risk Ranking, unpublished material, December 31, 2008, and January 7, 2009. FSIS provided the committee with two versions of its public-health risk ranking of slaughtering and processing establishments (E. Dreyling, FSIS, Data Analysis to Calculate Risk Ranking, unpublished material, December 31, 2008, and January 7, 2009). The fraction of E. coli O157:H7 cases attributable to specific FSIS product categories was revised in the January 7 version; the greatest changes were in ground beef and other raw ground meat. The public-health risk ranking of establishments changed with the change in attribution estimates. In the ranking of all establishments, 36% of the 50 top-ranked establishments retained their original ranking, 40% moved up four places, and four dropped more than 20 places. Only 24% of LOI 2 plants retained their original ranking, and two dropped five or more places. This example demonstrates that relatively small changes in attribution estimates can exert widespread changes in the public-health risk rankings of establishments. Because the attribution estimates contain considerable uncertainty, systematic sensitivity analyses are needed to explore the effect of that uncertainty on the stability and reliability of the public-health risk rankings. 4 Data analysis to calculate risk ranking. 21

OCR for page 1
The committee is also concerned that the proposed risk-ranking method may not be appropriate for ranking establishments and assigning resources across plants, and further validation of the model is needed. FSIS considers that production volume is important in estimating public-health attribution—the larger the production volume, the higher the potential public-health impact. For each plant, FSIS V V calculates the potential public health impact as, PPHI i  i  PHA , where i is the fraction of the total V V national volume of product produced by a specified establishment (i), and PHA is the public health attribution estimate for the specified product-pathogen pair (for example, Salmonella in raw poultry). FSIS estimated the public-health attribution as the fraction of illnesses attributed to all FSIS-regulated establishments for the corresponding pathogen-product pair. Fractional volume serves as a proxy for the fraction or proportion of consumers exposed to a given product with the assumption that contamination is randomly dispersed throughout the supply of the product independent of the establishment. The efficacy of the algorithm requires the assumption that the probability of illness, given consumption of contaminated product, does not depend on the product source (the establishment) or the specific product. This is implicitly addressed by categorizing by LOI. Therefore, for those establishments producing the same product, the ranking related to a specific V pathogen reflects differences in the fractional volume ( i ) of product produced. Comparisons across V product-pathogen pairs depends critically on the relative differences in the two components (public-health attribution and fractional volume) of the algorithm. Box 4 provides an illustration of how the fractional volume and pathogen attribution estimates interact using data from FSIS (E. Dreyling, FSIS, unpublished material, January 7, 2009) for a specific establishment (plant #2). The three largest public-health attribution estimates are for L. monocytogenes in RTE products (59%), E. coli O157:H7 in ground beef (39.2%) and E. coli O157:H7 in other ground meat (9.5%). Box 4 indicates that the corresponding fractional volumes are 0.002, 0.006, and 0.142, while the potential public health impacts are 0.0012, 0.0024, and 0.0135, respectively. Box 4 shows that for plant #2, the relative importance of the attribution estimates is the inverse of the overall public-health impact for the three pathogen-product pairs. This reflects the need for FSIS to validate the algorithm’s ability to capture the potential public health impact for each plant and assign resources accordingly. The validity of the algorithm depends on an evaluation of the assumptions in the model, which includes an evaluation of the LOI categorization and how it interacts with the potential public health impact. BOX 4 llustration of Interaction of Fractional Volume and Pathogen Attribution Estimates in Estimating Public Health Impact for Plant Number 2 RTE Ground Beef Other Ground Meat Salm. E. Coli Listeria Salm. E. Coli Listeria Salm. E. Coli Listeria Fractional Volume (V2/V) 0.002 0.002 0.002 0.006 0.006 0.006 0.142 0.142 0.142 Attribution Estimate (PHA) 0.022 0.031 0.589 0.039 0.392 0.003 0.013 0.095 0.001 Potential Impact (PPHI2) 0.00004 0.00006 0.0012 0.00023 0.0024 0.00002 0.0018 0.0135 0.00014 Source: E. Dreyling, FSIS, Data Analysis to Calculate Risk Ranking, unpublished material, January 7, 2009. 22

OCR for page 1
In addition, FSIS currently does not consider the severity of illness associated with a given pathogen in estimating attribution and public-health risk-ranking of establishments. Nonetheless, disease severity is an important consideration in many other risk-ranking methods, and its impact can be characterized using measures such as Disability-Adjusted Life Years (DALYs), Quality-Adjusted Life Years (QALYs) and willingness-to-pay (WTP), among others. These integrated measures of disease burden are detailed elsewhere (Batz 2007; Havelaar et al. 2007). It is likely that the rankings of establishments would differ if disease severity were considered in calculating them. Overall, the committee considers that severity of disease is an important component as FSIS seeks to measure “public health impact.” The committee is also concerned that FSIS needs to adequately evaluate and consider the rationale and implications of its risk-ranking algorithm. FSIS did not provide the committee with the algorithm and supporting data until well into its last meeting, and FSIS had to resubmit the algorithm to the committee afterward because its original submission contained calculation errors and was not adequately documented. It is important that FSIS present to the public a well-documented and transparent model to illustrate the algorithm. The algorithm (model) should be updated as new data and methods are developed. FSIS (2008a, p. 15) “recognizes that development of a health-based inspection model will be an ongoing process, and that the proposed algorithm may continue to evolve as more information about the risks associated with particular products and about the predictive indicators of food safety process controls at processing and slaughter establishments becomes available.” Performance Objectives The committee recognizes the importance of performance objectives as FSIS seeks to continuously evaluate its efforts in reducing foodborne illness and improving public health. However, the committee is concerned that the method that FSIS used to calculate performance objectives in the current report (FSIS 2008c) needs to be better documented, transparent, and readily reproducible and that the validity of such measures is hampered by limitations in attribution estimates (discussed above). In addition, FSIS does not present any uncertainty bounds surrounding its performance-objective estimates. It is important for performance objectives to be responsive to changes in attribution estimates that result from natural variations in disease incidence. Natural variations in disease incidence may result from changes in nonfood transmission sources and in food production, processing, and preparation practices, among others. The current attribution estimates remain constant when projecting performance objectives. FSIS (2008a) states that “prior to implementation of the proposed PHRBIS system, FSIS will develop its evaluation plan. The plan will include the types of outcome analyses to be conducted. The results of those analyses will be used to refine the PHRBIS” (FSIS 2008a, p. 34). The committee concludes that an integral part of the PBHRBIS should be an evaluation plan that contains specific metrics with which to evaluate the reduction in foodborne illness and with which to judge the success of the system. The data sources used to evaluate the metrics should be independent of those generated by FSIS. An example of such an approach could be one that uses FoodNet, PulseNet, and outbreak data to evaluate whether FSIS’s risk ranking is resulting in measurable improvements to public health. Use of Salmonella Serotype Data for Estimating Attribution Comprehensive Salmonella surveillance—including characterization of human, food, animal, and environmental isolates with a combination of serotyping and molecular subtyping methods—is critical for improving our understanding of Salmonella transmission and facilitating, in the long term, science-based approaches to Salmonella attribution. In particular, population and commodity-specific subtype data are required for the development of mathematical models for attribution of Salmonella to either different reservoirs (for example specific animal-host species) or possibly to specific commodities. FSIS has 23

OCR for page 1
adapted one subtype-based model developed in Denmark (the “Danish model”) to attribute public-health risk to different FSIS-regulated food sources. The committee applauds that effort to innovate; however, the committee is concerned about the ability of this particular approach to provide appropriate attribution to different FSIS-regulated foods. Specifically, there is limited biologic justification for associating Salmonella serotypes and different food items (except for Salmonella Enteritidis PT4, which is associated with transmission via intact eggs) other than their well-documented association with specific host species. Therefore, mathematical source attribution estimates based on subtype data for food isolates will largely reflect the underlying biologic association of subtypes with specific (animal) reservoirs and, as transmission of a specific host-associated serotype to humans can be achieved by different routes, this may prove unreliable for food-group attribution. For example, a particular serotype could be transmitted by consumption of contaminated meat or poultry products, by contamination of fruits and vegetables by manure runoff, or as a consequence of direct contact between humans and reservoir hosts. Subtype-based attribution efforts, such as the Danish model, also require large comprehensive data sets on Salmonella subtypes associated with different reservoirs and food commodities. Thus, application of subtype-based attribution to only a fraction of the food supply (FSIS-regulated foods) increases the chances of inaccurate attribution estimates. In addition, association of specific serotypes with certain reservoirs and foods may change over time; for example, while Salmonella Enteritidis was overwhelmingly associated with egg and egg products when it initially emerged, prevalence of this serotype in poultry products appears to have increased over time. Subtype-based attribution estimates thus need to be updated regularly to be accurate and useful. Finally, the Danish model used both serotype data and phage-typing data for a subset of serotypes, while the proposed FSIS approaches only use serotype data, thus relying on more limited subtype discrimination, particularly among the common Salmonella serotypes (for example, Typhimurium). Use of the modified Danish model to allow for appropriate subtype-based source-attribution estimates for the US thus will require additional data (for example, subtype data in addition to serotyping data), validation, and probably further improvements in subtype characterization to define clonal groups that may be associated with specific food types. On a broader scale, the data will also be critical for completion of an all-encompassing subtype- based Salmonella attribution effort for all foods and non-food-associated transmission pathways (for example, direct zoonotic transmission). The committee appreciates that the PulseNet database already provides a comprehensive subtype data set for human clinical Salmonella isolates but believes that further efforts are needed to enhance coverage of animal, food, and environmental Salmonella isolates in molecular typing databases, including design of data collection and analysis systems that are consistent among agencies. Other federal agencies (for example, the Animal Plant Health Inspection Service and FDA) and state agencies should be involved and integrated into those efforts. While FSIS appears to attempt an attribution to food source rather than reservoir, these collaborations are needed to provide serotype and PFGE data for Salmonella isolates associated with a variety of sources. These collaborations will be critical for accurate reservoir attribution, which, as detailed above, may be more feasible than food-source attribution. The committee also concludes that there are opportunities for better integration of FoodNet and PulseNet data in conjunction with subtype data to facilitate estimation of population-based attribution of sporadic cases to specific agents. For example, combining serotype (from the Public Health Laboratory Information System) and PFGE data (from PulseNet) on isolates from sporadic cases can facilitate estimation of population-based attribution of sporadic cases to specific agents; this would be similar to what has been done in the Danish model (Hald et al. 2004). In addition, comparing data from sporadic cases to that obtained from outbreaks should provide a better understanding of attribution differences between outbreak and sporadic cases, if such differences exist. While collection of subtype data for other foodborne-disease agents will provide for improved outbreak detection, subtype-based attribution may not necessarily be possible for other pathogens, particularly if there is no biologic basis for associations between pathogen subtypes and specific reservoirs and/or foods. While there are concerns whether the Danish model, which is a “reservoir” attribution effort, can be used as a model for “food-source attribution,” as proposed by FSIS, continued efforts using improved 24

OCR for page 1
subtyping methods and data sets may yield insights into the feasibility and reliability of subtype-based food-source attribution. However, it is important to recognize that even with improved subtype data for Salmonella strains associated with different food and animal sources, reliable food-source attribution may not be feasible, and instead only point-of- reservoir attribution will be possible. Consideration of Analysis in a Broader Context The foodborne-disease attribution model put forth by FSIS is to be used to categorize slaughtering and processing establishments with respect to their potential effects on public health. The purpose is to improve and facilitate priority-setting and resource allocation with respect to FSIS-inspected products that are the major contributors to human-foodborne illness. FSIS intends to use this approach, at least initially, for priority-setting among particularly risky pathogen-commodity pairs or promising mitigation strategies. It is expected that the approach will also be used to inform the development of new microbiologic criteria and related public-health goals and to document the efficacy of control strategies in reducing the overall foodborne-disease burden and improving public health. Indeed, as policy-makers move toward promulgating food-safety regulations in the framework of risk assessment, they are increasingly faced with the challenge of linking public-health goals with scientifically valid criteria, such as performance standards. However, there remain questions about how to achieve this. For instance, Can a performance standard be designed to fit a stated public-heath goal? How will the effectiveness of a regulation be measured in terms of specific public-health outcomes? Food attribution plays an important role in answering such questions. The committee considers that a more integrated approach to evaluating FSIS’s risk-based inspection system would be more effective. For example, the committee was tasked specifically with addressing attribution for the purposes of ranking establishments, and another committee addressed only process-control indicators. Understanding how these two components influence one another is vital in addressing the primary question about FSIS’s risk-based inspection system: Is it leading to improved public health? That is, does the risk-based inspection system enhance FSIS’s ability to detect and respond to food-safety hazards? Does it ensure that establishments that pose a risk to public health receive more attention? These are the questions that should be addressed to ensure that FSIS’s risk-based inspection system is enhancing public-health protection. CONCLUSIONS  The committee applauds FSIS’s efforts to develop a Public Health Risk-Based Inspection System; however, FSIS should present a more transparent algorithm to rank slaughtering and processing establishments according to public-health risk. Despite considerable effort, the committee had great difficulty in understanding the rationale behind the proposed approach and in precisely reproducing FSIS’s calculations because of a lack of transparency in the model. In addition, failure to characterize the uncertainty in the attribution estimates and other inputs of the risk-ranking algorithm is a critical weakness in the proposed PHRBIS.  The precision implied in FSIS’s public-health risk ranking, produced in part by using attribution estimates and production volume, appears to be quite low. Because FSIS estimates public-health effects on the basis of a small number of observations, the estimates have large uncertainties that should be communicated in the ranking algorithm. FSIS should also recognize that attribution estimates will need to be updated as disease incidence in humans changes to retain their relevance when used for risk-based inspection.  The data sources currently available for assessing attribution are insufficient to be used independently. FSIS has not used some data that are readily available to supplement the CDC outbreak data and expert elicitations. This could help in the development of better-informed attribution estimates. 25

OCR for page 1
 In the proposed public-health risk-based ranking algorithm, FSIS’s method of categorizing facilities on the basis of their LOI (indicators of process control) before incorporating public-health attribution (public-health effect) ranks facilities according to inspection-based risk (for example, recalls, enforcements, and verification testing). This ranking may not reflect public-health risk. In the current system, attribution has little influence on an establishment’s rank, inasmuch as rank is determined primarily by LOI categorization, which pinpoints hotspots, and the system then ranks establishments with a given LOI primarily according to the product of attribution and fractional volume. It is unclear how the public-health effect component of the algorithm will improve the ability to set priorities among high-risk facilities.  Attribution estimates based on outbreak data which reflect disease occurring at the “point of consumption” do not directly translate to attribution at the point of slaughter and processing. In fact, other points along the farm-to-fork continuum that are outside of FSIS’s jurisdiction (for example, the farm and the end-user) contribute substantially to disease associated with FSIS-regulated products. Because the risk-ranking algorithm does not explicitly consider the contribution of non-regulated attribution sources to FSIS-regulated products, it can under or over-estimate the proportion of illnesses actually attributable to slaughter and processing. This oversight may result in inappropriate risk-based allocation of resources.  The development of performance measures is premature, given the limitations of the attribution estimates and the lack of uncertainty characterization. That is of particular concern because imprecise estimates of attribution are being used to support specific performance objectives, and the proposed system may not reflect the changing or uncertain nature of the attribution estimates.  Salmonella serotyping and molecular subtyping not only will be critical for improved attribution efforts but will enhance the agency’s ability to monitor pathogen trends, such as emergence of new subtypes. Salmonella serotype-based and subtype-based attribution models are not yet at a stage where they should be used for policy decision-making. RECOMMENDATIONS  Recognizing that it is difficult to estimate food attribution given the small amount of available data and its relatively poor quality, FSIS should consider alternative prioritization methods for their PHRBIS. This might include ranking methods that do not rely on attribution data per se or risk-ranking models that approach the attribution problem in an alternative manner.  Once FSIS has selected a means of ranking, it should provide transparent documentation that describes the primary data used in the risk-ranking calculations; step-by-step instructions on how to perform the calculations, with examples; characterization of uncertainty in the data; sensitivity analysis of the risk-ranking algorithm; and strengths, limitations, and clear justification of the approach selected. To the extent practicable, the risk ranking should consider the importance of differences in disease severity associated with different pathogens. Documentation should be provided to allow interested stakeholders to reconstruct FSIS’s approach.  FSIS should state that it will update the risk-ranking algorithm and reevaluate the PHRBIS every 1-3 years, and the agency should specify how this will be done. The periodic evaluations should use newly available data and methods (for example, methods for risk ranking used by other regulatory agencies worldwide) and should evaluate model inputs and the model itself. A main focus of the regular evaluations should be to ensure that the dynamic nature of attribution is factored into the model. In addition, FSIS should articulate the metrics that it will use to demonstrate public-health outcomes; the metrics should be evaluated by using data sources that are independent of those generated by USDA.  If FSIS continues to include attribution as a component in its PHRBIS, FSIS in conjunction with CDC staff and others, should review the CDC outbreak database, including information not considered in the initial FSIS attribution model, to improve attribution of illnesses to regulated food products. Routine 26

OCR for page 1
use of the CDC outbreak data for purposes of food-attribution modeling may provide further incentives to state and local jurisdictions to report outbreaks accurately and quickly for use by CDC.  If FSIS continues to include attribution as a component in its PHRBIS, FSIS staff should work collaboratively with FoodNet and PulseNet staff to use sporadic-case and outbreak data in conjunction with subtype data more effectively to facilitate estimation of population-based attribution of sporadic cases to specific agents.  Recognizing that food-attribution data are of interest to many agencies, FSIS should work collaboratively with CDC, FDA, and other federal and state agencies to develop a common set of definitions for microbial foodborne-disease attribution; a coordinated approach to improve the quality and consistency of data used among agencies in determining food-attribution estimates; a process that allows for regular updating of attribution estimates; and a standardized coding scheme for food vehicles, including multi-component foods.  FSIS should continue to collaborate with CDC and other appropriate organizations in the serotyping and molecular subtyping of all Salmonella isolates, with emphasis on those obtained from specific food products. To the extent feasible, subtype data should also be collected for isolates from environmental samples and other sources of human exposure to Salmonella (for example, reptiles and pets). Recognizing that Salmonella serotyping and molecular subtyping will not only be critical for improved subtype-based attribution efforts, but will also enhance the agency’s ability to monitor pathogen trends (for example, emergence of new subtypes), FSIS should try to include serotyping and/or molecular subtyping in all of its future baseline studies. As part of these efforts, FSIS should establish and support collaborative arrangements with FDA to assure that Salmonella isolates obtained by USDA or FDA are characterized using the same molecular subtyping approaches and that results are available in a comprehensive database with harmonized nomenclature of human, animal, food, and environmental Salmonella isolates. In the future, it may be appropriate to expand such studies to other pathogens.  FSIS should continue to support the collection of serotype and molecular subtype data for Salmonella and perhaps other relevant pathogens, and the development of mathematical models that use these serotype and subtype data for understanding food (and source) attribution of human Salmonella infections. These efforts need to include research on developing new models, evaluating and validating existing models, and developing better quality data to populate the models. Attachments: A – Statement of Task B – Committee Membership C – References D – Acknowledgment of Reviewers E – Public Agendas for Meetings F – Electronic Foodborne Outbreak Reporting System G – Foodborne Disease Attribution 27