4
Framework for Evaluating Tools to Assess Dietary Risk

During early deliberations, the Committee on Dietary Risk Assessment in the WIC Program developed a framework for evaluating methods to assess dietary risk in WIC program applicants. The committee’s overall goal was to identify an assessment tool that could determine whether individuals did or did not meet the Dietary Guidelines for Americans (USDA/HHS, 2000) or more specifically (as discussed in Chapter 3) Let the Pyramid guide your food choices. It also considered the potential of tools to identify nutrient intakes in relation to cut-off points since diet adequacy is another allowed type of criterion (see Appendix A). This chapter outlines eight characteristics that together provide a framework for evaluating the usefulness and effectiveness of a dietary risk assessment tool in the WIC setting. Based on further deliberations, this framework has been modified slightly from that presented in the committee’s interim report (IOM, 2000c).

DESIRABLE CHARACTERISTICS OF AN ASSESSMENT TOOL

1. The Tools Should Identify Dietary Risks that are Related to Health or Disease

Ideally, any risk criterion adopted for dietary risk should be both predictive of the individual’s risk of health problems as well as indicative of nutrition and health benefit from program participation. When considering health outcomes for children, appropriate growth and development are key facets of health. Diet



The National Academies | 500 Fifth St. N.W. | Washington, D.C. 20001
Copyright © National Academy of Sciences. All rights reserved.
Terms of Use and Privacy Statement



Below are the first 10 and last 10 pages of uncorrected machine-read text (when available) of this chapter, followed by the top 30 algorithmically extracted key phrases from the chapter as a whole.
Intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text on the opening pages of each chapter. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.

Do not use for reproduction, copying, pasting, or reading; exclusively for search engines.

OCR for page 49
Dietary Risk Assessment in the WIC Program 4 Framework for Evaluating Tools to Assess Dietary Risk During early deliberations, the Committee on Dietary Risk Assessment in the WIC Program developed a framework for evaluating methods to assess dietary risk in WIC program applicants. The committee’s overall goal was to identify an assessment tool that could determine whether individuals did or did not meet the Dietary Guidelines for Americans (USDA/HHS, 2000) or more specifically (as discussed in Chapter 3) Let the Pyramid guide your food choices. It also considered the potential of tools to identify nutrient intakes in relation to cut-off points since diet adequacy is another allowed type of criterion (see Appendix A). This chapter outlines eight characteristics that together provide a framework for evaluating the usefulness and effectiveness of a dietary risk assessment tool in the WIC setting. Based on further deliberations, this framework has been modified slightly from that presented in the committee’s interim report (IOM, 2000c). DESIRABLE CHARACTERISTICS OF AN ASSESSMENT TOOL 1. The Tools Should Identify Dietary Risks that are Related to Health or Disease Ideally, any risk criterion adopted for dietary risk should be both predictive of the individual’s risk of health problems as well as indicative of nutrition and health benefit from program participation. When considering health outcomes for children, appropriate growth and development are key facets of health. Diet

OCR for page 49
Dietary Risk Assessment in the WIC Program has been shown to have both short- and long-term effects on behavior, cognitive development, physical growth, and general health status (Levitsky and Strupp, 1995; Pollitt, 1988). Inadequate energy intake in early life may be directly linked to poor outcomes in cognitive function, such as learning, or nutritional status in childhood (Gorman, 1995). Infants, preschool, and school-age children who are iron-deficient show deficits in mental development, attention, and learning, as well as in achievement test scores, when compared to iron-replete children (CDC, 1998b). In addition, essential fatty acids are necessary for proper brain development (Uauy et al., 2000). For children ages 2 to 5 years and pregnant or postpartum women, the 1996 IOM WIC report suggested using the indicator failure to meet Dietary Guidelines (IOM, 1996). As discussed in Chapter 3, this would involve using the updated consensus document, Dietary Guidelines for Americans (USDA/HHS, 2000), as a reference, specifically as related to the two guidelines, Let the Pyramid guide your food choices and Be physically active each day. In screening situations, one is assessing how an individual’s dietary intake compares with an appropriate cut-off point based on the Dietary Guidelines. The purpose is to conclude whether the individual “meets” or “does not meet” the Dietary Guidelines. For such vulnerable populations as pregnant women, postpartum women, and children ages 2 to 5 years, the committee decided that many criteria could be set, any one of which would provide evidence that the individual fails to meet either the Food Pyramid guideline or the physical activity guideline (see Chapters 3, 5, and 6). 2. The Tools are Appropriate for Age and Physiological Condition Several subgroups are served in the WIC program: pregnant, breastfeeding, and nonbreastfeeding women, along with their infants and children younger than 5 years of age. When assessing dietary risk, consideration needs to be given to the specific nutritional recommendations and appropriate dietary patterns for these groups. For example, if a tool were to assess whether a client consumes the recommended number of servings of fruits and vegetables, it would need to be designed to accommodate different recommendations for adult women as compared to young children and differences in common food choices by these two groups. A second consideration relates to the method of administration of the tool for assessing dietary risk. For example, young children cannot report their dietary intake, and proxy (parental) reports must be used. Hence, one must evaluate the suitability of tools for each client subgroup.

OCR for page 49
Dietary Risk Assessment in the WIC Program 3. The Tools Should Ideally Serve Three Purposes: Screening for Eligibility, Individualizing the Food Package, and Nutrition Education for Behavior Change As discussed in Chapter 2, dietary assessment tools are utilized for three reasons in the WIC setting: to define dietary risk as a criterion for eligibility in the WIC program, to identify eating patterns that influence the type of supplemental food package provided by the program, and as the starting point for nutrition education and counseling efforts. Ideally, one tool could be used for these purposes; however, the committee recognized that the standards for the effectiveness of a tool for screening would likely be higher than those required if the tool is used primarily for education. In the latter case, less well-performing tools would still have utility for education purposes if they provided the WIC nutrition professional with sufficient background on the individual’s food choices from which to begin a dialogue regarding dietary change. 4. The Tools Should have Acceptable Performance Characteristics All instruments should be evaluated prior to use to ensure that they perform adequately. Performance is assessed in quantitative terms by considering the validity and reliability of the instrument (Windsor et al., 1994), and related constructs that are defined in Table 4-1. Validity addresses whether one is really measuring what was intended. For example, 24-hour dietary recalls are intended to measure dietary intake for the previous 24-hour period, but several recent studies have revealed that as much as 30 percent of foods reported by children were not eaten the previous day (Baxter et al., 1997). Foods reported but not eaten are called intrusions or phantom foods (Domel et al., 1994). A method that systematically under- or overestimates consumption leads to biased estimates and is therefore not considered valid. Reliability relates to whether applying the same instrument two or more times provides the same results (Table 4-1). Reliability thereby indicates the degree of random error in the dietary assessment method. Random error could be caused by such conditions as the respondent or interviewer being upset at the time of assessment, multiple interviewers, excessive noise during assessment, the limitations of memory, or a person’s inability to properly average intake to provide a desired response on a food frequency questionnaire. Random error is always present; therefore, the question when evaluating a tool is whether the level of random error present is acceptable for the intended purpose. Chapter 5 provides further information about error in dietary data collection and the performance of different types of data collection tools used to assess diet. Error in the assessment of the dietary intake of an individual leads to misclassification in the determination of eligibility for the WIC program. Misclassification has serious consequences in that some truly eligible individuals

OCR for page 49
Dietary Risk Assessment in the WIC Program TABLE 4-1 Terms Used When Describing or Evaluating the Performance Characteristics of an Assessment Tool Term Definitions Validity Does the method measure what it is supposed to measure? Is the method accurate? That is, does it provide an unbiased estimate of usual dietary intake? Bias Also known as systematic error If biased, the estimated mean intake is not equal to the true mean intake Reliability Refers to the ability of the estimate to be reproduced when the measure is repeated The inability of the measure to be reproduced is a function of the amount of random error in the assessment procedure Reproducibility See Reliability Random error Variability in the measure when assessed over time Increases the variance around the mean of the measure, but does not affect the estimate of the mean Random error is inversely related to reliability Between-individual variability Variability across individuals in their usual dietary intakes Considered the true variability when estimating intakes of groups Within-individual variability Variability in dietary intakes within an individual from day to day Reduces the reliability of the measurement of usual intake Measurement error or imprecision Refers to error in dietary intake estimation due to the measurement process itself Includes interviewer differences, food composition database errors Reduces the reliability and validity of the measurement of usual intake Misclassification Quantification of error within the context of classifying individuals as being at dietary risk Quantified in terms of Sensitivity and Specificity of the measure Sensitivity Refers to the proportion or percent of individuals with dietary risk who are identified by the assessment tool as being at dietary risk Specificity Refers to the proportion or percent of individuals not at dietary risk who are identified by the assessment tool as not being at dietary risk Positive predictive value Refers to the proportion or percent of individuals identified at dietary risk who are truly at dietary risk Negative predictive value Refers to the proportion of individuals identified to not be at dietary risk who are truly not at dietary risk

OCR for page 49
Dietary Risk Assessment in the WIC Program may not be classified as eligible for the services (less than perfect sensitivity), or individuals not truly eligible for the services may receive them (less than perfect specificity). Chapter 5 provides examples of the effects of less than perfect sensitivity and specificity. In the absence of perfect tools (tools with 100 percent sensitivity and 100 percent specificity), policymakers and the public must decide how much and what type of misclassification error they are willing to tolerate when certifying people to receive or not receive federally funded WIC services. It is the view of the committee that less than perfect specificity should be tolerated in order to achieve perfect sensitivity; in other words, to ensure that all truly eligible individuals are identified as eligible with existing assessment tools, it is acceptable that some truly noneligible individuals receive WIC services. 5. The Tools Should be Suitable for the Culture and Language of the Population Served The WIC program serves a multiethnic, multicultural, heterogeneous population. Thirty-nine percent of WIC participants are Caucasian, 33 percent are Latino, 23 percent are African American, 3 percent are Asian or Pacific Islander, and 2 percent are American Indian or Alaskan Native (Bartlett et al., 2000). The percentages of non-Caucasians and the diversity of cultures are expected to increase. Diversity in heritage, geography, food consumed, and culture translates into diversity in dietary patterns and practices. To assess dietary intake and patterns effectively, dietary assessment tools would need to be developed with each specific culture in mind. Thus, many WIC agencies would require several dietary assessment tools to serve their population mix. Language translation alone would not provide an acceptable tool for a different culture because the types of foods consumed, the portion sizes, food combinations, and the way foods and eating are conceptualized are likely to differ. It is true that standardized 24-hour recalls and food records capture cultural preferences and foods consumed, provided that the interviewer is knowledgeable about reported foods, follows standard methods, and uses a food composition database that includes the foods. Thus, the need to consider specific development of tools for different cultural groups refers to the use of food frequency methods to determine usual dietary intakes. It was also recognized that effective administration of tools to different cultural groups would likely require special training and that little information exists to document successful adaptations of dietary assessment instruments for use in different cultures whose members wish to use WIC services.

OCR for page 49
Dietary Risk Assessment in the WIC Program 6. The Tools Should be Responsive to Operational Constraints in the WIC Setting Time constraints for both staff and participants necessitate the use of an assessment tool that can be administered, scored, and interpreted rapidly. It is imperative that the tools under consideration take into account the variety of skills and knowledge levels of the competent professional authorities (CPAs) who assess dietary intake in the WIC setting. Whether CPAs are paraprofessional or professional, the assessment and educational tools they use need to be linguistically and culturally appropriate for different population groups served by WIC clinics. A tool should provide consistent results regardless of the staff member who administers it. Subjective measures in scoring should be avoided to eliminate administrator bias. Furthermore, the tool should be constructed in a manner so as not to influence the client. Features that may influence responses inappropriately include scoring mechanisms placed directly on a self-administered form and phrasing that invites desirable or favorable responses rather than accurate ones. Additional points that need to be considered include the impact of the tools on the systems used by the WIC agency, and expected future changes to the system, such as automation or computerization. 7. The Tools Should be Standardized Across States and Agencies To some degree, tools used to determine eligibility for WIC participation based on dietary risk need to be standardized across state agencies for each of the categorical groups served by WIC. While differences in culture and language preclude the use of a single tool in all settings or even in a single setting, some form of standardization needs to occur to ensure equal access to program benefits regardless of the individual’s place of residence or cultural background. Moreover, if federal funding for the program is limited, standardization could help to ensure that individuals at greatest risk and with potential to benefit are served first. Standardization of dietary assessment tools and their interpretation can also facilitate tracking program benefits and comparing program activities and results across states. Program efficiencies may be gained by the broader use of standardized tools. These efforts could provide a stronger information base for the U.S. Department of Agriculture (USDA) and states to track program operations and uses of dietary risk assessment in WIC. For example, a few years ago, states were interested in having a common tool to assess the risk of food insecurity/hunger. USDA has since developed a food security module (USDA, 2001a) which, if used by states collecting this type of data, will allow comparison to data on a national level.

OCR for page 49
Dietary Risk Assessment in the WIC Program 8. The Tools Should Allow for Prioritization Within the Category of Dietary Risk Currently, funding for the supplemental food assistance portion of WIC is sufficient to meet current participation levels, and all who apply and meet eligibility criteria receive the food assistance component of WIC. However, if and when resources for WIC are insufficient to serve all those eligible, a tool should allow the prioritization of risk within the dietary risk category. The goal should be to ensure that those at greatest dietary risk and those most likely to benefit are served first. Meeting this goal requires a set of criteria that has different degrees of stringency reflecting different degrees of risk. SUMMARY These eight criteria formed the framework used by the committee for evaluating tools to assess dietary risk. In order to be a desirable tool, it must: use specific criteria that are related to health or disease; be appropriate for age and physiological condition; serve three purposes: screening for eligibility, tailoring of food packages, and nutrition education; have acceptable performance characteristics (validity and reliability); be suitable for the culture and language of the population served; be responsive to operational constraints in the WIC setting; be standardized across states and agencies; and allow prioritization within the category of dietary risk.

OCR for page 49
Dietary Risk Assessment in the WIC Program This page in the original is blank.