National Academies Press: OpenBook

Dietary Risk Assessment in the WIC Program (2002)

Chapter: 4 Framework for Evaluating Tools to Assess Dietary Risk

« Previous: 3 Using the Dietary Guidelines as the Basis of Dietary Risk Criteria
Suggested Citation:"4 Framework for Evaluating Tools to Assess Dietary Risk." Institute of Medicine. 2002. Dietary Risk Assessment in the WIC Program. Washington, DC: The National Academies Press. doi: 10.17226/10342.
×

4
Framework for Evaluating Tools to Assess Dietary Risk

During early deliberations, the Committee on Dietary Risk Assessment in the WIC Program developed a framework for evaluating methods to assess dietary risk in WIC program applicants. The committee’s overall goal was to identify an assessment tool that could determine whether individuals did or did not meet the Dietary Guidelines for Americans (USDA/HHS, 2000) or more specifically (as discussed in Chapter 3) Let the Pyramid guide your food choices. It also considered the potential of tools to identify nutrient intakes in relation to cut-off points since diet adequacy is another allowed type of criterion (see Appendix A). This chapter outlines eight characteristics that together provide a framework for evaluating the usefulness and effectiveness of a dietary risk assessment tool in the WIC setting. Based on further deliberations, this framework has been modified slightly from that presented in the committee’s interim report (IOM, 2000c).

DESIRABLE CHARACTERISTICS OF AN ASSESSMENT TOOL

1. The Tools Should Identify Dietary Risks that are Related to Health or Disease

Ideally, any risk criterion adopted for dietary risk should be both predictive of the individual’s risk of health problems as well as indicative of nutrition and health benefit from program participation. When considering health outcomes for children, appropriate growth and development are key facets of health. Diet

Suggested Citation:"4 Framework for Evaluating Tools to Assess Dietary Risk." Institute of Medicine. 2002. Dietary Risk Assessment in the WIC Program. Washington, DC: The National Academies Press. doi: 10.17226/10342.
×

has been shown to have both short- and long-term effects on behavior, cognitive development, physical growth, and general health status (Levitsky and Strupp, 1995; Pollitt, 1988). Inadequate energy intake in early life may be directly linked to poor outcomes in cognitive function, such as learning, or nutritional status in childhood (Gorman, 1995). Infants, preschool, and school-age children who are iron-deficient show deficits in mental development, attention, and learning, as well as in achievement test scores, when compared to iron-replete children (CDC, 1998b). In addition, essential fatty acids are necessary for proper brain development (Uauy et al., 2000).

For children ages 2 to 5 years and pregnant or postpartum women, the 1996 IOM WIC report suggested using the indicator failure to meet Dietary Guidelines (IOM, 1996). As discussed in Chapter 3, this would involve using the updated consensus document, Dietary Guidelines for Americans (USDA/HHS, 2000), as a reference, specifically as related to the two guidelines, Let the Pyramid guide your food choices and Be physically active each day.

In screening situations, one is assessing how an individual’s dietary intake compares with an appropriate cut-off point based on the Dietary Guidelines. The purpose is to conclude whether the individual “meets” or “does not meet” the Dietary Guidelines. For such vulnerable populations as pregnant women, postpartum women, and children ages 2 to 5 years, the committee decided that many criteria could be set, any one of which would provide evidence that the individual fails to meet either the Food Pyramid guideline or the physical activity guideline (see Chapters 3, 5, and 6).

2. The Tools are Appropriate for Age and Physiological Condition

Several subgroups are served in the WIC program: pregnant, breastfeeding, and nonbreastfeeding women, along with their infants and children younger than 5 years of age. When assessing dietary risk, consideration needs to be given to the specific nutritional recommendations and appropriate dietary patterns for these groups. For example, if a tool were to assess whether a client consumes the recommended number of servings of fruits and vegetables, it would need to be designed to accommodate different recommendations for adult women as compared to young children and differences in common food choices by these two groups. A second consideration relates to the method of administration of the tool for assessing dietary risk. For example, young children cannot report their dietary intake, and proxy (parental) reports must be used. Hence, one must evaluate the suitability of tools for each client subgroup.

Suggested Citation:"4 Framework for Evaluating Tools to Assess Dietary Risk." Institute of Medicine. 2002. Dietary Risk Assessment in the WIC Program. Washington, DC: The National Academies Press. doi: 10.17226/10342.
×

3. The Tools Should Ideally Serve Three Purposes: Screening for Eligibility, Individualizing the Food Package, and Nutrition Education for Behavior Change

As discussed in Chapter 2, dietary assessment tools are utilized for three reasons in the WIC setting: to define dietary risk as a criterion for eligibility in the WIC program, to identify eating patterns that influence the type of supplemental food package provided by the program, and as the starting point for nutrition education and counseling efforts. Ideally, one tool could be used for these purposes; however, the committee recognized that the standards for the effectiveness of a tool for screening would likely be higher than those required if the tool is used primarily for education. In the latter case, less well-performing tools would still have utility for education purposes if they provided the WIC nutrition professional with sufficient background on the individual’s food choices from which to begin a dialogue regarding dietary change.

4. The Tools Should have Acceptable Performance Characteristics

All instruments should be evaluated prior to use to ensure that they perform adequately. Performance is assessed in quantitative terms by considering the validity and reliability of the instrument (Windsor et al., 1994), and related constructs that are defined in Table 4-1. Validity addresses whether one is really measuring what was intended. For example, 24-hour dietary recalls are intended to measure dietary intake for the previous 24-hour period, but several recent studies have revealed that as much as 30 percent of foods reported by children were not eaten the previous day (Baxter et al., 1997). Foods reported but not eaten are called intrusions or phantom foods (Domel et al., 1994). A method that systematically under- or overestimates consumption leads to biased estimates and is therefore not considered valid. Reliability relates to whether applying the same instrument two or more times provides the same results (Table 4-1). Reliability thereby indicates the degree of random error in the dietary assessment method. Random error could be caused by such conditions as the respondent or interviewer being upset at the time of assessment, multiple interviewers, excessive noise during assessment, the limitations of memory, or a person’s inability to properly average intake to provide a desired response on a food frequency questionnaire. Random error is always present; therefore, the question when evaluating a tool is whether the level of random error present is acceptable for the intended purpose. Chapter 5 provides further information about error in dietary data collection and the performance of different types of data collection tools used to assess diet.

Error in the assessment of the dietary intake of an individual leads to misclassification in the determination of eligibility for the WIC program. Misclassification has serious consequences in that some truly eligible individuals

Suggested Citation:"4 Framework for Evaluating Tools to Assess Dietary Risk." Institute of Medicine. 2002. Dietary Risk Assessment in the WIC Program. Washington, DC: The National Academies Press. doi: 10.17226/10342.
×

TABLE 4-1 Terms Used When Describing or Evaluating the Performance Characteristics of an Assessment Tool

Term

Definitions

Validity

Does the method measure what it is supposed to measure?

Is the method accurate? That is, does it provide an unbiased estimate of usual dietary intake?

Bias

Also known as systematic error

If biased, the estimated mean intake is not equal to the true mean intake

Reliability

Refers to the ability of the estimate to be reproduced when the measure is repeated

The inability of the measure to be reproduced is a function of the amount of random error in the assessment procedure

Reproducibility

See Reliability

Random error

Variability in the measure when assessed over time

Increases the variance around the mean of the measure, but does not affect the estimate of the mean

Random error is inversely related to reliability

Between-individual variability

Variability across individuals in their usual dietary intakes

Considered the true variability when estimating intakes of groups

Within-individual variability

Variability in dietary intakes within an individual from day to day

Reduces the reliability of the measurement of usual intake

Measurement error or imprecision

Refers to error in dietary intake estimation due to the measurement process itself

Includes interviewer differences, food composition database errors

Reduces the reliability and validity of the measurement of usual intake

Misclassification

Quantification of error within the context of classifying individuals as being at dietary risk

Quantified in terms of Sensitivity and Specificity of the measure

Sensitivity

Refers to the proportion or percent of individuals with dietary risk who are identified by the assessment tool as being at dietary risk

Specificity

Refers to the proportion or percent of individuals not at dietary risk who are identified by the assessment tool as not being at dietary risk

Positive predictive value

Refers to the proportion or percent of individuals identified at dietary risk who are truly at dietary risk

Negative predictive value

Refers to the proportion of individuals identified to not be at dietary risk who are truly not at dietary risk

Suggested Citation:"4 Framework for Evaluating Tools to Assess Dietary Risk." Institute of Medicine. 2002. Dietary Risk Assessment in the WIC Program. Washington, DC: The National Academies Press. doi: 10.17226/10342.
×

may not be classified as eligible for the services (less than perfect sensitivity), or individuals not truly eligible for the services may receive them (less than perfect specificity). Chapter 5 provides examples of the effects of less than perfect sensitivity and specificity. In the absence of perfect tools (tools with 100 percent sensitivity and 100 percent specificity), policymakers and the public must decide how much and what type of misclassification error they are willing to tolerate when certifying people to receive or not receive federally funded WIC services. It is the view of the committee that less than perfect specificity should be tolerated in order to achieve perfect sensitivity; in other words, to ensure that all truly eligible individuals are identified as eligible with existing assessment tools, it is acceptable that some truly noneligible individuals receive WIC services.

5. The Tools Should be Suitable for the Culture and Language of the Population Served

The WIC program serves a multiethnic, multicultural, heterogeneous population. Thirty-nine percent of WIC participants are Caucasian, 33 percent are Latino, 23 percent are African American, 3 percent are Asian or Pacific Islander, and 2 percent are American Indian or Alaskan Native (Bartlett et al., 2000). The percentages of non-Caucasians and the diversity of cultures are expected to increase. Diversity in heritage, geography, food consumed, and culture translates into diversity in dietary patterns and practices. To assess dietary intake and patterns effectively, dietary assessment tools would need to be developed with each specific culture in mind. Thus, many WIC agencies would require several dietary assessment tools to serve their population mix. Language translation alone would not provide an acceptable tool for a different culture because the types of foods consumed, the portion sizes, food combinations, and the way foods and eating are conceptualized are likely to differ.

It is true that standardized 24-hour recalls and food records capture cultural preferences and foods consumed, provided that the interviewer is knowledgeable about reported foods, follows standard methods, and uses a food composition database that includes the foods. Thus, the need to consider specific development of tools for different cultural groups refers to the use of food frequency methods to determine usual dietary intakes. It was also recognized that effective administration of tools to different cultural groups would likely require special training and that little information exists to document successful adaptations of dietary assessment instruments for use in different cultures whose members wish to use WIC services.

Suggested Citation:"4 Framework for Evaluating Tools to Assess Dietary Risk." Institute of Medicine. 2002. Dietary Risk Assessment in the WIC Program. Washington, DC: The National Academies Press. doi: 10.17226/10342.
×

6. The Tools Should be Responsive to Operational Constraints in the WIC Setting

Time constraints for both staff and participants necessitate the use of an assessment tool that can be administered, scored, and interpreted rapidly. It is imperative that the tools under consideration take into account the variety of skills and knowledge levels of the competent professional authorities (CPAs) who assess dietary intake in the WIC setting. Whether CPAs are paraprofessional or professional, the assessment and educational tools they use need to be linguistically and culturally appropriate for different population groups served by WIC clinics.

A tool should provide consistent results regardless of the staff member who administers it. Subjective measures in scoring should be avoided to eliminate administrator bias. Furthermore, the tool should be constructed in a manner so as not to influence the client. Features that may influence responses inappropriately include scoring mechanisms placed directly on a self-administered form and phrasing that invites desirable or favorable responses rather than accurate ones. Additional points that need to be considered include the impact of the tools on the systems used by the WIC agency, and expected future changes to the system, such as automation or computerization.

7.

The Tools Should be Standardized Across States and Agencies

To some degree, tools used to determine eligibility for WIC participation based on dietary risk need to be standardized across state agencies for each of the categorical groups served by WIC. While differences in culture and language preclude the use of a single tool in all settings or even in a single setting, some form of standardization needs to occur to ensure equal access to program benefits regardless of the individual’s place of residence or cultural background. Moreover, if federal funding for the program is limited, standardization could help to ensure that individuals at greatest risk and with potential to benefit are served first.

Standardization of dietary assessment tools and their interpretation can also facilitate tracking program benefits and comparing program activities and results across states. Program efficiencies may be gained by the broader use of standardized tools. These efforts could provide a stronger information base for the U.S. Department of Agriculture (USDA) and states to track program operations and uses of dietary risk assessment in WIC. For example, a few years ago, states were interested in having a common tool to assess the risk of food insecurity/hunger. USDA has since developed a food security module (USDA, 2001a) which, if used by states collecting this type of data, will allow comparison to data on a national level.

Suggested Citation:"4 Framework for Evaluating Tools to Assess Dietary Risk." Institute of Medicine. 2002. Dietary Risk Assessment in the WIC Program. Washington, DC: The National Academies Press. doi: 10.17226/10342.
×

8. The Tools Should Allow for Prioritization Within the Category of Dietary Risk

Currently, funding for the supplemental food assistance portion of WIC is sufficient to meet current participation levels, and all who apply and meet eligibility criteria receive the food assistance component of WIC. However, if and when resources for WIC are insufficient to serve all those eligible, a tool should allow the prioritization of risk within the dietary risk category. The goal should be to ensure that those at greatest dietary risk and those most likely to benefit are served first. Meeting this goal requires a set of criteria that has different degrees of stringency reflecting different degrees of risk.

SUMMARY

These eight criteria formed the framework used by the committee for evaluating tools to assess dietary risk. In order to be a desirable tool, it must:

  • use specific criteria that are related to health or disease;

  • be appropriate for age and physiological condition;

  • serve three purposes: screening for eligibility, tailoring of food packages, and nutrition education;

  • have acceptable performance characteristics (validity and reliability);

  • be suitable for the culture and language of the population served;

  • be responsive to operational constraints in the WIC setting;

  • be standardized across states and agencies; and

  • allow prioritization within the category of dietary risk.

Suggested Citation:"4 Framework for Evaluating Tools to Assess Dietary Risk." Institute of Medicine. 2002. Dietary Risk Assessment in the WIC Program. Washington, DC: The National Academies Press. doi: 10.17226/10342.
×
This page in the original is blank.
Suggested Citation:"4 Framework for Evaluating Tools to Assess Dietary Risk." Institute of Medicine. 2002. Dietary Risk Assessment in the WIC Program. Washington, DC: The National Academies Press. doi: 10.17226/10342.
×
Page 49
Suggested Citation:"4 Framework for Evaluating Tools to Assess Dietary Risk." Institute of Medicine. 2002. Dietary Risk Assessment in the WIC Program. Washington, DC: The National Academies Press. doi: 10.17226/10342.
×
Page 50
Suggested Citation:"4 Framework for Evaluating Tools to Assess Dietary Risk." Institute of Medicine. 2002. Dietary Risk Assessment in the WIC Program. Washington, DC: The National Academies Press. doi: 10.17226/10342.
×
Page 51
Suggested Citation:"4 Framework for Evaluating Tools to Assess Dietary Risk." Institute of Medicine. 2002. Dietary Risk Assessment in the WIC Program. Washington, DC: The National Academies Press. doi: 10.17226/10342.
×
Page 52
Suggested Citation:"4 Framework for Evaluating Tools to Assess Dietary Risk." Institute of Medicine. 2002. Dietary Risk Assessment in the WIC Program. Washington, DC: The National Academies Press. doi: 10.17226/10342.
×
Page 53
Suggested Citation:"4 Framework for Evaluating Tools to Assess Dietary Risk." Institute of Medicine. 2002. Dietary Risk Assessment in the WIC Program. Washington, DC: The National Academies Press. doi: 10.17226/10342.
×
Page 54
Suggested Citation:"4 Framework for Evaluating Tools to Assess Dietary Risk." Institute of Medicine. 2002. Dietary Risk Assessment in the WIC Program. Washington, DC: The National Academies Press. doi: 10.17226/10342.
×
Page 55
Suggested Citation:"4 Framework for Evaluating Tools to Assess Dietary Risk." Institute of Medicine. 2002. Dietary Risk Assessment in the WIC Program. Washington, DC: The National Academies Press. doi: 10.17226/10342.
×
Page 56
Next: 5 Food-Based Assessment of Dietary Intake »
Dietary Risk Assessment in the WIC Program Get This Book
×
Buy Paperback | $46.00 Buy Ebook | $36.99
MyNAP members save 10% online.
Login or Register to save!
Download Free PDF

Dietary Risk Assessment in the WIC Program reviews methods used to determine dietary risk based on failure to meet Dietary Guidelines for applicants to the Special Supplemental Nutrition Program for Women, Infants, and Children (WIC). Applicants to the WIC program must be at nutritional risk to be eligible for program benefits. Although “dietary risk” is only one of five nutrition risk categories, it is the category most commonly reported among WIC applicants.

This book documents that nearly all low-income women in the childbearing years and children 2 years and over are at risk because their diets fail to meet the recommended numbers of servings of the food guide pyramid. The committee recommends that all women and children (ages 2-4 years) who meet the eligibility requirements based on income, categorical and residency status also be presumed to meet the requirement of nutrition risk. By presuming that all who meet the categorical and income eligibility requirements are at dietary risk, WIC retains its potential for preventing and correcting nutrition-related problems while avoiding serious misclassification errors that could lead to denial of services for eligible individuals.

  1. ×

    Welcome to OpenBook!

    You're looking at OpenBook, NAP.edu's online reading room since 1999. Based on feedback from you, our users, we've made some improvements that make it easier than ever to read thousands of publications on our website.

    Do you want to take a quick tour of the OpenBook's features?

    No Thanks Take a Tour »
  2. ×

    Show this book's table of contents, where you can jump to any chapter by name.

    « Back Next »
  3. ×

    ...or use these buttons to go back to the previous chapter or skip to the next one.

    « Back Next »
  4. ×

    Jump up to the previous page or down to the next one. Also, you can type in a page number and press Enter to go directly to that page in the book.

    « Back Next »
  5. ×

    Switch between the Original Pages, where you can read the report as it appeared in print, and Text Pages for the web version, where you can highlight and search the text.

    « Back Next »
  6. ×

    To search the entire text of this book, type in your search term here and press Enter.

    « Back Next »
  7. ×

    Share a link to this book page on your preferred social network or via email.

    « Back Next »
  8. ×

    View our suggested citation for this chapter.

    « Back Next »
  9. ×

    Ready to take your reading offline? Click here to buy this book in print or download it as a free PDF, if available.

    « Back Next »
Stay Connected!