The importance of evaluating the relative impact of obesity efforts is well known, and the challenges are not new. Those challenges remain, however, far from resolved. Even modest improvement in evaluation has the potential to provide clarity and refined direction in addressing the obesity epidemic. This report provides flexible evaluation plans for the national, state, and community levels (including indicators of status and progress) that can be implemented now. It also recommends a number of infrastructure changes at the national level that could make these evaluations even more effective in the future.
With funding from the Michael & Susan Dell Foundation, the Institute of Medicine (IOM) Committee on Evaluating Progress of Obesity Prevention Efforts was formed to develop a concise and actionable plan for measuring progress in obesity prevention efforts for the nation and adaptable guidelines for local community evaluation. The Committee’s assigned tasks were to
1. draw on the recommendations and recommended indicators of progress from the IOM committee and 2012 report, Accelerating Progress in Obesity Prevention (APOP); consider existing and new tools and metrics (e.g., trend analysis, community/local measures) to measure progress; and develop a plan for a national-level evaluation of obesity prevention efforts by sector and, if appropriate, across sectors;
2. develop a community-level measurement plan that adds detail and supports the national-level evaluation plan; and
3. identify measurement ideas that can determine the specific impact of the Home Box Office (HBO)/IOM campaign called The Weight of the Nation (TWOTN).
The intended audiences for the report’s recommended plans and measurement ideas are decision makers, community members, researchers, and evaluators at all levels and across all sectors of society.
The Committee relied on specific definitions for commonly used terms in this report. Interventions refer to policies, programs, systems, environmental changes, services, products, or any combination of these multi-faceted initiatives. Assessment is an effort to use data on the community or other jurisdiction to characterize the problem, its distribution, and efforts to address it. Monitoring is the tracking of the implementation of interventions compared to standards of performance. Surveillance is the ongoing systematic, collection, analysis, and interpretation of data that are tracked over time to detect patterns, disparities, and changes that may be associated with interventions or other causes. Summative evaluation is the effort to detect changes in output, outcomes, and impacts associated with the interventions and to attribute those changes to the interventions. In this report, the Committee sometimes uses the term evaluation to refer to all four of these functions. An indicator is a source of data or evidence that can be used to assess the status or trend of a person or population (a measurement, e.g., prevalence of obesity).
OBESITY EVALUATION PLANS
Evaluation plans (for assessment, monitoring, surveillance, and summative evaluation activities) are tools that contain guidance for planning, implementing, and evaluating obesity prevention efforts. A comprehensive evaluation plan can guide decision makers and users responsible for developing or funding evaluations to measure progress in national obesity prevention efforts. Community-level obesity evaluation plans can similarly support identification of key components in implementing evaluations at the local level. They offer guidance at the community level that is sensitive to local variation in needs, context, and resources, and they can help to support aggregation and dissemination of information across communities.
As described in the evaluation process framework in Figure 10-1 (detailed in Chapter 3), an evaluation plan is a key activity (found in Box 2 of Figure 10-1) that provides guidance for organizing and implementing evaluation-related efforts to achieve the intended outputs, outcomes, and impacts identified in the evaluation framework.
The successful implementation of the national and community evaluation plans recommended in this report will require the support of other activities, including a core set of indicators and common measures of success; resources for training, technical assistance, and dissemination; and an adequate evaluation infrastructure. To support these activities, the Committee (1) identifies existing indicators of progress that can be incorporated into the recommended plans, can be helpful in identifying gaps in existing data and information systems, and can provide examples of indicators that can be used by evaluators seeking to evaluate obesity prevention interventions and (2) recommends actions that will improve leadership and coordination, guidance, capacity, and infrastructure for evaluation efforts.
Actions to support and implement the plans and other activities will improve evaluation capacities in the short-term (e.g., use of a core set of existing indicators), increase evaluation activities in the intermediate-term (e.g., improve capacity and guidance), and enhance data use in the long-term to assess population-level changes and improvements that can result from widespread implementation of evidence-based interventions to prevent obesity (i.e., outcomes, Box 4 in the evaluation framework).
CONCLUSIONS AND RECOMMENDATIONS
Along with the framework in Chapter 3, the Committee details key findings in this report on dimensions of national- and community-level evaluation, including the information/data needs of those
FIGURE 10-1 Framework for evaluating progress of obesity prevention efforts.
a Evaluation refers to assessment, monitoring, surveillance, and summative evaluation activities.
NOTE: An Evaluation Plan is an activity listed in Box 2.
interested in obesity prevention and its results (Chapter 2), indicators that can serve as markers for assessing the progress of obesity prevention efforts recommended in the APOP report (Chapter 4), the infrastructure and capacity to support evaluation, and methods and protocols for conducting evaluations (Chapters 5, 6, 7, and 8), and a systems approach to obesity prevention evaluation efforts (Chapter 9). Based on these findings, the Committee has identified seven broad conclusions that serve as the context for the development of and guidance provided in the recommended plans and supporting actions that follow (see Box 10-1).
This chapter describes (1) indicators that are aligned with the recommendations in the APOP report from readily available data sources; (2) a National Obesity Evaluation Plan; and (3) a Community Obesity Evaluation Plan, and it recommends seven actions to support the implementation of the recommended plans. In the final part of the chapter the Committee identified measurement ideas for determining the impact of the HBO/IOM TWOTN campaign.
Broad Conclusions Informing Plans for Evaluation Regarding Obesity Prevention Efforts
• There is a pressing need to act on the problem of obesity, but there are gaps in the certainty of the effectiveness of actions or mixture of actions being implemented across the country. Systematic and comprehensive evaluations along with more routine assessments, monitoring, and surveillance offer valuable guidance for improving the quality and outcomes (or impact) of the actions being implemented and for defining the direction of further basic and implementation research.
• Information generated from current obesity prevention evaluation efforts, other than assessment of needs at the national and state levels, does not always address the needs and interests of the users of this information, often because of limited or outdated data (especially at the community level) and few presentations of the data in useful and timely formats.
• Current data (monitoring) systems do not adequately track progress of environmental and policy-related obesity prevention actions or systems changes recommended in the Accelerating Progress in Obesity Prevention report (IOM, 2012a). Such monitoring is needed at both the national and community levels, especially for populations at greatest risk for obesity. These limitations exist primarily because monitoring systems have traditionally focused on measuring individual behaviors, energy expenditure/energy intake, and overweight and obesity.
• Current investment in evaluation is too low and sporadic, presenting serious barriers to understanding the impact of and need for future investments in implementing interventions.
• A systems science approach to evaluation can help evaluation users identify and select combinations of actions and strategies to implement in multiple sectors, and at multiple levels, with available resources.
• Although many data systems exist, the current national systems for monitoring progress of recommended obesity prevention actions and for surveillance of their effects on obesity lack adequate leadership, coordination, infrastructure, guidance, accountability, and capacity.
• Communities lack adequate guidance, capacity, data, and resources necessary for assessing the status of obesity and its determinants, identifying prevention needs, monitoring obesity prevention actions, evaluating their short-term outcomes, and tracking (through surveillance) their long-term association with obesity reduction in the aggregate and differences among population segments.
Indicators of Progress
One clear gap in evaluation efforts is the identification of a set of core indicators to use at the national and community levels for measuring progress in obesity prevention efforts. As a key first step in identifying this core set, the Committee identified a list of indicators that currently exist. Based on available and ongoing data sources, the Committee identified several overarching and goal area–specific
indicators that were best aligned with the recommendations included in the APOP report. The 83 indicators identified provide a menu of possible indicators for use by evaluators. This exercise produced a list of indicators that can act as a starting point for the development of core indicators and related measures as well as for the identification of gaps in the existing data systems that can be incorporated into evaluation plans (national and community levels), and provide guidance to improve long-term evaluation infrastructure and capacities. In the short term, evaluators of obesity prevention programs, policies, and environments can use the indicators identified by the Committee to begin to comprehensively assess obesity prevention actions already being implemented across the country.
Of particular importance to the Committee was recognition that evaluating progress for the nation as a whole, and for regions of the nation, requires special attention to the disparities that are associated with the obesity epidemic. Although numerous challenges remain, the Committee found a small yet growing literature of tools and methodologies for monitoring progress toward obesity prevention among populations with health disparities.
A National Evaluation Plan for Obesity Prevention
A discrepancy exists between the importance and magnitude of the obesity problem and the level of action in the United States for developing a cohesive plan to evaluate implementation of efforts across the country and assess their impact. Currently U.S. efforts lag behind those in other countries to provide common guidance, support, and the appropriate infrastructure to support evaluation efforts. Although the Committee identified important strengths of the current monitoring, surveillance, and summative evaluation data systems, limitations of current national evaluation efforts exist, the following needs resulting in:
• coordinating leadership, integration, and accountability of evaluation efforts across federal agencies, within and between departments, across federal, state, and local governments, and with the nongovernmental and private sectors;
• maximizing and coordinating the use of data already being collected;
• identifying and prioritizing indicators at the national and community levels and developing new indicators where necessary;
• improving surveillance capacity and frequency, especially for policies and environmental factors, and evaluation capacity/leadership;
• improving training and support for monitoring, surveillance, and evaluation;
• improving access to and dissemination of data, findings from analysis, and other information for the consumer;
• collecting additional longitudinal data, including national incidence trends and local prevalence trends for obesity;
• tracking and monitoring disparities and their social determinants (i.e., differential exposures/ opportunities, vulnerabilities/capabilities, and consequences); and
• using best practices of evaluation design, including monitoring and feedback on progress on intermediate outcomes (i.e., community/system changes such as new policies, expanded programs, and environmental changes).
The Committee developed a National Evaluation Plan for Obesity Prevention (see Box 10-2) for the United States. The National Obesity Evaluation Plan is designed to organize the planning, implementation, and evaluation of the impact of obesity prevention interventions recommended in the APOP report at a national level. It is important to note here that the APOP report frames obesity prevention efforts in terms of policy, systems-level, and environmental approaches, which require new evaluation approaches, indicators, and measures. Box 10-2 identifies key components for organizing a National Obesity Evaluation Plan. Activities for achieving the objectives are outlined here; Chapter 6 provides detailed guidance to address each activity. This plan is intentionally broad to provide a starting point for the development, use, and support of core indicators (derived from the list in Chapter 4 organized around the APOP recommendations) and recommended methodologies, as well as flexibility for future innovations. The Committee stresses that the National Obesity Evaluation Plan activities should be prioritized to leverage existing resources that maximize the use of existing resources and efficiency of data collection and avoid duplication of efforts. Seven recommended actions to improve the national evaluation infrastructure necessary for implementing the plans are offered in a later section of this chapter.
The National Evaluation Plan for Obesity Prevention can be used as a model for state and multi-state regional evaluations. By using the National Obesity Evaluation Plan as a blueprint, states will be able to provide comparable data that can be used as benchmarks for state progress, when monitored over time, as well as when compared to other state and national data. However, state-level evaluation activities should be flexible enough to adapt to unique populations and regional characteristics requiring changes in measurement protocols or instruments. While a National Obesity Evaluation Plan can show changes in general trends over time, state-level plans have the potential to identify success stories using APOP strategies that can be disseminated broadly to accelerate obesity prevention progress.
The Community Evaluation Plan for Obesity Prevention
The Committee provides guidance for communities that are implementing or intend to implement obesity prevention interventions. The guidance, in the form of the Community Obesity Evaluation Plan consists of two distinct sets of activities: (1) community assessment and surveillance to describe the current health status, resources, and determinants of health in a community (assessment) and to track them over time (surveillance) and (2) more tailored community intervention summative evaluations that seek to establish and share what is being tried and implemented (monitoring) and to identify the effectiveness of local efforts to prevent obesity (summative evaluation). Together these activities provide baseline data and “diagnostic” data on the state of obesity and related “determinants” or conditions in the community. They offer opportunities to establish and share “what works.”
Many times, information captured locally is specifically tailored data that cannot be captured at the national level; community-level evaluation activities (assessment, surveillance, monitoring, and summative evaluation) provide an essential additional level of detail and local context-specific information that the National Obesity Evaluation Plan cannot. The learnings derived from this local information will allow greater return on national investments in obesity prevention as well as inform refinements to the national evaluation plan.
Core Components and Activities of the National Plan for Evaluating Progress in Obesity Prevention
Purpose: To evaluate progress at the national level in implementing strategies from the Institute of Medicine Accelerating Progress in Obesity Prevention (APOP) report and in achieving intended impacts as described in the Evaluation Framework (see Box 5 in Figure 10-1).
1. Identify leadership, infrastructure, resources, priorities, and timeline for implementing the plan.
2. Identify current national efforts for evaluation, including indicators (Chapter 4), and incorporate them selectively into national monitoring, surveillance, and summative evaluation data systems that are responsive to the needs of data users.
3. Propose data and infrastructure to add to existing monitoring and surveillance systems to fill gaps, and facilitate community obesity evaluation plans.
4. Propose additional assessment, monitoring, surveillance, and summative evaluation activities, new measures, and innovative strategies to implementation in the future.
5. Outline mechanisms for feedback to data users, assuring accessibility, privacy, and cost-efficiency.
6. Detail adaptations of the plan at the state level, with further applications at the regional level.
Activities (see seven recommended actions for implementation):
1. Designate a federal obesity evaluation task force or entity to oversee the implementation of the National Obesity Evaluation Plan and coordinate with relevant federal, state, local, and private-sector entities.
a. Identify and obtain the infrastructure necessary for implementing the plan and coordinate with appropriate partners
b. Ensure adequate benchmarks/goals, including a schedule for updates
c. Establish a process for prioritization, accountability, and adaptation of plan activities including an annual report to the agency responsible for leading the effort
d. Identify priorities and create an ongoing timeline for implementing the plan
i. Short-term objectives achievable within 1-3 years
ii. Intermediate-term objectives achievable within 3-5 years
iii. Long-term objectives achievable for 5 years or longer
2. Identify current national evaluation efforts, including indicators for monitoring and surveillance systems to minimize duplication, maximize use of data already being collected, and identify priorities to address evaluation gaps in a coordinated fashion.
a. Use the indicator list (Chapter 4) as a starting point to identify a core set of indicators
b. Match indicators as much as possible for common measurement across jurisdictions
c. Examine existing links to the Leading Health Indicators and other recommendations as consistent with APOP
d. Promote use of common measures through the National Collaborative on Childhood Obesity Research (NCCOR) (see Chapter 5) to facilitate harmonization of data across data-collection systems
e. Expand School Health Policies and Practices Study to include measures of additional settings such as worksite, child care centers, and schools on a rolling basis every 3 years rather than current settings every 6 years
f. Expand National Health and Nutrition Examination Survey (NHANES) sampling, analyses, and/or reporting to address gaps in developmental levels of children birth to 1 year, 2 to 5 years, 6 to 10 years, 11 to 13 years, and 14 to 19 years
g. Expand NHANES to oversample populations that are underserved or at greater risk for obesity
h. Standardize currently collected data and planned systems, such as electronic health records, for data aggregation
i. Incorporate data from birth certificates, Special Supplemental Nutrition Program for Women, Infants, and Children (WIC), Early Head Start, and Head Start into the National Obesity Evaluation Plan
j. Expand current monitoring and surveillance structures into existing data-collection systems at the national or state levels
3. Develop new data-collection infrastructure or systems, indicators, and measures to address gaps identified as priorities in areas such as policy and environment, physical activity, child care centers, worksites, health plans, federally qualified health centers, and community health centers/WIC clinics.
Community Assessments and Surveillance
Community assessment provides first-time assessment of status or trends overall in a community. Surveillance provides repeated or continuous assessment of progress overtime. Specific to this report, these data can assess what is being done in a community and reveal the factors that influence local obesity prevalence and incidence (e.g., demographics, social determinants).
4. Increase national and state capacity for assessment, monitoring, surveillance, and summative evaluation.
a. Standardize and provide training on measurement protocols (e.g., body mass index, waist circumference) and data-collection methods
b. Provide technical support for data utilization, statistical analysis, and reporting
i. Assess the impact of the data loss that resulted from discontinuation of the Centers for Disease Control and Prevention’s Pediatric Nutrition Surveillance System and Pregnancy Nutrition Surveillance System (state- and county-level data) and provide ongoing technical assistance to states that use existing data
c. Create lists of recommended standardized tools and methods for measurement
i. Expand and maintain the NCCOR Surveillance System and Measures Registry
5. Ensure that all relevant data systems include a mechanism for relevant and timely feedback to data users.
a. Expand Health Indicators Warehouse and other interactive sources of federal-level data
b. Expand and maintain Community Commons
c. Develop additional “dashboards” and “federal report card” formats that can be interactive and display data in easily understood infographics and tables
6. Ensure that evaluation plans in federally funded obesity-related grants and programs include common indicators and measures that can be aggregated across communities and inform the plan.
7. Encourage development and testing of alternative and emergent methods of collecting data, including
a. Real-time access of data from community-based organizations
b. Capitalization on the “quantified self” movement
c. Use of new technologies and geospatial modeling
Although the Committee identified several resources available to aid communities across the country, currently there is no consensus guidance for what indicators to measure or methodologies to use when conducting obesity-focused community assessment and surveillance. Based on a review of the existing infrastructure for conducting obesity-focused community-level assessment and surveillance, the Committee found
• a lack of data available at the local level for indicators relevant to measuring progress of APOP strategies. Especially needed are data for preschoolers and elementary school children, and sys-
tematic descriptions of determinants of obesity (e.g., environments, policies, other interventions, norms, and attitudes). Additional sources of data at the local level may exist in multiple sectors, such as health care, planning, and schools;
• a need to increase sample size of existing surveillance systems, add data on missing indicators, and develop new systems for policy, environmental, intervention indicators; and
• report data by race and socioeconomic status to the extent possible and by small areas affected by inequity for larger communities.
Other important findings include the following:
• There is a lack of a common set of indicators to allow cross-community comparisons and aggregation.
• Engaging stakeholders/community in assessment process is valuable.
• Capacity to develop assessments varies widely across communities.
• Improving the accessibility and dissemination of assessment data through multiple channels will improve their use for decision makers, media, and the public.
The intent of the Community Obesity Assessment and Surveillance Plan (see Box 10-3) is to provide guidance for local communities to identify and use a set of core indicators that measure obesity-related outcomes and impacts and to develop local capacity for these assessments, including common use and understanding of assessment protocols, descriptions of health disparities, community engagement, oversight, and public reporting on progress. The Committee developed the Community Obesity Assessment and Surveillance Plan to not only accommodate communities with varying resources and assets, but also provide a core set of indicators that can be measured comparably and aggregated across multiple jurisdictions. Recommended actions to support the development of local infrastructure and capacities for community assessments and surveillance are offered in a later section of this chapter. These enhanced capacities will lead to greater use of data and findings to inform local obesity prevention efforts, guide resource allocation, and engage stakeholders.
Monitoring and Summative Evaluation of Community Interventions
Community evaluations are critically important to developing knowledge about effective local interventions and for realistic implementation of local, state, and federal policies and funding initiatives. Community-level evaluation encompasses learning “what works” and also the relative effectiveness of various strategies—the extent to which they work (summative evaluation). In line with “what works,” summative evaluation also informs local implementers about ways to improve and manage interventions. It casts light on how and why these interventions may prevent obesity. Finally, it encompasses translating the effective interventions to be implemented on a broader scale and determining the contexts in which they are and are not effective (monitoring).
Box 10-4 identifies key components to develop and implement a Community-Level Obesity Intervention Monitoring and Summative Evaluation Plan. Considerable flexibility is needed within these components. As outlined, the core of any plan includes engaging stakeholders, identifying resources, having a logic model or theory of change, selecting the right focus, using appropriate measures, collecting
Components of a Community Obesity Assessment and Surveillance Plan
Purpose: To provide accurate and timely knowledge of local obesity-related conditions and relevant changes or trends over time as a result of implementing strategies in the Accelerating Progress in Obesity Prevention (APOP) report.
1. Define community boundaries.
a. Create specific geographic areas that reflect jurisdictions, key stakeholders, and community members’ perceptions of geographic boundaries.
2. Engage community members and other key stakeholders.
a. Include stakeholders to the extent possible, in defining community, identifying priorities, planning assessment, collecting data, interpretating and sense-making of results, and disseminating the findings.
3. Plan assessment/surveillance and include stakeholders and community members.
a. Identify lead agency or agencies responsible for conducting assessment/surveillance.
b. Clarify goals of assessment/surveillance.
c. Define audience and what information will move it to action.
d. Define topics to include in assessment/surveillance.
e. Identify sub-populations and small areas disproportionately affected by obesity, and develop approach to collecting information about them.
f. Select local data to be included about context, assets, interventions, barriers, and social determinants, and which data to schedule for ongoing surveillance.
4. Collect data.
a. Obtain existing data from Web-based platforms or published reports.
b. As resources permit, add other sources of data.
c. Create an inventory of local obesity prevention interventions.
5. Analyze and make sense of the data.
a. Include trends over time.
b. Present data for infants, children, adolescents, adults, and special populations.
c. Describe variation in indicators (e.g., across race/ethnicity/socioeconomic status/small areas).
d. Include comparison to benchmarks, state rates, and peer communities.
e. Compare extent of existing interventions identified to those recommended in APOP report.
f. Share data with community members and other stakeholders for their interpretations and suggested implications for action.
g. Visualize, or illustrate, data.
6. Disseminate findings.
a. Prepare reports, websites, infographics, and other dissemination tools.
b. Share findings with stakeholders and engage them in interpretation of findings.
c. Present findings at community meetings for further interpretation.
d. Implement a media advocacy strategy to gain media coverage.
e. Consider using social media to further increase awareness of findings.
NOTE: Steps are further detailed in Chapter 7.
Components of a Community-Level Obesity Intervention Monitoring and Summative Evaluation Plan
Purpose: To guide local action and to inform national choices about the most effective and cost-effective strategies identified in the Accelerating Progress in Obesity Prevention (APOP) report for funding, dissemination, and uptake by other communities.
1. Design stakeholder involvement.
a. Identify stakeholders
b. Consider the extent of stakeholder involvement
c. Assess desired outcomes of monitoring and summative evaluation
d. Define stakeholder roles in monitoring and summative evaluation
2. Identify resources for the monitoring and summative evaluation.
a. Person-power resources
b. Data-collection resources
3. Describe the intervention’s framework, logic model, or theory of change.
a. Purpose or mission
b. Context or conditions
c. Inputs: resources and barriers
d. Activities or interventions
e. Outputs of activities
f. Intended effects or outcomes
4. Focus the monitoring and summative evaluation plan.
a. Purpose or uses: What does the monitoring and summative evaluation aim to accomplish?
b. Priorities by end-user questions, resources, context
c. What questions will the monitoring and summative evaluation answer?
d. Ethical implications (benefit outweighs risk)
5. Plan for credible methods.
a. Stakeholder agreement on methods
b. Indicators of success
c. Credibility of evidence
6. Synthesize and generalize.
a. Disseminate and compile studies
b. Learn more from implementation
c. Ways to assist generalization
d. Shared sense-making and cultural competence
e. Disentangle effects of interventions
NOTE: Steps are further detailed in Chapter 8.
high-quality data, using appropriate analytic methods, engaging in sense-making, and disseminating findings. Chapter 8 provides detailed support and guidance for implementing each component of the Community-Level Obesity Intervention Monitoring and Summative Evaluation Plan. Recommended actions to support the development of local infrastructure and capacities for community-level intervention monitoring and summative evaluation are posited in a later section of this chapter.
Finally, the National Obesity Evaluation Plan and the Community Obesity Evaluation Plan are interdependent. The two plans have the potential to provide essential support and feedback to each other. Successful implementation of the community plan is supported by the components of the national plan, using indicators, sources of data, resources, and methodologies coordinated and developed with leadership at the national level. However, the community plan and its associated activities also provide an additional level of detail and local context-specific information that the national plan cannot measure. Indeed large-scale community-level evaluation efforts, which are intended to identify effective strategies that can be brought to scale in other communities, are already under way.
Considerations for Investing in Obesity Evaluation Plans
Evaluating obesity prevention is complex, and so is valuing the effort. The evaluation of obesity prevention may be a challenging proposition to implement when the gaps and recommendations identified by this report are considered. These challenges include financial resources, political factors, and different points of view of where to invest [scarce] resources. However, prevention of obesity may bring value to many stakeholders in the community and to society as a whole. As such, it would be useful to consider a valuation framework that brings transparency and legitimacy to the decision-making process of whether to invest in evaluation resources. The IOM recently published a report titled An Integrated Framework for Assessing the Value of Community-Based Prevention (IOM, 2012) that provides a blueprint for such a process. Using the proposed framework will allow local, regional, state, and national stakeholders to deal with reasonable disagreement and, in cases where such disagreement persists, identify and address potential legitimacy problems (Pronk et al., 2013).
Taking Action to Support the National and Community Obesity Evaluation Plans
Using the considerable number of indicators available through federal, state, and community efforts identified in this report (Chapter 4) and guided by methodologies and protocols outlined in the plans as a guide (Chapters 5, 6, 7, and 8), researchers, communities, and policy and other decision makers can take immediate action to begin comprehensive assessment of obesity prevention efforts recommended in the APOP report and already under way across the country.
As the study progressed and the Committee’s ideas matured, it became clear that the evaluation plans recommended by the Committee will not be fully realized without organizational changes and support across multiple federal, state, and local government agencies and departments in collaboration with other nonfederal partners responsible for obesity prevention–related activities. Given the existing gaps identified by the Committee, the following recommended actions will support the successful implementation of the components of the evaluation plans and will assure timely and meaningful data to inform and improve obesity prevention efforts at national, state, and community levels. The seven recommendations include aspects of leadership; data collection; guidance for identifying and using common indicators, mea-
sures, methods, and outcomes; dissemination of the information collected; workforce capacity development; assessment of disparities and health equity; and a systems approach to evaluation. The Committee offers a set of potential actions to guide each recommended action to fill existing gaps in the current evaluation infrastructure.
The resources needed to implement some of the recommendations (given the set of potential actions) range from minimal to substantial; some are occasional, and others require frequent to continuous measurement. As described in the prior section this means that some of the Committee’s recommendations to support implementation of the evaluation plans call for leadership and expenditures that will require trade-off decisions by government, organizations, and the private sector, with astute use of existing resources and prioritization of other necessary actions implemented with short-, intermediate-, and long-term time perspectives.
Improve Leadership and Coordination for Evaluation
The Committee believes that centralized leadership is necessary to coordinate the planning, implementation, and evaluation of the impact of obesity prevention efforts across the country. As described in Chapter 6, most of the existing data collection and support for evaluation exists across multiple federal agencies.1 The current decentralized structure provides limited authority, responsibility, or support and coordination for these efforts at the national level. The Committee views that gap in empowered leadership in coordinating resources for evaluating obesity prevention efforts at the federal level as a major obstacle to measuring obesity prevention efforts. Progress could be made if a federal task force or entity would take a leadership position in this coordination effort.
A number of relevant task forces/entities could serve in this coordination role. The Committee believes that one or a combination of these would be the best option for overseeing and implementing the National Obesity Evaluation Plan and for reporting to whatever agency is leading these efforts. Alternatively, the appointment of a new task force could also successfully address the need for improved leadership of evaluation of obesity prevention efforts, but the committee does not view it as necessary. It was not in the Committee’s charge or in its expertise to analyze various options and recommend a specific entity to take on this responsibility (e.g., Department of Health and Human Services [HHS] Healthy Weight Task Force, National Prevention Council, National Collaborative on Child Obesity Research, Interagency Committee for Human Nutrition Research, National Committee for Vital and Health Statistics).
Recommendation 1: An obesity evaluation task force or another entity should oversee and implement the National Obesity Evaluation Plan and provide support for the Community Obesity Evaluation Plan and should coordinate with other federal, state, and local public- and private-sector groups and other stakeholders who support, use, or conduct evaluations. The task force/entity could be a new or existing entity or a combination of existing entities.
1 Includes, but is not limited to, efforts in the following federal agencies: Corporation for National and Community Service; Departments of Agriculture, Commerce, Defense, Education, Health and Human Services, Interior, Labor, Transportation, and Veteran Affairs; Domestic Policy Council; Environmental Protection Agency; Federal Trade Commission; General Services Administration; and Office of Management and Budget.
The task force that oversees the National Obesity Evaluation Plan will have the following key roles:
• Identify and secure the infrastructure (i.e., effective leadership structure) necessary for implementation of the National Obesity Evaluation Plan
• Coordinate with appropriate federal partners and include representatives from major stakeholder groups (e.g., child care settings, schools, worksites, local and state government, public health departments, business/private sector, and communities)
• Ensure adequate national benchmarks/guidelines/goals (e.g., Dietary Guidelines for Americans, Healthy People 2020 objectives, Physical Activity Guidelines for Americans, reports of the U.S. Surgeon General)
• Create an ongoing timeline for implementation of the activities outlined in the National Obesity Evaluation Plan
• Establish a process for accountability, prioritization, and adaptation by agencies reporting periodically to the task force/entity on their activities, and the task force/entity reporting annually to the agency that is leading these efforts on coordination efforts, gaps in monitoring, recommendations for new measures and evaluations, and progress toward meeting goals.
Improve Data Collection for Evaluation
Recommendation 2: Using the recommended indicators and gaps identified in this report as guides (i.e., related to Accelerating Progress in Obesity Prevention report strategies), all federal agencies2 and state and local health departments responsible for collecting data relevant to obesity prevention efforts, in coordination with relevant private partners, should identify, coordinate, and maximize current efforts for ongoing collection of recommended indicators and, according to the priorities identified, address existing evaluation gaps at the national and local levels.
To guide the implementation of this recommendation, potential actions to coordinate efforts and address gaps include the following:
• Examine all relevant national survey activities and harmonize existing efforts (e.g., use of common metrics) across the federal agencies (see Appendix Table D-1).
• Identify linkages among current U.S. efforts identified in Recommendation 1 (and those of World Health Organization and European Union obesity-related evaluation plans) to enhance multinational coordination, comparison, and efficiency of evaluation plans.
• Ensure the ongoing collection and maintenance of existing data systems, leverage their use, and increase their capacity through connection and computing technology.
• Standardize currently collected data and data systems (e.g., electronic health records, data from departments of motor vehicles) to make aggregation and comparison of data feasible.
2 Agricultural Research Service, Economic Research Service, and Food and Nutrition Service of the U.S. Department of Agriculture; Census Bureau of the U.S. Department of Commerce; Agency for Healthcare Research and Quality, Centers for Disease Control and Prevention, Health Resources and Services Administration, and National Institutes of Health of the U.S. Department of Health and Human Services; Bureau of Labor Statistics of the U.S. Department of Labor; and Federal Highway Administration of the U.S. Department of Transportation.
• Expand existing surveys or develop new monitoring and surveillance systems (through other governmental funding or private funds or partnerships with commercial data providers) to address gaps at the national level, including increased frequency of key existing surveys; improved sampling or analysis and reporting for key intervention priorities in existing data surveys (all developmental levels of children and adolescents, populations at greater risk of obesity); in areas such as school, child care centers, worksites, health plans, clinics; increased sharing and use of commercially available data; developmental and implementation of policy and environmental indicators; cataloguing of interventions; and longitudinal data systems to monitor the incidence (i.e., new cases) of obesity (see Chapter 6, and Chapter 4, Table 4-4, for detailed list of gaps).
• Build, connect, and strengthen existing data systems and form partnerships to improve the availability of existing data to local jurisdictions, including improving sampling methodology and size; partnering with schools, health plans, and businesses to collect and make available relevant information to local jurisdictions; and including policy and environmental indicators (see Chapter 7 and Chapter 4, Tables 4-1 and 4-4).
• Assess the loss of data from discontinued or one-time surveillance systems (e.g., Centers for Disease Control [CDC] and Prevention’s Pediatric and Pregnancy Nutrition Surveillance Systems, National Youth Physical Activity and Nutrition Study) that could provide additional national-, state-, and local-level data and determine what support can be provided to states using existing data.
• Encourage state and local governments to develop the necessary infrastructure for creating data systems that will capture obesity prevention–related data below the national and, in some cases, state levels.
• Encourage states to disseminate existing data relevant to local jurisdictions and provide support to communities by developing community-level indicator estimates where data gaps remain.
Provide Common Guidance for Evaluation
Recommendation 3: Relevant federal agencies (e.g., in the U.S. Departments of Agriculture, Commerce, Health and Human Services, Labor, and Transportation) and state and local health departments, in collaboration with nonfederal partners, should standardize the collection and analysis of data, including common indicators, measures, methods, and outcomes used for assessment, monitoring, surveillance, and summative evaluation to assure aggregation among localities and back to the National Obesity Evaluation Plan.
To guide the implementation of this recommendation, potential actions to standardize the use of common indicators and measures include the following:
• Promote the use of sets of core indicators for assessment at the national, state, large community, and small community levels that, at a minimum, include indicators of obesity prevalence, physical activity, and nutrition to assess environmental and policy changes as recommended in the APOP report (see recommended indicators in Chapter 6, Table 6-3, for national- and state-level indicators and Chapter 7, Table 7-2, for large and small communities).
• Identify, develop, and disseminate a common measure for capturing each recommended indicator, maximizing feasibility and validity while minimizing cost and identifying what would be possible to collect at multiple levels—national, state, and local.
• Ensure that all federally funded grants and programs that include the recommended strategies to accelerate progress in obesity prevention (i.e., APOP report recommendations) include appropriate core indicators and common measures. Encourage similar metrics for research and summative evaluation funded by nongovernmental organization grants.
• Organizations that conduct mandated community assessments and surveillances should include the appropriate indicators recommended by the Committee (see Chapter 7, Table 7-2), including hospitals and their partners who are conducting required assessments and public health agencies who are meeting accreditation requirements.
To guide the implementation of this recommendation, potential actions to standardize the use of common methods and outcomes include the following:
• Create a standard national evaluation report template for assessing the progress of obesity prevention efforts that specifies obesity-related indicators, benchmarks, and subgroup analyses.
• In collaboration with the National Collaborative on Child Obesity Research, federal agencies (e.g., CDC, U.S. Department of Agriculture) should promote use of common tools and methods for measuring immediate and long-term outcomes.
• For community assessments and surveillance, promote the use of best practice templates for planning and implementing community assessments (see Box 10-3 and Chapter 7). Create a standard obesity community assessment and surveillance template that specifies obesity-related indicators, recommended analysis (e.g., subgroup, small areas, time trends), benchmarks and peer comparisons, and presentation format. It should include model language and provide a design template to minimize effort needed to produce reports. Specifications should include routine across-group comparisons (e.g., race, income) and comparisons with peer communities.
• Promote the use of best practice templates for community-level monitoring and summative evaluations of obesity prevention efforts (see Box 10-4 and Chapter 8), including the use of practical participatory engagement and use of a strong methodological study design and analyses. Document critical evaluation goals in a standardized format to improve the evidence base at levels practical for the resources of the community, including (1) monitoring and documenting implementation of policy, program, and environmental changes; (2) estimating collective impact of combinations of strategies including by characterizing and weighting their intensity (i.e., strength of intervention, reach, duration); and (3) measuring intermediate-term changes (e.g., in policies, systems, infrastructure, and capacity) and their association with long-term surveillance of population-level outcomes (e.g., behaviors related to physical activity and healthy nutrition, obesity).
• Examine new, alternative, and emerging methods of collecting data (e.g., real-time access of data from community-based organizations, crowd sourcing techniques, new technologies and handheld or worn devices, geographic information systems).
Improve Access to and Dissemination of Evaluation Data
Recommendation 4: Relevant federal agencies (e.g., in the U.S. Departments of Agriculture, Commerce, Health and Human Services, Labor, and Transportation) in collaboration with academics, nongovernmental organizations, and state and local health departments, should coordinate existing efforts to ensure that federal, state, and local assessment, monitoring, surveillance, and summative evaluation systems include a mechanism for feedback to users of evaluation data. In addition, local evaluations should continue to build the evidence base for the Accelerating Progress in Obesity Prevention report strategies; be stored, curated, synthesized, and shared to improve generalizable knowledge about implementation barriers and opportunities; and clarify “what works” in different contexts.
To guide the implementation of this recommendation, potential actions to improve access to and disseminate data include the following:
• Further develop or expand use of existing data warehouses and resource centers (e.g., HHS’s Health Information Warehouse, Data Resource Center for Child and Adolescent Health). The National Collaborative on Child Obesity Research is considering an evaluation registry that would describe and identify where major evaluation data can be obtained; this should be encouraged and expanded.
• Expand access to and increase functionality of data visualization tools (e.g., Community Commons, Data Resource Center for Child and Adolescent Health) so that users can add local data and produce charts and maps.
• Provide tools to use and access data for local data analysis, including support for generating synthetic estimates.
• Develop registries for gathering and disseminating the results of community-level evaluations of obesity prevention efforts.
• Create an online data entry, assessment, and monitoring system to support regular local food and physical activity environment, program, and policy scans.
Improve Workforce Capacity for Evaluation
Recommendation 5: The Centers for Disease Control and Prevention, National Institutes of Health, and the U.S. Department of Agriculture, through the National Collaborative on Child Obesity Research and other nongovernmental and professional organizations, should build on their existing evaluation resources to assure support for the diverse and inter-disciplinary workforce engaged in conducting assessments, surveillance, monitoring, and summative evaluation activities.
To guide the implementation of this recommendation, potential actions to improve workforce capacity include the following:
• Provide standardized training on planning and designing assessments, surveillance, monitoring, and summative evaluations for policy and environmental interventions, including the use of common indicators, measurement protocols, data collection methods, and the use of qualitative methods.
• Develop mechanisms for providing technical assistance for data access, statistical analysis, and reporting from state health departments, federal government (e.g., CDC), and nongovernmental organizations for states, territories, and local entities.
• Create a database of local evaluation expertise for use by stakeholders engaged in obesity prevention interventions.
• Link to a national network of knowledge brokers who can help to support and guide implementation of evaluation plans.
• Identify expertise at state and local universities and colleges for improved design and analysis of initiatives.
• Partner with state and local universities and professional organizations (e.g., National Association of County and City Health Officials) to offer online courses/webinars on how to conduct community assessments, surveillance, monitoring, and summative evaluations in different contexts.
• Strengthen university-community engagement and collaboration through community-based participatory assessments and participatory evaluation.
• Facilitate the development of resources provided to state and local health departments so they have necessary capacity to evaluate obesity prevention efforts.
• Increase skills in communicating findings with consumers, media, and decision makers in a relevant and understandable manner.
Improve Evaluations to Address Disparities and Health Equity
Recommendation 6: The U.S. Department of Health and Human Services in collaboration with nonfederal partners should increase its capacity to address health equity by practicing participatory and culturally competent evaluation, and it should standardize the collection, analysis, and reporting of data targeting disparities and health equity, and improve the accessibility of tools and methods for measuring social determinants that put populations at elevated risk for obesity.
To implement this recommendation,
• CDC as well as state and local health departments should strengthen assessment, monitoring, surveillance, and summative evaluation efforts through the following activities: (1) assure data samples are designed to allow analysis of differential and avoidable health outcomes related to race/ethnicity, income, and geographic sub-groups; (2) increase local data collection with an emphasis on disadvantaged populations and the differential exposures, vulnerabilities, and consequences that produce disparities; (3) make better use of data aggregation that allows for pooling of data across time and/or geographic area; (4) improve the methods for small area estimation that are often used to provide smoothed or synthetic estimates of risk; (5) standardize metrics to allow more effective pooling of data; (6) improve data collection methods via multi-method sampling (e.g., telephone, in-person, Internet); and (7) employ interviewers fluent in the language and culture of choice (for populations for whom English is not the first language).
• CDC, as well as state and local health departments, should strengthen assessment and monitoring of environmental conditions and systems relationships among them that produce disparities
through the following activities: (1) increase local data collection with an emphasis on differential exposures (e.g., access to healthful foods), vulnerabilities, and consequences; (2) adapt measures to allow for culturally appropriate foods, activities, and health-promoting environments; and (3) use methods for small-area analysis to examine associations between differential exposures/vulnerabilities and associated health disparities.
• The Secretary of HHS in collaboration with other federal agencies should (1) develop common conceptual and operational language, domain (i.e., individual dietary behavior, food environment, individual physical activity behavior, physical activity environment), and definitions to understand influences on disparities and health equity in obesity, taking into account expertise from multiple disciplines; (2) identify common tools (both qualitative and quantitative) for all target populations most at risk for obesity disparities across all levels of impact (e.g., individual, community, society); and (3) emphasize the quality of these recommended tools and methods for adapting them to specific contexts and systems.
• The National Collaborative on Child Obesity Research, a public-private partnership, should (1) identify best practices for both participatory and culturally competent evaluation; (2) expand the capability of its Registry to house and provide regular updates on core tools and methodologies to measure disparities and equity and improve the accessibility, utility, and dissemination of these tools; and (3) consider expanding the core tools and methodologies to include adults.
Support a Systems Approach in Evaluation
Recommendation 7: Evaluators, government, and private funders should incorporate a systems approach to evaluating obesity prevention efforts into their research-related activities through leadership, funding, and training support.
To implement this recommendation,
• Evaluators should embrace a systems approach—reflecting interactions among strategies in and across multiple sectors and levels—to guide their methods of research and evaluation of obesity prevention efforts.
• Government agencies should examine what combination(s) of indicators is most appropriate for evaluating progress in obesity prevention, focusing on categories of indicators that relate to the systems framework recommended in the APOP report.
• Government agencies and private organizations funding obesity prevention research and evaluation should (1) stimulate the use of systems science by integrating it into requests for proposals of research designed to address multi-sectoral, multi-level, and multi-component issues; (2) create requests for proposals that focus on systems science–based research in obesity prevention; and (3) stimulate research on the value proposition of a systems approach to obesity evaluation by creating calls for research that include numerous domains and elements to identify the “value” of community-based interventions (based on the recommendations of the Committee on Valuing Community-Based, Non-Clinical Prevention Policies and Wellness Strategies).
• Relevant federal agencies funding obesity prevention work (e.g., CDC, National Institutes of Health, Agency for Healthcare Research and Quality) should encourage and promote partner-
ships between federal/public and private organizations to train the evaluation workforce in the use of systems science for the purpose of obesity prevention evaluations by convening workshops, bringing together stakeholders, and providing pilot funding for developmental projects.
THE WEIGHT OF THE NATION MEASUREMENT IDEAS
TWOTN, a campaign produced by HBO and the IOM, is an illustrative contributor to a long-term national commitment to prevent obesity through policy and environmental changes. Along with many other events and vehicles, TWOTN attempts to make people aware of the obesity problem, raise their consciousness about policy and environmental forces that give rise to obesity, and, with the community screenings and school materials, engage them in strategies to address the problem. TWOTN utilized multiple components, including national (primarily the HBO television series and associated website) and community components (e.g., local screenings, school initiatives). According to its task, the Committee reviewed the components and goals/objectives of TWOTN and offered ideas for measurement of its impact.
Given the range of social media and advocacy efforts involved in TWOTN, it is a challenge to assess its contribution to the total mix among other components of the national efforts, let alone attribute change in physical activity, nutrition behaviors, or obesity to it; however, assessment of TWOTN can be illustrative of some of the challenges and opportunities that are inherent in evaluation of similar obesity prevention initiatives.
The initial national evaluation of TWOTN (see Box 6-3) will provide indicators of national dosage (or reach or exposure). The Committee concludes that further national-level evaluation is not warranted at this time because the extensive diffusion and secondary reach of the program has made comparison populations increasingly difficult to distinguish based on their exposure to elements of the campaign. The Committee provided some methods for national evaluation of future campaigns in Chapter 6. However, it will probably not be possible to disentangle TWOTN media campaign activities from other national and community activities that employ policy and environmental strategies to raise awareness and engage stakeholders in obesity prevention at this time.
Current community summative evaluation efforts by Prevention Research Centers (through University of North Carolina Chapel Hill) and Kaiser Permanente will provide indicators of local dosage or reach for TWOTN. Further community summative evaluations should be based on a logic-model approach. For example, if schools utilize TWOTN-derived products, such as the media kits and the three follow-on children’s film, then one might assess changes in knowledge about obesity before versus after viewing the film. The Committee emphasizes the importance of (1) using strong theoretical or logic models; (2) assessing reach or dosage, which is actually a critical step in the logic model for any health promotion program or mass media campaign; and (3) multiple waves of measurement, the more the better, preferably both before and after a campaign. Implementing these steps will require a commitment to resources for supporting the measurement of the community components of the campaign. Chapter 8 details these current efforts and suggests approaches to evaluating the community-level components of TWOTN.
The only real source of mass media data on reach, exposure, or dosage is from commercial sources (e.g., Nielsen). No government-sponsored repository of these data exists. This is distinct from most of
the other indicator areas, so it is worth highlighting as a potential area for strengthening the capacity for national evaluation of future media campaigns and media material for obesity prevention efforts (see Chapter 6).
In 2012, the IOM’s Accelerating Progress in Obesity Prevention: Solving the Weight of the Nation report provided recommended strategies and action steps for implementation by key stakeholders and sectors that individually have positive acceleration potential and that combined will create synergies that can further accelerate progress in preventing obesity over the next decade. These 20 strategies offer the focus for future evaluation efforts across the United States. This report builds on these strategies and offers an evaluation framework to inform and improve obesity prevention efforts. Key activities identified in the evaluation framework and provided in this report include obesity evaluation plans at the national and community levels that provide a tool for guiding the planning, implementation, and evaluation of obesity prevention efforts. A second key activity identified in the evaluation framework is a list of indicators and sources of data and provide a source of baseline data to begin to comprehensively assess obesity prevention actions already being implemented across the country. These indicators can be incorporated into the evaluation plans and will provide guidance for improving new targeted evaluations of the strategies recommended in the APOP report.
IOM (Institute of Medicine). 2012. An integrated framework for assessing the value of community-based prevention. Washington, DC: The National Academies Press.
Pronk, N. P., L. M. Hernandez, and R. S. Lawrence. 2013. An integrated framework for assessing the value of community-based prevention: A report of the Institute of Medicine. Preventing Chronic Disease 10:120323.
This page intentionally left blank.