10

Taking Action: Recommendations for Evaluating Progress of Obesity Prevention Efforts

The importance of evaluating the relative impact of obesity efforts is well known, and the challenges are not new. Those challenges remain, however, far from resolved. Even modest improvement in evaluation has the potential to provide clarity and refined direction in addressing the obesity epidemic. This report provides flexible evaluation plans for the national, state, and community levels (including indicators of status and progress) that can be implemented now. It also recommends a number of infrastructure changes at the national level that could make these evaluations even more effective in the future.

With funding from the Michael & Susan Dell Foundation, the Institute of Medicine (IOM) Committee on Evaluating Progress of Obesity Prevention Efforts was formed to develop a concise and actionable plan for measuring progress in obesity prevention efforts for the nation and adaptable guidelines for local community evaluation. The Committee’s assigned tasks were to

1.   draw on the recommendations and recommended indicators of progress from the IOM committee and 2012 report, Accelerating Progress in Obesity Prevention (APOP); consider existing and new tools and metrics (e.g., trend analysis, community/local measures) to measure progress; and develop a plan for a national-level evaluation of obesity prevention efforts by sector and, if appropriate, across sectors;

2.   develop a community-level measurement plan that adds detail and supports the national-level evaluation plan; and

3.   identify measurement ideas that can determine the specific impact of the Home Box Office (HBO)/IOM campaign called The Weight of the Nation (TWOTN).

The intended audiences for the report’s recommended plans and measurement ideas are decision makers, community members, researchers, and evaluators at all levels and across all sectors of society.



The National Academies | 500 Fifth St. N.W. | Washington, D.C. 20001
Copyright © National Academy of Sciences. All rights reserved.
Terms of Use and Privacy Statement



Below are the first 10 and last 10 pages of uncorrected machine-read text (when available) of this chapter, followed by the top 30 algorithmically extracted key phrases from the chapter as a whole.
Intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text on the opening pages of each chapter. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.

Do not use for reproduction, copying, pasting, or reading; exclusively for search engines.

OCR for page 273
10 Taking Action: Recommendations for Evaluating Progress of Obesity Prevention Efforts T he importance of evaluating the relative impact of obesity efforts is well known, and the challenges are not new. Those challenges remain, however, far from resolved. Even modest improvement in evalu- ation has the potential to provide clarity and refined direction in addressing the obesity epidemic. This report provides flexible evaluation plans for the national, state, and community levels (including indica- tors of status and progress) that can be implemented now. It also recommends a number of infrastructure changes at the national level that could make these evaluations even more effective in the future. With funding from the Michael & Susan Dell Foundation, the Institute of Medicine (IOM) Committee on Evaluating Progress of Obesity Prevention Efforts was formed to develop a concise and actionable plan for measuring progress in obesity prevention efforts for the nation and adaptable guide- lines for local community evaluation. The Committee’s assigned tasks were to 1. draw on the recommendations and recommended indicators of progress from the IOM com- mittee and 2012 report, Accelerating Progress in Obesity Prevention (APOP); consider existing and new tools and metrics (e.g., trend analysis, community/local measures) to measure progress; and develop a plan for a national-level evaluation of obesity prevention efforts by sector and, if appropriate, across sectors; 2. develop a community-level measurement plan that adds detail and supports the national-level evaluation plan; and 3. identify measurement ideas that can determine the specific impact of the Home Box Office (HBO)/IOM campaign called The Weight of the Nation (TWOTN). The intended audiences for the report’s recommended plans and measurement ideas are decision makers, community members, researchers, and evaluators at all levels and across all sectors of society. 273

OCR for page 273
The Committee relied on specific definitions for commonly used terms in this report. Interventions refer to policies, programs, systems, environmental changes, services, products, or any combination of these multi-faceted initiatives. Assessment is an effort to use data on the community or other jurisdic- tion to characterize the problem, its distribution, and efforts to address it. Monitoring is the tracking of the implementation of interventions compared to standards of performance. Surveillance is the ongoing systematic, collection, analysis, and interpretation of data that are tracked over time to detect patterns, disparities, and changes that may be associated with interventions or other causes. Summative evaluation is the effort to detect changes in output, outcomes, and impacts associated with the interventions and to attribute those changes to the interventions. In this report, the Committee sometimes uses the term evalu- ation to refer to all four of these functions. An indicator is a source of data or evidence that can be used to assess the status or trend of a person or population (a measurement, e.g., prevalence of obesity). Obesity Evaluation Plans Evaluation plans (for assessment, monitoring, surveillance, and summative evaluation activities) are tools that contain guidance for planning, implementing, and evaluating obesity prevention efforts. A comprehensive evaluation plan can guide decision makers and users responsible for developing or funding evaluations to measure progress in national obesity prevention efforts. Community-level obesity evalua- tion plans can similarly support identification of key components in implementing evaluations at the local level. They offer guidance at the community level that is sensitive to local variation in needs, context, and resources, and they can help to support aggregation and dissemination of information across communities. As described in the evaluation process framework in Figure 10-1 (detailed in Chapter 3), an evalu- ation plan is a key activity (found in Box 2 of Figure 10-1) that provides guidance for organizing and implementing evaluation-related efforts to achieve the intended outputs, outcomes, and impacts identified in the evaluation framework. The successful implementation of the national and community evaluation plans recommended in this report will require the support of other activities, including a core set of indicators and common mea- sures of success; resources for training, technical assistance, and dissemination; and an adequate evalua- tion infrastructure. To support these activities, the Committee (1) identifies existing indicators of progress that can be incorporated into the recommended plans, can be helpful in identifying gaps in existing data and information systems, and can provide examples of indicators that can be used by evaluators seeking to evaluate obesity prevention interventions and (2) recommends actions that will improve leadership and coordination, guidance, capacity, and infrastructure for evaluation efforts. Actions to support and implement the plans and other activities will improve evaluation capaci- ties in the short-term (e.g., use of a core set of existing indicators), increase evaluation activities in the intermediate-term (e.g., improve capacity and guidance), and enhance data use in the long-term to assess population-level changes and improvements that can result from widespread implementation of evidence- based interventions to prevent obesity (i.e., outcomes, Box 4 in the evaluation framework). Conclusions and recommendations Along with the framework in Chapter 3, the Committee details key findings in this report on dimensions of national- and community-level evaluation, including the information/data needs of those 274 Evaluating Obesity Prevention Efforts

OCR for page 273
1. Inputs • User/Stakeholder Needs • Existing Objectives Strategies • Context for Evaluation a • Guiding Principles for Evaluation a • Resources 5. Intended Impacts/ Improvements 2. Activities Evaluation • Identify Indicators/Measures of Success • Partnerships, Environments, Policies, Behaviors, Norms Framework • Develop Evaluation a Plans and Infrastructure • Energy Expenditure/Intake • Overweight/Obesity • Develop Resources for Training, Technical • Population Health and Well-Being; Equity Assistance, and Dissemination 4. Outcomes 3. Outputs • Short-term: Improved Evaluationa Capacity and Training • Core Indicators and Measures • Intermediate-term: Increased • Recommendations and Guidance Evaluationa Activities • Support for Implementation • Long-term: Enhanced Data Use FIGURE 10-1  Framework for evaluating progress of obesity prevention efforts. a Evaluation refers to assessment, monitoring, surveillance, and summative evaluation activities. NOTE: An Evaluation Plan is an activity listed in Box 2. Figures S-1, 4-1 and 10-1.eps i ­nterested in obesity prevention and its results (Chapter 2), indicators that can serve as markers for assessing the progress of obesity prevention efforts recommended in the APOP report (Chapter 4), the infrastructure and capacity to support evaluation, and methods and protocols for conducting evaluations (Chapters 5, 6, 7, and 8), and a systems approach to obesity prevention evaluation efforts (Chapter 9). Based on these findings, the Committee has identified seven broad conclusions that serve as the context for the development of and guidance provided in the recommended plans and supporting actions that f ­ollow (see Box 10-1). This chapter describes (1) indicators that are aligned with the recommendations in the APOP report from readily available data sources; (2) a National Obesity Evaluation Plan; and (3) a Community Obesity Evaluation Plan, and it recommends seven actions to support the implementation of the recommended plans. In the final part of the chapter the Committee identified measurement ideas for determining the impact of the HBO/IOM TWOTN campaign. Taking Action 275

OCR for page 273
BOX 10-1 Broad Conclusions Informing Plans for Evaluation Regarding Obesity Prevention Efforts • There is a pressing need to act on the problem of obesity, but there are gaps in the certainty of the effectiveness of actions or mixture of actions being implemented across the country. Systematic and comprehensive evaluations along with more routine assessments, monitoring, and surveillance offer valuable guidance for improving the quality and outcomes (or impact) of the actions being i ­mplemented and for defining the direction of further basic and implementation research. • Information generated from current obesity prevention evaluation efforts, other than assessment of needs at the national and state levels, does not always address the needs and interests of the users of this information, often because of limited or outdated data (especially at the community level) and few presentations of the data in useful and timely formats. • Current data (monitoring) systems do not adequately track progress of environmental and policy-­elated r obesity prevention actions or systems changes recommended in the Accelerating Progress in Obesity Prevention report (IOM, 2012a). Such monitoring is needed at both the national and com­ unity levels, m especially for populations at greatest risk for obesity. These limitations exist primarily because monitoring systems have traditionally focused on measuring individual behaviors, energy expenditure/energy intake, and overweight and obesity. • Current investment in evaluation is too low and sporadic, presenting serious barriers to understanding the impact of and need for future investments in implementing interventions. • A systems science approach to evaluation can help evaluation users identify and select combinations of actions and strategies to implement in multiple sectors, and at multiple levels, with available resources. • Although many data systems exist, the current national systems for monitoring progress of recom- mended obesity prevention actions and for surveillance of their effects on obesity lack adequate leader- ship, coordination, infrastructure, guidance, accountability, and capacity. • Communities lack adequate guidance, capacity, data, and resources necessary for assessing the status of obesity and its determinants, identifying prevention needs, monitoring obesity prevention actions, evaluating their short-term outcomes, and tracking (through surveillance) their long-term association with obesity reduction in the aggregate and differences among population segments. Indicators of Progress One clear gap in evaluation efforts is the identification of a set of core indicators to use at the national and community levels for measuring progress in obesity prevention efforts. As a key first step in identifying this core set, the Committee identified a list of indicators that currently exist. Based on available and ongoing data sources, the Committee identified several overarching and goal area–specific 276 Evaluating Obesity Prevention Efforts

OCR for page 273
indicators that were best aligned with the recommendations included in the APOP report. The 83 indica- tors identified provide a menu of possible indicators for use by evaluators. This exercise produced a list of indicators that can act as a starting point for the development of core indicators and related measures as well as for the identification of gaps in the existing data systems that can be incorporated into evaluation plans (national and community levels), and provide guidance to improve long-term evaluation infrastruc- ture and capacities. In the short term, evaluators of obesity prevention programs, policies, and environ- ments can use the indicators identified by the Committee to begin to comprehensively assess obesity pre- vention actions already being implemented across the country. Of particular importance to the Committee was recognition that evaluating progress for the nation as a whole, and for regions of the nation, requires special attention to the disparities that are associated with the obesity epidemic. Although numerous challenges remain, the Committee found a small yet grow- ing literature of tools and methodologies for monitoring progress toward obesity prevention among popu- lations with health disparities. A National Evaluation Plan for Obesity Prevention A discrepancy exists between the importance and magnitude of the obesity problem and the level of action in the United States for developing a cohesive plan to evaluate implementation of efforts across the country and assess their impact. Currently U.S. efforts lag behind those in other countries to provide common guidance, support, and the appropriate infrastructure to support evaluation efforts. Although the Committee identified important strengths of the current monitoring, surveillance, and summative evalua- tion data systems, limitations of current national evaluation efforts exist, the following needs resulting in: • coordinating leadership, integration, and accountability of evaluation efforts across federal agencies, within and between departments, across federal, state, and local governments, and with the nongovernmental and private sectors; • maximizing and coordinating the use of data already being collected; • identifying and prioritizing indicators at the national and community levels and developing new indicators where necessary; • improving surveillance capacity and frequency, especially for policies and environmental factors, and evaluation capacity/leadership; • improving training and support for monitoring, surveillance, and evaluation; • improving access to and dissemination of data, findings from analysis, and other information for the consumer; • collecting additional longitudinal data, including national incidence trends and local prevalence ­ trends for obesity; • tracking and monitoring disparities and their social determinants (i.e., differential exposures/ opportunities, vulnerabilities/capabilities, and consequences); and • using best practices of evaluation design, including monitoring and feedback on progress on intermediate outcomes (i.e., community/system changes such as new policies, expanded pro- grams, and environmental changes). Taking Action 277

OCR for page 273
The Committee developed a National Evaluation Plan for Obesity Prevention (see Box 10-2) for the United States. The National Obesity Evaluation Plan is designed to organize the planning, implementa- tion, and evaluation of the impact of obesity prevention interventions recommended in the APOP report at a national level. It is important to note here that the APOP report frames obesity prevention efforts in terms of policy, systems-level, and environmental approaches, which require new evaluation approaches, ­ indicators, and measures. Box 10-2 identifies key components for organizing a National Obesity Evaluation Plan. Activities for achieving the objectives are outlined here; Chapter 6 provides detailed guidance to address each activity. This plan is intentionally broad to provide a starting point for the development, use, and support of core indicators (derived from the list in Chapter 4 organized around the APOP recommendations) and recommended methodologies, as well as flexibility for future innovations. The Committee stresses that the National Obesity Evaluation Plan activities should be prioritized to lever- age existing resources that maximize the use of existing resources and efficiency of data collection and avoid duplication of efforts. Seven recommended actions to improve the national evaluation infrastructure necessary for implementing the plans are offered in a later section of this chapter. The National Evaluation Plan for Obesity Prevention can be used as a model for state and multi- state regional evaluations. By using the National Obesity Evaluation Plan as a blueprint, states will be able to provide comparable data that can be used as benchmarks for state progress, when monitored over time, as well as when compared to other state and national data. However, state-level evaluation activities should be flexible enough to adapt to unique populations and regional characteristics requiring changes in measurement protocols or instruments. While a National Obesity Evaluation Plan can show changes in general trends over time, state-level plans have the potential to identify success stories using APOP strate- gies that can be disseminated broadly to accelerate obesity prevention progress. The Community Evaluation Plan for Obesity Prevention The Committee provides guidance for communities that are implementing or intend to implement obesity prevention interventions. The guidance, in the form of the Community Obesity Evaluation Plan consists of two distinct sets of activities: (1) community assessment and surveillance to describe the cur- rent health status, resources, and determinants of health in a community (assessment) and to track them over time (surveillance) and (2) more tailored community intervention summative evaluations that seek to establish and share what is being tried and implemented (monitoring) and to identify the effectiveness of local efforts to prevent obesity (summative evaluation). Together these activities provide baseline data and “diagnostic” data on the state of obesity and related “determinants” or conditions in the community. They offer opportunities to establish and share “what works.” Many times, information captured locally is specifically tailored data that cannot be captured at the national level; community-level evaluation activities (assessment, surveillance, monitoring, and summative evaluation) provide an essential additional level of detail and local context-specific information that the National Obesity Evaluation Plan cannot. The learnings derived from this local information will allow greater return on national investments in obesity prevention as well as inform refinements to the national evaluation plan. 278 Evaluating Obesity Prevention Efforts

OCR for page 273
BOX 10-2 Core Components and Activities of the National Plan for Evaluating Progress in Obesity Prevention Purpose: To evaluate progress at the national level in implementing strategies from the Institute of Medicine Accelerating Progress in Obesity Prevention (APOP) report and in achieving intended impacts as described in the Evaluation Framework (see Box 5 in Figure 10-1). Components: 1. Identify leadership, infrastructure, resources, priorities, and timeline for implementing the plan. 2. Identify current national efforts for evaluation, including indicators (Chapter 4), and incorporate them selectively into national monitoring, surveillance, and summative evaluation data systems that are responsive to the needs of data users. 3. Propose data and infrastructure to add to existing monitoring and surveillance systems to fill gaps, and facilitate community obesity evaluation plans. 4. Propose additional assessment, monitoring, surveillance, and summative evaluation activities, new m ­ easures, and innovative strategies to implementation in the future. 5. Outline mechanisms for feedback to data users, assuring accessibility, privacy, and cost-efficiency. 6. Detail adaptations of the plan at the state level, with further applications at the regional level. Activities (see seven recommended actions for implementation): 1. Designate a federal obesity evaluation task force or entity to oversee the implementation of the National Obesity Evaluation Plan and coordinate with relevant federal, state, local, and private-sector entities. a. Identify and obtain the infrastructure necessary for implementing the plan and coordinate with appropriate partners b. Ensure adequate benchmarks/goals, including a schedule for updates c. Establish a process for prioritization, accountability, and adaptation of plan activities including an annual report to the agency responsible for leading the effort d. Identify priorities and create an ongoing timeline for implementing the plan i. Short-term objectives achievable within 1-3 years ii. Intermediate-term objectives achievable within 3-5 years iii. Long-term objectives achievable for 5 years or longer continued Taking Action 279

OCR for page 273
BOX 10-2 Continued 2. Identify current national evaluation efforts, including indicators for monitoring and surveillance systems to minimize duplication, maximize use of data already being collected, and identify priorities to address evaluation gaps in a coordinated fashion. a. Use the indicator list (Chapter 4) as a starting point to identify a core set of indicators b. Match indicators as much as possible for common measurement across jurisdictions c. Examine existing links to the Leading Health Indicators and other recommendations as consistent with APOP d. Promote use of common measures through the National Collaborative on Childhood Obesity Research (NCCOR) (see Chapter 5) to facilitate harmonization of data across data-collection systems e. Expand School Health Policies and Practices Study to include measures of additional settings such as worksite, child care centers, and schools on a rolling basis every 3 years rather than current settings every 6 years f. Expand National Health and Nutrition Examination Survey (NHANES) sampling, analyses, and/or reporting to address gaps in developmental levels of children birth to 1 year, 2 to 5 years, 6 to 10 years, 11 to 13 years, and 14 to 19 years g. Expand NHANES to oversample populations that are underserved or at greater risk for obesity h. Standardize currently collected data and planned systems, such as electronic health records, for data aggregation i. Incorporate data from birth certificates, Special Supplemental Nutrition Program for Women, Infants, and Children (WIC), Early Head Start, and Head Start into the National Obesity Evaluation Plan j. Expand current monitoring and surveillance structures into existing data-collection systems at the national or state levels 3. Develop new data-collection infrastructure or systems, indicators, and measures to address gaps identified as priorities in areas such as policy and environment, physical activity, child care centers, worksites, health plans, federally qualified health centers, and community health centers/WIC clinics. Community Assessments and Surveillance Community assessment provides first-time assessment of status or trends overall in a community. Surveillance provides repeated or continuous assessment of progress overtime. Specific to this report, these data can assess what is being done in a community and reveal the factors that influence local obesity prev- alence and incidence (e.g., demographics, social determinants). 280 Evaluating Obesity Prevention Efforts

OCR for page 273
4. Increase national and state capacity for assessment, monitoring, surveillance, and summative evaluation. a. Standardize and provide training on measurement protocols (e.g., body mass index, waist circumfer- ence) and data-collection methods b. Provide technical support for data utilization, statistical analysis, and reporting i. Assess the impact of the data loss that resulted from discontinuation of the Centers for Disease Control and Prevention’s Pediatric Nutrition Surveillance System and Pregnancy Nutrition Surveillance System (state- and county-level data) and provide ongoing technical assistance to states that use existing data c. Create lists of recommended standardized tools and methods for measurement i. Expand and maintain the NCCOR Surveillance System and Measures Registry 5. Ensure that all relevant data systems include a mechanism for relevant and timely feedback to data users. a. Expand Health Indicators Warehouse and other interactive sources of federal-level data b. Expand and maintain Community Commons c. Develop additional “dashboards” and “federal report card” formats that can be interactive and dis- play data in easily understood infographics and tables 6. Ensure that evaluation plans in federally funded obesity-related grants and programs include common indicators and measures that can be aggregated across communities and inform the plan. 7. Encourage development and testing of alternative and emergent methods of collecting data, including a. Real-time access of data from community-based organizations b. Capitalization on the “quantified self” movement c. Use of new technologies and geospatial modeling Although the Committee identified several resources available to aid communities across the country, currently there is no consensus guidance for what indicators to measure or methodologies to use when con- ducting obesity-focused community assessment and surveillance. Based on a review of the existing infrastruc- ture for conducting obesity-focused community-level assessment and surveillance, the Committee found • a lack of data available at the local level for indicators relevant to measuring progress of APOP strategies. Especially needed are data for preschoolers and elementary school children, and sys- Taking Action 281

OCR for page 273
tematic descriptions of determinants of obesity (e.g., environments, policies, other interventions, norms, and attitudes). Additional sources of data at the local level may exist in multiple sectors, such as health care, planning, and schools; • a need to increase sample size of existing surveillance systems, add data on missing indicators, and develop new systems for policy, environmental, intervention indicators; and • report data by race and socioeconomic status to the extent possible and by small areas affected by inequity for larger communities. Other important findings include the following: • There is a lack of a common set of indicators to allow cross-community comparisons and aggregation. • Engaging stakeholders/community in assessment process is valuable. • Capacity to develop assessments varies widely across communities. • Improving the accessibility and dissemination of assessment data through multiple channels will improve their use for decision makers, media, and the public. The intent of the Community Obesity Assessment and Surveillance Plan (see Box 10-3) is to provide guidance for local communities to identify and use a set of core indicators that measure obesity-related outcomes and impacts and to develop local capacity for these assessments, including common use and understanding of assessment protocols, descriptions of health disparities, community engagement, over- sight, and public reporting on progress. The Committee developed the Community Obesity Assessment and Surveillance Plan to not only accommodate communities with varying resources and assets, but also provide a core set of indicators that can be measured comparably and aggregated across multiple jurisdic- tions. Recommended actions to support the development of local infrastructure and capacities for com- munity assessments and surveillance are offered in a later section of this chapter. These enhanced capaci- ties will lead to greater use of data and findings to inform local obesity prevention efforts, guide resource allocation, and engage stakeholders. Monitoring and Summative Evaluation of Community Interventions Community evaluations are critically important to developing knowledge about effective local interventions and for realistic implementation of local, state, and federal policies and funding initiatives. Community-level evaluation encompasses learning “what works” and also the relative effectiveness of various strategies—the extent to which they work (summative evaluation). In line with “what works,” summative evaluation also informs local implementers about ways to improve and manage interventions. It casts light on how and why these interventions may prevent obesity. Finally, it encompasses translating the effective interventions to be implemented on a broader scale and determining the contexts in which they are and are not effective (monitoring). Box 10-4 identifies key components to develop and implement a Community-Level Obesity Intervention Monitoring and Summative Evaluation Plan. Considerable flexibility is needed within these components. As outlined, the core of any plan includes engaging stakeholders, identifying resources, hav- ing a logic model or theory of change, selecting the right focus, using appropriate measures, collecting ­ 282 Evaluating Obesity Prevention Efforts

OCR for page 273
BOX 10-3 Components of a Community Obesity Assessment and Surveillance Plan Purpose: To provide accurate and timely knowledge of local obesity-related conditions and relevant c­ anges or trends over time as a result of implementing strategies in the Accelerating Progress in Obesity h Prevention (APOP) report. 1. Define community boundaries. a. Create specific geographic areas that reflect jurisdictions, key stakeholders, and community m ­ embers’ perceptions of geographic boundaries. 2. Engage community members and other key stakeholders. a. Include stakeholders to the extent possible, in defining community, identifying priorities, planning assessment, collecting data, interpretating and sense-making of results, and disseminating the findings. 3. Plan assessment/surveillance and include stakeholders and community members. a. Identify lead agency or agencies responsible for conducting assessment/surveillance. b. Clarify goals of assessment/surveillance. c. Define audience and what information will move it to action. d. Define topics to include in assessment/surveillance. e. Identify sub-populations and small areas disproportionately affected by obesity, and develop approach to collecting information about them. f. Select local data to be included about context, assets, interventions, barriers, and social determi- nants, and which data to schedule for ongoing surveillance. 4. Collect data. a. Obtain existing data from Web-based platforms or published reports. b. As resources permit, add other sources of data. c. Create an inventory of local obesity prevention interventions. 5. Analyze and make sense of the data. a. Include trends over time. b. Present data for infants, children, adolescents, adults, and special populations. c. Describe variation in indicators (e.g., across race/ethnicity/socioeconomic status/small areas). continued Taking Action 283

OCR for page 273
high-quality data, using appropriate analytic methods, engaging in sense-making, and disseminating findings. Chapter 8 provides detailed support and guidance for implementing each component of the Community-Level Obesity Intervention Monitoring and Summative Evaluation Plan. Recommended actions to support the development of local infrastructure and capacities for community-level intervention monitoring and summative evaluation are posited in a later section of this chapter. Finally, the National Obesity Evaluation Plan and the Community Obesity Evaluation Plan are interdependent. The two plans have the potential to provide essential support and feedback to each other. Successful implementation of the community plan is supported by the components of the national plan, using indicators, sources of data, resources, and methodologies coordinated and developed with leader- ship at the national level. However, the community plan and its associated activities also provide an addi- tional level of detail and local context-specific information that the national plan cannot measure. Indeed large-scale community-level evaluation efforts, which are intended to identify effective strategies that can be brought to scale in other communities, are already under way. Considerations for Investing in Obesity Evaluation Plans Evaluating obesity prevention is complex, and so is valuing the effort. The evaluation of obesity prevention may be a challenging proposition to implement when the gaps and recommendations identified by this report are considered. These challenges include financial resources, political factors, and different points of view of where to invest [scarce] resources. However, prevention of obesity may bring value to many stakeholders in the community and to society as a whole. As such, it would be useful to consider a valuation framework that brings transparency and legitimacy to the decision-making process of whether to invest in evaluation resources. The IOM recently published a report titled An Integrated Framework for Assessing the Value of Community-Based Prevention (IOM, 2012) that provides a blueprint for such a process. Using the proposed framework will allow local, regional, state, and national stakeholders to deal with reasonable disagreement and, in cases where such disagreement persists, identify and address poten- tial legitimacy problems (Pronk et al., 2013). Taking Action to Support the National and Community Obesity Evaluation Plans Using the considerable number of indicators available through federal, state, and community efforts identified in this report (Chapter 4) and guided by methodologies and protocols outlined in the plans as a guide (Chapters 5, 6, 7, and 8), researchers, communities, and policy and other decision makers can take immediate action to begin comprehensive assessment of obesity prevention efforts recommended in the APOP report and already under way across the country. As the study progressed and the Committee’s ideas matured, it became clear that the evaluation plans recommended by the Committee will not be fully realized without organizational changes and sup- port across multiple federal, state, and local government agencies and departments in collaboration with other nonfederal partners responsible for obesity prevention–related activities. Given the existing gaps identified by the Committee, the following recommended actions will support the successful implementa- tion of the components of the evaluation plans and will assure timely and meaningful data to inform and improve obesity prevention efforts at national, state, and community levels. The seven recommendations include aspects of leadership; data collection; guidance for identifying and using common indicators, mea- 286 Evaluating Obesity Prevention Efforts

OCR for page 273
sures, methods, and outcomes; dissemination of the information collected; workforce capacity develop- ment; assessment of disparities and health equity; and a systems approach to evaluation. The Committee offers a set of potential actions to guide each recommended action to fill existing gaps in the current evaluation infrastructure. The resources needed to implement some of the recommendations (given the set of potential actions) range from minimal to substantial; some are occasional, and others require frequent to continuous mea- surement. As described in the prior section this means that some of the Committee’s recommendations to support implementation of the evaluation plans call for leadership and expenditures that will require trade-off decisions by government, organizations, and the private sector, with astute use of existing resources and prioritization of other necessary actions implemented with short-, intermediate-, and long- ­ term time perspectives. Improve Leadership and Coordination for Evaluation The Committee believes that centralized leadership is necessary to coordinate the planning, imple- mentation, and evaluation of the impact of obesity prevention efforts across the country. As described in Chapter 6, most of the existing data collection and support for evaluation exists across multiple federal agencies.1 The current decentralized structure provides limited authority, responsibility, or support and coordination for these efforts at the national level. The Committee views that gap in empowered leader- ship in coordinating resources for evaluating obesity prevention efforts at the federal level as a major obstacle to measuring obesity prevention efforts. Progress could be made if a federal task force or entity would take a leadership position in this coordination effort. A number of relevant task forces/entities could serve in this coordination role. The Committee believes that one or a combination of these would be the best option for overseeing and implement- ing the National Obesity Evaluation Plan and for reporting to whatever agency is leading these efforts. Alternatively, the appointment of a new task force could also successfully address the need for improved leadership of evaluation of obesity prevention efforts, but the committee does not view it as necessary. It was not in the Committee’s charge or in its expertise to analyze various options and recommend a spe- cific entity to take on this responsibility (e.g., Department of Health and Human Services [HHS] Healthy Weight Task Force, National Prevention Council, National Collaborative on Child Obesity Research, Interagency Committee for Human Nutrition Research, National Committee for Vital and Health Statistics). Recommendation 1: An obesity evaluation task force or another entity should oversee and implement the National Obesity Evaluation Plan and provide support for the Community Obesity Evaluation Plan and should coordinate with other federal, state, and local public- and private-sector groups and other stake- holders who support, use, or conduct evaluations. The task force/entity could be a new or existing entity or a combination of existing entities. 1  Includes, but is not limited to, efforts in the following federal agencies: Corporation for National and Community Service; Departments of Agriculture, Commerce, Defense, Education, Health and Human Services, Interior, Labor, Transportation, and Veteran Affairs; Domestic Policy Council; Environmental Protection Agency; Federal Trade Commission; General Services Administration; and Office of Management and Budget. Taking Action 287

OCR for page 273
The task force that oversees the National Obesity Evaluation Plan will have the following key roles: • Identify and secure the infrastructure (i.e., effective leadership structure) necessary for imple- mentation of the National Obesity Evaluation Plan • Coordinate with appropriate federal partners and include representatives from major stake­ holder groups (e.g., child care settings, schools, worksites, local and state government, public health departments, business/private sector, and communities) • Ensure adequate national benchmarks/guidelines/goals (e.g., Dietary Guidelines for Americans, Healthy People 2020 objectives, Physical Activity Guidelines for Americans, reports of the U.S. Surgeon General) • Create an ongoing timeline for implementation of the activities outlined in the National Obesity Evaluation Plan • Establish a process for accountability, prioritization, and adaptation by agencies reporting p ­ eriodically to the task force/entity on their activities, and the task force/entity reporting annu- ally to the agency that is leading these efforts on coordination efforts, gaps in monitoring, r ­ ecommendations for new measures and evaluations, and progress toward meeting goals. Improve Data Collection for Evaluation Recommendation 2: Using the recommended indicators and gaps identified in this report as guides (i.e., related to Accelerating Progress in Obesity Prevention report strategies), all federal agencies2 and state and local health departments responsible for collecting data relevant to obesity prevention efforts, in coordination with relevant private partners, should identify, coordinate, and maximize current efforts for ongoing collection of recommended indicators and, according to the priorities identified, address existing evaluation gaps at the national and local levels. To guide the implementation of this recommendation, potential actions to coordinate efforts and address gaps include the following: • Examine all relevant national survey activities and harmonize existing efforts (e.g., use of c ­ ommon metrics) across the federal agencies (see Appendix Table D-1). • Identify linkages among current U.S. efforts identified in Recommendation 1 (and those of World Health Organization and European Union obesity-related evaluation plans) to enhance multinational coordination, comparison, and efficiency of evaluation plans. • Ensure the ongoing collection and maintenance of existing data systems, leverage their use, and increase their capacity through connection and computing technology. • Standardize currently collected data and data systems (e.g., electronic health records, data from departments of motor vehicles) to make aggregation and comparison of data feasible. 2  Agricultural Research Service, Economic Research Service, and Food and Nutrition Service of the U.S. Department of Agriculture; Census Bureau of the U.S. Department of Commerce; Agency for Healthcare Research and Quality, Centers for Disease Control and Prevention, Health Resources and Services Administration, and National Institutes of Health of the U.S. Department of Health and Human Services; Bureau of Labor Statistics of the U.S. Department of Labor; and Federal Highway Administration of the U.S. Department of Transportation. 288 Evaluating Obesity Prevention Efforts

OCR for page 273
• Expand existing surveys or develop new monitoring and surveillance systems (through other governmental funding or private funds or partnerships with commercial data providers) to address gaps at the national level, including increased frequency of key existing surveys; improved sampling or analysis and reporting for key intervention priorities in existing data surveys (all developmental levels of children and adolescents, populations at greater risk of o ­ besity); in areas such as school, child care centers, worksites, health plans, clinics; increased sharing and use of commercially available data; developmental and implementation of policy and environmental indicators; cataloguing of interventions; and longitudinal data systems to monitor the incidence (i.e., new cases) of obesity (see Chapter 6, and Chapter 4, Table 4-4, for detailed list of gaps). • Build, connect, and strengthen existing data systems and form partnerships to improve the avail- ability of existing data to local jurisdictions, including improving sampling methodology and size; partnering with schools, health plans, and businesses to collect and make available rel- evant information to local jurisdictions; and including policy and environmental indicators (see Chapter 7 and Chapter 4, Tables 4-1 and 4-4). • Assess the loss of data from discontinued or one-time surveillance systems (e.g., Centers for Disease Control [CDC] and Prevention’s Pediatric and Pregnancy Nutrition Surveillance Systems, National Youth Physical Activity and Nutrition Study) that could provide additional national-, state-, and local-level data and determine what support can be provided to states using existing data. • Encourage state and local governments to develop the necessary infrastructure for creating data systems that will capture obesity prevention–related data below the national and, in some cases, state levels. • Encourage states to disseminate existing data relevant to local jurisdictions and provide support to communities by developing community-level indicator estimates where data gaps remain. Provide Common Guidance for Evaluation Recommendation 3: Relevant federal agencies (e.g., in the U.S. Departments of Agriculture, Commerce, Health and Human Services, Labor, and Transportation) and state and local health departments, in c ­ ollaboration with nonfederal partners, should standardize the collection and analysis of data, includ- ing common indicators, measures, methods, and outcomes used for assessment, monitoring, surveillance, and summative evaluation to assure aggregation among localities and back to the National Obesity Evaluation Plan. To guide the implementation of this recommendation, potential actions to standardize the use of common indicators and measures include the following: • Promote the use of sets of core indicators for assessment at the national, state, large community, and small community levels that, at a minimum, include indicators of obesity prevalence, physi- cal activity, and nutrition to assess environmental and policy changes as recommended in the APOP report (see recommended indicators in Chapter 6, Table 6-3, for national- and state-level indicators and Chapter 7, Table 7-2, for large and small communities). Taking Action 289

OCR for page 273
• Identify, develop, and disseminate a common measure for capturing each recommended indica- tor, maximizing feasibility and validity while minimizing cost and identifying what would be possible to collect at multiple levels—national, state, and local. • Ensure that all federally funded grants and programs that include the recommended strategies to accelerate progress in obesity prevention (i.e., APOP report recommendations) include appropri- ate core indicators and common measures. Encourage similar metrics for research and summa- tive evaluation funded by nongovernmental organization grants. • Organizations that conduct mandated community assessments and surveillances should include the appropriate indicators recommended by the Committee (see Chapter 7, Table 7-2), including hospitals and their partners who are conducting required assessments and public health agencies who are meeting accreditation requirements. To guide the implementation of this recommendation, potential actions to standardize the use of common methods and outcomes include the following: • Create a standard national evaluation report template for assessing the progress of obesity pre- vention efforts that specifies obesity-related indicators, benchmarks, and subgroup analyses. • In collaboration with the National Collaborative on Child Obesity Research, federal agencies (e.g., CDC, U.S. Department of Agriculture) should promote use of common tools and methods for measuring immediate and long-term outcomes. • For community assessments and surveillance, promote the use of best practice templates for planning and implementing community assessments (see Box 10-3 and Chapter 7). Create a standard obesity community assessment and surveillance template that specifies obesity-related indicators, recommended analysis (e.g., subgroup, small areas, time trends), benchmarks and peer comparisons, and presentation format. It should include model language and provide a design template to minimize effort needed to produce reports. Specifications should include ­ outine across-group comparisons (e.g., race, income) and comparisons with peer communities. r • Promote the use of best practice templates for community-level monitoring and summative evaluations of obesity prevention efforts (see Box 10-4 and Chapter 8), including the use of practical participatory engagement and use of a strong methodological study design and analy- ses. Document critical evaluation goals in a standardized format to improve the evidence base at levels practical for the resources of the community, including (1) monitoring and documenting implementation of policy, program, and environmental changes; (2) estimating collective impact of combinations of strategies including by characterizing and weighting their intensity (i.e., strength of intervention, reach, duration); and (3) measuring intermediate-term changes (e.g., in policies, systems, infrastructure, and capacity) and their association with long-term surveillance of population-level outcomes (e.g., behaviors related to physical activity and healthy nutrition, obesity). • Examine new, alternative, and emerging methods of collecting data (e.g., real-time access of data from community-based organizations, crowd sourcing techniques, new technologies and hand- held or worn devices, geographic information systems). 290 Evaluating Obesity Prevention Efforts

OCR for page 273
Improve Access to and Dissemination of Evaluation Data Recommendation 4: Relevant federal agencies (e.g., in the U.S. Departments of Agriculture, Commerce, Health and Human Services, Labor, and Transportation) in collaboration with academics, non­ governmental organizations, and state and local health departments, should coordinate existing efforts to ensure that federal, state, and local assessment, monitoring, surveillance, and summative evaluation systems include a mechanism for feedback to users of evaluation data. In addition, local evaluations should con- ­ tinue to build the evidence base for the Accelerating Progress in Obesity Prevention report strategies; be stored, curated, synthesized, and shared to improve generalizable knowledge about implementation barriers and opportunities; and clarify “what works” in different contexts. To guide the implementation of this recommendation, potential actions to improve access to and disseminate data include the following: • Further develop or expand use of existing data warehouses and resource centers (e.g., HHS’s Health Information Warehouse, Data Resource Center for Child and Adolescent Health). The National Collaborative on Child Obesity Research is considering an evaluation registry that would describe and identify where major evaluation data can be obtained; this should be encour- aged and expanded. • Expand access to and increase functionality of data visualization tools (e.g., Community Commons, Data Resource Center for Child and Adolescent Health) so that users can add local data and produce charts and maps. • Provide tools to use and access data for local data analysis, including support for generating synthetic estimates. • Develop registries for gathering and disseminating the results of community-level evaluations of obesity prevention efforts. • Create an online data entry, assessment, and monitoring system to support regular local food and physical activity environment, program, and policy scans. Improve Workforce Capacity for Evaluation Recommendation 5: The Centers for Disease Control and Prevention, National Institutes of Health, and the U.S. Department of Agriculture, through the National Collaborative on Child Obesity Research and other nongovernmental and professional organizations, should build on their existing evaluation resources to assure support for the diverse and inter-disciplinary workforce engaged in conducting assessments, sur- veillance, monitoring, and summative evaluation activities. To guide the implementation of this recommendation, potential actions to improve workforce capacity include the following: • Provide standardized training on planning and designing assessments, surveillance, monitoring, and summative evaluations for policy and environmental interventions, including the use of common indicators, measurement protocols, data collection methods, and the use of qualitative methods. Taking Action 291

OCR for page 273
• Develop mechanisms for providing technical assistance for data access, statistical analysis, and reporting from state health departments, federal government (e.g., CDC), and nongovernmental organizations for states, territories, and local entities. • Create a database of local evaluation expertise for use by stakeholders engaged in obesity pre- vention interventions. • Link to a national network of knowledge brokers who can help to support and guide implemen- tation of evaluation plans. • Identify expertise at state and local universities and colleges for improved design and analysis of initiatives. • Partner with state and local universities and professional organizations (e.g., National Association of County and City Health Officials) to offer online courses/webinars on how to conduct community assessments, surveillance, monitoring, and summative evaluations in ­different contexts. • Strengthen university-community engagement and collaboration through community-based p ­ articipatory assessments and participatory evaluation. • Facilitate the development of resources provided to state and local health departments so they have necessary capacity to evaluate obesity prevention efforts. • Increase skills in communicating findings with consumers, media, and decision makers in a r ­ elevant and understandable manner. Improve Evaluations to Address Disparities and Health Equity Recommendation 6: The U.S. Department of Health and Human Services in collaboration with non­ federal partners should increase its capacity to address health equity by practicing participatory and culturally competent evaluation, and it should standardize the collection, analysis, and reporting of data targeting disparities and health equity, and improve the accessibility of tools and methods for measuring social determinants that put populations at elevated risk for obesity. To implement this recommendation, • CDC as well as state and local health departments should strengthen assessment, monitoring, surveillance, and summative evaluation efforts through the following activities: (1) assure data samples are designed to allow analysis of differential and avoidable health outcomes related to race/ethnicity, income, and geographic sub-groups; (2) increase local data collection with an emphasis on disadvantaged populations and the differential exposures, vulnerabilities, and consequences that produce disparities; (3) make better use of data aggregation that allows for pooling of data across time and/or geographic area; (4) improve the methods for small area esti- mation that are often used to provide smoothed or synthetic estimates of risk; (5) standardize metrics to allow more effective pooling of data; (6) improve data collection methods via multi- method sampling (e.g., telephone, in-person, Internet); and (7) employ interviewers fluent in the language and culture of choice (for populations for whom English is not the first language). • CDC, as well as state and local health departments, should strengthen assessment and monitor- ing of environmental conditions and systems relationships among them that produce disparities 292 Evaluating Obesity Prevention Efforts

OCR for page 273
through the following activities: (1) increase local data collection with an emphasis on differ- ential exposures (e.g., access to healthful foods), vulnerabilities, and consequences; (2) adapt measures to allow for culturally appropriate foods, activities, and health-promoting environ- ments; and (3) use methods for small-area analysis to examine associations between differential exposures/vulnerabilities and associated health disparities. • The Secretary of HHS in collaboration with other federal agencies should (1) develop common conceptual and operational language, domain (i.e., individual dietary behavior, food environ- ment, individual physical activity behavior, physical activity environment), and definitions to understand influences on disparities and health equity in obesity, taking into account expertise ­ from multiple disciplines; (2) identify common tools (both qualitative and quantitative) for all target populations most at risk for obesity disparities across all levels of impact (e.g., individual, community, society); and (3) emphasize the quality of these recommended tools and methods for adapting them to specific contexts and systems. • The National Collaborative on Child Obesity Research, a public-private partnership, should (1) identify best practices for both participatory and culturally competent evaluation; (2) expand the capability of its Registry to house and provide regular updates on core tools and methodolo- gies to measure disparities and equity and improve the accessibility, utility, and dissemination of these tools; and (3) consider expanding the core tools and methodologies to include adults. Support a Systems Approach in Evaluation Recommendation 7: Evaluators, government, and private funders should incorporate a systems approach to evaluating obesity prevention efforts into their research-related activities through leader­ hip, funding, s and training support. To implement this recommendation, • Evaluators should embrace a systems approach—reflecting interactions among strategies in and across multiple sectors and levels—to guide their methods of research and evaluation of obesity prevention efforts. • Government agencies should examine what combination(s) of indicators is most appropriate for evaluating progress in obesity prevention, focusing on categories of indicators that relate to the systems framework recommended in the APOP report. • Government agencies and private organizations funding obesity prevention research and evalu- ation should (1) stimulate the use of systems science by integrating it into requests for proposals of research designed to address multi-sectoral, multi-level, and multi-component issues; (2) create requests for proposals that focus on systems science–based research in obesity prevention; and (3) stimulate research on the value proposition of a systems approach to obesity evaluation by creating calls for research that include numerous domains and elements to identify the “value” of community-based interventions (based on the recommendations of the Committee on Valuing Community-Based, Non-Clinical Prevention Policies and Wellness Strategies). • Relevant federal agencies funding obesity prevention work (e.g., CDC, National Institutes of Health, Agency for Healthcare Research and Quality) should encourage and promote partner- Taking Action 293

OCR for page 273
ships between federal/public and private organizations to train the evaluation workforce in the use of systems science for the purpose of obesity prevention evaluations by convening work- shops, bringing together stakeholders, and providing pilot funding for developmental projects. The Weight of the Nation Measurement Ideas TWOTN, a campaign produced by HBO and the IOM, is an illustrative contributor to a long-term national commitment to prevent obesity through policy and environmental changes. Along with many other events and vehicles, TWOTN attempts to make people aware of the obesity problem, raise their consciousness about policy and environmental forces that give rise to obesity, and, with the community screenings and school materials, engage them in strategies to address the problem. TWOTN utilized mul- tiple components, including national (primarily the HBO television series and associated website) and community components (e.g., local screenings, school initiatives). According to its task, the Committee reviewed the components and goals/objectives of TWOTN and offered ideas for measurement of its impact. Given the range of social media and advocacy efforts involved in TWOTN, it is a challenge to assess its contribution to the total mix among other components of the national efforts, let alone attribute change in physical activity, nutrition behaviors, or obesity to it; however, assessment of TWOTN can be illustrative of some of the challenges and opportunities that are inherent in evaluation of similar obesity prevention initiatives. The initial national evaluation of TWOTN (see Box 6-3) will provide indicators of national dosage (or reach or exposure). The Committee concludes that further national-level evaluation is not warranted at this time because the extensive diffusion and secondary reach of the program has made comparison populations increasingly difficult to distinguish based on their exposure to elements of the campaign. The Committee provided some methods for national evaluation of future campaigns in Chapter 6. However, it will probably not be possible to disentangle TWOTN media campaign activities from other national and community activities that employ policy and environmental strategies to raise awareness and engage stakeholders in obesity prevention at this time. Current community summative evaluation efforts by Prevention Research Centers (through University of North Carolina Chapel Hill) and Kaiser Permanente will provide indicators of local dosage ­ or reach for TWOTN. Further community summative evaluations should be based on a logic-model approach. For example, if schools utilize TWOTN-derived products, such as the media kits and the three follow-on children’s film, then one might assess changes in knowledge about obesity before versus after viewing the film. The Committee emphasizes the importance of (1) using strong theoretical or logic m ­ odels; (2) assessing reach or dosage, which is actually a critical step in the logic model for any health promotion program or mass media campaign; and (3) multiple waves of measurement, the more the b ­ etter, preferably both before and after a campaign. Implementing these steps will require a commitment to resources for supporting the measurement of the community components of the campaign. Chapter 8 details these current efforts and suggests approaches to evaluating the community-level components of TWOTN. The only real source of mass media data on reach, exposure, or dosage is from commercial sources (e.g., Nielsen). No government-sponsored repository of these data exists. This is distinct from most of 294 Evaluating Obesity Prevention Efforts

OCR for page 273
the other indicator areas, so it is worth highlighting as a potential area for strengthening the capacity for national evaluation of future media campaigns and media material for obesity prevention efforts (see Chapter 6). final thoughts In 2012, the IOM’s Accelerating Progress in Obesity Prevention: Solving the Weight of the Nation report provided recommended strategies and action steps for implementation by key stakeholders and s ­ ectors that individually have positive acceleration potential and that combined will create synergies that can further accelerate progress in preventing obesity over the next decade. These 20 strategies offer the focus for future evaluation efforts across the United States. This report builds on these strategies and offers an evaluation framework to inform and improve obesity prevention efforts. Key activities identified in the evaluation framework and provided in this report include obesity evaluation plans at the national and community levels that provide a tool for guiding the planning, implementation, and evaluation of obesity prevention efforts. A second key activity identified in the evaluation framework is a list of indica- tors and sources of data and provide a source of baseline data to begin to comprehensively assess obesity prevention actions already being implemented across the country. These indicators can be incorporated into the evaluation plans and will provide guidance for improving new targeted evaluations of the strate- gies recommended in the APOP report. references IOM (Institute of Medicine). 2012. An integrated framework for assessing the value of community-based prevention. Washington, DC: The National Academies Press. Pronk, N. P., L. M. Hernandez, and R. S. Lawrence. 2013. An integrated framework for assessing the value of c ­ ommunity-based prevention: A report of the Institute of Medicine. Preventing Chronic Disease 10:120323. Taking Action 295

OCR for page 273