National Academies Press: OpenBook
« Previous: 1 Introduction
Suggested Citation:"2 Perspectives of Evaluators." Institute of Medicine. 2009. Community Perspectives on Obesity Prevention in Children: Workshop Summaries. Washington, DC: The National Academies Press. doi: 10.17226/12705.
×

2
Perspectives of Evaluators

During the program evaluators’ session, a number of themes emerged regarding actions that hold promise for improving the ability of evaluators to measure the progress and outcomes of community-based obesity prevention programs:

  • Address the need for common measures for evaluating obesity prevention programs.

  • Sustain long-term commitments by foundations and policy makers so that impact can be demonstrated and better assessed.

  • Identify or develop relevant research models that can be applied to evaluation of community-based programs.

  • Ease the burden that is often imposed by evaluation, particularly on targeted community members who are frequently oversurveyed.

  • Develop solutions to the data burden associated with multifactorial obesity prevention measurements.

  • Improve opportunities for publishing and disseminating evaluation results.

ADDRESS THE NEED FOR COMMON MEASURES

A number of the evaluators expressed frustration at the lack of an accepted methodology and common measures for evaluating the complex program and policy changes involved in childhood obesity prevention. According to several evaluators, a set of common measures would facilitate the evaluation process, allow communities to learn from each other, and

Suggested Citation:"2 Perspectives of Evaluators." Institute of Medicine. 2009. Community Perspectives on Obesity Prevention in Children: Workshop Summaries. Washington, DC: The National Academies Press. doi: 10.17226/12705.
×

permit cross-program analysis. One participant pointed out the significant investments being made by communities in developing their own evaluation tools and assessment instruments, and suggested that the development of common measures would not only save money, but also help consolidate the experience of communities in complementary areas. To date, that kind of exchange has been difficult to achieve.

On the other hand, it was noted that individual obesity prevention programs use different evaluation measures for some good reasons. For example, an important benefit of community-based programming is that it can be targeted to a local population and environment, and evaluation measures need to be tailored to this context. Participants raised concern about applying a one-size-fits-all measure to very different and often diverse communities. Even within the same community, programs serve different needs for different policies, initiatives, and funders, so a variety of measures may be necessary. Some participants suggested that the lack of common measures is a consequence of the multiple factors involved in obesity, including the community environment, physical activity and the existence of safe areas for play, eating behaviors, and access to healthful foods. This multifactorial nature of obesity necessitates multiple interventions within a community, which in turn may require multiple evaluation approaches. A further challenge to the development and use of common measures arises in multisite studies, where the comparisons across sites made possible by such measures may be divisive, creating an atmosphere of competition rather than collaboration. Another issue is how one can compare two programs with the same measures when the two communities involved started from different baselines. For example, community A may already have obesity prevention policies and strategies in place at the start of an intervention, while community B has none. If both communities are evaluated using the same measures, how does the evaluator account for the fact that community B has made fewer gains in obesity prevention because it started from a different point? The end result may be that communities with the greatest need are evaluated as less successful than their counterparts, and their programs are therefore discontinued.

To address this tension between the need for and the pitfalls of using common measures, several evaluators suggested developing such measures as a starting point and giving the evaluator the flexibility to incorporate additional factors so as to tailor the evaluation to the specific program, population, and community environment. It was further suggested that, through the formation of a working group of experts in the field and community-based evaluators, and perhaps through regular workshops and seminars, a range of evaluator perspectives and experiences could be incorporated into the development of common measures.

Suggested Citation:"2 Perspectives of Evaluators." Institute of Medicine. 2009. Community Perspectives on Obesity Prevention in Children: Workshop Summaries. Washington, DC: The National Academies Press. doi: 10.17226/12705.
×

SUSTAIN LONG-TERM COMMITMENTS BY FOUNDATIONS AND POLICY MAKERS

Several evaluators noted that when funders or policy makers fail to see demonstrable outcomes in the short term, they are unlikely to continue funding a program. Yet the expectation of short-term progress is often unrealistic given the nature of community-based obesity prevention programs. Such programs take time to develop and implement, and hence time to show change. Moreover, many of the strategies used for childhood obesity prevention are new, and approaches and activities are continually being modified as new evidence and ideas emerge. Participants suggested, then, that the challenges of measuring progress toward a sustainable healthier lifestyle at the community level need to be better understood by sponsors of childhood obesity prevention programs. It was suggested that evaluation needs to focus on more than achievement of this ultimate goal. It is also important to capture some of the intermediate steps along the way, as well as such accomplishments as increased support for certain initiatives, engagement of key stakeholders, and other advocacy/coalition-building achievements.

IDENTIFY OR DEVELOP RELEVANT RESEARCH MODELS

A number of participants raised questions about how obesity prevention programs are assessed in the scientific community, particularly when use of the traditional randomized controlled trial (RCT) is considered the gold standard for all researchers. Participants suggested that RCTs often are not feasible for assessing community-based interventions—particularly interventions targeting a multifactorial condition such as obesity that is attributed to the community environment. Evidence-based research models that depend on a controlled environment are simply not appropriate for the problem at hand. One participant suggested that the monetary incentives used to encourage people to participate in academic research trials affect the results, and that such trials are not an accurate reflection of reality.

Thus participants stressed the limited applicability of current research models to the evaluation of programmatic and policy-based obesity prevention programs. One participant proposed that evaluators develop a journal supplement on evaluation tools for such programs. Another participant suggested that this issue needs to be raised more broadly within the scientific community.

EASE THE BURDEN THAT IS OFTEN IMPOSED BY EVALUATION

As noted, obesity prevention involves multiple interventions and therefore a range of measures of progress over time. One concern raised by

Suggested Citation:"2 Perspectives of Evaluators." Institute of Medicine. 2009. Community Perspectives on Obesity Prevention in Children: Workshop Summaries. Washington, DC: The National Academies Press. doi: 10.17226/12705.
×

participants was the burden this imposes on both community members and evaluators. The same individuals are often asked to respond to surveys multiple times, sometimes by different evaluators whose efforts are not coordinated. Further, funders frequently want progress reports, necessitating evaluation at various points throughout a program. This repeated evaluation has led to complaints by community members that they are continually filling out surveys.

Evaluators participating in the workshop acknowledged that progress reports are important. But they also emphasized the need to ensure that communities do not develop resistance to the research aspect of programs because they are constantly being surveyed, often by various funders that ask similar questions. Moreover, evaluators must work with site leaders on the administration of evaluations. A few evaluators reported that site leaders sometimes think they need to protect community members from feeling like research subjects. It was suggested that site leaders report all current evaluations to a database so as to document the evaluation load on their community. Funders could then determine the optimum timing for their evaluations to ease the burden on the community.

DEVELOP SOLUTIONS TO THE DATA BURDEN ASSOCIATED WITH MULTIFACTORIAL OBESITY PREVENTION MEASUREMENTS

Another issue related to data collection involves the above-noted multifactorial nature of childhood obesity prevention programs, which can necessitate multiple evaluations. This need to examine a range of variables, together with the push to have measurable outcomes or compare favorably with RCT standards, adds to the evaluation burden on community members. It also burdens evaluators by making surveys more complicated and difficult to design while weakening the measures collected as more and more measures are added. As one participant noted, the responses provided to surveys become less reliable as more questions are asked and the surveys grow longer. Although evaluators may have more measures to work with, the accuracy of answers and the difficulty of including so many variables are problematic for evaluators and those being evaluated. The balance between collecting an adequate amount of information and sustaining the strength of the evaluation by not over collecting information needs further discussion.

IMPROVE OPPORTUNITIES FOR PUBLISHING AND DISSEMINATING EVALUATION RESULTS

A number of participants cited the difficulty of publishing and disseminating the results of community- and policy-based obesity prevention

Suggested Citation:"2 Perspectives of Evaluators." Institute of Medicine. 2009. Community Perspectives on Obesity Prevention in Children: Workshop Summaries. Washington, DC: The National Academies Press. doi: 10.17226/12705.
×

interventions in the scientific literature. The inability to find journals or other venues in which to publish the results and lessons learned from these interventions makes it difficult to share the growing body of knowledge on obesity prevention strategies and programs with colleagues within and outside of the field. One participant attributed this difficulty with getting results published to the lack of comparison groups and traditional measurable outcomes, and suggested that the situation is driving many community-based evaluators to attempt to achieve the “gold standard” for evidence-based results when, as discussed above, its application is not appropriate.

In addition, several evaluators pointed out that articles in scientific journals are not a common source of information on evaluation techniques in community-based settings. Such publications frequently focus on short-term controlled studies involving a single activity that are not applicable in these settings. As a result, published data often are not helpful to those developing or evaluating community-based obesity prevention strategies. Instead, participants reported that they rely more heavily on “gray literature” or unpublished studies to inform their work in community-based settings.

In discussing publication bias, several evaluators expressed their belief that in selecting articles on childhood obesity interventions, journals favor programmatic interventions over sustainable environmental changes, which are more difficult to specify. Yet most of these participants were working to change the community’s environment rather than implementing a single, short-term program.

Suggested Citation:"2 Perspectives of Evaluators." Institute of Medicine. 2009. Community Perspectives on Obesity Prevention in Children: Workshop Summaries. Washington, DC: The National Academies Press. doi: 10.17226/12705.
×

This page intentionally left blank.

Suggested Citation:"2 Perspectives of Evaluators." Institute of Medicine. 2009. Community Perspectives on Obesity Prevention in Children: Workshop Summaries. Washington, DC: The National Academies Press. doi: 10.17226/12705.
×
Page 5
Suggested Citation:"2 Perspectives of Evaluators." Institute of Medicine. 2009. Community Perspectives on Obesity Prevention in Children: Workshop Summaries. Washington, DC: The National Academies Press. doi: 10.17226/12705.
×
Page 6
Suggested Citation:"2 Perspectives of Evaluators." Institute of Medicine. 2009. Community Perspectives on Obesity Prevention in Children: Workshop Summaries. Washington, DC: The National Academies Press. doi: 10.17226/12705.
×
Page 7
Suggested Citation:"2 Perspectives of Evaluators." Institute of Medicine. 2009. Community Perspectives on Obesity Prevention in Children: Workshop Summaries. Washington, DC: The National Academies Press. doi: 10.17226/12705.
×
Page 8
Suggested Citation:"2 Perspectives of Evaluators." Institute of Medicine. 2009. Community Perspectives on Obesity Prevention in Children: Workshop Summaries. Washington, DC: The National Academies Press. doi: 10.17226/12705.
×
Page 9
Suggested Citation:"2 Perspectives of Evaluators." Institute of Medicine. 2009. Community Perspectives on Obesity Prevention in Children: Workshop Summaries. Washington, DC: The National Academies Press. doi: 10.17226/12705.
×
Page 10
Next: 3 Perspectives of Site Leaders »
Community Perspectives on Obesity Prevention in Children: Workshop Summaries Get This Book
×
Buy Paperback | $41.00 Buy Ebook | $32.99
MyNAP members save 10% online.
Login or Register to save!
Download Free PDF

As the public health threat of childhood obesity has become clear, the issue has become the focus of local, state, and national initiatives. Many of these efforts are centered on the community environment in recognition of the role of environmental factors in individual behaviors related to food and physical activity. In many communities, for example, fresh produce is not available or affordable, streets and parks are not amenable to exercise, and policies and economic choices make fast food cheaper and more convenient than healthier alternatives.

Community efforts to combat obesity vary in scope and scale; overall, however, they remain fragmented, and little is known about their effectiveness. At the local level, communities are struggling to determine which obesity prevention programs to initiate and how to evaluate their impact.

In this context, the Institute of Medicine held two workshops to inform current work on obesity prevention in children through input from individuals who are actively engaged in community- and policy-based obesity prevention programs. Community perspectives were elicited on the challenges involved in undertaking policy and programmatic interventions aimed at preventing childhood obesity, and on approaches to program implementation and evaluation that have shown promise. Highlights of the workshop presentations and discussions are presented in this volume.

  1. ×

    Welcome to OpenBook!

    You're looking at OpenBook, NAP.edu's online reading room since 1999. Based on feedback from you, our users, we've made some improvements that make it easier than ever to read thousands of publications on our website.

    Do you want to take a quick tour of the OpenBook's features?

    No Thanks Take a Tour »
  2. ×

    Show this book's table of contents, where you can jump to any chapter by name.

    « Back Next »
  3. ×

    ...or use these buttons to go back to the previous chapter or skip to the next one.

    « Back Next »
  4. ×

    Jump up to the previous page or down to the next one. Also, you can type in a page number and press Enter to go directly to that page in the book.

    « Back Next »
  5. ×

    Switch between the Original Pages, where you can read the report as it appeared in print, and Text Pages for the web version, where you can highlight and search the text.

    « Back Next »
  6. ×

    To search the entire text of this book, type in your search term here and press Enter.

    « Back Next »
  7. ×

    Share a link to this book page on your preferred social network or via email.

    « Back Next »
  8. ×

    View our suggested citation for this chapter.

    « Back Next »
  9. ×

    Ready to take your reading offline? Click here to buy this book in print or download it as a free PDF, if available.

    « Back Next »
Stay Connected!