2
Perspectives of Evaluators

During the program evaluators’ session, a number of themes emerged regarding actions that hold promise for improving the ability of evaluators to measure the progress and outcomes of community-based obesity prevention programs:

  • Address the need for common measures for evaluating obesity prevention programs.

  • Sustain long-term commitments by foundations and policy makers so that impact can be demonstrated and better assessed.

  • Identify or develop relevant research models that can be applied to evaluation of community-based programs.

  • Ease the burden that is often imposed by evaluation, particularly on targeted community members who are frequently oversurveyed.

  • Develop solutions to the data burden associated with multifactorial obesity prevention measurements.

  • Improve opportunities for publishing and disseminating evaluation results.

ADDRESS THE NEED FOR COMMON MEASURES

A number of the evaluators expressed frustration at the lack of an accepted methodology and common measures for evaluating the complex program and policy changes involved in childhood obesity prevention. According to several evaluators, a set of common measures would facilitate the evaluation process, allow communities to learn from each other, and



The National Academies | 500 Fifth St. N.W. | Washington, D.C. 20001
Copyright © National Academy of Sciences. All rights reserved.
Terms of Use and Privacy Statement



Below are the first 10 and last 10 pages of uncorrected machine-read text (when available) of this chapter, followed by the top 30 algorithmically extracted key phrases from the chapter as a whole.
Intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text on the opening pages of each chapter. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.

Do not use for reproduction, copying, pasting, or reading; exclusively for search engines.

OCR for page 5
2 Perspectives of Evaluators During the program evaluators’ session, a number of themes emerged regarding actions that hold promise for improving the ability of evaluators to measure the progress and outcomes of community-based obesity preven- tion programs: • Address the need for common measures for evaluating obesity prevention programs. • Sustain long-term commitments by foundations and policy makers so that impact can be demonstrated and better assessed. • Identify or develop relevant research models that can be applied to evaluation of community-based programs. • Ease the burden that is often imposed by evaluation, particularly on targeted community members who are frequently oversurveyed. • Develop solutions to the data burden associated with multifactorial obesity prevention measurements. • Improve opportunities for publishing and disseminating evaluation results. ADDRESS THE NEED FOR COMMON MEASuRES A number of the evaluators expressed frustration at the lack of an accepted methodology and common measures for evaluating the complex program and policy changes involved in childhood obesity prevention. According to several evaluators, a set of common measures would facilitate the evaluation process, allow communities to learn from each other, and 

OCR for page 5
 COMMUNITY PERSPECTIVES ON OBESITY PREVENTION permit cross-program analysis. One participant pointed out the significant investments being made by communities in developing their own evaluation tools and assessment instruments, and suggested that the development of common measures would not only save money, but also help consolidate the experience of communities in complementary areas. To date, that kind of exchange has been difficult to achieve. On the other hand, it was noted that individual obesity prevention programs use different evaluation measures for some good reasons. For example, an important benefit of community-based programming is that it can be targeted to a local population and environment, and evaluation measures need to be tailored to this context. Participants raised concern about applying a one-size-fits-all measure to very different and often diverse communities. Even within the same community, programs serve different needs for different policies, initiatives, and funders, so a variety of mea- sures may be necessary. Some participants suggested that the lack of com- mon measures is a consequence of the multiple factors involved in obesity, including the community environment, physical activity and the existence of safe areas for play, eating behaviors, and access to healthful foods. This multifactorial nature of obesity necessitates multiple interventions within a community, which in turn may require multiple evaluation approaches. A further challenge to the development and use of common measures arises in multisite studies, where the comparisons across sites made possible by such measures may be divisive, creating an atmosphere of competition rather than collaboration. Another issue is how one can compare two programs with the same measures when the two communities involved started from different baselines. For example, community A may already have obesity prevention policies and strategies in place at the start of an intervention, while community B has none. If both communities are evaluated using the same measures, how does the evaluator account for the fact that commu- nity B has made fewer gains in obesity prevention because it started from a different point? The end result may be that communities with the great- est need are evaluated as less successful than their counterparts, and their programs are therefore discontinued. To address this tension between the need for and the pitfalls of using common measures, several evaluators suggested developing such measures as a starting point and giving the evaluator the flexibility to incorporate additional factors so as to tailor the evaluation to the specific program, population, and community environment. It was further suggested that, through the formation of a working group of experts in the field and community-based evaluators, and perhaps through regular workshops and seminars, a range of evaluator perspectives and experiences could be incor- porated into the development of common measures.

OCR for page 5
 PERSPECTIVES OF EVALUATORS SuSTAIN LONg-TERM COMMITMENTS by FOuNDATIONS AND POLICy MAKERS Several evaluators noted that when funders or policy makers fail to see demonstrable outcomes in the short term, they are unlikely to continue fund- ing a program. Yet the expectation of short-term progress is often unrealistic given the nature of community-based obesity prevention programs. Such programs take time to develop and implement, and hence time to show change. Moreover, many of the strategies used for childhood obesity preven- tion are new, and approaches and activities are continually being modified as new evidence and ideas emerge. Participants suggested, then, that the challenges of measuring progress toward a sustainable healthier lifestyle at the community level need to be better understood by sponsors of childhood obesity prevention programs. It was suggested that evaluation needs to focus on more than achievement of this ultimate goal. It is also important to capture some of the intermediate steps along the way, as well as such accomplishments as increased support for certain initiatives, engagement of key stakeholders, and other advocacy/coalition-building achievements. IDENTIFy OR DEvELOP RELEvANT RESEARCH MODELS A number of participants raised questions about how obesity preven- tion programs are assessed in the scientific community, particularly when use of the traditional randomized controlled trial (RCT) is considered the gold standard for all researchers. Participants suggested that RCTs often are not feasible for assessing community-based interventions—particularly interventions targeting a multifactorial condition such as obesity that is attributed to the community environment. Evidence-based research models that depend on a controlled environment are simply not appropriate for the problem at hand. One participant suggested that the monetary incentives used to encourage people to participate in academic research trials affect the results, and that such trials are not an accurate reflection of reality. Thus participants stressed the limited applicability of current research models to the evaluation of programmatic and policy-based obesity preven- tion programs. One participant proposed that evaluators develop a journal supplement on evaluation tools for such programs. Another participant suggested that this issue needs to be raised more broadly within the scien- tific community. EASE THE buRDEN THAT IS OFTEN IMPOSED by EvALuATION As noted, obesity prevention involves multiple interventions and there- fore a range of measures of progress over time. One concern raised by

OCR for page 5
 COMMUNITY PERSPECTIVES ON OBESITY PREVENTION participants was the burden this imposes on both community members and evaluators. The same individuals are often asked to respond to sur- veys multiple times, sometimes by different evaluators whose efforts are not coordinated. Further, funders frequently want progress reports, neces- sitating evaluation at various points throughout a program. This repeated evaluation has led to complaints by community members that they are continually filling out surveys. Evaluators participating in the workshop acknowledged that progress reports are important. But they also emphasized the need to ensure that communities do not develop resistance to the research aspect of programs because they are constantly being surveyed, often by various funders that ask similar questions. Moreover, evaluators must work with site leaders on the administration of evaluations. A few evaluators reported that site leaders sometimes think they need to protect community members from feeling like research subjects. It was suggested that site leaders report all current evaluations to a database so as to document the evaluation load on their community. Funders could then determine the optimum timing for their evaluations to ease the burden on the community. DEvELOP SOLuTIONS TO THE DATA buRDEN ASSOCIATED WITH MuLTIFACTORIAL ObESITy PREvENTION MEASuREMENTS Another issue related to data collection involves the above-noted multi- factorial nature of childhood obesity prevention programs, which can neces- sitate multiple evaluations. This need to examine a range of variables, together with the push to have measurable outcomes or compare favorably with RCT standards, adds to the evaluation burden on community mem- bers. It also burdens evaluators by making surveys more complicated and difficult to design while weakening the measures collected as more and more measures are added. As one participant noted, the responses provided to surveys become less reliable as more questions are asked and the surveys grow longer. Although evaluators may have more measures to work with, the accuracy of answers and the difficulty of including so many variables are problematic for evaluators and those being evaluated. The balance between collecting an adequate amount of information and sustaining the strength of the evaluation by not over collecting information needs further discussion. IMPROvE OPPORTuNITIES FOR PubLISHINg AND DISSEMINATINg EvALuATION RESuLTS A number of participants cited the difficulty of publishing and dis- seminating the results of community- and policy-based obesity prevention

OCR for page 5
 PERSPECTIVES OF EVALUATORS interventions in the scientific literature. The inability to find journals or other venues in which to publish the results and lessons learned from these interventions makes it difficult to share the growing body of knowledge on obesity prevention strategies and programs with colleagues within and outside of the field. One participant attributed this difficulty with get- ting results published to the lack of comparison groups and traditional measurable outcomes, and suggested that the situation is driving many community-based evaluators to attempt to achieve the “gold standard” for evidence-based results when, as discussed above, its application is not appropriate. In addition, several evaluators pointed out that articles in scientific journals are not a common source of information on evaluation techniques in community-based settings. Such publications frequently focus on short- term controlled studies involving a single activity that are not applicable in these settings. As a result, published data often are not helpful to those developing or evaluating community-based obesity prevention strategies. Instead, participants reported that they rely more heavily on “gray lit- erature” or unpublished studies to inform their work in community-based settings. In discussing publication bias, several evaluators expressed their belief that in selecting articles on childhood obesity interventions, journals favor programmatic interventions over sustainable environmental changes, which are more difficult to specify. Yet most of these participants were working to change the community’s environment rather than implementing a single, short-term program.

OCR for page 5