Perspectives of Evaluators
During the program evaluators’ session, a number of themes emerged regarding actions that hold promise for improving the ability of evaluators to measure the progress and outcomes of community-based obesity prevention programs:
Address the need for common measures for evaluating obesity prevention programs.
Sustain long-term commitments by foundations and policy makers so that impact can be demonstrated and better assessed.
Identify or develop relevant research models that can be applied to evaluation of community-based programs.
Ease the burden that is often imposed by evaluation, particularly on targeted community members who are frequently oversurveyed.
Develop solutions to the data burden associated with multifactorial obesity prevention measurements.
Improve opportunities for publishing and disseminating evaluation results.
ADDRESS THE NEED FOR COMMON MEASURES
A number of the evaluators expressed frustration at the lack of an accepted methodology and common measures for evaluating the complex program and policy changes involved in childhood obesity prevention. According to several evaluators, a set of common measures would facilitate the evaluation process, allow communities to learn from each other, and
permit cross-program analysis. One participant pointed out the significant investments being made by communities in developing their own evaluation tools and assessment instruments, and suggested that the development of common measures would not only save money, but also help consolidate the experience of communities in complementary areas. To date, that kind of exchange has been difficult to achieve.
On the other hand, it was noted that individual obesity prevention programs use different evaluation measures for some good reasons. For example, an important benefit of community-based programming is that it can be targeted to a local population and environment, and evaluation measures need to be tailored to this context. Participants raised concern about applying a one-size-fits-all measure to very different and often diverse communities. Even within the same community, programs serve different needs for different policies, initiatives, and funders, so a variety of measures may be necessary. Some participants suggested that the lack of common measures is a consequence of the multiple factors involved in obesity, including the community environment, physical activity and the existence of safe areas for play, eating behaviors, and access to healthful foods. This multifactorial nature of obesity necessitates multiple interventions within a community, which in turn may require multiple evaluation approaches. A further challenge to the development and use of common measures arises in multisite studies, where the comparisons across sites made possible by such measures may be divisive, creating an atmosphere of competition rather than collaboration. Another issue is how one can compare two programs with the same measures when the two communities involved started from different baselines. For example, community A may already have obesity prevention policies and strategies in place at the start of an intervention, while community B has none. If both communities are evaluated using the same measures, how does the evaluator account for the fact that community B has made fewer gains in obesity prevention because it started from a different point? The end result may be that communities with the greatest need are evaluated as less successful than their counterparts, and their programs are therefore discontinued.
To address this tension between the need for and the pitfalls of using common measures, several evaluators suggested developing such measures as a starting point and giving the evaluator the flexibility to incorporate additional factors so as to tailor the evaluation to the specific program, population, and community environment. It was further suggested that, through the formation of a working group of experts in the field and community-based evaluators, and perhaps through regular workshops and seminars, a range of evaluator perspectives and experiences could be incorporated into the development of common measures.
SUSTAIN LONG-TERM COMMITMENTS BY FOUNDATIONS AND POLICY MAKERS
Several evaluators noted that when funders or policy makers fail to see demonstrable outcomes in the short term, they are unlikely to continue funding a program. Yet the expectation of short-term progress is often unrealistic given the nature of community-based obesity prevention programs. Such programs take time to develop and implement, and hence time to show change. Moreover, many of the strategies used for childhood obesity prevention are new, and approaches and activities are continually being modified as new evidence and ideas emerge. Participants suggested, then, that the challenges of measuring progress toward a sustainable healthier lifestyle at the community level need to be better understood by sponsors of childhood obesity prevention programs. It was suggested that evaluation needs to focus on more than achievement of this ultimate goal. It is also important to capture some of the intermediate steps along the way, as well as such accomplishments as increased support for certain initiatives, engagement of key stakeholders, and other advocacy/coalition-building achievements.
IDENTIFY OR DEVELOP RELEVANT RESEARCH MODELS
A number of participants raised questions about how obesity prevention programs are assessed in the scientific community, particularly when use of the traditional randomized controlled trial (RCT) is considered the gold standard for all researchers. Participants suggested that RCTs often are not feasible for assessing community-based interventions—particularly interventions targeting a multifactorial condition such as obesity that is attributed to the community environment. Evidence-based research models that depend on a controlled environment are simply not appropriate for the problem at hand. One participant suggested that the monetary incentives used to encourage people to participate in academic research trials affect the results, and that such trials are not an accurate reflection of reality.
Thus participants stressed the limited applicability of current research models to the evaluation of programmatic and policy-based obesity prevention programs. One participant proposed that evaluators develop a journal supplement on evaluation tools for such programs. Another participant suggested that this issue needs to be raised more broadly within the scientific community.
EASE THE BURDEN THAT IS OFTEN IMPOSED BY EVALUATION
As noted, obesity prevention involves multiple interventions and therefore a range of measures of progress over time. One concern raised by
participants was the burden this imposes on both community members and evaluators. The same individuals are often asked to respond to surveys multiple times, sometimes by different evaluators whose efforts are not coordinated. Further, funders frequently want progress reports, necessitating evaluation at various points throughout a program. This repeated evaluation has led to complaints by community members that they are continually filling out surveys.
Evaluators participating in the workshop acknowledged that progress reports are important. But they also emphasized the need to ensure that communities do not develop resistance to the research aspect of programs because they are constantly being surveyed, often by various funders that ask similar questions. Moreover, evaluators must work with site leaders on the administration of evaluations. A few evaluators reported that site leaders sometimes think they need to protect community members from feeling like research subjects. It was suggested that site leaders report all current evaluations to a database so as to document the evaluation load on their community. Funders could then determine the optimum timing for their evaluations to ease the burden on the community.
DEVELOP SOLUTIONS TO THE DATA BURDEN ASSOCIATED WITH MULTIFACTORIAL OBESITY PREVENTION MEASUREMENTS
Another issue related to data collection involves the above-noted multifactorial nature of childhood obesity prevention programs, which can necessitate multiple evaluations. This need to examine a range of variables, together with the push to have measurable outcomes or compare favorably with RCT standards, adds to the evaluation burden on community members. It also burdens evaluators by making surveys more complicated and difficult to design while weakening the measures collected as more and more measures are added. As one participant noted, the responses provided to surveys become less reliable as more questions are asked and the surveys grow longer. Although evaluators may have more measures to work with, the accuracy of answers and the difficulty of including so many variables are problematic for evaluators and those being evaluated. The balance between collecting an adequate amount of information and sustaining the strength of the evaluation by not over collecting information needs further discussion.
IMPROVE OPPORTUNITIES FOR PUBLISHING AND DISSEMINATING EVALUATION RESULTS
A number of participants cited the difficulty of publishing and disseminating the results of community- and policy-based obesity prevention
interventions in the scientific literature. The inability to find journals or other venues in which to publish the results and lessons learned from these interventions makes it difficult to share the growing body of knowledge on obesity prevention strategies and programs with colleagues within and outside of the field. One participant attributed this difficulty with getting results published to the lack of comparison groups and traditional measurable outcomes, and suggested that the situation is driving many community-based evaluators to attempt to achieve the “gold standard” for evidence-based results when, as discussed above, its application is not appropriate.
In addition, several evaluators pointed out that articles in scientific journals are not a common source of information on evaluation techniques in community-based settings. Such publications frequently focus on short-term controlled studies involving a single activity that are not applicable in these settings. As a result, published data often are not helpful to those developing or evaluating community-based obesity prevention strategies. Instead, participants reported that they rely more heavily on “gray literature” or unpublished studies to inform their work in community-based settings.
In discussing publication bias, several evaluators expressed their belief that in selecting articles on childhood obesity interventions, journals favor programmatic interventions over sustainable environmental changes, which are more difficult to specify. Yet most of these participants were working to change the community’s environment rather than implementing a single, short-term program.