2

Improving the Usefulness of Obesity Evaluation Information to Potential Users

This chapter asks first what evaluation users require from evidence, specifically their preferences and needs for information. “Evaluation users” are the customers for data and information on progress in preventing obesity. The potential users are termed “stakeholders,” because they have an interest in evaluation and its results (Scriven, 1991). The Institute of Medicine (IOM) report Accelerating Progress in Obesity Prevention (IOM, 2012a) called on specific groups to take action on the problem: most notably federal and state policy makers (officials in executive, legislative, and increasingly judicial branches), federal and state government agency staff that manage programs and resources, nongovernmental organizations at all levels, advocates of policy changes at all levels, opponents of such advocacy, local coalitions, local officials and local program managers, researchers and evaluators, employers, and health care providers and insurers. Table 2-1 summarizes the roles and needs of the users of obesity evaluation information that are detailed in this chapter.

The table does not provide an all-inclusive list—for example, media are not included although they interpret and report on evaluations from time to time. Other stakeholders might emerge that are engaged and influential; good tools are available to identify such stakeholders (Preskill and Jones, 2009). Moreover, stakeholder roles can shift and blend into each other; both employers and community program managers can be part of community coalitions; mayors can serve both as decision makers and managers. What matters is the role that a potential user is playing in context. For example, any of the stakeholders described in this chapter could serve the role of advocate for obesity prevention; however certain stakeholders are identified primarily in this role through their activities in lobbying, blogging, op-ed pages, and other formats.

Evaluations need to be useful; that is their primary if not their only justification (Patton, 2008; Shadish et al., 1990; Yarbrough et al., 2011). Usefulness and utilization are a decades-long preoccupation for applied research, policy analysis, and program evaluation (Dunn, 2011; Lindblom and Cohen, 1979; Ottoson, 2009; Weiss, 1988), so it is familiar territory for the IOM (IOM, 2010, 2012b; NRC, 2012).



The National Academies | 500 Fifth St. N.W. | Washington, D.C. 20001
Copyright © National Academy of Sciences. All rights reserved.
Terms of Use and Privacy Statement



Below are the first 10 and last 10 pages of uncorrected machine-read text (when available) of this chapter, followed by the top 30 algorithmically extracted key phrases from the chapter as a whole.
Intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text on the opening pages of each chapter. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.

Do not use for reproduction, copying, pasting, or reading; exclusively for search engines.

OCR for page 43
2 Improving the Usefulness of Obesity Evaluation Information to Potential Users T his chapter asks first what evaluation users require from evidence, specifically their preferences and needs for information. “Evaluation users” are the customers for data and information on progress in preventing obesity. The potential users are termed “stakeholders,” because they have an interest in evalu- ation and its results (Scriven, 1991). The Institute of Medicine (IOM) report Accelerating Progress in Obesity Prevention (IOM, 2012a) called on specific groups to take action on the problem: most notably federal and state policy makers (officials in executive, legislative, and increasingly judicial branches), fed- eral and state government agency staff that manage programs and resources, nongovernmental organiza- tions at all levels, advocates of policy changes at all levels, opponents of such advocacy, local coalitions, local officials and local program managers, researchers and evaluators, employers, and health care provid- ers and insurers. Table 2-1 summarizes the roles and needs of the users of obesity evaluation information that are detailed in this chapter. The table does not provide an all-inclusive list—for example, media are not included although they interpret and report on evaluations from time to time. Other stakeholders might emerge that are engaged and influential; good tools are available to identify such stakeholders (Preskill and Jones, 2009). Moreover, stakeholder roles can shift and blend into each other; both employers and community program managers can be part of community coalitions; mayors can serve both as decision makers and managers. What matters ­ is the role that a potential user is playing in context. For example, any of the stakeholders described in this chapter could serve the role of advocate for obesity prevention; however certain stakeholders are identified primarily in this role through their activities in lobbying, blogging, op-ed pages, and other formats. Evaluations need to be useful; that is their primary if not their only justification (Patton, 2008; Shadish et al., 1990; Yarbrough et al., 2011). Usefulness and utilization are a decades-long preoccupation for applied research, policy analysis, and program evaluation (Dunn, 2011; Lindblom and Cohen, 1979; Ottoson, 2009; Weiss, 1988), so it is familiar territory for the IOM (IOM, 2010, 2012b; NRC, 2012). 43

OCR for page 43
TABLE 2-1  Users of Obesity Evaluation Information and Their Roles and Needs Evaluation User Role Needs Community • provide differing perspectives and • to know why it is important to take action on partners or priorities obesity prevention compared to other problems coalitions • efforts depend on partnerships for • knowledge of which strategies are effective for their sustainability specific situation • information about implementation and lessons learned from other places • clear communication strategies to convey information effectively • to know options for action • often require some guidance about how to implement options Local decision • may lead or be part of formal • to track progress to know when to apply course makers and community coalitions corrections, manage implementation, and emphasize managers • often are drivers for change or de-emphasize a course of action • innovate and share information • timely and accessible data at the local level about how to institute and • a good sense of “what works” implement relevant policies • assess strategies recommended by decision makers to determine whether the strategies are feasible, acceptable, and likely effective • be responsive and accountable to constituents and external funders Health care • health care providers: opportunity • health care providers: better information on “what providers and to guide patients about healthful works” for them to recommend, in the specific health insurance diet and physical activity context of their communities and health care plans • health insurance plans: interest settings in the evaluation to manage the • nonprofit hospitals: knowledge of “what works” at financial risk related to health a community level to assure good use of resources consequences of excess weight • health insurance plans: cost-effectiveness of various strategies for building the business case for employers and consumers • health insurance plans: standardized data collection • health insurance plans: information on community program resources • health insurance plans: data to target and refine communication Employers • control access to the workplace, • confidence that wellness programs will reduce not an important and pervasive setting only health care costs, but also absenteeism and for health promotion health-related productivity losses • knowledge to create the best program for their workforce 44 Evaluating Obesity Prevention Efforts

OCR for page 43
TABLE 2-1 Continued Evaluation User Role Needs Federal and state • power to greatly influence obesity • comparative effectiveness of alternative strategies policy makers prevention in government, along with cost and cost-effectiveness business, and nonprofit • geopolitical jurisdiction comparisons organizations • best way to define issues • make and administer policy at • funder organizations: need to hold grantee federal and state levels organizations accountable for the use of funds • clear and easily digestible information to help frame choices and correctly interpret evidence Advocates • essential to the policy • be visible and persistent development process, particularly • decide on which prevention strategies to focus for public health • information from the research community • often serve as knowledge brokers to support claims about “what works” and applicability to the populations at greatest risk • information on what similar communities and states are doing • knowledge of whether specific advocacy appeals or framing of the issues and stratagems work in different contexts • information on policy progress and the needs for improvement Federal and • oversee accountability and • a variety of data elements that are not always state agency reporting requirements for funds available administrators distributed to state and local levels • indicators such as changes in programs, policies, for initiatives or environments for planning and mid-course • dissemination, translation, and corrections local implementation • best available evidence of effectiveness • external validity and generalizability Funder • keep the policy conversation going • see indicators of progress on the way to health and Organizations • champion continued social and social changes system changes • tangible signs of progress both in interventions and • educate to encourage advocacy for outcomes to retain the interest of leadership and change at all levels boards of trustees • publicize progress • evidence about what works in community-level initiatives to invest resources Improving the Usefulness of Obesity Evaluation Information to Potential Users 45

OCR for page 43
Obesity prevention, however, is a relatively new area of inquiry, so the committee reviewed and synthesized findings from several available sources, including (1) studies of the users of obesity prevention data and their information needs, preferences, and use of evaluations; (2) several IOM reports on obesity prevention (IOM, 2009a,b, 2010); (3) basic texts on political science, government agencies, and nongov- ernmental organizations, and the dissemination and implementation of prevention strategies; and (4) a lit- erature search on the use of evaluation. In addition, the committee held a public workshop (see Appendix ­ I) and conducted interviews with evaluation users (see list of those interviewed in the Preface, p. ix). The workshop presenters were identified as experienced representatives of certain user groups: community decision makers (mayor), funders, health plans and employers, federal­agencies, community practitioners, ­ and advocates. Interviews were selective to fill in gaps in the Committee’s understanding, for example, in how community coalitions or federal policy advocates would use the information. The workshop and interviews were helpful to understand the concrete reality of these roles and the uncertainties about obe- sity prevention that needed to be addressed. They also confirmed and updated what the Committee had learned from other sources. In framing what users need to know, the Committee endorsed the L.E.A.D. framework (IOM, 2010) which stands for Locate evidence, Evaluate it, Assemble it, and inform Decisions. The framework starts by specifying the question the users want to answer. The content and methods of evaluation should derive from that question, not from some ideal of how evaluation should happen. The best available evaluation methods need to be used, consistent with current knowledge and the level of resources available. In the words of Rossi et al. (2004, p. 25), evaluation quality should be “good enough” for the question that is posed. And for each user group described in this chapter, quite a bit of information is available on what likely works and how to implement it, even while knowledge is still emerging. Community Coalitions as Evaluation USERS Why Community Coalitions? All obesity intervention is or eventually becomes local, especially for changes in educational or behavior-change programs, environment, and many policy initiatives. Community obesity prevention efforts generally involve an initiating organization, but frequently involve partnerships or coalitions of individuals and organizations with differing perspectives and priorities. The efforts depend on these part- nerships for sustainability. What Do Community Partners Need? Community organizations and partnerships first need to know why it is important to take action on obesity prevention compared to other problems they are facing. For this purpose, community assess- ments are helpful (see Chapter 7). Once obesity prevention is established as a priority, the particular issues and problems that a community is facing can be revealed through further community assessments and surveillance. According to our interviews, once community partners or coalitions are motivated to do some- thing about obesity prevention, they need to know which strategies are effective and what they should do in their specific situation, given the strengths and limitations revealed by the community assessments 46 Evaluating Obesity Prevention Efforts

OCR for page 43
and other planning exercises. In particular, stakeholders cannot necessarily visualize in advance how to implement interventions, policies, and environmental strategies to prevent obesity. Programmatic or direct-service strategies are more familiar to them. As described in one interview: “They need off-the-shelf models and also implementation support—direct, hands on translation of the evidence into what needs to be done.” The implication is that, beyond “what works,” they need information about implementa- tion and lessons learned from other places. However, evaluation in their own communities also benefits coalitions in several ways. Because prevention is a long-term goal, community members may be reluctant to continue participation because they see no progress toward the goal (IOM, 2012c). Evaluations help to maintain participation if they include shorter-term indicators of progress. Evaluations of implementa- tion (“monitoring”) and of outcomes provide coalitions with a basis for improvements, better training or supervision, as well as the ability to press for additional changes in interventions or environments or for the enforcement of agreed-upon policies. To convey information effectively, clear communication is essential. Visual presentations of data, such as maps from geographic information systems (GIS), or the Supermarket Need Index, are powerful tools for sharing research (Smith et al., 2011b). Visual presentations can also inform program design and engage policy makers and stakeholders—including community members (IOM, 2009a). Such presenta- tions, however, are not sufficient by themselves; at a minimum, people need to know their options for action and they often require some guidance about how to implement those options. Community leaders often benefit from lessons learned in other localities and appreciate when evaluation results are framed in terms of comparisons to other situations and locations and of knowledge of community conditions (IOM, 2012c; Kirkpatrick and McIntyre, 2009; Lebel et al., 2011). How Can Communities Develop Capacity to Use Evaluation? Now that guided tools and specific data such as GIS and community assessments are required activ- ities for health departments and nonprofit hospitals, they offer opportunities for community leaders and community coalitions to focus their obesity prevention efforts. However, no one knows how much these tools are used. Some jurisdictions require Health Impact Assessments (HIAs) of proposed interventions in other sectors. These requirements provide opportunities to work with other sectors on improving the posi- tive impact and minimizing the negative impact on health of their proposed interventions. Several HIAs have influenced decisions and, at a minimum, helped to frame policy debates (Henderson et al., 2011; Kids Safe & Healthful Foods Project, 2012). Yet, again, it is unclear how much community partnerships actually use such tools. Chapters 7 and 8 include these and other tools and strategies that may increase their use, such as community-based participatory research and policy mandates. American Public Health Association (2006) and the Council of State and Territorial Epidemiologists Executive Committee (2007) have called for evaluation of the impact of community assessments, yet only five studies of communities’ use of community assessments have been found as of 2012. The evidence for use appears to be mixed. Two surveys of health departments found an impressively high level of use: 100 percent of community health departments in Kansas reported using community assessments to iden- tify health priorities (Curtis, 2002) while 73 percent of community assessments conducted by local health departments in Washington state were used this way (Spice and Snyder, 2009). Community assessments also facilitated better communication among community groups, helped with the development of new Improving the Usefulness of Obesity Evaluation Information to Potential Users 47

OCR for page 43
partnerships, and facilitated understanding of problems (Curtis, 2002; Solet et al., 2009; Spice and Snyder, 2009). In Kansas, 72 percent of the communities completing community assessments reported starting efforts to address the identified health priorities (Curtis, 2002). In Washington, community assessments were used to develop health programs, strategies, or services (42 percent); develop or modify health poli- cies (21 percent); influence budget decisions (23 percent); and establish or modify agency strategy (26 ­ percent) (Spice and Snyder, 2009). Yet, in New York State, researchers piloting and field-testing an evalua- tion instrument had difficulty identifying community stakeholders outside of health departments who were knowledgeable about community assessments (Myers and Stoto, 2006; Stoto et al., 2009). Coalitions for community substance abuse control have been found to make little use of other technical assistance tools, resources, or consultation, even when offered without cost (Hallfors et al., 2002). The tools exist, and many are described in Chapter 7 and 8. There are certainly opportunities to increase their utility among community groups. COMMUNITY DECISION MAKERS and managers AS EVALUATION USERS Why Community Decision Makers and Managers? Community decision makers include mayors, city planners and managers, city councils, health departments, parks and recreation directors, transportation directors, school administrators, and school boards and other policy bodies. Administrators at this level may directly manage activities related to obesity prevention. They may lead or be part of formal community coalitions, or they may not, but they are often the drivers for change. (The needs of state policy and management actors are addressed later in Chapter 2.) Policies, interventions, and environmental changes instituted by community decision makers are b ­ urgeoning (IOM, 2012c; Ross et al., 2010). Community and state governments sometimes serve as labo- ratories that may innovate, implement, evaluate, and pave the way for federal policies. State and com­ munity public health departments and community coalitions are taking an increasing interest and role in the use, or potential use, of evaluative information about such policies (IOM, 2009a). Learning commu- nities and practitioner networks are beginning to emerge as policy makers innovate and share informa- tion about how to institute and implement relevant policies. Following on principles from Diffusion of Innovations (Rogers, 2003), several of the examples in this chapter relate to early adopters, often opinion leaders, who are taking actions to address obesity and often provide lessons to others. In many cases, com- munity actions are taking place in light of limited research-tested evidence on what works to prevent obe- sity, thus highlighting the need for strong evaluation resulting in so-called practice-based evidence (Green and Glasgow, 2006). Media attention to community or regional evaluations of innovations can accelerate their adoption and spread. This dynamic has important implications for innovations that need testing (Leviton et al., 2010a) and for generalizing about innovations that are promising (Leviton, 2001). For all these reasons, community and state policy agendas are quite advanced compared to the federal agendas on obesity pre- vention: examples include instituting incentives and disincentives for healthful eating; reconstructing built environments; and encouraging child care, health care, worksite, and school policies. As in the case of tobacco, bold innovations in policy and environmental change appear to be coming first from community and state levels. As in the case of tobacco, lobbying by forces opposed to these policies may be less effec- 48 Evaluating Obesity Prevention Efforts

OCR for page 43
tive at state and community levels than at the national level because the multiplicity of community initia- tives can outrun the lobbyists who are organized primarily to work with state and federal lawmakers. What Do Community Decision Makers and Managers Need? According to the Committee’s interviews and workshop, community decision makers need to track progress in preventing obesity so they know when to apply course corrections, manage implementation, and emphasize or de-emphasize a course of action. Yet, the data necessary to do so are often unavailable at the community level or not available in a timely or accessible manner. Community body mass index data in particular are often not available, although they are valued by the public and by school adminis- trators (Haboush et al., 2011). Like community coalitions, community decision makers also need a good sense of “what works” and what they should do given the situation of their particular community. They need to assess the strat- egies that might be recommended by federal and state decision makers to determine whether they are feasible for the cost, acceptable, and likely to be effective in their particular setting, with their particular population to be served (CDC, 2013c). Community policy makers and managers also need to be responsive and accountable to constituents and external funders. Yet accountability often takes the form of an evaluation report to government or private funders, which can impair stakeholders’ learning (about what works, about implementation, and about assumptions). Community program managers tend to regard evaluation as something they do for others, not for themselves (Patton, 2008; interviews), although evaluation has been associated with pro- gram sustainability (RWJF, 2009b). When practitioners and managers have an interest in or use for what is reported, the quality and relevance of the information is almost always higher. Community stakeholders are more likely to be interested in and have use for the evaluation results if they were engaged in posing the evaluation questions (Rossi et al., 2004). How Can Useful Evaluations Be Produced for Community Decision Makers and Managers? It is important to assure that those who are actually planning and implementing obesity preven- tion have a stake in evaluation as well. Too often, evaluations are not requested by community coalitions, d ­ ecision makers, or managers, but are rather imposed on them by funders or by higher levels of govern- ment. Those imposing evaluation from outside feel urgency to do so in order to hold community efforts accountable for the use of funds or the implementation of law. Accountability is an important function of evaluation, and users at the federal and state levels need better information for this purpose. Unfortunately, the accountability focus tends to be incompatible with optimal learning and program improvement (Chelimsky, 1997; Patton, 2008). Certainly if outsiders pose evaluation questions that are unimportant to communities, make erroneous or even dangerous assumptions about community context, or select incom- plete data sources, it should come as no surprise if communities see the reports as irrelevant. These prob- lems have occurred regularly throughout the history of modern program evaluation (Shadish et al., 1990). For this reason, a variety of participatory approaches to community assessment and summative evaluation have emerged to balance the accountability focus and offer practitioners and community pro- gram managers something of value from evaluation. These approaches include community-based par- Improving the Usefulness of Obesity Evaluation Information to Potential Users 49

OCR for page 43
ticipatory evaluation for affected community members and community coalitions (Green and Glasgow, 2006; Israel et al., 2012; Jagosh et al., 2012), empowerment evaluation geared primarily toward those implementing programs (Fetterman and Wandersman, 2005), and utilization-focused evaluation for all stakeholders (Patton, 2008). As noted in Chapter 7, these methods do not replace the importance of sys- tematic measurement to reveal needs; however, they assure that relevant perspectives and information are included. Community situations are complex; those conducting community assessments and summative evaluations will have a better chance of understanding that complexity and applying existing knowledge about “what works.” They will also have a better chance of educating community users about the com- plexities of obesity prevention in context. The capacity to use evaluation information, let alone conduct evaluations, is limited in many com- munity prevention settings. This issue appears to be a function both of the organizations themselves and of the relevance and quality of evaluative information (IOM, 2012c; Labin et al., 2012; Ohri-Vachaspati and Leviton, 2010). Also, in obesity prevention, many agencies cannot afford to collect recommended measures at the state or community levels (IOM, 2012c). “Knowledge brokers” become resources to help organizations apply the findings of evaluative reports. Such knowledge brokers at the community level can include the staff of health departments, universities or colleges, and nonprofit organizations that are organized for this purpose. State health departments and the more than 2,800 community health depart- ments in the United States have the potential to play a special and sustainable role in implementing com- munity obesity prevention, and in particular in the conduct and use of community obesity prevention evaluations (Blanck and Kim, 2012). However, their evaluation capacity is often limited (Cousins et al., 2011). Certain national websites and guides can help to serve the knowledge broker role for community users. For example, the Community Tool Box website,1 a public service of the University of Kansas, had more than 800,000 unique users in 2012, indicating its value to practitioners and planners (see Chapter 6) (personal communication, S. W. Fawcett, University of Kansas, October 9, 2012). Online data resources provide similar value. One example is the Data Resource Center for Child and Adolescent Health, which provides hands-on support to community and state policy makers across the country (The Child and Adolescent Health Measurement Initiative, 2012). HEALTH CARE PROVIDERS AND HEALTH INSURANCE PLANS AS EVALUATION USERS Why Health Care Providers and Health Insurance Plans? Nonprofit hospitals can participate in community initiatives for obesity prevention as part of their community benefit requirements under the Patient Protection and Affordable Care Act, Public Law 111-148, 111th Cong. (March 23, 2010). The Act revised the tax-exempt status of nonprofit hospitals to make their required “community benefit” activities transparent, concrete, measurable, and responsive to identified community needs. For this purpose they need to conduct community assessments and adopt an implementation ­ trategy. Health insurance plans have an interest in the evaluation of obesity preven- s tion because of their need to manage the financial risk related to the costly health consequences of excess weight, such as diabetes and hypertension. Reimbursement policies could be highly influential in deter- mining how much high-quality, effective individual counseling health providers give. 1  See http://ctb.ku.edu/en/default.aspx (accessed November 11, 2013). 50 Evaluating Obesity Prevention Efforts

OCR for page 43
Individual health care providers can be strong advocates for policy and environmental changes to give their patients a better chance to control weight (McPherson et al., 2012). Health care providers have the opportunity to guide adult patients and parents of pediatric patients about healthful diet and physical activity, although knowledge of energy balance guidelines and the assessment and behavioral management of overweight and obesity by primary care providers remain at a relatively low level considering the mag- nitude of the problem (Pronk et al., 2012; Smith et al., 2011a). In particular well child care offers oppor- tunities to address obesity prevention in the context of other advice on child rearing (National Initiative on Children’s Healthcare Quality, 2013). In other areas such as smoking cessation, provider advice to quit is effective at a population level (Stead et al., 2008). Providers, however, raise the issue of weight control with patients much less frequently than needed (Smith et al., 2011a). What Do Health Care Providers and Health Insurance Plans Need? Nonprofit hospitals want to know “what works” at a community level to assure good use of com- munity resources (IOM, 2012c). Based on their conduct of community assessments, they should be inter- ested in knowing what should be done, and given the nature of their bottom line, they are likely to be interested in cost. Health insurance plans see a challenge in accurately translating how reduction in risk factors can translate into improved health status and overall cost-savings. In particular, health insurance plans see a need for cost-effectiveness of various strategies for building the business case for employers and consumers. The Committee’s workshop revealed that users see a lack of standardized data collection as a major challenge to this goal (IOM, 2012c). Health insurance plans note that employers increasingly want their workers to have access to com- munity programs and are asking for information on those resources. Tracking the use of those resources is a challenge, and for health insurance plans the biggest obstacle is motivating participation and commit- ment by consumers to complete all aspects of prevention programs, especially if the benefits are slow to be realized. Health care providers and health plans also give a high priority to the measurement of, and improvements in, racial and ethnic disparities in health. Some health insurance plans are able to use “real- time” data to show participation and utilization of health care and community resources. Outcome data are helpful for targeting and refining communications to current and potential participants in programs. Individual health care providers need better information on “what works” to better enable them to make recommendations, in the specific context of their communities and health care settings (Green et al., 2012). Some evidence suggests that they believe most weight control interventions are ineffective and that family, cultural, social, and community factors are largely responsible (Leverence et al., 2007). Recent data from the National Survey of Energy Balance Related Care among Primary Care Physicians indicates that knowledge levels of energy balance guidelines (i.e., physical activity, diet, and weight) among primary care physicians who treat children are low. Among primary care physicians who treat adults, knowledge levels appear high for overweight and obesity guidelines but less so for physical activity and dietary guide- lines (Pronk et al., 2012). Hence, additional training and guidelines that may be integrated into clinical care delivery processes appear warranted. Improving the Usefulness of Obesity Evaluation Information to Potential Users 51

OCR for page 43
How Can Evaluations Be More Useful for Health Care Providers and Health Insurance Plans? The most important added value of evaluations for health care providers and health insurance plans is that they give specific evidence of the applicability and effectiveness of interventions as imple- mented under normal circumstances in the real-life, real-time context in which they are conducted. An e ­ valuation’s utility is enhanced if the users of the evaluation evidence are actively engaged as participants in planning the evaluation, in analyzing and interpreting the results, and in incorporating the results into the planning of program adaptations and extensions. Across communities, health insurance plans are uniquely positioned to align stakeholder inter- ests and generate outcomes of mutual interest. Key stakeholders include the health care providers, the purchasers of health benefits, and the insured people. To position obesity prevention evaluation as a valued and relevant activity, the incentives to pursue evaluations need to be aligned with the interests of each stakeholder (Pronk and Kottke, 2013). For the health insurance plan, the interest is an economic r ­ ationale. For the other listed stakeholders, interests include a quality-of-care rationale, a cost-savings and productivity rationale, and a function and health experience rationale, respectively. Making those inter- ests explicit and tangible through the use of evaluation may be of significant interest to any or all of these stakeholders. Employers as evaluation users Why Employers? Employers control access to the workplace, an important and pervasive setting for health promotion (Green and Kreuter, 2005). Employers show increasing interest in wellness programs because they attract competitive employees, have potential for cost savings, and are perceived as an important benefit and the right thing to do (Berry et al., 2010). With passage of the Patient Protection and Affordable Care Act, wellness programs are likely to expand further as more employers start to self-insure and begin to see pre- vention savings accrue directly to their bottom line. A RAND Employer Survey indicates that 51 percent of all employers offer wellness programs, and 79 percent of firms employing 50 or more employees pro- vide access to a wellness program (Mattke et al., 2013). The percentage of employers offering access to a wellness program increases markedly with the number of employees (39 percent for firms with 50-100 employees; 85 percent for firms with 1,001 or more). Obesity prevention and treatment for employees is a major focus, including body mass index screening at 69 percent of firms offering clinical screenings in their wellness programs. Incentives for workplace wellness programs may include reduced insurance pre- miums or waiver of copay and deductible or increased benefits. Of employers offering wellness programs, 25 percent and 28 percent offer incentives for employee participation in weight management programs and fitness programs respectively. Three percent of employers provide incentives for reaching a target body weight and 6 percent for reaching target fitness levels. Incentives for reaching these targets may become more pervasive because the Patient Protection and Affordable Care Act will increase the permitted limits on such incentives from 20 to 30 percent of the total cost of coverage in 2014 (Mattke et al., 2013). 52 Evaluating Obesity Prevention Efforts

OCR for page 43
What Do Employers Need? Employers express confidence that wellness programs will reduce not only health care costs, but also absenteeism and health-related productivity losses (Mattke et al., 2013). Certainly the clinical benefit from obesity treatment supports employer optimism (Powell et al., 2007), and a variety of analyses indi- cate savings from some, but not all, wellness activities (Mattke et al., 2013). Yet only about half of these employers surveyed by RAND had evaluated program impacts, and only 2 percent reported actual savings estimates (Mattke et al., 2013). The limitations in the data collected matters greatly because for preven- tion of obesity both impacts and savings depend on the design of the wellness programs. The employers’ version of the “what works?” question is about designing the best program for their employees. How Can Evaluations Be More Useful for Employers? Because so many claims have been made for employee wellness programs, employers can be skepti- cal of the benefits. Evaluations are more useful to employers when they provide insights about the best program design. For example, a systematic review indicated that environmental and policy changes by themselves are not effective in changing employee behavior; health education and other interventions are still needed (Kahn-Marshall and Gallant, 2012). The employee incentive component of wellness program design also needs evaluation. Because participation, retention, and adherence rates vary across worksites and segments of the employee population, employers might want to target incentives to prob- lem areas, such as dropouts from smoking cessation or sedentary lifestyles (Berry et al., 2010; Leviton, 1987). In general, strategies to increase participation are likely to be needed. The RAND Employer Survey indicates that among firms offering weight management programs, an average of only 11 percent of t ­argeted employees participated, and, among firms offering fitness programs, only 21 percent of targeted e ­ mployees participated (Mattke et al., 2013). Another way to make evaluations more useful to employers is to make explicit the cost and cost- effectiveness of different program options. In the RAND Employer Survey, the principal reason that employers gave for not providing wellness programs was the cost—yet some programs may be highly affordable (Mattke et al., 2013). Screenings range from free to costly; Mattke et al. (2013) concluded that for every $10 of incentive for weight loss, the average adult male employee would lose an additional 0.03 pounds or would increase exercise by more than 20 minutes for an additional 0.01 days. A final way to make evaluations more useful is to extend the evaluation of wellness programs to the families of employees, for whom employers also bear the cost of health coverage. Yet there is a surpris- ing lack of information about employer-based wellness programs for families—RAND’s 2013 report does not mention it at all (Mattke et al., 2013). Although the advantage of convenient access may be less in a family-based program, family-based approaches to weight management are strongly supported by research (Epstein et al., 2007; Gruber and Haldeman, 2009). FEDERAL and state Policy Makers as evaluation users Why Federal and State Policy Makers? Policy makers fill essential roles in government, business, and nonprofit organizations and have power to greatly influence obesity prevention. An example of the pervasive importance of federal agency Improving the Usefulness of Obesity Evaluation Information to Potential Users 53

OCR for page 43
concern over accountability often impairs, threatens, or crowds out important opportunities for learning and program improvement, for both the funder and the funding recipient (Chelimsky, 1997; Patton, 2008). In spite of the Government Accountability Office survey (2013) indicating that managers do use evaluation for program improvement, it is still reasonable to ask whether evaluation for accountability­has either the structure or content for optimal national or state program manager learning, except perhaps to point to ­ prevalent implementation problems. The answer, however, is not to abandon account­ bility, but to enhance a the process of evaluation so that it helps to improve, not merely prove, intervention effectiveness. FUNDER organizations AS USERS OF EVALUATION Why Focus on Funder Organizations? Governmental and philanthropic organizations across the United States have become concerned about the obesity problem, as seen in funding for the Department of Health and Human Services CPPW Initiative by American Relief and Reinvestment Act of 2009 (CDC, 2013a), CTG by the Patient Protection and Affordable Care Act’s Prevention and Public Health Fund (CDC, 2013b), Racial and Ethnic Approaches to Community Health (REACH) by the 2012 Prevention and Public Health Fund (CDC, 2012), and the activities of the IOM Standing Committee on Childhood Obesity by the Robert Wood Johnson Foundation, the California Endowment, the Michael & Susan Dell Foundation, and Kaiser Permanente. Other philanthropic funders include the W.K. Kellogg Foundation in the area of food sys- tems, the Kresge Foundation in the area of health disparities, and a variety of state and community foun- dations. These private and nonprofit funders can keep the policy conversation going in ways that federal and state agencies cannot. They can champion continued social and system changes conducive to healthy ­ weight, and they can educate to encourage advocacy for change at all levels (although they cannot lobby). They also can publicize progress, as in the recent case of “obesity bright spots” reporting by the media (e.g., Harper, 2013). What Do Funder Organizations Need to Know? Funders of obesity prevention aim at health and social change, so they need to see indicators of progress on the way to such changes. They want to build social movements so that their limited dollars can stimulate sustained change by others. The public and key influential individuals generally believe that personal responsibility is to blame for rising obesity rates. Funders believe that this perception is an obstacle to progress and attempt to reframe the cause of obesity as due to policy and environmental fac- tors (Brownell et al., 2010). Funders, like other users, need to see tangible signs of progress in obesity prevention both in the interventions and in the outcomes to retain the interest of leadership and boards of trustees. Both public and private funders have invested heavily in multi-component, complex community ini- tiatives to obesity prevention. As seen in Chapter 8, however, evaluation of these initiatives is particularly challenging, because of the dynamics of community coalitions, the range of program, environmental, and policy components, and the limitations of available designs. The evidence base is limited, and yet Institute of Medicine reports since 2003 have concluded that this approach is needed (IOM, 2004, 2009b, 2010, 2012a,b). The stakes are high. Funders include W.K. Kellogg Foundation’s Food and Fitness Initiative 62 Evaluating Obesity Prevention Efforts

OCR for page 43
(USDA, 2010); the Robert Wood Johnson Foundation’s Healthy Kids; Healthy Communities initiative (RWJF, 2013); the Kaiser Permanente Community Health initiative (Cheadle et al., 2010); the federal CPPW, CTG, and REACH initiatives; the First Lady Michelle Obama’s Let’s Move Campaign (Let’s Move, 2013); and the White House Task Force on Childhood Obesity (The White House, 2010). Indeed, federal funding priorities recognize the major importance of place-based initiatives and have included significant funding for CPPW, CTG, REACH, and others (CDC, 2012, 2013a,b). Therefore, all of these funders feel a pressing need to accumulate evidence about what works in community-level initiatives so that they can invest resources wisely and secure the best possible return on investment. How Can Evaluations Be More Useful for Funder Organizations? Funders respond to the same kinds of information as community and federal policy makers. They rely on trusted experts to advise them about investments, so linking them with the best scientists is criti- cal. Those scientists, however, also need to be able to translate research into feasible and relevant actions, another role for the “knowledge broker.” Evaluation can help bridge the research-to-action gap by testing the applicability of the research to the particular settings, populations, and circumstances in which the interventions recommended by the research would be applied or adapted. Funders can then assure their leadership and boards of trustees that their resources are having the intended impact. Evaluation can also be used to identify evidence gaps and testable hypotheses to be addressed through formal research. Such gaps in what is known may inform the next rounds of funding portfolios. GENERAL FACTORS AFFECTING USEFULNESS OF EVALUATION across types of Users A variety of factors affecting the utilization of evaluation and policy analysis have been identi- fied in the literature and appear to generalize across types of users. These are particularly important considerations for improving the usefulness of evaluation information on progress in preventing obe- sity. As ­ ummarized by Dunn (2011) and Johnson et al. (2009) , these factors may concern character- s istics of the evaluation, decision context, and user involvement. Evaluator competence and hence the q ­ uality of the evaluation is often paramount; poor quality evaluations may be used, but they are likely to be ­ egarded as less trustworthy. In addition, the quality of communications is critical: have findings been r conveyed in jargon-free language that is action oriented? Credibility depends on evaluation quality, but also on whether the findings are surprising or in line with other information from the body of evidence and experience on the topic, such as representativeness of the situation, population, and resources that were used. The particular findings and their relevance to decisions, as well as whether the information is on time for the window of opportunity, matter a great deal. Yet, timeliness is also a function of context and of user involvement. As described by Dunn (2011), findings need to be relevant to the particular activities of the policy development process. In the same way, community and state capacity matters: if program managers are not ready or able to receive infor- mation about what works, not willing to commit resources to, or capable of, implementing something that works, or have no capacity to improve their existing programs, evaluation findings from other set- tings can fall on deaf ears, and evaluation will not be undertaken in the absence of intervention in their own setting. Improving the Usefulness of Obesity Evaluation Information to Potential Users 63

OCR for page 43
BOX 2-7 Evaluation Users as Part of a Systems Approach to Evaluation Chapter 9 deals with the complexity of obesity prevention and outlines a systems approach. Consideration of the wide variety of evaluation users is integral to this approach. Emergent properties of complex systems force an evaluation approach to obesity prevention efforts to deal with reality as it unfolds. As a result, eval- uation efforts that focus too much on internal validity, and thereby lack generalizability, will suffer in their relevance to application. A complex systems approach will provide insights into the complex web of inter- relationships among multiple levels of activity, multiple sectors across communities or the nation, multiple stakeholder groups, multiple programmatic options, and other factors. It will also consider feedback loops and provide updates on progress based on the whole picture rather than a single element. As an example of consideration of the multiple interrelationships among many factors that affect obesity, the Committee refers to the 2012 Institute of Medicine report on valuing community-based prevention (IOM, 2012b) as well as to the obesity systems map in the Foresight report by the Government Office for Science in the United Kingdom (Vandenbroeck et al., 2007). As the obesity prevention field moves from research into practice, sys- tems approaches provide a realistic set of insights and learnings. Personal characteristics of the users matter, including whether they are accustomed to using data or to thinking analytically about programs and policies. In addition, their commitment to the evalua- tion, and the organization’s commitment or receptiveness to evaluation, will affect whether it is used. Characteristics of the decision, including feasibility of implementing recommendations based on evalua- tion findings, are factors in utilization. So is the political acceptability of potential solutions: if, for exam- ple, political sentiment is opposed to government regulation of food and physical activity environments, than it will greatly affect the interventions selected and the interpretation of evaluations. The information needs of the users, as well as competing or complementary information, all affect whether and how the information will be used. Clearly, all these factors can be enhanced in a particular intervention setting by the degree to which evaluation users in that setting can be engaged in planning and making sense of the evaluation. Their needs must be addressed. Consideration of the wide variety of evaluation users is critical to taking a sys- tems science approach to better understand the complexity of obesity prevention (see Box 2-7). A policy maker can facilitate access to information about timeliness, relevance, other information, and the basis for assessing credibility. A program manager committed to the evaluation is more likely to use evaluation results, feasibility and context permitting (Patton, 2008). Conclusions What Are the Priority Questions? Across the workshop, the interviews, and the literature, the various kinds of evaluation users identified a set of highest-priority questions: (1) “Why is this important?,” (2) “What works to prevent 64 Evaluating Obesity Prevention Efforts

OCR for page 43
o ­ besity?,” (3) “What should we do?,” and (4) “How are we doing?” (Farley and VanWye, 2012; IOM, 2012c; interviews; Rodgers and Collins, 2012). In addition, several potential user groups identified better cost information as important. Evaluation users operate at federal, state, and community levels, and in at least three contexts: the policy-making process (Dunn, 2011; Kingdon, 2011); dissemination and diffusion of obesity prevention strategies (Brownson et al., 2012; Rogers, 2003); and community-level implemen- tation, quality improvement, and sustainability of policies and programs (Fetterman and Wandersman, 2005; Ottoson, 2009; Scheirer and Dearing, 2011). What Actually Gets Used? An underlying assumption is that data “should” be used in policy and program development and implementation. Yet, the use of research, policy analysis, and evaluation is a process, not a discrete event, just as program planning and policy making are themselves processes that combine scientific evidence with other considerations. Evaluation requires users to interpret and draw out the implications of findings for action, considering both the purpose of the evaluation and the context within which the evaluation occurs (Dunn, 2011; Henry and Mark, 2003; Kirkhart, 2000; Leviton, 2003). Researchers and evalua- tors are often disappointed when their findings are not used immediately and concretely for funding or implementation decisions (Leviton and Hughes, 1981; Weiss, 1977). Although users sometimes act on findings in this immediate, instrumental way, the process depends on a host of other factors (Brownson et al., 2006; Johnson et al., 2009). Researchers can also become disillusioned when their findings are used to justify decisions that would have been made anyway, or in ways that go beyond the findings or without “fidelity” to the intervention as they had developed and tested it. Yet, their disappointment ignores the legitimate process of political persuasion that requires martialing a variety of arguments for or against a position, as well as the necessity of adapting some tested interventions to the very different people, set- tings, or circumstances in which they would be applied (Leviton and Hughes, 1981). Most commonly, findings are used conceptually along with other information, such as the experience of implementers, to better understand the nature of a problem, the operation of a program or policy, or the assumptions underlying a logic model or theory of change (Dunn, 2011; Weiss, 1977). Finally, users are often affected by their own participation in research, policy analysis, or evaluation to think more analytically—not ­ necessarily linked to any specific finding (Patton, 1997). The impact of their participation should not be underrated, because it can improve policy through simulations at the national or international levels (Gortmaker et al., 2011) and it can improve logic models and implementation in community obesity pre- vention programs (Leviton et al., 2010b). Ways to Improve Usefulness The literature, workshops, and interviews pointed to several areas for improvement in evaluating progress of efforts to prevent obesity. First, the field needs to develop better and more comparable data, especially at community levels, for indicators relevant to obesity. Also, data collection needs to be feasible for health departments and other organizations that are unlikely to have the resources for elaborate mea- surement of populations, policies, and environments. Better data will mean better comparisons across time and geo­ olitical areas, and may lead to better benchmarks or standards for progress. Good intermediate p indicators need to be agreed upon to help stakeholders to assess progress in achieving policy, environ­ Improving the Usefulness of Obesity Evaluation Information to Potential Users 65

OCR for page 43
mental, and behavioral changes in ways that will be most credible. “Knowledge brokers” can fill several roles, including providing brief, cogent summaries of available research, assisting researchers in making the implications of their findings for action clear and concrete, assessing applicability of the research and evaluations elsewhere to the community situation, drawing conclusions and options for action for stake- holders, and assisting them to envision what change would look like. More needs to be known about external validity as well as “what works.” The single-minded emphasis on requiring evaluations for accountability, however, may limit the potential of those reports to provide generalized knowledge about the populations, settings, and resources needed to adequately implement obesity prevention strategies. Structured differently, requirements for producing and presenting evaluative reports could be an enabling process and rich resource to more fully understand external validity. References APHA (American Public Health Association). 2006. Policy statement database: Conduct research to build an ­ vidence-base of effective community health assessment practice. http://www.apha.org/advocacy/policy/­ e policysearch/default.htm?id=1330 (accessed September 28, 2012). Beer, T., P. S. Ingargiola, and M. F. Beer. 2012. Advocacy & public policy grantmaking: Matching process to purpose. Washington, DC: The Colorado Trust. Berry, L. L., A. M. Mirabito, and W. B. Baun. 2010. What’s the hard return on employee wellness programs? Harvard Business Review 88(12):104-112, 142. Blanck, H. M., and S. A. Kim. 2012. Creating supportive nutrition environments for population health impact and health equity: An overview of the nutrition and obesity policy research and evaluation network’s efforts. American Journal of Preventive Medicine 43(3 Suppl 2):S85-S90. Brownell, K. D., R. Kersh, D. S. Ludwig, R. C. Post, R. M. Puhl, M. B. Schwartz, and W. C. Willett. 2010. Personal responsibility and obesity: A constructive approach to a controversial issue. Health Affairs 29(3):379-387. Brownson, R. C., C. Royer, R. Ewing, and T. D. McBride. 2006. Researchers and policymakers: Travelers in parallel universes. American Journal of Preventive Medicine 30(2):164-172. Brownson, R. C., G. A. Colditz, and E. K. Proctor, eds. 2012. Dissemination and implementation research in health: Translating science to practice. New York: Oxford University Press. CDC (Centers for Disease Control and Prevention). 2012. Racial and ethnic approaches to community health (REACH). http://www.cdc.gov/reach/about.htm (accessed April 10, 2013). CDC. 2013a. Communities Putting Prevention to Work. http://www.cdc.gov/communitiesputtingpreventiontowork (accessed April 4, 2013). CDC. 2013b. Community Transformation Grants (CTGs). http://www.cdc.gov/communitytransformation/index.htm (accessed April 4, 2013). CDC. 2013c. What is the Community Guide? http:// www.thecommunityguide.org/about/index.html (accessed May 24, 2013). CDC. 2013d. Replicating effective programs plus. http://www.cdc.gov/hiv/topics/prev_prog/rep/index.htm (accessed May 24, 2013). Cheadle, A., P. M. Schwartz, S. Rauzon, W. L. Beery, S. Gee, and L. Solomon. 2010. The Kaiser Permanente Community Health Initiative: Overview and evaluation design. American Journal of Public Health 100(11):2111-2113. Chelimsky, E. 1991. On the social science contribution to governmental decision-making. Science 254(5029):226-231. Chelimsky, E. 1997. The coming transformations in evaluation. In Evaluation for the 21st century: A handbook, edited by E. Chelimsky and W. R. Shadish. Thousand Oaks, CA: Sage Publications. Pp. 1-26. 66 Evaluating Obesity Prevention Efforts

OCR for page 43
Childhood Obesity Action Network. 2009. State obesity profiles. National Initiative for Children’s Healthcare Quality Child Policy Research Center, and Child and Adolescent Health Measurement Initiative. http://www.­ childhealthdata.org/browse/snapshots/obesity-2007 (accessed May 28, 2013). Choi, B. C. K., T. Pang, V. Lin, P. Puska, G. Sherman, M. Goddard, M. J. Ackland, P. Sainsbury, S. Stachenko, and H. Morrison. 2005. Can scientists and policy makers work together? Journal of Epidemiology and Community Health 59(8):632-637. Chriqui, J., E. Resnick, L. Schneider, R. Schermbeck, T. Adcock, V. Carrion, and F. Chaloupka. 2013. School district wellness policies: Evaluating progress and potential for improving children’s health five years after the federal ­ mandate. School years 2006-07 through 2010-11. Chicago, IL: Bridging the Gap Program, Health Policy Center, Institute for Health Research and Policy, University of Illinois at Chicago. Council of State and Territorial Epidemiologists Executive Committee. 2007. Research to study and disseminate evidence of effective community health assessments. http://c.ymcdn.com/sites/www.cste.org/resource/resmgr/ PS/07-EC-01.pdf (accessed May 24, 2013). Cousins, J. M., S. M. Langer, L. K. Rhew, and C. Thomas. 2011. The role of state health departments in supporting community-based obesity prevention. Preventing Chronic Disease 8(4):A87. Cronbach, L. J., and K. Shapiro. 1982. Designing evaluations of educational and social programs. San Francisco, CA: Jossey-Bass. Curtis, D. C. 2002. Evaluation of community health assessment in Kansas. Journal of Public Health Management and Practice 8(4):20-25. Dodson, E. A., C. Fleming, T. K. Boehmer, D. Haire-Joshu, D. A. Luke, and R. C. Brownson. 2009. Preventing child- hood obesity through state policy: Qualitative assessment of enablers and barriers. Journal of Public Health Policy 30:S161-S176. Dodson, E. A., A. A. Eyler, S. Chalifour, and C. G. Wintrode. 2012. A review of obesity-themed policy briefs. American Journal of Preventive Medicine 43(3 Suppl 2):S143-S148. Dorfman, L. 2013. Talking about sugar sweetened-beverage taxes: Will actions speak louder than words? American Journal of Preventive Medicine 44(2):194-195. Dunn, W. N. 2011. Public policy analysis: An introduction. 5th ed. Upper Saddle River, NJ: Pearson Prentice Hall. Epstein, L. H., R. A. Paluch, J. N. Roemich, and M. D. Beecher. 2007. Family-based obesity treatment, then and now: Twenty-five years of pediatric obesity treatment. Health Psychology 26(4):381-391. Evenson, K. R., R. C. Brownson, S. B. Satinsky, A. A. Eyler, and H. W. Kohl 3rd. 2013. The U.S. National Physical Activity Plan: Dissemination and use by public health practitioners. American Journal of Preventive Medicine 44(5):431-438. Farley, T. A., and G. Van Wye. 2012. Reversing the obesity epidemic: The importance of policy and policy research. American Journal of Preventive Medicine 43(3 Suppl 2):S93-S94. Fay W. Boozman College of Public Health. 2010. Year six evaluation: Arkansas Act 1220 of 2003 to combat child- hood obesity. Little Rock: University of Arkansas for Medical Sciences. Fetterman, D. M., and A. Wandersman. 2005. Empowerment evaluation principles in practice. New York: Guilford Press. Francisco, V. T., A. L. Paine, and S. B. Fawcett. 1993. A methodology for monitoring and evaluating community health coalitions. Health Education Research 8(3):403-416. GAO (Government Accountability Office). 2013. Program evaluation: Strategies to facilitate agencies’ use of evalua- tion in program management and policy making. Washington, DC: GAO-13-570. Ginsburg, A., and N. Rhett. 2003. Building a better body of evidence: New opportunities to strengthen evaluation utilization. American Journal of Evaluation 24(4):489-498. Improving the Usefulness of Obesity Evaluation Information to Potential Users 67

OCR for page 43
Glasgow, R. E., L. W. Green, L. M. Klesges, D. B. Abrams, E. B. Fisher, M. G. Goldstein, L. L. Hayman, J. K. Ockene, and C. Tracy Orleans. 2006. External validity: We need to do more. Annals of Behavioral Medicine 31(2):105-108. Gortmaker, S. L., B. A. Swinburn, D. Levy, R. Carter, P. L. Mabry, D. T. Finegood, T. Huang, T. Marsh, and M. L. Moodie. 2011. Changing the future of obesity: Science, policy, and action. Lancet 378(9793):838-847. Green, L. W., and R. E. Glasgow. 2006. Evaluating the relevance, generalization, and applicability of research: Issues in external validation and translation methodology. Evaluation and the Health Professions 29(1):126-153. Green, L. W., and M. W. Kreuter. 2005. Health promotion planning: An educational and ecological approach. 4th ed. New York: McGraw-Hill. Green, L. W., and M. Nasser. 2012. Attention to external validity. In Dissemination and implementation research in health: Translating science to practice, edited by R. Brownson, G. Colditz, and E. Parker. New York: Oxford University Press. Pp. 305-326. Green, L. W., F. L. Brancati, and A. Albright. 2012. Primary prevention of type 2 diabetes: Integrative public health and primary care opportunities, challenges and strategies. Family Practice 29(Suppl 1):i13-i23. Grob, G. F. 2010. Writing for impact. In Handbook of practical program evaluation, 3rd ed., edited by J. S. Wholey, H. P. Hatry, and K. E. Newcomber. San Francisco, CA: Jossey-Bass. Pp. 594-619. Gruber, K. J., and L. A. Haldeman. 2009. Using the family to combat childhood and adult obesity. Preventing Chronic Disease 6(3):A106. Haboush, A., T. Phebus, D. Tanata Ashby, H. Zaikina-Montgomery, and K. Kindig. 2011. Still unhealthy 2009: Building community research to identify risk factors and health outcomes in childhood obesity. Journal of Community Health 36(1):111-120. Hallfors, D., H. Cho, D. Livert, and C. Kadushin. 2002. Fighting back against substance abuse: Are community coalitions winning? American Journal of Preventive Medicine 23(4):237-245. Harper, M. 2013. Teaming up against obesity. Boston Globe, January 14, 2013. Henderson, T., M. Rader, B. Sorte, M. M. Ratcliffe, A. Lawrence, J. Lucky, and C. Harris. 2011. Health impact assessment: Farm to school and school garden policy, HB 2800. Upstream Public Health and the Health Impact Project. Henry, G. T., and M. M. Mark. 2003. Beyond use: Understanding evaluation’s influence on attitudes and actions. American Journal of Evaluation 24(3):293-314. Hirschman, J., and J. F. Chriqui. 2012. School food and nutrition policy, monitoring and evaluation in the USA. Public Health Nutrition 16(6):982-998. International Development Research Centre. 2008. The two-pager: Writing a policy brief. In The RM knowledge translation toolkit: A resource for researchers. Ottawa, Canada: International Development Research Centre. Chapter 8. IOM (Institute of Medicine). 2004. Preventing childhood obesity: Health in the balance. Washington, DC: The National Academies Press. IOM. 2008. School meals: Building blocks for healthy children. Washington, DC: The National Academies Press. IOM. 2009a. Community perspectives on obesity prevention in children: Workshop summaries. Washington, DC: The National Academies Press. IOM. 2009b. Local government actions to prevent childhood obesity. Washington, DC: The National Academies Press. IOM. 2010. Bridging the evidence gap in obesity prevention: A framework to inform decision making. Washington, DC: The National Academies Press. IOM. 2012a. Accelerating progress in obesity prevention: Solving the weight of the nation. Washington, DC: The National Academies Press. 68 Evaluating Obesity Prevention Efforts

OCR for page 43
IOM. 2012b. An integrated framework for assessing the value of community-based prevention. Washington, DC: The National Academies Press. IOM. 2012c. Evaluating progress of obesity prevention efforts: What does the field need to know? Public Session of the Committee on Evaluating Progress of Obesity Prevention Efforts. October 12, 2012, Washington, DC. Audio available at: http://www.iom.edu/Activities/Nutrition/EvaluatingProgressObesityPrevention/2012- OCT-12.aspx (accessed November 11, 2013). Israel, B. A., E. Eng, A. J. Schulz, and E. A. Parker. 2012. Methods for community-based participatory research for health. 2nd ed. San Francisco, CA: Jossey-Bass. Jagosh, J., A. C. Macaulay, P. Pluye, J. Salsberg, P. L. Bush, J. Henderson, E. Sirett, G. Wong, M. Cargo, C. P. Herbert, S. D. Seifer, L. W. Green, and T. Greenhalgh. 2012. Uncovering the benefits of participatory research: Implications of a realist review for health research and practice. Milbank Quarterly 90(2):311-346. Johnson, K., L. O. Greenseid, S. A. Toal, J. A. King, F. Lawrenz, and B. Volkov. 2009. Research on evaluation use. A review of the empirical literature from 1986 to 2005. American Journal of Evaluation 30(3):377-410. Jones, E., A. A. Eyler, L. Nguyen, J. Kong, R. C. Brownson, and J. H. Bailey. 2012. It’s all in the lens: Differences in views on obesity prevention between advocates and policy makers. Childhood Obesity 8(3):243-250. Kahn-Marshall, J. L., and M. P. Gallant. 2012. Making healthy behaviors the easy choice for employees: A review of the literature on environmental and policy changes in worksite health promotion. Health Education and Behavior 39(6):752-776. Kettel Khan, L. K., K. Sobush, D. Keener, K. Goodman, A. Lowry, J. Kakietek, and S. Zaro. 2009. Recommended community strategies and measurements to prevent obesity in the United States. MMWR Recommendations and Reports 58(RR-7):1-26. Kids’ Safe & Healthful Foods Project. 2012. Health impact assessment: National nutrition standards for snack and a la carte foods and beverages sold in schools. http://www.pewhealth.org/uploadedFiles/PHG/Content_Level_ Pages/Reports/KS%20HIA_FULL%20Report%20062212_WEB%20FINAL-v2.pdf (accessed June 27, 2013). Kingdon, J. 2011. Agendas, alternatives, and public policies. New York: Longman Publishing Group. Kirkhart, K. E. 2000. Reconceptualizing evaluation use: An integrated theory of influence. New Directions for Evaluation (88):5-23. Kirkpatrick, S. I., and L. McIntyre. 2009. The Chief Public Health Officer’s report on health inequalities: What are the implications for public health practitioners and researchers? Canadian Journal of Public Health 100(2):93-95. Kohl, H. W., 3rd, S. B. Satinsky, G. P. Whitfield, and K. R. Evenson. 2013. All health is local: State and local planning for physical activity promotion. Journal of Public Health Management and Practice 19(3 Suppl 1):S17-S22. Labin, S. N., J. L. Duffy, D. C. Meyers, A. Wandersman, and C. A. Lesesne. 2012. A research synthesis of the evalua- tion capacity building literature. American Journal of Evaluation 33(3):307-338. Lebel, A., M. Cantinotti, R. Pampalon, M. Theriault, L. A. Smith, and A. M. Hamelin. 2011. Concept mapping of diet and physical activity: Uncovering local stakeholders perception in the Quebec City region. Social Science and Medicine 72(3):439-445. Let’s Move. 2013. Let’s Move. America’s move to raise a healthier generation of kids. http://www.letsmove.gov (accessed June 3, 2013). Leverence, R. R., R. L. Williams, A. Sussman, and B. F. Crabtree. 2007. Obesity counseling and guidelines in primary care: A qualitative study. American Journal of Preventive Medicine 32(4):334-339. Leviton, L. C. 1987. The yield from cardiovascular risk reduction at the work place. Journal of Occupational Medicine 29(12):931-936. Leviton, L. C. 2001. External validity. In International encyclopedia of the social and behavioral sciences. 1st ed, edited by N. J. Smelser and P. B. Baltes. Oxford: Elsevier. Improving the Usefulness of Obesity Evaluation Information to Potential Users 69

OCR for page 43
Leviton, L. C. 2003. Evaluation use: Advances, challenges and applications. American Journal of Evaluation 24(4):525-535. Leviton, L. C., and R. F. Boruch. 1983. Contributions of evaluation to education programs and policy. Evaluation Review 7(5):563-598. Leviton, L. C., and E. F. X. Hughes. 1981. Research on the utilization of evaluations. A review and synthesis. Evaluation Review 5(4):525-548. Leviton, L. C., L. Kettel Khan, and N. Dawkins. 2010a. Editors notes. In The systematic screening and assessment method: Finding innovations worth evaluating, edited by L. C. Leviton, L. Kettel Khan, and N. Dawkins. New Directions in Evaluation 2010(125):1-5. Leviton, L. C., L. K. Khan, D. Rog, N. Dawkins, and D. Cotton. 2010b. Evaluability assessment to improve public health policies, programs, and practices. Annual Review of Public Health 31(1):213-233. Lindblom, C. E., and D. K. Cohen. 1979. Usable knowledge: Social science and social problem solving. New Haven, CT: Yale University. Litt, J., H. Reed, S. G. Zieff, R. G. Tabak, A. A. Eyler, N. O. Tompkins, R. Lyn, J. Gustat, K. V. Goins, and D. Bornstein. 2013. Advancing environmental and policy change through active living collaboratives: Compositional and stakeholder engagement correlates of group effectiveness. Journal of Public Health Management and Practice 19(3 Suppl 1):S49-S57. Mason, M., R. Schmidt, C. Gizzi, and S. Ramsey. 2010. Taking improvement action based on performance results: Washington State’s experience. Journal of Public Health Management and Practice 16(1):24-31. Mattke, S., H. Liu, J. P. Caloyeras, C. Y. Huang, K. R. Van Busum, D. Khodyakov, and V. Shier. 2013. Workplace wellness programs study: Final report. Santa Monica, CA: RAND. McPherson, M. E., R. Mirkin, P. N. Heatherley, and C. J. Homer. 2012. Educating health care professionals in advo- cacy for childhood obesity prevention in their communities: Integrating public health and primary care in the Be Our Voice project. American Journal of Public Health 102(8):e37-e43. Myers, S., and M. A. Stoto. 2006. Criteria for assessing the usefulness of community health assessments: A literature review. Santa Monica, CA: RAND Corporation. National Association for Family Child Care. 2013. Quality standards for NAFCC accreditation update 2013. http:// nafcc.org/media/pdf/accreditation/QS-date2013.pdf (accessed May 24, 2013). National Initiative for Children’s Healthcare Quality. 2013. Obesity. http://www.nichq.org/areas_of_focus/childhood_ obesity_topic.html (accessed March 12, 2013). NIH (National Institutes of Health). 2013. NIH roadmap for medical research. http://www.nei.nih.gov/funding/­ roadmap.asp (accessed January 2, 2013). NRC (National Research Council). 2012. Using science as evidence in public policy. Washington, DC: The National Academies Press. Ohri-Vachaspati, P., and L. C. Leviton. 2010. Measuring food environments: A guide to available instruments. American Journal of Health Promotion 24(6):410-426. Ottoson, J. M. 2009. Knowledge-for-action theories in evaluation: Knowledge utilization, diffusion, implementa- tion, transfer, and translation. In Knowledge utilization, diffusion, implementation, transfer, and translation: Implications for evaluation. New Directions for Evaluation 124:7-20. Ottoson, J. M., A. G. Ramirez, L. W. Green, and K. J. Gallion. 2013. Using a wider lens than legislation when evalu- ating whether a public health research or advocacy project has contributed to policy: The Salud America! Experience. American Journal of Preventive Medicine 44(3):S282-S289. Patton, M. Q. 1997. Utilization-focused evaluation: The new century text. 3rd ed. Thousand Oaks, CA: Sage Publications. Patton, M. Q. 2008. Utilization-focused evaluation. 4th ed. Thousand Oaks, CA: Sage Publications. 70 Evaluating Obesity Prevention Efforts

OCR for page 43
Powell, L. H., J. E. Calvin 3rd, and J. E. Calvin, Jr. 2007. Effective obesity treatments. American Psychologist 62(3):234-246. Preskill, H., and N. A. Jones. 2009. Practical guide for engaging stakeholders in developing evaluation questions. Princeton, NJ: Robert Wood Johnson Foundation. Pronk, N. P., and T. E. Kottke. 2013. Health promotion in health systems. In Lifestyle medicine. 2nd ed, edited by J. M. Ripper. Hoboken, NJ: CRC Press. Pp. 1311-1326. Pronk, N. P., S. M. Krebs-Smith, D. A. Galuska, B. Liu, R. F. Kushner, R. P. Troiano, S. B. Clauser, R. Ballard- Barbash, and A. W. Smith. 2012. Knowledge of energy balance guidelines and associated clinical care prac- tices: The U.S. National Survey of Energy Balance Related Care among Primary Care Physicians. Preventive Medicine 55(1):28-33. Rodgers, G. P., and F. S. Collins. 2012. The next generation of obesity research: No time to waste. Journal of the American Medical Association 308(11):1095-1096. Rogers, E. M. 2003. Diffusion of innovations. 5th ed. New York: The Free Press. Ross, R. K., R. J. Baxter, M. Standish, L. S. Solomon, M. K. Jhawar, P. M. Schwartz, G. R. Flores, and J. Nudelman. 2010. Community approaches to preventing obesity in California. American Journal of Public Health 100(11):2023-2025. Rossi, P. H., M. W. Lipsey, and H. E. Freeman. 2004. Evaluation: A systematic approach. 7th ed. Newbury Park, CA: Sage Publications. RWJF (Robert Wood Johnson Foundation). 2009a. Evaluation of communities creating healthy environments. http:// www.rwjf.org/en/research-publications/find-rwjf-research/2009/09/evaluation-of-communities-creating-healthy- environments.html (accessed March 21, 2013). RWJF. 2009b. Evaluation of local funding partnerships: Analyzing the factors that influence sustainability and rep- lication of projects after funding ends. http://www.rwjf.org/pr/product.jsp?id=38411 (accessed September 7, 2012). RWJF. 2013. Healthy kids, healthy communities. http://www.healthykidshealthycommunities.org (accessed April 10, 2013). Samuels, S. E., S. Lawrence Bullock, G. Woodward-Lopez, S. E. Clark, J. Kao, L. Craypo, J. Barry, and P. B. Crawford. 2009. To what extent have high schools in California been able to implement state-mandated nutri- tion standards? Journal of Adolescent Health 45(3):S38-S44. Samuels, S. E., K. S. Hutchinson, L. Craypo, J. Barry, and S. L. Bullock. 2010. Implementation of California state school competitive food and beverage standards. Journal of School Health 80(12):581-587. Scheirer, M. A., and J. W. Dearing. 2011. An agenda for research on the sustainability of public health programs. American Journal of Public Health 101(11):2059-2067. Scriven, M. 1991. Evaluation thesaurus. Newbury Park, CA: Sage Publications. Shadish, W. R., T. D. Cook, and L. C. Leviton. 1990. Foundations of program evaluation: Theories of practice. Newbury Park, CA: Sage Publications. Smith, A. W., L. A. Borowski, B. Liu, D. A. Galuska, C. Signore, C. Klabunde, T. T. Huang, S. M. Krebs-Smith, E. Frank, N. Pronk, and R. Ballard-Barbash. 2011a. U.S. primary care physicians diet-, physical activity-, and weight-related care of adult patients. American Journal of Preventive Medicine 41(1):33-42. Smith, L., C. Goranson, J. Bryon, B. Kerker, and C. Nonas. 2011b. Developing a supermarket need index. In Geospatial analysis of environmental health, edited by J. A. Maantay and S. McLafferty. Geotechnologies and the environment, Springer Vol. 4. Pp. 205-221. Solet, D., S. Ciske, R. Gaonkar, K. Horsley, M. McNees, P. Nandi, and J. W. Krieger. 2009. Effective commu- nity health assessments in King County, Washington. Journal of Public Health Management and Practice 15(1):33-40. Improving the Usefulness of Obesity Evaluation Information to Potential Users 71

OCR for page 43
Sorian, R., and T. Baugh. 2002. Power of information: Closing the gap between research and policy. Health Affairs 21(2):264-273. Spice, C., and K. Snyder. 2009. Reviewing self-reported impacts of community health assessment in local health juris- dictions. Journal of Public Health Management and Practice 15(1):18-23. Stamatakis, K. A., T. D. McBride, and R. C. Brownson. 2010. Communicating prevention messages to policy makers: The role of stories in promoting physical activity. Journal of Physical Activity and Health 7(Suppl 1):S99-S107. Stead, L. F., G. Bergson, and T. Lancaster. 2008. Physician advice for smoking cessation. Cochrane Database of Systematic Reviews (2):CD000165. Stoto, M. A., S. G. Straus, C. Bohn, and P. Irani. 2009. A web-based tool for assessing and improving the usefulness of community health assessments. Journal of Public Health Management and Practice 15(1):10-17. The Child and Adolescent Health Measurement Initiative. 2012. Data Resource Center for Child and Adolescent Health. http://www.childhealthdata.org (accessed April 4, 2013). The Community Guide. 2012. Obesity prevention and control. http://www.thecommunityguide.org/obesity/index. html (accessed May 24, 2013). The White House Office. 2010. Presidential memorandum: Establishing a task force on childhood obesity. http:// www.whitehouse.gov/the-press-office/presidential-memorandum-establishing-a-task-force-childhood-obesity (accessed June 3, 2013). USDA (U.S. Department of Agriculture). 2010. Food and Fitness Initiative. http://www.nifa.usda.gov/nea/food/ in_focus/health_if_food_fitness_initiative.html (accessed April 10, 2013). USDA. 2013. National school lunch program and school breakfast program: Nutrition standards for all foods sold in school as required by the Healthy, Hunger-Free Kids Act of 2010, proposed rule. Federal Register 78(27):9530-9567. Vandenbroeck, P., J. Goossens, and M. Clemens. 2007. Foresight tackling obesities: Future choices—building the obesity system map. London, UK: Government Office for Science. Wagenaar, A. C., A. L. Tobler, and K. A. Komro. 2010. Effects of alcohol tax and price policies on morbidity and mortality: A systematic review. American Journal of Public Health 100(11):2270-2278. Weiss, C. 1977. Research for policy’s sake: The enlightenment function of social research. Policy Analysis 3(4):531-545. Weiss, C. H. 1988. Evaluation for decisions: Is anybody there? Does anybody care? Evaluation Practice 9(1):5-19. Yarbrough, D. B., L. M. Shulha, R. K. Hobson, and F. A. Caruthers. 2011. The program evaluation standards. A guide for evaluators and evaluation users. 3rd ed. Thousand Oaks, CA: Sage Publications. YMCA. 2011. First-ever national healthy eating and physical activity standards developed for afterschool programs. http://www.ymca.net/news-releases/20110809-afterschool-standards.html (accessed May 24, 2013). 72 Evaluating Obesity Prevention Efforts