This chapter asks first what evaluation users require from evidence, specifically their preferences and needs for information. “Evaluation users” are the customers for data and information on progress in preventing obesity. The potential users are termed “stakeholders,” because they have an interest in evaluation and its results (Scriven, 1991). The Institute of Medicine (IOM) report Accelerating Progress in Obesity Prevention (IOM, 2012a) called on specific groups to take action on the problem: most notably federal and state policy makers (officials in executive, legislative, and increasingly judicial branches), federal and state government agency staff that manage programs and resources, nongovernmental organizations at all levels, advocates of policy changes at all levels, opponents of such advocacy, local coalitions, local officials and local program managers, researchers and evaluators, employers, and health care providers and insurers. Table 2-1 summarizes the roles and needs of the users of obesity evaluation information that are detailed in this chapter.
The table does not provide an all-inclusive list—for example, media are not included although they interpret and report on evaluations from time to time. Other stakeholders might emerge that are engaged and influential; good tools are available to identify such stakeholders (Preskill and Jones, 2009). Moreover, stakeholder roles can shift and blend into each other; both employers and community program managers can be part of community coalitions; mayors can serve both as decision makers and managers. What matters is the role that a potential user is playing in context. For example, any of the stakeholders described in this chapter could serve the role of advocate for obesity prevention; however certain stakeholders are identified primarily in this role through their activities in lobbying, blogging, op-ed pages, and other formats.
Evaluations need to be useful; that is their primary if not their only justification (Patton, 2008; Shadish et al., 1990; Yarbrough et al., 2011). Usefulness and utilization are a decades-long preoccupation for applied research, policy analysis, and program evaluation (Dunn, 2011; Lindblom and Cohen, 1979; Ottoson, 2009; Weiss, 1988), so it is familiar territory for the IOM (IOM, 2010, 2012b; NRC, 2012).
TABLE 2-1 Users of Obesity Evaluation Information and Their Roles and Needs
• provide differing perspectives and priorities
• efforts depend on partnerships for sustainability
• to know why it is important to take action on obesity prevention compared to other problems
• knowledge of which strategies are effective for their specific situation
• information about implementation and lessons learned from other places
• clear communication strategies to convey information effectively
• to know options for action
• often require some guidance about how to implement options
• may lead or be part of formal community coalitions
• often are drivers for change
• innovate and share information about how to institute and implement relevant policies
• to track progress to know when to apply course corrections, manage implementation, and emphasize or de-emphasize a course of action
• timely and accessible data at the local level
• a good sense of “what works”
• assess strategies recommended by decision makers to determine whether the strategies are feasible, acceptable, and likely effective
• be responsive and accountable to constituents and external funders
• health care providers: opportunity to guide patients about healthful diet and physical activity
• health insurance plans: interest in the evaluation to manage the financial risk related to health consequences of excess weight
• health care providers: better information on “what works” for them to recommend, in the specific context of their communities and health care settings
• nonprofit hospitals: knowledge of “what works” at a community level to assure good use of resources
• health insurance plans: cost-effectiveness of various strategies for building the business case for employers and consumers
• health insurance plans: standardized data collection
• health insurance plans: information on community program resources
• health insurance plans: data to target and refine communication
• control access to the workplace, an important and pervasive setting for health promotion
• confidence that wellness programs will reduce not only health care costs, but also absenteeism and health-related productivity losses
• knowledge to create the best program for their workforce
|Federal and state policy makers||
• power to greatly influence obesity prevention in government, business, and nonprofit organizations
• make and administer policy at federal and state levels
• comparative effectiveness of alternative strategies along with cost and cost-effectiveness
• geopolitical jurisdiction comparisons
• best way to define issues
• funder organizations: need to hold grantee organizations accountable for the use of funds
• clear and easily digestible information to help frame choices and correctly interpret evidence
• essential to the policy development process, particularly for public health
• often serve as knowledge brokers
• be visible and persistent
• decide on which prevention strategies to focus
• information from the research community to support claims about “what works” and applicability to the populations at greatest risk
• information on what similar communities and states are doing
• knowledge of whether specific advocacy appeals or framing of the issues and stratagems work in different contexts
• information on policy progress and the needs for improvement
|Federal and state agency administrators||
• oversee accountability and reporting requirements for funds distributed to state and local levels for initiatives
• dissemination, translation, and local implementation
• a variety of data elements that are not always available
• indicators such as changes in programs, policies, or environments for planning and mid-course corrections
• best available evidence of effectiveness
• external validity and generalizability
• keep the policy conversation going
• champion continued social and system changes
• educate to encourage advocacy for change at all levels
• publicize progress
• see indicators of progress on the way to health and social changes
• tangible signs of progress both in interventions and outcomes to retain the interest of leadership and boards of trustees
• evidence about what works in community-level initiatives to invest resources
Obesity prevention, however, is a relatively new area of inquiry, so the committee reviewed and synthesized findings from several available sources, including (1) studies of the users of obesity prevention data and their information needs, preferences, and use of evaluations; (2) several IOM reports on obesity prevention (IOM, 2009a,b, 2010); (3) basic texts on political science, government agencies, and nongovernmental organizations, and the dissemination and implementation of prevention strategies; and (4) a literature search on the use of evaluation. In addition, the committee held a public workshop (see Appendix I) and conducted interviews with evaluation users (see list of those interviewed in the Preface, p. ix). The workshop presenters were identified as experienced representatives of certain user groups: community decision makers (mayor), funders, health plans and employers, federal agencies, community practitioners, and advocates. Interviews were selective to fill in gaps in the Committee’s understanding, for example, in how community coalitions or federal policy advocates would use the information. The workshop and interviews were helpful to understand the concrete reality of these roles and the uncertainties about obesity prevention that needed to be addressed. They also confirmed and updated what the Committee had learned from other sources.
In framing what users need to know, the Committee endorsed the L.E.A.D. framework (IOM, 2010) which stands for Locate evidence, Evaluate it, Assemble it, and inform Decisions. The framework starts by specifying the question the users want to answer. The content and methods of evaluation should derive from that question, not from some ideal of how evaluation should happen. The best available evaluation methods need to be used, consistent with current knowledge and the level of resources available. In the words of Rossi et al. (2004, p. 25), evaluation quality should be “good enough” for the question that is posed. And for each user group described in this chapter, quite a bit of information is available on what likely works and how to implement it, even while knowledge is still emerging.
COMMUNITY COALITIONS AS EVALUATION USERS
Why Community Coalitions?
All obesity intervention is or eventually becomes local, especially for changes in educational or behavior-change programs, environment, and many policy initiatives. Community obesity prevention efforts generally involve an initiating organization, but frequently involve partnerships or coalitions of individuals and organizations with differing perspectives and priorities. The efforts depend on these partnerships for sustainability.
What Do Community Partners Need?
Community organizations and partnerships first need to know why it is important to take action on obesity prevention compared to other problems they are facing. For this purpose, community assessments are helpful (see Chapter 7). Once obesity prevention is established as a priority, the particular issues and problems that a community is facing can be revealed through further community assessments and surveillance.
According to our interviews, once community partners or coalitions are motivated to do something about obesity prevention, they need to know which strategies are effective and what they should do in their specific situation, given the strengths and limitations revealed by the community assessments
and other planning exercises. In particular, stakeholders cannot necessarily visualize in advance how to implement interventions, policies, and environmental strategies to prevent obesity. Programmatic or direct-service strategies are more familiar to them. As described in one interview: “They need off-the-shelf models and also implementation support—direct, hands on translation of the evidence into what needs to be done.” The implication is that, beyond “what works,” they need information about implementation and lessons learned from other places. However, evaluation in their own communities also benefits coalitions in several ways. Because prevention is a long-term goal, community members may be reluctant to continue participation because they see no progress toward the goal (IOM, 2012c). Evaluations help to maintain participation if they include shorter-term indicators of progress. Evaluations of implementation (“monitoring”) and of outcomes provide coalitions with a basis for improvements, better training or supervision, as well as the ability to press for additional changes in interventions or environments or for the enforcement of agreed-upon policies.
To convey information effectively, clear communication is essential. Visual presentations of data, such as maps from geographic information systems (GIS), or the Supermarket Need Index, are powerful tools for sharing research (Smith et al., 2011b). Visual presentations can also inform program design and engage policy makers and stakeholders—including community members (IOM, 2009a). Such presentations, however, are not sufficient by themselves; at a minimum, people need to know their options for action and they often require some guidance about how to implement those options. Community leaders often benefit from lessons learned in other localities and appreciate when evaluation results are framed in terms of comparisons to other situations and locations and of knowledge of community conditions (IOM, 2012c; Kirkpatrick and McIntyre, 2009; Lebel et al., 2011).
How Can Communities Develop Capacity to Use Evaluation?
Now that guided tools and specific data such as GIS and community assessments are required activities for health departments and nonprofit hospitals, they offer opportunities for community leaders and community coalitions to focus their obesity prevention efforts. However, no one knows how much these tools are used. Some jurisdictions require Health Impact Assessments (HIAs) of proposed interventions in other sectors. These requirements provide opportunities to work with other sectors on improving the positive impact and minimizing the negative impact on health of their proposed interventions. Several HIAs have influenced decisions and, at a minimum, helped to frame policy debates (Henderson et al., 2011; Kids Safe & Healthful Foods Project, 2012). Yet, again, it is unclear how much community partnerships actually use such tools. Chapters 7 and 8 include these and other tools and strategies that may increase their use, such as community-based participatory research and policy mandates.
American Public Health Association (2006) and the Council of State and Territorial Epidemiologists Executive Committee (2007) have called for evaluation of the impact of community assessments, yet only five studies of communities’ use of community assessments have been found as of 2012. The evidence for use appears to be mixed. Two surveys of health departments found an impressively high level of use: 100 percent of community health departments in Kansas reported using community assessments to identify health priorities (Curtis, 2002) while 73 percent of community assessments conducted by local health departments in Washington state were used this way (Spice and Snyder, 2009). Community assessments also facilitated better communication among community groups, helped with the development of new
partnerships, and facilitated understanding of problems (Curtis, 2002; Solet et al., 2009; Spice and Snyder, 2009). In Kansas, 72 percent of the communities completing community assessments reported starting efforts to address the identified health priorities (Curtis, 2002). In Washington, community assessments were used to develop health programs, strategies, or services (42 percent); develop or modify health policies (21 percent); influence budget decisions (23 percent); and establish or modify agency strategy (26 percent) (Spice and Snyder, 2009). Yet, in New York State, researchers piloting and field-testing an evaluation instrument had difficulty identifying community stakeholders outside of health departments who were knowledgeable about community assessments (Myers and Stoto, 2006; Stoto et al., 2009). Coalitions for community substance abuse control have been found to make little use of other technical assistance tools, resources, or consultation, even when offered without cost (Hallfors et al., 2002). The tools exist, and many are described in Chapter 7 and 8. There are certainly opportunities to increase their utility among community groups.
COMMUNITY DECISION MAKERS AND MANAGERS AS EVALUATION USERS
Why Community Decision Makers and Managers?
Community decision makers include mayors, city planners and managers, city councils, health departments, parks and recreation directors, transportation directors, school administrators, and school boards and other policy bodies. Administrators at this level may directly manage activities related to obesity prevention. They may lead or be part of formal community coalitions, or they may not, but they are often the drivers for change. (The needs of state policy and management actors are addressed later in Chapter 2.)
Policies, interventions, and environmental changes instituted by community decision makers are burgeoning (IOM, 2012c; Ross et al., 2010). Community and state governments sometimes serve as laboratories that may innovate, implement, evaluate, and pave the way for federal policies. State and community public health departments and community coalitions are taking an increasing interest and role in the use, or potential use, of evaluative information about such policies (IOM, 2009a). Learning communities and practitioner networks are beginning to emerge as policy makers innovate and share information about how to institute and implement relevant policies. Following on principles from Diffusion of Innovations (Rogers, 2003), several of the examples in this chapter relate to early adopters, often opinion leaders, who are taking actions to address obesity and often provide lessons to others. In many cases, community actions are taking place in light of limited research-tested evidence on what works to prevent obesity, thus highlighting the need for strong evaluation resulting in so-called practice-based evidence (Green and Glasgow, 2006).
Media attention to community or regional evaluations of innovations can accelerate their adoption and spread. This dynamic has important implications for innovations that need testing (Leviton et al., 2010a) and for generalizing about innovations that are promising (Leviton, 2001). For all these reasons, community and state policy agendas are quite advanced compared to the federal agendas on obesity prevention: examples include instituting incentives and disincentives for healthful eating; reconstructing built environments; and encouraging child care, health care, worksite, and school policies. As in the case of tobacco, bold innovations in policy and environmental change appear to be coming first from community and state levels. As in the case of tobacco, lobbying by forces opposed to these policies may be less effec-
tive at state and community levels than at the national level because the multiplicity of community initiatives can outrun the lobbyists who are organized primarily to work with state and federal lawmakers.
What Do Community Decision Makers and Managers Need?
According to the Committee’s interviews and workshop, community decision makers need to track progress in preventing obesity so they know when to apply course corrections, manage implementation, and emphasize or de-emphasize a course of action. Yet, the data necessary to do so are often unavailable at the community level or not available in a timely or accessible manner. Community body mass index data in particular are often not available, although they are valued by the public and by school administrators (Haboush et al., 2011).
Like community coalitions, community decision makers also need a good sense of “what works” and what they should do given the situation of their particular community. They need to assess the strategies that might be recommended by federal and state decision makers to determine whether they are feasible for the cost, acceptable, and likely to be effective in their particular setting, with their particular population to be served (CDC, 2013c).
Community policy makers and managers also need to be responsive and accountable to constituents and external funders. Yet accountability often takes the form of an evaluation report to government or private funders, which can impair stakeholders’ learning (about what works, about implementation, and about assumptions). Community program managers tend to regard evaluation as something they do for others, not for themselves (Patton, 2008; interviews), although evaluation has been associated with program sustainability (RWJF, 2009b). When practitioners and managers have an interest in or use for what is reported, the quality and relevance of the information is almost always higher. Community stakeholders are more likely to be interested in and have use for the evaluation results if they were engaged in posing the evaluation questions (Rossi et al., 2004).
How Can Useful Evaluations Be Produced for Community Decision Makers and Managers?
It is important to assure that those who are actually planning and implementing obesity prevention have a stake in evaluation as well. Too often, evaluations are not requested by community coalitions, decision makers, or managers, but are rather imposed on them by funders or by higher levels of government. Those imposing evaluation from outside feel urgency to do so in order to hold community efforts accountable for the use of funds or the implementation of law. Accountability is an important function of evaluation, and users at the federal and state levels need better information for this purpose. Unfortunately, the accountability focus tends to be incompatible with optimal learning and program improvement (Chelimsky, 1997; Patton, 2008). Certainly if outsiders pose evaluation questions that are unimportant to communities, make erroneous or even dangerous assumptions about community context, or select incomplete data sources, it should come as no surprise if communities see the reports as irrelevant. These problems have occurred regularly throughout the history of modern program evaluation (Shadish et al., 1990).
For this reason, a variety of participatory approaches to community assessment and summative evaluation have emerged to balance the accountability focus and offer practitioners and community program managers something of value from evaluation. These approaches include community-based par-
ticipatory evaluation for affected community members and community coalitions (Green and Glasgow, 2006; Israel et al., 2012; Jagosh et al., 2012), empowerment evaluation geared primarily toward those implementing programs (Fetterman and Wandersman, 2005), and utilization-focused evaluation for all stakeholders (Patton, 2008). As noted in Chapter 7, these methods do not replace the importance of systematic measurement to reveal needs; however, they assure that relevant perspectives and information are included. Community situations are complex; those conducting community assessments and summative evaluations will have a better chance of understanding that complexity and applying existing knowledge about “what works.” They will also have a better chance of educating community users about the complexities of obesity prevention in context.
The capacity to use evaluation information, let alone conduct evaluations, is limited in many community prevention settings. This issue appears to be a function both of the organizations themselves and of the relevance and quality of evaluative information (IOM, 2012c; Labin et al., 2012; Ohri-Vachaspati and Leviton, 2010). Also, in obesity prevention, many agencies cannot afford to collect recommended measures at the state or community levels (IOM, 2012c). “Knowledge brokers” become resources to help organizations apply the findings of evaluative reports. Such knowledge brokers at the community level can include the staff of health departments, universities or colleges, and nonprofit organizations that are organized for this purpose. State health departments and the more than 2,800 community health departments in the United States have the potential to play a special and sustainable role in implementing community obesity prevention, and in particular in the conduct and use of community obesity prevention evaluations (Blanck and Kim, 2012). However, their evaluation capacity is often limited (Cousins et al., 2011). Certain national websites and guides can help to serve the knowledge broker role for community users. For example, the Community Tool Box website,1 a public service of the University of Kansas, had more than 800,000 unique users in 2012, indicating its value to practitioners and planners (see Chapter 6) (personal communication, S. W. Fawcett, University of Kansas, October 9, 2012). Online data resources provide similar value. One example is the Data Resource Center for Child and Adolescent Health, which provides hands-on support to community and state policy makers across the country (The Child and Adolescent Health Measurement Initiative, 2012).
HEALTH CARE PROVIDERS AND HEALTH INSURANCE PLANS AS EVALUATION USERS
Why Health Care Providers and Health Insurance Plans?
Nonprofit hospitals can participate in community initiatives for obesity prevention as part of their community benefit requirements under the Patient Protection and Affordable Care Act, Public Law 111-148, 111th Cong. (March 23, 2010). The Act revised the tax-exempt status of nonprofit hospitals to make their required “community benefit” activities transparent, concrete, measurable, and responsive to identified community needs. For this purpose they need to conduct community assessments and adopt an implementation strategy. Health insurance plans have an interest in the evaluation of obesity prevention because of their need to manage the financial risk related to the costly health consequences of excess weight, such as diabetes and hypertension. Reimbursement policies could be highly influential in determining how much high-quality, effective individual counseling health providers give.
Individual health care providers can be strong advocates for policy and environmental changes to give their patients a better chance to control weight (McPherson et al., 2012). Health care providers have the opportunity to guide adult patients and parents of pediatric patients about healthful diet and physical activity, although knowledge of energy balance guidelines and the assessment and behavioral management of overweight and obesity by primary care providers remain at a relatively low level considering the magnitude of the problem (Pronk et al., 2012; Smith et al., 2011a). In particular well child care offers opportunities to address obesity prevention in the context of other advice on child rearing (National Initiative on Children’s Healthcare Quality, 2013). In other areas such as smoking cessation, provider advice to quit is effective at a population level (Stead et al., 2008). Providers, however, raise the issue of weight control with patients much less frequently than needed (Smith et al., 2011a).
What Do Health Care Providers and Health Insurance Plans Need?
Nonprofit hospitals want to know “what works” at a community level to assure good use of community resources (IOM, 2012c). Based on their conduct of community assessments, they should be interested in knowing what should be done, and given the nature of their bottom line, they are likely to be interested in cost. Health insurance plans see a challenge in accurately translating how reduction in risk factors can translate into improved health status and overall cost-savings. In particular, health insurance plans see a need for cost-effectiveness of various strategies for building the business case for employers and consumers. The Committee’s workshop revealed that users see a lack of standardized data collection as a major challenge to this goal (IOM, 2012c).
Health insurance plans note that employers increasingly want their workers to have access to community programs and are asking for information on those resources. Tracking the use of those resources is a challenge, and for health insurance plans the biggest obstacle is motivating participation and commitment by consumers to complete all aspects of prevention programs, especially if the benefits are slow to be realized. Health care providers and health plans also give a high priority to the measurement of, and improvements in, racial and ethnic disparities in health. Some health insurance plans are able to use “realtime” data to show participation and utilization of health care and community resources. Outcome data are helpful for targeting and refining communications to current and potential participants in programs.
Individual health care providers need better information on “what works” to better enable them to make recommendations, in the specific context of their communities and health care settings (Green et al., 2012). Some evidence suggests that they believe most weight control interventions are ineffective and that family, cultural, social, and community factors are largely responsible (Leverence et al., 2007). Recent data from the National Survey of Energy Balance Related Care among Primary Care Physicians indicates that knowledge levels of energy balance guidelines (i.e., physical activity, diet, and weight) among primary care physicians who treat children are low. Among primary care physicians who treat adults, knowledge levels appear high for overweight and obesity guidelines but less so for physical activity and dietary guidelines (Pronk et al., 2012). Hence, additional training and guidelines that may be integrated into clinical care delivery processes appear warranted.
How Can Evaluations Be More Useful for Health Care Providers and Health Insurance Plans?
The most important added value of evaluations for health care providers and health insurance plans is that they give specific evidence of the applicability and effectiveness of interventions as implemented under normal circumstances in the real-life, real-time context in which they are conducted. An evaluation’s utility is enhanced if the users of the evaluation evidence are actively engaged as participants in planning the evaluation, in analyzing and interpreting the results, and in incorporating the results into the planning of program adaptations and extensions.
Across communities, health insurance plans are uniquely positioned to align stakeholder interests and generate outcomes of mutual interest. Key stakeholders include the health care providers, the purchasers of health benefits, and the insured people. To position obesity prevention evaluation as a valued and relevant activity, the incentives to pursue evaluations need to be aligned with the interests of each stakeholder (Pronk and Kottke, 2013). For the health insurance plan, the interest is an economic rationale. For the other listed stakeholders, interests include a quality-of-care rationale, a cost-savings and productivity rationale, and a function and health experience rationale, respectively. Making those interests explicit and tangible through the use of evaluation may be of significant interest to any or all of these stakeholders.
EMPLOYERS AS EVALUATION USERS
Employers control access to the workplace, an important and pervasive setting for health promotion (Green and Kreuter, 2005). Employers show increasing interest in wellness programs because they attract competitive employees, have potential for cost savings, and are perceived as an important benefit and the right thing to do (Berry et al., 2010). With passage of the Patient Protection and Affordable Care Act, wellness programs are likely to expand further as more employers start to self-insure and begin to see prevention savings accrue directly to their bottom line. A RAND Employer Survey indicates that 51 percent of all employers offer wellness programs, and 79 percent of firms employing 50 or more employees provide access to a wellness program (Mattke et al., 2013). The percentage of employers offering access to a wellness program increases markedly with the number of employees (39 percent for firms with 50-100 employees; 85 percent for firms with 1,001 or more). Obesity prevention and treatment for employees is a major focus, including body mass index screening at 69 percent of firms offering clinical screenings in their wellness programs. Incentives for workplace wellness programs may include reduced insurance premiums or waiver of copay and deductible or increased benefits. Of employers offering wellness programs, 25 percent and 28 percent offer incentives for employee participation in weight management programs and fitness programs respectively. Three percent of employers provide incentives for reaching a target body weight and 6 percent for reaching target fitness levels. Incentives for reaching these targets may become more pervasive because the Patient Protection and Affordable Care Act will increase the permitted limits on such incentives from 20 to 30 percent of the total cost of coverage in 2014 (Mattke et al., 2013).
What Do Employers Need?
Employers express confidence that wellness programs will reduce not only health care costs, but also absenteeism and health-related productivity losses (Mattke et al., 2013). Certainly the clinical benefit from obesity treatment supports employer optimism (Powell et al., 2007), and a variety of analyses indicate savings from some, but not all, wellness activities (Mattke et al., 2013). Yet only about half of these employers surveyed by RAND had evaluated program impacts, and only 2 percent reported actual savings estimates (Mattke et al., 2013). The limitations in the data collected matters greatly because for prevention of obesity both impacts and savings depend on the design of the wellness programs. The employers’ version of the “what works?” question is about designing the best program for their employees.
How Can Evaluations Be More Useful for Employers?
Because so many claims have been made for employee wellness programs, employers can be skeptical of the benefits. Evaluations are more useful to employers when they provide insights about the best program design. For example, a systematic review indicated that environmental and policy changes by themselves are not effective in changing employee behavior; health education and other interventions are still needed (Kahn-Marshall and Gallant, 2012). The employee incentive component of wellness program design also needs evaluation. Because participation, retention, and adherence rates vary across worksites and segments of the employee population, employers might want to target incentives to problem areas, such as dropouts from smoking cessation or sedentary lifestyles (Berry et al., 2010; Leviton, 1987). In general, strategies to increase participation are likely to be needed. The RAND Employer Survey indicates that among firms offering weight management programs, an average of only 11 percent of targeted employees participated, and, among firms offering fitness programs, only 21 percent of targeted employees participated (Mattke et al., 2013).
Another way to make evaluations more useful to employers is to make explicit the cost and cost-effectiveness of different program options. In the RAND Employer Survey, the principal reason that employers gave for not providing wellness programs was the cost—yet some programs may be highly affordable (Mattke et al., 2013). Screenings range from free to costly; Mattke et al. (2013) concluded that for every $10 of incentive for weight loss, the average adult male employee would lose an additional 0.03 pounds or would increase exercise by more than 20 minutes for an additional 0.01 days.
A final way to make evaluations more useful is to extend the evaluation of wellness programs to the families of employees, for whom employers also bear the cost of health coverage. Yet there is a surprising lack of information about employer-based wellness programs for families—RAND’s 2013 report does not mention it at all (Mattke et al., 2013). Although the advantage of convenient access may be less in a family-based program, family-based approaches to weight management are strongly supported by research (Epstein et al., 2007; Gruber and Haldeman, 2009).
FEDERAL AND STATE POLICY MAKERS AS EVALUATION USERS
Why Federal and State Policy Makers?
Policy makers fill essential roles in government, business, and nonprofit organizations and have power to greatly influence obesity prevention. An example of the pervasive importance of federal agency
The National Prevention Council Action Plan
The Patient Protection and Affordable Care Act (Public Law 111-148, 111th Cong., March 23, 2010) requires coordination and leadership from 17 federal departments, agencies, and offices to implement the National Prevention Strategy, in which all sectors work together on evidence-based prevention, wellness, and health promotion. Obesity prevention is not an explicit focus of the Action Plan for the National Prevention Strategy, but the related issues of healthful diet and increased physical activity are pervasive in the Plan. The various agencies approach the Plan in ways that align with their own missions. For example, the Department of Transportation focuses on health in terms of encouraging active transportation such as bike lanes and Safe Routes to School. To guide their activities, the agencies need to know “what works” to promote health. Also, much of the research on health promotion focuses on individual behavior change, but several agencies regard structural changes as outcomes. Comparable data across federal agencies would help with policy development and alignment of federal activities.
SOURCES: Summary of the comments of Corrinne Graffunder (IOM, 2012c) and the National Prevention Council at http://www.surgeongeneral.gov/initiatives/prevention (accessed Noveber 11, 2013).
policies for obesity prevention can be seen in Boxes 2-1 and 2-2. Government officials make and administer policy at federal and state levels. For example, more than half the state health agencies have at least some regulatory powers and can influence policies related to obesity (Blanck and Kim, 2012).
Leadership of nongovernmental organizations also sets policy. For example, the YMCA and accrediting and licensing bodies like the National Association for Family Child Care set standards for physical activity in their programs based on best evidence and what is feasible (National Association for Family Child Care, 2013; YMCA, 2011). Businesses set policies for foods served in their cafeterias and for physical activity at the workplace. To inform this process, the Alliance of Community Health Plans and the National Business Group on Health rely on research and evaluation to assist them in discovering “what works” for obesity prevention at the workplace, as well as determining the reach and “dose” of a needed strategy, and documenting implementation (IOM, 2012c).
What Do Federal and State Policy Makers Need?
For policy makers, the most pressing questions are, “What is the comparative effectiveness of alternative strategies? What is their cost and cost-effectiveness? How does one geopolitical jurisdiction compare with others? How do trends inform us about the need for obesity prevention and the best way to define issues?” For funder organizations, the “How are we doing?” question often takes the form of a need to hold grantee organizations accountable for the use of funds, as in the case of the Centers for Disease Control and Prevention’s (CDC’s) Communities Putting Prevention to Work (CPPW) and Community Transformation Grants (CTGs) (CDC, 2013a,b). The accountability function is important, but it introduces problems for learning as described below.
Federal and state lawmakers and their staff are driven by the calendar for consideration and reauthorization of various policies and programs. The types of information that can be presented and absorbed depend critically on this cycle. Actual decisions are made within a relatively small window of time. Yet, evaluation evidence can influence decisions over a longer time period than this tight window (see Boxes 2-1 and 2-2 for examples). Information can play a role in setting the agenda for policy: as the time to make decisions nears, information can help to frame the choices; after policy enactment, it can assist implementation, help motivate adjustments, or provide a rationale for policy abandonment (Dunn, 2011; Ottoson et al., 2013). Box 2-2 illustrates this process for the federal school meals programs.
Like lawmakers, federal and state agency officials are often driven by the policy development cycle; unlike lawmakers, they often draw on a broad portfolio of research, policy, and evaluation information and experience to inform the process (Ginsburg and Rhett, 2003). In policy areas where research and evaluation are more fully developed than for obesity prevention, and more strategies have received adequate testing, federal managers oversee the process of vetting strategies for effectiveness such that they can be endorsed or financially supported for implementation at state and community levels (CDC, 2013c,d; NIH, 2013). For obesity prevention and physical activity, the Community Preventive Services Task Force
Policy Evaluation Improves Foods Sold and Served in Schools
Research and evaluation have long helped to shape policy for the federally funded child nutrition programs. Two examples illustrate this impact. First, the Healthy Hunger-Free Kids Act of 2010 (Public Law 111-296, 111th, Cong., 2nd sess. [December 13, 2010], 124, 3183) for the first time provided federal authority to regulate the sale of competitive foods (those that “compete” with the school lunch and breakfast). Prior to 2010, federal authority to regulate foods outside the school meals was limited to restrictions on the sale of “foods of minimal nutritional value” (e.g., carbonated beverages and certain candies). Analyses by the University of Illinois at Chicago’s Bridging the Gap Program found that both state and local district policies limiting these competitive foods and beverages were weak and inconsistent (Hirschman and Chriqui, 2012). In addition, these analyses demonstrated that strong policies limiting competitive foods have positive effects on student food consumption. These and other findings point to the need for the kinds of improvements incorporated into the law and the recently proposed federal rule governing competitive foods (USDA, 2013).
In the second example, since 1980, the U.S. Department of Agriculture has used research and evaluation studies to set standards and requirements for the school meals programs. For example, four separate waves of the School Nutrition Dietary Assessment Study (SNDA) have collected nationally representative data on meals offered and served, and two collected dietary intake information at school and over 24 hours on school days. SNDA-III, conducted in school year 2004-2005, was cited heavily in an Institute of Medicine report (IOM, 2008) that recommended updates to the dietary requirements for school meals. The 2008 IOM report provided the scientific basis for new regulations of school meals, requiring more whole grains, fruits, and vegetables; less sodium; only fat-free or low-fat milk; and age-appropriate calorie intake.
recommends several evidence-based strategies,2 and CDC has promoted additional policy and environmental changes to prevent obesity along with measures to assess those changes (Kettel Khan et al., 2009). However, the federal level does not yet drive the translation process for obesity prevention because, as seen in the systematic reviews conducted for the Community Preventive Services Task Force, most of the suggested policy and environment changes for obesity prevention are “evidence-informed” or “promising” rather than “evidence-based” at this time (The Community Guide, 2012). These ratings of the evidence and the occasional finding of “insufficient evidence” are sometimes interpreted by practitioners incorrectly as “ineffective.” They do not indicate ineffective interventions, but interventions for which the level of certainty of effectiveness and applicability do not permit a stronger recommendation.
How Can Evaluation Be More Useful to Federal and State Policy Makers?
Politicians have a markedly different frame of reference from scientists. To bridge this gap, repeated calls have been made for knowledge brokers who can translate research into policy, such as national experts, congressional agencies such as the Congressional Research Service or the U.S. Government Accountability Office, or advocates (Brownson et al., 2006; Choi et al., 2005; Lindblom and Cohen, 1979). The evaluations with the greatest documented effect on policy have systematically bridged this divide (Chelimsky, 1991; Leviton and Boruch, 1983). Lawmakers prefer very short jargon-free briefs with graphics and maps, but need substantiation by longer reports to verify the information if necessary (Grob, 2010; IOM, 2012c; Personal communication, M. Gutman, Gutman Research Associates, July 23, 2012). Examples include the state-by-state childhood obesity report cards developed to inform policy makers at the state level using data from the National Survey of Children’s Health in combination with state policy summaries (Childhood Obesity Action Network, 2009). Lawmakers view as useful information that contributes to a body of other evidence about programs (Dunn, 2011; Ginsburg and Rhett, 2003) and captures comparative effects and cost-effectiveness (IOM, 2012c). Maps and charts are particularly useful when they depict health effects about elected officials’ own constituents (IOM, 2012c). Policy makers are highly sensitive to media, and many prefer that personal interest stories accompany data (Sorian and Baugh, 2002). Evaluation findings need to be presented with clear and specific policy recommendations (Dodson et al., 2009; Grob, 2010). Unfortunately most presentations do not meet these criteria. In a recent review of 100 obesity-themed policy briefs, the majority had no tables and few figures, and only 36 percent included photos (Dodson et al., 2012). The average reading level was high, and data on evaluation of dissemination efforts and utilization were sparse. Box 2-3 provides a summary of recommendations to make policy briefs more effective.
ADVOCATES FOR OBESITY PREVENTION AS EVALUATION USERS
Advocates are essential to the policy development process, particularly for public health (Dorfman, 2013). They often serve as knowledge brokers: for example, in their window of opportunity to set the agenda and frame the issues, advocates will make “educational visits” with policy makers. Advocates need to be visible and persistent; legislators in states with less policy action are not as likely to identify
2 See http://www.thecommunityguide.org/CG-in-Action/table.html (accessed November 11, 2013).
Writing and Using Policy Briefs to Convey Evaluation Findings
A policy brief can best communicate research and evaluation by persuading the audience of the urgency of a problem and the need to adopt one of several viable alternatives. An effective policy brief should (1) make the evidence concise and understandable; (2) explain why the evidence is significant; and (3) describe evidence-informed policy options as suitable actions.
• The title should be catchy, informative, and encourage the reader to read on;
• The information in the brief should be clear and concise;
• Include information on the scale/importance of the problem and benefits of intervention;
• Aim for one to two pages, including tables, figures, and photos;
• When a brief is being tailored to a specific policy maker or region, include a compelling story;
• Include some action-oriented, “bottom-line” policy recommendation;
• Include a short list of references and contact information for follow-up;
• Authors of a policy brief should use active, targeted means of dissemination; and
• Dissemination of a brief should be monitored and evaluated.
SOURCES: Dodson et al., 2012; International Development Research Centre, 2008; Stamatakis et al., 2010.
the champions of obesity policy than those with more action (Jones et al., 2012). Yet counties and states ranked as having the lowest indicators of health would be happy to hear from advocates and would be responsive to their concerns (IOM, 2012c).
What Do Advocates Need?
Of course advocates rely on evaluation to persuade, but in the case of obesity, they also need to choose prevention strategies to focus on. As one advocate put it, “The range of possible ways to intervene is overwhelming. The socio-ecological model offers a multitude of different levels on which to intervene and numerous potential targets for intervention within those levels. Where do we start? Obviously, first recourse is with the things that have some support in the evaluation literature. But which ones are most relevant and culturally appropriate to the population at hand, and how many of them need to be done together or in tandem?” The Committee’s interviews and workshops also indicated that political opposition to many of the suggested strategies demands that advocates rely on the research community to be able to support claims about “what works” and their particular applicability to the populations at great-
est risk. Cost and cost-effectiveness findings help to make the business case for obesity prevention (IOM, 2012b). Also helpful is information on what similar communities and states are doing, because there is a desire not to be left behind on important health initiatives.
Advocates also need to know how specific advocacy appeals or framing of the issues and stratagems work in different contexts. Evaluative information can help advocates to track the supports, the allies and opposition, and the opportunities to frame the issues (Beer et al., 2012; RWJF, 2009a). Such tracking also provides intermediate outcomes that can be reported to funders and helps to identify strategic targets. Is advocacy best focused now on federal, state, or community issues? Which policies should be tracked, and at what points in time? How is the opposition framing the issues, and how might they be reframed?
Shifting the focus from obesity by itself to workforce productivity and health costs helps to capture policy maker attention (IOM, 2012c; interviews). Casting light on health inequities may help to advance action, particularly in communities of color and in low-income areas (Kirkpatrick and McIntyre, 2009).
Later in the policy process, when policies must be implemented, advocates rely on monitoring of enforcement to make the case for improvements. For example, ongoing work from the Robert Wood Johnson Foundation–supported Bridging the Gap Program fuels advocacy by providing a repository of information about problems in school implementation of wellness policies for obesity prevention (Chriqui et al., 2013). The Sarah Samuels Center for Public Health Research & Evaluation and University of California, Berkeley, Atkins Center for Weight and Health also fueled advocacy by monitoring food offerings in California schools after landmark legislation restricting competitive foods and beverages was passed (Samuels et al., 2009, 2010). Information on progress and the needs for improvement can also help to preserve the policies themselves, as described in Box 2-4.
How Can Evaluation Be More Useful to Advocates?
Because “all politics is local,” advocates try to rely on community assessment, surveillance, and evaluation data when they can get them. Research articles and even policy briefs must be boiled down to emails or one-page fact sheets. One advocate expressed unease about her ability to appraise a research study critically and her need to rely on a “middleman” (or knowledge broker) to do so. She said she uses the “arsenal of studies” provided by research programs funded by the Robert Wood Johnson Foundation to fuel the arguments for improving the school meals programs through legislation and regulation. Recognizing the need for such support, knowledge brokers such as the Data Resource Center for Child and Adolescent Health provide hands-on consultation to family advocates who want to integrate data findings quickly into their efforts around childhood obesity and related topics (The Child and Adolescent Health Measurement Initiative, 2012).
FEDERAL AND STATE AGENCY ADMINISTRATORS AS EVALUATION USERS
Why Agency Administrators?
Agency administrators oversee accountability and reporting requirements for funds distributed to state and community levels for initiatives such as the Communities Putting Prevention to Work initiative and the Community Transformation Grants (CDC, 2013a,b). Yet, they are also charged with dissemination, translation, and community implementation of “evidence-based,” “evidence-informed,” or “best”
Evaluation of Arkansas Act 1220 for School-Based Obesity Prevention
In 2003, Arkansas passed ambitious legislation to limit vending and à la carte food and beverage items in schools, and it established a state committee that recommended standards (adopted as regulation in 2005) for food offerings and physical activity. The law also required annual measurement of students’ body mass index (BMI) and notification of parents of the results. A 10-year evaluation of the law will be completed in 2013. Along with the BMI measurements themselves, the notification process has had several positive outcomes. Within a year, parents of children who were overweight and obese significantly improved their ability to identify their children’s weight status. Perhaps most importantly, parents and school officials realized that the problem existed in their home communities and schools. Two important political events followed: (1) many school districts implemented recommendations of the state committee well ahead of the regulations in 2007 and (2) public awareness of childhood obesity, progress in implementing the law, and evaluation findings apparently helped to prevent the repeal of the requirement for BMI measurement. The evaluation reports documented that BMI measurements were not controversial and did not increase harms, such as weight-based teasing or unhealthful diets. Student purchases of unhealthful items at school have declined significantly over time. The reports point to substantial changes in school policies, practices, and environments associated with nutrition and physical activity; however, they also reveal some continued violations of law and regulation.
SOURCE: Fay W. Boozman College of Public Health, 2010.
practices. Both public and private funders invest a great deal of time and money in assuring that effective strategies for obesity prevention are identified and shared with those who might adopt them (Brownson et al., 2012). To assure the right selection of strategies, agency administrators prioritize the related questions of a strategy’s reach (how many or what proportion of the population will be affected), the dose or exposure (duration, intensity, and relevance) of intervention that is needed to achieve effects, and fidelity/ adaptation (whether the strategy as implemented locally still retains the critical components that made it successful or promising in the first place) (Green and Glasgow, 2006).
The Government Accountability Office (2013) surveyed federal program managers to assess progress in implementing federal performance monitoring requirements. The survey revealed that only 37 percent of managers had evaluations conducted within the past 5 years, and another 40 percent were not aware of any. Of the managers that had evaluations, 80 percent reported that the evaluations contributed to improved program management or to assessment of program effectiveness. The most important barriers to using evaluations included lack of resources to implement the findings and program contexts, such as differences of opinion among program stakeholders. Like federal and state officials, the managers use bodies of evidence, rather than single evaluations, as a basis for changing programs (GAO, 2013).
Along with federal agencies, state health departments are charged with collecting and using surveillance data to set priorities for addressing health problems (see Box 2-5 for two examples) (Mason et al., 2010). They are well positioned to offer technical assistance and to leverage resources for prevention
Evaluation of Policies to Address Health Problems
The Nutrition and Obesity Policy Research and Evaluation Network (NOPREN), which is sponsored by the Centers for Disease Control and Prevention (CDC), evaluates policies to improve food and beverage environments. NOPREN identifies research gaps; develops common evaluative tools; and improves the evidence on reach, equity, cost-effectiveness, and sustainability of such policies. NOPREN works through six of CDC’s Prevention Research Centers as well as affiliates and collaborative members. For example, three local health departments in Washington State partner with the University of Washington to address policies for menu labeling, including developing lessons for working with restaurants and strategies to inform customer food selection (Blanck and Kim, 2012). Seattle-King County health department works with NOPREN to improve policies for child care and schools. Harvard University works with Boston’s Public Health Commission to provide access to water in Boston’s public schools, and it collaborates with the Massachusetts Department of Public Health and a range of local agencies to test both policy and practice for obesity prevention.
Another example of policy evaluation is the Physical Activity Policy Research Network (PAPRN), which was created by CDC to study the implementation and effectiveness of health policies related to increasing physical activity in communities. The network consists of one coordinating center, Prevention Research Center member centers, CDC technical advisors, and university members who collaborate on a variety of projects. The PAPRN works to identify policies that affect population physical activity, what determines policy success, what is the process of implementing policies, and finally what is the outcome of the policies. For example, PAPRN members at the University of Colorado, Denver, led a study of what makes a successful physical activity coalition or partnership by asking groups located across the country about their mission, history, process, success, and sustainability (Litt et al., 2013). The University of North Carolina, Chapel Hill, evaluated the National Physical Activity Plan to determine the extent to which states pursue and act upon recommendations in the plan and whether the plan is helping states to develop their own state physical activity plan (Evenson et al., 2013; Kohl et al., 2013).
efforts by other state organizations and community agencies or coalitions. State health departments sometimes have good capacity for providing evaluation and interpreting it to decision makers (Cousins et al., 2011). Some state health departments, however, suffer from the same evaluation capacity problems seen among community coalitions and decision makers.
What Do Agency Administrators Need?
Program administrators need a variety of data elements that are not always available for obesity prevention. Intermediate indicators such as changes in programs, policies, or environments are helpful for planning and mid-course corrections (see Box 2-6 for an example). To endorse best or evidence-based practices and provide meaningful technical assistance, agency administrators need the best evidence of effectiveness available. Such evidence sometimes comes not from evaluations, but from research studies that provide more experimental controls on threats to validity or alternative explanations. As noted later
in this report, however, the standard of evidence for such endorsement is difficult to discern in the area of comprehensive community initiatives on obesity prevention, as well as many policy and environmental changes (although there are notable exceptions [e.g., Wagenaar et al., 2010]). At the same time, external validity and generalizability is an emerging need (Glasgow et al., 2006; Green and Glasgow, 2006; Green and Nasser, 2012).
To offer optimal technical assistance, the “What works?” question becomes “Which strategies work, in what settings, with what resources, at what cost, and for what populations?” First posed decades ago in the context of mental health services and education, this is the classsic evaluation’s challenge to external validity (Cronbach and Shapiro, 1982). Yet, for obesity prevention, very little information is available about strategies most likely to be effective in the particular situation of a prospective user, much less about classes of situations for which particular strategies are optimal. How can federal program managers offer optimal technical assistance and training, or facilitate networking, when so little is known about prevalent patterns—types of settings and populations where particular strategies are more or less likely to be effective? How can community program managers choose optimal strategies for their own situation, when so little is known about what will work especially well in their context?
How Can Evaluation Be More Useful to Agency Administrators?
Federal and state administrators have a fiduciary responsibility to the public to assure that resources are used correctly, and they also are charged with making sure that the law is obeyed. Yet the legitimate
Use of the Community Dashboard by the Healthy Kids, Healthy Communities Program
Healthy Kids, Healthy Communities is funded by the Robert Wood Johnson Foundation (RWJF) to support policy and environmental changes in 49 communities nationwide. Program staff work with communities to identify targets for improvement, provide technical assistance on advocacy and resource development, and monitor progress in each community. The Dashboard, a tool that allows the community partners to network and share resources including assessment guides and policy examples, assists them in their efforts. Modeled on previous work by Francisco et al. (1993), each community agrees with program staff in advance about milestones for accomplishment and provides information about these milestones over time. Where progress has slowed, the staff can engage in problem solving with community coalitions. In addition, the Dashboard conveys to funders and to the coalitions themselves the amount of progress that is made, year by year. For example, the Dashboard permits a coalition to display the resources leveraged over time, the number of policies altered, or physical environments changed. This has been enormously helpful to the RWJF in its overall expectations about how quickly certain policies and environments can be expected to change.
SOURCE: Healthy Kids, Healthy Communities (Personal communication, August 2, 2012).
concern over accountability often impairs, threatens, or crowds out important opportunities for learning and program improvement, for both the funder and the funding recipient (Chelimsky, 1997; Patton, 2008). In spite of the Government Accountability Office survey (2013) indicating that managers do use evaluation for program improvement, it is still reasonable to ask whether evaluation for accountability has either the structure or content for optimal national or state program manager learning, except perhaps to point to prevalent implementation problems. The answer, however, is not to abandon accountability, but to enhance the process of evaluation so that it helps to improve, not merely prove, intervention effectiveness.
FUNDER ORGANIZATIONS AS USERS OF EVALUATION
Why Focus on Funder Organizations?
Governmental and philanthropic organizations across the United States have become concerned about the obesity problem, as seen in funding for the Department of Health and Human Services CPPW Initiative by American Relief and Reinvestment Act of 2009 (CDC, 2013a), CTG by the Patient Protection and Affordable Care Act’s Prevention and Public Health Fund (CDC, 2013b), Racial and Ethnic Approaches to Community Health (REACH) by the 2012 Prevention and Public Health Fund (CDC, 2012), and the activities of the IOM Standing Committee on Childhood Obesity by the Robert Wood Johnson Foundation, the California Endowment, the Michael & Susan Dell Foundation, and Kaiser Permanente. Other philanthropic funders include the W.K. Kellogg Foundation in the area of food systems, the Kresge Foundation in the area of health disparities, and a variety of state and community foundations. These private and nonprofit funders can keep the policy conversation going in ways that federal and state agencies cannot. They can champion continued social and system changes conducive to healthy weight, and they can educate to encourage advocacy for change at all levels (although they cannot lobby). They also can publicize progress, as in the recent case of “obesity bright spots” reporting by the media (e.g., Harper, 2013).
What Do Funder Organizations Need to Know?
Funders of obesity prevention aim at health and social change, so they need to see indicators of progress on the way to such changes. They want to build social movements so that their limited dollars can stimulate sustained change by others. The public and key influential individuals generally believe that personal responsibility is to blame for rising obesity rates. Funders believe that this perception is an obstacle to progress and attempt to reframe the cause of obesity as due to policy and environmental factors (Brownell et al., 2010). Funders, like other users, need to see tangible signs of progress in obesity prevention both in the interventions and in the outcomes to retain the interest of leadership and boards of trustees.
Both public and private funders have invested heavily in multi-component, complex community initiatives to obesity prevention. As seen in Chapter 8, however, evaluation of these initiatives is particularly challenging, because of the dynamics of community coalitions, the range of program, environmental, and policy components, and the limitations of available designs. The evidence base is limited, and yet Institute of Medicine reports since 2003 have concluded that this approach is needed (IOM, 2004, 2009b, 2010, 2012a,b). The stakes are high. Funders include W.K. Kellogg Foundation’s Food and Fitness Initiative
(USDA, 2010); the Robert Wood Johnson Foundation’s Healthy Kids; Healthy Communities initiative (RWJF, 2013); the Kaiser Permanente Community Health initiative (Cheadle et al., 2010); the federal CPPW, CTG, and REACH initiatives; the First Lady Michelle Obama’s Let’s Move Campaign (Let’s Move, 2013); and the White House Task Force on Childhood Obesity (The White House, 2010). Indeed, federal funding priorities recognize the major importance of place-based initiatives and have included significant funding for CPPW, CTG, REACH, and others (CDC, 2012, 2013a,b). Therefore, all of these funders feel a pressing need to accumulate evidence about what works in community-level initiatives so that they can invest resources wisely and secure the best possible return on investment.
How Can Evaluations Be More Useful for Funder Organizations?
Funders respond to the same kinds of information as community and federal policy makers. They rely on trusted experts to advise them about investments, so linking them with the best scientists is critical. Those scientists, however, also need to be able to translate research into feasible and relevant actions, another role for the “knowledge broker.” Evaluation can help bridge the research-to-action gap by testing the applicability of the research to the particular settings, populations, and circumstances in which the interventions recommended by the research would be applied or adapted. Funders can then assure their leadership and boards of trustees that their resources are having the intended impact. Evaluation can also be used to identify evidence gaps and testable hypotheses to be addressed through formal research. Such gaps in what is known may inform the next rounds of funding portfolios.
GENERAL FACTORS AFFECTING USEFULNESS OF EVALUATION ACROSS TYPES OF USERS
A variety of factors affecting the utilization of evaluation and policy analysis have been identified in the literature and appear to generalize across types of users. These are particularly important considerations for improving the usefulness of evaluation information on progress in preventing obesity. As summarized by Dunn (2011) and Johnson et al. (2009) , these factors may concern characteristics of the evaluation, decision context, and user involvement. Evaluator competence and hence the quality of the evaluation is often paramount; poor quality evaluations may be used, but they are likely to be regarded as less trustworthy. In addition, the quality of communications is critical: have findings been conveyed in jargon-free language that is action oriented? Credibility depends on evaluation quality, but also on whether the findings are surprising or in line with other information from the body of evidence and experience on the topic, such as representativeness of the situation, population, and resources that were used. The particular findings and their relevance to decisions, as well as whether the information is on time for the window of opportunity, matter a great deal.
Yet, timeliness is also a function of context and of user involvement. As described by Dunn (2011), findings need to be relevant to the particular activities of the policy development process. In the same way, community and state capacity matters: if program managers are not ready or able to receive information about what works, not willing to commit resources to, or capable of, implementing something that works, or have no capacity to improve their existing programs, evaluation findings from other settings can fall on deaf ears, and evaluation will not be undertaken in the absence of intervention in their own setting.
Evaluation Users as Part of a Systems Approach to Evaluation
Chapter 9 deals with the complexity of obesity prevention and outlines a systems approach. Consideration of the wide variety of evaluation users is integral to this approach. Emergent properties of complex systems force an evaluation approach to obesity prevention efforts to deal with reality as it unfolds. As a result, evaluation efforts that focus too much on internal validity, and thereby lack generalizability, will suffer in their relevance to application. A complex systems approach will provide insights into the complex web of interrelationships among multiple levels of activity, multiple sectors across communities or the nation, multiple stakeholder groups, multiple programmatic options, and other factors. It will also consider feedback loops and provide updates on progress based on the whole picture rather than a single element. As an example of consideration of the multiple interrelationships among many factors that affect obesity, the Committee refers to the 2012 Institute of Medicine report on valuing community-based prevention (IOM, 2012b) as well as to the obesity systems map in the Foresight report by the Government Office for Science in the United Kingdom (Vandenbroeck et al., 2007). As the obesity prevention field moves from research into practice, systems approaches provide a realistic set of insights and learnings.
Personal characteristics of the users matter, including whether they are accustomed to using data or to thinking analytically about programs and policies. In addition, their commitment to the evaluation, and the organization’s commitment or receptiveness to evaluation, will affect whether it is used. Characteristics of the decision, including feasibility of implementing recommendations based on evaluation findings, are factors in utilization. So is the political acceptability of potential solutions: if, for example, political sentiment is opposed to government regulation of food and physical activity environments, than it will greatly affect the interventions selected and the interpretation of evaluations. The information needs of the users, as well as competing or complementary information, all affect whether and how the information will be used.
Clearly, all these factors can be enhanced in a particular intervention setting by the degree to which evaluation users in that setting can be engaged in planning and making sense of the evaluation. Their needs must be addressed. Consideration of the wide variety of evaluation users is critical to taking a systems science approach to better understand the complexity of obesity prevention (see Box 2-7). A policy maker can facilitate access to information about timeliness, relevance, other information, and the basis for assessing credibility. A program manager committed to the evaluation is more likely to use evaluation results, feasibility and context permitting (Patton, 2008).
What Are the Priority Questions?
Across the workshop, the interviews, and the literature, the various kinds of evaluation users identified a set of highest-priority questions: (1) “Why is this important?,” (2) “What works to prevent
obesity?,” (3) “What should we do?,” and (4) “How are we doing?” (Farley and VanWye, 2012; IOM, 2012c; interviews; Rodgers and Collins, 2012). In addition, several potential user groups identified better cost information as important. Evaluation users operate at federal, state, and community levels, and in at least three contexts: the policy-making process (Dunn, 2011; Kingdon, 2011); dissemination and diffusion of obesity prevention strategies (Brownson et al., 2012; Rogers, 2003); and community-level implementation, quality improvement, and sustainability of policies and programs (Fetterman and Wandersman, 2005; Ottoson, 2009; Scheirer and Dearing, 2011).
What Actually Gets Used?
An underlying assumption is that data “should” be used in policy and program development and implementation. Yet, the use of research, policy analysis, and evaluation is a process, not a discrete event, just as program planning and policy making are themselves processes that combine scientific evidence with other considerations. Evaluation requires users to interpret and draw out the implications of findings for action, considering both the purpose of the evaluation and the context within which the evaluation occurs (Dunn, 2011; Henry and Mark, 2003; Kirkhart, 2000; Leviton, 2003). Researchers and evaluators are often disappointed when their findings are not used immediately and concretely for funding or implementation decisions (Leviton and Hughes, 1981; Weiss, 1977). Although users sometimes act on findings in this immediate, instrumental way, the process depends on a host of other factors (Brownson et al., 2006; Johnson et al., 2009). Researchers can also become disillusioned when their findings are used to justify decisions that would have been made anyway, or in ways that go beyond the findings or without “fidelity” to the intervention as they had developed and tested it. Yet, their disappointment ignores the legitimate process of political persuasion that requires martialing a variety of arguments for or against a position, as well as the necessity of adapting some tested interventions to the very different people, settings, or circumstances in which they would be applied (Leviton and Hughes, 1981). Most commonly, findings are used conceptually along with other information, such as the experience of implementers, to better understand the nature of a problem, the operation of a program or policy, or the assumptions underlying a logic model or theory of change (Dunn, 2011; Weiss, 1977). Finally, users are often affected by their own participation in research, policy analysis, or evaluation to think more analytically—not necessarily linked to any specific finding (Patton, 1997). The impact of their participation should not be underrated, because it can improve policy through simulations at the national or international levels (Gortmaker et al., 2011) and it can improve logic models and implementation in community obesity prevention programs (Leviton et al., 2010b).
Ways to Improve Usefulness
The literature, workshops, and interviews pointed to several areas for improvement in evaluating progress of efforts to prevent obesity. First, the field needs to develop better and more comparable data, especially at community levels, for indicators relevant to obesity. Also, data collection needs to be feasible for health departments and other organizations that are unlikely to have the resources for elaborate measurement of populations, policies, and environments. Better data will mean better comparisons across time and geopolitical areas, and may lead to better benchmarks or standards for progress. Good intermediate indicators need to be agreed upon to help stakeholders to assess progress in achieving policy, environ-
mental, and behavioral changes in ways that will be most credible. “Knowledge brokers” can fill several roles, including providing brief, cogent summaries of available research, assisting researchers in making the implications of their findings for action clear and concrete, assessing applicability of the research and evaluations elsewhere to the community situation, drawing conclusions and options for action for stakeholders, and assisting them to envision what change would look like. More needs to be known about external validity as well as “what works.” The single-minded emphasis on requiring evaluations for accountability, however, may limit the potential of those reports to provide generalized knowledge about the populations, settings, and resources needed to adequately implement obesity prevention strategies. Structured differently, requirements for producing and presenting evaluative reports could be an enabling process and rich resource to more fully understand external validity.
APHA (American Public Health Association). 2006. Policy statement database: Conduct research to build an evidence-base of effective community health assessment practice. http://www.apha.org/advocacy/policy/policysearch/default.htm?id=1330 (accessed September 28, 2012).
Beer, T., P. S. Ingargiola, and M. F. Beer. 2012. Advocacy & public policy grantmaking: Matching process to purpose. Washington, DC: The Colorado Trust.
Berry, L. L., A. M. Mirabito, and W. B. Baun. 2010. What’s the hard return on employee wellness programs? Harvard Business Review 88(12):104-112, 142.
Blanck, H. M., and S. A. Kim. 2012. Creating supportive nutrition environments for population health impact and health equity: An overview of the nutrition and obesity policy research and evaluation network’s efforts. American Journal of Preventive Medicine 43(3 Suppl 2):S85-S90.
Brownell, K. D., R. Kersh, D. S. Ludwig, R. C. Post, R. M. Puhl, M. B. Schwartz, and W. C. Willett. 2010. Personal responsibility and obesity: A constructive approach to a controversial issue. Health Affairs 29(3):379-387.
Brownson, R. C., C. Royer, R. Ewing, and T. D. McBride. 2006. Researchers and policymakers: Travelers in parallel universes. American Journal of Preventive Medicine 30(2):164-172.
Brownson, R. C., G. A. Colditz, and E. K. Proctor, eds. 2012. Dissemination and implementation research in health: Translating science to practice. New York: Oxford University Press.
CDC (Centers for Disease Control and Prevention). 2012. Racial and ethnic approaches to community health (REACH). http://www.cdc.gov/reach/about.htm (accessed April 10, 2013).
CDC. 2013a. Communities Putting Prevention to Work. http://www.cdc.gov/communitiesputtingpreventiontowork (accessed April 4, 2013).
CDC. 2013b. Community Transformation Grants (CTGs). http://www.cdc.gov/communitytransformation/index.htm (accessed April 4, 2013).
CDC. 2013c. What is the Community Guide? http://www.thecommunityguide.org/about/index.html (accessed May 24, 2013).
CDC. 2013d. Replicating effective programs plus. http://www.cdc.gov/hiv/topics/prev_prog/rep/index.htm (accessed May 24, 2013).
Cheadle, A., P. M. Schwartz, S. Rauzon, W. L. Beery, S. Gee, and L. Solomon. 2010. The Kaiser Permanente Community Health Initiative: Overview and evaluation design. American Journal of Public Health 100(11):2111-2113.
Chelimsky, E. 1991. On the social science contribution to governmental decision-making. Science 254(5029):226-231.
Chelimsky, E. 1997. The coming transformations in evaluation. In Evaluation for the 21st century: A handbook, edited by E. Chelimsky and W. R. Shadish. Thousand Oaks, CA: Sage Publications. Pp. 1-26.
Childhood Obesity Action Network. 2009. State obesity profiles. National Initiative for Children’s Healthcare Quality Child Policy Research Center, and Child and Adolescent Health Measurement Initiative. http://www.childhealthdata.org/browse/snapshots/obesity-2007 (accessed May 28, 2013).
Choi, B. C. K., T. Pang, V. Lin, P. Puska, G. Sherman, M. Goddard, M. J. Ackland, P. Sainsbury, S. Stachenko, and H. Morrison. 2005. Can scientists and policy makers work together? Journal of Epidemiology and Community Health 59(8):632-637.
Chriqui, J., E. Resnick, L. Schneider, R. Schermbeck, T. Adcock, V. Carrion, and F. Chaloupka. 2013. School district wellness policies: Evaluating progress and potential for improving children’s health five years after the federal mandate. School years 2006-07 through 2010-11. Chicago, IL: Bridging the Gap Program, Health Policy Center, Institute for Health Research and Policy, University of Illinois at Chicago.
Council of State and Territorial Epidemiologists Executive Committee. 2007. Research to study and disseminate evidence of effective community health assessments. http://c.ymcdn.com/sites/www.cste.org/resource/resmgr/PS/07-EC-01.pdf (accessed May 24, 2013).
Cousins, J. M., S. M. Langer, L. K. Rhew, and C. Thomas. 2011. The role of state health departments in supporting community-based obesity prevention. Preventing Chronic Disease 8(4):A87.
Cronbach, L. J., and K. Shapiro. 1982. Designing evaluations of educational and social programs. San Francisco, CA: Jossey-Bass.
Curtis, D. C. 2002. Evaluation of community health assessment in Kansas. Journal of Public Health Management and Practice 8(4):20-25.
Dodson, E. A., C. Fleming, T. K. Boehmer, D. Haire-Joshu, D. A. Luke, and R. C. Brownson. 2009. Preventing childhood obesity through state policy: Qualitative assessment of enablers and barriers. Journal of Public Health Policy 30: S161-S176.
Dodson, E. A., A. A. Eyler, S. Chalifour, and C. G. Wintrode. 2012. A review of obesity-themed policy briefs. American Journal of Preventive Medicine 43(3 Suppl 2):S143-S148.
Dorfman, L. 2013. Talking about sugar sweetened-beverage taxes: Will actions speak louder than words? American Journal of Preventive Medicine 44(2):194-195.
Dunn, W. N. 2011. Public policy analysis: An introduction. 5th ed. Upper Saddle River, NJ: Pearson Prentice Hall.
Epstein, L. H., R. A. Paluch, J. N. Roemich, and M. D. Beecher. 2007. Family-based obesity treatment, then and now: Twenty-five years of pediatric obesity treatment. Health Psychology 26(4):381-391.
Evenson, K. R., R. C. Brownson, S. B. Satinsky, A. A. Eyler, and H. W. Kohl 3rd. 2013. The U.S. National Physical Activity Plan: Dissemination and use by public health practitioners. American Journal of Preventive Medicine 44(5):431-438.
Farley, T. A., and G. Van Wye. 2012. Reversing the obesity epidemic: The importance of policy and policy research. American Journal of Preventive Medicine 43(3 Suppl 2):S93-S94.
Fay W. Boozman College of Public Health. 2010. Year six evaluation: Arkansas Act 1220 of 2003 to combat childhood obesity. Little Rock: University of Arkansas for Medical Sciences.
Fetterman, D. M., and A. Wandersman. 2005. Empowerment evaluation principles in practice. New York: Guilford Press.
Francisco, V. T., A. L. Paine, and S. B. Fawcett. 1993. A methodology for monitoring and evaluating community health coalitions. Health Education Research 8(3):403-416.
GAO (Government Accountability Office). 2013. Program evaluation: Strategies to facilitate agencies’ use of evaluation in program management and policy making. Washington, DC: GAO-13-570.
Ginsburg, A., and N. Rhett. 2003. Building a better body of evidence: New opportunities to strengthen evaluation utilization. American Journal of Evaluation 24(4):489-498.
Glasgow, R. E., L. W. Green, L. M. Klesges, D. B. Abrams, E. B. Fisher, M. G. Goldstein, L. L. Hayman, J. K. Ockene, and C. Tracy Orleans. 2006. External validity: We need to do more. Annals of Behavioral Medicine 31(2):105-108.
Gortmaker, S. L., B. A. Swinburn, D. Levy, R. Carter, P. L. Mabry, D. T. Finegood, T. Huang, T. Marsh, and M. L. Moodie. 2011. Changing the future of obesity: Science, policy, and action. Lancet 378(9793):838-847.
Green, L. W., and R. E. Glasgow. 2006. Evaluating the relevance, generalization, and applicability of research: Issues in external validation and translation methodology. Evaluation and the Health Professions 29(1):126-153.
Green, L. W., and M. W. Kreuter. 2005. Health promotion planning: An educational and ecological approach. 4th ed. New York: McGraw-Hill.
Green, L. W., and M. Nasser. 2012. Attention to external validity. In Dissemination and implementation research in health: Translating science to practice, edited by R. Brownson, G. Colditz, and E. Parker. New York: Oxford University Press. Pp. 305-326.
Green, L. W., F. L. Brancati, and A. Albright. 2012. Primary prevention of type 2 diabetes: Integrative public health and primary care opportunities, challenges and strategies. Family Practice 29(Suppl 1):i13-i23.
Grob, G. F. 2010. Writing for impact. In Handbook of practical program evaluation, 3rd ed., edited by J. S. Wholey, H. P. Hatry, and K. E. Newcomber. San Francisco, CA: Jossey-Bass. Pp. 594-619.
Gruber, K. J., and L. A. Haldeman. 2009. Using the family to combat childhood and adult obesity. Preventing Chronic Disease 6(3):A106.
Haboush, A., T. Phebus, D. Tanata Ashby, H. Zaikina-Montgomery, and K. Kindig. 2011. Still unhealthy 2009: Building community research to identify risk factors and health outcomes in childhood obesity. Journal of Community Health 36(1):111-120.
Hallfors, D., H. Cho, D. Livert, and C. Kadushin. 2002. Fighting back against substance abuse: Are community coalitions winning? American Journal of Preventive Medicine 23(4):237-245.
Harper, M. 2013. Teaming up against obesity. Boston Globe, January 14, 2013.
Henderson, T., M. Rader, B. Sorte, M. M. Ratcliffe, A. Lawrence, J. Lucky, and C. Harris. 2011. Health impact assessment: Farm to school and school garden policy, HB 2800. Upstream Public Health and the Health Impact Project.
Henry, G. T., and M. M. Mark. 2003. Beyond use: Understanding evaluation’s influence on attitudes and actions. American Journal of Evaluation 24(3):293-314.
Hirschman, J., and J. F. Chriqui. 2012. School food and nutrition policy, monitoring and evaluation in the USA. Public Health Nutrition 16(6):982-998.
International Development Research Centre. 2008. The two-pager: Writing a policy brief. In The RM knowledge translation toolkit: A resource for researchers. Ottawa, Canada: International Development Research Centre. Chapter 8.
IOM (Institute of Medicine). 2004. Preventing childhood obesity: Health in the balance. Washington, DC: The National Academies Press.
IOM. 2008. School meals: Building blocks for healthy children. Washington, DC: The National Academies Press.
IOM. 2009a. Community perspectives on obesity prevention in children: Workshop summaries. Washington, DC: The National Academies Press.
IOM. 2009b. Local government actions to prevent childhood obesity. Washington, DC: The National Academies Press.
IOM. 2010. Bridging the evidence gap in obesity prevention: A framework to inform decision making. Washington, DC: The National Academies Press.
IOM. 2012a. Accelerating progress in obesity prevention: Solving the weight of the nation. Washington, DC: The National Academies Press.
IOM. 2012b. An integrated framework for assessing the value of community-based prevention. Washington, DC: The National Academies Press.
IOM. 2012c. Evaluating progress of obesity prevention efforts: What does the field need to know? Public Session of the Committee on Evaluating Progress of Obesity Prevention Efforts. October 12, 2012, Washington, DC. Audio available at: http://www.iom.edu/Activities/Nutrition/EvaluatingProgressObesityPrevention/2012-OCT-12.aspx (accessed November 11, 2013).
Israel, B. A., E. Eng, A. J. Schulz, and E. A. Parker. 2012. Methods for community-based participatory research for health. 2nd ed. San Francisco, CA: Jossey-Bass.
Jagosh, J., A. C. Macaulay, P. Pluye, J. Salsberg, P. L. Bush, J. Henderson, E. Sirett, G. Wong, M. Cargo, C. P. Herbert, S. D. Seifer, L. W. Green, and T. Greenhalgh. 2012. Uncovering the benefits of participatory research: Implications of a realist review for health research and practice. Milbank Quarterly 90(2):311-346.
Johnson, K., L. O. Greenseid, S. A. Toal, J. A. King, F. Lawrenz, and B. Volkov. 2009. Research on evaluation use. A review of the empirical literature from 1986 to 2005. American Journal of Evaluation 30(3):377-410.
Jones, E., A. A. Eyler, L. Nguyen, J. Kong, R. C. Brownson, and J. H. Bailey. 2012. It’s all in the lens: Differences in views on obesity prevention between advocates and policy makers. Childhood Obesity 8(3):243-250.
Kahn-Marshall, J. L., and M. P. Gallant. 2012. Making healthy behaviors the easy choice for employees: A review of the literature on environmental and policy changes in worksite health promotion. Health Education and Behavior 39(6):752-776.
Kettel Khan, L. K., K. Sobush, D. Keener, K. Goodman, A. Lowry, J. Kakietek, and S. Zaro. 2009. Recommended community strategies and measurements to prevent obesity in the United States. MMWR Recommendations and Reports 58(RR-7):1-26.
Kids’ Safe & Healthful Foods Project. 2012. Health impact assessment: National nutrition standards for snack and a la carte foods and beverages sold in schools. http://www.pewhealth.org/uploadedFiles/PHG/Content_Level_Pages/Reports/KS%20HIA_FULL%20Report%20062212_WEB%20FINAL-v2.pdf (accessed June 27, 2013).
Kingdon, J. 2011. Agendas, alternatives, and public policies. New York: Longman Publishing Group.
Kirkhart, K. E. 2000. Reconceptualizing evaluation use: An integrated theory of influence. New Directions for Evaluation (88):5-23.
Kirkpatrick, S. I., and L. McIntyre. 2009. The Chief Public Health Officer’s report on health inequalities: What are the implications for public health practitioners and researchers? Canadian Journal of Public Health 100(2):93-95.
Kohl, H. W., 3rd, S. B. Satinsky, G. P. Whitfield, and K. R. Evenson. 2013. All health is local: State and local planning for physical activity promotion. Journal of Public Health Management and Practice 19(3 Suppl 1):S17-S22.
Labin, S. N., J. L. Duffy, D. C. Meyers, A. Wandersman, and C. A. Lesesne. 2012. A research synthesis of the evaluation capacity building literature. American Journal of Evaluation 33(3):307-338.
Lebel, A., M. Cantinotti, R. Pampalon, M. Theriault, L. A. Smith, and A. M. Hamelin. 2011. Concept mapping of diet and physical activity: Uncovering local stakeholders perception in the Quebec City region. Social Science and Medicine 72(3):439-445.
Let’s Move. 2013. Let’s Move. America’s move to raise a healthier generation of kids. http://www.letsmove.gov (accessed June 3, 2013).
Leverence, R. R., R. L. Williams, A. Sussman, and B. F. Crabtree. 2007. Obesity counseling and guidelines in primary care: A qualitative study. American Journal of Preventive Medicine 32(4):334-339.
Leviton, L. C. 1987. The yield from cardiovascular risk reduction at the work place. Journal of Occupational Medicine 29(12):931-936.
Leviton, L. C. 2001. External validity. In International encyclopedia of the social and behavioral sciences. 1st ed, edited by N. J. Smelser and P. B. Baltes. Oxford: Elsevier.
Leviton, L. C. 2003. Evaluation use: Advances, challenges and applications. American Journal of Evaluation 24(4):525-535.
Leviton, L. C., and R. F. Boruch. 1983. Contributions of evaluation to education programs and policy. Evaluation Review 7(5):563-598.
Leviton, L. C., and E. F. X. Hughes. 1981. Research on the utilization of evaluations. A review and synthesis. Evaluation Review 5(4):525-548.
Leviton, L. C., L. Kettel Khan, and N. Dawkins. 2010a. Editors notes. In The systematic screening and assessment method: Finding innovations worth evaluating, edited by L. C. Leviton, L. Kettel Khan, and N. Dawkins. New Directions in Evaluation 2010(125):1-5.
Leviton, L. C., L. K. Khan, D. Rog, N. Dawkins, and D. Cotton. 2010b. Evaluability assessment to improve public health policies, programs, and practices. Annual Review of Public Health 31(1):213-233.
Lindblom, C. E., and D. K. Cohen. 1979. Usable knowledge: Social science and social problem solving. New Haven, CT: Yale University.
Litt, J., H. Reed, S. G. Zieff, R. G. Tabak, A. A. Eyler, N. O. Tompkins, R. Lyn, J. Gustat, K. V. Goins, and D. Bornstein. 2013. Advancing environmental and policy change through active living collaboratives: Compositional and stakeholder engagement correlates of group effectiveness. Journal of Public Health Management and Practice 19(3 Suppl 1):S49-S57.
Mason, M., R. Schmidt, C. Gizzi, and S. Ramsey. 2010. Taking improvement action based on performance results: Washington State’s experience. Journal of Public Health Management and Practice 16(1):24-31.
Mattke, S., H. Liu, J. P. Caloyeras, C. Y. Huang, K. R. Van Busum, D. Khodyakov, and V. Shier. 2013. Workplace wellness programs study: Final report. Santa Monica, CA: RAND.
McPherson, M. E., R. Mirkin, P. N. Heatherley, and C. J. Homer. 2012. Educating health care professionals in advocacy for childhood obesity prevention in their communities: Integrating public health and primary care in the Be Our Voice project. American Journal of Public Health 102(8):e37-e43.
Myers, S., and M. A. Stoto. 2006. Criteria for assessing the usefulness of community health assessments: A literature review. Santa Monica, CA: RAND Corporation.
National Association for Family Child Care. 2013. Quality standards for NAFCC accreditation update 2013. http://nafcc.org/media/pdf/accreditation/QS-date2013.pdf (accessed May 24, 2013).
National Initiative for Children’s Healthcare Quality. 2013. Obesity. http://www.nichq.org/areas_of_focus/childhood_obesity_topic.html (accessed March 12, 2013).
NIH (National Institutes of Health). 2013. NIH roadmap for medical research. http://www.nei.nih.gov/funding/roadmap.asp (accessed January 2, 2013).
NRC (National Research Council). 2012. Using science as evidence in public policy. Washington, DC: The National Academies Press.
Ohri-Vachaspati, P., and L. C. Leviton. 2010. Measuring food environments: A guide to available instruments. American Journal of Health Promotion 24(6):410-426.
Ottoson, J. M. 2009. Knowledge-for-action theories in evaluation: Knowledge utilization, diffusion, implementation, transfer, and translation. In Knowledge utilization, diffusion, implementation, transfer, and translation: Implications for evaluation. New Directions for Evaluation 124:7-20.
Ottoson, J. M., A. G. Ramirez, L. W. Green, and K. J. Gallion. 2013. Using a wider lens than legislation when evaluating whether a public health research or advocacy project has contributed to policy: The Salud America! Experience. American Journal of Preventive Medicine 44(3):S282-S289.
Patton, M. Q. 1997. Utilization-focused evaluation: The new century text. 3rd ed. Thousand Oaks, CA: Sage Publications.
Patton, M. Q. 2008. Utilization-focused evaluation. 4th ed. Thousand Oaks, CA: Sage Publications.
Powell, L. H., J. E. Calvin 3rd, and J. E. Calvin, Jr. 2007. Effective obesity treatments. American Psychologist 62(3):234-246.
Preskill, H., and N. A. Jones. 2009. Practical guide for engaging stakeholders in developing evaluation questions.
Princeton, NJ: Robert Wood Johnson Foundation.
Pronk, N. P., and T. E. Kottke. 2013. Health promotion in health systems. In Lifestyle medicine. 2nd ed, edited by J. M. Ripper. Hoboken, NJ: CRC Press. Pp. 1311-1326.
Pronk, N. P., S. M. Krebs-Smith, D. A. Galuska, B. Liu, R. F. Kushner, R. P. Troiano, S. B. Clauser, R. Ballard-Barbash, and A. W. Smith. 2012. Knowledge of energy balance guidelines and associated clinical care practices: The U.S. National Survey of Energy Balance Related Care among Primary Care Physicians. Preventive Medicine 55(1):28-33.
Rodgers, G. P., and F. S. Collins. 2012. The next generation of obesity research: No time to waste. Journal of the American Medical Association 308(11):1095-1096.
Rogers, E. M. 2003. Diffusion of innovations. 5th ed. New York: The Free Press.
Ross, R. K., R. J. Baxter, M. Standish, L. S. Solomon, M. K. Jhawar, P. M. Schwartz, G. R. Flores, and J. Nudelman. 2010. Community approaches to preventing obesity in California. American Journal of Public Health 100(11):2023-2025.
Rossi, P. H., M. W. Lipsey, and H. E. Freeman. 2004. Evaluation: A systematic approach. 7th ed. Newbury Park, CA: Sage Publications.
RWJF (Robert Wood Johnson Foundation). 2009a. Evaluation of communities creating healthy environments. http://www.rwjf.org/en/research-publications/find-rwjf-research/2009/09/evaluation-of-communities-creating-healthy-environments.html (accessed March 21, 2013).
RWJF. 2009b. Evaluation of local funding partnerships: Analyzing the factors that influence sustainability and replication of projects after funding ends. http://www.rwjf.org/pr/product.jsp?id=38411 (accessed September 7, 2012).
RWJF. 2013. Healthy kids, healthy communities. http://www.healthykidshealthycommunities.org (accessed April 10, 2013).
Samuels, S. E., S. Lawrence Bullock, G. Woodward-Lopez, S. E. Clark, J. Kao, L. Craypo, J. Barry, and P. B. Crawford. 2009. To what extent have high schools in California been able to implement state-mandated nutrition standards? Journal of Adolescent Health 45(3):S38-S44.
Samuels, S. E., K. S. Hutchinson, L. Craypo, J. Barry, and S. L. Bullock. 2010. Implementation of California state school competitive food and beverage standards. Journal of School Health 80(12):581-587.
Scheirer, M. A., and J. W. Dearing. 2011. An agenda for research on the sustainability of public health programs. American Journal of Public Health 101(11):2059-2067.
Scriven, M. 1991. Evaluation thesaurus. Newbury Park, CA: Sage Publications.
Shadish, W. R., T. D. Cook, and L. C. Leviton. 1990. Foundations of program evaluation: Theories of practice. Newbury Park, CA: Sage Publications.
Smith, A. W., L. A. Borowski, B. Liu, D. A. Galuska, C. Signore, C. Klabunde, T. T. Huang, S. M. Krebs-Smith, E. Frank, N. Pronk, and R. Ballard-Barbash. 2011a. U.S. primary care physicians diet-, physical activity-, and weight-related care of adult patients. American Journal of Preventive Medicine 41(1):33-42.
Smith, L., C. Goranson, J. Bryon, B. Kerker, and C. Nonas. 2011b. Developing a supermarket need index. In Geospatial analysis of environmental health, edited by J. A. Maantay and S. McLafferty. Geotechnologies and the environment, Springer Vol. 4. Pp. 205-221.
Solet, D., S. Ciske, R. Gaonkar, K. Horsley, M. McNees, P. Nandi, and J. W. Krieger. 2009. Effective community health assessments in King County, Washington. Journal of Public Health Management and Practice 15(1):33-40.
Sorian, R., and T. Baugh. 2002. Power of information: Closing the gap between research and policy. Health Affairs 21(2):264-273.
Spice, C., and K. Snyder. 2009. Reviewing self-reported impacts of community health assessment in local health jurisdictions. Journal of Public Health Management and Practice 15(1):18-23.
Stamatakis, K. A., T. D. McBride, and R. C. Brownson. 2010. Communicating prevention messages to policy makers: The role of stories in promoting physical activity. Journal of Physical Activity and Health 7(Suppl 1):S99-S107.
Stead, L. F., G. Bergson, and T. Lancaster. 2008. Physician advice for smoking cessation. Cochrane Database of Systematic Reviews (2):CD000165.
Stoto, M. A., S. G. Straus, C. Bohn, and P. Irani. 2009. A web-based tool for assessing and improving the usefulness of community health assessments. Journal of Public Health Management and Practice 15(1):10-17.
The Child and Adolescent Health Measurement Initiative. 2012. Data Resource Center for Child and Adolescent Health. http://www.childhealthdata.org (accessed April 4, 2013).
The Community Guide. 2012. Obesity prevention and control. http://www.thecommunityguide.org/obesity/index.html (accessed May 24, 2013).
The White House Office. 2010. Presidential memorandum: Establishing a task force on childhood obesity. http://www.whitehouse.gov/the-press-office/presidential-memorandum-establishing-a-task-force-childhood-obesity (accessed June 3, 2013).
USDA (U.S. Department of Agriculture). 2010. Food and Fitness Initiative. http://www.nifa.usda.gov/nea/food/in_focus/health_if_food_fitness_initiative.html (accessed April 10, 2013).
USDA. 2013. National school lunch program and school breakfast program: Nutrition standards for all foods sold in school as required by the Healthy, Hunger-Free Kids Act of 2010, proposed rule. Federal Register 78(27):9530-9567.
Vandenbroeck, P., J. Goossens, and M. Clemens. 2007. Foresight tackling obesities: Future choices—building the obesity system map. London, UK: Government Office for Science.
Wagenaar, A. C., A. L. Tobler, and K. A. Komro. 2010. Effects of alcohol tax and price policies on morbidity and mortality: A systematic review. American Journal of Public Health 100(11):2270-2278.
Weiss, C. 1977. Research for policy’s sake: The enlightenment function of social research. Policy Analysis 3(4):531-545.
Weiss, C. H. 1988. Evaluation for decisions: Is anybody there? Does anybody care? Evaluation Practice 9(1):5-19.
Yarbrough, D. B., L. M. Shulha, R. K. Hobson, and F. A. Caruthers. 2011. The program evaluation standards. A guide for evaluators and evaluation users. 3rd ed. Thousand Oaks, CA: Sage Publications.
YMCA. 2011. First-ever national healthy eating and physical activity standards developed for afterschool programs. http://www.ymca.net/news-releases/20110809-afterschool-standards.html (accessed May 24, 2013).