C

Guiding Principles for Evaluation

Table C-1 provides detailed descriptions of the guiding principles identified in Chapter 3 of the Committee’s report. The Committee devised these principles to serve two aims: (1) to guide its deliberations and development of the national- and community-level evaluation plans and (2) to provide guidance to evaluators who will implement the national and community plans in their own settings. Recognizing that each evaluation is subject to its own unique context, constraints, and resources, the principles described below are intended to be suggestive. For each principle, the Committee has provided a plain language definition along with examples of end-user questions to help evaluators to interpret the relevance of a given principle for consideration when (1) identifying indicators of progress, (2) choosing appropriate evaluation processes, and (3) making decisions in regard to evaluating obesity prevention efforts.



The National Academies | 500 Fifth St. N.W. | Washington, D.C. 20001
Copyright © National Academy of Sciences. All rights reserved.
Terms of Use and Privacy Statement



Below are the first 10 and last 10 pages of uncorrected machine-read text (when available) of this chapter, followed by the top 30 algorithmically extracted key phrases from the chapter as a whole.
Intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text on the opening pages of each chapter. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.

Do not use for reproduction, copying, pasting, or reading; exclusively for search engines.

OCR for page 309
C Guiding Principles for Evaluation T able C-1 provides detailed descriptions of the guiding principles identified in Chapter 3 of the Committee’s report. The Committee devised these principles to serve two aims: (1) to guide its delib- erations and development of the national- and community-level evaluation plans and (2) to provide guidance to evaluators who will implement the national and community plans in their own settings. Recognizing that each evaluation is subject to its own unique context, constraints, and resources, the principles described below are intended to be suggestive. For each principle, the Committee has provided a plain language definition along with examples of end-user questions to help evaluators to interpret the relevance of a given principle for consideration when (1) identifying indicators of progress, (2) choosing appropriate evaluation processes, and (3) making decisions in regard to evaluating obesity prevention efforts. 309

OCR for page 309
TABLE C-1 Committee to Evaluate Progress in Obesity Prevention Guiding Principle Definitions and End-User Questions Guiding Principle (indicator/method/end user)* Relevant Definitions/Explanations Accuracy Plain language definition: The extent to which an indicator, measure, or evaluation plan is free from error or bias. Accuracy is derived from both reliability (replicability), (indicator/methods) which is the consistency of an indicator/measure to yield similar results under varying conditions, and validity, which is the extent to which an indicator/measure directly and without error represents a specific concept, construct, or variable. Validity includes internal validity, or the minimization of bias, and external validity, which is the extent to which evaluation findings can be generalized to broader and more diverse populations. Accuracy also encompasses the terms sensitivity, which is the proportion of true positives for a condition or indicator assessed by the measure relative to all those who have the condition or indicator, and specificity, which is the proportion of true negatives for a condition or indicator assessed by the measure relative to all those who do not have the condition or indicator. Examples of end-user questions: Does the information collected accurately represent what is being measured, e.g., is it valid? Does the information collected reflect the same results under different circumstances and across time periods, e.g., is it reliable or reproducible? Are the data analyzed appropriately? Is the design of the evaluation appropriate to answer the question being asked? Are the conclusions from the evaluation justified? Are the results valid and reliable for a specific population or across more diverse populations? What is the specificity and sensitivity of the measure? SOURCES: Adapted from Brownson et al., 2012; IOM, 2010; Yarbrough et al., 2011. Capacity Building Plain language definition: Providing the human resources and infrastructure necessary to conduct and sustain the evaluation including, but not limited to, training, mentoring, (methods/end user) identifying alternative funding, building internal assets, and forming partnerships. Ensuring that relevant end users understand the necessity of evaluation capacity at the outset and throughout the evaluation process. Examples of end-user questions: Do consultation and technical support enhance the competence of those doing the evaluation? Do they enable current and future generations to work together better on this and other evaluations? Are resources identified and used efficiently to continue relevant evaluation efforts? Are end users committed to providing the resources and infrastructure necessary to conduct and sustain the evaluation in both the short- and long-term? SOURCES: Adapted from Brownson et al., 2012; Fawcett, 2002. 310 Evaluating Obesity Prevention Efforts

OCR for page 309
TABLE C-1 Continued Guiding Principle (indicator/method/end user)* Relevant Definitions/Explanations Comparability Plain language definition: The comparison of an indicator/measure with a frame of reference, standard, or benchmark over time among different data sources, methods/ (indicator/methods) protocols, populations, and communities. Goals and benchmarks for obesity prevention in the United States can be found in Healthy People 2020 (HHS, 2010b), the Dietary Guidelines for Americans (HHS, 2010a), and the Physical Activity Guidelines for Americans (HHS, 2008). Examples of end-user questions: What sources of criteria, goals, or guidelines can be used for obesity-related measures? How does a community/group/population rate or rank in terms of obesity indicators/methods relative to other communities/groups/ populations or to the U.S. population in general? How do the obesity indicators/ measures in a community or the nation change over time? How far is a community or group from recommended guidelines or benchmarks for obesity-related measures? SOURCES: Adapted from Fawcett, 2002; IOM, 2009; Rossi and Freeman, 1993; Scriven, 1991. Context Plain language definition: Assessing the conditions, some more modifiable than others, that can help inform practice. The conditions can be political, cultural, social, and (methods/end user) organizational, and include end users’ needs, interpretation, or framing of the results of the evaluation. Consider the broader environment in which an intervention, program, or evaluation is being conducted or implemented. Understanding the context within which an evaluation is being conducted is necessary to identify probable influences on the evaluation design as well as the results of the evaluation. Understanding the context within which an evaluation is conducted also is an important factor in assessing the external validity (i.e., the extent to which evaluation findings can be generalized to broader and more diverse populations) of the evaluation. Examples of end-user questions: Do contextual factors affect the ability to carry out and evaluate a particular intervention? How does a practitioner or researcher best measure and track contextual factors? Does the evaluation design account for important contextual factors that may influence the outcome of the evaluation? From what context are end users operating? SOURCES: Adapted from CDC, 1999; Rabin et al., 2006; Waters et al., 2006; Yarbrough et al., 2011. Appendix C 311

OCR for page 309
TABLE C-1 Continued Guiding Principle (indicator/method/end user)* Relevant Definitions/Explanations Coordination and Plain language definition: Assuring all partner perspectives (including policy makers, Partnership evaluators, community members, representatives of various sectors, etc.) are involved in the development, implementation, and dissemination of the evaluation. (methods/end users) Maximizes identified strengths and assets of each partner, but also works to address needs and increase capacity of all partners including sharing of resources, risks, and responsibilities. Requires open lines of communication among all partners to ensure effective collaborations. Multi-sectoral collaborations are often necessary and can include members from local or state health departments, elected officials, urban planners, businesses or the Chamber of Commerce, school boards or schools, hospitals, universities, nongovernmental organizations such as local affiliates of the American Heart Association, and Cooperative Extension agents. Strong leadership is key to ensuring effective collaborations and partnerships. Partners can facilitate dissemination of evaluation findings through their respective networks and tailor the findings specific to their end-users’ needs. Examples of end-user questions: Are all relevant groups or end users involved in the partnership or collaborative? Are community members and other end users involved in determining what “success” would look like? Are sufficient resources devoted to maintenance of the partnership or collaborative? Is communication adequate to allow for effective coordination among end users within the partnership so that duplication of effort is avoided and scarce resources are leveraged to the maximum extent? During and after the evaluation, are the partnerships or collaboratives given opportunities to see the results and to help interpret their meaning? SOURCES: Adapted from Community-Campus Partnerships for Health, 2012; Fawcett, 2002; Hargreaves, 2010; IOM, 2009, 2010, 2012; WHO, 2010. Dissemination Plain language definition: The development of a systematic and effective approach to communicate and provide information about obesity-related indicators/measures to the (end users) priority population and end users. Examples of end-user questions: What is the plan for communicating obesity-related indicators/measures to the community or group in a timely fashion? Are the appropriate opinion leaders and end users included in the process? Are dissemination materials appropriate for the end users? Does the dissemination plan have adequate reach within the priority population? Is the process effective? SOURCES: Adapted from Brownson et al., 2012; Glasgow et al., 1999; Rogers, 2003. 312 Evaluating Obesity Prevention Efforts

OCR for page 309
TABLE C-1 Continued Guiding Principle (indicator/method/end user)* Relevant Definitions/Explanations Feasibility Plain language definition: The effectiveness and efficiency with which an indicator/ measure is capable of being measured with available resources. (indicator/methods) Examples of end-user questions: What tools, staffing, time, funding, or other resources are required to conduct a particular evaluation? Are the required resources accessible to the evaluator or practitioner? What data sources are available at the appropriate level? SOURCE: Adapted from Yarbrough et al., 2011. Health Disparities/Equity Plain language definition: The preventable differences in the burden of disease, injury, violence, and opportunities to achieve optimal health that are experienced by socially (indicator/methods/end users) disadvantaged populations. Examples of end-user questions: Did the evaluation prioritize and include measures that specifically focus on populations that are disproportionately affected by obesity based on geography, race/ethnicity, socioeconomic status, gender, and age? Are indicators/ measures appropriate for socially disadvantaged populations? Do indicators/measures and evaluation plans include input and feedback from end-users from these populations? Are appropriate contextual factors assessed and included in interpretation of results?   SOURCES: Adapted from CDC, 2008; IOM, 2012; WHO, 2010. Impact Plain language definition: The evaluation improves understanding of whether a program or policy causes changes in the desired direction for the outcome of interest and whether (end users) the program or policy has unintended consequences or negative outcomes. Impact assessments involve both qualitative and quantitative methods or a triangulation of methods. Examples of end-user questions: Do we have information about the contribution of community and systems changes (i.e., new or modified programs, policies, and practices) to valued outcomes? Can we see how (and whether) the amount and distribution of community and systems change is related to community-level outcomes? Are there any unintended consequences or negative outcomes associated with the program or policy? SOURCES: Adapted from Fawcett, 2002; Rossi and Freeman, 1993; Wholey et al., 2010. Implementation Plain language definition: The process of adopting and integrating appropriate and routine use of obesity-related indicators/measures and surveillance/evaluation plans into (methods) specific settings to provide ongoing feedback for evaluation of obesity prevention efforts. Examples of end-user questions: Which surveillance/evaluation plan and indicators/ measures should be adopted for a particular community? How can a surveillance/ evaluation plan be incorporated into routine use to provide data on a periodic basis? How can fidelity of implementation be assured? SOURCES: Adapted from Brownson et al., 2012; Glasgow et al., 2012. Appendix C 313

OCR for page 309
TABLE C-1 Continued Guiding Principle (indicator/method/end user)* Relevant Definitions/Explanations Parsimony Plain language definition: The principle that when several indicators/measures could provide similar information, the most succinct and simplest should be selected. (indicator/methods) Examples of end-user questions: Is there duplication among selected indictors/measures, or has the most parsimonious assessment been used? Are contextual measures necessary, e.g., do they measure closely interrelated concepts? Have methods been optimized to ensure the most direct measurement possible? SOURCES: Adapted from Nolan, 1997; Sober, 1981. Priority Setting Plain language definition: Involves development of guidelines, standards, and goals to guide evaluation design, indicator/measure development, and dissemination decisions. (indicators/methods/end users) Requires significant input from end users/partners early in the evaluation design process to ensure that relevant and necessary priorities are identified and accounted for throughout the evaluation design, implementation, and dissemination processes. Accounts for the individual and common goals and objectives of the end users. Includes prioritizing and setting aside necessary resources to support and sustain the evaluation. Examples of end-user questions: Have all end users specified their priorities at the outset of the evaluation design process? Are all end users’ priorities accounted for in the design, implementation, and dissemination of the evaluation findings? Are end-user priorities reflected in the selection of relevant indicators/measures? Have the end users committed the necessary resources to ensure that their priorities are maintained in the evaluation design, implementation, and dissemination? SOURCE: Adapted from WHO, 2010. Relevance Plain language definition: The extent to which the evaluation objectives and design, including the indicators, measures, and surveillance systems, are consistent with the (indicators/methods/end users) identified and emergent priorities, needs, concerns, and values of the end users. The extent to which the indicators, measures, and surveillance systems provide practical, timely, meaningful information consistent with identified and emergent needs of end users. Examples of end-user questions: Are the indicators that will be used to evaluate impact of the program/policy consistent with the end users’ identified and emergent needs? Are the necessary surveillance systems in place to provide the necessary indicators that are responsive to the end-user needs? Are end-user values accounted for in the design of the evaluation? SOURCES: Adapted from Fawcett, 2002; Wholey et al., 2010; Yarbrough et al., 2011. 314 Evaluating Obesity Prevention Efforts

OCR for page 309
TABLE C-1 Continued Guiding Principle (indicator/method/end user)* Relevant Definitions/Explanations Scalability Plain language definition: The extent to which a measure or evaluation method can be expanded to reach a larger population, yet still maintain accuracy and feasibility. (indicator/methods) Examples of end-user questions: Is the evaluation program or measure reaching the entire intended audience or only a subset? What additional resources are required to conduct measurements in the entire population of interest? Will the measure or method retain its validity and reliability when reach is expanded? SOURCES: Adapted from Brownson et al., 2012; Milat et al., 2012; Pronk, 2003. Surveillance Plain language definition: Ongoing, systematic, representative collection, analysis, interpretation, and dissemination of data on public health problems, policies, or (methods) environments of interest. Requires commitment on the part of end users to ensure that necessary surveillance systems are sustained throughout the life of the evaluation. Surveillance systems often are designed or tailored to respond to end users’ new and emergent needs. Requires end-user support and prioritization to ensure that indicators can be assessed over time. Examples of end-user questions: Are quantitative data compiled at regular intervals at the national and/or community levels to enable longitudinal tracking of the outcome or public health problem of interest? Are the data in the surveillance system readily available for immediate use at the national/community levels to identify when a public health problem is emerging, worsening, has reached a plateau, or is improving? Can the surveillance system data be extracted in ways to inform policy-relevant decisions? Are policy-tracking data readily available in systematic and reliable formats to indicate the extent to which communities have adopted given policy(ies) of interest? SOURCES: Adapted from German et al., 2001; Jacobs et al., 2012. Sustainability Plain language definition: The likelihood that monitoring and evaluation plans and indicators/measures will be continued or maintained over an extended period of time (indicator/methods/end users) after external support and funding is terminated. Examples of end-user questions: What factors (e.g., funds, community capacity/ infrastructure, partnerships, policies) are needed to promote sustainability of evaluation efforts? Are the benefits of and feedback obtained from the evaluation plan tangible enough to ensure community support and sustainability? Can the evaluation plan or measure be adapted for sustainability in diverse populations, yet maintain accuracy? SOURCES: Adapted from Scheirer and Dearing, 2011; Shediac-Rizkallah and Bone, 1998. Appendix C 315

OCR for page 309
TABLE C-1 Continued Guiding Principle (indicator/method/end user)* Relevant Definitions/Explanations Systems-oriented Plain language definition: An approach that recognizes the relationships between multiple interconnecting and interacting components within and across multiple levels (methods/end users) as well as other contextual factors in the broader environment. Related to coordination and partnerships among key end users. Requires an understanding and recognition that diet, physical activity, and obesity are each influenced by multiple, inter-connected environments and sectors including, but not limted to, urban planning, food and beverage industries, marketing, health care, education, sport and recreation, transport, commerce, business and industry, agriculture, trade and finance. Examples of end-user questions: How diverse are the perspectives of the end users involved in the program? Is the evaluation intended to find causal factors or explain observed relationships? Is the evaluation intended to support program development, summarize program impact, or monitor trends over time? What sectors are clearly involved in the proposed solutions? Are the roles and responsibilities clearly defined for each of the sectors? Who are the leaders that best represent the sectors and end users involved? To what extent are those leaders engaged in the efforts and connected with their constituencies? SOURCES: Adapted from Hargreaves, 2010; IOM, 2010, 2012; WHO, 2010. Transparency Plain language definition: Clear identification of end users and their objectives/needs early in the evaluation design process helps to communicate openness and provides an (methods/end users) opportunity to raise objections or concerns about the process. When evaluation results are reported, effectively communicating complete descriptions of findings (positive, negative, and neutral), limitations, conclusions, and potential sources of conflicts of interest (including funding sources). Examples of end-user questions: Was the evaluation fair, impartial, and just? Was the evaluation and its findings (1) responsive and inclusive, (2) clear and fair, (3) open, (4) free of conflicts of interest, and (5) considerate of fiscal responsibility? Were end users consulted in the evaluation design and question development processes? Were end users’ new and emergent needs accounted for in the evaluation design, implementation, and dissemination processes?  SOURCES: Adapted from AbouZahr et al., 2007; IOM, 2009; Preskill and Jones, 2009; WHO, 2010; Yarbrough et al., 2011. 316 Evaluating Obesity Prevention Efforts

OCR for page 309
TABLE C-1 Continued Guiding Principle (indicator/method/end user)* Relevant Definitions/Explanations Utility Plain language definition: Evaluations should be designed and conducted by individuals with expertise/experience in conducting evaluations. The evaluation should be designed (methods/end users) to be useful, relevant, and responsive to the full range of end users and their needs including those involved with the program being evaluated as well as those that will be affected by the outcome of the evaluation. The evaluation should be designed to account for individual and cultural values underlying the evaluation purpose, methods, and decisions. Careful attention should be placed on timely and appropriate reporting of evaluation progress and outcomes to the evaluation end users. The evaluation should anticipate potential consequences—both positive and unintended—and reporting should aim to guard against misuse or unintended consequences. End-user questions: To what extent do the evaluation end users find the evaluation methods, processes, and outputs (products) useful or valuable in meeting their needs? Are the results of the evaluation provided to evaluation end users in a timely fashion and in an appropriate format that can readily be used? SOURCE: Adapted from Yarbrough et al., 2011. Value Plain language definition: The relative utility of the surveillance and evaluation information, in relation to end-user needs and culture, while maintaining credibility and (end users) adaptability and avoiding unintended consequences. End-user questions: Does the surveillance or evaluation plan address identified and emerging needs of the community or group? Is the information from the surveillance or evaluation plan shared in a credible and relevant manner, without judgment? Do the end users have input into all facets of the surveillance and evaluation plan? Is care taken to avoid unintended consequences or judgment from the evaluation plan or resulting information? SOURCE: Adapted from Yarbrough et al., 2011. * (1) Indicator, (2) methods, and (3) end user indicate to what the guiding principle is applicable when making decisions about evaluating obesity prevention efforts. Appendix C 317

OCR for page 309
REFERENCES AbouZahr, C., S. Adjei, and C. Kanchanachitra. 2007. From data to policy: Good practices and cautionary tales. Lancet 369(9566):1039-1046. Brownson, R. C., G. A. Colditz, and E. K. Proctor. 2012. Dissemination and implementation research in health: Translating science to practice. New York: Oxford University Press. CDC (Centers for Disease Control and Prevention). 1999. Framework for program evaluation in public health. Morbidity and Mortality Weekly Report 48(RR11):1-40. CDC. 2008. Community health and program services (CHAPS): Health disparities among racial/ethnic populations. Atlanta, GA: U.S. Department of Health and Human Services. Community-Campus Partnerships for Health. 2012. Community-Campus Partnerships for Health: Promoting health equity and social justice. http://www.ccph.info (accessed December 27, 2012). Fawcett, S. B. 2002. Evaluating comprehensive community initiatives: A bill of rights and responsibilities. In Improving population health: The California Wellness Foundation’s health improvement initiative, edited by S. Isaacs. San Francisco, CA: Social Policy Press. Pp. 229-233. German, R. R., L. M. Lee, J. M. Horan, R. L. Milstein, C. A. Pertowski, M. N. Waller, Guidelines Working Group Centers for Disease Control and Prevention. 2001. Updated guidelines for evaluating public health surveil- lance systems: Recommendations from the guidelines working group. MMWR Recommendations and Reports 50(RR-13):1-35; quiz CE31-CE37. Glasgow, R. E., T. M. Vogt, and S. M. Boles. 1999. Evaluating the public health impact of health promotion inter- ventions: The RE-AIM framework. American Journal of Public Health 89(9):1322-1327. Glasgow, R. E., C. Vinson, D. Chambers, M. J. Khoury, R. M. Kaplan, and C. Hunter. 2012. National Institutes of Health approaches to dissemination and implementation science: Current and future directions. American Journal of Public Health 102(7):1274-1281. Hargreaves, M. 2010. Evaluating system change: A planning guide. Princeton, NJ: Mathematica Policy Research, Inc. HHS (Department of Health and Human Services). 2008. Physical activity guidelines for Americans. Washington, DC: U.S. Department of Health and Human Services. HHS. 2010a. Dietary guidelines for Americans. Washington, DC: Government Printing Office. HHS. 2010b. Healthy People 2020. http://www.healthypeople.gov/2020/default.aspx (accessed January 31, 2013). IOM (Institute of Medicine). 2009. Local government actions to prevent childhood obesity. Washington, DC: The National Academies Press. IOM. 2010. Bridging the evidence gap in obesity prevention: A framework to inform decision making. Washington, DC: The National Academies Press. IOM. 2012. Accelerating progress in obesity prevention: Solving the weight of the nation. Washington, DC: The National Academies Press. Jacobs, J. A., E. Jones, B. A. Gabella, B. Spring, and R. C. Brownson. 2012. Tools for implementing an evidence- based approach in public health practice. Preventing Chronic Disease 9:E116. Milat, A. J., L. King, A. E. Bauman, and S. Redman. 2012. The concept of scalability: Increasing the scale and potential adoption of health promotion interventions into policy and practice. Health Promotion International [epub ahead of print]. Nolan, D. 1997. Quantitative parsimony. British Journal for the Philosophy of Science 48:329-343. Preskill, H., and N. A. Jones. 2009. Practical guide for engaging stakeholders in the evaluation process. Princeton, NJ: Robert Wood Johnson Foundation. Pronk, N. P. 2003. Designing and evaluating health promotion programs: Simple rules for a complex issue. Disease Management and Health Outcomes 11(3):149-157. 318 Evaluating Obesity Prevention Efforts

OCR for page 309
Rabin, B. A., R. C. Brownson, J. F. Kerner, and R. E. Glasgow. 2006. Methodologic challenges in disseminat- ing evidence-based interventions to promote physical activity. American Journal of Preventive Medicine 31(4 Suppl):S24-S34. Rogers, E. M. 2003. Diffusion of innovations. 5th ed. New York: The Free Press. Rossi, P. H., and H. E. Freeman. 1993. Evaluation: A systematic approach. 5th ed. Newbury Park, CA: Sage Publications. Scheirer, M. A., and J. W. Dearing. 2011. An agenda for research on the sustainability of public health programs. American Journal of Public Health 101(11):2059-2067. Scriven, M. 1991. Evaluation thesaurus. Newbury Park, CA: Sage Publications. Shediac-Rizkallah, M. C., and L. R. Bone. 1998. Planning for the sustainability of community-based health pro- grams: Conceptual frameworks and future directions for research, practice and policy. Health Education Research 13(1):87-108. Sober, E. 1981. The principle of parsimony. British Journal for the Philosophy of Science 32:145-156. Waters, E., J. Doyle, N. Jackson, F. Howes, G. Brunton, A. Oakley, and C. Cochrane. 2006. Evaluating the effec- tiveness of public health interventions: The role and activities of the Cochrane Collaboration. Journal of Epidemiology and Community Health 60(4):285-289. WHO (World Health Organization). 2010. Population-based prevention strategies for childhood obesity. Report of a WHO forum and technical meeting, Geneva, 15–17 December 2009. Geneva: WHO. Wholey, J. S., H. P. Hatry, and K. E. Newcomber. 2010. Handbook of practical program evaluation. 3rd ed. San Francisco, CA: Jossey-Bass. Yarbrough, D. B., L. M. Shulha, R. K. Hopson, and F. A. Caruthers. 2011. The program evaluation standards: A guide for evaluators and evaluation users. 3rd ed. Thousand Oaks, CA: Sage. Appendix C 319

OCR for page 309