9

Systems and Evaluation: Placing a Systems Approach in Context

Why: To date, the majority of obesity prevention evaluation efforts have focused on methods that do not optimally recognize the interactions and relationships among the many factors that comprise a complex health issue such as obesity. As a result, important learnings may be missed. A systems approach to obesity prevention evaluation efforts can build on current evaluation methods and improve our understanding on the relationships and their impact among the Accelerating Progress in Obesity Prevention (IOM, 2012a) report strategies, outcomes, and stakeholders across a variety of settings.

What: Complex systems are a configuration of interacting, interdependent parts, connected through a web of relationships, that form a whole greater than the sum of its parts (Holland, 2000). The explicit recognition by evaluators that obesity is complex will help to facilitate the increased use of systems approaches in evaluation efforts for obesity prevention.

How: From a systems perspective, evaluating progress of obesity prevention efforts should include support for better understanding of the interactions and relationships among individuals, groups, communities, stakeholders, and national efforts across a variety of settings and throughout the evaluation process framework (see Chapter 3).

Obesity is complex. Attempts to treat or prevent it with individual behavior-change strategies (as a simple problem) have generally not been effective. Consideration of a complex systems approach to identify the best choices for obesity prevention is a worthwhile, arguably essential, endeavor. The importance of considering a systems approach to obesity is explicitly identified in the Institute of Medicine (IOM) report Accelerating Progress in Obesity Prevention (APOP) (IOM 2012a). The report notes that “a systems perspective helps to reveal, and create, the potential for combined impacts (or synergies) that can further accelerate progress in preventing obesity” (IOM, 2012a, p. 7). From an evaluation perspec-



The National Academies | 500 Fifth St. N.W. | Washington, D.C. 20001
Copyright © National Academy of Sciences. All rights reserved.
Terms of Use and Privacy Statement



Below are the first 10 and last 10 pages of uncorrected machine-read text (when available) of this chapter, followed by the top 30 algorithmically extracted key phrases from the chapter as a whole.
Intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text on the opening pages of each chapter. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.

Do not use for reproduction, copying, pasting, or reading; exclusively for search engines.

OCR for page 255
9 Systems and Evaluation: Placing a Systems Approach in Context Why: To date, the majority of obesity prevention evaluation efforts have focused on methods that do not optimally recognize the interactions and relationships among the many factors that comprise a complex health issue such as obesity. As a result, important learnings may be missed. A systems approach to obesity prevention evaluation efforts can build on current evaluation methods and improve our understanding on the relationships and their impact among the Accelerating Progress in Obesity Prevention (IOM, 2012a) report strategies, outcomes, and stakeholders across a variety of settings. What: Complex systems are a configuration of interacting, interdependent parts, connected through a web of relationships, that form a whole greater than the sum of its parts (Holland, 2000). The explicit recogni- tion by evaluators that obesity is complex will help to facilitate the increased use of systems approaches in evaluation efforts for obesity prevention. How: From a systems perspective, evaluating progress of obesity prevention efforts should include sup- port for better understanding of the interactions and relationships among individuals, groups, communities, stakeholders, and national efforts across a variety of settings and throughout the evaluation process frame- work (see Chapter 3). O besity is complex. Attempts to treat or prevent it with individual behavior-change strategies (as a simple problem) have generally not been effective. Consideration of a complex systems approach to identify the best choices for obesity prevention is a worthwhile, arguably essential, endeavor. The impor- tance of considering a systems approach to obesity is explicitly identified in the Institute of Medicine (IOM) report Accelerating Progress in Obesity Prevention (APOP) (IOM 2012a). The report notes that “a systems perspective helps to reveal, and create, the potential for combined impacts (or synergies) that can further accelerate progress in preventing obesity” (IOM, 2012a, p. 7). From an evaluation perspec- 255

OCR for page 255
tive, systems science provides quantitative methodologies that enable evaluators to consider the dynamic relationships of factors at multiple levels of analysis, but it also includes qualitative approaches to actively engage members of the community in a participatory process. Evaluation deals with challenges of the real world—dealing with reality as it unfolds demands a complex adaptive systems perspective. The complexity of obesity introduces notable challenges for evaluators. Simple or uncomplicated systems are likely to be homogenous, linear, static, and independent, with little feedback or connection to other levels or subsystems. On the other hand, complex systems are a configuration of interacting, inter- dependent parts, connected through a web of relationships, that form a whole greater than the sum of its parts (Holland, 2000). Complex systems are mostly nonlinear, dynamic, interdependent, generate feed- back, and have several connected levels. In a complex system, a given set of circumstances will not always yield the same result (Holmes et al., 2012). A systems approach to obesity may help to make sense of the complexity (discussions related to systems and complexity tend to use a unique lexicon, and many of the terms used in this discussion are defined in Appendix B). By identifying leverage points and simple rules among a network of interconnect- ing nodes, evaluators can advance their understanding of how different components work together under various circumstances to influence movement in weight status. This enables evaluators to better identify where to look, what to measure, when to measure, how to recommend action, and why the proposed approach is important for decision makers to consider. A systems approach elevates the importance of understanding contextual factors and relationships to describe what is happening, why it is happening, and what may be the best action (how) to achieve progress in the intended direction. To provide context in which a systems approach to evaluation may occur, the Committee con- sidered some situations in which a systems approach could be used to take action. It should be kept in mind that systems bring together a set of elements into a meaningful set of relationships—connected and i ­nterdependent—that act together as a whole. In effect, systems work by uniting various elements in a meaningful relationship that acts as a whole. To evaluate these complex systems is to evaluate patterns of relationships, how they are sustained, how they self-organize, and how they emerge. Systems, therefore, can only be understood as an integrated whole; the result of both the sum and the relationships among its elements. In the context of taking action to evaluate obesity prevention, action-oriented questions to ask include what happened, what will happen, and what is the best choice? All of these questions may apply to dealing with obesity as a system. However, as discussed below in more detail, the first two questions may be addressed by treating obesity as a predictable issue that may be solved through reductionist think- ing and, as such, represent simple or at most complicated systems. On the other hand, the third ques- tion addresses obesity as a complex system that is not so predictable and, as such, cannot be understood through reductionist approaches (Finegood, 2011). Evaluating the “What Happened?” Question To describe what happened, a basic reporting approach may be taken. This generally takes the form of surveillance and other monitoring techniques that describe the current state, which is attributed to activities conducted in the past. The “what happened” question context is representative of a static, i ­ndependent, linear situation that potentially describes some combination of what has happened, when it 256 Evaluating Obesity Prevention Efforts

OCR for page 255
might have happened, where it may have happened, and how much, how high, what level, or what score may have been attained. Evaluating the “What Will Happen?” Question To describe what will happen as a result of actions taken, evaluation approaches will generally use techniques that predict or estimate the likelihood of certain desired outcome(s) based on the observed association with another set of factors or the measured changes in proximal intervention targets that allow for the prediction of desired ultimate outcomes. To address this question, evaluators use several approaches, including randomized controlled trials and variations on experimental designs that attempt to control for many confounders (Mercer et al., 2007). These evaluation techniques also tend to address the problem in a reductionist manner, attempt to identify cause and effect, and often fall short in solving the ­ omplexity of the interactions among the many factors associated with obesity. On the other hand, c this type of approach may be very useful for addressing, to some level of confidence, the potential impact of actions proposed or taken, how to prevent adverse outcomes, or how to provide an estimate of antici- pated value of interventions. Evaluating the “What is the Best Choice?” Question To address the question what is the best choice of action, considering all available options, com- plexity science is needed. Complexity science is an attempt to understand how things influence each other within the context of the whole. The Committee considers obesity to be a complex system insofar as it possesses characteristics consistent with complexity that include being heterogeneous, nonlinear, s ­ tochastic, interdependent, generating feedback, self-organizing, and involving emergence (Finegood, 2011). A systems approach to taking action demands respecting the characteristics of systems noted above and considering leverage points that may influence change within a complex system—attempting to identify the “best choice” for action. The remainder of this chapter will focus on addressing the third ­question: “What is the best choice?” How Will Evaluation Efforts Benefit from a Systems Approach? Although relatively new, systems theory is already contributing to evaluations of obesity interven- tions (C3 Collaborating for Health, 2011). Evalations that do not consider systems dynamics or con­ itions d will likely miss aspects of the intervention and its environment that influence the intervention’s operation and success (Hargreaves, 2010). To optimize the development of practice-based evidence from such studies, ­ researchers and practitioners will need to develop evaluation systems that enable continuous learning and the pooling of findings and best practices (Holmes et al., 2012). To move toward systems-informed evalu- ation, researchers will need to reconsider the types of evidence needed from a systems point of view. For example, questions around effective leadership to facilitate systems change and leveraging of networks to advance health improvements become increasingly important with increased complexity (Holmes et al., 2012). Systems science (as it applies to community-based prevention) can address how causal structures change over time, including the effect of changes in the type or number of interventions implemented, or how changes in social norms, community practices, and leadership may affect outcomes. Examining these Systems and Evaluation: Placing a Systems Approach in Context 257

OCR for page 255
causal structures can increase understanding about both intended and unintended consequences of inter- ventions (IOM, 2012b; Meadows, 1999; Sterman, 2000; Ulrich, 2000). As one example of how a systems approach may be used to support evaluation, consider multiple uses of mapping the relationships among the many variables and factors of a complex system, including identification of leverage points, anticipation of unintended consequences and unexpected results, and comprehensive assessment of stakeholder views. Mapping and the Use of Mapping Systems maps (e.g., causal loop diagrams) help to visualize relationships. The process of developing a causal loop diagram is well suited for group work and facilitating communication among diverse stake­ holders (Homer et al., 2006; Milstein and Homer, 2009). A systems map represents a dynamic hypothesis, an evidence-based or evidence-informed depiction of relationships, but the map itself cannot be tested (Homer et al., 2006; Milstein and Homer, 2009). Systems causal maps may be created with specific appli- cation to obesity and can be the result of a quantitative or qualitative approach to complex systems map- ping. Excellent examples of complex systems maps for obesity are presented in Tackling Obesities: Future Choices—Building the Obesity Systems Map, a report from the United Kingdom Government’s Foresight Program (Vandenbroeck et al., 2007),1 the APOP report (IOM, 2012a), or An Integrated Framework for Assessing the Value of Community-Based Prevention report (IOM, 2012b). An example of the latter is provided in Figure 9-1. Identification of Leverage Points A map depicting the underlying theories of change and causal structures of a comprehensive obe- sity prevention policy system can increase the understanding of the multiple pathways of impact of ­ besity o preven­ion policies and actions. Mapping these relationships helps to identify key leverage points in the t system that may influence multiple other variables in the system. In turn these diagrams can help evalu- ators to identify and prioritize key variables and associated measures to be assessed. It guides evaluators on “where to look” for short-, intermediate-, and long-term effects of program and policy options, which may enhance the use and collection of relevant data (Diez Roux, 2011). Anticipation and Identification of Unintended Consequences and Unexpected Results The ideal, of course, would be to make only the best choices, but complex systems have emergent properties that make them unpredictable. Therefore, it is worthwhile to consider how to use systems approaches to identify and anticipate consequences of policies and actions taken that are not optimal or ideal but might be ameliorated. The use of obesity system causal maps can identify pathways that high- light actions and responses within the system that are more likely than others. For example, a map of the obesity prevention policy system can more explicitly elucidate the pathways from obesity prevention poli- cies and programs to health outcomes as well as the underlying structures that reinforce or hinder change processes (IOM, 2012b, p. 146). This type of mapping can support the entire proposed framework (see Figure 3-1) for evaluating progress in obesity prevention. The feedback identified through mapping, posi- tive and negative, reinforcing and weakening, anticipated and unexpected, provides opportunities for evaluators to explore the mechanisms that fuel or retard relationships and feedback in the system. This 1  See http://www.bis.gov.uk/assets/foresight/docs/obesity/12.pdf (accessed November 12, 2013). 258 Evaluating Obesity Prevention Efforts

OCR for page 255
FIGURE 9-1  Example causal loop diagram for value of community-based prevention policies. SOURCE: IOM, 2012b. Figure App 1--Bitmapped Broadside. Not much smaller means paying attention to the relationships among actors in the system, among program components, and for portrait among levels of the system. One example of unintended consequences in efforts to prevent obesity and increase fruit and vegetable intake among school children is illustrated in Box 9-1. Figure 9-1 from old report.eps Comprehensive View of Elements and Stakeholders Involved or Affected by Change Efforts To generate a useful systems model or map, multiple stakeholders and perspectives invested in the situation need to be engaged in the process (IOM, 2012a; Williams and Imam, 2006). Also known as par- ticipatory evaluation, systems-informed evaluators have a responsibility to “increase the voices of those in the margin” (Williams and Imam, 2006, p. 9) in all phases of development and evaluation of the inter- vention and establish common goals reflecting the various perspectives of the stakeholders. Their partici­ pation in building systems maps tends to unearth causal loops that experts and others from outside the community would not anticipate. Systems and Evaluation: Placing a Systems Approach in Context 259

OCR for page 255
BOX 9-1 yumPower School Challenge In 2011, a not-for-profit health plan in Minnesota announced a multi-year community collaboration called “yumPower” (www.yumpower.com) to increase “better-for-you” food consumption across a variety of com- munity settings, including elementary schools. During the 2012 school year, 32 elementary schools (about 15,000 students) participated in the yumPower School Challenge pilot program designed specifically to increase intake of fruits and vegetables among elementary school students. The program kicked off with a 45-minute, high-energy, interactive school assembly led by Radio Disney at each school site. Students were given fruit and veggie trackers to record their consumption each day for 4 weeks. Schools received tracking supplies and pencils and notepads to provide students with reminders and incentives for tracking. Some key findings: 16 extra tons of fruits and vegetables were consumed by the students over this 4-week time frame, students increased their fruits and vegetables consumption by 11 percent overall and by 22 percent during weekend days, participation rates were as high as 76 percent, and finally 93 percent of teachers and school staff said they would recommend the program to other schools. An unintended consequence occurred as well: food service staff faced challenges dealing with the increased demand for fruits and vegetables. This conundrum prompted one food service staff person to say “be sure to warn food service before you start because we ran out of fruits and vegetables and needed to order more!” Programs designed to increase healthy food consumption for one group must consider all stakeholders affected by the changes that the program may bring about. SOURCES: HealthPartners, 2012; Isham et al., 2013. A Systems Perspective on the Challenge of Evaluating Obesity Prevention Complex systems are a configuration of interacting, interdependent parts, connected through a web of relationships, that represent a whole greater than the sum of the parts (Holland, 2000). Research s ­ upports the obesity-related influences that stem from genetics, neurobiology, physiology, psychology, ­ behavior, family structure and influences, social context and social norms, environment, economics, ­ m ­ arkets, and public policy. However, the mechanisms of these relationships are not always well researched (Hammond, 2009). Because obesity is more likely to be the result of a multitude of factors interacting with each other over time than any single causal factor (Glass and McAtee, 2006) drives the field to recognize that obesity is complex and that progress in obesity prevention may gain significant ben- efits when evaluation strategies include a systems perspective. An earlier IOM report, Bridging the Evidence Gap in Obesity Prevention: A Framework to Inform Decision Making (IOM, 2010), discusses the need to apply a systems perspective to obesity prevention. A systems approach to obesity prevention makes sense because multiple properties specific to obesity need to be considered in combination in order to address obesity prevention effectively (although this list of properties may not be all-inclusive). Properties to consider include the following: 260 Evaluating Obesity Prevention Efforts

OCR for page 255
• Obesity is related to social networks. • Intervention outcomes for obesity prevention are affected by unanticipated, emergent situations and baseline assumptions that may not hold over time. • Significant inter-individual variability exists in biological, physiological, and emotional ­ esponses r to interventions. • A significant knowledge gap exists related to weight accumulation. • Resource allocation that addresses obesity prevention is related to other areas competing for the same resources. With all of these properties to consider, using a systems perspective for evaluating obesity preven- tion efforts is challenging. Decisions around what systems approaches to consider may be informed by an understanding of which simple rules govern the behavior of the system and at what level of the sys- tem provides the most opportunity to make a difference. In addition, costs, scalability, and sustainability of interventions are important considerations (Hammond, 2009; Huang et al., 2009; Levy et al., 2011; Mabry et al., 2010; Nader et al., 2012). To discuss this in more detail, the next two sections address the concepts of “simple rules” and “system levels.” Simple Rules Complex systems, given their dynamic and interacting characteristics, are inherently unpredictable, and interacting with them leads to continually emerging novel behavior (Plsek and Greenhalgh, 2001). Nevertheless, it appears that this emergent property follows an inherent self-organization facilitated through simple, locally applied rules. Examples of simple rules identified to explain complex behavior in, for example, the context of health systems include those associated with large system transformation and those that attempt to explain how health systems create value for their stakeholders. Best et al. (2012) identified five simple rules that were likely to enhance the success of transformation initiatives: “1) blend designated leadership with distributed leadership, 2) establish feedback loops, 3) attend to history, ­ 4) engage physicians, and 5) include patients and families” (p. 421). Kottke et al. (2012) identified five simple rules that generate value for the health system’s stakeholders: “1) the stakeholders agree on a set of mutual, measurable goals for the system; 2) the extent to which the goals are being achieved is reported to the public; 3) resources are available to achieve the goals; 4) stakeholder incentives, imperatives, and sanc- tions are aligned with the agreed-upon health goals; and 5) leaders of all stakeholders endorse, promote, and honor the agreed-on health system goals.” Identifying the simple rules that govern the behavior of the system in question may point to a limited number of intervention opportunities that are likely to influence the outcomes and overall impact on the goal of obesity prevention. Simple rules also relate to the concept of self-organization, that is, the capacity of a system to make its own structure more complex. For example, a researcher may consider the organization of a person’s social network. People who have friends who are overweight are more likely to be overweight themselves (Valente et al., 2009). Furthermore, if their social system changes over time, the likelihood they may gain or lose weight is highly correlated with the weight of the friends to whom they are connected (Christakis and Fowler, 2007). As such, the relationships a person has may be considered a simple rule related to body weight, but caution must be exercised in not equating correlation with causation. Systems and Evaluation: Placing a Systems Approach in Context 261

OCR for page 255
System Levels In addition to simple rules, system levels also provide a means to make systems approaches more actionable. For example, when obesity interventions are only applied at the level of the individual and do not take into consideration what happens at the level of behavioral settings (e.g., worksites, schools, health care), sectors influencing (e.g., public health, education, business, government), or social norms and values, it is unlikely that the changes induced will be sustainable (IOM, 2012a). Malhi et al. (2009) simplified an original list described by Meadows (2008) into five system levels at which obesity interven- tions could be applied. These levels include (1) paradigm (a system’s deepest belief), (2) goals (what the ­ system is trying to achieve), (3) structure (enhancing connections across most of the system as a whole), (4) feedback and delays (to effect self-regulation, self-reinforcement, and adaptation of the system), and (5) structural elements (to affect subsystems, actors, and the physical structure of the system). The para- digm level is the most difficult at which to intervene, whereas the structural elements level is the least dif- ficult. However, interventions at the more difficult levels tend to be more effective and impactful. In the context of childhood obesity, Malhi et al. (2009) found that most recommendations for interventions are made at the structural elements level. This is the level that includes a strong evidence base and is associ- ated with evidence of cause and effect. Paradoxically, it is the level least likely to be effective in creating and sustaining change in a complex system. Table 9-1 describes the intervention levels and associates them with objectives for evaluation. A clear example of interventions implemented at multiple levels of a system is the tobacco experi- ence over the past several decades. The interrelated set of activities that eventually connected several levels of interventions represents an evolution toward a systems approach to address tobacco use in the United States. As outlined in Bridging the Evidence Gap in Obesity Prevention: A Framework to Inform Decision Making (IOM, 2010), the combined and coordinated set of activities implemented at the individual, com- TABLE 9-1  Places to Intervene in a Complex System and the Associated Evaluation Objectives Level Definition/Description Evaluation Objective Structural elements Subsystems, actors, and the physical structure Assess level of coherence among the components of the system of the system Feedback and delays Self-regulation, self-reinforcement, and Assess degree to which the system operates in adaptation of the system. Reinforcing loops a continuous manner, and assess the likelihood for driving growth and balancing loops for that the system will or will not grow constraining goals Structure Enhancing connections across most of the Assess degree to which interdependent elements system as a whole in a network work together Goals What the system is trying to achieve Evaluate how well the system adapts to its environment Paradigm A system’s deepest belief Understand the degree to which the system acts as a learning environment SOURCES: Adapted from Finegood, 2011; Malhi et al., 2009; and Meadows, 2008. 262 Evaluating Obesity Prevention Efforts

OCR for page 255
munity, population, system, and public education levels all contributed to overall tobacco use reductions in the country. Figure 9-2 presents this set of activities. Simple rules and system levels can be used to identify opportunities to take action and may be r ­ egarded as a fundamentally different way of thinking about how to approach complex problems, such as obesity. Using a Systems Perspective to Address Population Health: A Review of Prior IOM Reports Previous IOM consensus committees have applied systems thinking to identify the types of evidence and research needed to guide decision making around obesity prevention; to identify promising strategies and actions; to anticipate the effects and unanticipated consequences of actions; and to explore different pathways through which interventions can lead to changes—for better or worse—in population behavior and health outcomes. This report builds from this knowledge base by applying a systems approach to evaluation design and implementation. The Committee built on three reports that present applicable dis- cussion of system approaches to obesity prevention; they are briefly presented in Table 9-2 below. Evaluation Tools and Systems Science A diverse set of qualitative, quantitative, and mixed-methods evaluation and analytic tools are avail- able to help to evaluate actions to address a complex problem such as obesity. The tools can be used to System-level Intervention • Epidemiological models • System models, networks, knowledge • NCI ISIS project Population-level Intervention • Clean air laws • Tobacco taxation • NCI ASSIST project Community-level Intervention • Community-based coalitions • Workplace smoking policies • NCI COMMIT project Individual-level Intervention • Smoking cessation tools • Physician counseling • Telephone quitlines Public education–level Intervention • Surgeon General’s reports • Public service announcements • Media campaigns FIGURE 9-2  Evolution of tobacco control approaches toward system thinking. Figure 9-2.eps NOTE: ASSIST = American Stop Smoking Intervention Study; COMMIT = Community Intervention Trial for Smoking Cessation; ISIS = Initiative on the Study and Implementation of Systems; NCI = National Cancer Institute. SOURCE: Adapted from NCI, 2007. Systems and Evaluation: Placing a Systems Approach in Context 263

OCR for page 255
TABLE 9-2  IOM Reports That Used Systems Perspectives IOM Report Description Emphasis on Systems Bridging the Evidence This report developed the L.E.A.D. • Frame the problem Gap in Obesity (locate, evaluate, assemble, inform • Understand potential causes Prevention: A Framework decisions) framework to help researchers • Identify critical leverage points of influence to Inform Decision locate, evaluate, and analyze evidence • Take effective action Making to inform and advance decision making. • Maintain a systems perspective throughout (IOM, 2010) This framework can help evaluators • Give consideration to external validity to maximize opportunities to generate of the evidence for applicability in other evidence from policy and practice, and it populations or settings can help other end users—such as public health practitioners—collect, analyze, and present information. Accelerating Progress The APOP recommendations addressed To understand the process of system in Obesity Prevention: five environments—physical activity, food, change, the committee proposed monitoring Solving the Weight of the message, health care and work, school— engagement, communication, and leadership Nation (APOP) with the understanding that synergies among all sectors to increase the development, (IOM, 2012a) across actions in these environments implementation, and coordination of common would advance greater movement in messages, processes, and strategies. preventing obesity than action in any one area alone. An Integrated Framework This report developed a framework for This report highlighted the utility of a for Assessing the Value assessing the value of community-based systems science approach to understanding of Community-Based nonclinical prevention policies and the collective impact of community-based Prevention wellness strategies. interventions. It discussed the value of a (IOM, 2012b) systems approach to coalition formation, research design, and analytical descriptions and evaluations. identify changes in patterns of system behavior over time and to advance an understanding of the factors and conditions underpinning these changes. The tools range from sophisticated computer models, such as systems dynamics modeling (Homer and Hirsch, 2006; Sterman, 2006), dynamic microsimulation model- ­ ing (Mitton et al., 2000), and agent-based modeling (Axelrod and Tesfatsion, 2006; Epstein, 2006), as well as qualitative approaches such as focus groups and Concept Mapping (Kane and Trochim, 2007), which require little technological support. Some examples of tools presented in Box 9-2 are discussed in more detail below. Simulated Virtual Worlds: A Quantitative Model Simulated virtual worlds, an example of a quantitative method that is based on formal models, are widely used to advance hypotheses about how a system behaves over time. High-quality empiri- cal research, especially well-designed experimental and quasi-experimental studies, is needed to inform systems models (IOM, 2010), which use data-driven assumptions to guide their predictions (Levy et al., ­ 264 Evaluating Obesity Prevention Efforts

OCR for page 255
BOX 9-2 Examples of Evaluation Tools/Methods for Complex Systems Quantitative Methods Qualitative Methods Agent-based modeling Case studies, interviews, and focus groups Time-trend analysis Observation of activities Observational or cross-sectional studies Document reviews Retrospective analyses Outcome mapping, concept mapping Adaptive learning measurement systems Analyses of emergent system-wide patterns, tracking of   events, encounters, and policy changes Geographic information system spatial analysis Use of simple rules and conditions of self-organization Simulated virtual worlds Soft systems methodology Systems dynamic modeling Appreciative inquiry, reflective practice Dynamic microsimulation modeling Group model building Systems mapping/causal loop model building SOURCE: Adapted from Hargreaves, 2010. 2011). The ReThink Health dynamics simulation model was designed to help planners to test different quantitatively estimated effects of scenarios by tracking changes in health status, utilization, costs, and equity following different intervention options (ReThink Health, 2012). To varying degrees, the proposed interventions address risk, care, capacity, cost, funding, and/or economic and health care trends (e.g., prevalence of uninsured individuals, local economic climate). These simulated scenarios are also useful for evaluators because they shed light on what types of impacts can be expected under different conditions in the short- and long-term. As a result, they point to where in the system evaluators should monitor and measure change, and at what intervals of time. In essence, they can help to identify the best choice among a variety of options. Group Model Building: A Qualitative Model Group model building (GMB), a qualitative method example, presents a participatory process for developing and analyzing a formal model. This iterative approach, conducted over several meetings, fosters collaboration and stakeholder involvement, and it can help to address issues of transparency by articulating underlying assumptions. It may be especially valuable when working with marginalized com- munities (Hovmand et al., 2012). GMB requires teamwork, distinct roles (e.g., facilitator, recorder), and facilitation to help to advance collaboration and manage conflict. Process maps can help teams to visual- ize and plan the overall sequence of GMB sessions. Systems and Evaluation: Placing a Systems Approach in Context 265

OCR for page 255
Systems in the Context of the Committee’s Proposed Evaluation Framework The framework, as presented in Chapter 3, outlines five components, including inputs, activities, ­ o ­ utputs, intended outcomes, and impacts (see Figure 3-1). Each of these components can be viewed through a complexity lens. The various evaluation methods and tools discussed above can be applied to each component’s set of activities shown in Figure 9-3 to support the evaluation and measurement effort. Some tools or methods can be applied to multiple sections or the entire framework at once, whereas ­ thers ­ o may be applied to a single step in the model. For example, a systems map can encompass sources of input, activities in the evaluation efforts, identified outputs and intended outcomes, and anticipated impacts. Examples of systems approaches to obesity prevention evaluation that may be applied to the frame- work are provided in Figure 9-3. The examples are presented as part of each step of the model as well as in the context of the entire model. The “Inputs” box dialogue sessions may be one example of collecting insights that follow a systems approach methodology. The “Impacts” box the need to collect School Health Policies and Practices Study and School Nutrition Dietary Assessment Study–like data on child care settings on a national and periodic basis represents another opportunity for cross-agency collaboration. In addition ­ the ability to identify sociocultural and socioeconomic determinants, timing of exposure, and living and working conditions across populations illustrates another important multi-level relationship that follows ­ a systems approach methodology (further described in Chapter 5 and applied in Chapters 6, 7, and 8). Looking at the framework as a whole, and recognizing that complex systems include, for example, charac- teristics such as nonlinearity, interdependency, feedback loops, and emergence (Finegood, 2011), one real- izes that complex social networks serve as an example of a systems approach to evaluation that includes Inputs: Stakeholders’ needs/goals; current context for systems-oriented evaluations; need for systems-oriented evaluation Vision: Networks and Impacts: Multi-sector/multi-level Activities: Need for identifying multi-level feedback loops that partnerships; improved physical, policy, process-oriented indicators; develop the include individuals, and social systems; reduction of plan to monitor/evaluate; and develop families, peers, geographic (spatial) disparities resources for technical assistance communities, regions, states, and the nation Outputs: Recommended set of core environmental change indicators and measures leveraging networks, identifying Outcomes: Improved evaluation/surveillance capacity and leaders, and enabling continuous learning infrastructure; activities; and data use and pooling of findings and best practices; with a systems lens protocols and guidance for systems evaluations; participatory evaluation to engage stakeholders FIGURE 9-3  Examples of how various systems approaches to evaluation may be applied to obesity prevention efforts in the context of the Committee’s evaluation framework. Figure 9-3.eps 266 Evaluating Obesity Prevention Efforts

OCR for page 255
many of the systems characteristics noted and applies to the entire framework. Complex social networks, and their inherent feedback loops, have been shown to affect obesity. Using social network analysis, Christakis and Fowler (2007) were able to examine the spread of obesity over time among subjects of the Framingham Heart Study. These researchers showed that obesity was clearly associated with social rela- tionships (e.g., friends, spouses, and siblings) and that the geographic distance among people (an example of physical environment) was less important than social environment. They were also able to show that obesity followed the friendships and concluded that obesity is, in fact, contagious. Case Examples Change in body weight is associated with an imbalance between energy content of food stuffs c ­ onsumed and energy expended. Hence, to consider the potential of any obesity intervention, its impact on both energy intake and expenditure needs to be properly accounted for. Hall et al. (2011) provide an example of how predictions about the potential effect of an intervention may differ between more tradi- tional approaches and systems approaches to obesity prevention. These researchers compared the results of a projected impact of taxing caloric sweetened beverages on obesity prevalence based on the extrapola­ tion of a static weight loss prediction model to the results of a dynamic simulation model. The static model, presented in a report by the U.S. Department of Agriculture (USDA) (Smith et al., 2010), esti- mated the impact of a 20 percent tax to be an overall decrease in energy intake of about 40 kcal per day and a 1.8 kg body weight loss per year. Extrapolated over 5 years, this would result in almost 10 kg body weight loss. In comparison, the dynamic simulation model estimated that the overall impact of a decrease in energy intake as a result of the tax increase would reduce average body weight by about 1.8 kg over 5 years and about 1 kg of weight loss after 1 year (Hall et al., 2011). This result is in stark contrast to the estimates of the USDA analysis and highlights the importance of inter-individual variability of weight loss (caused by the same intervention), uncertainty of the assumptions made about energy expenditure, and the trajectory of human weight change. This example in the context of obesity prevention recognizes the importance of a careful consideration of the interactive and dynamic relationships among all factors and variables that influence the outcomes of interest over time. Another case example outlines how a systems approach to obesity prevention evaluation may be used in a larger context to support decision making and dissemination of successful interventions. Figure 9-4 illustrates how a complex obesity systems map has been used to inform action (e.g., planning, decision making, and implementation) around obesity prevention. The case example starts with the explic- it recognition that the workplace may be regarded as a complex adaptive system and that addressing obe- sity, considered complex in its own right, should be considered from a systems perspective. The Foresight obesity system causal map, created by the UK government (Butland et al., 2007), is used as the starting point to identify the best choice(s) for addressing obesity at the workplace. The workplace is mentioned in the context of physical activity (energy expenditure) in two of the seven main thematic clusters—the indi- vidual physical activity and physical activity environment clusters. More specifically, these clusters indicate the issues of overall level of occupational activity and dominance of sedentary employment. Hence, two places that can provide leverage to address obesity prevention successfully include the potential to increase person-level activity during the workday and to change the work environment by reducing the sedentary nature of the work tasks in which employees are engaged. Systems and Evaluation: Placing a Systems Approach in Context 267

OCR for page 255
Step 1: Recognition of both obesity and the workplace as complex adaptive systems, intervention goals set on obesity (energy balance) and associated behaviors and consider person-, policy-, and physical and social environment Following identification of best choices for level solutions. Focus of interest is physical activity (Pronk, 2009). intervention, a decision was made to invest in an intervention designed to change the sedentary mature of the work and to reduce sitting time throughout the day for employees with sedentary job types. Whereas Step 2: this is unlikely to result in weight loss, per se, it may be Consider the obesity causal map helpful in increasing energy expenditures and (Vandenbroeck et al., 2007). vasomotion, addressing back and neck pain due to Decision to invest in an intervention static tension, and overall mobility that may aid in prevention of weight gain. Step 5: Design intervention concepts that change the work environment and address prolonged sedentary time. Step 3: Simplify the map to its main Step 6: thematic clusters and Conduct pilot feasibility study and disseminate results: identify the places where The “Take-A-Stand” project (Pronk et al., 2012). physical activity related variables are located Review outcomes and impacts and consider broad - (Vandenbroeck et al., 2007). based dissemination: • Reduced prolonged sitting time • Reduced upper back and neck pain Step 4: • Improved mood states Identify thebest choices • Increased self-reported productivity for action. Dominance of • Worker perceptions of enhanced energy levels, feeling sedentary employment more focused, happier, and less stressed (physical activity Step 7: environment cluster) is Disseminate findings to larger organization: Application associated with level of of solution to entire company (Koster, 2012). occupational activity (individual physical activity cluster) which, in turn, links to overall physical activity. FIGURE 9-4  Case example of an obesity system casualFigure 9-4.eps a workplace setting to identify intervention opportuni- map applied to broadside ties and evaluate energy balance (physical activity) interventions SOURCE: Vandenbroeck et al., 2007. Foresight: Tackling Obesities: Future Choices—Project Report, 2nd Edition. Final Project Report. The Government Office for Science: London. As a result of this evaluation of the Foresight obesity maps, program planners decided to invest resources in programmatic solutions. They designed and tested intervention concepts in a pilot project (Pronk et al., 2012). In the context of a multi-component, comprehensive health and well-being program, sit-stand devices were deployed in the work stations of employees with sedentary job types (call-center employees). Employees were engaged in a participatory manner, a supportive environment was established to optimize participation, and minimally invasive measurement methods were deployed to monitor per- formance. The results of the pilot study proved meaningful from the perspective of both the employees and management, and, as a result, the solution was deployed across all areas of the company that housed sedentary jobs (Koster, 2012). Summary This chapter presents a systems approach to evaluate obesity prevention efforts in context and con- siders it from an end user’s perspective by asking the question, “what is the best choice?” when taking 268 Evaluating Obesity Prevention Efforts

OCR for page 255
action. It describes what a systems approach entails, what are its key components, how opportunities for action may be identified, where readers may find other discussions of systems applied to obesity, what evaluation tools and methods are available, and what benefits a systems approach may provide to evalu- ation efforts in this area. Finally, the chapter considers how systems may be applied to the Committee’s proposed evaluation framework. A systems approach may be useful to describe interrelationships among the many variables and ele- ments that are involved in obesity and prevention efforts. It can help policy makers and implementers to identify the most important leverage points to achieve progress to determine “what is the best choice?” regarding what action to take and how best to invest resources. Taking a systems approach to evaluating the APOP strategies will enable researchers and decision ­ makers to think about evaluation efforts in new ways—identifying the types of data to collect and the methods for analyzing the data, identifying core indicators and measures that leverage networks and feed- back loops among different sectors of society, identifying leaders, and engaging stakeholders in all aspects of evaluation. The result of these activities will improve the capacity and infrastructure of the collective evaluation system for obesity. And because obesity itself is an integral part of the larger systems of public and population health, it may be viewed as a major element in the larger complex system of the health of populations in communities, across states, and across the nation. References Axelrod, R., and L. Tesfatsion. 2006. Appendix A. A guide for newcomers to agent-based modeling in the social sci- ences. In Handbook of computational economics, vol. 2: Agent-based computational economics, edited by L. Tesfatsion and K. L. Judd. Amsterdam, the Netherlands: North-Holland. Pp. 1647-1659. Best, A., T. Greenhalgh, S. Lewis, J. E. Saul, S. Carroll, and J. Bitz. 2012. Large-system transformation in health care: A realist review. Milbank Quarterly 90(3):421-456. Butland, B., S. Jebb, P. Kopelman, P. McPherson, S. Thomas, J. Mardell, and V. Parry. 2007. Foresight: Tackling ­ besities: Future choices–project report, 2nd edition. London, UK: Government Office for Science. o C3 Collaborating for Health. 2011. Beyond Foresight: Can systems thinking help us address complex problems like obesity and chronic disease? http://www.c3health.org/wp-content/uploads/2011/03/Diane-Finegood-seminar- FINAL-20110322.pdf (accessed January 30, 2013). Christakis, N. A., and J. H. Fowler. 2007. The spread of obesity in a large social network over 32 years. New England Journal of Medicine 357(4):370-379. Diez Roux, A. V. 2011. Complex systems thinking and current impasses in health disparities research. American Journal of Public Health 101(9):1627. Epstein, J. M. 2006. Remarks on the foundations of agent-based generative social science. In Handbook of com- putational economics, vol. 2: Agent-based computational economics, edited by L. Tesfatsion and K. L. Judd. Amsterdam, the Netherlands: North-Holland. Pp. 1585-1604. Finegood, D. T. 2011. The complex systems science of obesity. In The Oxford handbook of the social science of ­obesity, edited by J. Cawley. New York: Oxford University Press. Pp. 208-236. Glass, T. A., and M. J. McAtee. 2006. Behavioral science at the crossroads in public health: Extending horizons, envisioning the future. Social Science and Medicine 62(7):1650-1671. Hall, K. D., G. Sacks, D. Chandramohan, C. C. Chow, Y. C. Wang, S. L. Gortmaker, and B. A. Swinburn. 2011. Quantification of the effect of energy imbalance on bodyweight. Lancet 378(9793):826-837. Systems and Evaluation: Placing a Systems Approach in Context 269

OCR for page 255
Hammond, R. A. 2009. Complex systems modeling for obesity research. Preventing Chronic Disease 6(3):A97. http://www.cdc.gov/pcd/issues/2009/jul/09_0017.htm (accessed June 6, 2013). Hargreaves, M. 2010. Evaluating system change: A planning guide. Princeton, NJ: Mathematica Policy Research, Inc. HealthPartners. 2012. YumPower school challenge. Summary of key findings. Bloomington, MN: HealthPartners. Holland, J. H. 2000. Emergence: From chaos to order. Oxford, UK: Oxford University Press. Holmes, B. J., D. T. Finegood, B. L. Riley, and A. Best. 2012. Systems thinking in dissemination and implementation research. In Dissemination and implementation research in health: Translating science to practice, edited by R. C. Brownson, G. A. Colditz, and E. K. Proctor. New York: Oxford University Press. Pp. 175-191. Homer, J. B., and G. B. Hirsch. 2006. System dynamics modeling for public health: Background and opportunities. American Journal of Public Health 96(3):452-458. Homer, J., B. Milstein, W. Dietz, D. Buchner, and E. Majestic. 2006. Obesity population dynamics: Exploring historical growth and plausible futures in the U.S. Paper presented at 24th International System Dynamics Conference, Nijmegen, Netherlands. Hovmand, P. S., D. F. Andersen, E. Rouwette, G. P. Richardson, K. Rux, and A. Calhoun. 2012. Group model-­ building “scripts” as a collaborative planning tool. Systems Research and Behavioral Science 29(2):179-193. Huang, T. T., A. Drewnowski, S. K. Kumanyika, and T. A. Glass. 2009. A systems-oriented multilevel framework for addressing obesity in the 21st century. Preventing Chronic Disease 6(3):A82. http://www.cdc.gov/pcd/ issues/2009/jul/09_0013.htm (accessed June 7, 2013) IOM (Institute of Medicine). 2010. Bridging the evidence gap in obesity prevention: A framework to inform decision making. Washington, DC: The National Academies Press. IOM. 2012a. Accelerating progress in obesity prevention: Solving the weight of the nation. Washington, DC: The National Academies Press. IOM. 2012b. An integrated framework for assessing the value of community-based prevention. Washington, DC: The National Academies Press. Isham, G. J., D. J. Zimmerman, D. A. Kindig, and G. W Hornseth. 2013. HealthPartners adopts community business model to deepen focus on nonclinical factors of health outcomes. Health Affairs 32(8):1446-1452. Kane, M., and W. M. K. Trochim. 2007. Concept mapping for planning and evaluation. Thousand Oaks, CA: Sage Publication. Koster, K. 2012. Sit-stand workstations incorporate movement into work days of sedentary employees. Employee Benefit News, December 1. Kottke, T. E., N. P. Pronk, and G. J. Isham. 2012. The simple health system rules that create value. Preventing Chronic Disease 9:110179. http://dx.doi.org/10.5888/pcd9.110179 (accessed June 7, 2013). Levy, D., P. Mabry, Y. Wang, S. Gortmaker, T. T. K. Huang, T. Marsh, M. Moodie, and B. Swinburn. 2011. Simulation models of obesity: A review of the literature and implications for research and policy. Obesity Reviews 12(5):378-394. Mabry, P. L., S. E. Marcus, P. I. Clark, S. J. Leischow, and D. Mendez. 2010. Systems science: A revolution in public health policy research. American Journal of Public Health 100(7):1161-1163. Malhi, L., O. Karanfil, T. Merth, M. Acheson, A. Palmer, and D. T. Finegood. 2009. Places to intervene to make complex food systems more healthy, green, fair, and affordable. Journal of Hunger and Environmental Nutrition 4(3-4):466-476. Meadows, D. 1999. Leverage points: Places to intervene in a system. Hartland, VT: The Sustainability Institute. Meadows, D. H. 2008. Thinking in systems: A primer, edited by D. Wright. White River Junction, VT: Chelsea Green Publishing Company. Mercer, S. L., B. J. DeVinney, L. J. Fine, L. W. Green, and D. Dougherty. 2007. Study designs for effectiveness and translation research: Identifying trade-offs. American Journal of Preventive Medicine 33(2):139-154. 270 Evaluating Obesity Prevention Efforts

OCR for page 255
Milstein, B., and J. Homer. 2009. System dynamics simulation in support of obesity prevention decision-making. Paper read at Institute of Medicine Workshop on the Application of Systems Thinking to the Development and Use of Evidence in Obesity Prevention Decision-Making, March 16, 2009, Irvine, CA. http://www.iom. edu/~/media/Files/Activity%20Files/PublicHealth/ObesFramework/IOMIrvine16Mar09v52MilsteinHomer.ashx (accessed September 23, 2013). Mitton, L., H. Sutherland, and M. Weeks, eds. 2000. Microsimulation modelling for policy analysis: Challenges and innovations. Cambridge, UK: University Press. Nader, P. R., T. T. Huang, S. Gahagan, S. Kumanyika, R. A. Hammond, and K. K. Christoffel. 2012. Next steps in obesity prevention: Altering early life systems to support healthy parents, infants, and toddlers. Childhood Obesity 8(3):195-204. NCI (National Cancer Institute). 2007. Greater than the sum: Systems thinking in tobacco control. Tobaco con- trol monograph no. 18. NCI Tobacco Control Monograph Series. NIH Pub. No. 06-6085. Bethesda, MD: Department of Health and Human Services, National Institutes of Health, National Cancer Institute. Plsek, P. E., and T. Greenhalgh. 2001. Complexity science: The challenge of complexity in health care. British Medical Journal 323(7313):625-628. Pronk, N. P. 2009. Physical activity promotion in business and industry: Evidence, context, and recommendations for a national plan. Journal of Physical Activity and Health 6(Suppl 2):S220-S235. Pronk, N. P., A. S. Katz, M. Lowry, and J. R. Payfer. 2012. Reducing occupational sitting time and improving worker health: The Take-a-Stand Project, 2011. Preventing Chronic Disease 9:E154. http://dx.doi.org/10.5888. pcd9.110323 (accessed June 7, 2013). ReThink Health. 2012. Summary of the ReThink Health dynamics simulation model. http://rippelfoundation.org/ docs/RTH-Dynamics-Model-Summary-v6.pdf (accessed November 16, 2012). Smith, T. A., B.-H. Lin, and J.-Y. Lee. 2010. Taxing caloric sweetened beverages: Potential effects on beverage consumption, calorie intake, and obesity. Economic Research Report No. 100. Washington, DC: Economic Research Service, U.S. Department of Agriculture. Sterman, J. D. 2000. Business dynamics. Systems thinking and modeling for a complex world. Boston, MA: McGraw-Hill. Sterman, J. D. 2006. Learning from evidence in a complex world. American Journal of Public Health 96(3):505-514. Ulrich, W. 2000. Reflective practice in the civil society: The contribution of critically systemic thinking. Reflective Practice 1(2):247-268. Valente, T. W., K. Fujimoto, C. P. Chou, and D. Spruijt-Metz. 2009. Adolescent affiliations and adiposity: A social network analysis of friendships and obesity. Journal of Adolescent Health 45(2):202-204. Vandenbroeck, P., J. Goossens, and M. Clemens. 2007. Foresight tackling obesities: Future choices—building the obesity system map. London, UK: Government Office for Science. Williams, B, and I. Imam, eds. 2006. Systems concepts in evaluation: An expert anthology. Los Angeles: EdgePress, American Evaluation Association. Systems and Evaluation: Placing a Systems Approach in Context 271

OCR for page 255