5

A Framework for Assessing the Value of Community-Based Prevention

This chapter proposes and describes a framework for assessing the value of community-based prevention. It addresses the need to take a comprehensive view of benefits and resources used, which is central to recognizing the far-reaching effects of community-based prevention; it proposes the development of summary measures to support a comprehensive perspective; it explains the need to base valuation on changes in benefits and resources used; and it describes the prospective use of the framework for decision making, and its retrospective use to evaluate programs and policies once they have been implemented. The chapter then reviews the data needed to quantify value within this framework along with the limitations of the data, discusses how communities and other stakeholders can use the framework to value community-based prevention, and concludes with a discussion of the implications for state and national policy.

A FRAMEWORK FOR ASSESSING VALUE

Existing frameworks for community-based prevention interventions neglect one or more of the elements previously identified as being key to the success of such a framework. In this chapter the committee assembles various elements of existing frameworks into a new framework that measures in a way that suits the unique aspects of community-based prevention.



The National Academies | 500 Fifth St. N.W. | Washington, D.C. 20001
Copyright © National Academy of Sciences. All rights reserved.
Terms of Use and Privacy Statement



Below are the first 10 and last 10 pages of uncorrected machine-read text (when available) of this chapter, followed by the top 30 algorithmically extracted key phrases from the chapter as a whole.
Intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text on the opening pages of each chapter. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.

Do not use for reproduction, copying, pasting, or reading; exclusively for search engines.

OCR for page 119
5 A Framework for Assessing the Value of Community-Based Prevention This chapter proposes and describes a framework for assessing the value of community-based prevention. It addresses the need to take a comprehensive view of benefits and resources used, which is central to recognizing the far-reaching effects of community-based prevention; it proposes the development of summary measures to support a comprehensive perspective; it explains the need to base valuation on changes in benefits and resources used; and it describes the prospective use of the framework for decision mak- ing, and its retrospective use to evaluate programs and policies once they have been implemented. The chapter then reviews the data needed to quantify value within this framework along with the limitations of the data, discusses how communities and other stakeholders can use the framework to value community-based prevention, and concludes with a discussion of the implications for state and national policy. A FRAMEWORK FOR ASSESSING VALUE Existing frameworks for community-based prevention interventions neglect one or more of the elements previously identified as being key to the success of such a framework. In this chapter the committee assembles various elements of existing frameworks into a new framework that mea- sures in a way that suits the unique aspects of community-based prevention. 119

OCR for page 119
120 ASSESSING THE VALUE OF COMMUNITY-BASED PREVENTION The goals of this framework are (1) to incorporate the full scope of ben- efits into the value of interventions so that in addition to health benefits and harms, the benefits and harms from community well-being and community process are included; (2) to emphasize that value requires a comparison of the benefits and harms of an intervention with the resources used for that intervention; (3) allow the specific characteristics and context of individual communities to be reflected in the valuation of community-based preven- tion; (4) to promote the quantification of value in terms of projected or actual changes due to the intervention; and (5) to encourage the develop- ment of evidence so as to make understanding the effects of interventions easier and more reliable. The committee’s proposed framework for assessing the value of com- munity-based prevention is shown in schematic form in Figure 5-1. For the assessment of value, the framework proposes a comprehensive consider- ation of benefits and harms in the context of health, community well-being, and community process as well as an inclusive and comprehensive consid- eration of the resources used. Valuing—What Should Be Counted? Benefits Quality-Adjusted Life Years Health (QALYs) or Health-Adjusted B enefits and Harms Harms Life Expectancy (HALE) Benefits Community Community Well-Being Community Benefit Well-Being Indicator Harms Benefits Community Community Process Process Indicator Value Harms The Value of an Intervention Considers Its Benefits, Harms, and Costs R es ourc es Us ed Valuing—What Should Be Counted? Identification of Savings Community Costs Investments Monetary Units (USD) and Costs Resources FIGURE 5-1  Conceptual framework for valuing community-based prevention interventions.

OCR for page 119
A FRAMEWORK FOR ASSESSING THE VALUE 121 The Framework Should Take a Comprehensive View To value an intervention one should take into account its outcomes (i.e., benefits and harms), the resources used in the intervention itself, and the downstream costs and savings. The valuation of community-based prevention interventions should take a comprehensive perspective—that is, that the measurement of benefits, harms, and resources should include impacts on all members of the community as well as on stakeholders who may be outside the community. There are many stakeholders involved in community-based prevention, each of which is likely to be impacted differently: individuals, families, communities, businesses, taxpayers, and governments at the local, state, and federal level. It can be helpful to assess value from one or more of these other perspectives, but those assessments should be presented and considered relative to the comprehensive perspective. For example, if an intervention is federally funded but the benefits occur locally, the value of the intervention from the perspective of the community will be greater than its value from the federal perspective because the community does not pay all the costs of the intervention. Good decisions will be based on informa- tion about benefits and costs to all stakeholders. To support a comprehensive framework and focus attention on the breadth of benefits that such a framework encompasses, the committee proposes that benefits be grouped into the three broad categories of health, community well-being, and community process, as discussed in Chapter 3. The measurement of benefits and harms should occur in all three areas. Re- sources used are a fourth major category of consequences to be considered in valuing community-based prevention. This comprehensive perspective should be the reference point in decision making. Recommendation 1: The committee recommends that those seeking to assign value to community-based prevention interventions take a com- prehensive view that includes the benefits and harms in the three major domains of health, community well-being, and community process as well as the resource use associated with such interventions. Table 5-1 provides a summary of the proposed framework compared to the eight existing frameworks discussed in Chapter 4. Proposed Summary Measures There are a variety of sources of data on health, including surveys (e.g., the National Health Interview Survey and the Behavioral Risk Factor Surveillance System), cohort studies (e.g., the Framingham Heart Study),

OCR for page 119
122 ASSESSING THE VALUE OF COMMUNITY-BASED PREVENTION TABLE 5-1  Nine Frameworks Summarized Includes Comprehensive Compares Set of Valued Benefits with Accounts for Differences Outcomes Costs Among Communities Benefit–cost analysis Yes, can account Yes Low; can account for context for all benefits Cost-effectiveness No, health only Yes Low; can account for context analysis Congressional Budget No, only federal Yes Low; designed for Office scoring spending and Congressional budget process revenue PRECEDE–PROCEED No, although it No High; used in communities framework includes both health and community process RE-AIM framework No, health only No High; used by evaluators Health Impact No, health only No High; used in communities Assessment framework Community Preventive No, health only No Moderate; focus on community Services Task Force guidelines Lomas model No, valued No Moderate; focus on decision- outcomes not making process specified Proposed framework Yes Yes High; involves communities in process registries, health services data, and vital statistics and data collected by state public health agencies. Unfortunately, there are several limitations when at- tempting to use these data for local, community-based measurement (IOM, 2011). Identifying measures and sources of information for community well-being and community process elements is even more challenging than identifying these items for health. Such efforts will require an increased focus on identifying appropriate information gaps and data sources. Recommendation 2: The committee recommends that the CDC a. develop an inventory of existing data sources for health, commu- nity well-being, and community process; b. identify gaps in data sources; and c. develop data sources to fill those gaps.

OCR for page 119
A FRAMEWORK FOR ASSESSING THE VALUE 123 Different metrics are appropriate for measuring the different domains of value. Chapter 3 describes many of the outcomes that can be important in each domain. Health impacts, for example, can be measured by changes in intermediate outcomes such as blood pressure or weight, by changes in the numbers of cases of disease, or by changes in health-related quality of life, the numbers of deaths, or quality-adjusted life years (QALYs). Community well- being encompasses an even wider range of possible outcomes and measures. Community process adds another set. The resources used for community- based interventions—and any savings that may result from them—add still another set. Choosing among community-based prevention policies and pro- grams can be difficult when programs have so many effects and those effects take so many different forms. The larger the menu of interventions, and the larger the number of valued outcomes, the more difficult choices become. Decision makers can end up focusing on one outcome, such as health, and ignoring others simply because it is too difficult to take them all into account. A few overall metrics can help prevent this narrowing of focus. The committee proposes that four metrics be developed to assess the value of community-based prevention: changes in health, changes in com- munity well-being, changes in community process, and changes in resources used. Health outcomes in the population can be valued with QALYs or health-adjusted life expectancy (HALE). These metrics are well developed and widely used. Each incorporates important domains such as physical well-being, mental well-being, role function, and social function. The committee is unaware of generally accepted single metrics for the domains of community well-being and community process, although a set of relevant elements and an algorithm for each could be developed similar to those used for the “EuroQol,” Quality of Well Being, or the Health Utility Index for health-related quality of life. Measures of community well- being, such as the Urban Hardship Index1 and the Community Well-Being (CWB) indices (e.g., the Canadian Arctic CWB Index2), or the county health rankings could serve as starting points, but they have significant limitations in scope. The committee recognizes that developing these single indicators is a complex task that will require expertise from outside of the field of health. The committee also recognizes that the development of the single indicator is a long-term goal, since such indicators do not currently exist for the com- munity well-being and community process domains. The National Preven- tion, Health Promotion, and Public Health Council (Prevention Council) is 1 See http://www.rockinst.org/pdf/cities_and_neighborhoods/2004-08-an_update_on_­urban_ hardship.pdf. 2  See http://www.aadnc-aandc.gc.ca/eng/1100100016579.

OCR for page 119
124 ASSESSING THE VALUE OF COMMUNITY-BASED PREVENTION an interagency group established by the Affordable Care Act and chaired by the Surgeon General. The Prevention Council recognizes that the health of a community is influenced by a number of factors outside of the health care and public health sectors, including education, housing, and transporta- tion. Such a group is well positioned to encourage the research needed in the multiple sectors that need to be involved in developing a community benefit indicator. Recommendation 3: The committee recommends that the National Prevention, Health Promotion, and Public Health Council and other public and private sponsors support research aimed at developing a. a single metric for appraising a community’s well-being, b. a single metric for appraising community processes, and c. a single metric for combining indicators of community well-being and community process with health into a single indicator of com- munity benefit that can be considered in the context of costs and used to determine the value of a community-based prevention intervention. The committee envisions a well-being and process index that is roughly parallel to QALYs and HALE for health. QALYs and HALE can account for differences in how people value health and other outcomes, much as can be done with willingness to pay (WTP). QALYs calculate not just years of life saved but the quality of each life year as determined by an elicitation of preferences. QALY weights can be determined using community, indi- vidual, or patient preferences depending upon the context of the analysis. In addition, QALYs are specific to the desired outcome of the intervention. For example, QALYs used to compare an intervention aimed at lowering blood pressure would be different from an intervention aimed at reducing automobile injuries. The QALY is a common single indicator that can be used in many different contexts (Gold et al., 1996). Likewise, HALE is also a measure of both duration and quality of life. According to a previous IOM committee (2011) HALE weights “have the ability to take into account the effects of particular illnesses; provide insight into regional differences associated with social, environmental, and behavioral risk factors; and allow examination of the health experiences of subpopulations by race/ethnicity.” In other words, HALE can account for differences in preferences and population. Although recognizing the challenges, it is worth pursuing the develop- ment of a single indicator that can aid in valuing community well-being as well as an indicator for community process. These indicators, like QALYs and HALE, would combine objective measures of well-being and process status with subjective measures of preference.

OCR for page 119
A FRAMEWORK FOR ASSESSING THE VALUE 125 Multi-Attribute Utility (MAU) and the Analytic Hierarchy Process (AHP) are examples of methodologies that can be used to develop the community well-being, community process, and community benefit metrics. Health-Related Quality of Life (HRQoL) is a set of metrics that is used to value the multiple dimensions of health, such as mental function, physical function, and role function, and to combine them into a single, preference- based measure, such as the Quality of Well-Being (QWB), Health Utilities Index (HUI), and EuroQoL. They were developed using MAU techniques, much the same as used for other complex decision models such as those for assessing business decisions and defense strategies. The same approach can be used to identify key components of community well-being and to value them. The MAU approach “combines multiple attributes, such as health, equity and empowerment, by eliciting importance (trade-off) weights for attributes” (Peacock et al., 2007). The six steps in the methodology are to (1) identify relevant attributes; (2) describe the levels of the attributes (for example, civic participation could be described as active involvement, me- dium involvement, low involvement); (3) the levels within each attribute are scaled from 1-100; (4) the attributes are assessed in terms of their relative importance; (5) an intervention is evaluated in terms of how well it con- tributes to each relevant attribute; and (6) scores are combined to calculate the combined benefit score (Peacock et al., 2007). The AHP is a method used to arrange options in a hierarchy in order to assist in decision making. The steps in the AHP are similar to those of MAU, and include breaking down the decision into interrelated decision elements (for example, the elements within each domain affected by an intervention); “collecting data by pair-wise comparisons of the decision elements”; and estimating the relative weights of the decision elements (Zahedi, 1986). As applied to community well-being, the steps would include identifying the components that the community values, such as aesthetics, ability to meet basic needs, and resilience; developing a scale for each; and then to value (weight) each of the components. One can then assess interventions based on their ability to affect each of the components and combine them into a total “score” so that interventions can be compared. Unlike the health domain, which has developed the QALY and the HALE, a single metric does not currently exist for the domain of com- munity well-being nor does a single metric exist for the domain of com- munity process. Therefore, other options must be used until such time as those metrics are developed. It is reasonable to consider using a mixture of quantitative and qualitative approaches. Such a mixed model would allow for an action-oriented approach to reach a quantifiable solution. That is, the mixed model may provide information that ultimately will inform how to come to such a solution.

OCR for page 119
126 ASSESSING THE VALUE OF COMMUNITY-BASED PREVENTION Measures of the built and natural environments; and measures of educa- tion, crime, employment, and equity; and various other elements of inter- est could be combined into a measure of overall community well-being by weighting the changes produced by an intervention in the component out- comes by the community’s preferences for each specific outcome. Similarly an aggregate measure for the process-related benefits could be developed that would reflect the value placed on the way that deliberations occur regarding community-based programs and policies, about the manner in which deci- sions are made and reported, and the manner in which community-based interventions are implemented. Box 5-1 provides an example. BOX 5-1 Valuing the Construction of a Greenway Using the Proposed Framework A community concerned about obesity is looking for ways to encourage more physical activity. One proposal is to convert unused public land along a mass tran- sit corridor into a greenway with a series of pedestrian and bike trails along with other recreational facilities, as several other cities throughout the United States have done or proposed to do. The greenway would pass through several differ- ent communities, some affluent and some poor, linking diverse parts of the city. Using the framework, each affected community as well as other stakeholders would have to decide which outcomes or elements it valued within the proposed three domains of health, community well-being, and community process. Im- proved health could be one possible outcome and there is ample evidence that increased physical activity leads to better health in the long term. This may be the outcome of highest priority for potential funders, such as government agencies or private foundations. But taking a comprehensive view, as recommended by the committee could lead to the identification of other valued outcomes. For example, for one community the greenway has the potential to improve the communities aesthetically as well as to provide greater opportunity for social and community engagement. This community may value increased recreation facilities more than any other outcome. City leaders may value the development of under-utilized land and the addition of amenities to the city. In addition, the greenway could provide an alternate transportation path that could decrease the number of trips made by cars, thereby improving air quality and decreasing traffic. Finally, the construction of the greenway provides local communities with the opportunity to participate in the implementation of a project that reflects their preferences and values, thereby promoting community empowerment. Along with the potential benefits, however, there are potential harms. Pe- destrians and cyclists on the path face a risk of injury. Members of the various communities would be inconvenienced and annoyed by the construction of the greenway and they may find that their differing preferences and values lead to conflict at the planning level. Moreover, communities may find that the placement of recreational facilities or some other aspect of building the greenway creates or

OCR for page 119
A FRAMEWORK FOR ASSESSING THE VALUE 127 The resources used for an intervention can be summarized in dollars or other currency. As presented in the framework, costs are inclusive of goods and services purchased in markets—what everyone recognizes as costs because money must be paid out—and also donated goods and services. As a comprehensive summary measure, these costs could be captured and represented as a “Community Costs” indicator and used in arriving at the value of an intervention. Given that the outcomes in the four domains are—or will be once they are developed—measured in different units, it is currently not possible to provide a single widely-used indicator of the value of community-based highlights disparities in the distribution of resources between communities. Some communities may fear that the greenway will bring strangers or outsiders into their community and make it less desirable. All of these potential benefits and harms should be identified and weighed by the communities and decision makers considering the proposal. For the health benefits and harms, measures such as QALYs and HALE offer well-established means of valuation. In the community well-being and community process domains there are no universally accepted measures. Until these measures are developed there is value in community identification of the constituent benefits and harms because this allows the community to consider the full range of consequences of the proposed intervention. For example, as discussed in Chapter 3, there are positive outcomes in perceived general health associated with access to green space. A shift in transportation preferences away from car trips could, over time, lead to better outdoor air quality. From these projected changes from the current baseline, decision makers could derive an idea of the potential net community benefit even if widely accepted summary measures of community well-being and community process are currently unavailable. In addition to identifying the benefits and harms of the intervention within the three domains, communities and decision makers must also identify the costs of the intervention. As discussed in Chapter 3 the costs of an intervention should be considered from a comprehensive perspective in order to encourage a full ac- counting and to discourage double counting. The costs of the proposed greenway would include the short-term capital outlay for construction and landscaping and long-term maintenance and security costs. They would also include the costs of unpaid volunteer time within the community for maintaining and managing the greenway. There are widely accepted methods of capturing these costs and ex- pressing them using a common metric. After determining the community benefit and the community cost, decision makers are in a better position to value the proposed intervention. In addition, they can determine which indicators will be valuable in evaluating the interven- tion. Many of the projected benefits of the greenway will not occur for a number of years, such as lowering obesity rates. Thus, the time horizons used in valuing the greenway must be appropriate for the valued outcomes. Decision makers should take this into account when valuing and evaluating the intervention.

OCR for page 119
128 ASSESSING THE VALUE OF COMMUNITY-BASED PREVENTION prevention. However, once a single indicator of Community Benefit is de- veloped, it should be considered alongside the Community Cost indicator, and value could be expressed as units of Community Benefit per dollar. It should be noted that if the community benefit indicator is determined to be negative, no further valuation need be conducted. Summary measures for each of the three domains of benefit and for resources used are a first step toward a possible future overall summary measure. Valuation Is Based on Changes in Benefits and Costs Value is based on changes relative to some baseline or to an alterna- tive intervention. The value of a community-based prevention intervention reflects its impacts relative to what would have happened in its absence or relative to an alternative community-based prevention intervention. Changes due to adding an intervention are usually referred to as incremen- tal changes, whereas applying an existing intervention more intensively is usually referred to as a marginal change. In principle, the assessment of the value of an intervention should include changes in everything that has a reasonable chance of changing by more than a trivial amount as a result of the intervention or its intensification. It is important to assess the actual changes that are projected to occur as a result of an intervention and to express them in absolute terms, such as QALYs or HALE, rather than in relative terms, such as a percentage change. Stakeholders often assess the value of an intervention in terms of the overall burden of the health problem or the size of the effect of an intervention. However, these metrics are, by themselves, inadequate for measuring value since they do not consider the overall health impact of an intervention. A problem may be large, but if none of the available interven- tions is effective against it, then they have little value despite the size of the problem; similarly, an intervention may have a large effect size (e.g., it may reduce an adverse health impact by 90 percent), but if the number of indi- viduals affected is very small, then the overall health impact will be small as well. The preventable burden (effectiveness times size of the problem) is a better measure of impact than either the effect size or the size of the affected population considered alone. Recommendation 4: The committee recommends that those assessing value should include in their assessments the expected or demonstrated changes, both positive and negative, that result from the intervention.

OCR for page 119
A FRAMEWORK FOR ASSESSING THE VALUE 129 Prospective Assessment for Decision Making The central task of the framework is to support decision making about choices and options between various possible community-based prevention interventions. An assessment of the value of an intervention can be prospec- tive (before the intervention occurs) so as to inform decisions about which interventions to choose, or concurrent (while the intervention is ongoing), or retrospective (after the intervention concludes) to inform decisions about whether and how to continue an intervention. The prospective assessment of the value of an intervention requires three steps: (1) the identification of factors that are valued by the com- munity, (2) projection of the changes in outcomes (impact) expected as a result of the proposed intervention and of the resources to be used, and (3) estimation of the value of the projected changes. Each step is critical to estimating the value of an intervention for the purpose of decision making. Each step is explored below. Understanding what the community cares about in each of the three domains (health, community well-being, and community process) is critical for designing and proposing interventions that address areas of importance to the community. This assessment will not only identify important health (and non-health) factors in the community, but it will also identify those factors for which improvement is preferred by community members. What is important for one community may not be important for another. Recommendation 5: The committee recommends that those involved in decision making ensure that the elements included in valuing c ­ommunity-based prevention interventions reflect the preferences of an inclusive range of stakeholders. The second step is the projection of changes. This includes both the projected costs of the intervention itself and the benefits, including savings, and harms that are projected to occur as a result. To support good decisions it is essential to list all the outcomes of importance and to show how much each intervention contributes to each outcome. Health is usually projected in terms of intermediate outcomes, such as changes in weight or blood pres- sure, cases of disease, and deaths, and it can be summarized, as noted earlier, by QALYs gained, or by HALE. Community well-being might be measured by reductions in crime using statistics already established by police systems, by increases in educational attainment, by additions to green space, or through surveys of how people feel about their community and the changes in it. Community process might be measured, very roughly, by the number of people participating in local planning activities or the hours of work volunteered in a year, or, more carefully through surveys that ask people in

OCR for page 119
130 ASSESSING THE VALUE OF COMMUNITY-BASED PREVENTION what ways and how much, beyond the consequences for health, they value the activities in which they participate or the transparency with which the decisions are made. Indicators for these two domains do not yet exist and the committee recommends their development (Recommendation 3). Program costs—the resources used in implementing the policy or strategy—should also be projected. All resources used should be projected, regardless of the source of funds used to purchase them or whether they are non-monetary, such as donated time. In this framework the commit- tee specifically recognizes that context (which includes the implementation process, the determinants of health, and the community’s characteristics) is critical to the success and thus to the projected impact of the intervention. The focus on intervention context encourages those engaged in the valuing process to recognize that community-based prevention can be a complex system within which prevention policies, programs, determinants, stake- holders, and strategies interact dynamically. The third step is the estimation of the value of the projected changes. Once community members have listed all the outcomes they value and measured the effects of interventions in those terms, they have to choose among interventions. That can be hard to do when the valued outcomes take different forms. Policy A may provide safer streets and community participation among parents and children. Strategy B may provide meals and social interaction for isolated elderly people. Interventions C, D, and E may offer still other things of value. Which are most valuable? Which should be done if not all can be? Which should be done first? Valuation typically involves assigning weights to the projected changes or ranking the changes, in order to summarize the value of an intervention. If there are only a few choices and only a few valued outcomes, listing the contributions of each intervention to each outcome, as described in step 2, can be sufficient to allow people to choose among them. But the larger the menu of choices, and the larger the number of valued outcomes, the more difficult the choice becomes. In that case people can end up focusing on one outcome, such as health, and ignoring the others simply because it becomes too difficult to take them all into account. As noted, a single indi- cator can help prevent this narrowing of focus. The three domains of value recommended by the committee are a step in the direction of the necessary summary measures. Cost–benefit analysis offers one approach to summarizing benefits (and costs). In the cost–benefit approach all the individual components are converted to dollar values. However, because many people are reluctant to convert health outcomes into dollars and because of the challenge in valu- ing intangible benefits and harms such as social and environmental changes and changes in process, it may be desirable to develop several intermediate indicators, as recommended by the committee. These indicators will make it

OCR for page 119
A FRAMEWORK FOR ASSESSING THE VALUE 131 possible to reduce the important outcomes of community-based prevention to a more manageable—but still understandable—number. In either case the goal is to encourage the adoption of interventions for which the value of the benefits exceeds the value of the resources required to produce them. Re-Valuation Prospective valuation feeds directly into the process of deciding whether or not to allocate resources, which may require deciding among priorities competing for the same resources. If the decision is not to invest, the process ends there. If the decision is to invest in the proposed community- based prevention intervention, the process continues with the implementa- tion of the program in the community. While the framework for valuing is complete at this point, implementation should be accompanied by an evaluation of the intervention’s performance and of how well it delivers value to the community, as well as by ongoing monitoring and reporting of progress. The ultimate effects of an intervention depend on the quality of its design and implementation. Context is a powerful determinant of the ultimate outcomes and practice-based evidence of effectiveness becomes an important source of data for those contemplating investment of resources in other communities (Pronk, 2012). Evaluation, which can be thought of as re-valuation, is thus key. For example, the effect of a given initiative could change over time. It could become stronger as community norms and expectations change, as in the decreasing acceptability of smoking in public spaces. Or it could become weaker as the initial enthusiasm for a change wears off, as with the fitness initiatives of the early 1960s. The evaluation process follows projection processes closely. In particu- lar, as the program or policy is in operation and actual resources are being used for it one component of evaluation focuses on quantifying the results of the program. Measurement includes an assessment of the resources actu- ally deployed and the extent to which the projected changes in intermediate and long-term outcomes are achieved. The duration of the intervention may affect its costs. Fixed costs and variable, recurring costs should both be considered, with appropriate discounting. For example, a school-based educational campaign will need to be repeated for each new cohort of students; the fixed costs of developing the curriculum can be spread over the years it is used. When both costs and benefits—and, therefore, their ratio to each other or the costs relative to added years (or added quality) of life—can vary over time, it is important to include time in the evaluation of the intervention. Measurement of the baseline state before the intervention is also re- quired so that changes may be determined at intermediate steps in the process of implementing the intervention or after completion. For example,

OCR for page 119
132 ASSESSING THE VALUE OF COMMUNITY-BASED PREVENTION it is important to know if necessary resources were mobilized across the community and to what degree the intervention was applied to the target audience(s). Measurements should be made across all the factors in the health, community well-being, and community process domains that were previously identified by the community as being valued. Including such an examination in the evaluation process also provides information that can be used to determine how well the framework accurately projected change. The evaluation component considers the short-, intermediate-, and long-term impacts of the community-based prevention intervention and provides ongoing reports of progress over time. Program administrators should consider how this intervention may be assessed in terms of its imple- mentation fidelity (the degree of fit between the developer-defined elements of a prevention program and its actual implementation in a given organiza- tion or community setting), how it may be generalized to other audiences or communities, whether or not the intervention is scalable and sustainable in the long run, and how to share what is learned of the actual process of implementation with others so that this knowledge can be disseminated as practice-based evidence of effectiveness. Finally, ongoing reporting of experience and progress should be shared with all stakeholders, used for continuous improvement, and aligned with surveillance efforts of health indicators across the community. INFORMATION NEEDED TO ASSESS VALUE USING THIS FRAMEWORK According to the way that the committee has presented the frame- work, there are three types of information needed to assess the value of an intervention: (1) What is the baseline state? (2) What impacts might an intervention have? (3) What impacts did an intervention have? The answers to all three questions should represent the best information available. In Chapter 4 the committee noted the importance of being explicit about the data sources and criteria for admissible evidence to be used in the assess- ment of value. One important methodological note is that formal program evaluation requires a reasonable way to determine what the progress in those outcomes would have been had the intervention not taken place. Measurement of a baseline is needed in order to project such changes or to measure the relative impact of intervention implementations at various times. Baseline measurement needs to occur in the health, community well-being, and community process domains and, ideally, be reflected in the summary mea- sures proposed. Therefore, the committee urges that evaluations include a reasonable control or comparison group or other methodology (interrupted

OCR for page 119
A FRAMEWORK FOR ASSESSING THE VALUE 133 time series, quantitative and qualitative mixed methods, etc.) to support the evaluation of the impact of the intervention. Systematic literature reviews on the evidence of effectiveness provide the highest quality information for projecting changes in outcomes, but other sources are often needed, particularly when estimating the value of in- terventions prospectively. Good-quality cost data are also important (Luce et al., 1996; Polsky and Glick, 2009). As empirical evidence accumulates, that information can be used to refine the analyses. Information related to the impact of the intervention should be con- sidered in the context of a model of causation (or a logic framework) that allows the outcomes valued by the community to be connected (both posi- tively and negatively) to the various activities included in the intervention. For example, the causal loop subsystem related to tobacco use prevention policies described in Appendix B specifies a variety of factors important in generating changes. Based on this type of approach, information may be gathered that is, at a minimum, inclusive of the factors outlined in the causal model. HOW TO USE THE FRAMEWORK IN THE COMMUNITY CONTEXT An assessment of the value of an intervention is just one piece within the larger decision-making process. The proposed framework is intended to aid decision making about adopting community-based prevention interven- tions in a broad range of contexts. It is also intended to assist in the task of monitoring and evaluating community-based prevention interventions once they have been adopted and implemented. Decision makers may operate at the level of higher-level funders, whether public (national or state agencies) or private (foundations or businesses), or they may operate at the level of a community initially deciding what to propose to a local government or to a decision maker at a higher level considering competing proposals. At any level, an early decision must be made about the group of stakeholders that should be included in the process of planning and valuing the proposal. Al- though who should be included will vary with the nature of the intervention and the level at which the decision about adoption will be made, in general a broader group of decision makers will give voice to a broader range of values to be considered. The framework encourages a comprehensive valu- ation process and so encourages the broad inclusion of various stakeholders in a decision. Furthermore, the different voices need to be balanced so that some stakeholders do not have undue influence at the expense of others. The framework encourages a broad consideration of the benefits and harms as well as the costs of a community-based prevention intervention. It seeks some agreement on the net benefit or value of any intervention

OCR for page 119
134 ASSESSING THE VALUE OF COMMUNITY-BASED PREVENTION while recognizing two prominent features of the situation that make such agreement on its value something that may take thoughtful deliberation by decision makers, including some negotiation and compromise. The two features are that there is often disagreement about just what values must be considered; the second is that some of the values may be difficult to quan- tify, making it difficult to compare them and arrive at agreement. These are unavoidable features of a situation with a range of kinds of benefits, kinds of costs, and kinds of harms. Accordingly, the framework does not offer a decision procedure or algorithm for making choices about competing interventions. The frame- work nevertheless can help decision makers and others identify the types of outcomes that contribute to the value of an intervention. If the intervention is setting aside lane space for bicycling, for example, there are quantifiable health effects, not only from the exercise, but also from the reduction of bicycling accidents. There may also be non-quantifiable effects of people bicycling, such as the pleasure of activity and the respect for those engaged in it that is shown by setting aside space for it. The framework reminds deliberators to think about the broad range of valued consequences of the community-based prevention intervention. One dimension of the health outcomes that affects value is the possible conflict between equity and improving aggregate health for a population. Sometimes these two goals of health policy pull in the same direction, and sometimes they conflict. A community-based prevention intervention may be good at improving aggregate health, but it may have a greater effect on those already better off in some important way—say by income or residential location or occupational status—and this may increase health disparities. The willingness of people to trade off increased inequality for aggregate improvement may vary significantly. Reasonable disagreement about how to weigh these two values may persist, and the framework can make the source of that disagreement more visible. Furthermore, community-based interventions often focus their gains and harms on particular groups within a community. Likely targets of an intervention include people with low income, people with disabilities, and racial and ethnic minorities. An important question is whether or not spe- cial weight should be given to gains or harms accruing to or imposed on particular groups. If particular communities prefer to attach special import to the gains or harms that accrue to certain target groups, efforts should be made to establish explicit weights that could be attached to these gains and harms. In this way, the desire to reflect equity concerns in the valua- tion effort can be achieved in the overall valuation process. However, it is important that the process be clear and explicit. This requires that a clear set of weights or values be established before attaching the weights to the projections of gains and harms to particular groups.

OCR for page 119
A FRAMEWORK FOR ASSESSING THE VALUE 135 The persistence of such disagreement around values suggests there may be a legitimacy problem for decision makers. Even if they are the appropri- ate authorities for making such decisions, they need to make them in the “right” way if legitimacy is to be obtained. They must listen to the appro- priate voices expressing different value commitments. Their process should search for rationales that take the relevant values into consideration, and the rationales must explain the basis for giving them the weight that the de- cision reflects. The framework can help identify the value components that need to be considered, and it can even help clarify who might value those issues and therefore who should be listened to. Applying the framework to alternative interventions may thus clarify the various ways in which the value of these interventions differs. As stated above, the committee recom- mends that the value of community-based prevention interventions should reflect the preferences of an inclusive range of stakeholders. Transparency improves the deliberative process, and the framework emphasizes its importance. Determining the value of an intervention in a transparent way can enhance legitimacy. In particular, it is important that the rationales for decisions be made publicly available. The framework can also be used in revisiting a decision in light of new evidence and arguments. In this context it can add consistency to the deliberation by helping decision makers consider again the range of values that influenced the original decision. Monitoring and evaluation of an intervention can answer the question, does the value it initially promised and that was the basis for adopting it emerge in the process of implementation? The framework can guide the design of monitoring and evaluation that should be part of good plan- ning for any community-based prevention intervention. The comprehensive identification of the specific benefits, harms, and resources used that were included in the value of the intervention should guide the monitoring and evaluation process, for it will track the resulting intervention to see if esti- mated net benefits are realized. Ideally, a good monitoring and evaluation process can identify ways to improve the implementation or revise the in- tervention so that negative effects, or costs can be reduced. The framework urges such ongoing assessment of the value of a community-based preven- tion intervention. Recommendation 6: The committee recommends that, to assure transparency, a. analysts make publicly available the evidence used for valuation and provide estimates of the uncertainty of their results, and b. decision makers make publicly available the rationale for their decisions.

OCR for page 119
136 ASSESSING THE VALUE OF COMMUNITY-BASED PREVENTION IMPLICATIONS FOR STATE AND NATIONAL POLICY Frameworks for valuation, such as the one presented in this paper, have the potential to impact federal, state, and local policy making in significant ways. Chapter 4 reviewed eight existing valuation frameworks: benefit–cost analysis, cost-effectiveness analysis, Congressional Budget Office scoring, the PRECEDE–PROCEED framework, the RE-AIM framework, Health Impact Assessment, the Community Preventive Services Task Force guidelines, and the Canadian Health Services Research Foundation (Lomas) model. These eight frameworks have several elements in common: (1) They have passed through many rounds of validation and refinement, (2) they are broadly ac- cepted among researchers and policy makers, and (3) they are incorporated into the formal process of policy making and not merely used piecemeal to advocate for or against specific proposals. As with the frameworks discussed in Chapter 4, the committee’s frame- work has limitations. The framework presented in this report is in its very early stages, and so its near-term impact on policy making is likely limited. Because of the importance of contextual factors and the limited scope and generalizability of evidence on the effects of community-based prevention, the framework does not yet provide a detailed roadmap for valuation. Clear, consistent measurement of the elements of value are important. Yet comprehensive data are often not available to measure tangible benefits adequately and the measurement of the many intangible benefits is not yet well developed. Such a broadly inclusive framework may seem overly abstract or unreliable to some observers. As the framework is applied, new measures and data sources will need to be developed as will an ap- propriate methodology for creating valid single indicators for community well-being and community process. Old measures and data sources will need to be applied in new ways, a process that will take time to establish validity and gain acceptance. The committee has recommended several steps to take to promote progress on these fronts. Although much work remains, the committee’s proposed framework is designed to capture the value of community-based prevention by taking a comprehensive approach, comparing benefits, harms, and resources used in three domains, and taking into account community context. Expanding the influence of this framework will require building a consensus that the outcomes on which it focuses (health, community well- being, community process, and resources used) are broadly important and not just the narrow interests of a specific group. First and foremost, such validation involves testing whether or not this model is useful to com- munities and stakeholders in general as an organizing framework. Next, one would need to examine how the framework, in general, responds to such factors as utility, feasibility, propriety, and accuracy. Furthermore,

OCR for page 119
A FRAMEWORK FOR ASSESSING THE VALUE 137 validation of the framework could include a consideration of its scalability and sustainability, whether it can support capacity building for health in communities, whether it can address health equity effectively, and whether it is generalizable across many contexts and settings. It will also be important to validate the framework by showing re- peatedly that it correctly distinguishes between interventions that improve community well-being and those that do not. This process of validation will almost certainly require refining the framework and expanding the under- lying evidence base. Following consensus and validation, the framework can be formally incorporated into the policy-making process. This formal role could consist of a requirement that legislative or grant proposals be accompanied by an objective impact assessment based on the framework or of a requirement that executive branch agencies use the framework in evaluating the output of their programs. A formal role could also consist of a requirement that discretionary funding be distributed based on valuations that use the framework. Although that type of role may be many years off, the existing frameworks described in Chapter 4 provide clear precedents for such a progression. CONCLUSION Two transitions have led to changes in perspective about the kinds of interventions needed to address today’s challenges to living a healthy life: (1) the shift in major causes of illness and death from communicable diseases to chronic diseases, and (2) an increased emphasis on the social determinants of health. Community-based prevention interventions seek to address the distribution of health and risk factors in populations (e.g., the social determinants of health) that contribute to today’s primary causes of death and disease. But determining the value of community-based interven- tions has proven difficult. Existing frameworks for valuing interventions fall short, and the committee concluded that what is needed is a framework that focuses on population-level impact and that can take account of inter- sectoral action, community participation and empowerment, context, and systems thinking. The framework proposed by the committee is comprehensive and includes the assessment of the benefits, harms, and resource use of c ­ommunity-based prevention interventions in the three major domains of health, community well-being, and community process. The framework also proposes that summary measures or single indicators be developed to assess value in these three areas and that these be compared with a sum- mary measure of resource use. Until such time as a single indicator for each domain exists, however, it will be appropriate to use different metrics for

OCR for page 119
138 ASSESSING THE VALUE OF COMMUNITY-BASED PREVENTION measuring the different domains of value. Chapter 3 describes many of the outcomes that can be measured and weighted in each domain. The assessment of value of an intervention usually takes place within a decision-making context. Stakeholders and decision makers come from different perspectives and emphasize different factors. It is important, there- fore, that the value assessment reflect the preferences of an inclusive range of stakeholders. It is also important that there be transparency in the use of the framework so that there is understanding about the rationale and evidence used for making decisions. As stated earlier, the proposed framework is in its very early stages and much is yet to be learned. However, the framework identifies critical areas for valuing and the report proposes additional areas where work needs to be undertaken. REFERENCES Gold, M., J. Siegel, L. Russell, and M. Weinstein. 1996. Cost-effectiveness in health and medicine. New York: Oxford University Press. IOM (Institute of Medicine). 2011. A nationwide framework for surveillance of cardiovascular and chronic lung diseases. Washington, DC: The National Academies Press. Luce, B., W. Manning, J. Siegel, and J. Lipscomb. 1996. Estimating costs in cost-effectiveness analysis. In Cost effectiveness in health and medicine, edited by M. Gold and J. Siegel. New York: Oxford University Press. Peacock, S., J. Richardson, R. Carter, and D. Edwards. 2007. Priority setting in health care using Multi-Attribute Utility Theory and Programme Budgeting and Marginal Analysis (PBMA). Social Science & Medicine 64(4):897-910. Polsky, D., and H. Glick. 2009. Costing and cost analysis in randomised trials: Caveat emptor. Pharmacoeconomics 27(3):179. Pronk, N. P. 2012. The power of context: Moving from information and knowledge to practi- cal wisdom for improving physical activity and dietary behaviors. American Journal of Preventive Medicine 42(1):103-104. Zahedi, F. 1986. The analytic hierarchy process—a survey of the method and its applications. Interfaces 16(4):96-108.