Below are the first 10 and last 10 pages of uncorrected machine-read text (when available) of this chapter, followed by the top 30 algorithmically extracted key phrases from the chapter as a whole.
Intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text on the opening pages of each chapter. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.
Do not use for reproduction, copying, pasting, or reading; exclusively for search engines.
OCR for page 89
4 Existing Frameworks This chapter reviews eight frameworks that are currently used to assess the value of community-based prevention: benefit–cost anal- ysis, cost-effectiveness analysis, Congressional Budget Office scor- ing, PRECEDE–PROCEED, RE-AIM, health impact assessment, the Community Preventive Services Task Force (CPSTF) guidelines, and the Canadian Health Services Research (Lomas) Model. The committee concluded that existing frameworks are inadequate for assessing the value of community-based prevention because none meet all of the most important criteria outlined in Chapter 3 and this chapter. Most lack community well-being measures other than health, some do not assess the value of the community processes by which prevention activities are planned and undertaken, many do not consider costs, and some do not give sufficient attention to the individual community context. WHAT IS A FRAMEWORK FOR ASSESSING VALUE? The committee concluded that a framework for assessing value is a structure for gathering and processing information to aid intelligent deci- sion making and, more specifically, to help decide whether an activity or intervention is worthwhile. (Frameworks for implementation are different: They focus on how best to implement a program. See Chapter 2 for a description of the most important frameworks for the implementation of community-based prevention.) 89
OCR for page 90
90 ASSESSING THE VALUE OF COMMUNITY-BASED PREVENTION A framework for assessing value can aid decision making by • requiring that goals be stated clearly; • integrating incomplete and sometimes conflicting information and beliefs; • avoiding decision making based on arbitrary impressions or self-interest; • clarifying trade-offs; • promoting transparency; and • exposing legitimate sources of disagreement and helping to work through them. Frameworks for assessing value can be geared toward prospective or retrospective assessments of value. A prospective assessment of value is performed before an intervention takes place and is designed to help policy makers decide whether to undertake the intervention. An example of a prospective assessment is a cost estimate produced by the Congressional Budget Office. Program evaluations are concurrent or retrospective assess- ments of value: What can the evaluators say about an intervention’s value while it is being implemented or after it has occurred? (Stufflebeam, 1999). Benefit–cost analysis, cost-effectiveness analysis, and some other valuation frameworks, may be either prospective or retrospective (Nash et al., 1975). The committee concluded that a framework for assessing value should include the following elements: • A decision-making context o Who are the decision makers, what are the decisions they are making, and what are the formal and informal mechanisms by which assessments of value feed into the decision-making process? • A list of valued outcomes o What does the user of the framework care about? What should the user of the framework care about? • A list of admissible sources of evidence o What information does the user of the framework use to build the model of causation that links interventions to valued outcomes? • A method for weighting and summarizing o How is information on all the valued outcomes boiled down and made digestible for decision makers?
OCR for page 91
EXISTING FRAMEWORKS 91 A Framework for Assessing Value Is Embedded Within a Decision-Making Context Frameworks have evolved to feed into specific decision-making con- texts. Examples of decision-making contexts include Congress deciding whether to enact a piece of legislation, a local health department deciding how to allocate its budget to specific health promotion activities, and a community group deciding whether to organize its volunteers to undertake a specific health-related project. A framework that is appropriate and help- ful in one decision-making context may not be helpful in another. As a result, the description of a framework must take into account the decision- making context in which it is or will be used. Different decision makers come from different perspectives and empha- size different factors. Possible factors to consider include legal and ethical issues, the nature of the condition, resource availability, administrative fac- tors, and idiosyncratic factors. These can sometimes be taken into account in the valuation framework. At other times, decision makers must consider them outside of—and in addition to—the assessment of value. For example, while family-planning activities are legal and may be a valued outcome, they can be constrained by ethical attitudes toward abor- tion and contraception. Differing ethical and religious views in the com- munity may need to be considered outside the valuation framework. The nature of the health condition may also need to be considered separately from valuation. Some conditions, such as conditions that affect young chil- dren, may evoke more sympathy and a greater sense of urgency than others. Another factor that may need to be considered separately from the valuation is resource availability. Lack of the right resources may interfere with the adoption of an intervention, even when its assessed value is high relative to its costs; the right facilities and people may not be available. The availability of administrative mechanisms that can enhance the acceptability of an intervention is another factor that is considered in decision making but that is outside the valuing framework. For example, the Supplemental Nutrition Assistance Program (SNAP, formerly the Food Stamp Program), with its eligibility requirements to assure that benefits reach the intended population, could serve as the administrative base of a community interven- tion to improve the nutrition of low-income people. Finally, idiosyncratic factors, such as powerful advocates or vested interests, can outweigh assess- ments of value that the larger community places on interventions. All of these factors must be taken into account in actual decisions. A framework for valuation provides a way of focusing attention on the valued outcomes and costs of interventions in a systematic way and helps make those outcomes and costs clear to the larger community in a way that promotes better choices.
OCR for page 92
92 ASSESSING THE VALUE OF COMMUNITY-BASED PREVENTION A Framework Includes a List of Valued Outcomes To provide effective support for decision making it is critical to list all the valued outcomes and to show, perhaps in a table, how much each inter- vention contributes to each outcome. That means that there must be some measure of how each outcome is affected by the intervention. For example, health might be measured in years of life gained or quality-adjusted life years gained. Community participation might be measured by the number of people who attend events or the hours of work volunteered in a year. Reductions in crime or in health risk factors might be represented by the statistics already established by police systems or by disease registries or health surveys. If decision makers are choosing among a number of possible interven- tions, it helps if measures of valued outcomes can be devised that work across interventions so that the outcomes of the different interventions can be compared. That is, it helps if each health outcome can be measured in the same way for all interventions, if each community process outcome can be measured in the same way for all interventions, and so on. If each program has its own measures that are different from those of every other, it becomes more difficult to compare interventions. Program costs (i.e., the value of the resources used) should also be measured. All resources should be measured, whether or not they are pur- chased. The time donated by community volunteers is a major example of a resource used in community-based programs, often in large quantities, that is rarely counted as a cost when choices are evaluated. The true cost of volunteer time, as with any other resource, is that if it is used for one program, it is not then available for other programs or for other activities that the volunteer might engage in. So the program chosen should be a worthwhile use of that time, preferably the best use. A Framework Includes a List of Sources of Evidence and a Standard for Admissible Evidence Every assessment of value is built on a model of causation, i.e., a theory of how the world works. Those models of causation can be built up from many different sources and types of evidence. Some frameworks make explicit the sources of evidence that are taken into consideration and the standards that each source of evidence must meet. The Community Preventive Services Guide, for example, includes a clear description of the sources from which the task force draws evidence and the standards that are used to grade the quality of different pieces of evidence (Carande-Kulis et al., 2000). Other frameworks, such as benefit–cost analysis, have clear criteria for what is to be counted and what is not, but the execution of
OCR for page 93
EXISTING FRAMEWORKS 93 the analysis must still rely on the analyst’s judgment (Nash et al., 1975; Weisbrod, 1983). A Framework Includes a Method for Weighting and Summarizing Ultimately, after the users of a framework have listed all the outcomes they value and have measured program outcomes in those terms, they must choose among the programs. That can be hard to do when the valued outcomes take different forms and the strength and weight of evidence supporting them vary across outcomes. Intervention A may provide safer streets and community participation among parents and children; interven- tion B may provide meals and social interaction for isolated elderly people; interventions C, D, and E may offer still other things of value. Which pro- grams are most valuable? Which should be done if not all can be? Which should be done first? If there are only a few interventions and only a few outcomes, listing the contributions of each intervention to each outcome can be sufficient to allow people to choose among them. But the more interventions and the more outcomes, the more difficult the choice becomes. In that case people can end up focusing on one outcome, such as health, and ignoring the oth- ers, simply because it becomes too difficult to know how to take them all into account. An overall summary measure can help prevent this narrowing of focus, although this is not always possible. How Do We Know if a Framework Works? A framework works if it supports an intelligent decision-making pro- cess, that is, a process that clarifies trade-offs, reminds decision makers of the things that are important, and helps decision makers explore and work through, rather than gloss over, disagreements. Of course, a particular deci- sion may seem intelligent to one person, while it seems an awful mistake to another. Disagreements on ultimate decisions are inevitable. One sign that a framework works well is if it is perceived as valid and useful by people who disagree vehemently about what decision should be made. EIGHT EXISTING FRAMEWORKS The committee has identified eight existing frameworks that have been used to assess the value of community-based preventions: 1. benefit–cost analysis, 2. cost-effectiveness analysis, 3. Congressional Budget Office (CBO) scoring,
OCR for page 94
94 ASSESSING THE VALUE OF COMMUNITY-BASED PREVENTION 4. the PRECEDE–PROCEED framework, 5. the RE-AIM framework, 6. the Health Impact Assessment (HIA) framework, 7. the Community Preventive Services Task Force (CPSTF) guidelines, and 8. the Canadian Health Services Research Foundation (Lomas) Model. Three of the existing frameworks emerged from the field of econom- ics (benefit–cost analysis, cost-effectiveness analysis, and CBO scoring), while the rest have their roots in the field of public health planning and promotion (PRECEDE–PROCEED, RE-AIM, HIA, CPSTF, and the Lomas model). The committee’s task in analyzing these frameworks is to identify whether they work well for assessing the value of community-based preven- tion and, if not, why not. The following sections discuss each framework in terms of its decision-making context, its list of valued outcomes, its criteria for admissible evidence, its weights and summarizing, and its limitations. Benefit–Cost Analysis Unless otherwise noted, information in the following section was ob- tained from Carlson et al. (2011) and Weisbrod (1983). The benefit–cost analysis (BCA) framework grew out of the belief that society’s problems can be solved systematically through the rigorous application of quantita- tive scientific principles. BCA was originally developed to guide decisions regarding large-scale government infrastructure projects, such as dam build- ing and flood control projects, and it is geared toward deciding whether a major capital investment is worthwhile (Subcommittee on Evaluation Standards, 1958). (See Box 4-1 for a thorough description of the BCA methodology.) Decision-Making Context In the United States BCA has been used mainly by executive branch agencies of the federal government to guide decisions on whether or not to implement infrastructure projects, job training programs, and other social programs. It has also been applied in regulatory impact analyses to deci- sions such as limits on toxins in drinking water standards. List of Valued Outcomes In principle BCA takes a societal perspective, meaning that it takes into account all the things that all the individuals in society care about. This
OCR for page 95
EXISTING FRAMEWORKS 95 completeness of perspective is BCA’s core strength. Ideally the list of val- ued outcomes includes market-traded goods and services that can be easily expressed in dollars as well as things like fairness and risk avoidance that are either difficult or impossible to express in dollars. In practice analysts find it difficult to document and quantify the value of things like fairness. Generally, a sound BCA will describe the fairness effects in a separate sec- tion, sometimes under the label of “intangibles.” Criteria for Admissible Evidence In general, it is up to the analyst to decide what evidence to include in measuring the quantitative effects of an intervention and the value of those effects. The evidence that is most clearly admissible is high-quality published quantitative evidence on the effect of the policy on some out- come of relevance. Price data for market-traded goods and services that are program outcomes are also admissible. A difficult problem occurs when an important outcome that is affected by the intervention is not traded in markets and, therefore, has no price. In this case, the analyst must rely on systematic reviews of published academic literature on such topics as willingness to pay as well as on expert opinion and on his or her personal judgments. That flexible approach is essential given the comprehensive list of valued outcomes in BCA and the very wide scope of projects to which BCA can be applied. As with some other frameworks for assessing value, this flexibility inevitably requires users to apply judgments regarding the reliability of estimates of benefits and costs. Weighting and Summarizing BCA uses a single metric—dollars (or other currency)—to summarize the good and bad effects of an intervention and the resources used to under- take the intervention. Dollar values are assigned in a straightforward way to market-traded goods and services, but they are also assigned to things, such as extended life expectancy that are not traded directly in markets. The concept of “willingness to pay” is used to provide a dollar value for such things as the chance of a better health outcome due to a proposed in- tervention. All future benefits of a project are summarized in present-value dollars, as are all future costs. The present value of a cost or benefit that occurs in the future is deflated to the present using a discount factor. The BCA technique uses methods for measuring and describing the degree of uncertainty in the assessment of value. One prominent technique is called Monte Carlo simulation analysis, in which the value of an inter- vention is assessed repeatedly, each time using assumptions that are drawn randomly (hence “Monte Carlo”) from a range of possible values defined
OCR for page 96
96 ASSESSING THE VALUE OF COMMUNITY-BASED PREVENTION BOX 4-1 Benefit–Cost Analysis: Theoretical Basis and Practical Considerations Benefit–cost analysts seek to identify both private and public decisions in which the benefits (or outputs) are greater than the costs (or inputs). Market prices should be used to measure the value of both benefits and costs, except where market prices do not exist or when there is a good reason to believe that market prices do not accurately reflect true value. Among the reasons for questioning the appropriateness of observed prices are the existence of monopoly power in particular sectors, economies of scale, a serious lack of information, or external effects (or spillovers) not reflected in market values. Where the prices of inputs or outputs do not exist, analysts strive to construct values that reflect people’s valuation of inputs and outcomes. (These are known as shadow values or shadow prices.) Although economic efficiency should be regarded as the primary objec- tive, when decisions have important equity (and other) effects, these should be recorded and, if possible, entered into the benefit–cost analysis itself. An alterna- tive way of describing the efficiency criterion is to state that projects should be designed to maximize total (or per capita) national economic welfare, which is often assumed to be equivalent to national income. To do this the project should maximize the net benefits that it generates. Benefits (or costs) that will not occur until sometime into the future should be valued (weighted) as less important (per dollar) than benefits or costs expected to be incurred immediately because (and only if) this reflects how people feel about future benefits and costs relative to present ones. The process employed for mak- ing benefits and costs which occur at different points in time commensurable is called discounting and requires the use of a discount (or interest) rate to reflect the diminished value today of benefits or costs not expected to occur until some future time period. For projects that generate a stream of future benefits or costs, the benefit–cost ratio (B/C) is B/C = Total discounted value of future expected benefits/ Total discounted value of future expected costs, or Net Present Value of Program = Present value of benefits/ Present value of costs by the analyst. This enables the user to see the range of possible outcomes under various combinations of assumptions as well as the likelihood that they will be realized (Savvides, 1994). More Details and Examples Among the many published BCA studies, two of the most compre- hensive are Weisbrod’s (1983) analysis of non-institutional care for the
OCR for page 97
EXISTING FRAMEWORKS 97 An intervention has externality, or spillover, effects if it affects individuals who do not directly participate. Externalities, which can be positive or negative, should be accounted for in a benefit–cost analysis, even if observed market prices are not available for the valuation of the effect. The concept of “benefit” (which when negative becomes a “cost”) underlies all benefit cost analyses; a clear understanding of the meaning of “benefit” is the – fundamental requirement for undertaking any sound benefit–cost study of public activities. It is useful to think of the benefits of a public intervention as the extent to which the program produces desirable results. What is or is not desirable de- pends, in turn, on the goals or objectives of the program. This is to say, the first step in the process of project evaluation or policy analysis should be a statement of goals. The second step should be an attempt to state these goals in operation- ally measurable form. The third step in the analysis is the development of a set of weights that reflect judgments about the comparative importance of progress toward each of the goals—the goal trade-offs. Valued outcomes in BCA can be grouped into two principal categories: (1) those related to economic allocative efficiency and (2) those related to distribu- tional equity. Allocative efficiency as an economic goal reflects the fact that it is some- times possible to reallocate resources—perhaps increasing or decreasing the amount of resources used in any expenditure program in ways that will bring about an increase in the net value of output produced by those resources. For such reallocations the increase in the value of the output of the good whose production is expanded must be greater than the decrease in the value of the output of the good whose production is decreased. Insofar as benefit–cost analysis is directed at allocative efficiency, it can be viewed as an attempt to replicate for the public sector the decisions that would be made if private mar- kets worked satisfactorily (Haveman and Weisbrod, 1977). However, allocative efficiency ignores considerations of which particular people are made better off or worse off. The issue of how alternative resource allocations affect the well- being of particular people is captured by the distributional—or equity—goals. The goals can and should be incorporated into benefit–cost analysis, but must be done explicitly. One way of examining distributional effects is through sen- sitivity analysis, which varies the values of certain inputs to determine their influence on output. mentally ill and Carlson et al.’s (2011) analysis of Section 8 housing subsidies. Shortcomings of BCA When Applied to Assessing the Value of Community-Based Prevention BCA requires a monetary assessment of how a community trades one outcome for another. However, some outcomes, such as social cohesion or
OCR for page 98
98 ASSESSING THE VALUE OF COMMUNITY-BASED PREVENTION civic participation are not readily monetized, and there are other factors, such as years of human life, that can (and have been monetized), but for which monetization is controversial. An example of the latter can be seen in the value of a year of human life assigned in the regulatory impact analyses conducted by the Environmental Protection Agency (IOM, 2006). Cost-Effectiveness Analysis Cost-effectiveness analysis (CEA) begins with many of the core ele- ments of BCA framework, but it is tailored to the assessment of medical and health interventions. The key difference between CEA and BCA is that CEA focuses on health as the valued outcome and measures health by methods that avoid the use of dollars (Donaldson, 1998; Gold et al., 1996). The measures of health often used in CEA include cases of disease, life-years, and health-adjusted life expectancy. The core question that CEA answers is how much it costs to produce an additional unit of health using one particular intervention versus a second intervention with which it is compared (Bleichrodt and Quiggin, 1999; Donaldson, 1998; Drummond et al., 2005; Gold et al., 1996). The answer to this question is given by what is termed the “cost- effectiveness ratio.” An intervention’s cost-effectiveness ratio is, in effect, the dollars spent for an additional unit of health. The health measure that is the standard of good practice is quality-adjusted life years (QALYs), so that the cost-effectiveness of an intervention is thus expressed as the additional cost to achieve an additional QALY (Gold et al., 1996). A cost-effectiveness ratio, sometimes called an incremental cost-effectiveness ratio, is not a fixed or single number associated with an intervention. It is instead defined with reference to some alternative intervention, and thus its value depends on what the alternative is. For example, vaccinating children once in early childhood may be compared with not vaccinating them but instead treat- ing the disease when it occurs. Or providing a single vaccination could be compared with providing multiple vaccinations over time, perhaps includ- ing a booster in adulthood. The cost-effectiveness of vaccinating once will differ depending on which comparison is chosen. Decision-Making Context CEA is geared toward maximizing the health improvements achieved among a target population and to analyzing the resources or costs likely to be required to achieve those health improvements. An example of this sort of decision-making context is provided by the Department of Veter- ans Affairs, which operates a health system on a fixed budget with the goal of improving the health of its enrolled population. CEA can help the
OCR for page 99
EXISTING FRAMEWORKS 99 administrator of that program prioritize technology adoption and choose the treatment guidelines that produce health improvements most efficiently. Cost-effectiveness analysis is not used explicitly in the development of cov- erage policy and practice guidelines in the United States. In other countries cost-effectiveness is often explicitly used to help set standards of care and coverage. The UK National Institute for Health and Clinical Excellence (NICE) uses cost-effectiveness analysis as one element in deciding whether the National Health Service (NHS) will pay for new technologies; its purpose is to help ensure that everyone in the country has access to proven medical care (Steinbrook, 2008). NICE sometimes uses cost-effectiveness analysis to negotiate prices with manufacturers who can improve the cost-effectiveness of their product by reducing its price (Kanis et al., 2008; Steinbrook, 2008). Australia’s Pharmaceutical Benefits Advi- sory Committee (PBAC) uses cost-effectiveness analysis to develop guidance for the Minister of Health on the medications that should be covered by the national pharmacy benefits plan (Department of Health and Aging, 2007; Henry et al., 2005). List of Valued Outcomes Health is the primary valued outcome in CEA. CEA takes into account the health improvements and adverse effects from the intervention that oc- cur over a specific time horizon, often the lifetime of patients, to calculate the net health benefits, which are defined as improvements minus adverse effects. As noted, health improvements include both longer life and better quality of life. CEA calculates the resources used to produce the health improvements separately. Resources include market-traded items, such as physician labor, hospital care, and pharmaceuticals, as well as non-market- traded items, such as patients’ time and the time of unpaid caregivers. The health improvements and costs of an intervention are then compared with an alternative intervention—vaccination with waiting and treating illness, screening annually with screening less often, and so on—in order to arrive at the net addition to health and the net addition to costs (or savings) of the intervention compared to the alternative. The cost-effectiveness ratio is the net costs divided by the net addition to health (Gold et al., 1996; Weinstein and Stason, 1977). Criteria for Admissible Evidence In general the modeling team uses the best available evidence, with preference given to published peer-reviewed literature (Gold et al., 1996). When published data are not available, the team is expected to use judg- ment and expert opinion to fill in key parameters when published data are
OCR for page 108
108 ASSESSING THE VALUE OF COMMUNITY-BASED PREVENTION List of Valued Outcomes HIAs do not have a specific list of health outcomes to evaluate. Rather, a scoping process, including literature reviews and expert consultation, is used to identify the potential health impacts that are of importance to the affected stakeholders. The level of stakeholder participation varies with the type of policy or project. For example, the development of new residential and commercial infrastructure would engage those currently living in the area as well as individuals in the business and other communities. Consid- erations might include the effects of displacement, the disruption of social networks and cohesion, increased housing cost, changes in access to public transportation, and changes in job prospects as well as the impact of com- mercial development (London’s Health, 2000; UCLA HIA-CLIC, 2012). Criteria for Admissible Evidence In general the best available information is used to assess the health impact. Evidence may be qualitative or quantitative. The degree of rigor is often contingent on the nature of the project, the analytic resources avail- able, the urgency of the decision makers, and the time available (Snowdon et al., 2010). Weighting and Summarizing Changes in health status are usually displayed in natural health units and are sometimes summarized using QALYs. More Details and Examples The 2011 NRC report Improving Health in the United States provides a summary of the purposes, methods, and uses of HIAs. Shortcomings of HIA When Applied to Assessing the Value of Community-Based Prevention The primary shortcoming of HIA is that it does not capture the costs associated with an intervention. Analyses are adapted to the specific interven- tions and stakeholders and thus they can vary significantly (Kemm, 2003). Community Preventive Services Task Force Guidelines The Community Preventive Services Task Force (CPSTF) is an indepen- dent, nonfederal, volunteer body with members appointed by the Director
OCR for page 109
EXISTING FRAMEWORKS 109 of the Centers for Disease Control and Prevention (CDC). Those members represent a broad range of research, practice, and policy expertise in com- munity preventive services, public health, health promotion, and disease prevention (Community Guide, 2012d). See Box 4-2 for CPSTF Prioritiza- tion Process. Decision-Making Context The CPSTF recommendations influence decisions of the CDC and of other funders regarding which activities to fund. Local health departments, community groups, and health systems also use the recommendations to decide which interventions to undertake (Community Guide, 2012b). BOX 4-2 The CPSTF Prioritization Process The task force prioritization committee is responsible for overseeing the process of prioritizing topics. The process begins with formally requesting stake- holders to suggest high-priority topics. The task force then collects and evaluates information on each potential topic using the following criteria: • potential magnitude of preventable morbidity, mortality, and health care burden for the U.S. population as a whole based on estimated reach, impact, and feasibility; • potential to reduce health disparities across varied populations based on age, gender, race/ethnicity, income, disability, setting, context, and other factors; • degree and immediacy of interest expressed by major Community Guide audiences and constituencies, including public health and health care practitioners, community decision makers, the public, and policy makers; • alignment with other strategic community prevention initiatives, including, but not limited to, Healthy People 2020, the National Prevention Strategy; the County Health Rankings, and America’s Health Rankings; • synergies with topically related recommendations from the U.S. Pre- ventive Services Task Force and Advisory Committee on Immunization Practices; • availability of sufficient research to support informative systematic evi- dence reviews; and • the need to balance reviews and recommendations across health topics, risk factors, and types of services, settings, and populations. SOURCE: Community Guide, 2011.
OCR for page 110
110 ASSESSING THE VALUE OF COMMUNITY-BASED PREVENTION List of Valued Outcomes The CPSTF evaluates the effectiveness of interventions (programs and policies). The key valued outcome is the health of a population, assessed as the sum of the health of individuals. The CPSTF guidelines also mea- sures how effective interventions are in different populations (distributional and equity effects), generalizability, and acceptability (Community Guide, 2012c). For those interventions where there is evidence of effectiveness the resource use and cost-effectiveness are assessed where studies are available, although these are not used for making the primary recommendation (Hahn et al., 2004). Criteria for Admissible Evidence The CPSTF guidelines are based on systematic reviews of the published academic literature. The admissibility of published studies is determined according to the appropriateness of the study design for the intervention being examined and the quality of execution. The criteria recognize that randomized controlled trials may not be the most appropriate study design and that they are often impractical for assessment of community-level inter- ventions. Thus, well-done observational studies are often included (Norris et al., 2002). Weighting and Summarizing The findings from the literature review are summarized on two dimen- sions: Is the evidence strong enough to draw a conclusion (i.e., are there enough studies of suitable design and execution)? And, if so, does the evi- dence indicate that the intervention improves health outcomes? (The Com- munity Guide, 2012a). More Details and Examples For a detailed description of CPSTF’s methodology see Briss et al. (2000) and Carande-Kulis et al. (2000). The CPSTF’s recommendations are available at http://www.thecommunityguide.org/index.html. Shortcomings of the CPSTF Guidelines When Applied to Assessing the Value of Community-Based Prevention The CPSTF guidelines have several shortcomings: The list of valued outcomes focuses only on outcomes associated with health; the process for determining which interventions are studied is highly centralized; and
OCR for page 111
EXISTING FRAMEWORKS 111 conclusions do not consider the tradeoff between benefits and costs, al- though where cost effectiveness information is available that information is summarized. The Canadian Health Services Research Foundation Model The Canadian Health Services Research Foundation (CHSRF) model— also referred to as the Lomas model after Jonathan Lomas, the first chief executive officer of the CHSRF—provides a conceptual framework for com- bining evidence of different types to inform health system decision making. It is not a full-fledged framework for assessing value because it does not specify a list of valued outcomes. Instead, it enumerates the different types of evidence used by different decision makers (Lomas et al., 2005). Decision-Making Context The model focuses on the use of evidence in real-world decision making and recognizes that a broad range of information, correct or incorrect, is used by decision makers. The primary focus is on how those who formulate guidance use different types of evidence in making their recommendations, setting their targets, and providing guidance (Lomas et al., 2005). List of Valued Outcomes The Lomas model does not specify a set of valued outcomes. Criteria for Admissible Evidence The model describes different types of evidence, but it does not have specific criteria for what may be included. All types of evidence are, in general, admissible, which is the generally accepted practice of the relevant disciplines and decision makers. The Lomas model distinguishes three types of evidence: scientific evi- dence, social science scientific evidence, and colloquial evidence. Scientific evidence is considered context-independent—that is, the information is knowable and broadly true. It provides information about whether an intervention can work. The efficacy result of a randomized controlled trial is a typical example. Social science scientific evidence is considered to be context dependent—that is, the information is knowable, but the result depends on the context in which it occurs. Such evidence provides infor- mation about whether an intervention does work in a given community. Effectiveness studies are an example. The final type of evidence, colloquial evidence, “can usefully be divided into evidence about resources, expert and
OCR for page 112
112 ASSESSING THE VALUE OF COMMUNITY-BASED PREVENTION professional opinion, political judgment, values, habits and traditions, lob- byists and pressure groups, and the particular pragmatics and contingencies of the situation” (Lomas et al., 2005, p. 1). Weighting and Summarizing The Lomas model does not prescribe a single approach for weight- ing or summarizing the various pieces of evidence, and it recognizes the importance of all types of evidence and the lack of a technical solution to making the best choice. This framework incorporates a deliberative process of relevant stakeholders to consider and weigh all the different types of information. More Details and Examples See Lomas et al. (2005). Shortcomings of the Lomas Model When Applied to Assessing the Value of Community-Based Prevention The Lomas model does not specify a list of valued outcomes or a method for weighting and summarizing an intervention’s impacts. The Lomas model is more descriptive of the decision-making process and is not meant to be prescriptive or normative (Lomas et al., 2005). VALUING COMMUNITY-BASED PREVENTION: IS A NEW FRAMEWORK NEEDED? This chapter has identified eight existing frameworks for assessing value. Given the profusion of frameworks, is it really necessary to define another one? The answer depends on how well each of the eight frame- works addresses the special characteristics of community-based prevention described in Chapters 2 and 3. The committee concluded that a framework for evaluating community preventive programs and policies should meet at least three criteria: 1. The framework should account for benefits and harms in three domains: health, community well-being, and community process (see Chapter 3). Community-based prevention can create value not only through improvements in the health of individuals but also by increasing the investment individuals are willing and able
OCR for page 113
EXISTING FRAMEWORKS 113 to make in themselves, in their family and neighbors, and in their environment. Furthermore, community-based prevention, by defi- nition, involves decisions among groups of people about how to live in society, how the physical environment should be built, what food should be served in schools, and so on. Thus, the process by which interventions are decided upon and undertaken needs to be treated as a valued outcome. If a community decides to tell people what they can or cannot do or what they should or should not do, the decisions need to have the legitimacy—the added value—that comes from an open and inclusive group decision-making process. 2. The framework should consider the resources used and compare benefits and harms with those resources. To make that compari- son, and to compare different interventions with each other, it is essential not only to know that some benefit is likely, but also to be aware of the magnitude of the benefits and of the costs associated with each intervention. 3. The framework needs to be sensitive to differences among commu- nities and to take them into account in valuing community-based prevention. In part this reflects the reality that, because communi- ties vary so much in their characteristics, the causal links between interventions and valued outcomes may be different for different communities. None of the eight frameworks meets all three criteria—that is, accounts for benefits and harms in all three domains identified in Chapter 3, com- pares benefits with costs, and is sensitive to differences among communities (see Table 4-1). Only three of the eight are comprehensive in accounting for benefits and can thus assess value in all three domains of health, com- munity well-being, and community process. Only three estimate costs as a matter of course. Four are moderate or high in their attention to differ- ences among communities. Benefit–cost analysis, which is comprehensive in accounting for benefits and always estimates costs, does not routinely consider the unique characteristics of the decision-making context and the community. PRECEDE–PROCEED, which measures health and community process benefits and takes the unique characteristics of the community into account, does not require that costs be estimated. The committee concluded that a new framework is necessary to guide the assessment of value for community-based prevention, one that measures benefits comprehensively, compares benefits with costs, and takes into ac- count the differences among and within communities. Chapter 5 describes such a new framework.
OCR for page 114
114 ASSESSING THE VALUE OF COMMUNITY-BASED PREVENTION TABLE 4-1 Eight Frameworks Summarized Includes Compares Accounts for Comprehensive Set of Benefits Differences Among Valued Outcomes with Costs Communities Benefit–cost analysis Yes, can account for Yes Low; can account for (BCA) all benefits context Cost-effectiveness No, health only Yes Low; can account for analysis context Congressional Budget No, only federal Yes Low; designed for Office scoring spending and revenue Congressional budget process PRECEDE–PROCEED No, although it No High; used in framework includes both health communities and community process RE-AIM framework No, health only No High; used by evaluators Health Impact No, health only No High; used in Assessment framework communities Community Preventive No, health only No Moderate; focus on Services Task Force community guidelines Lomas model No, valued outcomes No Moderate; focus not specified on decision-making process REFERENCES Birch, S., and A. Gafni. 2003. Economics and the evaluation of health care programmes: Generalisability of methods and implications for generalisability of results. Health Policy 64(2):207-219. Bleichrodt, H., and J. Quiggin. 1999. Life-cycle preferences over consumption and health: When is cost-effectiveness analysis equivalent to cost–benefit analysis? Journal of Health Economics 18(6):681-708. Briss, P. A., S. Zaza, M. Pappaioanou, J. Fielding, L. Wright-De Agüero, B. I. Truman, D. P. Hopkins, P. D. Mullen, R. S. Thompson, and S. H. Woolf. 2000. Developing an evidence- based guide to community preventive services: Methods. The Task Force on Community Preventive Services. American Journal of Preventive Medicine 18(1 Suppl):35-43. Buta, B., L. Brewer, D. L. Hamlin, M. W. Palmer, J. Bowie, and A. Gielen. 2011. An innovative faith-based healthy eating program: From class assignment to real-world application of PRECEDE–PROCEED. Health Promotion Practice 12(6):867-875. Carande-Kulis, V. G., M. V. Maciosek, P. A. Briss, S. M. Teutsch, S. Zaza, B. I. Truman, M. L. Messonnier, M. Pappaioanou, J. R. Harris, and J. Fielding. 2000. Methods for system- atic reviews of economic evaluations for the Guide to Community Preventive Services. American Journal of Preventive Medicine 18:75-91.
OCR for page 115
EXISTING FRAMEWORKS 115 Carlson, D., R. Haveman, T. Kaplan, and B. Wolfe. 2011. The benefits and costs of the Section 8 housing subsidy program: A framework and estimates of first year effects. Journal of Policy Analysis and Management 30(2):233-255. CBO (Congressional Budget Office). 2008. Family Smoking Prevention and Tobacco Control Act. Washington, DC: Congressional Budget Office. Cochrane Collaboration. 2012. About us: The Cochrance Collaboration. http://www.cochrane. org/about-us (accessed May 21, 2012). Community Guide. 2011. Community Preventive Services Task Force first annual report to Congress. Atlanta, GA: Community Preventive Services Task Force. Community Guide. 2012a. The Community Guide—systematic review methods. http://www. thecommunityguide.org/about/methods.html (accessed May 22, 2012). Community Guide. 2012b. The Community Preventive Services Task Force. http://www. thecommunityguide.org/about/task-force-members.html (accessed May 22, 2012). Community Guide. 2012c. The guide to community preventive services. http://www. thecommunityguide.org/index.html (accessed May 22, 2012). Community Guide. 2012d. What is the Community Preventive Services Task Force? http:// www.thecommunityguide.org/about/aboutTF.html (accessed May 22, 2012). Department of Health and Aging. 2007. Pharmaceutical Benefits Advisory Committee. http:// www.health.gov.au/internet/main/publishing.nsf/content/health-pbs-general-listing- committee3.htm (accessed May 21, 2012). Donaldson, C. 1998. The (near) equivalence of cost-effectiveness and cost-benefit analyses: Fact or fallacy? Pharmacoeconomics 13(4):389-396. Drummond, M., M. Schulper, G. Torrance, B. Obrien, and G. Stoddart. 2005. Methods for economic evaluation of health care programmes. 3rd Ed. New York: Oxford University Press Inc. Glasgow, R. E., T. M. Vogt, and S. M. Boles. 1999. Evaluating the public health impact of health promotion interventions: The RE-AIM framework. American Journal of Public Health 89(9):1322-1327. Gold, M., J. Siegel, L. Russell, and M. Weinstein. 1996. Cost-effectiveness in health and medicine. New York: Oxford University Press. Green, L. W. 2012a. Bibliographies. http://lgreen.net/bibliog.htm (accessed May 22, 2012). Green, L. W. 2012b. Furthering dissemination and implementation research: The need for more attention to eternal validity. In Dissemination and implementation research in health: Translating science to practice, edited by R. C. Brownson, G. A. Colditz, and E. K. Proctor. New York: Oxford University Press. Pp. 305-326. Green, L. W. 2012c. PRECEDE applications. http://lgreen.net/precede%20apps/preapps-NEW. htm (accessed May 22, 2012). Green, L. W., and R. E. Glasgow. 2006. Evaluating the relevance, generalization, and applica- bility of research. Evaluation and the Health Professions 29(1):126-153. Green, L. W., and M. W. Kreuter. 2005. Health program planning: an educational and ecologi- cal approach, 4th ed. New York: McGraw-Hill. Green, L. W., R. E. Glasgow, D. Atkins, and K. Stange. 2009. Making evidence from research more relevant, useful, and actionable in policy, program planning, and practice: Slips “twixt cup and lip”. American Journal of Preventive Medicine 37(6):S187-S191. Hahn, R. A., J. Lowy, O. Bilukha, S. Snyder, P. Briss, A. Crosby, and P. Corso. 2004. Thera- peutic foster care for the prevention of violence. Morbidity and Mortality Weekly Report 53(RR10):1-8. Harris, R. P., M. Helfand, S. H. Woolf, K. N. Lohr, C. D. Mulrow, S. M. Teutsch, and D. tkins. 2001. Current methods of the U.S. Preventive Services Task Force: A review A of the process. American Journal of Preventive Medicine 20(3 Suppl):21-35.
OCR for page 116
116 ASSESSING THE VALUE OF COMMUNITY-BASED PREVENTION Harvey, I., and M. O’Brien. 2011. Addressing health disparities through patient education: The development of culturally tailored health education materials at Puentes de Salud. Journal of Community Health Nursing 28(4):181-189. Haveman, R., and B. A. Weisbrod. 1977. Public expenditure and policy analysis: An overview. In Public expenditure and policy analysis, edited by R. Haveman and J. Margolis. Skokie, IL: Rand McNally College. Pp. 1-24. Henry, D. A., S. R. Hill, and A. Harris. 2005. Drug prices and value for money. JAMA 294(20):2630-2632. IOM (Institute of Medicine). 2006. Valuing health for regulatory cost-effectiveness analysis. Washington, DC: The National Academies Press. IOM. 2011. For the public’s health: Revitalizing law and policy to meet new challenges. Washington, DC: The National Academies Press. Kanis, J. A., J. Adams, F. Borgström, C. Cooper, B. Jönsson, D. Preedy, P. Selby, and J. Compston. 2008. The cost-effectiveness of alendronate in the management of osteo- porosis. Bone 42(1):4-15. Kemm, J. 2003. Perspectives on health impact assessment. Bulletin of the World Health Or- ganization 81(6):387-387. Kling, J. R. 2011. CBO’s use of evidence in analysis of budget and economic policies. http:// www.cbo.gov/sites/default/files/cbofiles/attachments/11-03-APPAM-Presentation_0.pdf (accessed May 21, 2012). Lomas, J., T. Culyer, C. McCutcheon, L. McAuley, and S. Law. 2005. Conceptualizing and combining evidence for health system guidance. Ottawa, Ontario: Canadian Health Services Research Foundation. London’s Health. 2000. A short guide to health impact assessment: Informing healthy deci- sions. London: NHS Executive London. Nash, C., D. Pearce, and J. Stanley. 1975. An evaluation of cost-benefit analysis criteria. Scot- tish Journal of Political Economy 22(2):121-134. NCI (National Cancer Institute). 2012. RE-AIM publications. http://publications.cancer.gov/ dipubs/reaim.aspx (accessed May 22, 2012). Norris, S. L., P. J. Nichols, C. J. Caspersen, R. E. Glasgow, M. M. Engelgau, L. Jack, S. R. Snyder, V. G. Carande-Kulis, G. Isham, and S. Garfield. 2002. Increasing diabetes self- management education in community settings: A systematic review. American Journal of Preventive Medicine 22(4):39-66. NRC (National Research Council). 2011. Improving health in the United States: The role of health impact assessment. Washington, DC: The National Academies Press. Pew Charitable Trusts. 2009. Overview: Health impact project. http://www.pewtrusts.org/ news_room_detail.aspx?id=55601 (accessed May 22, 2012). RWJF (Robert Wood Johnson Foundation). 2009. RWJF, the Pew Charitable Trusts launch health impact project. http://www.rwjf.org/publichealth/product.jsp?id=50088 (accessed May 22, 2012). Savvides, S. 1994. Risk analysis in investment appraisal. Project Appraisal Journal 9(1):3-18. Snowdon, W., J.-L. Potter, B. Swinburn, J. Schultz, and M. Lawrence. 2010. Prioritizing policy interventions to improve diets? Will it work, can it happen, will it do harm? Health Promotion International 25(1):123-133. Steinbrook, R. 2008. Saying no isn’t nice: The travails of Britain’s National Institute for Health and Clinical Excellence. New England Journal of Medicine 359(19):1977-1981. Stufflebeam, D. L. 1999. Foundational models for 21st Century program evaluation. Kalama- zoo, MI: Evaluation Center, Western Michigan University. Subcommittee on Evaluation Standards. 1958. Proposed practices for economic analysis of river basin projects. Washington, DC: Inter-Agency Committee on Water Resources.
OCR for page 117
EXISTING FRAMEWORKS 117 UCLA HIA-CLIC (University of California, Los Angeles, Health Impact Assessment- Clearinghouse Learning and Information Center). 2012. Phases of HIA: 2. Scoping. http://www.hiaguide.org/methods-resources/methods/phases-hia-2-scoping (accessed May 22, 2012). U.S. Preventive Services Task Force. 2012. Procedure manual (USPSTF). http://www. uspreventiveservicestaskforce.org/uspstf08/methods/procmanual3.htm (accessed May 21, 2012). Watson, M. R., A. M. Horowitz, I. Garcia, and M. T. Canto. 2001. A community participa- tory oral health promotion program in an inner-city Latino community. Journal of Public Health Dentistry 61(1):34-41. Weinstein, M. C., and W. B. Stason. 1977. Foundations of cost-effectiveness analysis for health and medical practices. New England Journal of Medicine 296(13):716-721. Weisbrod, B. A. 1983. A guide to benefit-cost analysis, as seen through a controlled experiment in treating the mentally ill. Journal of Health Politics, Policy and Law 7(4):808-845. Woolf, S. H., C. G. Husten, L. S. Lewin, J. Marks, J. Fielding, and E. Sanchez. 2009. The economic argument for disease prevention: Distinguishing between value and savings. Washington, DC: Partnership for Prevention.
OCR for page 118