Three of the major economic evaluation methods that can be applied to interventions1 serving children, youth, and families identified in Chapter 2 are cost analysis (CA), cost-effectiveness analysis (CEA), and benefit-cost analysis (BCA). These methods can be used to address a number of important questions relevant to decisions about intervention investments. For example, What does it cost to fully implement a given intervention? If an investment is made, what can be expected to be gained in return (e.g., outcomes, dollars, or overall better quality of life)? Is the investment a justifiable use of scarce resources relative to other investments?
Economic evidence generated by these methods can inform investment decisions, but barriers to using this evidence exist. As noted in Chapter 2, some of these barriers relate to the quality of the economic evidence produced. High-quality economic evidence can be difficult to derive because economic evaluation methods are complex and entail many assumptions (Crowley et al., 2014; Lee and Aos, 2011; Vining and Weimer, 2009a). Moreover, methods are often applied inconsistently in different studies, making results difficult to compare and use appropriately in policy and investment contexts (Drummond and Sculpher, 2005; Foster et al., 2007; Institute of Medicine and National Research Council, 2014; Karoly, 2012; Weinstein et al., 1997). Results also may be communicated in a way that obscures important findings or is not suited for nonresearch audiences, or a way in which decision makers may not deem them reliable or compelling
(National Research Council and Institute of Medicine, 2009; Oliver et al., 2014; Pew-MacArthur Results First Initiative, 2013). Shortcomings in these areas may not only limit decision makers’ use of economic evidence but also reduce their demand for such evidence, as well as other types of evidence, in the future.
The primary aim of this chapter is to examine issues associated with the quality of economic evidence, and thus to address the first of this study’s two guiding principles, as described in Chapter 1: quality counts. As noted in Chapter 2, the quality of economic evidence is essential to its utility and ongoing use. Thus, a major goal of this chapter is to help current and would-be producers of economic evidence understand when interventions are ready for economic evaluation and what it takes to produce and report high-quality economic evidence. In several instances, the chapter identifies emerging issues—such as the importance of incorporating the impact of intervention investments on participants’ quality of life—that merit further investigation to determine their applicability to economic evaluation of investments in children, youth, and families.2
In focusing on the quality of economic evidence, the committee drew on the literature and the expertise of its members to identify best practices that can both support high-quality economic evaluation and potentially lead to greater standardization of evaluation methods. Standardization is particularly important because decisions to invest in interventions for children, youth, and families typically involve weighing alternatives in the face of limited budgets; other constraints; and, perhaps, competing values. The use of differing methods to estimate the costs and benefits of alternative investments impedes understanding the economic trade-offs involved and limits the utility of the evidence. At the same time, it is important to recognize the potential disconnect between ideal practice and the real-world analytic issues and constraints that producers of economic evidence encounter. Where possible, this chapter provides strategies for addressing such practical limitations. In addition, the best practices for producing high-quality evidence recommended at the end of the chapter are divided into those that can be viewed as “core” and readily implemented in most circumstances and those the committee characterizes as “advancing,” to be pursued when feasible.
The focus of this chapter extends to highlighting best practices for re-
2 In this chapter, the committee discusses at some length both the strengths and limitations of economic evaluations and the economic evidence produced. The committee recognizes that, based on the current state of the field, there is no perfect solution for every issue that is discussed herein. Although economic evidence has its limitations, the hope is that stakeholders, to the extent possible, follow good practices, are transparent about these practices, and understand—whether they are producers or consumers—what can and cannot be derived from economic evaluations.
porting the results of economic evaluations in a consistent and transparent manner. Findings need to be communicated in ways that facilitate understanding, acknowledge limitations, and support their appropriate use in investment decisions. Achieving such transparency and utility is not a small task given the complexity, multiple assumptions, and various sources of uncertainty entailed in the use of economic evaluation methods. Nonetheless, the chapter offers guidelines that in the committee’s view can enhance the utility and use of economic evidence while maintaining scientific rigor.
It should be noted that a well-established literature on best practices in the conduct and reporting of CAs provided a solid foundation for the CA-related conclusions and recommendations offered in this chapter. Best practices in CEA in health and medicine, initially established in 1996 (Gold et al., 1996), are currently under review by the 2nd Panel on Cost-Effectiveness Analysis in Health and Medicine.3 In addition to the best practices pertinent to CEA identified in this chapter, interested readers are encouraged to turn to this panel’s recommendations when they are available. Best practices in the application of BCA to investments in children, youth, and families have just begun to appear in the literature, so the committee’s conclusions and recommendations on this method are based on the consensus view of the committee members, incorporating perspectives from the available literature and papers and panels sponsored for this study.
Finally, although much of this chapter is directed at producers of economic evidence, its content should also be of interest to consumers of the evidence. Consumers can benefit from understanding the analytic issues associated with planning for and conducting economic evaluations, the best practices for the production and reporting of economic evidence, and the limitations of economic evaluation methods. Similarly, producers of the evidence would benefit from understanding the issues raised in Chapter 4, which deals with how consumers use the economic evidence they receive, even if it is of the highest quality, and the context in which investment decisions are made.
The first two sections of this chapter outline issues pertinent to all types of economic evaluation: determining whether an intervention is ready for economic evaluation and defining the scope of the evaluation. Next is a discussion of issues specific to evaluating intervention cost (relevant to CA), and by extension, CEA and BCA, determining intervention impacts (relevant to CEA and BCA), and valuing outcomes (relevant particularly to BCA and related methods). Sections then follow on the development and reporting of summary measures for the results of CA, CEA, and BCA; how the uncertainty intrinsic to economic evaluations can be handled; and how equity
considerations can be addressed. The chapter closes with the committee’s recommendations regarding best practices for producing and reporting high-quality economic evidence.
As discussed in Chapter 2, economic evaluation encompasses an array of methods used to answer questions about the economic value of the resources required to implement an intervention, alone or with reference to the intervention’s impact, measured in terms of the outcomes affected or the economic value of those outcomes. Determining whether an intervention is ready for economic evaluation and if so, which evaluation method to use, depends on the question(s) of interest and the information available. This section highlights the requirements for undertaking a high-quality economic evaluation, beginning with the most general requirements and then focusing on those that are specific to different economic evaluation methods. Figure 3-1 provides a decision tree used to guide the discussion.
Intervention Specificity, Counterfactual, and Other Contextual Features
For all types of economic evaluation, whether ex post or ex ante, two essential requirements are that the intervention be clearly defined and the counterfactual condition be well specified (Figure 3-1).
Intervention specificity means that the intervention’s specific purpose, intended recipients, approach to implementation, causal mechanisms, and intended impact can be described in sufficient detail. For an ex post analysis, this specificity means that others can replicate the intervention or apply it in new settings or with new populations (Calculating the Costs of Child Welfare Services Workgroup, 2013; Foster et al., 2007; Gottfredson et al., 2015). For an ex ante analysis, it means that consumers of the analysis understand the nature of the intervention being analyzed.
In the context of an ex post analysis, a logic model describing the intervention’s theory of change, or mechanisms by which its impact is achieved, is useful in establishing specificity, as are written curricula, manuals, detailed policy plans, and other documents outlining how the intervention is to be implemented and how staff implementing it are to be trained and supported in carrying it out effectively. Many interventions meet this requirement and have published manuals and logic models or explicit theories of change (Gill et al., 2014; Hawkins et al., 2014; Hibbs et al., 1997; Smith et al., 2006). Guidelines for developing logic models where they do not exist are also readily available (Centers for Disease Control and Prevention, 2010; W.K. Kellogg Foundation, 2004).
NOTES: This decision tree highlights the major types of economic evaluation based on how the estimate will inform the intervention investment and the available information. BCA = benefit-cost analysis, BIA = budgetary impact analysis, CA = cost analysis, CEA = cost-effectiveness analysis, DALY = disability-adjusted life year, QALY = quality-adjusted life year, ROI = return on investment.
*Payers may include employers, government, the health care system, or recipients of the intervention.
The second key requirement is defining the counterfactual, the alternative with which the intervention is being compared in the economic evaluation, whether the evaluation is an ex post or ex ante CA, CEA, or BCA. In the context of program evaluation, this is usually referred to as the control, status quo or baseline, or comparison condition. The counterfactual condition may be no intervention, the status quo, or business as usual (e.g., an existing intervention), or it may be a less intensive version of the intervention of interest. For example, a school-based teen pregnancy prevention intervention might be evaluated in a community where there was no current intervention, where there was an existing intervention (school- or community-based), or where there was an intervention that provided information materials only but no other services. Defining the counterfactual is key, as a CA will be based on measuring the resources used to implement the intervention relative to the counterfactual condition. If a CEA or BCA is to be performed, intervention impacts should be measured relative to the same counterfactual condition as that used for the CA.
Clarifying other aspects of the context in which the intervention has been or will be carried out is necessary for interpreting the results of economic evaluation. Additional contextual details—such as the sociodemographic characteristics of the population targeted and served; the time, place, and scale of implementation; and other elements detailed in Consolidated Standards of Reporting Trials (CONSORT) guidelines (Schulz et al., 2010)—can also aid interpretation, help consumers understand the circumstances under which economic evidence is likely to apply, and guide appropriate use of the evidence in decision making. Without a clear understanding of the base case, the counterfactual, and other contextual factors—“what is delivered for whom, under what conditions, and relative to what alternative”—interpretation of the results of economic evaluation will be muddy.
Other Requirements for Economic Evaluation
Provided that an intervention is well defined and the counterfactual and other contextual factors can be specified, a CA can be performed to understand the economic cost of the resources required for implementation or to provide the foundation for a CEA or BCA. As discussed later in this chapter, conducting a CA, CEA, or BCA requires estimates of the resources used in intervention implementation and the economic values to attach to those resources (Figure 3-1). Later in the chapter, in the discussion of CA as a stand-alone analysis or as a component of CEA or BCA, best practices for measuring the resources used and their values are reviewed.
When investors have more complex questions than cost, such as which interventions are expected to yield the greatest impact for a given invest-
ment or which investments are likely to generate positive returns, evidence of intervention impact also is needed so that CEA or BCA can be performed (Figure 3-1) (Jamison et al., 2006; Lee and Aos, 2011; Levin and McEwan, 2001). Issues related to the nature of the evidence of impact are considered later in this chapter. When CA is possible but evidence of intervention impact is not available, Figure 3-1 shows that a break-even analysis can be performed to determine how large impacts would need to be for an intervention to be deemed cost-effective or cost-beneficial, provided that potential intervention impacts can be monetized. When intervention impacts are available and the impacts can be monetized, Figure 3-1 indicates that a BCA can be conducted; otherwise, a CEA is a feasible alternative.
CONCLUSION: Key requirements for all types of economic evaluation are that the intervention can be clearly defined, the counterfactual well specified, and other contextual features delineated. To conduct cost analysis, cost-effectiveness analysis (CEA), and benefit-cost analysis (BCA), information on the resources used to implement the intervention is required. For CEA or BCA, credible evidence of impact also is needed.
Once it has been determined that an economic evaluation is feasible, an essential next step is to define key elements of the evaluation’s scope. These include the perspective for the analysis, the time horizon and discount rate, and several other analytic features.
The perspective for an economic evaluation is determined by the question(s) to be answered and the audience(s) for the analysis (Figure 3-1). The broadest perspective is the societal perspective, which captures the public and private sectors and includes individuals who may be the focus of the intervention, as well as those who may be affected only indirectly. CA, CEA, and BCA all can be conducted from a societal perspective, with all costs being captured regardless of who bears them, and the economic values associated with all outcomes accounting for all who gain or lose. An economic evaluation conducted from the societal perspective can be disaggregated to consider the results from the perspective of specific stakeholder groups: the individuals who are targeted or served by the intervention; other individuals in society who are not targeted or served by the intervention; and the public sector at all levels of government combined or further disaggregated to consider the federal, state, and local levels separately or
even different agencies at a given level. The public sector can also be viewed as representing the costs and benefits borne by individuals as taxpayers. Providing this detail is particularly useful in showing how costs and benefits of an intervention are distributed to various interested parties. For example, an intervention with a small but positive net benefit could mask losses to participants that were offset by public sector savings. Though a favorable investment overall, an intervention with such a distribution of costs and benefits may not be appealing to investors valuing gains to participants over government savings. Further discussion of perspectives is included later in this chapter in the section on best practices for conducting cost analyses.
For some economic evaluations, the primary focus may reflect mainly or solely a government perspective, which is just one component of the societal perspective. As noted in Chapter 2, cost-savings analysis is a BCA from the government perspective (Figure 3-1). The government perspective may be even more narrowly focused, such as for a specific government agency or level of government (e.g., federal, state, or local). Economic evaluations also can be conducted from the private perspective of a specific stakeholder, such as a business, philanthropy, or private investor. When an analysis is conducted for a specific stakeholder, its conclusions will reflect that stakeholder’s perspective but may fail to capture the full range of costs and benefits of the intervention being analyzed, and the excluded costs and benefits may be substantial. For example, focusing on the government perspective may fail to provide important information about how an intervention impacts the participants involved, in terms of costs borne or gains received from participating in the intervention. In contrast, the advantage of the societal perspective is that in the ideal, it provides a comprehensive accounting of all costs and benefits.
At the same time, when the societal perspective is adopted, it is important to disaggregate societal costs and benefits into those that accrue to the private sector (e.g., to intervention participants and other members of society) and those that accrue to the public sector (e.g., to the government as a whole or subdivisions of the public sector). The justification for public-sector investments in children, youth, and families is strongest when there are positive net benefits to the public sector and the rest of society, in addition to any private returns to the individual participants. Private individuals may underinvest in those areas (e.g., health, education) where the private returns are less than the social returns (i.e., there are also returns to the public sector or other members of society). Conversely, if the only returns to an investment are private, there is little justification for a public-sector investment.4
4 Disaggregating can show how costs, benefits, and net benefits are distributed to different stakeholders, both in total as well as over the time period of interest (e.g., annual costs and
Time Horizon and Discounting
One feature of investments in children, youth, and families is that an intervention may take place over multiple years, and its impact may extend over long periods of time, sometimes covering the entire life course and even affecting future generations. For this reason, it is important to define the time horizon that will be applied to the economic evaluation and whether the stream of future values associated with the resources used to implement the intervention and the outcomes that result will be discounted.
Establishing the time horizon is relatively straightforward. At a minimum, the time horizon will typically include the period over which the intervention is implemented. For discrete interventions, such as an early childhood or youth development program, implementation will occur over a fixed number of years. Depending on the length of the follow-up period, outcomes may be observed only during the intervention period, or they may extend further into the future if participants are followed after the intervention ends. There may also be interest in projecting outcomes beyond the period when outcomes were last measured. Such projections may extend over an individual’s lifetime or even to future generations. As discussed later in the chapter, such projections introduce additional uncertainty into the results of an economic evaluation.
The issue of discounting arises because economists often assume that individuals and society place a higher value on costs and outcomes occurring in the present than on those that will occur in the future. Two common arguments to justify this assumption are (1) that money and other resources available today can be invested or used in some way to enjoy more benefits later on than would be realized if the same resources were available only in the future, and (2) that having those resources today eliminates any uncertainty of having them in the future (Miller and Hendrie, 2013).
Discounting is a technique used in economic evaluations to adjust costs and outcomes to account for this premium placed on benefits accrued closer to the present. For economic evaluations focused on children, youth, and families, the social discount rate is appropriate (Boardman and Greenberg, 1998). The standard approach to discounting is analogous to the process of compounding interest: a stream of costs or outcomes is reduced to its present value by applying a compounded discount rate to future streams.
benefits to each stakeholder from intervention’s start through year 5). Stakeholders included in the analysis should be meaningful to the program and/or policy question. For example, costs and benefits to participants and to taxpayers who may finance and also benefit from the intervention are often included in stakeholder analysis, and there may be additional groups to incorporate as well. It is important to align the stakeholder groups on the cost and benefit sides of the analysis. That is, both the costs and benefits to each stakeholder group of interest should be estimated.
Because higher discount rates lead to higher valuation of outcomes occurring in the present relative to those occurring in the future, the discount rate is a key choice in economic evaluation, especially for interventions with significant impacts over long periods of time. The discount rate used in studies reflects the value of a dollar today versus that of a dollar tomorrow at a particular margin, determined largely by what return is required to attract the last dollar of saving. A discount rate also may vary by whether one uses a risky or riskless return. Discount rates used in economic evaluation have varied widely, although recommendations in recent years appear to be settling in the range of 3-7 percent (Drummond et al., 2005; Gold et al., 1996; Haddix et al., 2003; Hunink et al., 2001; Office of Management and Budget, 2003; Washington State Institute for Public Policy, 2015). A later section of this chapter describes the practice of using a base discount rate and then assessing the sensitivity of the results of the economic evaluation using a range of alternative discount rates.
Although there is little disagreement on the validity of discounting intervention costs, more controversy is associated with the issue of whether other outcomes—health in particular—should be discounted, and at what rate. One argument in favor of discounting health outcomes focuses on uncertainty: individuals would prefer to postpone illness because (1) they may not even be alive in the future, and (2) future medical progress could reduce the negative effects of the same illness occurring today (Miller and Hendrie, 2012). Some argue, moreover, that health outcomes should be discounted at the same rate as costs to avoid a paradox that arises when health is discounted at a lower rate than costs: the economic performance of an intervention may sometimes be improved by delaying its implementation, since the same health benefits could be achieved at a lower (discounted) cost simply by waiting (Gold et al., 1996; Keeler and Cretin, 1983; Weinstein and Stason, 1977). As noted by Drummond and colleagues (2005, p. 111), most guidelines for economic evaluation of health interventions, including those of the U.S. Panel on Cost-Effectiveness in Health and Medicine and the World Health Organization, recommend discounting of both costs and health outcomes using the same rate. For example, if an obesity prevention intervention is conducted when children are 10 years old and the expected impacts are expected to reduce the probability of cardiovascular disease (CVD) in the 50th future year, then the benefits of the intervention—either in natural units in a CEA, in cases of CVD avoided, or in monetary benefits in a BCA—are valued at benefit/(1 + discount rate)^50. At a 10-percent discount rate, $1,000 spent today would be worth $9; at a 3 percent rate, $228. Note that future health costs and outcome benefits are discounted at the same rate as the cost saving and outcome benefit. For obesity prevention, for example, costs and benefits will be the same for people of different ages.
With investments in health interventions for children, of course, most benefits accrue over time, as in better educational and then work outcomes due to better health. It is the accumulation of all those benefits over time that is typically to be compared with current costs. For many studies, the danger is more that the future benefits simply are not estimated or cannot easily be estimated—less an issue than that long-term benefits are discounted too heavily.
A related and more complex ethical issue arises from discounting any intervention with impacts that affect future generations, or comparing benefits for a younger generation with costs to an older one. Miller and Hendrie (2013, pp. 356-357) give the example of a hypothetical environmental regulation targeting global climate change, which could affect outcomes of today’s children and of children centuries into the future. Even using a low discount rate, outcomes just a century away would have present values so low that an economic evaluation would likely favor investments that would avoid even small sacrifices in the present, at the cost of potentially significant harm for future generations.
Part of the complication here is that discount rates assume investments apply at the margin, so that an extra benefit (valued in dollar terms) may be worth less to a future generation, expected to be richer, than to the current one. But if comparisons are made with respect to the value of a life today versus a life tomorrow, then the implicit assumption that the calculation applies at the margin no longer obtains. Put another way, there is no case for valuing a life tomorrow less than a life today, even if an extra lifetime dollar is worth more to the older of two generations.
Solutions suggested for avoiding this problem include starting the “discounting clock” when those affected are born, using a zero discount rate, and eliminating the need for discounting by assuming an explicit social utility function (Cowen and Parfit, 1992; Miller and Hendrie, 2013; Schelling, 1995). A broader suggestion, acknowledging that there is no satisfactory solution to this issue, is to consider moral obligations to future generations separately from the question of discounting practice (Institute of Medicine, 2006).
Finally, Karoly (2012) highlights an issue especially relevant to early childhood. Early programs can start at various child ages, from before birth up to age 5. If discounting originates at the age a program starts, some studies will discount to birth, while others will discount to as late as age 4. In such cases, present-value estimates will not be comparable across studies. The same concern applies in comparing interventions at other stages of
development. Unless interventions are discounted to the same age, present-value estimates will not be comparable.5
Other Analytic Features
Two other analytic features to determine at the outset of an economic evaluation are (1) the monetary unit and year in which all economic values will be denominated, and (2) whether to account for the deadweight cost of taxation.6 For the United States, economic evaluations typically use dollars as the currency measure, but any currency is feasible provided resources used and the value of intervention outcomes can be denominated in that currency. To adjust for changes in prices over time, economic evaluations measure the opportunity cost of resources and the economic value of outcomes in inflation-free monetary units, using a base year as reference. Thus, prices of resources used or outcome values before the base year are inflated using changes in relevant price indices (e.g., the consumer price index or employment cost index in the United States), and prices of resources used or outcome values in the future are held constant at the base year levels. The year of measurement may be specific to the point in time at which costs and outcomes were measured, or monetary values may be inflated to a more recent year so that findings can be expressed in current monetary values. As discussed later in this chapter in the section on reporting, the year for which monetary units are valued—whether intervention costs or the value of intervention outcomes—needs to be clearly stated.
When interventions for children, youth, and families involve new taxes for financing the intervention or produce impacts that affect taxes (e.g., an increase in taxes because of higher earnings or a reduction in welfare payments because of reduced welfare participation), there is a corresponding change in the deadweight cost associated with the distortionary effects of taxes on economic behavior and the costs associated with administering the tax—i.e., the dollars of welfare loss per tax dollar (Vining and Weimer, 2010). Producers of economic evidence may account for this deadweight
5 Maynard and Hoffman (2008) highlight another approach in their analysis of teen pregnancy prevention: assuming that an intervention had been fully implemented (from birth to adulthood for everyone) and then providing a steady-state analysis.
6 Every dollar of government revenue raised through taxes typically costs society more than one dollar in resources because taxes induce changes in behavior (e.g., reduced work effort) that represent an opportunity cost to society and because of the administrative costs of tax collection. The deadweight loss (also known as excess burden) measures those costs and is usually expressed as a percentage of the revenue raised. Although the costs of administering government-transfer payment programs conceptually can be viewed as a deadweight loss, such changes in administrative costs are best handled as costs or benefits in the cost-outcome equation.
cost of taxation as an additional cost when taxes are increased to pay for an intervention or when taxes rise as a result of an intervention. Conversely, the deadweight loss is reduced when the intervention produces a reduction in taxes. While economic evaluations often assume no deadweight loss, a few recent evaluations have produced results assuming different levels of deadweight loss as part of a sensitivity analysis (e.g., as in Heckman et al.  and Washington State Institute for Public Policy , in which deadweight costs are assumed to be 0 percent, 50 percent, and 100 percent).
CONCLUSION: Once an intervention has been determined to be ready for an economic evaluation, an essential next step entails establishing the perspective; the time horizon for capturing costs (all types of analyses) and outcomes (cost-effectiveness analysis and benefit-cost analysis); the baseline discount rate; the monetary unit and reference year; and the assumed magnitude of the deadweight loss parameter, if deadweight loss will be evaluated.
A systematic CA gives stakeholders important insight into the operation of interventions that impact children, youth, and families, including the overall cost of implementing and sustaining an intervention, costs for specific intervention activities, and costs per intervention participant (Crowley et al., 2012; Foster et al., 2007; Haddix et al., 2003). Beyond assessing actual costs, a CA may serve to facilitate planning, maximizing the efficiency of resource use, replication, dissemination, and implementation of efficacious and effective interventions. Chapter 2 describes the place of CA within evaluation and economic evaluation frameworks. CA relies on information about an intervention’s implementation, such as the specific programmatic activities, the types and quantities of resources used in delivering intervention services, the number and characteristics of providers delivering and individuals or families receiving services, and the intensity or dosage of services provided. This information on intervention inputs is also the focus of process evaluation, which answers the questions of “what is done,” “when,” “by whom,” and “to whom.”
In addition, as discussed in Chapter 2, CA establishes the foundation for other types of economic evaluation, such as CEA and BCA. As detailed later in this chapter, CEA examines the relationship between an intervention’s costs and a relevant unit of intervention effectiveness, while BCA quantifies intervention benefits in monetary terms and assesses whether they exceed intervention costs. The precision of these analyses depends, in part, on accurate analysis of intervention costs.
When a consistent and accurate approach is used to collect and ana-
lyze cost data, CA also can support comparisons of costs across services, interventions, and agencies. Increasingly, federal agencies require that evaluations of the interventions they fund include cost analyses. For example, a number of program announcements of the Administration for Children and Families requires that applicants propose a reasonable cost evaluation design that (1) allows for analyses of personnel and nonpersonnel resources among cost categories and program activities, (2) allows for analyses of direct services and of management and administrative activities, (3) includes both case-level and aggregate data that can reasonably be obtained and tracked, and (4) identifies anticipated and potential strategies for addressing these issues.
Similarly, at the Department of Education, Office of Innovation and Improvement, applicants for Investing in Innovation funding are required to provide detailed information about how they will evaluate whether their proposed projects are cost-effective when implemented.7 This evaluation may include assessing the cost of comparable or alternative approaches. To receive competitive preference points, applicants addressing this priority must provide a detailed budget, an examination of different types of costs, and a plan for monitoring and evaluating cost savings, all of which are essential to improving productivity.
Best Practices for Conducting Cost Analyses
The goal of a CA is to quantify the full economic value of the resources required to implement an intervention relative to the status quo or control condition. The characteristics of a high-quality CA necessarily include (1) defining the purpose and scope of the analysis, (2) defining the intervention, (3) providing comprehensive and valid cost estimates, (4) applying widely accepted best practices in the field, and (5) acknowledging the limitations of the analysis. The discussion of best practices in this section draws on a review and synthesis of guidelines for conducting CA in the literature. In so doing, it provides additional support for practices discussed earlier in the chapter that are relevant to economic evaluation methods in general, such as defining the purpose and scope of the analysis and the intervention to be analyzed. This section addresses these issues specifically in the context of CA and the best practices identified in the literature.
Defining the Purpose and Scope of a Cost Analysis
According to the U.S. Children’s Bureau guide for assessing the costs of child welfare programs (Calculating the Costs of Child Welfare Services Workgroup, 2013), internal and external stakeholders should be engaged prior to the CA to (1) clarify the goals and audience for the analysis, (2) clearly define the intervention to be analyzed, and (3) specify the time period to be covered. The goals of the study help define who needs the CA (audience) and the intended uses of its results. This information in turn determines the perspective for the analysis, dictating which cost categories to consider. The perspective selected for the study guides all subsequent decisions around how best to estimate intervention costs. Many guidelines in the existing literature do not offer recommendations for a specific study perspective, but rather state that it should arise from the interests of the stakeholders or audience for the analysis and/or the research question (Detsky and Naglie, 1990; Drummond and Jefferson, 1996; European Commission, 2008; Graf von der Schulenburg and Hoffman, 2000; Hjelmgren et al., 2001; Honeycutt et al., 2006; Task Force on Community Preventive Services, 2005; Vincent et al., 2000). If a study perspective is recommended, it is most commonly the societal perspective (Barnett, 2009; Capri et al., 2001; Haddix et al., 2003; Honeycutt et al., 2006, Graf von der Schulenburg and Hoffmann, 2000; Hjelmgren et al., 2001; Laupacis et al., 1992; Luce et al., 1996; Ontario Ministry of Health and Long-Term Care, 1994; Pritchard and Sculpher, 2000; Suter, 2010; Task Force on Community Preventive Services, 2005; World Health Organization, 2012). Further, guidelines state that economists prefer the societal perspective (Chatterji et al., 2001; Drummond and Jefferson, 1996; Gray et al., 2010), and almost always recommend this perspective for BCAs (Calculating the Costs of Child Welfare Services Workgroup, 2013; Commonwealth of Australia, 2006; European Regional Development Fund, 2013; Treasury Board of Canada Secretariat, 2007; World Health Organization, 2006).
When the societal perspective is used to guide the CA, additional information is often gained by disaggregating overall costs into subperspectives showing how costs are borne by various stakeholders. Subperspectives may reflect the potential investors in an intervention (agencies, private organizations, taxpayers) or those impacted by the intervention (e.g., participants, potential victims). For CEAs of interventions provided by the health care sector, for example, several guidelines additionally recommend a health system or payer perspective (Academy of Managed Care Pharmacy, 2012; Graf von der Schulenburg and Hoffmann, 2000; Haute Autorité de Santé, 2012; Hjelmgren et al., 2001; Institute for Quality and Efficiency in Health Care, 2009; Marshall and Hux, 2009; National Institute for Health and Care Excellence, 2013; Walker, 2001). If the full societal costs of an inter-
vention are not estimated, however, the subperspective may provide only a partial picture of the value of all resources required to implement an intervention. Indeed, multiple perspectives for an analysis are often preferred and expected. From a provider perspective, for example, the costs of an intervention may equate to actual monetary expenditures. From a societal perspective, however, the value of all resources required to implement an intervention is included in the analysis regardless of to whom they accrue, so that, for instance, costs would include in-kind donations in addition to monetary expenditures. They might also include the cost to participants of spending their time on program activities instead of alternatives, such as work or leisure.
Defining the Intervention
Defining the intervention to be delivered is another critical step in the analysis that needs to include stakeholders who know the intervention model well. Many options exist for analyzing intervention costs as part of broader evaluation efforts (Yates, 2009), and the collection and analysis of cost data are more likely to be successful if included in evaluation planning from the outset. Logic models are a convenient evaluation tool that can help delineate intervention inputs with a bearing on the CA.
Specifying the time period over which cost data will be collected is also important (Brodowski and Filene, 2009). CAs may cover a time horizon of several years to provide information on how costs vary over time, or they may focus on a single year that is considered to be representative of the intervention’s typical operating state. Evaluators also need to specify the intervention’s stage of implementation during the CA because costs are likely to differ between a startup or planning period (preimplementation) and a period of steady-state implementation, when the intervention is operating at or near full capacity (Miller and Hendrie, 2015). The potential existence of economies of scale implies that differences in output level need to be taken into account in comparing operating efficiency across intervention sites, and cost projections may be inaccurate if they fail to take into account the decrease in average cost that occurs as output expands (Mansley et al., 2002).
CONCLUSION: The societal perspective is the most commonly recommended perspective for researchers conducting cost analysis (CA). Subperspectives can be used to tailor cost estimates to specific audiences, but do not necessarily provide a comprehensive estimate of costs and may be inadequate for supporting intervention replication. In addition, CA requires carefully defining the intervention and identifying any of its activities that consume its resources. Best practice further requires that the time horizon for the CA be clearly defined.
Providing Comprehensive and Valid Cost Estimates
Developing accurate estimates of the cost of an intervention for children, youth, and families requires carefully quantifying and valuing the resource needs to replicate intervention effects. There are a number of methods for costing an intervention (Barnett, 2009; Calculating the Costs of Child Welfare Services Workgroup, 2013; Gray et al., 2010; Haddix et al., 2003; Honeycutt et al., 2006; Luce et al., 1996; Muenning and Khan, 2002; Yates, 1996). These methods represent one of two general approaches. The first is a macro, top-down approach that uses total public spending (or individual site budget or expenditure) data to provide gross average estimates of intervention costs.8 The other is a bottom-up approach known as micro costing that relies on identifying all resources required to implement an intervention and then valuing those resources in monetary units to estimate intervention costs. The methods used for micro costing—ingredients- and activity-based allocation—are generally considered the methods of choice because, relative to the macro approach, they are more accurate and provide investors with greater detail about intervention costs so that resource needs for success can be projected. This detail includes robust estimates of the marginal and steady-state (average) costs of the intervention (see the section later in this chapter on “Getting to Results” for additional information on summary measures for CA), which allow for estimation of the intervention’s per-unit cost (e.g., per family or child served). However, micro costing can be more difficult and time-consuming to implement than other costing methods (Levin and Belfield, 2013), requiring that an infrastructure be in place with which to collect data on resource use at the unit level.9
To fully understand resource needs and to ensure that all stages of implementation are covered, combining logic models with the micro costing approach is a good solution to avoid “hidden” costs (e.g., for adoption, development, training, technical assistance, and sustainability). Hidden
8 Budgetary information can be a useful data source for conducting cost analyses, but estimating the economic costs of interventions requires more than a simple accounting of budgetary expenditures. Specifically, while budgets can be used to estimate the quantity of some resources consumed to implement an intervention, it should not be assumed that they reflect all the resources needed to adopt, implement, and sustain an intervention. Further, the price information that can be extracted from a budget may be representative only of local market prices. Adjustments may be needed to estimate intervention costs in new settings or for national dissemination.
9 Many resources are available that can provide comprehensive and field-specific listings of types of costs. For instance, the new Costing-Out tool from Columbia’s Center for BenefitCost Analyses of Education can be used for interventions delivered in educational settings; the Drug Abuse Treatment Cost Analysis Program (DATCAP) can be used for interventions for children, youth, and families delivered in social service and clinical settings; and the Children’s Bureau offers a free Guide for Child Welfare Researchers and Service Providers (Cost Analysis in Program Evaluation).
costs of an intervention also may include resources required beyond the intervention to ensure full implementation. A CA conducted from the societal perspective, for example, may need to include the value of systems-level resources required for implementation, beyond those resources required only at the local level. Considering implementation costs is especially important when comparing differing approaches to intervention. For example, implementation costs are quite different for passing an underage drinking law and issuing regulations to implement it and for adopting a school-based alcohol education program.
Cost Categories Typical cost categories for consideration in micro costing are personnel, space, materials, and supplies. The categorization of costs may be strengthened by consideration of these major cost categories within specified activities associated with an intervention. It may be helpful, for example, to consider an intervention’s costs within the broad categories of the preimplementation and implementation phases of intervention delivery, or startup versus ongoing maintenance costs. It may also be useful to consider direct versus indirect costs. Direct costs may refer to those resources required to provide services directly to participants, such as classroom time for a bullying prevention curriculum or home visits to prevent child maltreatment. When estimating costs at the level of the unit of the participant, a CA may need to allocate more resources to the estimation of direct personnel time (Yates, 1996). Indirect costs typically denote overhead costs related to administrative functions of an intervention or to services not provided directly to but on behalf of the participants. Often these costs are shared by more than one intervention or used to create more than one output, or may be defined as expenses that directly benefit the agency (American Humane Association, 2009; Calculating the Costs of Child Welfare Services Workgroup, 2013; Capri et al., 2001; Chatterji et al., 2001; Cisler et al., 1998; Derzon et al., 2005; European Regional Development Fund, 2013; Federal Accounting Standards Advisory Board, 2014; Foster et al., 2003; Graf von der Schulenburg and Hoffmann, 2000; Haute Autorité de Santé, 2012; Institute for Quality and Efficiency in Health Care, 2009; Leonard, 2009; National Center for Environmental Economics, 2010; Pritchard and Sculpher, 2000; Suter, 2010; Task Force on Community Preventive Services, 2005; Treasury Board of Canada Secretariat, 2007).
The consensus in the literature is that analysts should include program and administrative or overhead costs for programmatic CAs (Barnett, 2009; Calculating the Costs of Child Welfare Services Workgroup, 2013; Greenberg and Appenzeller, 1998; Her Majesty’s Treasury, 2003; Office of Management and Budget, 2004a, 2004b; World Health Organization, 2012). Indirect costs can be distributed using the proportion of time spent in direct delivery of each service. Once the fraction of time devoted by each
staff member to various activities is known, this information can readily be monetized by multiplying the fractions by the staff members’ compensation (salaries and other benefits) over an appropriate time period, such as 1 year (Greenberg and Appenzeller, 1998). Alternatively, the total annual expenditures on each indirect cost (e.g., support staff salaries, supervisors’ salaries, computers, rental space, telephone, electricity, water, maintenance) can be multiplied by the fraction of the organization’s total staff costs devoted to each activity (Greenberg and Appenzeller, 1998; World Health Organization, 2012). However, it is important to assume that not all management tasks are indirect costs (Calculating the Costs of Child Welfare Services Workgroup, 2013); some are directly related to an intervention, and managers may be able to estimate the amount of time they spend on such tasks.
Another important consideration in CA is fixed versus variable costs, particularly when the evaluator is interested in an intervention’s marginal and steady-state (average) costs. Fixed costs are the value of those resources required only occasionally for the intervention, which do not vary with the number of participants served. Typical fixed costs—such as costs to train providers and to buy furniture—occur in the preimplementation phase of an intervention. By annualizing costs of capital equipment over their useful life, it is possible to allocate a fair portion of those costs to each person served. Variable costs are the value of those resources required for each person served by the intervention. Table 3-1 (Ritzwoller et al., 2009) shows a typical valuation of fixed and variable costs in a CA.
Unit Prices The most important determinant of the comprehensiveness of a CA is how well the resources required to implement an intervention are inventoried and then valued. That is, the costing of an intervention is really a function of resources (Q) and their prices (P). But what unit prices should be used? Budget sheets that show intervention expenditures for a given fiscal year include similar resource categories and are often a convenient, but perhaps incomplete, way to value the resources. Stakeholders also may play a role in determining the appropriate unit prices to use, based on the audience for the analysis. If a national intervention is being valued, for example, unit prices may need to reflect national averages for such costs as wage rates, space rental, and supply purchases. Local interventions may need to rely on local unit prices. Either way, transparency of unit prices is critical for replicability of a CA across sites. Moreover, within any analysis, the use of a consistent set of prices (e.g., state, local, federal) and a common reference year is important.
Nearly all recommendations for conducting CAs suggest that resources be valued by their opportunity cost. Often, the market price for a resource is a good approximation for its opportunity cost. However, when a market price does not exist or is suspected not to reflect the opportunity cost, one
|Cost Element||Variable ($)||Fixed ($)||Total ($)|
|Tailored news letters||10,102||10,102|
|Interviewers training and supervision||23,865||23,865|
|Phone counseling/data management||11,872||11,872|
|Project meetings and e-mail||5,667||5,667|
|Equipment and materials||2,890||2,890|
|Total Recruitment plus 3-Month Intervention||112,848|
aOverhead includes office tasks, such as printing, copy making, unscheduled staff meetings, phone conversations, intervention preparation time, commute to the intervention site where calls are made and newsletters are produced, etc.
SOURCE: Example from Ritzwoller et al. (2009), reprinted with permission.
method for valuing the resource is to use a shadow price (Commonwealth of Australia, 2006; European Commission, 2008; Gray et al., 2010; Joint United Nations Programme on HIV/AIDS, 2000; The World Bank, 2010; Treasury Board of Canada Secretariat, 2007; Walter and Zehetmayr, 2006; World Health Organization, 2006, 2012). Examples of the use of shadow prices are the shadow wage rate for adjusting labor prices to account for distortions in the labor market and the shadow price of capital, which is used to adjust the valuation of costs for the effects of government projects on resource allocation in the private sector (European Commission, 2008; National Center for Environmental Economics, 2010; Office of Management and Budget, 1992; World Health Organization, 2006). Examples of the shadow price of wages include the value of the time friends or family spend providing unpaid care (Gray et al., 2010) and the value of volunteer
time, which are based on the wage rate for someone carrying out similar work (World Health Organization, 2006). An example of the shadow price of capital is the use of the price of comparable private-sector land for the price of government-owned land (Commonwealth of Australia, 2006).
Sensitivity Analyses A recent and notable addition to the list of steps for conducting a cost evaluation (whether CA or some other method), from the Children’s Bureau10 and others (Haddix et al., 2003; Yates, 2009), is to conduct sensitivity analysis and examine cost variation (Corso et al., 2013; Crowley et al., 2012). The consensus in the literature is that sensitivity analysis should be performed whenever estimates, data, or outcomes are uncertain. It is accepted that providing the results of sensitivity analysis when reporting the results of cost analysis is best practice, both internationally and domestically (Benefit-Cost Analysis Center;11Hjelmgren et al., 2001; Levin and McEwan, 2001; Luce et al., 1996; Marshall and Hux, 2009; Messonnier and Meltzer, 2003; Office of Management and Budget, 1992; Pharmaceutical Benefits Board, 2003; Ramsey et al., 2005; Siegel et al., 1996; Walker, 2001; World Health Organization 2000). Recommendations on sensitivity analyses are usually generic and often are centered on the discussion of discount rates. In some instances, however, especially in international contexts, particular methods are specified (Hjelmgren et al., 2001; Marshall and Hux, 2009; Walker, 2001); Canada, for example, encourages the use of Monte Carlo simulations (Walker, 2001). Further discussion of sensitivity analysis is provided later in this chapter.
CONCLUSION: According to best practices, after establishing the perspective, defining the intervention, and specifying the base year and time period over which the intervention will be assessed, cost analysis includes the following steps:
- inventorying the resources, in specific units (which may vary across different resources), required for all activities entailed in the intervention;
- calculating the real (adjusted for inflation) cost per unit of each resource used, including fringe benefits associated with wages (P);
- counting the number of units of each resource used (Q) in the specified time period for the number of children, youth, or families served;
10 For more information, see http://www.acf.hhs.gov/programs/cb/resource/cost-workgroup [March 2016].
11 Available: http://evans.uw.edu/sites/default/files/public/Federal_Agency_BCA_PS_Social_Programs.pdf [March 2016].
- calculating the total costs of the intervention by multiplying all resources used by their unit costs (sum of all P × Q);
- calculating the expected cost per child, youth, or family served—i.e., average costs—by dividing P × Q by the number served during the specified time period of the intervention;
- calculating the expected cost per one more child, youth, or family served—that is, marginal costs—by differentiating between fixed and variable costs; and
- conducting sensitivity analysis to test the uncertainty of assumptions made about quantity and price.
Reporting the Results of Cost Analyses
Acknowledging the limitations of a CA requires transparency as to the methods used and the assumptions made. An overall goal is to achieve so much transparency that another community can implement the same intervention with complete understanding of all resources required (even if some resources are donated). As noted earlier, therefore, all resources need to be inventoried, even if all cannot be valued. For example, if one is conducting a cross-site evaluation of the costs to deliver a home visiting intervention and training costs are not available across all sites, these costs may be excluded for purposes of comparability across sites. However, the CA still needs to note that these costs are an important resource required to implement the intervention, even if they are not explicitly included in the CA results.
Because CAs generate such a wide array of estimates that the level of information can overwhelm even the most discerning reader and obscure useful information, reporting transparent and generalizable results is essential to ensure that the results of the analysis can be translated into effective policy. It is important in reporting, then, to balance detail with useful information. Also important is acknowledging that some unit cost estimates are more robust than others. Specifying where data are limited sets the stage for sensitivity analysis of cost estimates based on those variables and creates a research agenda for those implementing the intervention in the future.
Investors in interventions for children, youth, and families may want to know more than an intervention’s cost; they may also desire economic analysis of the return on investment in the intervention, which can be measured in various ways. As described briefly in Chapter 2, CEA compares resource investments with intervention impacts measured in their natural units, while
BCA compares investments with impacts that have been monetized. Cost-utility analysis is a form of CEA that compares investments with impacts expressed in quality-adjusted life years (QALYs) or disability-adjusted life years (DALYs).12 The common thread in these approaches is that they all rest on evidence that the intervention caused one or more favorable outcomes to occur. Evidence of impact may be drawn from a completed program evaluation, as in an ex post economic evaluation, or it may be presumed, as in the case of an ex ante economic evaluation, conducted, for example, for planning purposes.
Without valid evidence of causal impact, there is no reliable return on investment to capture in economic evaluation (Karoly, 2008; Washington State Institute for Public Policy, 2015). Causality, the main topic of this section, is one of several major impact-related issues that producers of economic evaluations need to consider in preparation for estimating cost-effectiveness or the return on intervention investment. Other issues common in economic evaluations of interventions serving children, youth, and families include how to combine evidence when multiple evaluations of a given intervention exist, which effects to include in a CEA or BCA, and how to handle uncertainty in intervention effect sizes—all addressed in subsequent sections of this chapter.
Research Designs and Evidence of Intervention Impact
An underlying premise of CEA and BCA is that the outcomes being subjected to economic evaluation were caused by the intervention.13 Logic models can help articulate an intervention’s putative causal mechanisms, but evidence that the intervention caused an outcome comes from certain research designs used in program evaluation. Some research designs can increase confidence that an observed difference was caused by an intervention, such as using many repeated measures over time and space; making comparisons with jurisdictions, population groups, or outcomes that should not be affected; replicating; and establishing that a
12 A QALY is defined as a measure of quality of life where 1 is a year lived in perfect health and 0 is death. In some circumstances, values less than 0 (fates worse than death) are allowed. Absent equity weights, a QALY is 1 minus a DALY. Quality-of-life measurement is anchored in part in functional capacity, which gives it some objectivity (Wilson and Cleary, 1995).
13 Causal inference is one of many factors that is relevant to the validity of a study or a set of studies for any given decision. It can be challenging to address certain factors beyond causal inference because they are often dependent upon concerns that the researcher cannot reasonably foresee or control (e.g., generalizability of study context).
Many, if not most, methodologists believe that the clearest unbiased evidence of causal impact comes from well-conducted experimental designs or randomized controlled trials because these designs minimize threats to internal validity, or the chance that something other than the intervention caused the observed differences in outcomes between intervention participants and nonparticipants (Cook and Campbell, 1979; Fisher et al., 2002; Gottfredson et al., 2015; Jones and Rice, 2011). Although research has shown that “real-world” randomized controlled trials of complex programs and policies are more feasible than originally thought (Cook and Payne, 2002; Donaldson et al., 2008; Gerber et al., 2013), relevant concerns have been raised regarding their limitations. These potential limitations include issues related to external validity (e.g., artificial circumstances defined by eligibility criteria, participants failing to represent particular populations, trials by definition only including volunteers who agree to participate for treatment); issues related to the number of treatment and control groups used;16 high costs; potential ethical concerns; and trade-offs with respect to generalizability and statistical power (Wagenaar and Komro, 2013).
Given the limitations of randomized controlled trials, certain methodologists have recommended the use of other research designs. For example, researchers have studied the extent to which quasi-experimental designs can provide unbiased causal evidence (Bloom et al., 2005; Cook et al., 2008; Shadish et al., 2008). These researchers present convincing examples in which well-designed prospective evaluations of laws and regulations using comparison time series designs—where randomization was not a possibility—provided strong causal evidence through careful attention to other design elements.
In addition, regression discontinuity designs (where individuals are compared with those who receive an intervention based on an arbitrary cut-off point) have also been shown to provide unbiased causal estimates (Gottfredson et al., 2015).17 For instance, children who miss eligibility for
14 The term “dose-response relationship” refers to the change in an outcome resulting from different degrees of exposure to an intervention.
15 With increasing occurrence, these types of alternative designs are used in both implementation and effectiveness studies. The ways in which these designs impact CAs, CEAs, and BCAs, although noteworthy, are not a primary focus of this chapter.
16 This limitation makes it difficult to ascertain what aspect(s) of the treatment are responsible for the observed effect. The ideal trial design would have multiple different treatment groups, with the potential for multiple control groups; however, these efforts are often a challenge to implement because of cost and logistical issues.
17 As is noted throughout this report, describing the research design and sample on which an economic evaluation is based aids in accurate interpretation of the evaluation results. For regression discontinuity designs, this is particularly important as impacts apply to participants
preschool by 1 or 2 months (based on birthdate cut-offs) can be compared with children whose birthdays are 1 or 2 months on the other side of the cut-off. Nonrandomized controlled trial designs such as these have been used to evaluate the effectiveness of universal pre-K programs in Boston, Tulsa, and in the state of Georgia (Yoshikawa et al., in press). Additionally, two other nonrandomized controlled trial designs, difference-in-difference and fixed effects, also have been shown to be effective and have been used to examine the efficacy of preschool (e.g., Bassok et al., 2014; Magnuson et al., 2007).
Propensity matching, propensity scoring, and instrumental variable designs also are popular alternatives, but at times are misapplied (Austin, 2009; Basu et al., 2007) or yield questionable results. Too often, propensity matching studies match to an intervention serving nonequivalent people (e.g., those who declined the intervention), and the instrumental variables chosen violate essential independence requirements. Quasi-experimental studies are intended to serve a useful purpose; however, the literature has several examples of such studies that are poorly designed, and many grapple with the same issues as those encountered with randomized controlled trials.
Ultimately, analysts conducting economic evaluations need to assess and describe the overall quality of the impact evaluation evidence that forms the basis for the economic evaluation, whether that evidence comes from experimental or quasi-experimental designs or both. For some interventions, strong evidence will come from experimental designs. For others, a randomized control trial is not feasible, but other strong quasi-experimental designs can contribute credible evidence.18 Often, there may be multiple evaluations and analysts can use the overall body of evidence as part of the economic evaluation. For example, researchers often pool data from a variety of interventions derived from randomized controlled trials as well as other useful research designs (see blueprintsprograms.com for examples). When multiple impact studies exist, systematic reviews and/or meta-analyses may be necessary to draw valid conclusions about impacts. For a discussion of major issues involved in systematic reviews and meta-
who are at or near the eligibility threshold, or cut-off score, for receiving an intervention. Reporting of information about the entire group served by the intervention, the cut-off value, and the portion of the group to which the impact estimates apply is encouraged to add transparency to economic evaluation findings that are based on impacts from these designs.
18 As an example, the research examining the intended and unintended effects of the Earned Income Tax Credit (EITC) on employment and other outcomes has relied on several quasi-experimental methods because an experimental design for evaluating this federal program has not been an option. Analysts have used natural experiments, such as the expanded eligibility for the program in the 1990s, the adoption of state EITC add-ons, and the calendar timing of receipt of the lump sum tax credit (Bitler and Karoly, 2015).
analyses of interventions for children, youth, and families, see the paper by Valentine and Konstantopoulos (2015) commissioned for this study.
Issues in Practice
As research on quasi-experimental designs suggests, standards for evidence evolve over time. Best research design practices and methods also may differ across disciplines, for which different concerns may apply. In the real world, moreover, the only available evidence of intervention impact may be from research designs that are not optimal, and impact estimates may indeed be biased. In such cases, should the economic evaluation go forward? Sometimes the answer is no. For example, if the evidence was produced with no comparison group, if measurement was very weak, or if the evaluation is judged to be of poor quality for other reasons, proceeding with a BCA or CEA is probably unwise. In other situations, such as when several quasi-experimental designs consistently suggest positive impact, it may make sense to proceed.19 In such cases, however, conducting sensitivity analyses with varying effect sizes to test the robustness of conclusions about impact is important. Another approach that can be taken in the absence of causal or unbiased evidence is to estimate how strong intervention impacts would have to be to produce economically favorable results and then judge whether such effect sizes appear feasible. If only a small effect size is required, it may be relatively easy to conclude that the investments in the intervention are economically sound.
Placing the goal of using only the highest-quality, unbiased impact estimates in BCA and CEA in the context of real-world realities and imperfections leads to several practical suggestions for economic analysts. First, seek impact evaluations conducted in accordance with the strongest research designs and best practices within a given field or discipline. Second, critically evaluate all design elements and methods used to estimate impact. Third, when available evidence is based on nonoptimal designs or impact estimates are likely to be biased, it may be possible to proceed with caution, including acknowledging possible bias and conducting sensitivity analyses on economic findings. Finally, if economic evaluations appear too speculative to be valid, the next best step may be to attempt a higher-quality efficacy analysis that produces higher-quality evidence of impact.
19 See the paper by Valentine and Konstantopoulos (2015) commissioned for this study, which emphasizes the importance of identifying all evidence about an intervention’s impact (published and not; positive, negative, and null) before conducting an economic evaluation to avoid publication bias.
CONCLUSION: The credibility of cost-effectiveness analyses, benefit-cost analyses, and related methods is enhanced when estimates of intervention impact are based on research designs shown to produce unbiased causal estimates. Meta-analysis may be used when impact estimates are available from multiple studies of the same or similar interventions. When evidence of impact comes from designs that may be biased because of selectivity or other methodological weaknesses, it is important for researchers to conduct sensitivity analyses to test the robustness of their findings to variation in effect size and to acknowledge the limitations of the underlying evidence base in their reports.
To assess return on investment in an intervention, producers of economic evidence may choose CEA, BCA, or the related methods outlined in Chapter 2. In CEA, results are expressed as the cost per unit of a single outcome, such as dollars per life saved, dollars per incarceration prevented, or dollars per additional college graduate. Costs come from a CA and outcomes from an impact evaluation. As discussed above, the ideal is for outcomes to be linked causally to the intervention being subjected to CEA; when this is not the case, it is important to disclose the fact and to interpret results of the analysis with caution.
In contrast to CEA, BCA compares the economic value of an intervention’s outcomes, expressed in the selected monetary unit, with the costs of the intervention as determined by a CA. When the value of the outcomes exceeds the intervention costs (after both have been adjusted as necessary for inflation and discounting), an intervention can be said to be “cost-beneficial.” For example, a BCA of an obesity prevention intervention would involve estimating the economic gains, or benefits, from reducing obesity and determining whether they exceeded intervention investments (after both had been adjusted for inflation and discounting). As discussed earlier, comprehensive BCAs take a societal perspective, but some BCAs are conducted from a more narrow perspective, such as that of the government or an agency. These more narrow BCAs may under- or overstate both costs and benefits, a limitation that needs to be disclosed when results are communicated.
BCA is an attractive method for interventions impacting multiple outcomes, as is common in interventions for children, youth, and families, because the dollar benefits of each outcome can be summed to produce an estimated total economic impact. Yet while BCA is a powerful tool that increasingly has become part of evidence-based decision making (White and VanLandingham, 2015), monetizing outcomes can be a complex and time-consuming process. Many organizations lack the capacity to under-
take these analyses without consulting experts in the method. Moreover, the valuation of some outcomes—for example, a human life—can be controversial and lead to skepticism regarding the findings and conclusions of the analysis. Haddix and colleagues (2003) suggest this is why CEA became the dominant analytic method for economic evaluation in health care after the 1980s. In other fields concerned with positive development for children and youth, however, interest in BCA is growing.
Consistent with the overall goal of this chapter, this section addresses issues related to valuing intervention outcomes as part of a high-quality BCA (Boardman et al., 2011; Crowley et al., 2014; Karoly, 2012; Vining and Weimer, 2010). As the discussion proceeds, the economic value of the outcomes from an intervention is referred to as the resulting “benefits.” In practice, however, an intervention may generate some favorable outcomes that result in higher costs (e.g., an intervention that increases educational attainment adds education costs as an outcome), or an intervention may generate some unanticipated unfavorable outcomes that translate into higher costs (e.g., an increased use of special education rather than the expected decrease).
Typically, benefit streams are estimated over time and then discounted back to the present to reflect monetary time preferences according to the following formula:
The size of economic impact in each year (Qy) and the price per unit of economic impact in each year (Py) both need to be estimated and a discount rate (d) selected. The total present-value benefit is the sum of all intervention-related discounted benefits streams. Producing high-quality estimates requires careful attention to each component: quantity, price, changes in each over time, and discount rate. It also involves assessing the implications of uncertainty for the estimates and summary measures derived from the evaluation. This section describes issues involved in estimating economic impacts and their prices over time. The basic rationale for discounting was addressed earlier in the chapter, while uncertainty and summary measures are discussed in subsequent sections.
Quantifying Outcomes for BCA
Undertaking a BCA requires several decisions with respect to the outcomes that result from the intervention being analyzed. As discussed below, these decisions include which outcomes to include in the analysis; whether the outcomes can be valued directly or linked to other outcomes with economic value; and what the appropriate time horizon for the analysis
is, including whether outcomes can be projected beyond the point of last observation to capture expected future outcomes.
Which Impacts to Include in the Analysis
In conducting a BCA, the analysis needs to identify all outcomes impacted by the intervention and determine which ones to include in the analysis. BCAs typically incorporate a subset of outcomes because not all outcomes can be monetized. For example, interventions may impact the quality of parent-child relationships, but the economic consequences of these gains have not been studied. Other reasons for exclusion of outcomes relate to methodological shortcomings, as there may not be a satisfactory approach to or precedent for monetizing them.20 For instance, social and emotional learning outcomes, which may be instrumental in healthy development, generally are not included in BCAs, although research in this area is progressing (Belfield et al., 2015). There may be too much uncertainty in how the outcomes are estimated, such as effects on populations not directly targeted by the intervention (Institute of Medicine and National Research Council, 2014; Karoly, 2012; Vining and Weimer, 2009c). Finally, another common factor in the choice of outcomes is convenience: analysts may focus on the outcomes that are easiest to work with or on those that are more relevant to their field of study or to the intended audience for the analysis (Institute of Medicine, 2006). An important consequence of the exclusion of outcomes for any of these reasons is potential underestimation of the economic benefits of an intervention (or overestimation, if excluded outcomes indicated harms from the intervention). BCA reports need to include information about which intervention outcomes were not included and why.21
Once monetizable outcomes have been identified, an additional consideration is which ones to include in the BCA. Some argue that all measured impacts, both significant and nonsignificant, should be included in the BCA, provided they can be monetized. Monte Carlo methods that account for standard errors associated with estimates can be used to evaluate the implications of uncertainty in the estimates (Farrow and Zerbe, 2013; Vining and Weimer, 2009b). This approach avoids the selection of impacts based on a somewhat arbitrary level of statistical significance. The drawback is that it may mean including impacts that are theoretically plausible but not well supported by evidence. Haddix and colleagues (2003) recommend
20 See, for example, Vining and Weimer (2009a), who provide a list of shadow prices that need to be estimated (or improved) and are important to the economic evaluation of social programs.
21 Double counting also becomes an issue if some outcomes are inputs for other outcomes. For example, it would be tricky to include both education and earnings as outcomes for an intervention.
that studies include all benefits and harms that have a meaningful impact on the results of the analysis. In its recent report on standards of evidence for efficacy, effectiveness, and scale-up in prevention science, the Society for Prevention Research concludes that prevention programs can be deemed efficacious only with respect to impacts with a “consistent pattern of non-chance findings in the desired direction” (Gottfredson et al., 2015, p. 908). These studies would suggest a more narrowly constructed BCA.
A related concern is that randomized controlled trials of interventions by the interventions’ developers often collect longitudinal data on scores of outcomes indicated by program logic models, test significance by time period and subgroup, and then produce confidence intervals by time period-subgroup combination. Statistical test results may or may not be corrected for multiple comparisons and the possibility of chance findings. The first Nurse-Family Partnership trial, for example, has tracked many child outcomes through age 19. Publications report multiple outcomes across time and for relevant subgroups. Additional outcomes are reported in interim or final grant reports (Miller, 2015). In such cases, some filtering of outcomes is necessary before a BCA is conducted.
Absent agreement within the field about whether to include all or a subset of measured impacts, a way forward is to adopt an explicit rationale for which impacts to include, employ the rationale in a consistent manner, and disclose the rationale when reporting on methods and results. It is important for each included impact to follow from the intervention’s theory of change. Additional criteria might include the statistical significance of the impact, as well as the power to detect effects given the research design and sample size. If conventional levels of significance (p < .05 or p < .10) appear to be too restrictive (Institute of Medicine and National Research Council, 2014), analysts can explain why additional impacts were included. If the outcome change is large and preferably supported by significant changes in intermediate outcomes (e.g., significant gains in math and reading achievement scores could justify including a rise in high school graduation rate with a significance of p = .18), and the BCA includes a probabilistic sensitivity analysis, one might want to include this impact.22 The variance of the overall benefit-cost estimate will account for its large variance. Further sensitivity analysis with and without the nonsignificant impacts also will aid in understanding the implications for the BCA’s bottom line.23
22 The power to detect an impact of a given size could be quite high even if significance levels were moderate, for example, p < .15.
23 It is important that evaluators not cherry pick results but rather formulate an approach that is applied consistently to all impacts. This means that the same rules apply to intervention impacts that are in the expected direction and those that are counter to what was expected, or iatrogenic impacts.
Types of Impacts
Intervention impacts may have direct economic implications or may be linked to other impacts that have economic value. Comprehensive estimates consider both. Direct economic impacts are measured in the impact evaluation and reflect changes in the outcomes targeted by the intervention. For example, a job training program may lead to an increase in employment among its participants, which has economic consequences in the form of higher wages for participants, increased taxes to government, and the like.
Linked economic impacts are not measured directly in the impact evaluation but are caused by other, direct intervention impacts.24 If a truancy prevention program causes lower absenteeism and higher graduation rates, for example, its linked impacts may be increased college enrollment, better-paying jobs, etc. As another example, expanding Medicaid and the State Children’s Health Insurance Program is associated with increased tax receipts (Brown et al., 2015) and greater rates of high school and college completion (Cohodes et al., 2014). Linked impacts can be numerous, vary in magnitude, and cover a wide range of domains (e.g., employment, health, crime). They also can be intangible, such as improved quality of life from obtaining employment or reductions in pain and suffering from lower rates of crime or child abuse. To include linked impacts in their analysis, researchers conducting BCAs typically rely on evidence established by other research (e.g., meta-analysis, databases linking smoking or alcohol use to illness rates and/or death) that makes it possible to estimate the size of the linked relationship. As with evidence of intervention impact, evidence of causal linkage is desirable but not always feasible. Evidence from correlational studies or other noncausal designs needs to be used with appropriate caution, and conducting sensitivity analyses incorporating different effect sizes is important. Table 3-2 provides several examples of BCA studies including direct impacts, linked impacts, or a combination of the two.
Time Horizon for Estimated Impacts
The goal of many interventions for children, youth, and families is to improve lives not just in the present but over time. In conducting a BCA, it is important to describe the time horizon over which intervention impacts are being estimated. Strong theory linking present to future behavior, supported by empirical data, can help justify these choices (Crowley et al., 2014; Institute of Medicine and National Research Council, 2014). Some outcomes, such as the effect of increasing educational attainment on wages,
24 If a study is long enough, researchers may be able to gather evidence on linked outcomes directly. Often this is not the case, so producers need to rely on evidence from the research literature.
|Benefits and Costs of Intensive Foster Care Services: The Casey Family Programs Compared with State Services (Kessler et al., 2008; Zerbe et al., 2009)||The High/Scope Perry Preschool Program Cost-Benefit Analysis Using Data from the Age 40 Follow-up (Belfield et al., 2006)||Benefit-Cost Analysis of Communities That Care Outcomes at Grade 12 (Kuklinski et al., 2015)|
|Description||Comparison of adult educational, health, and social outcomes among children who received private intensive foster care services compared with children who received public foster care services||Comparison of adult outcomes at age 40 among children who participated in the Perry Preschool Program compared with children who did not||Comparison of sustained effects of the Communities That Care (CTC) prevention system on grade 12 outcomes observed in a panel of youth involved in a community-randomized trial of CTC efficacy|
|Treatment, Control, or Comparison Group||
Randomized controlled trial:
Measured Program Outcomes
|Program Outcomes with Direct Economic Impacts||
|Linked Economic Impacts||
|Actual or Estimated Economic Impacts||Estimated||Combination||Estimated|
may last over an individual’s working life; others, such as the association between early test scores and educational attainment, may fade over time (Washington State Institute for Public Policy, 2015). Even when the empirical literature indicates long-term economic impact, the analyst may be pressed to consider economic consequences over shorter time periods. This pressure may reflect decision makers’ values and priorities, but it also can reflect that uncertainty increases over time because the world changes in unpredictable ways.
Some argue that the best way to understand the long-term effects of interventions is to collect real data by following participants over time and learning what happened to them (National Research Council and Institute of Medicine, 2009). Studies that used administrative or other data for long-term follow-up have demonstrated important long-term economic impact from investments made in childhood. The age 40 follow-up of the Perry Preschool Program documented persistent economic impacts including higher earnings for participants, higher tax revenues, lower criminal justice system expenditures, and lower welfare payments (Belfield et al., 2006). Several studies examining the implications of expanded childhood Medicaid coverage also showed long-term improvements to a range of health-related outcomes (Miller and Wherry, 2014; Wherry and Meyer, 2015; Wherry et al., 2015). However, interventions rarely produce these data because resource constraints prevent long-term follow-up, and measurement typically is limited to outcomes closely related to the intervention’s goals (Karoly, 2012). Because unmeasured linked impacts can represent a significant portion of an intervention’s benefits—for example, 70 percent of the Perry Preschool Program’s estimated benefits were due to long-term reductions in criminal activity among males (Belfield et al., 2006)—ignoring them is likely to bias BCA estimates. Constant follow-up evaluations would help in acknowledging long-term benefits, as well as changes in needs, interests, or contexts.
The alternative is to estimate or project impacts over time. In some cases, developing projections is relatively straightforward. For example, the long-term impact of adolescent smoking consists mainly of poorer health in adulthood. Good estimates of this effect are readily available from epidemiologic studies; thus, results from a short-term evaluation of an adolescent smoking prevention intervention can be combined with estimates from the literature on the long-term effects of smoking to project the intervention’s long-term impact. In other cases, projections are more complex. For example, the Washington State Institute for Public Policy has developed a sophisticated model for estimating the impact of reducing delinquency and crime that involves the marginal and operating costs for different types of crime, recidivism rates and related costs, victimization costs related to dif-
ferent types of crimes, and other elements (Washington State Institute for Public Policy, 2015)
At present there are no standards or widely accepted guidelines on how to generate projections. More straightforward approaches involve combining estimates found in the literature with results estimated from program data; for example, the analysis of an intervention that increases academic achievement may project impact on lifetime earnings based on published studies of how academic achievement causally affects earnings (Heckman et al., 2014). More complex approaches may involve the development of large microsimulation models using parameters derived from program evaluations, administrative and survey data, and the literature (Thomas, 2012). Variation also exists in how researchers decide which outcomes to project. If the focus is on outcomes that are easier to project, benefits are likely to be under- or conservatively estimated. If the focus is on including outcomes that are more difficult to project, estimates may be imprecise and have wide confidence intervals. Relatedly, projecting outcomes adds uncertainty to a study’s results, especially over longer time periods or when empirical evidence of causal relationships is not well developed (e.g., intervention impacts on peers or siblings [Institute of Medicine and National Research Council, 2014; Karoly, 2012]).
CONCLUSION: Preparation for a high-quality benefit-cost analysis requires consideration of which impacts to value in the analysis, including whether to include outcomes that are not statistically significant. It is also necessary to determine which outcomes may justifiably be linked to other outcomes that can be valued and that may be projected into the future. In each of these areas, standards or widely accepted guidelines for best practice are lacking. At a minimum, best practice requires transparency in reporting the approach taken. Consideration may also be given to conducting sensitivity analyses to determine whether results are robust to alternative approaches.
Valuing Outcomes for Economic Evaluation
To estimate the economic benefits of an intervention (e.g., avoided costs, increased income, increased tax revenues), the analyst must assign prices to its impacts, a process that includes consideration of price changes over time. Some outcomes have market prices that reflect their economic value; that is, the prices reflect an individual’s or society’s willingness to pay them. Examples include wages paid for different types of jobs, health insurance costs, and home prices.
However, some outcomes of interest to children, youth, and families (e.g., lower student absenteeism, greater self-respect, reduced loss of free-
dom) do not have market prices, or their market prices are distorted (e.g., prices do not reflect the outcomes’ actual economic value). An example of the latter is health care utilization as measured by hospital charges, which often are poor indicators of costs or even of what hospitals actually are paid for their services. In these cases, researchers can estimate the outcomes’ economic value by turning to shadow prices, which, as discussed earlier, are indirect estimates of how much individuals or society would be willing to pay for the outcomes. At times, assignment of shadow prices is relatively straightforward. For example, although there is not a direct market for educational attainment, its shadow price is routinely captured in BCAs through higher wages associated with higher levels of attainment (Oreopoulos and Petronijevic, 2013; Washington State Institute for Public Policy, 2015).
Other approaches for estimating shadow prices, such as revealed preferences and stated preferences, are more complex.25 The revealed preferences approach looks at what people pay for products with the attributes that need to be valued but are not priced directly. For example, regression analyses of how home prices vary with water quality or neighborhood safety can reveal how much homebuyers value drinkable water or safety (Bickel and Rainer, 2005; Gayer et al., 2002; Nelson, 1978; World Health Organization, 2012), outcomes with import for children, youth, and families. Other revealed preference studies have valued family safety by examining how demand for home smoke alarms grew as their price dropped (Miller, 1990) and estimated how parents value children by analyzing the market for child bicycle helmets (Jenkins et al., 2001).
The stated preferences approach uses survey data to estimate prices where markets do not exist or are distorted. Survey questions ask respondents what they are willing to pay for the attribute that needs to be valued. Alternatively, respondents are asked to choose their preference among scenarios that differ in the attribute. The attribute may be tangible, such as high-quality childcare for children, or intangible, such as honesty. Because surveys allow for detailed descriptions of what is being valued, preferences can be elicited for very specific features of interest (Ryan et al., 2008).
The two most common stated preferences methods are (1) contingent valuation, where individuals are asked directly how much they would be willing to pay (or be paid) for a certain outcome (e.g., a reduction in the risk of death), contingent on a hypothetical scenario; and (2) discrete choice experiments, in which respondents’ preferences are assessed from a sequence of hypothetical scenarios that vary along several attributes, which can include the price of the nonmarket commodity of interest. Widely ac-
25 A more complete discussion of pricing and monetization methods used in BCAs can be found in Vining and Weimer (2009b).
cepted guidelines exist for contingent valuation survey design and procedures (Arrow et al., 1993; Bateman et al., 2002). Examples of how these methods have been used include estimations of the value of environmental policies, food safety, injury prevention, and various health-related outcomes (Boxall et al., 1996; Hanemann, 1994; Lindhjem et al., 2011; Loureiro and Umberger, 2007; Olsen and Smith, 2001; Schwab-Christe and Soguel, 1995).
A major challenge in applying contingent valuation methods is that respondents have difficulty putting a price on intangibles reliably and reproducibly (Mitchell and Carson, 1989). Often, the methods yield overstated values (Hausman, 2012). Hausman (2012, p. 43) concludes the method has intractable “problems of embedding and scope. . . . [r]espondents to contingent valuation surveys are often not responding out of stable or well-defined preferences, but are essentially inventing their answers on the fly, in a way which makes the resulting data useless for serious analysis.” Discrete choice models address these problems by asking individuals to choose among alternatives with different sets of tangible and intangible attributes (Bridges et al., 2011). Among stated preference methods, the discrete choice approach has “become the most frequently applied approach in healthcare” (Johnson et al., 2013). Consensus panels also have developed guidelines for discrete choice survey design and procedures (Bridges et al., 2011; Johnson et al., 2013).
Despite having strong theoretical foundations and being viewed favorably by such agencies as the U.S. Office of Management and Budget (2003), revealed and stated preference methods only recently became used more widely in health care economics, primarily in international health, and they remain rare in studies of social interventions. Examples of such work relevant to investments in children, youth, and families include analyses of parents’ willingness to pay for preschool programs for their children (Escobar et al., 1988) or for protecting their children against house fires, road crashes, human papillomavirus, the common cold, asthma, and fatal diseases (Alberini and Šcasný, 2011; Blomquist et al., 1996; Brandt et al., 2012; Brown et al., 2010; Hammitt and Haninger, 2010; Jarahi et al., 2011; Liu et al., 2000; Miller, 1990; Takeuchi et al., 2008). Studies with valuations not limited to respondents’ own families have examined the value of crime reduction; employer-sponsored childcare; library services; community cultural, sports, and leisure activities; arts performances; preventing a child’s death due to maltreatment; and publicly funded health interventions (Bosworth et al., 2010, 2015; Cohen et al., 2004; Connelly et al., 2004; Corso et al., 2011, 2013; Hendrie, 2013; Ludwig and Cook, 2001; Thompson et al., 2002).
Revealed and stated preference methods can facilitate the inclusion of important intangible benefits, such as increased motivation or self-
confidence (Belfield, 2014), in BCAs of interventions for children, youth, and families. Failing to monetize intangible benefits may lead to underestimating the value of outcomes considered in a BCA, particularly for interventions that have few or difficult-to-measure tangible effects. Examples of such outcomes include those related to behavioral/emotional problems, marriage and divorce, mental health, and reproductive health (Karoly, 2008).
Several recent studies have incorporated intangibles in the valuation of intervention outcomes. A recent study using willingness-to-pay methods, for example, estimated a value of $18.3 million for preventing a child’s death due to maltreatment, in contrast to earlier estimates of $1 million calculated from future productivity losses and $7.7 million based on juries’ willingness to award (Corso et al., 2007, 2011; Miller et al., 2001, 2006). In BCAs of health care interventions, including intangibles, allow researchers to go beyond valuing health outcomes simply by aggregating future health care cost savings and increases in productivity; rather, the intangible value of improved health status to patients and their families (e.g., Gentilello et al., 2005; Nichol, 2001; Takenaga et al., 1985) becomes part of the equation, resulting in dramatic increases in estimated intervention benefits (Dickie and Gerking, 2002; Miller and Hendrie, 2012).
Valuing children’s lives directly using willingness to pay would be problematic (Office of Management and Budget, 2003). Young children have limited ability to perceive risk (Barton and Schwebel, 2007), much less value it. The brain’s prefrontal cortex, which regulates emotions and promotes sound decision making, is still developing in late adolescence to early adulthood (Gogtay et al., 2004; Nelson et al., 2006). Adolescents consequently take irrational risks, with both their revealed and stated preferences for risk being skewed downward from their preferences at maturity. Using those immature values in societal decision making seems unwise, and many countries have strong traditions of safety legislation to constrain adolescent risk decision making.
A better option is to use the values parents or the broader society place on children (Alberini et al., 2010). The literature provides more than a dozen U.S. value estimates (Alberini et al., 2010; Scapecchi, 2006; Williams, 2013), as well as results of and stated preference surveys for a variety of risks in the Czech Republic and Italy (Alberini et al., 2010), cold protection in Taiwan (Liu et al., 2000), vaccination in Bangladesh (Islam et al., 2008) and the Philippines (Palanca-Tan, 2014), and arsenic-free water in Bangladesh (Aziz, 2007). Although not unanimous, these studies generally find that parents place a greater value on the safety of their children than on themselves (Alberini et al., 2010). OECD (2012) and Williams (2013) recommend valuing children’s lives at 1.5 to 2.0 times adult values.
CONCLUSION: In support of high-quality benefit-cost analysis, valuing intervention outcomes requires measuring society’s willingness to pay for each outcome. In some cases, market prices may provide the required economic values, but many outcomes affected by interventions for children, youth, and families do not have market prices. Indirect economic values, or shadow prices, are used to capture economic values using various methods, such as linking the outcome of interest to another outcome that can be valued. Revealed and stated preference methods can be used to estimate willingness to pay, potentially enabling both tangible and intangible outcomes to be valued. Using valuation methods that fail to account for the value of intangible outcomes may result in biased estimates (typically undervaluation) of intervention benefits.
Estimating shadow prices can be costly. It also can yield values with wide variance and even means that differ considerably among high-quality studies. Consider, for example, the published estimates of willingness to pay to avoid a homicide, in 2014 dollars, in Table 3-3. The means range from $6.3 million to $47.5 million, with the 95 percent confidence intervals from three studies not even overlapping. The value chosen obviously could make a major difference in the BCA outcome for a crime reduction intervention. One way to facilitate quality analyses, encourage sensitivity analysis around uncertain values, reduce expenditures on shadow pricing, and encourage consistent value choices that support benefit-cost comparisons across studies is to catalog quality values in a clearinghouse. The Tufts Cost-Effectiveness Registry (Thorat et al., 2012) has taken this approach with QALY estimates for health conditions, with the result that studies increasingly are using established values instead of developing their own.
|Data Sources||Mean||Lower Confidence Limit||Upper Confidence Limit|
|Cohen et al. (2004)||14.2||12.5||16.2|
|Corso et al. (2013) (child maltreatment death)||18.3||16.1||20.8|
|Kochi and Taylor (2011)||6.3||1.7||10.9|
|Scotton and Taylor (2011)||11.6||0.0||64.3|
|Scotton and Taylor (2011)||47.5||7.4||87.7|
Similarly, meta-analyses of outcomes developed by the Cochrane Collaborative (The Cochrane Public Health Group, 2011), the Community Guides to Preventive Services, the U.S. Substance Abuse and Mental Health Services Administration’s National Registry of Evidence-based Programs and Practices, and the U.S. Office of Justice Programs’ CrimeSolutions.gov increase the quality and completeness of effectiveness evidence readily available for use in economic analysis.
CONCLUSION: Registries can increase uniformity of practice, reduce the costs and time required for shadow pricing, and increase the accessibility and comparability of cost-outcome estimates for interventions targeting children, youth, and families. To support these functions, registries could be established to evaluate the quality of and archive (1) values for common nonmarket outcomes such as those related to crime and special education, and (2) cost-outcome estimates for interventions for children, youth, and families.
Valuing Quality of Life
Quality of life is a particularly important intangible that is likely to be relevant in valuing outcomes for interventions for children, youth, and families. Indeed, virtually all such interventions affect quality of life to some degree. Moreover, quality-of-life impacts may constitute a large share of the benefits of such interventions (see, e.g., the benefit-cost estimates for various interventions shown in Table 36.6 in Miller and Hendrie ). Consequently, an economic evaluation that excludes these impacts may fail to capture the full economic value of the intervention being analyzed, typically underestimating its benefits. In addition, if the value of quality-of-life impacts is omitted, it is difficult to compare the economic returns on investments in different sectors, such as health interventions versus prevention or other social interventions. Nonetheless, the difficulty of measuring and valuing quality-of-life impacts is a long-standing concern regarding economic evaluations of interventions serving children, youth, and families.
In the case of health interventions, quality of life accounts for a large portion of the benefits. For this reason, according to the first Panel on Cost-Effectiveness in Health and Medicine (Gold et al., 1996, p. 84), “In order to capture health outcomes beyond simple survival it is necessary to obtain information on the health-related quality-of-life associated with different interventions.” Health and environmental analysts generally have taken two approaches to measuring quality-of-life gains. The first examines what people are willing to pay—and actually do pay—for goods that represent the effects on quality of life (e.g., a reduced risk of death or criminal victimization) (Dréze, 1964; Schelling, 1968). The second measures impact on
An advantage of quality-adjusted life years (QALYs) or disability-adjusted life years (DALYs) is that one can apply a single quality-of-life measure to track changes across a range of interventions instead of collecting willingness-to-pay data on multiple outcomes. This advantage is especially important in valuing such things as reductions in drunk driving crashes versus distracted driving crashes, which produce different profiles of injuries (Blincoe et al., 2015). It would be difficult to describe those injury mixes to survey respondents in a way that would enable them to provide well-informed willingness-to-pay estimates, and much easier to use a standardized instrument to collect objective data on the functional capacity lost to the injuries and convert that loss into QALYs. QALYs also are not constrained by what respondents can afford or their difficulty in accurately pricing something that cannot be purchased.
On the other hand, an advantage of willingness to pay is that the valuation is in dollars. It is difficult to interpret a cost-utility estimate of, say, $75,000 per QALY saved. Often the value choice is left to decision makers, with country-specific yet arbitrary maximum acceptable value ranges emerging over time (Harris et al., 2008; World Health Organization, 2001). To better inform those choices, more than a third of recent cost-utility studies conducted a sensitivity analysis using a cost-effectiveness acceptability curve (CEAC), which displayed the probability that an intervention yielded a positive return at different monetary values of a QALY (Meckley et al., 2010). Another alternative is to place a dollar value on a QALY. Popular approaches to QALY valuation are to use a willingness-to-pay survey (e.g., Gyrd-Hanson, 2003) or simply to divide the value of statistical life by the number of QALYs left in an average life (discounted to present value) (e.g., Cutler and Richardson, 1998; Miller et al., 1989). Both valuation approaches ignore economic theory and supporting empirical evidence, which suggest that the value of a QALY is situational (Hammitt, 2007; Hammitt and Haninger, 2010, Haninger and Hammitt, 2011; Johannesson and Johansson, 1997; Mortimer and Segal, 2008; von Stackelberg and Hammitt, 2009).
The Committee to Evaluate Measures of Health Benefits for Environmental, Health, and Safety Regulation (Institute of Medicine, 2006) concluded that although DALY/QALY “measures are based on surveys reflecting individual choices, these choices may not fully reflect individual preferences and are not entirely consistent with the tenets of utility theory that underlie welfare economics” (p. 36). The committee concluded that since these measures were not homogeneous across ages or permanency of the impairment, the dollar value of a QALY/DALY was not fixed. Ironically, those same concerns arise with willingness to pay for a statistical life, a year of life expectancy, or avoidance of functional capacity loss, but the patterns of variation differ, which Hammitt (2007) concludes means using willingness to pay for a QALY violates social welfare theory. An alternative view is that a QALY is not a homogeneous good, and its value thus is specific to the details of the QALY. That conclusion is equally valid for a statistical life or many other intangible goods. Policy analysts sometimes handle this issue by using an average value but warning of its imprecision.
While interest in the use of happiness or well-being measures has grown, such measures have been found to be subjective, framed by a person’s disposition, expectations, adaptation, and coping strategies (Andrews and Withey, 1976; Diener et al., 1999). Indeed, Richardson and colleagues (2013) found that across six countries, a well-being scale explained only 7 percent of the variation in QALY scores among 8,022 respondents stratified to oversample people with chronic health problems. Happiness correlates poorly with income (Andrews and Withey, 1976; Argyle, 2001) but tends to be strongly influenced by personal disposition, and in the short to medium term by family events. In addition, people tend to have happiness set-points. Over time, adaptation causes happiness changes resulting from most major life events to fade, although unemployment tends to have a lasting effect (Diener and Oishi, 2005; Lucas et al., 2004; Robinson et al., 2012). Thus, well-being measures appear better suited for use as mediators and moderators than as outcome measures. Lacking an objective component to keep them in balance, they are better at predicting how people will react to change than at measuring change.
health-related quality of life directly using nonmonetary measures—either DALYs reduced (Murray and Lopez, 1996) or QALYs saved (Fanshel and Bush, 1970; Torrance et al., 1972; Weinstein et al., 2009). Researchers also have experimented with happiness or well-being measures, but their subjective nature can make their use in program evaluation challenging (Andrews and Withey, 1976; Diener et al., 1999). The considerations associated with each of these approaches are described more fully in Box 3-1.
Several generic QALY or DALY scales are commonly used in economic evaluation of health services (McDowell, 2006; Miller, 2000). Both systematic reviews (Spicer et al., 2011) and surveys that collected data using multiple scales (Richardson et al., 2015) provide insight into consistency and equivalences across scales and even support translation between some scales (Gray et al., 2006). The committee reviewed the major scales to see whether any of them accounted for aspects of well-being that extend beyond health. One widely used set of QALY scales did: the World Health Organization’s (WHO) 100-item quality-of-life instrument (WHOQOL) (World Health Organization Quality of Life Group, 1998) and a reduced 26-question WHOQOL-BREF (Skevington et al., 2004) that usually can be administered in less than 5 minutes. The WHOQOL-BREF instrument encompasses four domains—physical, psychological, social, and environmental—and includes questions about shelter and educational opportunities. It has been administered in more than 40 languages, and more than 11,000 respondents worldwide were polled to test its validity, reliability,
and discriminant ability. WHOQOL development and refinement extended from 1991 to 2004 and was a joint effort of 24 WHO collaborating centers.
The WHOQOL instruments originally were designed for adult respondents. However, they have been used successfully with adolescents in a range of countries, validated for adolescent use, and shown to be concordant with parent proxy reports (with a question about satisfaction with one’s sex life often being omitted or modified to be age- and culture-appropriate) (Agnihotri et al., 2010a, 2010b; Al-Fayez and Ohaeri, 2011; Chau et al., 2012; Cruz et al., 2014; Izutsu et al., 2005; Jirojanakul and Skevington, 2000).
Given this evidence, the WHOQOL-BREF appears to be a promising tool for estimating QALY gains from interventions for children, youth, and families that are not targeted to health. It provides proof of concept that aspects of well-being beyond health can be captured in a valid and reliable way. Indeed, it may be the forerunner of a family of quality-of-life measurement tools that are sensitive to gains in outcomes extending well beyond health to include educational, employment, social welfare, and other domains of well-being. Especially in the United States, such tools have been applied too rarely in impact evaluations and related economic evaluations of interventions for children, youth, and families to judge whether their promise can become a reality.
CONCLUSION: Quality-of-life impacts may be an important outcome of interventions for children, youth, and families, yet they are rarely accounted for in economic evaluations of such interventions, largely because of the difficulty of measuring and valuing them. Approaches for this purpose have been developed for use in economic evaluations in other policy areas, such as health and the environment, which may provide promising strategies for use in economic evaluations of social programs. A quality-of-life measure embedded in a willingness-to-pay estimate or one that can be compared with the quality-adjusted life years/disability-adjusted life years measures used in economic evaluations of health interventions would facilitate the ability to compare across sectors.
Aggregating Economic Values
As this discussion suggests, estimating direct and linked impacts and assigning prices to them, including choosing an appropriate time period for the analysis, is often a challenging, time-consuming endeavor that involves many decisions and assumptions on the part of the analyst. Once the process has been completed, however (at least for monetized benefits), valuing
interventions for purposes of a BCA is relatively straightforward.26 After ensuring that impact quantities and prices are in comparable units, flows over time can be calculated, summed, and discounted according to equation (3.1) presented earlier in this chapter to determine the present value of intervention benefits. Issues involved in comparing intervention benefits with costs to determine whether an intervention investment is cost-beneficial are described next.
The final step in producing economic evidence related to investments in interventions for children, youth, and families is the development of summary measures appropriate to the audience, perspective, and scope of the analysis. In CA, summary measures may include total costs, average costs, and marginal costs. In CEA, summary measures may include the cost-effectiveness ratio, the measure of net health benefits or net monetary benefits, or the cost-effectiveness acceptability curve. In BCA, summary measures may include a measure of net benefits, a benefit-cost, ratio, the internal rate of return, or return on investment. Different summary measures convey different information, and they have different strengths and limitations. In each type of analysis, the type of summary measure that is developed needs to be informed by the study question, the scope of the analysis, and the audience. Further, discussing these measures a priori is critical for successful utilization by policy makers.
Summary Measures: Cost Analysis
Total costs of an intervention are its aggregate costs, calculated by multiplying all resources used by their unit costs and then summing these totals. As noted earlier, if costs are incurred over time, decisions about inflation and discounting are applied. Total costs can be disaggregated by typical cost categories, such as personnel versus nonpersonnel, or by activity category, such as direct client services versus administrative costs. It also may be helpful to aggregate according to intervention startup, implementation, and sustainability phases (Crowley et al., 2014). A measure of total costs is useful for summarizing costs over time, across intervention sites, and across funders, and is helpful in providing key feedback to intervention administrators. However, a measure of total costs may not explain why an intervention in one site costs more than that in another because differences in aggregate costs often are due to differences in the number of children,
26 See the later discussion of nonmonetized benefits.
youth, or families served. To understand difference in intervention costs across sites, one needs to examine the average costs of an intervention.
Average costs of an intervention express the expected cost per child, youth, or family served and are calculated by dividing total costs by the number served during the specified time period. This measure can be used to compare resource needs and costs across sites. Differences in average per participant costs by site suggest that factors other than the numbers served are driving cost differentials. Costs may vary as a result of differences in the individual characteristics of those served (e.g., non-English-speaking versus native-English-speaking, having a disability versus able-bodied), the intensity of services (e.g., part-day or full-day programming), the characteristics of the providers (e.g., nurses versus social workers), the characteristics of the intervention (e.g., startup versus ongoing, enhanced versus standard), or the characteristics of the community (e.g., urban versus rural).
Another important summary measure in CA is marginal costs, which are derived by calculating the expected cost per additional child, youth, or family served. As noted earlier, calculating marginal costs requires differentiating between fixed and variable costs in the CA. Marginal cost summary measures are helpful when budget planning is possible or necessary because they can be used to determine how many more individuals could be served, for example, if the budget were to increase or decrease by a certain percentage. Some resources—such as the building where the intervention is being delivered if not operating at capacity (a fixed cost)—would not need to be considered in an expansion of the intervention scale, while other resources—such as reading materials provided to a child in a reading-ready intervention—would need to be considered.
Summary Measures: Cost-Effectiveness Analysis
The main summary measure in a CEA is the cost-effectiveness (CE) ratio, derived by dividing the intervention costs (discounted and adjusted for inflation, as appropriate) net of monetized outcomes by the change in a focal outcome measured in its natural unit (e.g., cases prevented, percent reduction in crime, percent increase in high school completion rate, QALYs saved). An incremental CE analysis compares a set of two or more interventions arrayed in order of increasing effectiveness (with “no intervention” or “usual standards of care” often being the least effective). The set of comparative ratios is called incremental CE ratios (ICERs). The Panel on Cost-Effectiveness in Health and Medicine (Gold et al., 1996) and others (Drummond et al., 2005; Haddix et al., 2003; Levin and McEwan, 2001) provide detailed, cogent guidance on how to calculate ICERs.
Cost-effectiveness analysis is increasingly used to guide resource allocation decisions in high- as well as low- and middle-income countries.
These decisions typically concern allocations to social-sector interventions in the larger context of other government expenditures. For development interventions, decisions are made by national governments, as well as by global funding organizations and aid agencies. To support decision making, it is necessary to know the extent of additional resources a payer should be prepared to devote to incremental gains in the desired outcome areas. The cost-effectiveness threshold is the standard means for assessing the acceptability or affordability of an estimated incremental cost-effectiveness ratio. Box 3-1, presented earlier, includes discussion of the lack of theoretical justification for any threshold for a maximum acceptable cost per QALY or DALY and the ad hoc approaches used to handle the question.
Summary Measures: Benefit-Cost Analysis
The preferred summary measure for BCA is net present-value (NPV) benefits, which reflects the present value of all benefits attributable to an intervention less the present value of the costs required to conduct the intervention (after adjusting for inflation). When the NPV is positive, it provides an economic argument for investing in the intervention because it implies that the benefits of the intervention outweigh the costs after discounting. An attractive feature of this measure is that it is not sensitive to how costs and benefits are labeled (e.g., if negative benefits are treated as costs). It also provides information about the magnitude of benefits expected. Both total NPV and NPV per unit (e.g., child, youth, family) are informative.
The results of a BCA also are commonly expressed in terms of the benefit-cost ratio (BCR), which is calculated by dividing total benefits by total intervention costs after relevant discounting and inflationary adjustments have been performed. If the resulting ratio is greater than 1, benefits exceed costs, and an economic case can be made for the investment. The BCR also is popular with stakeholders because it allows for concise statements about the “dollar return per dollar invested,” making it possible to focus on where the next dollar should be spent or the last dollar shifted among comparable interventions. However, the BCR can be misleading for several reasons. First, as a ratio, it fails to capture the magnitude of benefits or costs. For example, a 3:1 BCR could result from an intervention that cost $100 per child and returned $300 or one that cost $10,000 but returned $30,000. Decision makers facing limited budgets, pressures to reach certain numbers of children, and other constraints benefit from more information than is captured by a BCR alone. Further, the ability to treat some costs as “negative benefits” or benefits as “negative costs” means that the ratio can be manipulated.27
27 This may not be a relevant issue for certain subfields.
Another summary measure in BCA is return on investment (ROI), which captures the percentage of return for every dollar invested. It is calculated by dividing intervention benefits net of intervention costs (as opposed to total benefits as in the BCR) by intervention costs. Common in the business sector, ROI summary measures often are narrowly constructed, focusing on financial gain to the intervention provider or funder over a short time horizon rather than overall benefits for society over a short or long time period. At times it also is used as a first-approximation measure of what might be gained from further investments of a similar type, although see the discussion above about issues surrounding projections. More recently, the concept of ROI has increasingly been used to describe investments in health and education (Psacharopoulos and Patrinos, 2004; Robert Wood Johnson Foundation, 2013). In reality, many of the interventions that target children, youth, and families may not necessarily save money in the short term.
Somewhat similar to ROI, the internal rate of return (IRR), calculated as the discount rate that makes the NPV equal to 0, is yet another summary measure used in BCAs—typically in BCAs of business investments. An intervention with an IRR above some predetermined threshold (the social discount rate or the rate of return required by the business or organization providing the intervention) is deemed acceptable. The IRR provides valuable information as to the sensitivity of the NPV or BCR to the choice of the discount rate. For a given discount rate (say, 3%), if the estimated net benefits are positive, the IRR shows how high the discount rate could go before the NPV was zero. Conversely, if the estimated net benefits are negative, the IRR indicates how low the discount rate would have to be for the NPV to become positive. As a summary measure, however, IRR can be problematic because interventions can have multiple IRRs, depending on the timing of costs and benefits, and the IRR will not necessarily rank interventions in the same order as the NPV (Zerbe and Dively, 1974).28 Further, unlike net benefits as a summary measure but like the BCR, the IRR does not provide information on the scale of the intervention—that is, the scale of costs required to implement the intervention.
Economic evaluations producing summary measures based on nonlocal samples and prices may not inform local decisions. Understanding how investments yield benefits depends critically on preintervention problem levels as well as an intervention’s reach; that is, context matters (see Chapter 4). The local context may be very different from the context that produced a favorable NPV or BCR. As a specific example, if a community has a base-
28 To address this issue, Zerbe (2010) proposes a reliable IRR (IRRr). Zerbe and Dively (1974) also discuss an approach for adjusting for projects with different costs because the BCR can provide different rankings.
line rate of youth substance misuse that is much lower than the national average, a school-based substance misuse intervention in that community is likely to have a lower BCR than the same intervention in a community with a higher baseline rate. Similarly, if a community implements the same intervention but in a different setting (e.g., family-centered rather than school-based), the intervention may have a different cost structure relative to that in other communities; population characteristics may differ as well, including different baseline levels of the problem behavior. Data needed to fully inform local decision making typically are not available.
Reporting Summary Measures
Regardless of the method of economic evaluation employed, all the methods involve CA, so it is important to, at a minimum, report results framed in the context of the unit cost of the investment. Specifically, a CA would report the average and marginal cost for each participant unit. In addition, a CEA would report the average and incremental cost-effectiveness per unit of outcome achieved (e.g., cost per emergency department visit prevented), while a BCA would report the NPV per participant along with the BCR. Together, such estimates can prevent consumers of the evidence derived from BCAs, in particular, from assuming that the intervention with the lowest cost or highest total savings is inherently the better choice.
CONCLUSION: The literature supports a number of summary measures for economic evaluation:
- Cost analysis—In addition to total cost, informative summary measures include the unit cost of the investment (e.g., cost per participant or average cost) and marginal cost.
- Cost-effectiveness analysis (CEA)—The preferred summary measures for a CEA are the average and incremental cost-effectiveness per unit of outcome achieved (CE ratio). Cost-effectiveness thresholds may also inform the analysis, although clearer guidance is needed on the valid thresholds to apply based on the outcomes and the setting.
- Benefit-cost analysis (BCA)—The preferred primary summary measure for a BCA is net-present value benefits. The benefit-cost ratio, return on investment, and the internal rate of return also can be valuable and may be required in many decision-making contexts, including projections of returns on further investments.
Uncertainty is intrinsic to evaluation in general and economic evaluation in particular. Vining and Weimer (2010, p. 17) explain, as follows:
The application of BCA to social policies typically involves producing predictions of net benefits based on imprecise predictions of numerous effects and their monetization with relatively uncertain shadow prices. Effectively, net benefits are the sums of products of random variables.
Even if the research design used to evaluate an intervention provided causal evidence of impact, uncertainty in the magnitude of that impact would remain an issue. A wide range of factors—for example, sampling error; selection bias; sample attrition, particularly differential attrition; measurement error due to imprecision or weak instrumentation; failure to control adequately for covariates—all may contribute to uncertainty about the true magnitude of intervention impact (Trochim and Donnelly, 2006).
Crowley and colleagues (2014) identify three different sources of uncertainty: (1) model uncertainty, related to decisions on which costs and outcomes are included in the analysis and assumptions regarding the relationships between them; (2) measurement uncertainty, which originates from imprecision in the measurement of analysis inputs and from sampling practices; and (3) parameter uncertainty, related to such assumptions as the discount rate or the probabilities used in state-transition models. Yet despite the ubiquity of uncertainty in economic evaluation, studies often fail to acknowledge uncertainty in their main results. In a summary of methodological choices made in 14 BCAs published between 1996 and 2010, Karoly (2012) found that only 4 of them reported the standard errors of their results.
Sensitivity analysis, as alluded to earlier, is used to address uncertainty in an economic evaluation. Although different methods can be used to conduct sensitivity analysis, in general they all are designed to test the robustness of results to variability in the study’s assumptions. Probabilistic methods (more formally, a Monte Carlo simulation analysis) that address multiple sources of uncertainty simultaneously provide a comprehensive assessment of the implications of uncertainty for economic estimates (Vining and Weimar, 2009a). In contrast, one-way (or multiway) sensitivity analysis varies values of a single parameter (or multiple parameters) over a range the analyst considers plausible (e.g., the magnitude of the effect size for a given outcome, the magnitude of the relationship between an observed and linked or projected outcome, the discount rate), leaving all other parameters and assumptions unchanged, and recalculates results for each new value. Repeating this exercise for all major parameters allows the analyst to understand how robust baseline results are to uncertainty in each parameter or
combination of parameters. This method, however, provides a less complete picture of the implications of uncertainty than that obtained with a Monte Carlo simulation (Boardman et al., 2011; Weinstein et al., 1997), a limitation that needs to be acknowledged when this method is used.
Monte Carlo simulation is becoming standard practice for analyzing measurement uncertainty, as reflected in its being recommended by the U.S. Office of Management and Budget (2003) and the Canadian government (Treasury Board of Canada Secretariat, 2007). It is also standard practice in BCAs conducted by the Washington State Institute for Public Policy (Aos et al., 2004). The method entails repeated sampling, often thousands of times, of the probability distributions for each parameter in the cost or cost-outcome equation to estimate the variance around the economic estimate. Typically, a Monte Carlo analysis uses standard deviations of outcomes drawn from the intervention evaluation, although other probability distributions are possible. For unit costs, standard deviations often come from administrative data sets that are not program-specific (e.g., on school cost per pupil day) or even may simply assume a normal distribution with a standard deviation of 10 percent of the mean. A more complex example is a sensitivity analysis involving the value of preventing a homicide, in which the mean and standard deviation of each of three conflicting stated preference surveys is assumed to have an equal probability of being the correct distribution.
A limitation of the use of Monte Carlo methods for estimating the standard deviation of a complex cost-outcome equation involving multiple outcomes is that the procedure typically samples the probability distribution for each outcome independently. In reality, intervention performance often will be good or bad based on multiple outcomes simultaneously. If Monte Carlo methods do not take this covariance into account, they may underestimate the standard deviation.
Analysts may choose to supplement Monte Carlo analysis with one-way or multiway sensitivity analyses of key parameters. Such analyses, for example, can demonstrate the effects on economic evaluation results of smaller outcomes expected in intervention replication, a shorter time horizon for benefits, omission of an outcome with weak evidence, or an alternative discount rate. One-way sensitivity analyses also can provide information that facilitates comparisons across analyses. Notably, the first Panel on Cost-Effectiveness in Health and Medicine (Gold et al., 1996) recommended that all CEAs provide an estimate at a 3 percent discount rate either as the base case or in a sensitivity analysis. Because of that recommendation, the Tufts Cost-Effectiveness Registry has been able to catalog thousands of CEA estimates from around the world that can be compared without distortion by differing discount rates. Thus, the comparison case at a 3 percent discount rate has served the CEA community well. Of course
at different discount rates, interventions could end up being ranked differently in terms of such criteria as net benefits, but this qualification does not mitigate the need to use a common discount rate to compare interventions. Decision makers employing economic evaluations of interventions serving children, youth, and families would also be expected to benefit from having a 3 percent standard discount rate, either in a base case or as part of a sensitivity analysis.
A final important consideration in the analysis of uncertainty is the communication of results. A thorough sensitivity analysis may require that the analyst expend a great deal of effort on summarizing the implications of the analysis for decision makers, who may have a low tolerance for complex technical discussions (Robinson and Hammitt, 2011). In the case of BCA, Vining and Weimer (2010) recommend communicating a simple indicator of uncertainty to decision makers: the proportion of Monte Carlo trials yielding a positive net benefit. They argue that if the central concern in a BCA is whether an intervention produces positive net benefits, decision makers can be confident about the likelihood of making a favorable investment if the proportion of trials with zero or negative benefits is small. The Washington State Institute for Public Policy has adopted this practice in its BCA reports (Washington State Institute for Public Policy, 2015).
CONCLUSION: For all economic evaluation methods, one or more types of uncertainty usually are associated with the evaluation findings. The literature supports the following practices for addressing uncertainty in high-quality economic evaluations:
- An emerging best practice for providing a comprehensive assessment of the implications of multiple sources of uncertainty is the use of Monte Carlo methods—either alone or in combination with one-way sensitivity analyses.
- In the case of benefit-cost analyses, a recommended summary measure from Monte Carlo simulations is the proportion of trials with positive net benefits.
- Conducting economic evaluations of interventions for children, youth, and families using a 3 percent discount rate as a base case or in a sensitivity analysis will facilitate the ability to make more and consistent comparisons across studies.
In Chapter 2, equity is identified as an important concern in resource allocation targeting children, youth, and families. Equity considerations range from progressivity (vertical equity), to equal treatment of equals
(horizontal equity or equal justice), to individual equity (the right to returns from one’s efforts and labor). Yet, as acknowledged in Chapter 2 and in the paper by Cookson (2015) commissioned for this study, the economic evaluation methods discussed in this chapter typically do not take distributional issues into account. Results are reported in the aggregate for society or the stakeholder whose perspective is adopted. To some extent, the distributional consequences of an intervention can be discerned by disaggregating the total results by different stakeholder groups, such as intervention participants, the rest of society, and the government sector. There may be interest, however, in taking into account the distribution of costs and benefits according to particular characteristics of the participants, such as their income level, race-ethnicity, geographic location, or other defining features that are valued by the decision maker.
Accordingly, some economic evaluations assign differential weights to the costs and benefits accruing to different subgroups affected by an intervention based on the differential means or needs of the subgroups or variation in other socially relevant characteristics. The challenge in defining and applying such weights is to ascertain the appropriate weights to use, as they may vary across different members of the target audience for an economic evaluation. To address this issue, the weights may be determined through formal mechanisms or formulas. The commissioned paper by Cookson (2015) provides much greater detail on how some of these formal mechanisms have evolved to date. Usually they involve some weighting of outcomes according to the initial well-being of the child, youth, or family being served. For instance, a particular dollar of expenditure may be given greater weight if spent on a child in a low-income family rather than a child in a high-income family. Likewise, if the World Bank seeks to allocate some efficient health expenditure on the basis of the capability of recipient countries, it may rank countries along some scale, such as per capita income. Yet even that scalar requires determining how differently to treat desperately poor countries and those with income somewhat below the median.
Of course, income is not the only measure of progressivity. Consider educational attainment. It is not uncommon for an educational intervention to rate its success by the percentage of those below some educational standard who are brought above that standard. A BCA examining such an intervention, however, needs somehow to count the gains to those already above the standard, or losses to the extent that resources were shifted from them.
Often more informal methods will be applied that may or may not involve some amount of quantitative assessment. For instance, a decision may be made to allocate spending to interventions with a wide range of progressivity but not to regressive interventions. Even here, care needs to be exercised in reporting whether an intervention is progressive. A regressive
intervention making use of progressive taxation may still be progressive overall. By some measures, for example, Social Security survivor benefits provided to children might generate higher benefits for children from higher-income families, but through the intervention’s tax structure, still redistribute benefits from those who are better off to those who are worse off (Steuerle and Bakija, 1994).
Ultimately, both formal and informal approaches to incorporating equity weights can be controversial. Yet it is known that decision makers often bring such equity considerations to the table in making decisions about resource allocation. Explicit attempts to incorporate equity considerations, especially as part of sensitivity analyses, can help illuminate the distributional consequences of a given intervention while still acknowledging that the weights any given decision maker would use may differ from those applied in the analysis (Karoly, 2012).
CONCLUSION: Acknowledging equity concerns can enhance the quality and usefulness of economic evaluations. Presenting results disaggregated by key stakeholder groups provides one way of demonstrating the distributional consequences of an intervention (e.g., the costs and benefits for intervention participants versus the rest of society or for relevant subgroups of intervention participants). Applying distributional weights is a more controversial approach but one that can be informative, especially in the context of sensitivity analyses.
“Regarding comparability, if you have a system where the studies are put in looking at everything in a comparable way, it is several orders of magnitude easier to do those kinds of comparisons.”
—Dan Rosenbaum, senior economist, Economic Policy Division, Office of Management and Budget, in the committee’s open session on March 23, 2015.
“One of the things that generates the most angst among ReadyNation members is that the estimates are all over the ballpark. One of the issues that we work on is early childhood education. What is the benefit for preschool: $7.00 for every $1.00; $14.00; $100.00? When members hear that, it makes them very suspicious. Having some standards around that is really important.”
—Sandra Bishop-Josef, deputy director of research, ReadyNation, in the committee’s open session on March 23, 2015.
“There is a need for some type of standardization when it comes to displaying information about the evidence, the costs, and the benefits, and then ultimately the economic evidence. We have seen that there are very limited standards for the documentation of evidence.”
—Danielle Berfond, consultant, The Bridgespan Group, in the committee’s open session on June 1, 2015.
The purpose of this chapter has been to address issues involved in producing high-quality evidence from three major economic evaluation methods: CA, CEA, and BCA. The committee determined this focus was warranted because to truly inform and be useful for decisions about investing in interventions for children, youth, and families, evidence needs to be derived using the best available methods and practices for the question(s) at hand. The evidence also needs to be communicated in a way that supports its credibility and fosters its appropriate use by decision makers. Achieving each of these goals can be a challenge given the complexity of the methods and the multiple decisions entailed in carrying them out, as well as the inevitable uncertainty in the estimates produced. This section presents the committee’s recommended best practices for producing high-quality economic evidence, drawing on the material presented in previous sections. It also offers recommended best practices for reporting the results of economic evaluations so as to achieve transparency, consistency, and usefulness to decision makers. Although this discussion is geared toward those producing economic evidence, it also should be helpful to consumers of the evidence, particularly with respect to assessing the quality and completeness of the evidence presented to them.
Recommended Best Practices for Producing High-Quality Economic Evidence
As the prior sections of this chapter should make clear, conducting an economic evaluation requires careful consideration of a number of assumptions, decisions, and possible practices to produce economic evidence that is of high quality. In broad terms, attention is needed to several different aspects of the evaluation, each of which was discussed earlier in this chapter: requirements for conducting and defining the scope of the evaluation; issues specific to evaluating intervention costs (relevant to CA, CEA, and BCA), determining intervention impacts (relevant to CEA and BCA), and valuing outcomes (relevant particularly to BCA); choices among several possible summary measures, depending on the method and the evaluation question(s); methods for handling uncertainty; and, where relevant, equity considerations.
The best practices identified by the committee are based on the material presented previously in this chapter and reflect a review of the existing literature, as well as the expert consensus views of the committee members. In many areas, the literature and committee members’ consensus provide clear guidance on best practices. These practices are identified as “best practices for all economic evaluation methods” and as “core practices” for specific evaluation methods. Adopting these practices will help ensure the production of sound economic evidence, facilitate comparisons across different evaluations, and provide information that can help decision makers evaluate alternative investment options. However, the core practices will not fully resolve limits on study comparability because of the many possible sources of difference among interventions and economic evaluation methods and assumptions.
In other areas, the literature and committee members’ views were not as clear-cut. Nonetheless, the committee concluded that several “advancing practices” could be identified for each method. Although not essential or even possible in every evaluation, these practices have the potential to improve the quality and utility of the evidence produced for some evaluation questions and investment decisions. Producers of economic evidence are encouraged to adopt these methods when possible.
Finally, it should be noted that the practices identified by the committee reflect the current state of economic evaluation methods. As the field moves forward, some of the advancing practices listed are likely to become core practices, and additional best practices will undoubtedly be identified.
RECOMMENDATION 1: In support of high-quality economic evaluations, producers29 of economic evidence should follow the best practices delineated in the checklist below for conducting cost analyses (CAs), cost-effectiveness analyses (CEAs), benefit-cost analyses (BCAs), and related methods. Producers should follow the core practices listed and, where feasible and applicable, the advancing practices as well. Consumers of economic evidence should use these recommended best practices to assess the quality of the economic evidence available to inform the investment decisions they are seeking to make.
Checklist of Best Practices for Producing High-Quality Economic Evidence
For All Economic Evaluation Methods, Report the Following:
- — Specify the intervention for the economic evaluation, including a description of the intervention’s purpose, its intended recipients, the intensity and duration of services provided, the approach to implementation, the causal mechanisms, and the intended impact(s).
- — Specify the context in which the intervention was or will be implemented, such as characteristics of the population served; the time, place, and scale of implementation; and other relevant contextual factors.
- — Specify the counterfactual condition, including whether the alternative is no intervention, an alternative intervention, or business as usual. In the case of cost-effectiveness analysis (CEA) and benefit-cost analysis (BCA), ensure that the same counterfactual applies to the cost analysis (CA) and the impacts used for the CEA or BCA.
- — Determine the scope of the economic evaluation, including the type of method to be used and the perspective (and any subperspectives) for the analysis; if the societal perspective is not adopted, discuss limitations of the evidence and/or generate results from the societal perspective in a sensitivity analysis.
- — Determine the currency and reference year for all monetary values.
- — If new taxes will be used to fund the intervention, determine the assumed deadweight loss parameter. If a 0 percent rate is selected (i.e., no deadweight loss), generate results in a sensitivity analysis using loss parameters greater than 0 when accounting for new revenue required to pay for an intervention or for impacts on taxes paid or transfer payments.
- — Determine the time horizon for the analysis, and when costs or outcomes accrue over multiple years, the base case discount rate and age or point in time to which to discount (e.g., start of the intervention or a standardized child age). If a 3 percent discount rate is not selected, generate results using a 3 percent discount rate in a sensitivity analysis.
- — Determine the method for addressing uncertainty, and apply it to generate standard errors and confidence intervals for all summary measures, such as estimates of total (present-discounted-value [PDV]) costs, total (PDV) benefits, net (PDV) benefits, cost-effectiveness and benefit-cost ratios, and internal rate of return.
- — Employ sensitivity analyses to test the robustness of estimates under a variety of assumptions, including alternative discount rates, deadweight loss parameters, and estimates of the societal perspective if not the main perspective.
- — Determine whether equity issues need to be addressed.
- — Follow the reporting guidelines on the checklist for best practices for reporting economic evidence below.
- — Value all resources needed to implement the intervention, including infrastructure needs.
- — Use shadow prices to derive an accurate estimate of the value of a resource when a market price is not available.
- — Allocate overhead costs based on use.
- — Annuitize capital investments.
- — Calculate total costs and cost components: fixed, variable, and marginal costs.
- — Calculate unit costs (e.g., cost per participant) to facilitate implementation and replication.
Advancing Practices (all core practices plus the following):
- — Prospectively plan for cost analyses to be integrated into program evaluation.
- — Use micro costing procedures whenever possible to improve the quality of intervention cost estimates and facilitate implementation and replication.
- — Define major intervention activities and identify costs associated with each, including who bears those costs.
- — Estimate costs for intervention planning, development, and adoption separately from those for intervention implementation.
- — Use Monte Carlo methods to evaluate simultaneously the implications of multiple sources of uncertainty.
- — Develop or modify budgetary and other management information systems to include relevant cost categories.
For CEA and Related Methods (in addition to best practices for CA)
- — Determine an explicit rationale for including intervention impacts
in the CEA and selecting the focal impact that will not be valued in the monetary unit. All included impacts should be attributable to the intervention’s theory of change. When available and relevant to the evaluation question(s), use information from well-conducted systematic reviews and/or meta-analyses to inform intervention impact estimates.
- — Determine whether the CEA will use a quality-of-life measure (e.g., quality-adjusted life years [QALYs], disability-adjusted life years [DALYs]) as the focal impact and what method will be used for scoring that measure.
- — Determine whether the CEA will be limited to direct, observable economic impacts, or linked or projected impacts also will be included.
- — For impacts valued in the monetary unit (if any), use willingness-to-pay methods to calculate their prices. This may mean using a combination of market prices and shadow prices.
- — Calculate the average cost-effectiveness ratio and, where feasible, the incremental cost-effectiveness ratio.
Advancing Practices (all core practices plus the following):
- — Conduct CEA only when an intervention has been evaluated using research designs that can produce unbiased causal estimates of impact.
- — Conduct CEA from a societal perspective to produce the most comprehensive economic estimates.
- — Link or project observed outcomes only when strong causal evidence of the assumed relationship exists.
- — Estimate costs and benefits separately by perspective (e.g., participant, agency, government, other beneficiary) and by category (e.g., income, crime, health care).
- — Use Monte Carlo methods to evaluate simultaneously the implications of multiple sources of uncertainty.
For BCA and Related Methods (in addition to best practices for CA)
- — Determine an explicit rationale for including intervention impacts in the BCA. All included impacts should be attributable to the intervention’s theory of change. When available and relevant to the evaluation question(s), use information from well-conducted systematic reviews and/or meta-analyses to inform intervention impact estimates.
- — Determine whether the BCA will be limited to direct, observable economic impacts, or linked or projected impacts also will be included.
- — Determine whether the BCA will include intangible as well as tangible economic impacts.
- — Use willingness-to-pay methods to calculate prices for impacts. This may mean using a combination of market and shadow prices.
- — Estimate linked or projected economic impacts using the strongest available theoretical and empirical literature. When available, use information from well-conducted systematic reviews and/or meta-analyses to inform estimates used for linking and projections.
- — Calculate PDV costs, benefits, and net benefits (total and unit). Where relevant, also calculate benefit-cost ratio, return on investment, and internal rate of return.
- — When there is concern that impact estimates may be biased (e.g., nonexperimental design, quasi-experimental design), test the robustness of findings to variation in effect size.
Advancing Practices (all core practices plus the following):
- — Conduct BCA only when an intervention has been evaluated using research designs that can produce unbiased causal estimates of impact.
- — Conduct BCA from a societal perspective to produce the most comprehensive economic estimates.
- — Link or project observed outcomes only when strong causal evidence of the assumed relationship exists.
- — Generate tangible and intangible values separately.
- — Estimate costs and benefits separately by perspective (e.g., participant, agency, government, other beneficiary) and by category (e.g., income, crime, health care).
- — Use Monte Carlo methods to evaluate simultaneously the implications of multiple sources of uncertainty.
Recommended Best Practices for Reporting Economic Evidence
How the results of an economic evaluation are reported can greatly influence the credibility of the findings and the usefulness of the evidence for the target audience. Throughout this chapter, in the discussion of methods for producing high-quality economic evidence, it has been noted that best practice also dictates reporting the evidence in a comprehensive and transparent manner. Recommendation 2 includes best practices for reporting the results of economic evaluations. As in the best practices under Recom-
mendation 1, those practices listed first are appropriate regardless of the economic evaluation method employed, while those listed subsequently are specific to CA, CEA, and BCA.
Adopting these reporting practices will ensure that information about the methods and findings of an economic evaluation is made available in a thorough and consistent manner, and that issues pertinent to assessing the internal and external validity of the findings, as well as the comparability of findings across different studies, are conveyed. To avoid overwhelming users with analytic details that could obscure the bottom line, it may be helpful to prepare a brief summary report along with a separate technical appendix detailing assumptions and methods. Producing clear and comprehensive reports will strengthen the credibility of the evidence derived from economic evaluation for users and facilitate its appropriate use by decision makers.
RECOMMENDATION 2: In support of high-quality and useful economic evaluations of interventions for children, youth, and families, producers of economic evidence should follow the best practices delineated in the checklist below for reporting the results of cost analyses, cost-effectiveness analyses, and benefit-cost analyses, and related methods.
Checklist of Best Practices for Reporting Economic Evidence
For All Economic Evaluation Methods, Report the Following:
- — The features of the intervention analyzed (e.g., logic model, intended recipients, intensity and duration of services, implementation, and other intervention features)
- — The context in which the intervention was or will be implemented (e.g., population served; time, place, and scale of operation)
- — The counterfactual (baseline or status quo) with which the intervention is compared
- — The perspective for the analysis and any subperspectives examined, with associated results
- — The currency and reference year for all monetary values
- — The assumed deadweight loss parameter, if one was used
- — The horizon for measuring economic values and, when discounting is used, the discount rate and time (or age) to which discounted
- — Summary measures of the economic evaluation results (see below for each specific method)
- — When relevant, results disaggregated by stakeholder
- — The approach for addressing uncertainty, details on how the method was implemented, and the associated standard errors or confidence intervals for all summary measures
- — Sensitivity analyses performed and associated results*
- — When relevant, any equity considerations
For CEA, BCA, and Related Methods That Employ Impact Estimates Also Report:
- — The evaluation method, the intervention impacts* and their statistical significance,* potential biases in estimates of causal effects, and any adjustments to estimated intervention impacts
- — All limitations resulting from the strength of the evidence of causal intervention impacts
In Addition to the Elements for All Methods, for CA and the CA Component of a CEA or BCA Also Report:
- — The costing method (e.g., micro costing)
- — The inventory of resources used and those that are valued versus not valued in the CA
- — The method for obtaining information on how much of each resource is used, any related assumptions made, and how much of each resource is used
- — The method for obtaining unit costs, prices, or shadow prices for each type of resource; any related assumptions made; and the resulting values*
- — Total costs and unit cost (e.g., cost per participant)
- — Fixed, variable, and marginal costs
- — The implications of methods (e.g., omission of resources, prices applied) for under- or overestimating intervention costs
In Addition to the Elements for All Methods and for CA, for a CEA Also Report:
- — Which impacts measured in the evaluation are valued in the CEA and which are not*
- — Which impacts are observed versus linked or projected, for whom they are linked or projected, and the linking or projection method — For the impacts valued in the monetary unit (if any), the prices used,* their derivation, and the geographic or jurisdictional boundary to which the valuations apply*
- — If the focal impact is a quality-of-life measure (e.g., QALYs, DALYs), how that measure was scored
- — The average cost-effectiveness ratio and, where feasible, the incremental cost-effectiveness ratio
- — The implications of methods (e.g., omission of resources in CA, prices applied in CA, causal evidence on outcomes, linkages or projections of outcomes, valuation for outcomes) for under- or overestimating cost-effectiveness
In Addition to the Elements for All Methods and for CA, for a BCA Also Report:
- — Which impacts measured in the evaluation are valued in the BCA and which are not*
- — Which impacts are observed versus linked or projected, for whom they are linked or projected, and the linking or projection method
- — For each impact valued, the price or shadow price used,* its derivation, and the geographic or jurisdictional boundary to which the valuation applies*
- — PDV societal costs, benefits, and net benefits
- — Benefit-cost ratio, return on investment, and/or internal rate of return
- — The PDV benefits (or costs) of each outcome valued,* with disaggregation by outcomes observed versus projected and, where possible and relevant, by tangible versus intangible benefits (e.g., for crime or child abuse and neglect)
- — The implications of methods (e.g., omission of resources in CA, prices applied in CA, causal evidence on outcomes, exclusion of outcomes, linkages or projections of outcomes, valuation for outcomes) for under- or overestimating intervention net benefits
NOTE: An asterisk denotes reporting that may be suitable for a table.
Academy of Managed Care Pharmacy. (2012). A Format for Submission of Clinical and Economic Evidence of Pharmaceuticals in Support of Formulary Consideration. Alexandria, VA: Academy of Managed Care Pharmacy.
Agnihotri, K., Awasthi, S., Singh, U., Chandra, H., and Thakur, S. (2010a). A study of concordance between adolescent self-report and parent-proxy report of health-related quality of life in school-going adolescents. Journal of Psychosomatic Research, 69(6), 525-532.
Agnihotri, K., Awasthi, S., Chandra, H., Singh, U., and Thakur, S. (2010b). Validation of WHO QOL-BREF instrument in Indian adolescents. The Indian Journal of Pediatrics, 77(4), 381-386.
Alberini, A., and Ščasný, M. (2011). Context and the VSL: Evidence from a stated preference study in Italy and the Czech Republic. Environmental and Resource Economics, 49(4), 511-538.
Alberini, A., Loomes, G., Ščasný, M., and Bateman, I. (2010). Valuation of Environment-Related Health Risks for Children. Paris, France: OECD.
Al-Fayez, G.A., and Ohaeri, J.U. (2011). Profile of subjective quality of life and its correlates in a nationwide sample of high school students in an Arab setting using the WHOQOLBref. BMC Psychiatry, 11(71).
American Humane Association. (2009). Replicating the Family Connections Program: Lessons learned. Protecting Children, 24(3), 1-88.
Andrews, F.M., and Withey, S.B. (1976). Social Indicators of Well-Being. New York: Plenum Press.
Aos, S., Lieb, R., Mayfield, J., Miller, M., and Pennucci, A. (2004). Benefits and Costs of Prevention and Early Intervention Programs for Youth. Olympia: Washington State Institute for Public Policy.
Argyle, M. (2001). The Psychology of Happiness. New York: Taylor & Francis.
Arrow, K., Solow, R., Portney, P.R., Leamer, E.E., Radner, R., and Schuman, H. (1993). Report of the NOAA Panel on Contingent Valuation. Federal Register, 58(10), 4601-4614.
Austin, P.C. (2009). The relative ability of different propensity score methods to balance measured covariates between treated and untreated subjects in observational studies. Medical Decision Making. 29(6), 661-677.
Aziz, S.N., Boyle, K.J., and Crocker, T.D. (2007). Valuation of Avoiding Arsenic in Drinking Water in Rural Bangladesh: An Averting Behavior Analysis (Doctoral dissertation, Doctoral thesis, The University of Maine). Available: http://www.ncbi.nlm.nih.gov/pmc/articles/PMC4058837/pdf/fpubh-02-00057.pdf [March 2016].
Barnett, P.G. (2009). An improved set of standards for finding cost for cost-effectiveness analysis. Medical Care, 47(7, Suppl. 1), S82-S88.
Barton, B.K., and Schwebel, D.C. (2007). The roles of age, gender, inhibitory control, and parental supervision in children’s pedestrian safety. Journal of Pediatric Psychology, 32(5), 517-526.
Bassok, D., Fitzpatrick, M., and Loeb, S. (2014). Does state preschool crowd-out private provision? The impact of universal preschool on the childcare sector in Oklahoma and Georgia. Journal of Urban Economics, 83, 18-33.
Basu, A., Heckman, J.J., Navarro-Lozano, S., and Urzua, S. (2007). Use of instrumental variables in the presence of heterogeneity and self-selection: An application to treatments of breast cancer patients. Health Economics, 16(1), 1133-1157.
Bateman, I.J., Carson, R.T., Day, B., Hanemann, M., Hanley, N., Hett, Jones-Lee, M., Loomes, G., Mourato, S., Ozedemiroglu, E., Pearce D.W., Sugden, R., and Swanson, J. (2002). Economic Valuation with Stated Preference Techniques: A Manual. Cheltenham, UK: Edward Elgar.
Belfield, C.R. (2014). Cost-benefit analysis and cost-effectiveness analysis. In H.F. Ladd and M.E. Goertz (Eds.), Handbook of Research in Education Finance and Policy (pp. 141-156). New York: Routledge.
Belfield, C.R., Nores, M., Barnett, S., and Schweinhart, L. (2006). The High/Scope Perry Preschool Program. Journal of Human Resources, XLI(1), 162-190.
Belfield, C.R., Bowden, B., Klapp, A., Levin, H., Shand, R., and Zander, S. (2015). The Economic Value of Social and Emotional Learning. Available: http://blogs.edweek.org/edweek/rulesforengagement/SEL-Revised.pdf [March 2016].
Benefit-Cost Analysis Center. (2012). Federal Agency Benefit-Cost Analysis Principles and Standards for Social Programs. Available: http://evans.uw.edu/sites/default/files/public/Federal_Agency_BCA_PS_Social_Programs.pdf [March 2016].
Bickel, P., and Rainer, F. (2005). ExternE Externalities of Energy: Methodology 2005 Update. Available: https://ec.europa.eu/research/energy/pdf/kina_en.pdf [February 2016].
Bitler, M.P. and Karoly L.A. (2015). Intended and unintended effects of the war on poverty: What research tells us and implications for policy. Journal of Policy Analysis and Management, 34(3), 639-696.
Blincoe, L.J., Miller T.R, Zaloshnja, E., and Lawrence, B.A. (2015). The Economic Impact of Motor Vehicle Crashes, 2010. Washington, DC: National Highway Traffic Safety Administration.
Blomquist, G.C. (2004). Self-protection and averting behavior, values of statistical lives, and benefit-cost analysis of environmental policy. Review of Economics of the Household, 2(1), 89-110.
Blomquist, G.C., Miller, T.R., and Levy, D.T. (1996). Values of risk reduction implied by motorist use of protection equipment: New evidence from different populations. Journal of Transport Economics and Policy, 55-66.
Blomquist, G.C., Dickie, M., and O’Conor, R.M. (2011). Willingness to pay for improving fatality risks and asthma symptoms: Values for children and adults of all ages. Resource and Energy Economics 33(2), 410-435.
Bloom, H.S., Michalopoulos, C., and Hill, C.J. (2005). Using experiments to assess nonexperimental comparison-group methods for measuring program effects. In H.S. Bloom (Ed.), Learning More from Social Experiments (pp. 173-235). New York: Russell Sage Foundation.
Boardman, A.E., and Greenberg, D.H. (1998). Discounting and the social discount rate. In F. Thompson and M.T. Green (Eds.), Handbook of Public Finance (pp. 269-318). New York: Marcel Dekker.
Boardman, A.E., Greenberg, D.H., and Vining, A.R. (2011). Cost-Benefit Analysis: Concepts and Practice. Upper Saddle River, NJ: Prentice Hall.
Bosworth, R., Cameron, T.A., and DeShazo, J.R. (2010). Is an ounce of prevention worth a pound of cure? Comparing demand for public prevention and treatment policies. Medical Decision Making, 30(4), E40-E56.
Bosworth, R., Cameron, T.A., and DeShazo, J.R. (2015). Willingness to pay for public health policies to treat illnesses. Journal of Health Economics, 39, 74-88.
Boxall, P.C., Adamowicz, W.L., Swait, J., Williams, M., and Louviere, J. (1996). A comparison of stated preference methods for environmental valuation. Ecological Economics, 18(3), 243-253.
Brandt, S., Vásquez Lavín, F., and Hanemann, M. (2012). Contingent valuation scenarios for chronic illnesses: The case of childhood asthma. Value in Health, 15(8), 1077-1083.
Bridges, J.F.P., Hauber, A.B., Marshall, D., Lloyd, A., Prosser, L.A., Regier, D.A., Johnson, F.R., and Mauskopf, J. (2011). Conjoint analysis applications in health—a checklist: A report of the ISPOR Good Research Practices for Conjoint Analysis Task Force. Value in Health, 14(4), 403-413.
Brodowski, M.L., and Filene, J.H. (2009). Engaging program staff in economic evaluation: Lessons learned and recommendations for practice. Protecting Children, 24(3), 70-77.
Brown, D.S., Johnson, F.R., Poulos, C., and Messonnier, M.L. (2010). Mothers’ preferences and willingness to pay for vaccinating daughters against human papillomavirus. Vaccine, 28(7), 1702-1708.
Brown, D.W., Kowalski, A.E., and Lurie, I.Z. (2015). Medicaid as an investment in children: what is the long-term impact on tax receipts? NBER Working Paper 20835. Cambridge, MA: National Bureau of Economic Research.
Calculating the Costs of Child Welfare Services Workgroup. (2013). Cost Analysis in Program Evaluation: A Guide for Child Welfare Researchers and Service Providers. Washington, DC: Children’s Bureau, Administration for Children and Families, U.S. Department of Health and Human Services. Available: http://www.acf.hhs.gov/sites/default/files/cb/cost_analysis_guide.pdf [March 2016].
Capri, S., Ceci, A., Terranova, L., Merlo, F., and Mantovani, L. (2001). Guidelines for economic evaluations in Italy: Recommendations from the Italian Group of Pharmacoeconomic Studies. Therapeutic Innovation & Regulatory Science, 35(1), 189-201.
Centers for Disease Control and Prevention. (2010). Healthier Worksite Initiative: Logic Model. Available: http://www.cdc.gov/nccdphp/dnpao/hwi/programdesign/logic_model.htm [July 2015].
Chatterji, P., Caffray, C.M., Jones, A.S., Lillie-Blanton, M., and Werthamer, L. (2001). Applying cost analysis methods to school-based prevention programs. Prevention Science, 2(1), 45-55.
Chau, K., Baumann, M., Kabuth, B., and Chau, N. (2012). School difficulties in immigrant adolescent students and roles of socioeconomic factors, unhealthy behaviours, and physical and mental health. BMC Public Health, 12(453).
Cisler, R., Holder, H.D., Longabaugh, R., Stout, R.L., and Zweben, A. (1998). Actual and estimated replication costs for alcohol treatment modalities: Case study from Project MATCH. Journal of Studies on Alcohol and Drugs, 59(5), 503-512.
Cohen, M.A., Rust, R.T., Steen, S., and Tidd, S.T. (2004). Willingness-to-pay for crime control programs. Criminology, 42(1), 89-110.
Cohodes, S., Grossman, D., Kleiner, S., and Lovenheim, M.F. (2014). Effect of Child Health Insurance Access on Schooling. Working Paper 20178. Cambridge, MA: National Bureau of Economic Research.
Commonwealth of Australia. (2006). Handbook of Cost-Benefit Analysis. Australia: Commonwealth of Australia Department of Finance and Administration, Financial Management Group. Available: http://www.finance.gov.au/sites/default/files/Handbook_of_CB_analysis.pdf [March 2016].
Connelly, R., Degraff, D.S., and Willis, R.A. (2004). The value of employer-sponsored child care to employees. Industrial Relations, 43(4), 759-792.
Cook, T.D., and Campbell, D.T. (1979). Quasi-Experimentation: Design and Analysis Issues for Field Settings. Boston, MA: Houghton Mifflin.
Cook, T.D., and Payne, M.R. (2002). Objecting to the objections to using random assignment in educational research. Evidence Matters: Randomized Trials in Education Research, 150-178.
Cook, T.D., Shadish, W.R., and Wong, V.C. (2008). Three conditions under which experiments and observational studies produce comparable causal estimates: New findings from within-study comparisons. Journal of Policy Analysis and Management, 27(4), 724-750.
Cookson, R. (2015). Methods for Incorporating Equity into Economic Evaluation of Social Investments. Commissioned paper for the Committee on the Use of Economic Evidence to Inform Investments in Children, Youth, and Families. Available: http://sites.nationalacademies.org/cs/groups/dbassesite/documents/webpage/dbasse_171854.pdf [June 2016].
Corso, P.S., Mercy, J., Simon, T., Finkelstein, E., and Miller, T. (2007). Medical costs and productivity losses due to interpersonal and self-directed violence in the United States. American Journal of Preventive Medicine, 32(6), 474-482.
Corso, P.S., Fang, X., and Mercy, J.A. (2011). Benefits of preventing a death associated with child maltreatment: Evidence from willingness-to-pay survey data. American Journal of Public Health, 101(3), 487-490.
Corso, P.S., Ingels, J., and Roldos, M. (2013). A comparison of willingness to pay to prevent child maltreatment deaths in Ecuador and the United States. International Journal of Environmental Research and Public Health, 10(4), 1342-1355.
Cowen, T., and Parfit, D. (1992). Against the social discount rate. In P. Laslett and J. Fishkin (Eds.), Philosophy, Politics, and Society (pp. 144-161). New Haven, CT: Yale University Press.
Crowley, D.M., Jones, D.E., Greenberg, M.T., Feinberg, M.E., and Spoth, R.L. (2012). Resource consumption of a diffusion model for prevention programs: The PROSPER delivery system. Journal of Adolescent Health, 50(3), 256-263.
Crowley, D.M., Hill, L. G., Kuklinski, M.R., and Jones, D.E. (2014). Research priorities for economic analyses of prevention: Current issues and future directions. Prevention Science, 15(6), 789-798.
Cruz, M.L.S., Cardoso, C.A.A., Darmont, M.Q., Souza, E., Andrade, S.D., D’al Fabbro, M.M., Fonseca, R., Bellido, J.G., Monteiro, S.S., and Bastos, F.I. (2014). Viral suppression and adherence among HIV-infected children and adolescents on antiretroviral therapy: Results of a multicenter study. Jornal de Pediatria, 90(6), 563-571.
Cutler, D.M., and Richardson, E. (1998). The value of health: 1970-1990. American Economic Review, 88(2), 97-100.
Derzon, J.H., Miller, T., and Zaloshnja, E. (2005). Recommended Cost Bands for Substance Abuse Prevention. Calverton, MD: Pacific Institute for Research and Evaluation.
Detsky, A.S., and Naglie, I.G. (1990). A clinician’s guide to cost-effectiveness analysis. Annals of Internal Medicine, 113(2), 147-154.
Dickie, M., and Gerking, S. (2002). Willingness to Pay for Reduced Morbidity. Presentation at the workshop Economic Valuation of Health for Environmental Policy: Assessing Alternative Approaches, March, 18-19, Orlando, FL.
Dickie, M., and Gerking, S. (2009). Family behavior: Implications for health benefits transfer from adults to children. Environmental and Resource Economics, 43(1), 31-43.
Dickie, M., and Messman, V.L. (2004) Parental altruism and the value of avoiding acute illness: are kids worth more than parents? Journal of Environmental Economics and Management, 48(3), 1146-1174.
Dickie, M., and Ulery, V.L. (2001, June). Valuing health in the household: Are kids worth more than parents? Association of Environmental and Resource Economists 2001 Workshop: Assessing and Managing Environmental and Public Health Risks, 13-15.
Diener, E., and Oishi, S. (2005). The nonobvious social psychology of happiness. Psychological Inquiry, 16(4), 162-167.
Diener, E., Suh, E.M., Lucas, R.E., Smith, H.E. (1999). Subjective well-being: Three decades of progress. Psychological Bulletin, 125(2), 276-302.
Donaldson, S.I., Christie, C.A., and Mark, M.M. (2008). What Counts as Credible Evidence in Applied Research and Evaluation? London: Sage. Available: http://cgu.edu/PDFFiles/sbos/Donaldson_Credible_Evidence_Epilogue.pdf [March 2016].
Dréze, J.H. (1964). Some postwar contributions of French economists to theory and public policy: With special emphasis on problems of resource allocation. Amercian Economic Review, 54(4), 2-64.
Drummond, M.F., and Jefferson, T.O. (1996). Guidelines for authors and peer reviewers of economic submissions to the BMJ. British Medical Journal, 313(7052), 275-283.
Drummond, M.F., and Sculpher, M. (2005). Common methodological flaws in economic evaluations. Medical Care, 43(Suppl. 7), II-5-II-14.
Drummond, M.F., Sculpher, M.J., Torrance, G.W., O’Brien, B.J., and Stoddart, G.L. (2005). Methods for the Economic Evaluation of Health Care Programmes. New York: Oxford University Press.
Escobar, C.M., Barnett, W.S., and Keith, J.E. (1988). A contingent valuation approach to measuring the benefits of preschool education. Educational Evaluation and Policy Analysis, 10(1), 13-22.
European Commission. (2008). Guide to Cost-Benefit Analysis of Investment Projects: Structural Funds, Cohesion Fund and Instrument for Pre-Accession. Available: http://ec.europa.eu/regional_policy/sources/docgener/guides/cost/guide2008_en.pdf [February 2016].
European Regional Development Fund. (2013). Guidance Manual for Cost Benefit Analysis Appraisal in Malta. Malta: Parliamentary Secretariat for the EU Presidency 2017.
Fanshel, S., and Bush, J.W. (1970). A health-status index and its application to health-services outcomes. Operations Research, 18(6), 1021-1066.
Farrow, S., and Zerbe, R.O. (Eds.). (2013). Principles and Standards for Benefit-Cost Analysis. Cheltenham, UK: Edward Elgar.
Federal Accounting Standards Advisory Board. (2014). FASAB Handbook of Federal Accounting Standards and Other Pronouncements, as Amended. Washington, DC: Federal Accounting Standards Advisory Board.
Fisher, C.B., Hoagwood, K., Boyce, C., Duster, T., Frank, D.A., Grisso, T., Levine, R.J., Macklin, R., Spencer, M.B., Takanishi, R., Trimble, J.E., and Zayas, L.H. (2002). Research ethics for mental health science involving ethnic minority children and youths. American Psychologist, 57(12), 1024-1040.
Foster, E.M., Dodge, K.A., and Jones, D. (2003). Issues in the economic evaluation of prevention programs. Applied Developmental Science, 7(2), 76-86.
Foster, E.M., Porter, M.M., Ayers, T.S., Kaplan, D.L., and Sandler, I. (2007). Estimating the costs of preventive interventions. Evaluation Review, 31(3), 261-286.
Gayer, T., Hamilton, J.T., and Viscusi, W.K. (2002). The market value of reducing cancer risk: Hedonic housing prices with changing information. Southern Economic Journal, 69(2), 266-289.
Gentilello, L.M., Ebel, B.E., Wickizer, T.M., Salkever, D.S., and Rivara, F.P. (2005). Alcohol interventions for trauma patients treated in emergency departments and hospitals: A cost-benefit analysis. Annals of Surgery, 241(4), 541-550.
Gerber, A.S., Greena, D.P., and Carnegie, A.J. (2013). Evaluating public health law using randomized experiments. In A.C. Wagenaar and S.C. Burris (Eds.), Public Health Law Research: Theory and Methods (pp. 283-305). Somerset, NJ: Wiley.
Gill, A.M., Dishion, T.J., and Shaw, D.S. (2014). The family check-up. In Wellbeing: A Complete Reference Guide (pp. 1-21). Hoboken, NJ: Wiley-Blackwell.
Gold, M.R., Siegel, J.E., Russell, L.B., and Weinstein, M.C. (Eds.). (1996). Cost-Effectiveness in Health and Medicine. New York: Oxford University Press.
Gogtay, N., Giedd, J.N., Lusk, L., Hayashi, K.M., Greenstein, D., Vaituzis, A.C., Nugent, T.F., Herman, D.H., Clase, L.S., Toga, A.W., and Rapoport, J.L. (2004). Dynamic mapping of human cortical development during childhood through early adulthood. Proceedings of the National Academy of Sciences, 101(21), 8174-8179.
Gottfredson, D.C., Cook, T.D., Gardner, F.E.M., Gorman-Smith, D., Howe, G.W., Sandler, I.N., and Zafft, K.M. (2015). Standards of evidence for efficacy, effectiveness, and scale-up research in prevention science: Next generation. Prevention Science, 16(7), 893-926.
Graf von der Schulenburg, J.M., and Hoffmann, C. (2000). Review of European guidelines for economic evaluation of medical technologies and pharmaceuticals. Health Economics in Prevention and Care, 1(1), 2-8.
Gray, A.M., Rivero-Arias, O., and Clarke, P. (2006). Estimating the association between SF-12 responses and EQ-5D utility values by response mapping. Medical Decision Making, 26(1), 18-29.
Gray, A.M., Clarke, P.M., Wolstenholme, J.L., and Wordsworth, S. (2010). Applied Methods of Cost-Effectiveness Analysis in Healthcare. New York: Oxford University Press.
Greenberg, D.H., and Appenzeller, U. (1998). Cost Analysis Step by Step: A How-to Guide for Planners and Providers of Welfare-to-Work and Other Employment and Training Programs. New York: Manpower Demonstration Research Corporation.
Gyrd-Hanson, D. (2003). Willingness to pay for a QALY. Health Economics, 12(12), 1049-1060.
Haddix, A.C., Teutsch, S.M., and Corso, P.S. (2003). Prevention Effectiveness: A Guide to Decision Analysis and Economic Evaluation. New York: Oxford University Press.
Hammitt, J.K. (2007). Valuing changes in mortality risk: Lives saved versus life years saved. Review of Environmental Economics and Policy, 1(2), 228-240.
Hammitt, J.K., and Haninger, K. (2010). Valuing fatal risks to children and adults: Effects of disease, latency, and risk aversion. Journal of Risk and Uncertainty, 40(1), 57-83.
Hanemann, W.M. (1994). Valuing the environment through contingent valuation. Journal of Economic Perspectives, 19-43.
Haninger, K., and Hammitt, J.K. (2011). Diminishing willingness to pay per quality-adjusted life year: Valuing acute foodborne illness. Risk Analysis, 31(9), 1363-1380.
Harris, A.H., Hill, S.R., Chin, G., Li, J.J., and Walkom, E. (2008). The role of value for money in public insurance coverage decisions for drugs in Australia: A retrospective analysis 1994-2004. Medical Decision Making, 28(5), 713-722.
Hausman, J. (2012). Contingent valuation: From dubious to hopeless. Journal of Economic Perspectives, 26(4), 43-56.
Haute Autorité de Santé. (2012). Choices in Methods for Economic Evaluation. Saint-Denis La Plaine, France: Department of Economics and Public Health Assessment, Haute Autorité de Santé.
Hawkins, J.D., Catalano, R.F., and Kuklinski, M.R. (2014). Communities that care. Encyclopedia of Criminology and Criminal Justice, 393-408.
Heckman, J.J., Moon, S.H., Pinto, R., Savelyev, P., and Yavitz, A. (2010). A New Cost-Benefit and Rate of Return Analysis for the Perry Preschool Program: A Summary. NBER Working Paper 16180. Cambridge, MA: National Bureau of Economic Research.
Heckman, J.J., Humphries, J.E., Veramendi, G., and Urzua, S. (2014). Education, Health and Wages. Cambridge, MA: National Bureau of Economic Research.
Hendrie, D. (2013). Economic Analysis of Health Sponsorship in Australia. Unpublished Doctoral Thesis.
Her Majesty’s Treasury. (2003). The Green Book: Appraisal and Evaluation in Central Government: Treasury Guidance. London, UK: Stationery Office.
Hibbs, E.D., Clarke, G., Hechtman, L., and Abikoff, H.B. (1997). Manual development for the treatment of child and adolescent disorders. Psychopharmacology Bulletin, 33(4), 619.
Hjelmgren, J., Berggren, F., and Andersson, F. (2001). Health economic guidelines—similarities, differences and some implications. Value in Health, 4(3), 225-250.
Honeycutt, A., Clayton, L., Khavjou, O., Finkelstein, E., Prabhu, M., Blitstein, J., Evans, W., and Renaud, J. (2006). Guide to Analyzing the Cost-Effectiveness of Community Public Health Prevention Approaches. Available: http://aspe.hhs.gov/pdf-report/guide-analyzing-cost-effectiveness-community-public-health-prevention-approaches [June 2016].
Hunink, M.G.M., Glasziou, P.P., Siegel, J.E., Weeks, J.C., Pliskin, J.S., Elstein, A.S., and Weinstein, M.C. (2001). Decision Making in Health and Medicine. Cambridge, UK: Cambridge University Press.
Institute for Quality and Efficiency in Health Care. (2009). General Methods for the Assessment of the Relation of Benefits to Costs. Cologne, Italy: Institute for Quality and Efficiency in Health Care.
Institute of Medicine. (2006). Valuing Health for Regulatory Cost-Effectiveness Analysis. W. Miller, L.A. Robinson, and R.S. Lawrence (Eds.). Committee to Evaluate Measures of Health Benefits for Environmental, Health, and Safety Regulation; Board on Health Care Services. Washington, DC: The National Academies Press.
Institute of Medicine and National Research Council. (2014). Considerations in Applying Benefit-Cost Analysis to Preventive Interventions for Children, Youth, and Families. S. Olson and K. Bogard (Rapporteurs). Board on Children, Youth, and Families. Washington, DC: The National Academies Press.
Islam, Z., Maskery, B., Nyamete, A., Horowitz, M.S., Yunus, M., and Whittington, D. (2008). Private demand for cholera vaccines in rural Matlab, Bangladesh. Health Policy, 85(2), 184-195.
Izutsu, T., Tsutsumi, A., Islam, M., Matsuo, Y., Yamada, H., Kurita, H., and Wakai, S. (2005). Validity and reliability of the Bangla version of WHOQOL-BREF on an adolescent population in Bangladesh. Quality of Life Research, 14(7), 1783-1789.
Jamison, D.T., Breman, J.G., Measham, A.R., Alleyne, G., Claeson, M., Evans, D.B., Jha, P., Mills, A., and Musgrove, P. (2006). Priorities in Health. Washington, DC: The International Bank for Reconstruction and Development/The World Bank Group.
Jarahi, L., Karbakhsh, M., and Rashidian, A. (2011). Parental willingness to pay for child safety seats in Mashad, Iran. BMC Public Health, 11(1), 281.
Jenkins, N.R., Owens, N., and Wiggins, E. (2001). Valuing reduced risks to children: The case of bicycle safety helmets. Contemporary Economic Issues, 19(4), 397-408.
Jirojanakul, P., and Skevington, S. (2000). Developing a quality-of-life measure for children aged 5-8 years. British Journal of Health Psychology, 5(3), 299-321.
Johannesson, M., and Johansson P.-O. (1997). Is the valuation of a QALY gained independent of age? Some empirical evidence. Journal of Health Economics, 16(5), 589-599.
Johnson, F.R., Lancsar, E., Marshall, D., Kilambi, V., Mühlbacher, A., Regier, D.A., Bresnahan, B.W., Kanninen, B., and Bridges, J.F.P. (2013). Constructing experimental designs for discrete-choice experiments: Report of the ISPOR Conjoint Analysis Experimental Design Good Research Practices Task Force. Value in Health, 16(1), 3-13.
Joint United Nations Programme on HIV/AIDS. (2000). Costing Guidelines for HIV Prevention Strategies. Available: http://data.unaids.org/Publications/IRC-pub05/jc412-costguidel_en.pdf [March 2016].
Jones, A.M., and Rice, N. (2011). Econometric evaluation of health policies. In S. Glied and P.C. Smith (Eds.), The Oxford Handbooks of Health Economics. Oxford, UK: Oxford University Press.
Karoly, L.A. (2008). Valuing Benefits in Benefit-Cost Studies of Social Programs. Technical Report. Santa Monica, CA: RAND.
Karoly, L.A. (2012). Toward standardization of benefit-cost analysis of early childhood interventions. Journal of Benefit-Cost Analysis, 3(1).
Keeler, E.B., and Cretin, S. (1983). Discounting of life-saving and other nonmonetary effects. Management Science, 29(3), 300-306.
Kessler, R.C., Pecora, P.J., Williams, J., Hiripi, E., O’Brien, K., English, D., White, J., Zerbe, R., Downs, A.C., Plotnick, R., Hwang, I., and Sampson, N.A. (2008). Effects of enhanced foster care on the long-term physical and mental health of foster care alumni. Archives of General Psychiatry, 65(6), 625.
Kochi, I., and Taylor, L. (2011). Risk heterogeneity and the value of reducing fatal risks: Further market-based evidence. Journal of Benefit-Cost Analysis, 2(3), Article 1.
Kuklinski, M.R., Fagan, A.A., Hawkins, J.D., Briney, J.S., and Catalano, R.F. (2015). Benefit-cost analysis of a randomized evaluation of communities that care: Monetizing intervention effects on the initiation of delinquency and substance use through grade 12. Journal of Experimental Criminology, 11(2), 165-192.
Laupacis, A., Feeny, D., Detsky, A.S., and Tugwell, P.X. (1992). How attractive does a new technology have to be to warrant adoption and utilization? Tentative guidelines for using clinical and economic evaluations. Canadian Medical Association Journal, 146(4), 473-481.
Lee, S., and Aos, S. (2011). Using cost-benefit analysis to understand the value of social interventions. Research on Social Work Practice, 21(6), 682-688.
Leonard, B. (2009). GAO Cost Estimating and Assessment Guide: Best Practices for Developing and Managing Capital Program Costs. Collingdale, PA: Diane.
Leung, J., and Guria, J. (2006). Value of statistical life: Adults versus children. Accident Analysis and Prevention, 38(6), 1208-1217.
Levin, H.M., and Belfield, C. (2013). Guiding the Development and Use of Cost-Effectiveness Analysis in Education. New York: Center for Benefit-Cost Studies of Education, Columbia University. Available: http://cbcse.org/wordpress/wp-content/uploads/2013/08/Guiding-the-Development-And-Use-of-Cost-effectiveness-Analysis-in-Education.pdf [February 2016].
Levin, H.M., and McEwan, P.J. (2001). Cost-Effectiveness Analysis (2nd Edition). Thousand Oaks, CA: Sage.
Lindhjem, H., Navrud, S., Braathen, N.A., and Biausque, V. (2011). Valuing mortality risk reductions from environmental, transport, and health policies: A global meta-analysis of stated preference studies. Risk Analysis, 31(9), 1381-1407.
Liu, J.-T., Hammitt, J.K., Wang, J.-D., and Liu, J.-L. (2000). Mother’s willingness to pay for her own and her child’s health: A contingent valuation study in Taiwan. Health Economics, 9(4), 319-326.
Loureiro, M.L., and Umberger, W.J. (2007). A choice experiment model for beef: What U.S. consumer responses tell us about relative preferences for food safety, country-of-origin labeling, and traceability. Food Policy, 32(4), 496-514.
Lucas, R.E., Clark, A.E., Georgellis. Y., and Diener, E. (2004). Unemployment alters the set point for life satisfaction. Psychologial Science, 15(1), 8-13.
Luce, B.R., Manning, W.G., Siegel, J.E., and Lipscomb, J. (1996). Estimating costs in cost-effectiveness analysis. In M. Gold, M. Weinstein, J. Siegel, M. Kamlet, and L. Russell (Eds.), Cost-Effectiveness in Health and Medicine (pp. 176-213). New York: Oxford University Press.
Ludwig, J., and Cook, P.J. (2001). The benefits of reducing gun violence: Evidence from contingent-valuation survey data. Journal of Risk and Uncertainty, 22(3), 207-226.
Magnuson, K.A., Ruhm, C., and Waldfogel, J. (2007). The persistence of preschool effects: Do subsequent classroom experiences matter? Early Childhood Research Quarterly, 22(1), 18-38.
Mansley, E.C., Duñet, D.O., May, D.S., Chattopadhyay, S.K., and McKenna, M.T. (2002). Variation in average costs among federally sponsored state-organized cancer detection programs: Economies of scale? Medical Decision Making, 22(5), S67-S79.
Marshall, D.A., and Hux, M. (2009). Design and analysis issues for economic analysis alongside clinical trials. Medical Care, 47(7, Suppl. 1), S14-S20.
Maynard, R.A., and Hoffman, S.D. (2008) The costs of adolescent childbearing. In R.A. Maynard and S.D. Hoffman (Eds.) Kids Having Kids: Economic Costs and Social Consequences of Teen Pregnancy (2nd Edition). Washington, DC: Urban Institute Press.
McDowell, I. (2006). Measuring Health (3rd Edition). New York: Oxford University Press.
Meckley, L.M., Greenberg, D., Cohen, J.T., and Neumann, P.J. (2010). The adoption of cost-effectiveness acceptability curves in cost-utility analyses. Medical Decision Making, 30(3), 314-319.
Messonnier, M., and Meltzer, M. (2003). Cost-benefit analysis. In A.C. Haddix, S.M. Teutsch, and P.S. Corso (Eds.), Prevention Effectiveness: A Guide to Decision Analysis and Economic Evaluation (2nd Edition). New York: Oxford University Press.
Miller, S., and Wherry, L. (2014). The Long-Term Health Effects of Early Life Medicaid Coverage. Working paper. Available: http://www-personal.umich.edu/~mille/millerwherry_prenatal2014.pdf [March 2016].
Miller, T.R. (1990). The plausible range for the value of life––Red herrings among the mackerel. Journal of Forensic Economics, 3(3), 17-39.
Miller, T.R. (2000). Valuing non-fatal quality of life losses with quality-adjusted life years: The health economist’s meow. Journal of Forensic Economics, 13(2), 145-168.
Miller, T.R. (2015). Projected outcomes of Nurse-Family Partnership home visitation during 1996-2013, USA. Prevention Science, 16(6), 765-777.
Miller, T.R., and Hendrie, D.V. (2012). Economic Evaluation of Public Health Laws and Their Enforcement. Available: http://publichealthlawresearch.org/sites/default/files/downloads/resource/EconomicEvaluationPHL-Monograph-MillerHendrie2012.pdf [February 2016].
Miller, T.R., and Hendrie, D. (2013). Economic evaluation of public health laws and their enforcement. In A.C. Wagenaar and S.C. Burris (Eds.), Public Health Law Research: Theory and Methods (pp. 347-378). San Francisco, CA: Jossey-Bass.
Miller, T.R., and Hendrie, D.V. (2015). Nurse family partnership: Comparing costs per family in randomized trials versus scale-up. The Journal of Primary Prevention, 36(6), 419-425.
Miller, T.R., Calhoun, C., and Arthur, W.B. (1989). Utility-adjusted impairment years: A low-cost approach to morbidity valuation. In Estimating and Valuing Morbidity in a Policy Context: Proceedings of June 1989 AERE Workshop. EPA-230-08-89-065. Washington, DC: U.S. Environmental Protection Agency.
Miller, T.R., Fisher, D.A., and Cohen, M.A. (2001). Costs of juvenile violence: Policy implications. Pediatrics, 107(1), E3.
Miller, T.R., Levy, D.T., Cohen, M.A., and Cox, K.L. (2006). The costs of alcohol and drug-involved crime. Prevention Science, 7(4), 333-342.
Mitchell, R.C., and Carson, R.T. (1989). Using Surveys to Value Public Goods: The Contingent Valuation Method. Washington, DC: Resources for the Future.
Mortimer, D., and Segal L. (2008). Is the value of a life or life-year saved context specific? Further evidence from a discrete choice experiment. Cost Effectiveness and Resource Allocation, 6(8).
Mount, T., Weng, W., Schulze, W., and Chestnut, L. (2001, June). Automobile safety and the value of statistical life in the family: Valuing reduced risk for children, adults and the elderly. Association of Environmental and Resource Economists Workshop, Bar Harbor, Maine (Vol. 1315). Available: https://www.researchgate.net/publication/237278864_AUTOMOBILE_SAFETY_AND_THE_VALUE_OF_STATISTICAL_LIFE_IN_THE_FAMILY_VALUING_REDUCED_RISK_FOR_CHILDREN_ADULTS_AND_THE_ELDERLY [June 2016].
Muenning, P., and Khan, K. (2002). Designing and Conducting Cost Effectiveness Analyses in Medicine and Health Care. San Francisco, CA: Jossey-Bass.
Murray, C.J.L., and Lopez, A.D. (Eds.). (1996). The Global Burden of Disease: A Comprehensive Assessment of Mortality and Disability from Diseases, Injuries and Risk Factors in 1990 and Projected to 2020. Cambridge, MA: Harvard University Press.
National Center for Environmental Economics. (2010). Guidelines for Preparing Economic Analyses. Washington, DC: U.S. Environmental Protection Agency.
National Institute for Health and Care Excellence. (2013). Guide to the Methods of Technology Appraisal 2013. London, UK: National Institute for Health and Care Excellence.
National Research Council and Institute of Medicine. (2009). Strengthening Benefit-Cost Analysis for Early Childhood Interventions. A. Beatty (Rapporteur). Committee on Strengthening Benefit-Cost Methodology for the Evaluation of Early Childhood Interventions. Board on Children, Youth, and Families. Division of Behavioral and Social Sciences and Education. Washington, DC: The National Academies Press.
Nelson, C.A., de Haan, M., and Thomas, K.M. (2006). Neuroscience and Cognitive Development: The Role of Experience and the Developing Brain. New York: Wiley.
Nelson, J.P. (1978). Residential choice, hedonic prices, and the demand for urban air quality. Journal of Urban Economics, 5(3), 357-369.
Nichol, K.L. (2001). Cost-benefit analysis of a strategy to vaccinate healthy working adults against influenza. Archives of Internal Medicine, 161(5), 749-759.
OECD. (2012). Mortality Risk Valuation in Environment, Health and Transport Policies. Paris, France: OECD.
Office of Management and Budget. (1992). Circular A-94: Guidelines and Discount Rates for Benefit-Cost Analysis of Federal Programs. Washington, DC: Office of Management and Budget.
Office of Management and Budget. (2003). Circular A-4: Regulatory Analysis. Washington, DC: Office of Budget and Management.
Office of Management and Budget. (2004a). Circular A-21: Cost Principles for Educational Institutions. Available: https://www.whitehouse.gov/omb/circulars_a021_2004 [November 2015].
Office of Management and Budget. (2004b). Circular A-122: Cost Principles for Non-Profit Organizations. Available: https://www.whitehouse.gov/omb/circulars_a122_2004 [November 2015].
Oliver, K., Innvar, S., Lorenc, T., Woodman, J., and Thomas, J. (2014). A systematic review of barriers to and facilitators of the use of evidence by policymakers. BMC Health Services Research, 14(1), 2.
Olsen, J.A., and Smith, R.D. (2001). Theory versus practice: A review of willingness-to-pay in health and health care. Health Economics, 10(1), 39-52.
Ontario Ministry of Health and Long-Term Care. (1994). Ontario Guidelines for Economic Analysis of Pharmaceutical Products. Toronto, ON: Ontario Ministry of Health and Long-Term Care.
Oreopoulos, P., and Petronijevic, U. (2013). Making college worth it: A review of the returns to higher education. The Future of Children, 23(1), 41-65.
Palanca-Tan, R. (2014). Estimating the value of statistical life for children in Manila. In S. Barrett, K.-G., Mäler, and E. S. Maskin (Eds.), Environment and Development Economics. Oxford University Press. Available: https://idl-bnc.idrc.ca/dspace/bitstream/10625/46114/1/132605.pdf [March 2016].
Pew-MacArthur Results First Initiative. (2013). States’ Use of Cost-Benefit Analysis: Improving Results for Taxpayers. Washington, DC: Pew-MacArthur Results First Initiative.
Pharmaceutical Benefits Board. (2003). General Guidelines for Economic Evaluations from the Pharmaceutical Benefits Board. Available: http://www.tlv.se/Upload/English/Guidelinesfor-economic-evaluations-LFNAR-2003-2.pdf [March 2016].
Pritchard, C., and Sculpher, M. (2000). Productivity Costs: Principles and Practice in Economic Evaluation. London, UK: Office of Health Economics.
Psacharopoulos, G., and Patrinos, H.A. (2004). Returns to investment in education: A further update. Education Economics, 12(2), 111-134.
Ramsey, S., Willke, R., Briggs, A., Brown, R., Buxton, M., Chawla, A., Cook, J., Glick, H., Liljas, B., Petitti, D., and Reed, S. (2005). Good research practices for cost-effectiveness analysis alongside clinical trials: The ISPOR RCT-CEA Task Force report. Value in Health, 8(5), 521-533.
Richardson, J., Iezzi, A., Khan, M., and Maxwell, A. (2013). Subjective Wellbeing, Utility and Quality of Life: Results from the Multi Instrument Comparison (MIC) Project. Melbourne, Australia: Centre for Health Economics, Monash University.
Richardson, J., Iezzi, A., and Khan, M.A. (2015). Why do multi-attribute utility instruments produce different utilities: The relative importance of the descriptive systems, scale and ‘micro-utility’ effects. Quality of Life Research, 24(8), 2045-2053.
Ritzwoller, D.P., Sukhanova, A., Gaglio, B., and Glasgow, R.E. (2009). Costing behavioral interventions: A practical guide to enhance translation. Annals of Behavioral Medicine, 37(2), 218-227.
Robert Wood Johnson Foundation. (2013). Return on Investments in Public Health: Saving Lives and Money. Available: http://www.rwjf.org/en/library/research/2013/12/return-oninvestments-in-public-health.html [February 2016].
Robinson, K., Kennedy, N., and Harmon, D. (2012). Happiness: A review of evidence relevant to occupational science. Journal of Occupational Science, 19(2), 150-164.
Robinson, L.A., and Hammitt, J.K. (2011). Behavioral economics and regulatory analysis. Risk Analysis, 31(9), 1408-1422.
Ryan, M., Gerard, K., and Amaya-Amaya, M. (Eds.). (2008). Using Discrete Choice Experiments to Value Health and Health Care. Dordrecht, The Netherlands: Springer.
Scapecchi, P. (2006). Economic Valuation of Environmental Health Risks to Children. Paris, France: OECD.
Schelling, T. (1968). The life you save may be your own. In S.B. Chase (Ed.), Problems in Public Expenditure Analysis (pp. 127-162). Washington, DC: Brookings Institution.
Schelling, T.C. (1995). Intergenerational discounting. Energy Policy, 23(4-5), 395-401.
Schulz, K.F., Altman, D.G., and Moher, D. (2010). CONSORT 2010 statement: Updated guidelines for reporting parallel group randomised trials. BMC Medicine, 8(1), 18.
Schwab-Christe, N.G., and Soguel, N.C. (Eds.). (1995). Contingent Valuation, Transport Safety and the Value of Life. Boston, MA: Kluwer Academic.
Scotton, C.R., and Taylor, L.O. (2011). Valuing risk reductions: Incorporating risk heterogeneity into a revealed preference framework. Resource and Energy Economics, 33(2), 381-397.
Shadish W.R., Cook, T.D., and Campbell, D.T. (2002). Experimental and Quasi-Experimental Designs for Generalized Causal Inference. Boston, MA: Wadsworth Cengage Learning.
Shadish, W.R., Clark, M.H., and Steiner, P.M. (2008). Can nonrandomized experiments yield accurate answers? A randomized experiment comparing random and nonrandom assignments. Journal of the American Statistical Association, 103(484), 1334-1344.
Siegel, J.E., Weinstein, M.C., Russell, L.B., and Gold M.R. (1996). Recommendations for reporting cost-effectiveness analyses. Panel on Cost-Effectiveness in Health and Medicine. Journal of the American Medical Association, 276(16), 1339-1341.
Skevington, S.M., Lotfy, M., and O’Connell, K.A. (2004). The World Health Organization’s WHOQOL-BREF quality-of-life assessment: Psychometric properties and results of the international field trial. Quality of Life Research, 13(2), 299-310.
Smith, T., Scahill, L., Dawson, G., Guthrie, D., Lord, C., Odom, S., Rogers, S., and Wagner, A. (2006). Designing research studies on psychosocial interventions in autism. Journal of Autism and Developmental Disorders, 37(2), 354-366.
Spicer, R.S., Miller, T.R., Hendrie, D., and Blincoe, L (2011). Quality-adjusted life years lost to road crash injury: Updating the Injury Impairment Index. Annals of Advances in Automotive Medicine, 55, 365-377.
Steuerle, C.E., and Bakija, J.M. (1994). Retooling Social Security for the 21st Century: Right and Wrong Approaches to Reform. Washington, DC: Urban Institute.
Suter, C.R. (2010). Economic Evaluation of a Community-Based, Family-Skills Prevention Program. Pullman: Washington State University.
Takenaga, N., Kai, I., and Ohi, G. (1985). Evaluation of three cervical cancer detection programs in Japan with special reference to cost-benefit analysis. Cancer, 55(10), 2514-2519.
Takeuchi, K., Kishimoto, A., and Tsuge, T. (2008). Altruism and Willingness to Pay for Reducing Child Mortality. Kobe, Japan: Graduate School of Economics, Kobe University.
Task Force on Community Preventive Services. (2005). The Guide to Community Preventive Services: What Works to Promote Health? New York: Oxford University Press.
The Cochrane Public Health Group. (2011). Guide for Developing a Cochrane Protocol. London, UK: The Cochrane Public Health Group.
Thomas, A. (2012). Three strategies to prevent unintended pregnancy. Journal of Policy Analysis and Management, 31(2), 280-311.
Thompson, E., Berger, M., Blomquist, G., and Allen, S. (2002). Valuing the arts: A contingent valuation approach. Journal of Cultural Economics, 26(2), 87-113.
Thorat, T., Cangelosi, M., and Neumann, P.J. (2012). Skills of the trade: The Tufts Cost-Effectiveness Analysis Registry. Journal of Benefit-Cost Analysis, 3(1), 1-9.
Torrance, G.W., Thomas, W.H., and Sackett, D.L. (1972). A utility maximization model for evaluation of health care programs. Health Services Research, 7(2), 118-133.
Treasury Board of Canada Secretariat. (2007). Canadian Cost-Benefit Analysis Guide: Regulatory Proposals. Available: https://www.tbs-sct.gc.ca/rtrap-parfa/analys/analys-eng.pdf [March 2016].
Trochim, W.M.K., and Donnelly, J.P. (2006). Research Methods Knowledge Base (3rd Edition). Boston, MA: Cengage Learning.
Valentine, J.C., and Konstantopoulos, S. (2015). Using Systematic Reviews and Meta-Analyses to Inform Public Policy Decisions. Commissioned paper for the Committee on the Use of Economic Evidence to Inform Investments in Children, Youth, and Families. Available: http://sites.nationalacademies.org/cs/groups/dbassesite/documents/webpage/dbasse_171853.pdf [June 2016].
Vincent, D., Oakley, D., Pohl, J., and Walker, D.S. (2000). Cost effectiveness analysis: An essential tool for practice management. The Nurse Practitioner, 25(9), 95-96.
Vining, A.R., and Weimer, D.L. (2009a). An agenda for promoting and improving the use of CBA in social policy. In D.L. Weimer and A.R. Vining (Eds.), Investing in the Disadvantaged: Assessing the Costs and Benefits of Social Policies (pp. 249-271). Washington, DC: Georgetown University Press.
Vining, A.R., and Weimer, D.L. (2009b). Assessing the costs and benefits of social policies. In D.L. Weimer and A.R. Vining (Eds.), Investing in the Disadvantaged: Assessing the Costs and Benefits of Social Policies (pp. 1-16). Washington, DC: Georgetown University Press.
Vining, A.R., and Weimer, D.L. (2009c). Overview of the state-of-the-art CBA in social policy. In D.L. Weimer and A.R. Vining (Eds.), Investing in the Disadvantaged: Assessing the Costs and Benefits of Social Policies (pp. 219-248). Washington, DC: Georgetown University Press.
Vining, A.R., and Weimer, D.L. (2010). An assessment of important issues concerning the application of benefit-cost analysis to social policy. Journal of Benefit-Cost Analysis, 1(1), 1-40.
von Stackelberg, K., and Hammitt, J. (2009). Use of contingent valuation to elicit willingness-to-pay for the benefits of developmental health risk reductions. Environmental and Resource Economics, 43(1), 45-61.
Wagenaar, A.C., and Komro, K.A. (2013). Natural experiments: Research design elements for optimal causal inference without randomization. In A.C. Wagenaar and S.C. Burris (Eds.), Public Health Law Research: Theory and Methods (Ch. 14, pp. 307-324). San Francisco, CA: Jossey-Bass.
Walker, D. (2001). Cost and cost-effectiveness guidelines: Which ones to use? Health Policy and Planning, 16(1), 113-121.
Walter, E., and Zehetmayr, S. (2006). Guidelines on Health Economic Evaluation. Consensus Paper. Vienna, Austria: Institut für Pharmaökonomische Forschung. Available: http://www.ispor.org/peguidelines/source/Guidelines_Austria.pdf [March 2016].
Washington State Institute for Public Policy. (2015). Benefit-Cost Technical Documentation. Olympia: Washington State Institute for Public Policy.
Weinstein, M.C., and Stason, W.B. (1977). Foundations of cost-effectiveness analysis for health and medical practices. New England Journal of Medicine, 296(13), 716-721.
Weinstein, M.C., Siegel, J.E., Gold, M.R., Kamlet, M.S., and Russell, L.B. (1997). Recommendations of the panel on cost-effectiveness in health and medicine. Survey of Anesthesiology, 41(6), 331-332.
Weinstein, M.C., Torrance, G., and McGuire, A. (2009). QALYs: The basics. Value Health, 12(Suppl. 1), S5-S9.
Wherry, L.R., and Meyer, B.D. (2015). Saving teens: Using a policy discontinuity to estimate the effects of Medicaid eligibility. The Journal of Human Resources. doi:10.3368/ jhr.51.3.0913-5918R1.
Wherry, L.R., Miller, S., Kaestner, R., and Meyer, B.D. (2015). Childhood Medicaid Coverage and Later Life Health Care Utilization. NBER Working Paper 20929. Cambridge, MA: National Bureau of Economic Research.
White, D., and VanLandingham, G. (2015). Benefit-cost analysis in the states: Status, impact, and challenges. Journal of Benefit-Cost Analysis, 6(2), 369-399.
Williams, S. (2013). Statistical children. Yale Journal on Regulation, 30(1), 63-124.
Wilson, I.B., and Cleary, P.D. (1995). Linking clinical variables with health-related quality of life. A conceptual model of patient outcomes. Journal of the American Medical Association, 273(1), 59-65.
W.K. Kellogg Foundation. (2004). Logic Model Development Guide. Battle Creek, MI: W.K. Kellogg Foundation.
The World Bank. (2010). Cost-Benefit Analysis in World Bank Projects. Washington, DC: The World Bank.
World Health Organization. (2000). Workbook 8: Economic Evaluations. Geneva, Switzerland: World Health Organization.
World Health Organization. (2001). Macroeconomics and Health: Investing in Health for Economic Development. Report of the Commission on Macroeconomics and Health. Geneva, Switzerland: World Health Organization.
World Health Organization. (2006). Guidelines for Conducting Cost-Benefit Analysis of Household Energy and Health Interventions. G. Hutton and E. Rehfuess (Eds.). Geneva, Switzerland: World Health Organization. Available: http://www.who.int/indoorair/publications/guideline_household_energy_health_intervention.pdf [March 2016].
World Health Organization. (2012). WHO Guide to Cost-Effectiveness Analysis. T.T.-T. Edejer, R. Baltussen, T. Adam, R. Hutubessy, A. Acharya, D. Evans, and C. Murray (Eds.). Geneva, Switzerland: World Health Organization.
World Health Organization Quality of Life Group. (1998). The World Health Organization Quality of Life (WHOQOL) assessment: Development and general psychometric properties. Social Science & Medicine, 46(12), 1569-1585.
Yates, B.T. (1996). Analyzing Costs, Procedures, Processes, and Outcomes in Human Services: An Introduction (Vol. 42). Thousands Oak, CA: Sage.
Yates, B.T. (2009). Cost-inclusive evaluation: A banquet of approaches for including costs, benefits, and cost-effectiveness and cost-benefit analyses in your next evaluation. Evaluation and Program Planning, 32(1), 54-56.
Yoshikawa, H., Weiland, C., and Brooks-Gunn, J. (in press). When does preschool matter? Submitted to The Future of Children.
Zerbe, R.O., Jr. (2010). A Reliable Internal Rate of Return. Unpublished manuscript.
Zerbe, R.O., Jr. , and Dively, D.D. (1974). Benefit-Cost Analysis in Theory and Practice. New York: Harper and Row.
Zerbe, R.O., Plotnick, R.D., Kessler, R.C., Pecora, P.J., Hiripi, E.V.A., O’Brien, K., Williams, J., English, D., and White, J. (2009). Benefits and costs of intensive foster care services: The Casey Family Programs compared to state services. Contemporary Economic Policy, 27(3), 308-320.