IOM ROUNDTABLE ON EVIDENCE-BASED MEDICINE
September 2007 version. This Issue Overview was prepared at the request of the IOM Roundtable Working Group on Sustainable Capacity by J. Michael McGinnis, LeighAnne Olsen, Katharine Bothner, Daniel O’Neill, and Dara Aisner.
MARCH 2009 UPDATE
The American Recovery and Reinvestment Act of 2009
In the time since the preparation of this white paper, $1.1 billion of federal funds have been provided by Congress, through the American Recovery and Reinvestment Act of 2009 (ARRA), to increase national capacity for clinical effectiveness research. AHRQ (Agency for Healthcare Research and Quality) has received $700 million of these funds, of which $400 million will be transferred to the Office of the Director of NIH (National Institutes of Health) to conduct or support comparative effectiveness research (CER) activities.
An additional $400 million will be allocated at the discretion of the Secretary of HHS (Department of Health and Human Services) to:
“…accelerate the development and dissemination of research assessing the comparative effectiveness of health care treatments and strategies, through efforts that: (1) conduct, support, or synthesize research that compares the clinical outcomes, effectiveness, and appropriateness of items, services, and procedures that are used to prevent, diagnose, or treat diseases, disorders, and other health conditions; and (2) encourage the development and use of clinical registries, clinical data networks, and other forms of electronic health data that can be used to generate or obtain outcomes data.”
The recommendations from an Institute of Medicine consensus committee report and from a newly established Federal Coordinating Council on CER within HHS will be considered by the secretary’s office in designating activities to receive funds. Members of the 15-member council will be federal employees or officers appointed by the President, at least half of which will have clinical expertise.
LEARNING WHAT WORKS BEST
THE NATION’S NEED FOR
EVIDENCE ON COMPARATIVE
EFFECTIVENESS IN HEALTH CARE
This Issue Overview on current and needed capacity for comparative effectiveness research was developed by staff of the IOM Roundtable on Evidence-Based Medicine, initially as background material for the activities of the Roundtable’s Sustainable Capacity Working Group, and to inform discussion at the March 19, 2007 Roundtable meeting.
Guidance on structure and content were provided by members of an advisory group including: Carmella Bocchino (AHIP), Queta Bond (Burroughs Wellcome Fund), Kathy Buto (Johnson & Johnson), Steve Galson (FDA), Mark McClellan (AEI-Brookings Joint Center for Regulatory Studies), Lisa Payne-Simon (Blue Shield of California Foundation), Diana Petitti (University of Southern California), Jean Slutsky (AHRQ), Sean Tunis (Center for Medical Technology Policy), and Gail Wilensky (Project Hope).
We also extend special thanks to those who took the time to review and comment on various sections or draft versions of this paper, including: Wade Aubry (HealthTech), Tanisha Carino (Avalere Health), Nancy Featherstone (AstraZeneca), Mark Gibson (OHSU), Carmen Hooker Odom (North Carolina HHS), Michael Johns (Emory), Peter Juhn (Johnson & Johnson), Doug Owens (Stanford), Steve Pearson (AHIP), Steve Phurrough (CMS), Eugene Rich (House Committee on Ways and Means, Health Subcommittee), Wayne Rosenkrans, Jr. (AstraZeneca), and Jeff Shuren (FDA).
IOM staff who contributed include: Dara Aisner, Katharine Bothner, Michael McGinnis, LeighAnne Olsen, and Daniel O’Neill.
Roundtable sponsors: The work of the Roundtable is supported by the
Agency for Healthcare Research and Quality, AHIP (America’s Health Insurance Plans), AstraZeneca, Blue Shield of California Foundation, Burroughs Wellcome Fund, California Healthcare Foundation, Centers for Medicare and Medicaid Services, Department of Veterans Affairs, Food and Drug Administration, the HWG Fund, Johnson & Johnson, sanofi-aventis, and Stryker.
Suggested citation Institute of Medicine. 2007. Learning what works best: The nation’s need for evidence on comparative effectiveness in health care. http://www.iom.edu/ebm-effectiveness.
IOM ROUNDTABLE ON EVIDENCE-BASED MEDICINE
Working Group on Sustainable Capacity*
LEARNING WHAT WORKS BEST
THE NATION’S NEED FOR EVIDENCE ON
COMPARATIVE EFFECTIVENESS IN HEALTH CARE
A core objective for the nation is achieving the best health outcome for every patient. This objective simply cannot be accomplished until we have better evidence on which to base healthcare decisions, as well as more effective application of the knowledge we have. Each is vitally important. We know, for example, that failure to deliver proven interventions is a substantial challenge to the quality of health care for Americans—and is a key concern of the IOM Roundtable on Evidence-Based Medicine. Yet, with the current pace of change, the most rapidly growing problem is our inability to produce the needed evidence in a timely fashion. This paper provides background for discussion about the evidence gap—the fact that the nation’s capacity has fallen far short of the need to produce reliable and practical information about the care that works best. Medical care decision-making is now strained, at both the level of the individual patient and the level of the population as a whole, by the growing number of diagnostic and therapeutic options for which evidence is insufficient to make a clear
* Jack Rowe, Columbia University (Chair); Adam Bosworth, Google; Helen Darling, National Business Group on Health; Michael Johns, Emory University; Steve MacMillan, Stryker Corporation; Mark McClellan, AEI-Brookings; Richard Platt, Harvard University; Steve Udvarhelyi, Independence Blue Cross; Bill Weldon, Johnson & Johnson; Janet Woodcock, FDA. The material here is a staff paper prepared at the request of the working group. Information on the IOM Roundtable on Evidence-Based Medicine may be obtained at www.iom.edu/ebm.
choice. Biomedical insights and medical innovation continue to advance opportunities to increase the health and life-span of the American public, yet to capitalize fully on this potential requires enhanced capacity to ensure that decisions, in the face of increasing complexity, can be supported and guided by the best available scientific information.
Health care in the United States underperforms on many dimensions. At the macro level, with per capita expenditures more than 20 percent higher than any other country in the world and more than twice the average expenditure for European countries, the nation ranks well below others on key health indices—28th in overall life expectancy at birth and 23rd in infant survival [2, 3]. In part this is because people often do not receive the care they need. One study found that, where evidence exists, only about 55 percent of recommended services were actually delivered . In part it is also because the services people receive are not always necessary or the right ones for them. The intensity of services for similar conditions with similar results—in particular, for procedures such as lumbar surgery, hysterectomy, and bypass surgery, where discretion plays a stronger role—can vary by as much as a factor of 20 depending simply on where one lives. In Idaho Falls, Idaho, 4.6 lumbar fusions were reported per 1,000 Medicare enrollees annually, as compared with 0.2 in Bangor, Maine, with no difference in the outcomes . Similarly, wide geographic variations have been reported for conditions such as hip fracture, colorectal cancer, and acute myocardial infarction as well as end-of-life care , with a nearly fourfold difference in cardiac bypass surgery rates, a phenomenon primarily related to the region’s number of cardiac catheterization labs per capita rather than illness rates . One estimate suggested that only 27 percent of the weighted discrepancy in Medicare spending across regions could be explained by population illness levels , and if all regional spending levels matched those of the lowest decile, Medicare could see savings of up to $40 billion (1996 dollars) without compromising health status . Clearly, more does not by itself equate to better—and the variation is greater for conditions in which the evidence is more limited.
Ultimately, the central challenge is not primarily a matter of overuse or underuse of services but instead is related to the lack of available evidence to achieve the right care for any given patient. Information on which to compare the results from drugs with the same purpose is often not available. For example, both Lucentis and Avastin are promising new drugs for treatment of macular degeneration, but head-to-head information on the relative outcomes is not available—and one costs about 20 times as much as the other. Similarly, different approaches to radiation therapy—intensity-modulated radiotherapy and conformal radiotherapy—have very different costs but currently inadequate information on which to base clinical judgments. And the pace of introduction of new genetic prognostic tests is on
an exponential course without the necessary evidence about the results of clinical decisions and outcomes.
Estimates range widely concerning the proportion of medical care in the United States that is based on, or supported by, adequate evidence [9-14]. However, given concerns about the extent to which this information may be generalized and the quality of the evidence that is used, some place this figure at well below half. Regardless of the precise level, there is no question about the need for improvement. Part of the challenge is the appropriate delivery of what has already been proven effective. Medical care is becoming more complex with the increase in multifaceted chronic diseases, the development of new interventions, and the pressures to reduce the time of patient-provider interaction in the face of greater administrative burdens. New care management approaches, decision support systems, and incentives will be required to help providers and patients work together to ensure that the care delivered is the care that is known to be most effective.
The most rapidly growing problem may soon relate not so much to shortfalls in applying what is known—a clearly significant problem—as to the inability for evidence of comparative clinical advantage to keep pace with innovation. It is both a capacity investment and a resource allocation problem. Of the nation’s more than $2 trillion annual health expenditure—nearly half of it borne by government—currently less than 0.1 percent [15, 16] is invested in assessing the comparative effectiveness of available interventions. Although about 5 percent of overall health expenditures is devoted to research, most is devoted either to basic research or to product development , as opposed to assessing how well medical treatment options perform. If trend data were kept, it would likely reveal that the proportion of expenditures devoted to this assessment “budget” was actually shrinking every year, yet the complexity of clinical decisions continues to compound.
A testament to the power of innovation is the fact that new pharmaceuticals, medical devices, biologics, and procedures are introduced constantly, and the pace is quickening. From 1991 to 2003 the number of medical device patents per year doubled, from 4,500 to nearly 9,100 . From 1992 to 2001 the total biotechnology patents granted per year tripled, from more than 2,500 to nearly 7,800, and the number of biopharmaceutical patents granted in the United States increased nearly four-fold, from about 1400 to nearly 5,200 [19, 20]. In the same period, annual sales of biologics and pharmaceuticals more than doubled . Between 1993 and 2004 there was a more than 80 percent increase in the number of prescriptions received by Americans . Data for the growth in procedures are more difficult to obtain; however, as one example, between 1989 and 1995 specialized procedures in major teaching hospitals nearly tripled . A recent review by the Kaiser Family Foundation suggests that half or more of the
growth in medical spending in recent years is attributable to technological change .
Much, but certainly not all, of this change has resulted in better care. Diagnostic imaging services, for example, grew more rapidly than any other type of physician service under Medicare. Between 2000 and 2005 spending on radionuclide imaging (RNI) doubled from $6.6 billion to $13.7 billion . Yet an American College of Cardiology Foundation technical panel convened in 2005 to assess the appropriateness of cardiovascular RNI imaging for 52 indications  found that the lack of clear evidence on the best and most effective uses yielded strong disagreement on the appropriate circumstances.
In addition to the growth in use of drugs, devices, biologics, and procedures, the world of health care is about to experience dramatic new insights concerning the variation in individual response to different diagnostic and treatment interventions. The 3 billion base pairs of the human genome have now been sequenced, revealing the 99.9 percent of the genetic code that is common to all humans. Additional differences, such as the gain or loss of regions of the genetic code, increase the variation between two random individuals by five- to ten-fold. Cataloging and characterizing these differences by haplotype mapping and other initiatives is currently in progress and will begin to reveal how people vary in their susceptibilities to diseases and their responses to diagnosis and treatment.
The age of personalized medicine will soon be a reality, if the capacity can be developed to contend with its insights. Today the average clinical encounter already requires a health provider to manage more variables than would be considered reasonable given what is known about the capabilities of the human mind, and over the next decade that same encounter will require contending with perhaps an order of magnitude more . The traditions of developing evidence through one-at-a-time studies and relying for quality assurance on the recall capacity of an individual provider are no longer practical.
Over the long term, substantial changes will emerge in the way the nation goes about generating and applying evidence for clinical decision making. A learning healthcare system is one in which the clinical research paradigm depends more judiciously on the serial conduct of randomized controlled trials—important, but often too expensive, untimely, and of limited applicability—and draws more heavily on electronic health records (EHRs) to generate evidence as a natural by-product of the clinical experience. But while these longer-term capacities emerge, substantial near-term improvement will be necessary in our capacity to assess the relative effectiveness of different interventions—to understand what works best for whom under what circumstances. We need better understanding, agreement, and focus on the value we get from our health care—including what
constitutes value and how to measure it. Without this capability, it is likely that the inefficiencies that currently characterize the U.S. healthcare system will be compounded, perhaps considerably. Conversely, a more systematic and sustained effort to develop evidence on comparative and “real-world” effectiveness should stimulate more investment in research on innovation that will deliver better outcomes and greater value.
Engaging the immediate need for a much stronger and sustained capacity to meet the need for guidance on the clinical effectiveness of medical interventions is the subject of this paper. Discussion follows on the perspectives of the various stakeholders, the current capacity and activities on clinical effectiveness research, the key functional needs to be met, and, finally, some possible approaches to addressing the issues, including consideration of decision principles, governance, funding, and public support.
Better evidence is essential to securing trust in our healthcare system. In the face of uncertainty borne of insufficient evidence, patients, providers, insurers, and health product companies frequently find themselves at odds and distrustful of each others’ motives. Concern about the shortfall in the national capacity to determine what medical care is actually best for whom is shared among many stakeholders. Most important, of course, are the patients who receive medical care and the health providers who deliver it, but large stakes are also held by healthcare delivery organizations, insurers, manufacturers, and others engaged in various aspects of health care, with the shared goals of improving patient health and delivering the best value. Increasing the level of investment in clinical effectiveness research, and doing so in a comparative fashion, is key to facilitating significant gains toward these shared goals. Roundtable teams are currently reviewing the perspectives and action prospects as the various sectoral stakeholders work to improve the prospects for the development of a learning healthcare system. Following are some of the more important implications of accruing substantially better information on clinical effectiveness.
Each patient should be able to feel confident that there is solid evidence that the care received is the appropriate care for his or her circumstances. Yet, increasingly, this notion is strained. The American public has traditionally expressed strong support for medical care, research, and technology development, while also expressing a strong interest in both individual patient prerogative and better information to aid decision making. But with the increasing complexity of care and an increasing awareness of its
shortfalls, confidence in healthcare delivery is beginning to wane. Some of the concern may relate to increased costs borne directly by patients as a result of increased prices, related coverage reductions, and system inefficiencies—an increase of about 50 percent in out-of-pocket expenditures from 1994 to 2004 , on top of an 87 percent increase in premium costs for family coverage alone since 2000 —but much is based in perceptions about quality. In a 2005 Research!America survey, 60 percent of Americans said they didn’t believe that the United States has the best healthcare system in the world, 41 percent said they knew of a time when they or a family member had received the wrong care, and 56 percent said there should be more investment in clinical and health services research . A variety of recent initiatives, some by patient groups and some by medical and scientific groups, including the Institute of Medicine’s 2001 report Crossing the Quality Chasm , have emphasized the need for health care to focus more on the delivery of individualized, patient-centered care—including an active role for patients in making informed choices about their health care, with careful consideration of known risks and benefits, transparency in the quality of care, and continual assessment of the performance. A central precondition for each of these is a substantial enhancement of the evidence on the interventions that are most effective for any given circumstance. Most of the challenge in this respect is in ramping up the capacity to generate and apply the necessary evidence, but other issues related to patient perceptions must also be addressed. Because some patients may view the results of comparative effectiveness studies as potentially limiting to their choices, care must be taken in the application and interpretive processes to ensure understanding and appropriateness. Similarly, in order to reduce patient confusion as recommendations change and to improve the support for use of clinical data for new insights on effectiveness, it is important to help patients better understand the dynamic nature of evidence and the need, in a learning healthcare system, to draw upon the healthcare delivery experience to continually refine scientific understanding of the safety and effectiveness of healthcare interventions in a practical setting.
Health providers remain the central decision makers in health care, although increasingly they are making those decisions in close consultation with their patients. Ultimately, expanded efforts to make better information available on the comparative effectiveness of interventions are undertaken to better equip physicians and other health providers with the information needed to deliver the right care and to foster a more informed and supportive practice context. No health professional should be put in the position of uncertainty about the evidence in support of the care provided at his or
her behest. Yet, with the current pace of advances in medical procedures, pharmaceuticals, devices, and biotechnology, a sometimes confusing array of choices is presented for patients and their healthcare providers, making decisions about the best care for the circumstances increasingly complex. The lack of critical information and studies necessary to inform these decisions already place in jeopardy both the effectiveness and the efficiency of the medical care delivered to Americans. The pace at which this problem will grow with new innovations and insights is uncertain, but routine availability of comparative effectiveness information is a basic need for health professionals seeking to deliver high-value care to patients.
Healthcare Delivery Organizations
The integrity and reputation of healthcare delivery organizations is dependent on their ability to ensure the quality and appropriateness of the care delivered within their walls. In part, this is a function of the health professionals they employ and the culture they foster. Increasingly, it is also a function of the soundness of the systems they build to ensure that the necessary care is delivered and the delivered care is necessary. Any decision support system is only as good as the information built into the model and should include the comparative advantages or disadvantages of different diagnostic and therapeutic options. Especially as healthcare delivery organizations become more skilled in team management and as they build systems to improve and measure the consistency of the care with established performance standards, the rate-limiting factor will be the baseline information on the comparative effectiveness of available options.
The sector with the largest economic stake in better capacity to generate and apply information about the effectiveness of clinical interventions is the healthcare manufacturing sector—the companies that make the pharmaceuticals, devices, and biological products that provide the backbone for much of health care and its progress. It is a given that healthcare manufacturers, focused as they are on returns on investment, inherently understand the importance of improving the value proposition in patient care. But their stake goes deeper. Manufacturers directly bear the economic burden of delays and inefficiencies related to market entry requirements that are poorly linked to coverage approval processes; the absence of a clinical safety and effectiveness monitoring process that functions beyond the one-at-a-time tracking of new interventions, and the consequences of public and shareholder backlash when problems are identified too late. Given the aggregate size of the healthcare investment, improvements should
be achieved with economies of scale, better coordination of the efforts to produce the studies and systems necessary, and more transparency in the use of clinical data to draw conclusions. A stronger capacity to identify earlier when new interventions yield superior results should yield advances in both value and the pace of progress. Without a sizable improvement in our evaluative capacity, the slower pace of understanding how and when interventions work best will retard the application of potential innovations.
Over half of the nation’s health expenditures are borne by the private sector, including a sizable share by employers . For the fourth consecutive year, chief executive officers of U.S. companies have cited healthcare costs as their number one economic concern —a concern compounded by questions about the quality of the return on investment, when per capita expenditures rank 50 percent higher than any other country in the world. Employers now pay 78 percent more for health care than 5 years ago, and it has been suggested by some that this increased financial burden makes it more difficult for American companies and workers to compete in the global marketplace . Increasingly, healthcare costs are associated with reductions in the depth and breadth of employer-based health insurance coverage for U.S. workers and are often cited as a factor reducing the ability of companies to remain competitive. Without better information on which to improve the focus on innovations that work, rising costs and growing utilization will continue to contribute to the upward trajectory of healthcare spending, eroding the efficiency and value of our health expenditures. Those expenditures are projected to reach 20 percent of the gross domestic product by 2015—raising the burden for households, businesses, and government, yet not yielding concomitant gains in value .
Insurers represent the front line of the economic choices that have to be made on payment for healthcare services, with a fiduciary responsibility to purchasers to ensure that payments are devoted to care that is most appropriate and that returns the most value to their clients. This means drawing conclusions about the comparative advantages or disadvantages of proposed diagnostic or treatment interventions, in the face of a paucity of such information . The larger insurers maintain analytic staffs to assess the existing literature, and most also contract with organizations that conduct formal systematic reviews, but such reviews are limited by the shallowness of primary research studying the effectiveness of interventions in a practical setting—either compared to a placebo or to other alternative
choices. Insurers perhaps most acutely feel the need for much more reliable, rigorous, transparent, and impartial comparative effectiveness information to make decisions in the growing marketplace of medical interventions.
Government currently accounts for about 45 percent of health expenditures in the United States, although if foregone tax revenues for employer-based health benefits are factored in, the number may be closer to 58 percent [37-40]. Most of the direct government expenditures are for reimbursement for Medicare and Medicaid clients, but the federal government also provides services directly to 9.2 million members of the uniformed services and their dependents, to 5.3 million patients who are served by the Veterans Health Administration, and to 1.5 million clients of Indian Health Service facilities. The federal government pays for the health coverage of approximately 3.7 million employees through the Federal Employees Health Benefit Plan (FEHBP), with enrollees selecting health plans from a number of competing insurance carriers. Whether acting as a payer or a provider, the government also has a basic responsibility to ensure that its clients receive the care that is most appropriate and of the greatest value. Those responsible for the relevant decisions encounter the same disadvantages as private sector decision makers with respect to the lack of information on the comparative effectiveness of candidate interventions. In addition, the Food and Drug Administration, which is required by law to make its judgments based on the safety and efficacy of a given intervention, is increasingly under pressure to provide perspectives on the relative advantages of proposed new approaches but generally lacks the studies on which to base such counsel.
Activities currently under way to assess the effectiveness of healthcare interventions may be generally characterized as broad, based in multiple loci, largely uncoordinated, far short of the need, and under-resourced. Presented here are definitions of the basic terms used to describe the various categories of activity, followed by a summary of the major institutions engaged in the work. A more detailed description of the activities is presented in Appendix One.
Clinical effectiveness research can be described as either primary or secondary. For our purposes, primary refers to the direct generation of
evidence through the use of a specific experimental methodology. Secondary refers to the systematic gathering and evaluation of primary research information to further the understanding of common conclusions or disparate results.
Primary Clinical Effectiveness Research
In this respect, primary clinical effectiveness research refers to the specific design and implementation of structured research protocols to produce data on the results of one or more diagnostic or therapeutic interventions of interest. Examples include certain randomized controlled trials, practical clinical trials, cluster randomized trials, observational studies, and cohort studies, including registries. Some of these studies focus only on the efficacy of an intervention—the extent to which an intervention produces a beneficial result under ideal circumstances. But many also examine the effectiveness of an intervention when used under ordinary circumstances—including evaluations in broader patient populations and healthcare delivery settings or analyses of the relative risks and benefits of competing therapies. Both types of evaluation are important to an understanding of which interventions work best, for whom, and under what circumstances.
Evidence synthesis or secondary clinical effectiveness research refers to the structured assessment of evidence from multiple primary studies to derive conclusions which are considered to have greater weight than an individual study alone. The types of evidence synthesis include systematic review and technology assessment, both of which describe a systematic method of identifying, assembling, and interpreting a body of data to validate or extend the interpretation of single trials, lend context to individual trials, and, where possible, arrive at common conclusions. Systematic reviews are frequently published through the peer-reviewed literature, while many assessments are tailored more narrowly to assist in policy or practice decision making.
Within the overall umbrella of clinical effectiveness research, the most practical need is for studies of comparative effectiveness, the comparison of one diagnostic or treatment option with one or more others. In this respect, primary comparative effectiveness research involves the direct generation of clinical information on the relative merits or outcomes of one intervention in comparison with one or more others, and secondary comparative
effectiveness research involves the synthesis of primary studies to allow conclusions to be drawn. Secondary comparisons of the relative merits of different diagnostic or treatment interventions can be done through a collective analysis of the results of multiple head-to-head studies, or else indirectly, in which case the treatment options have not been directly compared to each other in a clinical evaluation, and inferences must be drawn based on the effect of each intervention relative to a specific comparison, often a placebo.
Other Related Terms
Other relevant terms used in the context of clinical effectiveness research discussions include cost-effectiveness analysis and cost-utility analysis. In cost-effectiveness analysis, the economic cost per specified unit of health gain—e.g., reduced mortality or morbidity—is determined for a given intervention or family of interventions. This allows for one measure of the relative value of an intervention to be estimated in comparison to alternatives. Cost-utility analysis is a form of cost-effectiveness analysis that estimates the cost of a specific utility gain, usually to the patients targeted—e.g., quality-adjusted life-years—for an individual intervention. Finally, the term health services research refers broadly to the multidisciplinary field of scientific investigation that studies how the effectiveness of health care for different populations is shaped by various systemic factors such as social factors, financing systems, organizational structures and processes, health technologies, and personal behaviors affecting access to health care, the quality and cost of health care, and measures of population health and well-being .
Clinical Effectiveness Research in the United States
It is difficult to characterize precisely the national expenditures on clinical effectiveness research, but the investment is clearly far short of the need. If only 1 percent of the nation’s $2 trillion healthcare bill were devoted to understanding the effectiveness of the care purchased, the total would come approximately to $20 billion annually. In contrast, only $15 million annually has been specifically appropriated by Congress to the Agency for Healthcare Research and Quality (AHRQ), under section 1013 of the Medicare Prescription Drug, Improvement, and Modernization Act (MMA), for comparative effectiveness research, the issue for which the shortfall is greatest. The total of all appropriations to all federal agencies—the National Institutes of Health (NIH), the Veterans Health Administration (VHA), the Department of Defense (DOD), the Centers for Medicare and Medicaid Services (CMS), the Food and Drug Administration (FDA),
AHRQ, and the Centers for Disease Control and Prevention (CDC)—for all health services research amounts to about $1.5 billion, only a small portion of which is devoted to clinical effectiveness research .
In addition to these federal appropriations, the insurance industry currently invests substantially in assessment of clinical interventions, and healthcare manufacturers account for the largest contribution to clinical effectiveness research. It is reported, for example, that Pharmaceutical Research and Manufacturers of America (PhRMA) member companies spend in the range of $15 billion annually for Phase III and IV clinical trials of pharmaceuticals, including examination of clinical effectiveness . Data are not currently available on the amount of direct expenditures by professional societies on primary and secondary clinical effectiveness research. Even accounting for the support from all involved public and private institutions, the aggregate national commitment to assessing the effectiveness of clinical interventions is likely well under 1 percent—far below the standard that any company would expect to invest in work to evaluate and improve its products.
Primary Clinical Effectiveness Research
At the federal level, several agencies of the Department of Health and Human Services (HHS) sponsor research and maintain databases that provide insights on the clinical effectiveness of healthcare interventions. NIH is the largest federal sponsor of clinical research, and its component institutes have supported some work on the comparative effectiveness of health interventions, although these types of assessments are not a major focus of attention—an estimated $660 million total since 1982, in comparison with the aggregate NIH budget of $250 billion in that period . Through the National Library of Medicine the NIH also maintains, in collaboration with FDA, the national inventory of clinical trials at the ClinicalTrials.gov Web site. CMS does not conduct clinical research directly, but its data systems, demonstration and evaluation activities, and coverage policies offer powerful resources for assessing and monitoring clinical effectiveness. Claims data are maintained on the more than 42 million Americans served under Medicare and on the 47 million low-income people covered under Medicaid. With the passage of the Medicare Modernization Act Part D benefit, which makes Medicare enrollees eligible for prescription drug coverage, extensive new opportunities were presented to assess clinical effectiveness in a post market environment, by linking Part D data to data from Parts A, B, and C in the conduct of public health research. Recently, CMS has launched the Coverage with Evidence Development initiative, beginning with the development of a registry that will track the experience of Medical patients receiving implantable cardioverter defibrillators (ICDs). This
allows coverage of services for certain populations on which existing effectiveness information is limited, contingent upon clinical effectiveness data collection via a registry or other mechanism.
FDA, CDC, and AHRQ are HHS agencies that also contribute to clinical effectiveness research. FDA contributions come through requirements that manufacturers establish basic safety and efficacy information as part of the drug, biologic, and device approval processes; it also collects data as part of its adverse event reporting process and related post-marketing surveillance work. Recently FDA has proposed the development of an integrated national network, a Sentinel Network, formed through a series of public-private partnerships using new electronic information technology systems to collect and analyze medical product safety information and then disseminate it to healthcare practitioners and patients at the point of care. In general, the FDA does not require comparative effectiveness information to approve the marketing of an agent or innovation. CDC funds some health services research to guide decisions on public health services and systems, and some of this research may include an examination of the effectiveness of some therapies in the area of infectious disease or vaccines . In addition, CDC maintains a number of national data systems—vital statistics, health examinations, and health interview surveys—that are important resources for certain types of clinical studies.
AHRQ, the lead federal agency for health services research, has a mandate from section 1013 of the Medicare Prescription Drug, Improvement, and Modernization Act along with a related $15 million appropriation to perform research with a focus on the outcomes, comparative clinical effectiveness, and appropriateness of pharmaceuticals, devices, and healthcare services. The provision has led to the creation of AHRQ’s Effective Health Care Program, which has three components: synthesizing existing studies into Comparative Effectiveness Reports by Evidence-based Practice Centers; developing evidence, including research aimed at filling knowledge gaps about treatment effectiveness (DEcIDE centers); and improving communication of complex scientific findings to a variety of audiences (Eisenberg Center).
At the Department of Veterans Affairs, the VHA has an active clinical research program in its 157 medical centers and more than 1,300 ambulatory, residential, and home and community-based sites of care, facilitated by a state-of-the-art computerized clinical data system, with 8 million patient records, which allows the rendering of large quantities of patient data for analyses on a wide variety of clinical research topics. With more than 3,000 researchers throughout the system and specially designated “centers of excellence,” the VHA devoted some $400 million in 2005 to clinical research in addition to the resources devoted to the related medical care.
A substantial share of this work focuses on clinical effectiveness, including some on comparative effectiveness.
Outside the federal government, primary and secondary (evidence synthesis) research is both conducted and sponsored by health manufacturers, insurers, healthcare delivery organizations, and professional societies. Industry-sponsored trials represent a significant proportion of health manufacturer investments in research and development and about 40 percent of pharmaceutical R&D investments goes to the phase 3 and 4 trials that have particular relevance to clinical effectiveness . Many of these studies are conducted with academic investigators and others are managed by contract research organizations. Relatively few of the studies are comparative, or head-to-head studies [37, 38].
Large healthcare systems, such as large health maintenance organizations, also have the capacity to use both their clinical and administrative data systems for primary clinical effectiveness research—some sponsored out of their own resources, others sponsored by NIH or commercial concerns. For example, Kaiser Permanente, Geisinger Health, and Group Health all have efforts to use their large databases to learn what works best for which patients, extending insights from primary studies of efficacy or effectiveness. The Health Maintenance Research Network (HMORN) is composed of 15 HMOs covering over 15 million individuals and working cooperatively on effectiveness research. Data from their large record systems and registries are useful in defining appropriate use of interventions in subgroups of patients, including the post-introduction monitoring of results from new diagnostics and treatments. Similarly, payer-related data consortia probe the use of linked health insurance claims information to assess the clinical effectiveness of various interventions. In July 2006, the AQA Alliance and the Hospital Quality Alliance (HQA) announced the formation of a joint effort to combine resources to identify, collect, and report data across the variety of care settings they represent; a variety of data are being aggregated, including information on the quality of physician performance, cost-of-care measures, and the quality of care for specific conditions, such as heart attack or pneumonia, as well as other measures.
Recently, as a bridge of sorts between primary research and the use of existing data, more emphasis has been given to the potential for decision models in estimating the relative benefits and harms for different interventions . Most cost-effectiveness analyses depend on the construction of decision models, but such models can also be useful in estimating purely clinical outcomes that might have important implications for decisions to use or not to use a particular intervention. Modeling may be particularly helpful in decisions on the application of diagnostic testing and screening tests, by revealing the likely yield from such tests when applied on a population basis. For example, such models proved important to the develop-
ment of recommendations on the ages and periodicity for cancer screening . Similarly, they can be useful in determining the optimal sequencing of radiologic tests and of other tests used in sequence for diagnosis, particularly when supplemental information is available at large databases . With the growth of information from registries and other large databases, models will likely be deployed much more frequently as evidence sources and decision tools.
Much of the work to marshal evidence for conclusions about clinical effectiveness takes the form not of primary data generation and analysis but of systematic reviews and meta-analyses of existing studies. Such secondary clinical effectiveness research is sponsored and conducted by a variety of organizations with overlapping and intersecting activities and interests, including federal agencies, state agencies, insurer and insurer-related organizations, independent assessment centers, professional groups and societies, university centers, and consortia. At the federal level perhaps the best known is the work of the U.S. Preventive Services Task Force (USPSTF), which has since 1984 conducted systematic reviews of the evidence in support of clinical preventive services, applied rigorous criteria to classify and rate the level of the evidence, and, based on the overall strength of the evidence for a given condition, offered conclusions and recommendations. The USPSTF is now sponsored by AHRQ, and its approach has set a standard reference point for much of the subsequent work on synthesizing evidence and making clinical recommendations. Building on this work, AHRQ has established a network of 13 AHRQ-sponsored evidence-based practice centers (EPCs) which review literature, perform technology assessments, and produce evidence reports including comparative effectiveness reviews. One of the AHRQ EPCs, at Oregon Health & Sciences, operates the Drug Effectiveness Review Project (DERP) on behalf of 13 state Medicaid programs to assess the pharmaceuticals to be provided under Medicaid. Consumers Union uses DERP findings as the basis for its Best Buy Drugs Program. In a related AHRQ- and FDA-sponsored effort, the 11 Centers for Education and Research on Therapeutics (CERTS) each focus on a specific patient population or therapeutic area in conducting research on ways to advance the optimal use of drugs, biologics, and medical devices by identifying best practices. CMS draws directly on AHRQ-sponsored technology assessments for use by its Medicare Coverage Advisory Committee to inform CMS coverage decision making. Other federal evidence synthesis efforts include the NIH consensus development conferences and state-of-the-science conferences, and the VA Technology Assessment Program for devices, drugs, procedures, and organizational and supportive systems used in health care.
In addition to the federally sponsored efforts, the largest insurers, such as United Healthcare, Wellpoint, Aetna, Humana, and CIGNA, have substantial in-house capabilities for the conduct of evidence reviews, and they also commission more formal assessments from technology assessment entities, such as the Blue Cross Blue Shield Association’s Technology Evaluation Center (TEC), ECRI, Hayes, and Cochrane (see below). Another insurer-related activity is the Academy of Managed Care Pharmacy (AMCP), through which health plans utilize comparative analysis when developing formularies. Currently, most clinical practice guidelines are produced in association with physician specialty societies and published through multiple modalities. There are more than 150 medical specialty societies in the United States, and many are engaged in some form of evidence review and guideline development. The work is also performed beyond the medical societies. Altogether, more than 300 organizations have published at least one guideline on the AHRQ-supported Web site, guidelines.gov. The national guideline clearinghouse currently contains more than 1,900 individual summaries. There is substantial variability in the approaches of the various societies, with some of them conducting the reviews by informal staff-generated activities and others engaged in large, multifaceted, and structured consultations involving the multiple related organizations with similar interests in an issue. Because of the overlapping nature of the interests, issues, and approaches, a number of collaborative efforts have emerged. The best known of these is the Cochrane Collaboration, an international effort (see Appendix Two), including a U.S. center, which sponsors systematic reviews using carefully developed common standards on a wide variety of issues in health care. Groups such as the Grading of Recommendations Assessment, Development and Evaluation Working Group (GRADE) and Appraisal of Guidelines Research and Evaluation (AGREE) have formed to develop standards for evidence syntheses and clinical practice guidelines. Similarly, the Evidence-Based Medicine Roadmap Group, an effort led by AHIP including participation from several interested organizations, is working to develop a more consistent and transparent system for assessing comparative clinical effectiveness.
Clinical effectiveness assessment activities have a utility that crosses national borders, and a growing involvement in technology assessment and clinical effectiveness evaluation has developed internationally. The explicit incorporation of cost-effectiveness evaluation tends to be a common feature of the work abroad, prompted by the need for better information on which to make decisions about the use of public monies. Implicit in the systematic evaluation of cost for an intervention is the comparison of the intervention
with alternatives for care, as a means to arrive at an assignment of value for investment. Thus, the majority of comparative effectiveness analysis performed in an international context is based, to some degree, on the cost of the intervention. Programs in Australia, Canada, France, Germany, Sweden, and the United Kingdom are described in Appendix Two, as are the multinational efforts under the European Union, EUnetHTA, and the Cochrane Collaboration.
As broad and variegated as the interest and activity around clinical effectiveness are, the aggregate capacity is very thin and substantially short of the need. Because there are scant resources for the support of primary comparative effectiveness research (CER)—head-to-head studies—much of the work done is secondary evidence synthesis. Even the systematic reviews and technology assessments are often uncoordinated and draw on inconsistent standards for effectiveness determinations, underscoring the need for a substantially greater and more systematic approach to the work. The areas of activity and changes needed have been characterized in various ways [45-51], and can be grouped as indicated in Table A-1 below, closely reflecting those discussed at a recent Health Industry Forum meeting .
TABLE A-1 Prominent CER Activities and Needs
|1.||Studies of comparative effectiveness (“head to head”)|
|2.||Systematic reviews of comparative effectiveness|
|3.||Assessment of comparative value/cost effectiveness|
|4.||Coordinated priority setting and execution|
|5.||Improved study designs and research methods|
|6.||Better linkage of studies of efficacy, safety, and effectiveness|
|7.||Appropriate evidence standards consistently applied|
|8.||Consistent recommendations for clinical practice|
|9.||Guidance for coverage and funding|
|10.||Dissemination, application, and public communication|
1. Studies of comparative effectiveness
The most rapidly growing problem in health care today is the need for better information on the head-to-head effectiveness of diagnostic and therapeutic interventions. Care related to breast cancer offers a good example. Not long ago breast cancer was treated as a single entity with one therapeutic approach—radical mastectomy—but now the condition is seen as a complex cluster of conditions whose many different forms require different diagnostic approaches, ranging from characterization of tumor cell
mass and estrogen sensitivity to genetic predisposition to metastases, and also require different surgical, radiologic, and chemotherapeutic treatments. Despite the fact that breast cancer is one of the most extensively studied conditions, the rapid and encouraging development of different diagnostic approaches and treatment interventions is substantially outstripping the ability to understand what is best for a given individual. Indeed, bone marrow transplantation for breast cancer represents one of the most prominent examples of an aggressive intervention applied prematurely because adequate studies had not been done—and it was ultimately proved ineffective. The challenge of too little information on the relative effectiveness of different tools for diagnosis and treatment is a growing problem across the spectrum of diseases and conditions. When people become ill, the question that rivets the attention is, “What’s best for me?” Yet, as options increase, providers increasingly do not have the answer to that question. Most biomedical research investments are devoted to understanding basic biological processes and mechanisms, which ironically both feed the development of new approaches, but, by virtue of the tacit displacement of research on clinical effectiveness, impede the ability to learn what actually works best for individuals and for populations. Even within an overall pool of research on clinical effectiveness that is inadequate, the share devoted to comparative assessments of interventions may be fairly characterized as very small. With a need this compelling, the consequences are also compelling, ranging from the prospect of lost opportunities and harm for patients to the wholesale dissipation of resources and to public and provider confusion. Put simply, there is an acute need for: (1) a substantial increase in support for primary research into the comparative effectiveness of diagnostic and treatment interventions; (2) a means of determining the priority for the studies that are most compelling; and (3) entirely different ways of conducting the research to accelerate the generation of clinical effectiveness insights, including improved approaches stemming from the use of electronic health records to generate evident from routine clinical experiences (see study designs and research methods below).
2. Systematic reviews of comparative effectiveness
There are currently multiple means by which secondary research on the evidence of effectiveness is performed in the United States, as described earlier. The primary limitation on systematic reviews of the comparative effectiveness of various interventions is the paucity of primary research results on which to draw. There are also several general categories of issues in need of additional systematic evaluation both for determinations related to the evidence, where it exists, and recommendations relating to additional studies needed. These include issues related to the comparative evaluation of different drugs within a single class, evaluation of different
drugs in different classes for the same condition, evaluation of medical procedures, comparison of the effectiveness of procedural versus pharmaceutical approaches to individual diseases, and the comparative effectiveness of different approaches to diagnosis given a suspected disease. Other challenges exist as well, including those related to coordinating studies of interest, ensuring consistency of approaches, and improving the compatibility of findings. For example, systematic reviews by different institutions on the use of epidural steroid injections for the treatment of sciatica and on stress ulcer prophylaxis  in critically ill patients have yielded discordant results [48, 53]. In part, these sorts of inconsistencies arise from the possibility of legal impediments to payer organizations collaborating on analyses that relate to their coverage determinations, but they also relate to the absence of a consensus on the conceptual framework of key decision categories and how standards for necessary levels of evidence might vary by category. So the need exists both for additional support for systematic reviews of comparative effectiveness and for application of consensus approaches and standards.
3. Assessment of comparative value/cost effectiveness
Understanding the value returned for investment is a basic obligation in the stewardship of resources. The essential elements of value include knowing whether and on what dimensions an intervention works (outcome), knowing how well it works in comparison to alternatives (relative outcome), and knowing the economic cost (cost per outcome unit). There are multiple ways in which cost can be considered as part of medical decision making. Methodologies such as cost-effectiveness and cost-utility analysis use cost data in specific ways to place an economic metric on individual interventions. In the United States, the use of economic considerations as a determining factor in medical decision making remains controversial. CMS, the largest single payer for healthcare services, does not have the authority to use cost as a factor in making national coverage determinations, although it does use costs to the Medicare system in choosing which technologies to evaluate, and local CMS contractors are empowered to use cost in certain determinations (e.g., “lowest cost alternative”). The notion of knowing the relative cost-effectiveness of interventions is becoming more important as the costs of health care continue to rise, consume a greater part of the national economy, and have a significant impact on individual, business, and government financial health. This is particularly true for considering interventions of comparable clinical effectiveness but very different costs. As economic implications become ever more acute, so does the need for consistent, validated, and transparent means of assessing value.
4. Coordinated priority setting and execution
With resources as scarce as they are for comparative effectiveness assessments, and with the need for directing them rapidly to the most pressing issues, it will be of key importance to develop a means of coordinating the establishment and execution of needed studies and assessment priorities. While there are many organizations involved in activities related to clinical effectiveness research and comparative effectiveness evaluation, these efforts are fragmentary, frequently occur in isolation, and, as a result, can be duplicative or even contradictory. A systematic approach to linking the research agendas and outputs of these various organizations would facilitate an effort to increase the national capacity for research in regards to both primary and secondary research. A focus on high-volume or high-cost interventions is natural, but many considerations pertain. Various factors enter into the consideration of priorities, such as the impact of the condition in play (morbidity and mortality), the number of intervention options, their expense, the potential for significant impact on outcome, cost, or quality of life, and more. A first step in this respect would be the development of a mechanism whereby the perspectives of the key stakeholders can be reflected and criteria developed for making determinations. In addition, some consideration should be given to the appropriate methodology for analysis. Certain issues warrant higher priority for head–to-head trials, but others might be more appropriate for observational, registry, or database studies. This issue is further examined in the section on expanded methodologies.
5. Improved study designs and research methods
Clearly central to progress is the development of improved tools. Chief among the needs in this respect is the development of new study designs and statistical tools that will allow an expanded use of electronic health records in order to generate insights on clinical effectiveness. As our ability to manage large quantities of data electronically continues to increase, the use of electronic records to drive observational databases and registries will increase. Thus, another key element to consider, in terms of methodology, is a more informed and practical approach to the use of observational studies, clinical registries, and data aggregation. The randomized controlled trial (RCT) has been so successful in bringing order and validity to insights from the clinical experience that it has come to be known as the gold standard for studies of clinical effectiveness. But the RCT has many designs and applications that vary depending on the circumstances. Similarly, there are many approaches to gathering and assessing observational data, and the nature of the datasets themselves is changing rapidly. The pace of EHR development has the potential to expand the data pool dramatically, as does the focus on improving and standardizing the data inputs and the development of new statistical approaches to evaluating the data. When underlying
data are adequate, modeling can be used to make determinations of relative effectiveness within populations, and it has the potential to contend with the large populations needed to substratify patients and substantially change the approach to conducting comparative studies . In many ways the notion of a “gold standard” is a misleading characterization. The basic challenge is to select the approach that is most informative from those practical to accomplish, identifying and acknowledging the potential for flaws. Certainly this is a more feasible approach, given the impossibility of relying on serial RCTs to provide all, or even a sizable fraction, of the comparative insights needed. So, both improved study methods and a capacity for the provision of technical assistance in the design of clinical studies are vital to progress.
6. Better linkage of studies of efficacy, safety, and effectiveness
With the need being so acute for a larger study base on which to base determinations of clinical effectiveness, the merits of better linkage of existing resources and activities are obvious. Viewed in vertical and horizontal dimensions, this might be termed “working to enhance the efficiencies of scope and the efficiencies of scale.” With respect to scope, the parameters of any study design will affect the parameters of its applicability. Noted earlier were some of the problems encountered, for example, by manufacturers when studies designed to test safety and efficacy came up short with respect to the proof of effectiveness needed for coverage decisions by payers. Better communication at the outset among manufacturers, product approval authorities, and payers might expedite the evaluation of an intervention’s performance in comparison with existing and alternative approaches, thus expediting the decision-making process. Some have termed this planning for the full life cycle of clinical research, from the early stages of testing through post-introduction monitoring. There is a need for vehicles to foster this type of communication. Similarly, with respect to scale, there is potential benefit in enhancing mutual awareness of similar clinical studies in order to foster the potential for collaborative or synergistic work. One existing resource of this sort is clinicaltrials.gov, described earlier as a mechanism for registering all clinical trials, both to increase patient recruitment as well as to facilitate information gathering once the trials are in process or have been recently completed. Perhaps even more important than the prospects for benefits from linking clinical trial work are the potential benefits from linking clinical data. Several efforts are currently under way in this regard, including the work of AHIP to foster data sharing through the ambulatory and hospital quality alliances and also the work of the RHIOs (Regional Health Information Organization) to advance platform and language compatibility. The VHA, with its vanguard work to implement electronic health records, has dramatically improved the linkage of records throughout its
system, setting the stage for broader use in clinical effectiveness research. The pace of progress in such linkage activities is dependent on a number of factors, including progress in standardizing terminology, improving platform compatibility, developing novel search algorithms, and patient privacy through mechanisms to de-identify data. Accordingly, there is a need for a well-positioned entity to steward and coordinate the activities of the various participants dealing with the different components of these interlocking issues.
Post-approval monitoring Particular mention is warranted of the implications of enhanced data system linkage—the scale dimension—for improved post-approval monitoring of new interventions and for fostering the pace of innovation and learning. Phase IV studies on the performance of new products are now usually independently designed and funded and are generally implemented on a product-by-product basis. As the prospect expands for linked clinical information systems, many of which are already well into application (e.g., VHA, DOD, Kaiser, EPIC users), so does the prospect for the availability of information of sufficient scale to provide much earlier insights into sub-group safety and effectiveness issues not fully resolved before an intervention’s introduction into practice. An intermediate variation on this theme is the development of patient registries to monitor the performance of new interventions as they are introduced into new populations for which pre-introduction trial evidence is suggestive of effectiveness, but not conclusive—as in the CMS initiative on Coverage with Evidence Development. Once a registry is established, in principle its use could be expanded to monitor certain other interventions as well. In each of these circumstances, there is a need for better capacity to coordinate, monitor, and validate the work.
7. Appropriate evidence standards consistently applied
The impact of reliable research results is ultimately determined by the routes to their application, and the first stop along such a route is the set of standards used to judge the reliability and implications of the findings. Currently, several issues slow progress in this respect, including the limited flexibility of established concepts in adapting to different clinical circumstances and forms of evidence; the lack of a vehicle to foster agreement on the approach to application of standards of evidence; and a certain amount of resulting inconsistency in the determinations. Concerning the first issue, in part the need is to adapt approaches to the different nature of devices, diagnostics, pharmaceuticals, and procedures. For example, evaluation of many devices needs to take account of provider training and facility experience as well as of their sometimes short life cycle. Similarly, “blinding” in an RCT on a surgical procedure is largely impractical. Indeed, concerning what has come to be known as the evidence pyramid—with RCTs at the
top and professional opinion at the base—while the pyramid offers a key conceptual touchstone, it cannot be applied without adaptation to circumstance, and the rules or guidance or adaptive principles do not yet exist. Some collaborative efforts are under way to harmonize the approach to the grading of evidence and recommendations, such as those of the international GRADE working group, and to find more appropriate approaches to matching standards with needs, such the AHIP Evidence-Based Medicine Roadmap Group project to develop a matrix or mosaic approach to characterizing evidence. But work of this sort is still early in its development and limited in its acceptance. Consequences to clinical effectiveness determinations resulting from inconsistencies related to standards of evidence are relatively few, but they do occur. Perhaps the most substantial problem in this respect relates to the uncertainty those developing new interventions have concerning the standards they will have to meet in providing proof of effectiveness.
8. Consistent recommendations for clinical practice
There exists substantial heterogeneity in the work of various professional societies and other groups to develop guidelines for clinical practice. Some organizations and activities have well-established and formalized protocols for their guideline development activities. Examples include the U.S. Preventive Services Task Force (USPSTF), operated with AHRQ sponsorship, and the guidelines activities jointly sponsored by the American College of Cardiology and the American Heart Association. But other activities are often more ad hoc in nature. Guidelines.gov has served as a clearinghouse of clinical practice guidelines and, in association with the USPSTF, has fostered greater consistency in formatting. But the quality and specificity of individual guidelines remain variable, and although the format seems more standardized, the approach is often not. Some prominent examples of inconsistency in guidelines include those related to mammography for average-risk women over 40, prostate-specific antigen (PSA) testing for men, and Human immunodeficiency virus (HIV) screening for the general population . Fostering greater consistency in the application of standards to produce clinical guidelines is an obvious priority.
9. Guidance for coverage and funding
The goal of increasing the capacity for research on comparative effectiveness is to improve the basis for medical decision making at many levels, not the least of which is for those who must make determinations on funding and coverage for interventions. Each payer has developed its own approach to obtaining the best available analyses on which to base coverage determinations, but they also generally report that the majority of the decisions they make have to be made with what they consider insufficient
evidence in hand. One of the problems in the current fragmented and under-resourced activities to establish clinical effectiveness is that studies focused on the issue of efficacy and safety for product approval purposes often lack the information needed for coverage decisions down the line—e.g., how well an intervention works in different sub-populations, or how results apply to populations with multiple, interacting conditions and treatments, or how it compares to alternatives. Examples of manufacturers who reported having received FDA approval for use but having delayed the because of coverage uncertainties include those who manufacture certain artificial spinal discs and ankles, Regranex for wound healing, and the Mammotome Breast Biopsy System . Approaches are therefore needed both to improve the extent to which the information requisite for coverage decision making is incorporated early in the research planning process and to foster the capacity for using the post-approval monitoring period to further elucidate issues of effectiveness and safety.
10. Dissemination, application, and public communication
Ultimately, the extent to which better evidence drives healthcare improvement in this country will be determined by the public’s clear and strong demand for better evidence that is more fully applied according to what is most appropriate for an individual patient’s circumstances. As a result, a major key to progress will be effective communication—both on the specifics of what is learned about effective treatment as results are gleaned and on the dynamic nature of evidence, its evolution, and how it is gauged. Currently, the frequency with which recommendations are in conflict, previously accepted practices are disproved, and messages—commercial and other—are broadcast advocating one intervention or another, often leaves the public, and sometimes providers, uncertain, skeptical, and even fearful. Some of these issues will be ameliorated if greater consistency can be achieved in the standards used to interpret evidence and develop guidelines, perhaps through the development of a trusted and reliable source that might validate determinations or through which determinations and recommendations might be filtered. But more will be necessary in using communication and marketing principles more effectively both to make validated determinations more accessible to the consumer public and to help inform and educate the public about how evidence continually evolves and how to judge its state of play for a given issue at a given point.
Noted below in Table A-2 is a summary of these key challenges in redressing the pressing shortfalls in the development and application of clinical evidence.
TABLE A-2 Prominent CER Activities and Needs—Key Challenges
|Head-to-head studies||Scant resources; rapidly increasing need; comparison choice|
|Systematic reviews||Few primary studies; inconsistent methods; uncoordinated|
|Comparative value insights||Little agreement on metrics or role of costs; cost fluctuation|
|Priority setting||Fragmentation; inefficiency; no mechanism for coordination|
|Study designs and tools||Clinical trial time/cost/limits; large dataset mining methods|
|Research lifecycle links||Efficacy-effectiveness disjuncture; post-approval surveillance|
|Evidence standards||Standards not adapted to needs; inconsistency in application|
|Practice guidance||Disparate approaches; conflicting recommendations|
|Coverage guidance||Narrow evidence base; limited means for provisional coverage|
|Application tools||Public misperceptions; incentive structures; decision support|
The compelling potential health and economic consequences for Americans call for a substantial response to narrow the rapidly growing gap between the available evidence on clinical effectiveness and the evidence necessary for sound clinical decision making. From a practical perspective, over the long run the rate of progress in both the development and the application of evidence on effective care will depend on advances in the use, standards, and interoperability of health information technology throughout all practice settings. In the meantime, and even when that capacity is in place, much stronger investment in dedicated comparative studies is essential. Various organizations and recent public articles have called for a quantum increase—several billion dollars—in the level of investment in comparative effectiveness research and the creation of a sustainable new capacity to ensure that the highest priorities are addressed most effectively and efficiently and that new approaches are developed to accelerate the pace, reliability, and consistency of the results [41, 46, 47]. Publicly conducted reviews of the concerns related to the problems with the COX-2 inhibitors have underscored the fact that the need is not only for improvements in the framework for pre-market testing of interventions,
but for improved assessment of interventions in active use and an integrated process that brings the two into closer alignment. Because of inconsistencies and overlaps even at its currently limited level of work, the need is also not only for substantially increasing the level of assessment activity but also for restructuring its administration to ensure the efficiency of implementation—in effect to ensure the presence of a trusted, independent capacity to ensure the successful execution of a program of work in a transparent fashion, with the active engagement of key stakeholders as part of the process. The notion of independence is central because of past instances of political will intervening in scientific processes, as is scientific credibility to the patients and health professionals who will ultimately determine the success of the work.
Presented below are several approaches to developing the sustainable capacity needed for studies on the comparative effectiveness of healthcare diagnosis and treatment. They are grouped into four categories according to the nature and source of the funds for their support—incremental funding, public funding, private funding, and public-private funding (Table A-3). Each of the approaches is based on an existing or recent model of some kind—ranging from government agencies such as the Agency for Healthcare Research and Quality to the user fee model of the Food and Drug Administration to the operation of the highway trust-funded Transportation Research program of the National Research Council in The National Academies. Although presented as discrete models for discussion purposes, they are clearly not mutually exclusive. Many variations on the themes are possible, and several have been the focus of recent discussion and recommendations [46-48, 51]. All are succinctly described below, with elaboration on some presented in Appendices Three through Six. A brief
TABLE A-3 Models for Enhancing Capacity
|Incremental funding augmentations|
|• Incremental model|
|Public funded entity|
|• Executive branch agency model|
|• Independent government commission model|
|• Legislative branch office model|
|Private funded entity|
|• Operating foundation model|
|• Investment tax credit cooperative model|
|Public-private funded entity|
|• User fee public model|
|• FFRDC public model|
|• Independent cooperative model|
|• Independent quasi-governmental authority model|
summary of some of the pros and cons raised for the three most prominent proposals is presented in the following section on Decision and Implementation Considerations.
Incremental Funding Augmentations
The approach most resembling the status quo is to incrementally grow existing activities in both the public and private sectors. In this scenario, the expectation would be for AHRQ to receive gradually increased appropriations for its comparative effectiveness research program, NIH to steadily increase its priority on investing in clinical effectiveness research as part of its Roadmap Initiative, industry working earlier with FDA and payers in conversations to anticipate post-market needs from the outset of pre-market testing, CMS to expand its Coverage with Evidence Development initiative, and insurers and manufacturers to increase their investment in comparative effectiveness research. The formation of an interagency collaborative is one that might seek to utilize the strengths of the individual agencies while reducing the burden on any single agency. A partnership between AHRQ, NIH, and FDA, for example, could potentially carry out a number of the functional elements described in the previous section. While there are ample precedents for interagency collaboration, most are on a scale too limited for true coordination, collaboration, and harmony in determinations, in particular given the potential scope of the activities required.
Publicly Funded Entity
An alternative to the incremental approach is to establish a dedicated entity with substantial funding (see Implementation Considerations below) as a largely public-funded executive branch agency or independent government agency, or as an adjunct to Congress. Three of the major reviews to date of this issue have expressed the view that the work is substantially in the public interest and is therefore best funded directly out of public funds [41, 46, 47].
Executive Branch Agency Model
The most straightforward public funded model is an expanded and appropriated mandate to an existing or newly created federal agency.
AHRQ-based The single agency whose mandate most closely parallels these priorities is AHRQ, which undertakes many of the functional components
discussed in the previous section, including prioritization of conditions or interventions to be evaluated and conducting systematic reviews. As AHRQ has an existing framework for many components through its Effective Health Care program, it may have the greatest flexibility to accommodate a mandate with increased functionality. A concern about this approach is that the current mandate of AHRQ goes far beyond the Effective Health Care program, and an expansion of functionality, such as taking on significant components of an increased capacity for comparative effectiveness research, could become the dominant force within AHRQ and supersede its other activities.
NIH-based There are other agencies that address some of the issues related to comparative effectiveness, such as NIH. NIH has strong credibility and experience with both primary and secondary clinical effectiveness research. However, the structure of NIH with institute divisions that each have oversight on their specific research agendas, combined with the mandate of the NIH to conduct basic and translational research, is less in line with the outcomes-oriented approach of comparative effectiveness research.
Other HHS-based Another option is the formation of an entirely new agency with HHS to undertake the elements of comparative effectiveness information. This new agency would rely on collaboration with other agencies, but as a lead organization would have the capacity to take on many of the functional components described. A clear advantage to this approach is that in creating a new agency, a new approach could be used to establish the governance of the agency to maximally insulate it from political influence.
Other executive-based Many other examples exist of independent, freestanding operational federal government agencies with presidentially appointed chief executives, including the National Science Foundation, the National Aeronautics and Space Administration, the Export-Import Bank, and the Small Business Administration. An expressed concern relevant to all Executive Branch models is that a Cabinet-level agency using funds appropriated by Congress, and subject to political oversight, may be especially vulnerable to political influence in its work.
Independent Government Commission Model
Several models exist of federally funded independent government commissions or agencies with operational programs and responsibilities. Although not operating major research programs of the sort considered here, their governance seeks shared non-partisan oversight of the program
of policies and activities. Two such agencies are the Consumer Product Safety Commission and the Federal Trade Commission.
Consumer Product Safety Commission (CPSC) The CPSC is a three-commissioner independent agency charged with ensuring the safety of consumer products. Although it does not maintain an extensive research program, CPSC maintains a consumer hotline and reporting system for product problems and product-related injuries, and it also operates the National Electronic Injury Surveillance System which monitors the injuries treated in about 100 active hospital emergency rooms throughout the nation. CPSC courses of action include developing voluntary and mandatory standards, requiring the issuance of consumer warnings, issuing recalls of products, and banning dangerous consumer products.
Federal Trade Commission (FTC) The FTC is an independent government agency charged with consumer protection and the elimination of anti-competitive business practices. Its five commissioners are appointed to 7-year terms by the President. No more than three of the commissioners can belong to any single political party. Its three main divisions are the bureaus of consumer protection, competition, and economics, each with analytic and enforcement responsibilities. Other examples include the Consumer Product Safety Commission, the Federal Communications Commission, the Occupational Safety and Health Review Commission, the Securities and Exchange Commission, the Postal Rate Commission, and others.
Legislative Branch Office Model
Office of Technology Assessment (OTA) From 1972 to 1995 the OTA operated to provide Congress with objective and independent analysis and advice on matters of science and technology with policy implications. OTA was governed by the Technology Assessment Board (TAB), made up of six senators and six representatives with equal representation from each party. The TAB appointed the OTA Director and a member advisory council from industry, academia, and elsewhere outside the federal government. The comptroller general of the United States and the director of the Congressional Research Service also served as members. OTA was funded out of appropriations.
Medicare Payment Advisory Commission (MedPAC) MedPAC is a 17-member committee, with its member appointed to 3-year terms by the comptroller general to serve as independent federal advisors to advise Congress on issues affecting the Medicare program. The MedPac mandate is broad, including issues related to payments for Medicare services,
evaluation of access to care and quality of care. It meets publicly to discuss policy issues and seeks input through a variety of other mechanisms. Its recommendations to Congress are issued in reports and briefings. It does not conduct a primary clinical research program, nor does it address issues outside the target Medicare population, but its mandate could be expanded to do so.
Privately Funded Entity
A clinical effectiveness research entity could be established entirely with private funds. The primary advantage of a private sector approach is that it would be perceived as not being unduly influenced by the government and political considerations. Conversely, it could be seen as having undue influence from the private organizations which fund it, and the resulting recommendations could be seen as playing to the interests of those organizations. In addition, the sustainability of the endeavor would not be guaranteed, nor would the philosophical notion of the importance of the information as a public good. Examples of two possible approaches to substantially increasing private funding are noted below.
Operating Foundation Model
A variety of foundation models exist that could serve as the basis for pooled funds from private sector stakeholders interested in reducing duplicated effort, effecting economies of scale, and accelerating the conduct of clinical research. Foundations dedicated to activities that serve the common good have preferential tax status and can be established either as non-profit corporations or as charitable trusts. They can be established as grant-making entities, for charitable activities, or they can conduct their own programs of activities, as long as they meet the provisions that determine their tax status. The several types include private, corporate, and public foundations. The corporate model is the one likely most familiar to the stakeholders cooperating in a shared program of clinical effectiveness research. In 2004 there were more than 2,500 corporate foundations operating in the United States, and the total giving of the top 50 of these foundations by total giving was nearly $2 billion dollars. Of these 50 top foundations by giving, seven were from pharmaceutical or biotechnology foundations .
Investment Tax Credit Model
Another approach to expanding private support, which could be used in conjunction with the establishment of a cooperation foundation enter-
prise, would be to expand the provisions governing corporate investment tax credits to encourage investments in comparative effectiveness research. The federal research and development tax credit was established to allow private organizations to take a credit against their tax liability equal to 20 percent of certain qualified expenses—specifically those expenses in excess of a predetermined base amount in order to encourage additional R&D above and beyond what a company might otherwise undertake. It has been successful in encouraging R&D investments and, if paired with the development of an independent mechanism for shared governance, pooling, priority setting, and quality control, could provide a basis for expansion of the work needed.
Public-Private Funded Entity
User Fee Public Model
One of the most prominent examples of shared public-private funding in the health arena is the user fee model to expedite the review of drug and biological products. In the 1992 Prescription Drug User Fee Act (PDUFA), the FDA was authorized to collect fees from companies, regardless of the review outcome, for each proposed new drug or biologic. In addition, companies pay annual fees for each manufacturing establishment and for each prescription drug product marketed. Industry provides the funding in exchange for FDA agreement to meet drug-review performance goals, which emphasize timeliness . User fees currently fund about half of new drug review costs—which totaled about $219,841,000 in 2006, with a total budget for drug review of $515,557,000 . By providing needed funds, PDUFA began to address what industry viewed as slow and unpredictable review and approval of new drug applications, while keeping FDA’s high standards. The additional resources allowed FDA to devote more time for guidance to help minimize unnecessary trials and generally improve drug development, with the result that now 50 percent of new drugs are being launched first in the United States, compared to only 8 percent in the years prior to the establishment of PDUFA. This type of user fee model could be employed to support clinical effectiveness studies.
One type of quasi-governmental organization is the Federally Funded Research and Development Center (FFRDC), which is funded at least in part by the federal government, functions as a nonprofit private organization, and is managed by nongovernmental organizations, usually universities or other nonprofit institutions. FFRDCs were first established during
World War II to assist government agencies in meeting specific well-defined technical needs that could not be met by existing government agencies or normal contractor relationships. Currently there are 37 FFRDCs, and, while their missions are quite varied, there are general rules applied to their administration. The sponsoring federal agency is responsible for overall policy and oversight, following guidelines set forth by Office of Federal Procurement Policy (OFPP) Policy Letter 84-1. FFRDC funds come from the sponsoring agency requesting the work or from a line item on congressionally appropriated budgets of the sponsoring agency. In addition, an FFRDC can receive up to of 30 percent of its funding from private sources. Limits have been placed on competition for other government or commercial business with the intent of fostering a strategic relationship between an FFRDC and its sponsor as well as limiting the potential for conflict of interest that this special access may create. Existing FFRDCs fall into four general categories: policy-focused study and analysis centers (e.g., the National Defense Research Institute, administered by RAND, and the Homeland Security Institute, administered by Analytic Services, Inc.), research and development laboratories and research laboratories (e.g., Lawrence Livermore National Laboratory, administered by the University of California; the National Cancer Institute lab at Frederick, administered by Science Applications International Corp.; and others), and systems engineering and integration centers (e.g., the Aerospace Federally Funded Research and Development Center, administered by the Aerospace Corporation).
Independent Cooperative Model
Formal public-private partnerships have been established to facilitate research, and there are two good examples in the health area: the Health Effects Institute and the NIH Public-Private Partnership Program.
Health Effects Institute Chartered in 1980, the Health Effects Institute (HEI) is a nonprofit corporation that provides unbiased, independent research on the health effects of air pollution. Since its inception, HEI has published over 200 reports on over 250 funded studies that inform policy making on various pollutants, such as carbon monoxide, nitrogen oxides, and diesel exhaust. HEI is funded jointly, with roughly half of its funds from the U.S. Environmental Protection Agency (EPA) and half from the 27 companies of the worldwide motor vehicle industry. To accomplish its mission, HEI prioritizes research needs, funds and oversees projects, provides independent review of the projects, and disseminates findings. It is governed by an independent board of directors, a group of science and policy leaders not tied to HEI sponsors that sets goals and priorities, makes final funding decisions, and oversees research. The Health Research Committee, appointed
by the board of directors, selects and oversees projects, and an independent review board helps HEI staff evaluate research. In 2005 EPA grants for the Health Effects of Air Pollution Program totaled $2.2 million, and the total HEI research investments were $7.4 million. Special projects of specific programs can also be funded by other public and private institutions.
NIH public-private partnerships The NIH has for some time maintained an active program of collaborative and co-funded activities with the private sector to address issues that require outside resources. As part of the 2002 Roadmap Initiative, NIH formally established the Program on Public-Private Partnerships (PPPs) as a central resource to provide guidance and advice in their development. Partnerships are established either directly with the NIH or through the Foundation for the NIH (FNIH), an independent public foundation chartered by Congress in 1990 to support the mission of the NIH. As a public foundation, FNIH can solicit donations from nongovernmental sources and can also conduct grant or contract solicitations, reviews, awards, and management. Partners can include foundations, patient advocacy groups, and industry (pharmaceutical, biotechnology, devices, diagnostics, informatics, and other). The governance and specific roles of the partners are also tailored to the particular aims of the partnership. For example, contributions from NIH might include development of tests, assays, and diagnostics; development of databases, biobanks, and repositories; or organization of clinical trials. Industry contributions range from intellectual and fiscal support to samples and data from clinical trials. In May 2005 FNIH was involved in approximately 50 PPPs totaling $280 million. Examples are detailed in Appendix 4 and include the Biomarkers Consortium, a partnership managed by FNIH and governed by an executive committee involving NIH, FDA, CMS, industry, voluntary organizations, and advocacy groups, with the purpose of accelerating the identification, validation, and application of new biomarkers; the Osteoarthritis Initiative, a $60 million initiative partnering NIH institutes and centers with four industry partners ($22 million from Pfizer, Merck, Novartis, and GlaxoSmithKline) in a 7-year study collecting clinical, radiological, and biological data from 5,000 patients with osteoarthritis; and the Grand Challenges in Global Health, a $450 million project administered by the FNIH and prompted by a $200 million commitment from the Bill & Melinda Gates Foundation, with contributions from the U.K. Wellcome Trust and the Canadian Institutes of Health Research, supporting 43 separate research projects aimed at solving the 14 most important challenges in global health.
Independent Quasi-Governmental Authority Model
There are two prominent existing examples of approaches to the establishment of a quasi-governmental entity on clinical effectiveness: The National Academies model and the Federal Reserve model. Although the Federal Reserve does not run a research program, both The National Academies and the Federal Reserve have government mandates, are funded independently, and share certain governance features relevant to the operation of a clinical effectiveness research entity, including: public-private character, non-partisan, independent governance, shared stakeholder priority setting, and central policy authority.
National Academies One approach that takes advantage of existing independence, capacity, and reputation would be to work through The National Academies, which are governmentally chartered and have an established model for managing a cooperative national program of research in the work of the Transportation Research Board (TRB). The TRB has operated since 1920 as a division of the National Research Council and began regularly managing research projects with the establishment in 1962 of the National Cooperative Highway Research Program. In the 1990s Congress, the U.S. Department of Transportation, and the state departments of transportation asked TRB to undertake additional tasks, including management responsibilities for the Transit Cooperative Research Program, guidance of ongoing research programs such as the Long-Term Pavement Performance studies, and management of the Innovations Deserving Exploratory Analysis (IDEA) programs. The policies and activities of the Transportation Research Board are determined and directed by its executive committee acting within the overall policies of The National Academies. Executive committee members are appointed with the approval of the chairman of the National Research Council. The executive committee is composed of representatives from government, industry, and academia who are active in the areas of interest of TRB. Representatives from states, transit organizations, and universities are appointed to liaise between TRB and the organization or institution. The program is funded through resources from state transportation departments, industry associations, the U.S. Department of Transportation, and other organizations interested in the effectiveness of transportation tools and strategies. Applying this model for a program of comparative effectiveness research would place responsibility with the Institute of Medicine (IOM), perhaps in cooperation with the National Research Council. The membership of the IOM includes national and international leaders in health care, and its studies are highly regarded. As with any organization, taking on an operational role of the scope anticipated to meet the need for comparative effectiveness research would require careful consideration
and the development of new administrative and procedural approaches. However, the IOM has indicated a willingness to explore housing such an activity, possibly along the lines of the TRB precedent.
Federal Reserve The Federal Reserve System is a blended quasi-governmental system that serves as the nation’s central banking system with the charge of conducting the nation’s monetary policy, regulation of banking institutions, maintaining the stability of the economy, and providing financial services to depository institutions. The Fed is comprised of the Board of Governors, 12 regional Federal Reserve Banks, numerous private banks, and the Federal Open Market Committee. The Board of Governors, appointed by the President of the United States for 14-year terms, is an independent governmental agency responsible for monetary policy and overall supervision of the integrity of the banking system. It does not receive federal appropriations, and members are not accountable to any specific official, except to the President on matters of misconduct. The regional federal reserve banks are private entities, controlled by local member banks. The Federal Open Market Committee, which determines and administers the sale of government securities, is comprised of the seven members of the Board of Governors and five representatives from the Federal Reserve Banks.
Establishment of a substantially expanded capacity for clinical and comparative effectiveness research will need to address a variety of implementation considerations, including, if established as a new entity, those related to funding, governance, priority setting, research conduct, and finding validation. Some of the relevant principles and administrative issues in identifying the organization’s location, support, structure, and function include those that follow.
Given the complexity of the issues and the intensity of the public interest, concerns, and stakes, scientific credibility is the most important characteristic of the organization vested with the responsibility of managing the conduct of research aimed at determining which health care works best, for whom, and under what circumstances. The organization that functions best in this respect will be the organization that is best able to gain the
trust and confidence of the public, the scientific community, and the other stakeholders involved.
Similarly, the conduct of the scientific enterprise must be insulated from the political processes. In any public endeavor, virtually all interests will seek—and have sought—to use political pressure to affect policies and processes. Whether the focus is funding for different projects, the wording of policies and recommendations, or the make-up of advisory committees, political influence and action has the ability to distort the integrity of the scientific process. Insulation from that influence is key.
Inherent in the notion of scientific credibility is that of stakeholder neutrality. Patients, providers, employers, manufacturers, and insurers all have driving perspectives whose interests are understandable, but whose impact must not intrude on the integrity of the scientific process. Both the organization and its management structure must be well insulated from the disproportionate sway of any particular stakeholder.
Political independence and stakeholder neutrality do not equate to the absence of their engagement in the agenda- and priority-setting process. The determination of the priorities to pursue is a policy exercise in which all relevant stakeholders have a right to engage and to which they can add value. This amounts to an appropriate involvement in the governance process, while safeguarding against interference with the structure or conduct of scientific reviews.
While agenda and priority setting may be shared and brokered responsibilities, management and conduct of the research processes and the determination and validation of research results must be completely insulated from stakeholder influence. Vesting of responsibility for this work ought therefore to be placed, if not in an organization with a history of protecting against such influence, then in one that can construct the necessary procedural firewalls between the processes of policy and priority setting and the design, conduct, and reporting of the scientific studies.
Current work is far short of the need, but described in this paper are a number of organizations now operating with related mandates and activities. Advantage must be taken, where possible, of existing capacity for the establishment of scientific standards, the oversight and conduct of comparative effectiveness research, and the development of the requisite findings and recommendations.
Although a certain measure of process is a precondition for ensuring the integrity of scientific determinations, the organizational decision making, resource allocation, and program conduct must have enough flexibility to be able to respond quickly to emerging issues and changing circumstances.
Transparency of Processes and Results
The credibility, integrity, and usefulness of the studies carried out under the auspices of the responsible organization, and the determinations made, will depend upon the transparency of the work. An assumed character of any determinations, and the studies on which they are based, should be specification and availability of the data on which determinations are based, and clarity as to the processes and tools used in their evaluation.
Support and Management Issues
According to various estimates, funding adequate to meet the information needs of clinical decision makers must increase substantially, in the several billions above current levels [16, 46]. Estimates of this sort are consistent with the fact that simply investing a relatively modest 1 percent of healthcare expenditures to discern which care works best would amount to some $20 billion annually. This is a figure 10 to 20 times more than current federal expenditures for this work, but, if realized, it would clearly improve our ability to get better value from health care. Under the publicly funded approach, the two primary funding options are direct annual appropriations or a set-aside from the Medicare Trust Fund. With a public-private funded approach, proportionately matching contributions could be structured in a variety of ways, including blending small set-asides from Medicare fund expenditures, from private health insurance premiums, or
from manufacturer R&D expenditures. There can be many variations on these themes, but the key concepts are less related to the structure of the funding support than to the value of the yield for better health—the return for the outcomes and efficiency of the nation’s health care.
Governance and Priority Setting
With multiple stakeholders involved, ranging from individuals like patients and providers to institutions like manufacturers and insurers, governance and priority setting should be a shared responsibility. Accomplishing this will require a governance structure that allows for broad stakeholder involvement in setting the agenda, determining priorities, and shaping and executing the decision-making principles.
Research Conduct, Findings and Recommendations
Several important considerations pertain to the conduct of the research supported, including insulating the research process from the governance and priority-setting responsibilities to ensure that the conduct of the science is fully independent of the political dynamics of priority setting; building on the research infrastructure that currently exists for the conduct of the studies; and full transparency in the conduct of the research and availability of data. Administratively, this suggests several elements to the task: identification of study priorities; triage of the research responsibilities to capable institutions; ensuring the quality and consistency of approaches among those institutions; validating the results; and fashioning guidelines and recommendations based on the results.
Summary of Common Approaches to the Independent Model
Because of the challenges to the notion of increasing comparative effectiveness research primarily through a straightforward federal appropriation to the Agency for Healthcare Research and Quality—difficulty of marshaling an appropriation at a level commensurate with the need, lack of political independence, limited ability to draw on other agencies—much of the recent discussion has focused on independent models, often with blended public and private funding. Table A-4 reviews some possible features of the three such models most commonly discussed: an agency (including an agency-linked FFRDC) model, an independent board model, and a hybrid model.
As independent entities, each of these approaches assumes the establishment of a governing board charged with priority setting, broad budget allocation, and fiduciary responsibility for execution of the program of
TABLE A-4 Comparative Effectiveness Research Enterprise Models
|Reference model||NIH or AHRQ||Federal Reserve Board||NAS (IOM/TRB)|
|Priority setting||Agencya/HHS Board||Governing board/staff||Governing board/ISOb|
|Budget allocation||Agency/HHS Board||Governing board/staff||Governing board/ISO|
|Study selection||Agency||Governing board/staff||ISO|
|Agency designation||Agency||Governing board/staff||Governing board/ISO|
|Study certification||Agency||Governing board/staff||ISO|
|Study conclusions||Agency||Governing board/staff||ISO|
|Advantages||Builds on current||Independent||Independent Builds on current|
|Disadvantages||Politically vulnerable||No established credibility||Other missions of ISO|
|Linked to one agency||Duplicate capacity|
a Some proposals suggest creating an agency-associated but privately operated FFRDC (Federally Funded Research and Development Center) to give the work quasi-insulated status.
b ISO, independent scientific organization (e.g., Institute of Medicine, on the model of the NAS Transportation Research Board).
activities. Depending on the nature of support base for the work, this board would presumably be comprised of some mix of key stakeholders, who would be appointed in a fashion that would assure their political independence. Not explicitly discussed here are a number of important issues that also will need to be addressed, such as the loci for fostering new and improved study designs and research methods, the relationship of the findings from the proposed center to those of existing professional organizations and societies, and how the determinations might be used in payment decisions.
Agency or Agency-Linked FFRDC Model
Because an FFRDC is linked to a sponsoring federal agency that assumes responsibility for policy and operational oversight, the most logical agency sponsor of a comparative effectiveness research entity is the Agency for Healthcare Research and Quality, which has both a formal mandate and an existing, albeit small, program of such research [37, 42]. The components of AHRQ’s Effective Health Care Program could be drawn closely and naturally into the work, such as the Evidence-Based Practice Centers, the Centers for Education & Research on Therapeutics, and the Developing Evidence to Inform Decisions about Effectiveness network. On the other hand, even an FFRDC operating outside the federal structure is still substantially associated with its sponsoring agency and not immune from political vulnerability. Although many public-funded entities have demonstrated the capacity for long-term stability and relative insulation from political influence, experiences indicate the difficulties faced when topics sensitive to certain constituencies are engaged. The Congressional Office of Technology Assessment was eliminated, in part on the grounds that it intruded on market forces, and the Agency for Health Care Policy and Research (now AHRQ) was nearly eliminated in 1995 when spine surgeons hostile to its recommendation on management of low back pain brought political pressure—which led to the removal of policy from the name. In addition, the primary association of the FFRDC with one agency might limit the engagement of other federal research agencies with important clinical effectiveness activities, such as NIH, FDA, CDC, and the VA. Furthermore, if the research is to be supported through combined public and private funds, the current FFRDC maximum of 30 percent private funds would have to be altered.
Independent Board Model
An alternative to the federal agency-based approach is to create an entirely new, free-standing entity along the lines of the Federal Reserve Board which governs the nation’s banking system . The philosophy behind the Federal Reserve model is that an enterprise that will affect the direction of 16 percent of the national economy and the health care of every citizen warrants a status with sufficient independence, size, and power to control the landscape. Although most discussions of this approach do not propose the same legislated mandates related to oversight, accountability, and reporting that are vested in the Federal Reserve Board, there are clear advantages to the creation of a capacity with similar independence and reach. In addition to a governing board for priority setting and budget allocation, some versions of this scenario would vest this board with operating
capacity for selecting specific studies, designating the locus of their conduct, and certifying the results and conclusions for dissemination. The possible disadvantages to this model include the need, as a newly established entity, to build credibility from the ground up, and the duplication of existing capacity for independent scientific review that exists in places such as the National Institutes of Health and the National Academies’ Institute of Medicine and that is growing in AHRQ.
A hybrid model, blending reliance on existing independent scientific capacity with the creation of a new free-standing priority-setting entity, would establish a governing board to set priorities and allocate the necessary funds but draw on the IOM to execute the scientific activities. This would have the advantages of ensuring stakeholder involvement in the policy-determining elements of the governing board work, while constructing a firewall between the determination of policy by the governance apparatus and the conduct and oversight of the science. Based on the priorities established by the governing board, the IOM (or AHRQ/NIH, if preferred) would select the necessary studies, identify the study design and approach, designate the appropriate agency (e.g., NIH, AHRQ, CDC, FDA, VA) to identify the research organization and manage the conduct of the work, then the IOM would certify the validity of the results, fashion the conclusions, and initiate the dissemination. Additional advantages to this approach include that, with IOM, it uses an organization that has the established and acknowledged credibility and independence, and draws the capacity of all existing federal agencies into play, as appropriate. The principal disadvantage is that, since one of its basic principles is the notion of the desirability of a firewall to ensure the integrity of the science, the relationship between the governing board and the National Academy of Sciences would have to be solidly established.
As ever-increasing options evolve in health care, current gaps in knowledge and practice about which care works best will persist or worsen without the appropriate information on which to base healthcare decisions. Innovations that are beginning to find clinical application, such as genetic profiling, are on the front end of a wave of technology yet to come. As the boom in pharmaceuticals, devices, and biologics has left us with a need for information on clinical utility, and as innovations in the pipeline come to fruition, the information gap will widen. The rate with which new interventions are introduced into the medical marketplace is currently outpacing
the rate at which information is generated on their effectiveness and the circumstances of best use. If trends continue, the ability to deliver appropriate care will be strained and may be overwhelmed.
A substantially increased capacity to conduct and evaluate research on clinical effectiveness of interventions brings many potential opportunities for significant improvement across a wide spectrum of healthcare needs. An integrated system to examine clinical effectiveness can significantly improve the speed with which evidence is developed on medical interventions through coordination of research activities, and further function to align evident clinical information gaps with research priorities. Clinical effectiveness research, through establishing the best practices in health care, serves as the cornerstone of quality improvement, and an increased ability to determine what has the most clinical utility will drive faster and more rigorous quality improvements. As electronic health records move from concept to reality, the need for coordinated efforts for clinical effectiveness research will become even more pressing but also more feasible. Interoperability among data systems will provide a wealth of data, the power of which will only be realized by a harmonized infrastructure that allows for the scale of clinical effectiveness studies required. The enhanced capability to know what works best will generate substantial and valuable information to support innovation by identifying the key areas where it is needed the most. The options reviewed here offer a sense of the possibilities and opportunities, but the need for swift action is pressing.
1. OECD Health Data. 2006. How Does the United States Compare?
2. Mathers, C., et al. 2001. Estimates of healthy life expectancy for 191 countries in the year 2000: Methods and results. Global Programme on Evidence for Health Policy Discussion Paper No. 38. World Health Organization.
3. Anderson, G. F. 1997. In search of value: An international comparison of cost, access, and outcomes. Health Affairs 16(6):163-171.
4. McGlynn, E. A., et al. 2003. The quality of health care delivered to adults in the United States. New England Journal of Medicine 348(26):2635-2645.
5. Sung, N. S., et al. 2003. Central challenges facing the national clinical research enterprise. JAMA 289(10):1278-1287.
6. Fisher, E. S., and J. E. Wennberg. 2003. Health care quality, geographic variations, and the challenge of supply-sensitive care. Perspectives in Biology and Medicine 46(1):69-79.
7. Wennberg, J. E., E. S. Fisher, and J. S. Skinner. 2002. Geography and the debate over Medicare reform. Health Affairs. Suppl Web Exclusives: W96-114.
8. Wennberg, J. E., and M. M. Cooper, eds. 1999. The Dartmouth Atlas of Health Care. Chicago: American Hospital Association.
9. Gill, P., et al. 1996. Evidence based general practice: A retrospective study of interventions in one training practice. BMJ 312(7034):819-821.
10. Lee, H. Y., et al. 2005. Comparison of evidence-based therapeutic intervention between community- and hospital-based primary care clinics. International Journal of Clinical Practice 59(8):975-980.
11. Lee, J. S., D. M. Urschel, and J. D. Urschel. 2000. Is general thoracic surgical practice evidence based? Annals of Thoracic Surgery 70(2):429-431.
12. Moyer, V. A., A. K. Gist, and E. J. Elliott. 2002. Is the practice of paediatric inpatient medicine evidence-based? Journal of Paediatrics and Child Health 38(4):347-351.
13. Myles, P. S., et al. 1999. Is anaesthesia evidence-based? A survey of anaesthetic practice. British Journal of Anaesthesia 82(4):591-595.
14. Smith, R. 1991. Where is the wisdom…? BMJ 303(6806):798-799.
15. Moses, H., 3rd, et al. 2005. Financial anatomy of biomedical research. JAMA 294(11): 1333-1342.
16. AcademyHealth. 2005. Placement, coordination, and funding of health services research within the federal government. AcademyHealth Report. September.
17. Research!America. 2006. 2005 Investment in U.S. Health Research. Online. Available at http://www.researchamerica.org/publications/appropriations/healthdollar2005.pdf.
18. Advanced Medical Technology Association. 2004. The Medical Technology Industry at a Glance. p. 16, Chart 3.6.
19. Biotechnology Industry Organization. 2006. Chart: Total Biotechnology Patents Granted per Year. Online. Available at http://www.bio.org/speeches/pubs/er/statistics.asp.
20. Foster, L., D. Nanu, and S. Alcock. 2002. Patenting in the biopharmaceutical industry—Comparing the US with Europe. Drug Plus International 1(1).
21. PhRMA (Pharmaceutical Research and Manufacturers of America). 2006. Pharmaceutical Industry Profile. p. 49, Table 4.
22. FDA (Food and Drug Administration). 2005. Prescription Drug User Fee Act (PDUFA): Adding Resources and Improving Performance in FDA Review of New Drug Applications (Executive Summary).
23. Levin, R., E. Moy, and P. F. Griner. 2000. Trends in specialized surgical procedures at teaching and nonteaching hospitals. Health Affairs 19(1):230-238.
24. Kaiser Family Foundation. 2007. How Changes in Medical Technology Affect Health Care Costs.
25. Kuhn, H. 2006. Testimony on Payment for Imaging Services under the Medicare Physician Fee Schedule, July 18, 2006. Online. Available at http://www.hhs.gov/asl/testify/t060718.html. Accessed January 22, 2007.
26. Brindis, R. G., et al. 2005. ACCF/ASNC appropriateness criteria for single-photon emission computed tomography myocardial perfusion imaging (SPECT MPI): A report of the American College of Cardiology Foundation Quality Strategic Directions Committee Appropriateness Criteria Working Group and the American Society of Nuclear Cardiology endorsed by the American Heart Association. Journal of the American College of Cardiology 46(8):1587-1605.
27. Stead, W. W. 2007. The Learning Health Care System: Workshop Summary. Washington, DC: The National Academies Press.
28. Kaiser Family Foundation. 2006. Trends and Indicators in the Changing Health Care Marketplace: Section One. p. 8, Exhibit 1.8.
29. Kaiser Family Foundation and Health Research and Educational Trust. 2006. Summary of Findings: National Survey of Private and Public Employers of Three or More Workers.
30. Research!America. 2005. National Survey. Charlton Research Company.
31. IOM (Institute of Medicine). 2001. Crossing the Quality Chasm. Washington, DC: National Academy Press: Washington, DC.
32. Robinson, J. C. 2006. The commercial health insurance industry in an era of eroding employer coverage. Health Affairs 25(6):1475-1486.
33. Business Roundtable. 2006. New CEO Survey Predicts Slow, Steady Growth for First Half of 2007; More Than Half of CEOs Say Health Care is Greatest Cost Pressure on Their Companies. Washington, DC.
35. Borger, C., et al. 2006. Health spending projections through 2015: Changes on the horizon. Health Affairs 25(2):w61-w73.
36. IOM (Institute of Medicine). 2006. The Role of Purchases and Payers in the Clinical Research Enterprise: Workshop Summary. Washington, DC: The National Academies Press.
37. Heres, S., et al. 2006. Why olanzapine beats risperidone, risperidone beats quetiapine, and quetiapine beats olanzapine: an exploratory analysis of head-to-head comparison studies of second-generation antipsychotics. American Journal of Psychiatry 163(2):185-194.
38. Lieberman, D. A., et al. 2000. Use of colonoscopy to screen asymptomatic adults for colorectal cancer. Veterans Affairs Cooperative Study Group 380. New England Journal of Medicine 343(3):162-168.
39. McDonnell, K., and P. Fronstin. 1999. EBRI Health Benefits Data Book. Washington, DC: Employee Benefit Research Institute.
40. NIH (National Institutes of Health). 2007. Fact Sheet: Research Into What Works Best.
41. Academy Health. 2005. Placement, Coordination, and Funding of Health Services Research within the Federal Government. Washington, DC.
42. Eddy, D. M. 2007. Linking electronic medical records to large-scale simulation models: Can we put rapid learning on turbo? Health Affairs 26(2):w125-136.
43. Eddy, D. 1989. Screening for breast cancer. Annals of Internal Medicine 1;111(5): 389-399.
44. Petitti, D. 2007. Personal communication. April 2.
45. Clancy, C. M. 2006. Getting to ‘smart’ health care. Health Affairs 25(6):w589-w592.
46. Wilensky, G.R. 2006. Developing a center for comparative effectiveness information. Health Affairs 25(6):w572-w585.
47. Kupersmith, J., et al. 2005. Creating a new structure for research on health care effectiveness. Journal of Investigative Medicine 53(2):67-72.
48. Hopayian, K. 2001. The need for caution in interpreting high quality systematic reviews. BMJ 323(7314):681-684.
49. Buto, K., and P. Juhn. 2006. Can a center for comparative effectiveness information succeed? Perspectives from a health care company. Health Affairs 25(6):w586-w588.
50. Rowe, J. W., D. A. Cortese, and J. M. McGinnis. 2006. The emerging context for advances in comparative effectiveness assessment. Health Affairs 25(6):w593-w595.
51. The Health Industry Forum. 2006. Comparative Effectiveness Forum Executive Summary. November 30. Washington, DC.
52. Cook, D. J., et al. 1996. Stress ulcer prophylaxis in critically ill patients. Resolving discordant meta-analyses. JAMA 275(4):308-314.
53. Jadad, A. R., D. J. Cook, and G. P. Browman. 1997. A guide to interpreting discordant systematic reviews. CMAJ 156(10):1411-1416.
54. Eddy, D., and L. Schlessinger. 2003. Validation of the archimedes diabetes model. Diabetes Care 26(11):3102-3110.
55. Owens, D. 2007. Personal communication. February 13.
56. Juhn, P. 2007. Personal communication. February 25.
58. IOM (Institute of Medicine). 2006. The Future of Drug Safety: Promoting and Protecting the Health of the Public. Washington, DC: The National Academies Press.
Information is not currently available to characterize precisely the national expenditure on clinical effectiveness research, but the investment is clearly far short of the need. If only 1 percent of the nation’s $2 trillion health care bill was devoted to understanding the effectiveness of the care purchased, the total would come approximately to $20 billion annually. In contrast, only $15 million annually has been specifically appropriated by Congress to the Agency for Healthcare Research and Quality (AHRQ), under section 1013 of the Medicare Prescription Drug, Improvement, and Modernization Act (MMA), for comparative effectiveness research, the issue for which the shortfall is greatest. The total of all appropriations to all federal agencies—the National Institutes of Health (NIH), the Veteran’s Health Administration (VHA), the Department of Defense (DOD), the Centers for Medicare and Medicaid Services (CMS), the Food and Drug Administration (FDA), AHRQ, and the Centers for Disease Control and Prevention (CDC)—for all health services research amounts to about $1.5 billion, only a small portion of which is devoted to clinical effectiveness research . Apart from federal appropriations, the insurance industry currently undertakes significant effort and investments annually in assessment of clinical interventions, and healthcare manufacturers devote more than $15 billion dollars annually to their sponsorship of Phase III and IV clinical trials that provide insights on clinical effectiveness . Data are not currently available on the amount of direct expenditures by professional societies on primary and secondary clinical effectiveness research. Even accounting for all the support from all of the sources involved, the aggregate national commitment to assessing the effectiveness of clinical interventions is likely well under 1 percent—far below the standard that any company would expect to invest in work to evaluate and improve its products. Presented below is a summary of the institutions and activities currently engaged in trying to meet the need for better information on the
effectiveness of healthcare interventions, beginning with a short description of relevant terms and definitions.
As used in this paper, clinical effectiveness research can be described as either primary or secondary. For our purposes, primary refers to the direct generation of evidence through the use of a specific experimental methodology. Secondary refers to the systematic gathering and evaluation of primary research information to further the understanding of common conclusions or disparate results.
Primary Clinical Effectiveness Research
In this respect, primary clinical effectiveness research refers to the specific design and implementation of structured research protocols to produce data on the results of one or more diagnostic or therapeutic interventions of interest. Examples include certain randomized controlled trials, practical clinical trials, cluster randomized trials, observational studies, and cohort studies, including registries. Some of these studies focus only the efficacy of an intervention—the extent to which an intervention produces a beneficial result under ideal circumstances. But many also examine the effectiveness of an intervention when used under ordinary circumstances—including evaluation in broader patient populations and healthcare delivery settings, or the relative risks and benefits of competing therapies. Both types of evaluation are important to gain an understanding of which interventions work best, for whom, and under what circumstances.
Evidence synthesis, or secondary clinical effectiveness research, refers to the structured assessment of evidence from multiple primary studies to derive conclusions, which are considered to have greater weight than an individual study alone. Terms sometimes used to describe activities of secondary clinical effectiveness research include systematic review and technology assessment, which both describe a systematic method of identifying, assembling, and interpreting a body of data to validate or extend the interpretation of single trials, lend context to individual trials, and, where possible, arrive at common conclusions. For the purpose of this discussion the two are seen as equivalent processes, although in general practice the audiences for these products are different. Systematic reviews are frequently published through the peer-reviewed literature or other mechanisms to inform policy and practice, while technology assessment is often carried out
to assist in decision making on a policy level regarding the effectiveness of a new intervention.
Within the overall umbrella of clinical effectiveness research, the most practical need is for studies of comparative effectiveness, the comparison of one diagnostic or treatment option with one or more others. In this respect, primary comparative effectiveness research involves the direct generation of clinical information on the relative merits or outcomes of one intervention in comparison to one or more others, and secondary comparative effectiveness research involves the synthesis of primary studies to allow conclusions to be drawn. Secondary comparisons of the relative merits of different diagnostic or treatment interventions can be done either directly, through collective analysis of the results of multiple head-to-head studies, or indirectly, in which case the treatment options have not been directly compared to each other in a clinical evaluation, and inferences must be drawn based on the relative effect of each intervention to a specific comparison, often a placebo.
Other Related Terms
Other relevant terms often used in the context of clinical effectiveness research discussions include cost-effectiveness analysis and cost-utility analysis. In cost-effectiveness analysis the economic cost per specified unit of health gain—e.g., reduced mortality or morbidity—is determined for a given intervention or family of interventions. This allows for one measure of the relative value of an intervention to be estimated in comparison to alternatives. Cost-utility analysis is a form of cost-effectiveness analysis estimating the cost of a specific utility gain, usually to the patients targeted—e.g., quality-adjusted life-years—for an individual intervention. Finally, the term health services research refers broadly to the multidisciplinary field of scientific investigation that studies how systemic factors shape the effectiveness of health care for different populations, including how social factors, financing systems, organizational structures and processes, health technologies, and personal behaviors affect access to health care, the quality and cost of health care, and measures of population health and well-being .
Primary Clinical Effectiveness Research
Conduct of primary research that produces data on the clinical effectiveness of candidate diagnostic and therapeutic interventions is sponsored and managed by a variety of federal agencies, including NIH, CMS, VHA,
FDA, CDC, and AHRQ. Similarly, a number of private initiatives provide information important to primary clinical effectiveness research, including the sponsorship by the healthcare industry of study protocols for their products, the coordinated work of health maintenance organizations to use their clinical databases to study clinical effectiveness, and work involving various payers in data consortia to improve the availability of information for tracking intervention safety and effectiveness. Efforts are under way through the network of regional health information organizations (RHIOs) to advance the progress and interoperability of electronic health data systems, which will enhance the capacity for primary clinical effectiveness research [3-5]. Finally, the use of mathematical modeling to construct clinical scenarios is a form of primary research in that it creates new information by building on established work and uses biological science and probability assumptions to project and test outcomes.
National Institutes of Health
NIH is the largest federal sponsor of clinical research. No direct mandate or prioritization process exists at the NIH for performing comparative effectiveness trials, but many trials of this nature have been conducted, sometimes in collaboration with CMS or AHRQ. Over the past quarter century, several institutes have sponsored or co-sponsored large multicenter comparative effectiveness studies, including studies of balloon angioplasty plus carotid stenting vs. carotid endarterctomy for stroke prevention; behavioral therapy vs. pharmacotherapy for alcoholism; various antipsychotics for schizophrenia; newer vs. older antihypertensive and lipid-lowering agents to prevent heart attacks; lung volume reduction surgery vs. standard medical care for emphysema patients; use of intensive therapy vs. routine glucose monitoring and control in the prevention of diabetes complications; addition of Nevirapine vs. Azidothymidine (AZT) alone in reducing mother-to-child transmission of HIV; combination therapy vs. single drug therapy (doxaziosin, finasteride) to treat benign prostatic hyperplasia; and use of implantable cardiac defibrillators (ICD) vs. antiarrythmic drugs in preventing sudden cardiac death. These comparative effectiveness trials represent an investment of about $660 million since 1982—representing a small share of its total budget of approximately $300 billion over that period [6, 7]. Through the National Library of Medicine, the NIH, in collaboration with the FDA, also maintains a national computer searchable database, ClinicalTrials.gov, to track all federal and privately sponsored clinical trials. This resource was mandated as part of the FDA Modernization Act and requires that any clinical trial being conducted to support an investigational new drug application be included in the database. A related effort, GlobalTrialBank.org, is under way under the auspices of the Ameri-
can Medical Informatics Association with the goal of forming a worldwide repository of clinical trial protocols and data, with the aim of facilitating evidence generation, as well as comparison and evaluation of trials.
Centers for Medicare and Medicaid Services
CMS does not conduct clinical research directly, but its data systems, demonstration and evaluation activities, and coverage policies offer powerful resources for assessing and monitoring clinical effectiveness. Claims data are maintained on the more than 42 million Americans served under Medicare and on the 47 million low-income people covered under Medicaid. Medicare and Medicaid each maintain databases of administrative data which can be accessed by authorized researchers. These data have been a rich resource for the study of practice variation patterns, the effects of volume on surgical outcomes, disparities in medical care, and other issues related to profiles of health services delivery and quality. In addition, they have been used to study the relative effectiveness of different interventions, such as bariatric surgery and treatments for myocardial infarction and asthma [8-10]. The VHA databases have been recently linked with Medicare data to create a merged database, the Diabetes Epidemiology Cohort (DepIC), which allowed an increased accuracy in the determination of diabetes prevalence in the VA population and which has the potential for significant contributions to insights on the comparative effectiveness of interventions . With the passage of the Medicare Modernization Act (MMA) Part D benefit, which makes Medicare enrollees eligible for prescription drug coverage, extensive new opportunities are presented for assessing clinical effectiveness in a post-market environment by linking Part D data to data from Parts A, B, and C in the conduct of public health research. CMS has thus far been largely precluded from using its demonstration and evaluation authority to provide clinical effectiveness insights because this authority is required to be used only for delivery-related demonstrations that are budget neutral. With the recent launching of the CMS initiative on Coverage with Evidence Development, the prospect is emerging for a new capacity to generate important information on the most effective application of clinical interventions by requiring that coverage, for certain populations on which existing effectiveness information is limited, be contingent on subsequent data collection via a registry or other mechanism. Under the CED initiative, for example, CMS has entered into partnership with the American College of Cardiology to maintain a registry that will track the experience of medical patients receiving implantable cardioverter defibrillators (ICDs). Other CED possibilities include FDG-PET for suspected dementia, off-label uses of colorectal cancer drugs (co-sponsored with NCI), and use of home oxy-
gen therapy in moderate COPD (co-sponsored with National Heart, Lung, and Blood Institute [NHLBI]).
Food and Drug Administration
Although FDA supports selective, relatively small-scale intramural research projects, it does not conduct primary clinical effectiveness research. It requires such work by manufacturers to establish basic safety and efficacy information as part of the drug, biologic, and device approval processes (see Healthcare Industry below). FDA also collects and maintains data directly as part of its adverse event reporting process and related post-marketing surveillance work, and it frequently seeks commitments from sponsors to conduct post-market surveillance as a condition of approval. One key issue, which has been the subject of recent scrutiny, is the absence of a practical mechanism to ensure the conduct of the post-market study commitments, with the result that many are never completed. An IOM report on drug safety notes that of over 1,200 drug and biologic post-market commitments, about two-thirds had yet to be initiated . Recently, FDA has proposed the development of an integrated national network, a sentinel network, formed through a series of public-private partnerships using developing electronic information technology systems to collect, analyze, and disseminate medical product safety information to healthcare practitioners and patients at the point of care. In general, the FDA does not require comparative effectiveness information to grant market approval of an agent or innovation.
Veterans Health Administration
The VHA has a strong clinical research program as a result of its 157 medical centers and more than 1,300 ambulatory, residential, and home- and community-based sites of care, and a state-of-the-art computerized clinical data system with 8 million patient records that allows the rendering of large quantities of patient data for analyses on a wide variety of clinical research topics. With more than 3,000 researchers throughout the system and specially designated “centers of excellence,” the VHA devoted over $400 million in 2005 to clinical research, with a nearly equivalent amount coming from its medical care account to support research efforts. In addition to VHA-appropriated funds, research activities are supported through NIH grants and through the conduct of research sponsored by pharmaceutical companies. A substantial share of this research focuses on clinical effectiveness. Multicenter clinical trials investigate the best therapy for various conditions, including AIDS, alcoholism, schizophrenia, stroke, and Parkinson’s disease, as well as health and rehabilitation services research.
Some examples of this research that have yielded valuable information on comparative clinical effectiveness include coronary artery bypass grafts vs. percutaneous coronary intervention for revascularization of high-risk patients , intravenous administration vs. subcutaneous administration of erythropoietin for severe anemia in hemodialysis patients , colonoscopy vs. sigmoidoscopy for primary screening of colon cancer , arthroscopic lavage vs. arthroscopic débridement vs. placebo , and new vs. old drugs for schizophrenia .
Centers for Disease Control and Prevention
CDC funds some health services research to guide decisions on public health services and systems, and some of this research may also examine effectiveness of some therapies in the area of infectious disease or vaccines . In addition, CDC maintains a number of national data systems that are important resources for certain types of studies. National Vital Statistics are collected by CDC through a cooperative arrangement between states and the National Center for Health Statistics (NCHS). NCHS also operates the National Health Interview Survey and the National Health and Nutrition Examination Survey, which use personal interviews and health examinations to track important trends in health status and health-related behavior. Along with other information collected from health records, these data are used to evaluate the impact of health policies and programs and support other research activities related to health trends, services, and status. Also operated out of CDC, the Vaccine Safety Datalink (VSD) is a collaborative project with America’s Health Insurance Plans (AHIP), eight large managed care organizations, and the Immunization Safety Office of CDC. Using administrative data sources, VSD’s linked database has been used to provide comprehensive medical and immunization histories, provide guidance on immunization policy decisions, and provide a platform for research on the safety of vaccines .
Agency for Healthcare Research and Quality
AHRQ uses its more than $300 million annual budget to support health services research, improve the quality of health care, and promote evidence-based decision making. Among federal agencies, AHRQ carries a mandate through section 1013 of the Medicare Prescription Drug, Improvement, and Modernization Act (MMA), and a related $15 million appropriation, to perform research with a focus on the outcomes, comparative clinical effectiveness, and appropriateness of pharmaceuticals, devices, and healthcare services. The provision has led to the creation of AHRQ’s Effective Health Care Program, which has three components: synthesizing
existing studies into comparative effectiveness reports (CERs) by evidence-based practice centers, developing evidence including research aimed at filling knowledge gaps about treatment effectiveness (DEcIDE centers), and improving communication of complex scientific findings to a variety of audiences (Eisenberg Center). Of these, the Developing Evidence to Inform Decisions about Effectiveness (DEcIDE) network supports work in 13 centers to perform primary research on clinical effectiveness.1 The DEcIDE centers have, collectively, access to de-identified medical data for over 50 million patients, including Medicare’s 42 million beneficiaries, which allows accelerated practical studies about the outcomes, comparative clinical effectiveness, safety, and appropriateness of healthcare items and services. Initial research focuses on the outcomes of prescription drug use and other interventions for which randomized controlled trials would not be feasible or timely or else would raise ethical concerns that are difficult to address. Comparative effectiveness studies initiated through DEcIDE include atypical vs. conventional antipsychotic medications, beta-adrenergic antagonists, new therapies for glucose control in diabetes mellitus, and extended use of clopidogrel in drug-eluting stents. AHRQ also maintains the Medical Expenditure Panel Survey (MEPS)—including the Household Component Survey, the Medical Provider Component Survey, and the Insurance Component Survey—which are used in policy-related and behavioral research on healthcare use, spending, and insurance coverage .
Industry-sponsored trials represent a significant proportion of the billions invested by health manufacturers in research and development. For example, well over half of the $37 billion that pharmaceutical companies spent in 2004 on R&D was devoted to the conduct of Phase I to IV clinical trials. Of this total, about 40 percent went to the Phase III and IV trials that have particular relevance to clinical effectiveness . Total R&D spending (including engineering research and pre- and post-marketing activities) by the top 100 device manufacturers was estimated to be $47.8 billion in 2003. Many of these studies are conducted with academic investigators and others are managed by contract research organizations (CROs). The data from these studies are considered proprietary and reside with the companies sponsoring the trials. While some industry-sponsored work is devoted to
1 University of Illinois at Chicago; Acumen, LLC—Data Development for Part D; Johns Hopkins University; University of Maryland at Baltimore; Brigham and Women’s Hospital—Antipsychotic Treatment in Older Adults, Outcome Sciences; University of North Carolina; Duke University; RTI International; University of Pennsylvania School of Medicine; Harvard Pilgrim Health Care, Inc.; University of Colorado at Denver; and Health Sciences Center, Vanderbilt University Medical Center.
comparative, or head-to-head, studies with other products, they have been a small percentage of the studies that industry sponsors, perhaps because of the sense of potential bias from a commercially sponsored comparative study [15, 20].
Integrated Delivery Systems
Large healthcare systems, such as large health maintenance organizations (HMOs), have the capacity to use both their clinical and administrative data systems for primary clinical effectiveness research—some sponsored out of their own resources, others sponsored by NIH or commercial concerns. Kaiser Permanente, Geisinger Health, and Group Health, for example, have efforts to utilize their large databases for improved understanding of what works best for which patients. Kaiser Permanente has integrated participation in multiple networks for information gathering into its routine approach to cancer care, including its own detailed database as well as participation in several national databases, such as the NIH-sponsored Cancer Research Network (CRN) . At Intermountain healthcare, evaluation of large patient databases on diabetes management allowed for a rapid series of process changes which dramatically improved diabetes control . Leadership has grown out of these systems for collaborative work on a number of dimensions. The Health Maintenance Organization Research Network (HMORN) is composed of 15 HMOs covering over 15 million individuals. There are formal research programs that utilize the electronic medical records within each individual organization, and the HMORN is also a designated center for effectiveness. Through this linkage, projects are examining asthma drug use, safety during pregnancy, survival of COPD patients exposed to specific therapies, antibiotic use in children, and many other areas. Similarly, the National Cancer Institute–funded HMO Cancer Research Network is a collaborative effort whose aim is to improve the effectiveness of preventive, therapeutic, and supportive interventions for cancer care (crn.cancer.gov).
Patient registries provide an ever-expanding primary source of information about the performance of different interventions and are sponsored and maintained by many groups—e.g., companies, professional societies, academic researchers, and government agencies. AHRQ defines a patient registry as an organized system with a stated predetermined purpose that uses observational study methods to collect uniform data in order to evaluate specified outcomes for populations defined by a particular disease, condition, or exposure. There are multiple types of registries, including
product, health services, and disease and condition. Many of the hundreds of patient registries in the United States are sponsored at individual academic medical centers or by organizations centered around specific diseases, while others are product registries maintained for post-marketing surveillance. For example, the IVAX Clozapine registry, mandated by the FDA, is designed to detect adverse events associated with use of the drug. Another example is the Cystic Fibrosis Foundation patient registry, which has been in existence for nearly 40 years and is used to help improve care for patients with cystic fibrosis. The quality of patient registries—and their utility for evaluating clinical effectiveness—varies. To begin improving the consistency, quality, and utility of registries, AHRQ has recently developed a guide for the creation and utilization of registries.
Payer-Related Data Consortia
If they are appropriately linked, health insurance claims can also provide a potentially strong resource for assessing the clinical effectiveness of various interventions. In July 2006 the AQA Alliance and the Hospital Quality Alliance (HQA) announced the formation of a joint effort to combine resources to identify, collect, and report data across the variety of care settings they represent through data aggregation, including information on quality of physician performance; cost-of-care measures; quality of care for specific conditions, such as heart attack or pneumonia; and other measures. A related variation on this theme, typical of the potential for such consortia, is the emerging Health Plan Consortium for Public Health, composed of health plans covering an estimated 100 million individuals, which is aimed initially at the use of participating organizations’ databases to track the long-term safety of vaccines. The scale of this project will greatly improve the timeliness, generalizability, and statistical power of the record linkage studies, and, if successful, will expand to similarly track the experience for pharmaceuticals.
One example of emerging work to generate evidence in a novel fashion is the use of mathematical models. At Kaiser Permanente, for example, the Archimedes project is a mathematical modeling system that creates a virtual healthcare universe in which hospitals, doctors, patients, organs, and cells are modeled using mathematical equations and other formalisms. “Trials” may then be performed against this virtual medical universe in order to make predictions about the consequences. The model spans all levels, from physiology of disease to clinical treatment and organizational operations. Data were obtained from a number of different data sources, including
epidemiological studies, clinical trials, medical center records, and hospital accounting departments. Archimedes uses this complex model to offer consulting services regarding healthcare questions such as how to design a clinical trial, which drug shows better effectiveness, or how much money an organization might save through a specific treatment plan. Because modeling builds upon existing trial data, systems like Archimedes might offer an efficient way to develop comparative information on competing therapies.
Evidence Synthesis (Secondary Clinical Effectiveness Research)
Much of the work to marshal evidence for conclusions about clinical effectiveness takes the form not of primary data generation and analysis, but of systematic reviews and meta-analyses of existing studies. Such secondary clinical effectiveness research is sponsored and conducted by a variety of organizations with overlapping and intersecting activities and interests, including federal agencies, state agencies, insurer and insurer-related organizations, independent assessment centers, professional groups and societies, university centers, and consortia. Despite the effort and expense that goes into these analyses, often the results of the assessments are inconclusive with respect to a given intervention. Furthermore, those technology assessments that include a comparative component generally compare a new or evolving technology to an established technology, as opposed to head-to-head comparisons with other emerging technologies .
Most federal health agencies are involved at some level in the sponsorship of evidence synthesis for the purpose of drawing conclusions, making recommendations, or crafting policy related to the use of clinical diagnostics and treatment interventions. Perhaps the best known is the work of the U.S. Preventive Services Task Force (USPSTF), which has since 1984 conducted systematic reviews of the evidence in support of clinical preventive services, applied rigorous criteria to classify and rate the level of the evidence, and, based on the overall strength of the evidence for a given condition, offer conclusions and recommendations. The USPSTF is now sponsored by AHRQ, and its approach has set a standard reference point for much of the subsequent work to synthesize evidence and make clinical recommendations. Building on this work, AHRQ has established a network of 13 AHRQ-sponsored evidence-based practice centers2 (EPCs) that review
2 Blue Cross and Blue Shield Association Technology Evaluation Center; Duke University; ECRI; Johns Hopkins University; McMaster University; Oregon Health & Science University; RTI International—University of North Carolina; Southern California; Stanford University—
literature, perform technology assessments, and produce evidence reports including comparative effectiveness reviews. The EPCs are located at the Technology Evaluation Center of the Blue Cross and Blue Shield Association, ECRI, and a number of universities: Duke, Johns Hopkins, Oregon Health & Sciences, North Carolina, Southern California, Stanford-UCSF, Tufts-New England Medical Center, and Minnesota, as well as McMaster, Alberta, and Ottawa in Canada. Now part of the AHRQ Effective Healthcare Program, EPCs produce studies that span broad issues ranging from clinical to social and behavioral, economic, and other healthcare organization and delivery issues. The EPCs often compare the relative effectiveness of different treatments, including drugs, and also identify gaps in the research. A related AHRQ- and FDA-sponsored effort, the 11 Centers for Education and Research on Therapeutics (CERTS) each focus on a specific patient population or therapeutic area in the conduct of research on ways to advance the optimal use of drugs, biologics, and medical devices by identifying best practices. Although the research typically does not compare specific treatments in a head-to-head fashion, implicit conclusions are drawn through the definitions of best practices. To keep track of the various clinical guideline development efforts, AHRQ also sponsors the National Guideline Clearinghouse (www.guideline.gov), which was originally developed in conjunction with the AMA and AHIP and is managed by ECRI (see below).
CMS draws directly on AHRQ-sponsored technology assessments for the use of its Medicare Coverage Advisory Committee (MEDCAC) to inform CMS coverage decision making. MCAC is charged with determining whether scientific evidence is adequate to support the routine clinical use of certain interventions in the population of Medicare beneficiaries as well as with assessing how the effectiveness of a candidate intervention compares with the effectiveness of established practice. Occasionally the coverage group in CMS will conduct its own evidence syntheses or build off related work done through the Cochrane Collaboration and related technology assessment organizations.
Various NIH institutes sponsor evidence syntheses as part of their initiatives to translate research results for provider and public use. The National Heart Lung and Blood Institute has, for example, supported a variety of committees charged with developing guidance on issues such as high blood pressure treatment, blood cholesterol management, tobacco cessation programs, and asthma management. Other institutes have sponsored guideline development in a variety of areas, including management of hepatitis C, celiac disease, and total knee replacement. In addition, since
University of California, San Francisco; Tufts University—New England Medical Center; University of Alberta, Edmonton; University of Minnesota, Minneapolis; University of Ottawa.
1977 the NIH has sponsored consensus development conferences and state-of-the-science conferences to offer public and professional guidance on matters of important clinical concern. Recently AHRQ’s EPCs have begun providing the evidence reviews for the use of the panels convened to offer the recommendations.
The Department of Veterans Affairs (VA) also sponsors evidence reviews through the VA Technology Assessment Program (VATAP). VATAP is a national program within the Office of Patient Care Services dedicated to advancing evidence-based decision making in the VA. It provides systematic reviews on key healthcare technology issues to help in the service profile decisions of senior VA policy makers. VATAP evaluations encompass devices, drugs, procedures, and organizational and supportive systems used in health care.
The Department of Defense’s TRICARE Management Activity also has a process in place to evaluate medical interventions to make coverage determinations. Often, these evaluations arise out of the appeals process, through physician or beneficiary inquiry, or through TMA’s monitoring of the scientific literature. The group that formulates these evaluations reviews the scientific evidence; utilizes technology assessments, which are either accessed through subscription to a private service (currently Hayes) or obtained as custom reports from a private service (currently ECRI); and reviews published policy statements. A wide variety of interventions are assessed, including pharmaceuticals, devices, and procedures.
Certain states have established programs to conduct technology assessments and systematic reviews to assist in their decisions about Medicaid coverage approaches and the policies of state health insurance agencies. California, for example, has established the California Health Benefits Review Program with a legislative mandate to analyze the medical, financial, and public health impacts of proposed changes in mandated health benefits. The analyses have separate components for medical effectiveness, utilization and cost analysis, and coverage impact on public health. In addition, 13 state Medicaid programs have joined in support of the Drug Effectiveness Review Project (DERP) at the Center for Evidence-Based Policy at Oregon Health & Sciences University (OHSU) to provide assistance in assessing the pharmaceuticals to be provided under Medicaid. OHSU has also initiated a parallel program, the Medicaid Evidence-Based Decisions Project (MED), designed to create a research pool of evidence-based tools, such as technology assessments (see University Centers below).
Insurers and Insurer-Related Organizations
The largest insurers, such as United Healthcare, Wellpoint, Aetna, Humana, and CIGNA, have substantial in-house capabilities for the conduct of evidence reviews. Although the volume of reviews conducted internally by these organizations tends to be much larger than the number from outside organizations, they also license more formal assessments from technology assessment entities such as the Blue Cross Blue Shield Association’s Technology Evaluation Center (TEC), ECRI, Hayes, and Cochrane (see below), as well as the health technology assessment work of other countries, such as NICE in the United Kingdom. In addition, these larger groups will consult with appropriate specialty societies and academic centers for guidance and help on clinical effectiveness research. Smaller plans and state and regional Blue Cross/Blue Shield organizations generally do not maintain this analytic capacity and instead look elsewhere for guidance. As a result, they depend more on organizations like TEC, which conducts evidence syntheses for clients in both the public and private sectors (including Kaiser Permanente and CMS). TEC was one of the first private sector agencies devoted to assessing evidence of clinical effectiveness of new technologies in direct response to requests from healthcare decision makers. TEC performs about 30 clinical effectiveness and appropriateness reviews per year for various procedures, devices, or drugs, using five criteria, also adopted by a number of other organizations: regulatory approval; effect on net health outcome; applicability of benefits in a real-world setting; impact relative to established alternatives (in health outcomes, not cost); and availability of the intervention. The resulting reports are reviewed and ratified by TEC’s Medical Advisory Panel (MAP) and then distributed to subscribing health plans and provider groups. Another insurer-related activity is the Academy of Managed Care Pharmacy (AMCP), through which health plans utilize comparative analysis when developing formularies. AMCP guidelines provide detailed information on a drug’s economic value relative to alternative therapies. Particular challenges faced by payers include the determination of patient selection criteria, technologies not identifiable in the claim payment process, and the evolution of devices (once a technology has received medical necessity approval, the next generation device is generally not reviewed, although there may be significant cost issues).
Independent Assessment Centers
In addition to TEC, several other independent assessment centers provide evidence syntheses for payers, healthcare delivery organizations, providers, employers, and manufacturers. Two of the most active in this respect are ECRI and Hayes. ECRI (formerly Emergency Care Research Institute)
is an AHRQ EPC that also provides fee-based content, such as commercial information services and technical assistance. ECRI performs some comparative technical assessments, mostly on devices and technologies, but also on some drugs and procedures. Subscribing member organizations also fund ECRI’s Health Technology Assessment Information Service (HTAIS), which develops and disseminates evidence-based health technology assessment information. Similarly, Hayes, Inc. is a health technology assessment company targeted primarily at healthcare professionals and, to a lesser extent, to healthcare consumers. About 275 reports, including horizon scanning (80) and technology assessments (190), are issued annually by Hayes; they evaluate medical and surgical procedures, drugs, biologics, diagnostic and screening tests, medical devices and equipment, and complementary and alternative therapies in order to facilitate decision making in coverage policy, contracting, acquisition, and health technology deployment. Reports focus on new and emerging technologies likely to have a significant impact on the cost, utilization, and quality of health care. Costs to produce a single report range from $20,000 to $500,000, depending on the complexity of the analysis and scope of the literature .
Two other independent activities provide services intended to set the stage for identifying priority future evidence review efforts. HealthTech, a non-profit California-based research and education organization, conducts horizon scanning to help participating healthcare and health industry organizations plan for emerging developments. HealthTech’s research covers new and emerging pharmaceuticals, biotechnology, medical devices, and information technology, focusing in particular on ways to engage emerging technologies for which, by definition, good evidence is lacking for decisions about purchase, deployment, and use. Through literature review, expert interviews, stakeholder analysis, expert panels, and product/developer reviews, HealthTech develops research forecasts as decision-making tools projecting possible technologies and impacts. A related activity, currently positioned as a HealthTech affiliate, is the Center for Medical Technology Policy (CMTP). CMTP is convening groups with interests in identifying critical clinical effectiveness knowledge gaps and plans to facilitate the design and implementation of related studies.
Professional Groups and Societies
Many clinicians rely on clinical practice guidelines for advice on best practices for specific clinical situations, in some cases providing advice on how to choose between options. Currently, most clinical practice guidelines are produced in association with physician specialty societies and published through multiple modalities. There are more than 150 medical specialty societies in the United States, and many are engaged in some form
of evidence review and guideline development, and the work spans beyond medical societies. Altogether, more than 300 organizations have published at least one guideline on the AHRQ-supported website, guidelines.gov. The national guideline clearinghouse currently contains more than 1,900 individual summaries. There is substantial variability in the approaches of the various societies, some conducting the reviews by informal staff-generated activities, and others engaged in large, multifaceted and structured consultations involving the multiple related organizations with similar interests in an issue. For example, the American College of Cardiology and the American Heart Association have a carefully specified cooperative protocol for conducting their reviews and developing their guidelines.
Much of the work of various organizations sponsoring evidence syntheses, from AHRQ and NIH to states, professional societies, and manufacturers, is conducted by university researchers, some of whom reside in university centers substantially devoted to such efforts. These primarily include the 12 universities involved as EPCs, although individual academic researchers do conduct various forms of evidence syntheses within their areas of expertise. For example, the EPC at the University of Oregon does many of the systematic reviews for the work of the AHRQ-supported U.S. Preventive Services Task Force (noted above), and it also does the reviews for the Drug Effectiveness Review Program (DERP) (noted above), which compares evidence about effectiveness and safety of drugs of the same class, and uses information to inform health policy. Between 2003 and 2006, 26 reports were commissioned, and 12 were completed. Reports are also periodically updated. Final reports are delivered to subscribers and are also publicly available. Consumers Union also uses DERP findings as the basis for its Best Buy Drugs Program.
Because of the overlapping nature of the interests, issues, and approaches, a number of collaborative efforts have emerged. The best known of these is the Cochrane Collaboration, an international effort (see next section), including a U.S. center, which sponsors systematic reviews using carefully developed common standards on a wide variety of issues in health care. Groups such as the Grading of Recommendations Assessment, Development and Evaluation Working Group (GRADE), and Appraisal of Guidelines Research and Evaluation (AGREE) have formed to develop standards for evidence syntheses and clinical practice guidelines. Other consortia involved in coordinating their work on clinical effectiveness research
include international technology assessment groups, including EUnetHTA and INAHTA (see Appendix Two below), as well as those mentioned earlier, including the HMO Research Network, the Cancer Research Network, and the Vaccine Safety Datalink at CDC.
1. Academy Health. 2005. Placement, Coordination, and Funding of Health Services Research within the Federal Government. Washington, DC.
2. PhRMA (Pharmaceutical Research and Manufacturers of America). 2006. Pharmaceutical Industry Profile. p. 49, Table 4.
3. The White House. Transforming Health Care: The President’s Health Information Technology Plan, Promoting Innovation and Competitiveness: A New Generation of Innovation. 2004 Online. Available at http://www.whitehouse.gov/infocus/technology/economic_policy200404/chap3.html. Accessed February 14, 2007.
4. HHS (U.S. Department of Health and Human Services). 2007. Office of the National Coordinator for Health Information Technology (ONC): Goals of Strategic Framework. Online. Available at www.hhs.gov/healthit/goals.html. Accessed February 13, 2007.
5. Bartschat, W., et al. 2006. Surveying the RHIO landscape. A description of current RHIO models, with a focus on patient identification. Journal of AHIMA 77(1):64A-64D.
6. NIH (National Institutes of Health). 2007. The NIH Almanac—Appropriations—Section 2. Online. Available at www.nih.gov/about/almanac/appropriations/part2.htm. Accessed February 13, 2007.
7. NIH (National Institutes of Health). 2007. Fact Sheet: Research Into What Works Best.
8. Flum, D. R., et al. 2005. Early mortality among Medicare beneficiaries undergoing bariatric surgical procedures. JAMA 294(15):1903-1908.
9. Udvarhelyi, I. S., et al., Acute myocardial infarction in the Medicare population. Process of care and clinical outcomes. JAMA 268(18):2530-2536.
10. Heaton, P. C., et al. 2006. Analysis of the effectiveness and cost benefit of leukotriene modifiers in adults with asthma in the Ohio Medicaid population. Journal of Managed Care Pharmacy 12(1):33-42.
11. Kupersmith, J., et al. 2007. Advancing evidence-based care for diabetes: Lessons from the Veterans Health Administration. Health Affairs.
12. IOM (Institute of Medicine). 2006. The Future of Drug Safety: Promoting and Protecting the Health of the Public. Washington, DC: The National Academies Press.
13. Stroupe, K., et al. 2006. Investigators of Veterans Affairs Cooperative Studies Program #385 (AWESOME: Angina With Extremely Serious Operative Mortality Evaluation). Circulation 114(12):1251-1257.
14. Kaufman, J. S., et al. 1998. Subcutaneous compared with intravenous epoetin in patients receiving hemodialysis. Department of Veterans Affairs Cooperative Study Group on Erythropoietin in Hemodialysis Patients. New England Journal of Medicine 339(9):578-583.
15. Lieberman, D. A., et al. 200. Use of colonoscopy to screen asymptomatic adults for colorectal cancer. Veterans Affairs Cooperative Study Group 380. New England Journal of Medicine 343(3):162-168.
16. Moseley, J. B., et al. 2002. A controlled trial of arthroscopic surgery for osteoarthritis of the knee. New England Journal of Medicine 347(2):81-88.
17. Rosenheck, R., et al. 2003. Effectiveness and cost of olanzapine and haloperidol in the treatment of schizophrenia: A randomized controlled trial. JAMA 290(20):2693-2702.
18. CDC (Centers for Disease Control and Prevention). 2007. Vaccine Safety Datalink Project (VSD). Online. Available at www.cdc.gov/od/science/iso/research_activities/vsdp.htm. Accessed January 31, 2007.
19. AHRQ (Agency for Healthcare Research and Quality). 2007. Medical Expenditure Panel Survey (MEPS): Overview of the MEPS Web Site. Online. Available at www.ahrq.ov/data/mepsweb.htm. Accessed January 31, 2007.
20. Heres, S., et al. 2006. Why olanzapine beats risperidone, risperidone beats quetiapine, and quetiapine beats olanzapine: An exploratory analysis of head-to-head comparison studies of second-generation antipsychotics. American Journal of Psychiatry 163(2):185-94.
21. Wallace, P. J. 2007. Reshaping cancer learning through the use of health information technology. Health Affairs 26(2):w169-w177.
22. IOM (Institute of Medicine). 2007. Learning Health Care System: Workshop Summary. Washington, DC: The National Academies Press.
23. Hayes, W. 2007. Personal communication.
Clinical effectiveness assessment activities have utility that crosses national borders. A growing involvement in technology assessment and clinical effectiveness evaluation has developed internationally. The explicit incorporation of cost-effectiveness evaluation tends to be a more common feature of the work abroad, prompted by the need for better information with which to make decisions about the use of public monies. Implicit in the systematic evaluation of cost for an intervention is the comparison of the intervention with alternatives for care as a means to arrive at an assignment of value for investment. Thus, the majority of comparative effectiveness analysis that is performed in an international context is based, to some degree, on the cost of the intervention. Noted below are examples of some of the work under way in individual countries and through multicountry collaborations.
Country Specific Programs
The Australian government’s Department of Health and Ageing maintains several parallel pathways for the evaluation of various interventions: the Therapeutic Goods Administration (TGA), the Medical Services Advisory Committee (MSAC), the Pharmaceutical Benefits Advisory Com-
mittee (PBAC), and the Prostheses and Devices Committee (PDC). The Therapeutic Goods Agency is responsible for the regulation of multiple classes of therapeutic products, including pharmaceuticals (prescription and over-the-counter), devices, cellular therapies, and blood and tissue products, and it serves as the primary gateway for introduction of these interventions. Once a product is approved for use, there is a separate process for determination of coverage of regulated products. The PBAC reviews applications from pharmaceutical companies for pharmaceuticals to be subsidized under the Pharmaceutical Benefits Scheme (PBS). Each application undergoes an economic cost and utilization review. A similar process is undertaken for the evaluation of medical technologies and procedures, in which the MSAC provides advice to the Australian government as to whether new technologies and procedures should be considered for reimbursement under the Medicare Benefits Schedule (MBS). The primary aim of the MSAC is to ensure that new and existing medical procedures are supported by evidence of their safety, clinical effectiveness, and cost effectiveness. The option of interim funding can be linked to a requirement for additional data acquisition.
The Canadian Agency for Drugs and Technologies in Health (CADTH) is charged with providing Canada’s federal, provincial, and territorial healthcare decision makers with counsel on the effectiveness and efficiency of healthcare interventions. CADTH operates as an independent, not-for-profit organization with three main programs: Health Technology Assessment (HTA), Common Drug Review (CDR), and the Canadian Optimal Medication Prescribing and Utilization Service (COMPUS). The HTA program conducts assessments of new and existing technologies for the technology’s impact on health, comparisons with alternatives, value for investment, and other health services implications. HTA work is performed both through internal capacity and through a program of extramural research contracts and grants. HTA also operates a horizon scanning program. Since 2003 the CDR has provided a centralized process for evaluation of pharmaceuticals in order to reduce the duplication inherent in each locality independently preparing an evaluation. The recommendations of the CDR advisory committee are advisory and non-binding on local decision making. A technical assistance institution, COMPUS, was initiated in 2004 to help identify and promote evidence-based best practices in drug prescribing and utilization, including maintenance of a virtual library of stakeholder-submitted best-practices guidelines.
In 2004 the French government established the Haute Authorie de Sante (HAS) to bring together a number of activities related to patient care improvement: evaluation of drugs, procedures, and devices; publication of clinical guidelines; accreditation of healthcare providers and doctors; and dissemination of medical information. It carries out these activities through seven special committees: the Transparency Committee, which evaluates medicines; the Committee for Assessment of Devices and Health Technologies (CEPP); the Committee for Assessment of Diagnostic and Therapeutic Procedures (CEAP); Committee for Healthcare Cover for Chronic Conditions; the Committee for Practice Guidelines and Practice Improvement; the Committee for Medical Information Quality and Dissemination; and the Committee for Accreditation of Healthcare Organizations. The Transparency Committee reviews approved pharmaceuticals for reimbursement and use under national health insurance, based on both the baseline and marginal clinical benefit. It also organizes post-market surveillance for drugs. While the functions of the Transparency Committee and CEPP have existed for some time, the introduction of CEAP added the capacity to study diagnostic and therapeutic procedures with respect to their benefit, comparison with other strategies, temporal placement of the procedure in the timeline of care, necessary skills for operators, and economic assessment. HAS is supported by taxes on promotional spending by pharmaceutical companies, health insurance funds, state funding, accreditation fees, and application fees.
The Institute for Quality and Efficiency (IQWiG) has been operating in Germany since October 2004 as an independent, non-profit organization responsible for the evaluation of diagnostic and therapeutic services, pharmaceuticals, clinical practice guidelines, and healthcare services. It also makes recommendations to disease management programs, and disseminates information to patients and consumers. IQWiG was established by the Federal Joint Committee (G-BA), a self-governing organization made up of physicians and health insurance funds that is responsible for the administration of health services in Germany and which is funded by contributions from statutory health insurance funds. Under the new framework, IQWiG is responsible for conducting assessments on topics requested by G-BA or the Federal Ministry of Health, and the conclusions of these reports are used for decision making regarding coverage. Most interventions require a randomized controlled trial that demonstrates efficacy in order to receive a positive recommendation from IQWiG. For the purpose of reimbursement,
medical interventions are grouped into clusters, and reference pricing is used for all treatments which are determined to fall within a specific cluster.
The National Institute for Health and Clinical Excellence (NICE) was established in 1999 to provide patients, health professionals, and the public with authoritative guidance on public health, health technologies, and clinical practices. Three centers guide the programs at NICE: the Centre for Public Health Excellence, which develops public health guidance; the Centre for Clinical Practice, which develops clinical practice guidelines; and the Centre for Health Technology Evaluation, which assesses new and existing medicines, treatments, and procedures. The assessment process involves a technical committee (the Technology Appraisal Committee) as well as economic modeling of cost effectiveness. Technology approvals by NICE mandate coverage for those who meet the approved indications for a treatment. The work of NICE is facilitated by the availability of multiple large databases such as the General Practice Research Database, which contains longitudinal anonymized primary care data for 35 million patient-years and 3 million active patients.
The Ministry of Health and Social Affairs and the National Board of Health and Welfare are two centralized agencies that oversee operation of the Swedish healthcare system. However, the system is decentralized into localities which are primarily responsible for decision making. The Swedish Council on Technology Assessment in Health Care (SBU), formed in 1987, is an agency that undertakes health technology assessments to aid in the decision-making process for the localities; it works in cooperation with the National Board of Heath and Welfare as well as other governmental agencies such as the Medical Products Agency and the Pharmaceutical Benefits Board. Economic evaluations are included in the assessments undertaken by SBU.
European Union 
The European Medicines Agency (EMEA) was established in 1995 as a centralized means of establishing marketing authority for pharmaceuticals in the European Union. A single application process conducted through the EMEA allows for marketing approval for pharmaceuticals and biophar-
maceuticals based upon an evaluation of the quality, safety, and efficacy of the medicinal products submitted. The EMEA coordinates the evaluation of medicinal products through its network of roughly 3,500 experts. Guidance from the EMEA is utilized by member states in varying ways, and member states are free to adopt additional measures, such as those related to cost, with regard to approved agents. The EMEA has adopted a standard emphasizing the use of comparator evaluation when an established pharmacological alternative is available. Processes used by member states to arrive at relative clinical effectiveness and cost-effectiveness have not been fully clarified.
A collaborative project that began in January 2006, EUnetHTA seeks to develop an organizational framework for a sustainable European Network for Health Technology Assessment. EUnetHTA involves 59 partners from 31 countries (including the United States, Canada, Australia, and Israel), and is coordinated and led by the Danish Centre for Evaluation and Health Technology Assessment. It will operate until at least 2008 in the work to create a permanent capacity. It builds on a number of predecessor efforts: EUR-ASSESS (1994 to 1997), HTA-Europe (1997 to 1999), and the European Collaboration for Assessment of Health Interventions-Health Technology Assessment (1999 to 2001). Functions under exploration include facilitation of networking, identification of opportunities for joint assessments and priority setting, development of methodology, gathering and dissemination of information, and educational activities.
The International Network of Agencies for Health Technology Assessment (INAHTA) was established in 1993 and has 45 member organizations from 23 countries. Member organizations must be nonprofit, have a relationship with a local or national government, and be at least 50 percent publicly funded. Through a Web-based service, INAHTA seeks to facilitate the extent to which various HTA organizations can benefit from the work of other centers, through networking, communication, and process improvement.
Cochrane Collaboration 
Perhaps the best known multinational organization working to evaluate evidence of healthcare interventions is the Cochrane Collaboration. Founded in 1993, its major product is the Cochrane Database of System-
atic Reviews. Review groups clustered around medical specialties follow a standardized procedure to produce the systematic reviews, including the preparation, formulation of the problem, locating and selecting studies, assessment of study quality, analysis, and interpretation. The reviews are published electronically and are periodically updated as new information becomes available. A methods advisory group provides advice and assistance to the collaboration in the methodology of systematic review. Organized around disease topics, the Cochrane Collaboration now has more than 4,500 systematic reviews in its database. Among the 12 Cochrane centers worldwide, one is based in the United States, at Johns Hopkins Bloomberg School of Public Health. The central organizing functions of the collaboration are funded through subscription revenue, while individual entities are funded through a wide variety of other sources. including governmental, institutional, and private funding.
3. O’Malley, S. P. 2006. The Australian experiment: The use of evidence based medicine for the reimbursement of surgical and diagnostic procedures (1998-2004). Australian and New Zealand Health Policy 3(3).
8. Pearson, S. D., and M. D. Rawlins. 2005. Quality, innovation, and value for money: NICE and the British National Health Service. JAMA 294(20):2618-2622.
Federally Funded Research and Development Center (FFRDC)
Organizations spanning both the public and private domains exist in many forms. One such quasi-governmental entity is the Federally Funded Research and Development Center (FFRDC). FFRDCs are congressionally chartered to meet the specific needs of a government agency, and strict federal oversight maintains the course of FFRDC research. An FFRDC is funded by the federal government and functions as a non-profit private organization. FFRDCs were first established during World War II to assist government agencies in meeting specific, well-defined technical needs that could not be met by existing government agencies or by the private sector. The mission and core competencies of an FFRDC are developed to support the needs of the sponsoring governmental agency. Currently there are 37 FFRDCs, and, while their missions are quite varied, there are general rules applied to their administration. The sponsoring agency is responsible for overall policy and oversight following guidelines set forth by Office of Federal Procurement Policy (OFPP) Policy Letter 84-1. FFRDCs are managed by non-governmental organizations, usually universities or other nonprofit institutions. These operating parent organizations may also operate multiple FFRDCs or non-FFRDC organizations. According to the OFPP, federal agencies should only establish an FFRDC when the agency determines that “[e]xisting alternative sources for satisfying agency requirements cannot effectively meet the sponsor’s special research and development needs.”
FFRDC funding comes from the organization within the sponsoring agency requesting the work or from a line item on congressionally appropriated budgets of the sponsoring agency. An FFRDC can receive up to 30 percent of its funding from private sources. Competition for other government or commercial business is limited with the intention of fostering a strategic relationship between an FFRDC and its sponsor as well as reducing the potential for conflict of interest that this special access may create. In FY 1993 the federal government spent $67.3 billion on R&D, of which $5.9 billion, or 8.7 percent, was obligated for FFRDCs.
Existing FFRDCs fall into four general categories: policy-focused studies and analyses centers, research and development (R&D) centers, research laboratories, and systems engineering and integration centers. Research laboratories differ from R&D centers in that they are usually limited to basic and applied research that seeks solutions to specific problems but
do not engage in major developmental activities. At this time the government agencies sponsoring FFRDCs include the Department of Defense, the Department of Energy, the National Science Foundation, the Federal Aviation Administration, the National Aeronautics and Space Administration, the National Institutes of Health, the Nuclear Regulatory Commission, and the Internal Revenue Service.
Six FFRDCs are briefly described below, with details offered about their governance structure, funding mechanism, and output.
National Defense Research Institute (NDRI)3
Sponsored by Department of Defense (DoD)
Administered by RAND Corporation
Purpose Established as an FFRDC in 1984, NDRI is the primary source of analytic expertise regarding national defense policy and strategy problems for the Office of the Secretary of Defense (OSD), the Joint Staff, the Unified Combatant Commands, the defense agencies, the United States Marine Corps, and the United States Navy. NDRI identifies and evaluates new policy options and develops alternative approaches to implement current policy. Corresponding with the responsibilities of four OSD under secretaries, NDRI research is conducted within RAND’s National Security Research Division in four interconnected centers: International Security and Defense Policy Center, Acquisition and Technology Policy Center, Forces and Resources Policy Center, and Intelligence Policy Center.
Governance An advisory board made up of 13 senior DoD civilians oversees the NDRI research program as well as the DoD budget request.
Funding NDRI is funded primarily by its sponsor, DoD, with a renewable 5-year contract. Additional funding comes from the departments of its Advisory Board members . Non-DoD agencies that fund NDRI include agencies from the intelligence community, the U.S. Coast Guard, the Department of State, allied governments and their ministries of defense, and foundations and private contributors. RAND supports some NSRD research with its discretionary funds. The fiscal year 2001 research and development (R&D) expenditure was $25.5 million, all of which was federal .
Output NDRI research findings are documented as reports, both classified and unclassified. Examples of recently published reports include Review of Military Health Benefit Design, China’s Military Modernization and the
Cross-Strait Balance, Monitoring the Progress of Shipbuilding Programs, and How Can the Defense Procurement Agency More Accurately Monitor Progress?
Homeland Security Institute (HSI)4
Sponsored by Department of Homeland Security (DHS)
Administered by Analytic Services, Inc.
Purpose Established in 2004, HSI evaluates homeland security systems and technologies during development, deployment, and use for DHS and its operating elements. HSI has four core competencies: systems evaluations; technology assessments; operational assessments; and resource and support analyses. Within Title III, Science and Technology in Support of Homeland Security, of the Homeland Security Act (P.L. 107-296) there are two provisions that call for the establishment of FFRDCs. Section 312 directs the secretary of DHS to establish an FFRDC known as the Homeland Security Institute. According to the legislation, the institute should, among other things, conduct systems analysis, risk analysis, and simulation and modeling in order to assess the vulnerability of the nation’s critical infrastructure.
Governance Acting as the primary sponsor, the DHS under secretary for science and technology designates the executive agent who provides DHS oversight of HSI. The executive agent oversees HSI activities, chairs and designates the membership of the HSI Advisory Group, and designates the program manager. The advisory group oversees the research program. The program manager works to implement the decisions of the advisory group.
Funding HSI receives $10 million in annual core funding . HSI core tasks are funded by the Science and Technology (S&T) Directorate. Analytic tasks are funded by various sponsors throughout DHS, including the S&T directorate.
Output HSI supplies research, studies, analyses, analytic and computational models, simulations, and other technical and analytical support. HSI also distributes weekly newsletters and publishes the Journal of Homeland Security.
Science and Technology Policy Institute (STPI)5
Sponsored by National Science Foundation (NSF)
Administered by Institute for Defense Analyses (IDA)
Purpose Established in 1991, STPI (named the Critical Technologies Institute until 1998) provides technical analytical support to inform the Executive Branch as it formulates federal science and technology policy. STPI conducts research for the Office of Science and Technology Policy (OSTP) and other government users. In addition to providing information regarding science and technology developments and trends, STPI identifies roles for the federal government and other sectors to ensure long-term U.S. competitiveness. Science and technology issues addressed by STPI include national security, homeland security, critical infrastructure protection, health and the environment, space and transportation, information/telecommunication infrastructure and technology, education and training, physical sciences, engineering and technology, research and development portfolio, and development of new analytic methods and tools.
Governance The director of STPI reports to the president of IDA. IDA leadership includes a board of trustees and corporate officers. OSTP approves all projects requested by other agencies. Other agencies requesting projects transfer funds to NSF, which then contracts with STPI.
Funding STPI is funded on a project basis. The fiscal year 2001 R&D expenditure was $7.3 million, all of which was federal . The NSF budget request for fiscal year 2008 for projects requested by OSTP provides $3.04 million for STPI.
Output STPI produces informal briefings as well as formal reports for the sponsoring agency. Decisions concerning public availability are strictly up to the sponsoring agency.
Lawrence Livermore National Laboratory (LLNL)6
Sponsored by Department of Energy (DOE)
Administered by University of California
(contract expires September 2007)
Purpose Established in 1952 as a research and development FFRDC, LLNL is one of three national laboratories that are part of National Nuclear
Security Administration (NNSA); the other two are Los Alamos National Laboratory and Sandia National Laboratories. LLNL is an applied science laboratory that provides scientific and technical expertise to ensure the safety and reliability of the nation’s nuclear weapons. LLNL also supports national security needs by developing capabilities for nonproliferation. LLNL principle activities include stockpile surveillance, stockpile refurbishment, and integrated program management. LLNL’s eight core competencies are physics, computing, biology, engineering, national security, lasers and optics, chemistry and materials science, and energy and environment.
Governance DOE determines the mission of LLNL. LLNL determines how best to carry out the research. University of California (UC) management provides the appropriate long-term research environment. The UC board of regents appoints the laboratory director and senior executive team (the director, the deputy director for operations, the deputy director for science and technology, the laboratory executive officer, the associate director at large, and the director’s chief of staff). Historically, the amount of DOE oversight of LLNL has varied, with increases in DOE oversight in the 1980s due to environmental issues at nuclear weapons sites. The subsequent development of performance-based management by UC and DOE resulted in an increase in UC oversight.
Funding LLNL support comes largely from the NNSA Office of Defense Programs for nuclear weapons stockpile stewardship activities. National security and homeland security work is also funded by the NNSA Office of Defense Nuclear Nonproliferation, the Department of Homeland Security, various Department of Defense sponsors, and other federal agencies. The National Aeronautics and Space Administration, Nuclear Regulatory Commission, National Institutes of Health, Environmental Protection Agency, California state agencies, and private industry provide additional LLNL funding. The fiscal year 2001 R&D expenditure was $1.1 billion, of which 96 percent was federal . The FY 2005 LLNL annual budget was $1.6 billion.
Output Recent examples of LLNL-developed advanced technologies include a device to detect highly enriched uranium inside cargo containers, a radiation detector, and a bioagent monitor. LLNL also provides an annual assessment report for nuclear weapons systems.
National Cancer Institute at Frederick (NCI-Frederick)7
Sponsored by Department of Health and Human Services (HHS)
Administered by Science Applications International Corp. (SAIC);
Charles River Laboratories, Inc.; Data Management
Services, Inc.; and Wilson Information Services, Inc.
Purpose A part of the National Institutes of Health (NIH), NCI-Frederick is one of two NCI campuses. The NCI’s clinical researchers and the NIH Clinical Center are located on the NIH campus in Bethesda, Maryland. The NCI’s Frederick campus is located within Fort Detrick, a U.S. Army base in Frederick, Maryland. NCI-Frederick is a GOCO (government-owned, contractor-operated) facility operated by four separate contractors providing services and support for (1) operations and technical support, (2) animal production, (3) computer and statistical services, and (4) scientific library services. NCI-Frederick employs more than 100 scientists to provide scientific and technical support to various NCI and NIH programs involved in the advancement of knowledge and tools for diagnosis, treatment, and prevention of human cancer and AIDS. Researchers work toward understanding the underlying genetic, molecular, environmental, and behavioral factors that contribute to human cancers, as well as developing novel tools for cancer diagnosis, treatment, and prevention. Additionally, NCI-Frederick provides scientific expertise and technology development to the NIH with its Research Technology Program (RTP) and other programs.
Governance The associate director at NCI-Frederick provides leadership and strategic planning, working with the NCI director to ensure that resources are being used to fulfill the FFRDC’s mission. The Office of the Director at NCI through its Office of Scientific Operations develops and coordinates contractor requirements, providing overall scientific administrative management and program planning. The Office of the Director for the Center for Cancer Research (CCR) is also involved in strategic planning and determines CCR’s scientific needs for NCI-Frederick resource allocation.
Funding The fiscal year 2001 R&D expenditure was $187 million, of which $183 million was federal . The fiscal year 2008 proposed budget for the NCI provides up to $8 million for facilities repairs and improvements at NCI-Frederick .
Output Scientific research is published and disseminated to the research community in peer-reviewed publications. The following research support services are available at NCI-Frederick: animal resources; bioinformatics and biostatistics; chemistry and structural biology; mass spectrometry center; clinical research support; Economy Act/Work For Others; flow cytometry; genetics, genomics, DNA sequencing; imaging/microscopy; mass spectrometry; microarray and quantification of gene expression; protein chemistry and biophysics; proteomics; repository services; and scientific graphics and media.
Aerospace Federally Funded Research and Development Center
Sponsored by Department of Defense
Administered by the Aerospace Corporation
Purpose Established in 1960, Aerospace is the systems engineering and integration center type of FFRDC. Aerospace provides scientific and engineering support for both the long-term programs and the immediate needs of U.S. military and reconnaissance space programs for National Security Space. Aerospace mainly works for the Space and Missile Systems Center of Air Force Space Command and the National Reconnaissance Office. Aerospace is involved in the concept, design, acquisition, development, deployment, and operation of missions. Aerospace’s core competencies, as defined by DoD, are launch certification, system-of-systems engineering, systems development and acquisition, process implementation, and technology application.
Governance The board of trustees of the Aerospace Corporation elects corporate officers and sets policy for its FFRDC. The board also supervises and directs the general management of the corporation. The 20 board members are elected from business, scientific, academic and public-service sectors to 3-year terms. The chief executive officer and president are ex officio members and are elected annually by the board.
Funding The total funding for Aerospace was $381 million for FY 1994, which includes $365.5 million from DoD obligations.
Output Aerospace has been involved in almost all national-security launch and satellite programs. For example, in fiscal year 2005 Aerospace pro-
vided oversight to 43 operational launches for the Space Missile Systems Center and the Air Force. Aerospace also provides technical analyses and assessments.
1. GAO (United States General Accounting Office). 1996. Federally Funded R&D Centers: Information on the Size and Scope of DoD-Sponsored Centers.
2. NSF (National Science Foundation). 2002. Federally Funded Research and Development Centers’ R&D Expenditures: Fiscal Year 2001. Table 9. Online. Available at www.nsf.gov/statistics/nsf03313/ pdf/tab9.pdf. Accessed 2007.
3. House of Representatives Committee on Science. 2007. An Overview of the Federal R&D Budget for Fiscal Year 2007. Online. Available at http://commdocs.house.gov/committees/science/hsy25938.000/hsy25938_0.HTM. Accessed 2007.
4. OMB (Office of Management and Budget). 2007. FY 2008 Budget Appendix. Online. Available at http://www.whitehouse.gov/omb/budget/fy2008/pdf/appendix/hhs.pdf. Accessed 2007.
The National Institutes of Health (NIH) has long realized the importance of collaborations to tackle those issues that require outside resources, and public-private partnerships (PPPs) were set up ad hoc to fulfill such needs. One outcome of the 2002 NIH Roadmap, which recognized that partnerships between PPPs could help push scientific discoveries from the bench to the bedside, was the creation of the Program on PPPs as a central resource to provide guidance and advice to NIH and prospective partners on the development of a PPP.
Partnerships are established either directly with the NIH, often facilitated by the Program for PPP, or through the Foundation for the NIH (FNIH), an independent public foundation. The FNIH was chartered by Congress in 1990 to support the mission of the NIH. As a public foundation, FNIH can solicit donations from non-governmental sources. FNIH has many roles in PPPs beyond fund raising and donor interactions, including the management of scientific projects and the solicitation, review, award, and management of grants or contracts. In May 2005 FNIH was involved in approximately 50 PPPs totaling $280 million.
NIH PPPs are formed to address specific scientific issues, with the larger
goal of improving public health. Thus PPPs range widely in size and scope to accommodate the goals and mandates of the partners. Partners from the public/non-profit sector may include academic institutions, foundations, patient advocacy organizations, U.S. government agencies, and foreign governments. Private-sector partners may include foundations, patient advocacy groups, and members of industry (pharmaceutical, biotechnology, devices, diagnostics, informatics, and other). Governance of PPPs is individualized and includes all sectors of the partnership. All NIH PPPs comply with federal laws and NIH policies and maintain transparency, fairness, inclusiveness, and scientific rigor. Partner contributions to the scientific and fiscal aspects of the projects also vary greatly, depending on project needs.
Brief descriptions of three PPPs are provided below, with information on their governance structures, funding mechanisms, and output. The three PPPs are the Biomarkers Consortium, the Osteoarthritis Initiative, and Grand Challenges in Global Health.
The Biomarkers Consortium9
The Biomarkers Consortium (BC) is a public-private partnership launched in October 2006 which aims to identify and validate new biological markers, or biomarkers. The goal is to use these novel biomarkers to promote the “delivery of successful new technologies, medicines, and therapies for prevention, early detection, diagnosis, and treatment of disease.” In practice, the BC will fund clinical research centers to generate data pertinent to biomarkers that will be publicly available to encourage the use of biomarkers to improve public health.
Partners BC is a partnership between public entities (NIH, FDA, and Centers for Medicare and Medicaid Services), industry (pharmaceuticals, biotechnology, diagnostics, and medical devices), non-profit organizations and associations, and advocacy groups. The partnership is managed by the FNIH.
Contributions NIH contributes to the partnership its PPP Program staff effort and Office of General Council (OGC) and Office of Technology Transfer (OTT) involvement where necessary. Individual NIH institutes and centers (ICs) provide intellectual and scientific contributions to steering committees and project teams in the design or conduct of experiments. FDA contributes intellectual and scientific input to the steering committees and project teams. Additionally, FDA’s involvement helps attract partici-
9 Information summarized from the FNIH BC Web site http://www.fnih.org/Biomarkers%20Consortium/Biomarkers_home.shtml.
pation from companies that hope to gain exposure or insight into FDA. Pharmaceutical Research and Manufactures of America (PhRMA) and Biotechnology Industry Organization (BIO) participated in establishing the BC structure. They have contributed funding for infrastructure development and support, and they are coordinating the effort and participation from all industry partners. Participating industry organizations, including biotechnology, diagnostics, and imaging industries, contribute to the steering committees and project teams. They also contribute data, assays, equipment, drugs, and other reagents. Advocacy groups representing the public are critical to fundraising, patient recruitment, public education, and public relations. FNIH provides oversight and coordination of BC activities among the partners. FNIH issues solicitations and reviews the administration and awarding of BC/FNIH grants and contracts.
Governance BC is governed by a 15-member executive committee (EC) representing founding partners and additional stakeholders: three members from NIH ICs, three from FDA, three from the pharmaceutical industry, one from CMS, one from the biotech/diagnostics/imaging industry, one from advocacy groups representing the public, and three non-voting FNIH members. The EC is involved in decision making and advising the FNIH board on financial commitments. Decision making is in the hands of the founding partners (EC), with any partner having veto ability, and all projects needing assent from all three founding partners. EC has approval checkpoints related to project concept, project plan, intellectual property, data sharing, data access, conflict of interest, and antitrust. Steering committees, which are content area focused, identify and oversee specific project activities. Project teams determine priorities and implement projects. BC policies are negotiated with principals and legal counsels concerning issues pertaining to intellectual property, antitrust issues, grantee/contractor selection, confidentiality, and conflict of interest.
Funding FNIH is responsible for fund raising and financial management. At its launching in October 2006, $1.2 million was committed by BC funding members: the Alzheimer’s Association, AstraZeneca, the Biotechnology Industry Organization, Bristol-Myers Squibb, GlaxoSmithKline, the Leukemia & Lymphoma Society, Johnson & Johnson, Eli Lilly & Company, Pfizer Inc., the Pharmaceutical Research and Manufacturers of America, and F. Hoffmann-La Roche. NIH ICs also contribute funding if it is appropriate agency-wide. Specific projects may have funding from additional sources.
Output BC seeks to identify and validate biomarkers useful for disease diagnosis, including early detection, and for predicting response in order to better guide treatment decisions.
The Osteoarthritis Initiative10
The Osteoarthritis Initiative (OAI) is a partnership established in 2001 to bring together public and private scientific expertise and funding to generate a research resource of osteoarthritis patient data, radiologic information, and biospecimens. The OAI goal is to “create a public resource to validate prospective biomarkers, obtain early-stage input from the FDA as to acceptability of biomarkers as clinical endpoints, and ensure that validated biomarkers are as widely available as possible to further product development and the public health.” OAI awarded NIH peer-reviewed contracts to four clinical centers and one coordinating center to perform a comprehensive longitudinal study of approximately 5,000 patients over 5 to 7 years.
Partners OAI is a partnership with multiple NIH ICs (National Institute of Arthritis and Musculoskeletal and Skin Diseases, National Institute on Aging, National Center for Complimentary and Alternative Medicine, National Center on Minority Health and Health Disparities, National Institute of Biomedical Imaging and Bioengineering, National Institute of Dental and Craniofacial Research, Office of Research on Women’s Health, and Office of the Director). FNIH is coordinating the private sector participation of GlaxoSmithKline, Merck, Novartis, and Pfizer. The four clinical centers are at the University of Maryland/Johns Hopkins University, Memorial Hospital Rhode Island/Brown University, Ohio State University, and University of Pittsburgh. The Data Coordinating Center is University of California San Francisco.
Contributions Academic centers that receive NIH funding carry out the research and provide access to public databases and biospecimens. Private sector industry partners provide study planning, scientific input, financial support, data, and biospecimens. The public partners provide financial support, contract funding, and a biospecimen repository as well as housing the public database. FNIH is coordinating the private sector participation.
Governance An OAI Steering Committee provides scientific oversight. There is also an Observational Study Monitoring Board (OSMB).
Funding NIH and private sector participants have contributed to the funding of this $60 million PPP, with approximately $22 million support from private industry . According to the FNIH, private-sector funding is $3.2
10 Information summarized from the FNIH Web site http://www.fnih.org/programs/research_environment/osteoarthritis.shtml and the OAI Web site http://www.oai.ucsf.edu/datarelease/StudyOverview.asp.
million annually, with equal contributions from GlaxoSmithKline, Merck, Novartis, and Pfizer, and $5 million per year from the NIH.
Output OAI set out to create a public-domain resource for researchers to be used to “identify and compare the effectiveness of new treatments and to define the state of the disease.” The first release of such data was in August 2006.
Grand Challenges in Global Health11
Launched in 2003 by the Bill & Melinda Gates Foundation, the Grand Challenges in Global Health (GCGH) initiative provides grants for a broad range of research aimed at solving the 14 grand challenges identified by scientists and health experts worldwide. As of 2005, there were 43 research projects under way in 33 countries, each working toward scientific or technical breakthroughs in one of seven project areas to improve vaccines, create new vaccines, control insect vectors, improve nutrition, limit drug resistance, cure infection, and measure health status.
Partners GCGH partners include the Bill & Melinda Gates Foundation, the Canadian Institutes of Health Research (CIHR), the Wellcome Trust, NIH, and FNIH.
Contributions The Bill & Melinda Gates Foundation, the CIHR, and the Wellcome Trust provide funding and management. FNIH also participates in managing the GCGH initiative. NIH provides scientific advice and expertise to the FNIH. Additionally, NIH may participate in identifying and funding related projects or follow-up activities resulting from GCGH studies.
Governance The GCGH scientific board includes 22 scientists and public health experts from 13 countries. Twenty of the 43 grants were managed by the FNIH as of 2005.
Funding The Bill & Melinda Gates Foundation has committed $450 million to the GCGH initiative. Additional funding was contributed from the Wellcome Trust and CIHR ($27.1 million and $4.5 million, respectively).
Output The GCGH research projects are aimed at developing and improv-
11 Information summarized from he GCGH Web site http://www.gcgh.org/AboutUs/, the FNIH Web site http://www.fnih.org/programs/programs.shtml, the NIH PPP Web site http://ppp.od.nih.gov/pppinfo/examples.asp, and links therein.
ing methods and techniques to bring effective, affordable health care to developing countries.
1. Foundation for the National Institutes of Health. 2005. Annual Report.
The National Academies is a private, non-profit organization that provides independent advice to the federal government and others on scientific and technological issues. It includes the National Academy of Sciences (NAS), the National Academy of Engineering (NAE), the Institute of Medicine (IOM), and the National Research Council (NRC). NAS was congressionally chartered in 1863, and, under this charter, the NRC was created in 1916, the NAE in 1964, and the IOM in 1970. NRC is the working arm of the NAS and NAE, and it is administered jointly by NAS, NAE, and IOM. A division of the NRC, the Transportation Research Board (TRB) is considered to be the nation’s primary transportation research body. Formerly named the National Advisory Board on Highway Research, TRB was established in 1920 as a source of information and research results about highway technology. Starting in the 1950s, TRB moved beyond information exchange to the management of research projects.
The stated mission of the TRB is to promote innovation and progress in transportation through research. TRB is involved in a wide range of activities to fulfill this mission. TRB activities are organized into six divisions: technical activities; studies and special programs; administration and finance; cooperative research programs; SHRPII (Strategic Highway Research Programs II); and activities of the Marine Board. The Cooperative Research Programs Division houses the transportation research activities that parallel in funding and approach several proposals for support and conduct of comparative effectiveness research under the auspices of the IOM/NRC. It includes five contract research programs that aim to identify near-term, practical solutions to problems facing transportation agencies. The five programs are the National Cooperative Highway Research Program (NCHRP), the Transit Cooperative Research Program (TCRP), the Airport Cooperative Research Program (ACRP), the National Cooperative
Freight Research Program (NCFRP), and the Hazardous Materials Cooperative Research Program (HMCRP). The programs are administered by TRB and sponsored by federal or state agencies. Of these the NCHRP is the largest and oldest program, with expenditures of $35 million in 2007. All Cooperative Research Programs expenditures totaled $57 million in fiscal year 2007 .
The National Cooperative Highway Research Program, started in 1962, was the first continuing research management activity of the NRC. TRB research management capacity has continued to grow significantly at the request of Congress, the U.S. Department of Transportation, and the state departments of transportations through the American Association of State Highway and Transportation Officials (AASHTO). The state departments of transportation are the sponsors of the NCHRP through voluntary set-asides—at an allocation that amounts to 5.5 percent of the apportionment from the states’ Federal-Aid Highway State Planning and Research (SPR) funds.
Every year the AASHTO Standing Committee on Research solicits potential research topics from the member state transportation departments, the various AASTHO committees, and the Federal Highway Administration. The NCHRP uses evaluation panels to assess the problem statements, which are fed back for revision, and the problems are given priority rankings based on the feedback. The Standing Committee on Research then selects specific research projects based on the level of funding available. After the projects are selected, an announcement of research projects is issued, technical oversight panels are formed for each project, scopes of work are finalized, RFPs are issued, research proposals are received and evaluated, and awards are made. The other cooperative research programs noted above are run in a similar fashion, in association with their respective public agency constituencies.
A related branch, the Studies and Special Programs Division produces policy studies, syntheses of current transportation practices, information services, and the Long-Term Pavement Performance Studies. This division supports the IDEA program (Innovations Deserving Exploratory Analysis), which provides grants to investigator-initiated proposals.
SHRPII is a new program that will focus on applied research in four areas of highway transportation—safety, renewal, reliability, and capacity—with the goal of advancing highway performance and safety. The program will be funded at approximately $150 million over 7 years. SHRPII will make recommendations about procedures, practices, and applications, which may be adopted as standards or guides to recommended practices at the local, state, or federal levels.
Apart from research-related support, TRB is funded through fees provided by the U.S. Department of Transportation and other federal agen-
cies, the state transportation departments, industry associations, and other interested organizations and individuals. Sixty-seven sponsors provide the bulk of funding for TRB’s core technical activities, each with a minimum annual fee of $60,000. Five sustaining affiliates provide minimum annual fees of $15,000. Additionally, 110 organizational affiliates also contribute to TRB financial support via annual fees.
1. Transportation Research Board. 2006. Annual Report. Washington, DC: The National Academies Press.
The Federal Reserve System was created by Congress through the Federal Reserve Act in 1913 to serve as the central bank for the United States. Established as an entity independent from the federal government, the Federal Reserve promotes economic stability through formulating monetary policy, overseeing and regulating banks, and providing financial services to depository institutions, all with minimal political pressure. The three major components of the Federal Reserve—the board of governors, 12 regional federal reserve banks, and various member banks located throughout the country—work in concert to carry out its charge.
Consistent with its independence from the federal government, the Federal Reserve receives funding not through congressional appropriations but through interest on government securities acquired through open market operations as well as fees charged for banking services. With this income, the reserve banks pay their expenses, and any remaining earnings are turned over to the U.S. Treasury. In 2005 payments to the treasury amounted to $21.5 billion.12
To achieve a balance between the Federal Reserve System’s independence from the federal government and the need to keep the system accountable, the Federal Banking Agency Audit Act gives the Government Accountability Office authority to audit the board of governors, the reserve banks, and the branches. Congress also requires the Federal Reserve to produce reports
12 U.S. Federal Reserve System. 2006 (June). 2005 Annual Report of the Board of Governors of the Federal Reserve System.
and conduct semi-annual hearings. Federal Reserve officials often testify on a wide range of issues affecting the economy and banking industry.
In addition to the reporting requirements, Congress exercises its oversight role during the appointment process for the seven members of the board of governors, who are appointed by the President with the guidance and approval of the Senate. When selecting a governor, the President carefully considers each candidate to ensure fair representation of the various economic interests and geographical divisions of the country. For instance, only one governor from any one of the Federal Reserve regions may be appointed. Furthermore, candidates cannot have held a position in a member bank for 2 years prior or at any point during their term, if appointed. The term of service for a board seat is 14 years, and the terms are staggered with one term ending every 2 years. Two governors hold the positions of chairman and vice-chairman of the board. These terms are only 4 years long, and they do not necessarily coincide with the President’s term or that of each other. Each of the seven governors, including the chairman and vice-chairman, has one vote on the board.
The board of governors formulates monetary policy through open market operations carried out by the Federal Open Market Committee (FOMC). The FOMC is made up of the seven board members as well as 5 of the 12 federal reserve bank presidents. This body increases and decreases the supply of money by altering interest rates, with the long-term goal of promoting non-inflationary economic expansion. Several open market operations exist, such as the purchase and sale of government securities, the shifting of legal reserve requirements, and the resizing of the discount window. For example, when securities are purchased by the Federal Reserve, the money supply is increased, relaxing credit conditions, and when securities are sold, the money supply is decreased and credit conditions are tightened. Similarly, legal reserve requirements, the amount that depository financial institutions are required to set aside in proportion to deposits, can be increased or decreased, causing banks to have less and more money to lend, respectively. Also, rates charged by the discount window, the Federal Reserve facility for lending to eligible depository institutions, can be augmented to encourage or discourage borrowing and, consequently, lending.
In addition to formulating monetary policy, through the Truth in Lending Act, the Electronic Funds Transfer Act, and the Fair Housing Act, the board of governors oversees member banks as well as the 12 federal reserve banks.
The 12 federal reserve banks reside in Boston, New York, Philadelphia, Cleveland, Richmond, Atlanta, Chicago, St. Louis, Minneapolis, Kansas City, Dallas, and San Francisco. Each of these banks is managed by a nine-member board of directors, who serve 3-year terms. Each board of directors is divided into three classes to ensure that the interests of member banks
and the general public are represented. The federal reserve banks act as fiscal agents to the federal government, issuing, transferring, exchanging, and redeeming government securities and savings bonds. They also provide transaction accounts for the treasury and collect and disburse funds for the federal government. Furthermore, the banks carry out such routine duties as moving coin and currency out of circulation, collecting and processing checks, and tracking electronically originated credits and debits.
The stock of these 12 Federal Reserve Banks is owned entirely by the member banks of their respective districts, purchased at 6 percent of the member banks’ own capital and interest, with the understanding that 3 percent must be paid and 3 percent is subject to call by the board of governors.13 All national banks are required to be members of the Federal Reserve System. This membership is optional for state-chartered banks. As of June 30, 2005, there were 1,861 national banks and 903 state-chartered Federal Reserve member banks.14
A Commissioned AnAlysis
Professor, Department of Health Care Policy
Harvard Medical School
Professor, Department of Internal Medicine
University of Michigan Medical School
Acknowledgment: We thank David Helms, Sean Tunis, Peter Juhn, and reviewers from the Institute of Medicine for their helpful comments.
13 U.S. Federal Reserve System. 2006 (June). 2005 Annual Report of the Board of Governors of the Federal Reserve System.
14 U.S. Federal Reserve System. 2006 (June). 2005 Annual Report of the Board of Governors of the Federal Reserve System.
Although the United States spends a far greater amount on healthcare services than other countries, it is well documented that Americans underachieve in terms of health measures such as life expectancy and infant mortality, as well as other vital health statistics. For instance, in a 2002 ranking of 23 industrialized countries, the United States tied for last with 4 other countries on healthy life expectance at age 60. Similarly, the United States’ infant mortality rate ranked last, with rates more than double the average of the three leading countries (France, Japan, and Spain) . While some of these discrepancies may be explained by genetic and social factors, it is well accepted that inefficiencies in clinical care—which medical services are used and to whom they are provided—contribute substantially to the discrepancy, particularly the extra spending. Systemic factors in the organization, financing, and delivery of care are part of the problem. The mere fact that Americans consistently under-use medical interventions rigorously proven to provide important health benefits and, at the same time, use substantially more services of unproven value, warrants a careful reassessment of the evidence base that informs medical decision making as well as the payment system that likely rewards inefficient behavior.
Given the complexity of clinical medicine and the financing system, the demands on the evidence base are large. When assessing the value of a specific clinical intervention, the natural tension between quality improvement and cost containment compels a transparent discussion of not only the clinical advantages a certain service may provide, but also the economic ramifications of its use. Moreover, it must be acknowledged that the value of a specific medical intervention will vary across patient groups (e.g., colonoscopy for a 55-year-old woman with a first-degree relative with colon cancer as compared to a low-risk 35-year-old woman). Thus effectiveness is not an attribute purely of the intervention, but of both the intervention and the patient. While acquiring the scientific evidence to improve medical decision making may seem a daunting task, an investment in a clinical research agenda with the specific aim of determining the right medical intervention for the right person at the right time is critical to ensure a more efficient system.
Comparative effectiveness research (CER) refers to the clinical and/or economic evaluations of specific medical interventions (including pharmaceuticals, medical devices, and medical procedures) relative to other available alternatives for a selected clinical indication. CER is very broad. In some cases, it may compare two similar approaches to a particular clinical issue (drug A to drug B, or procedure A to procedure B). In other cases, it may compare available—and dissimilar—clinical approaches (e.g., watchful waiting, radiotherapy, and surgery for prostate cancer) to determine the optimal management of a specific clinical problem. CER typically will focus on realistic decisions confronting patients and their clinicians in actual prac-
tice, and thus seldom compares interventions to placebo controls. Because of this focus on effectiveness as opposed to efficacy, these investigations will likely rely on both prospective trials and observational data to determine relative value in real-world settings, in contrast to the carefully manipulated environment of a controlled clinical trial.
Many countries other than the United States already support a centrally coordinated CER structure, typically financed with public funds. For example, the United Kingdom’s National Institute for Health and Clinical Excellence (NICE) produces national guidance on the appropriate use of health technologies and the treatment and prevention of ill health . In the United States, federally funded agencies such as NIH and AHRQ support CER as a part of their established practices, and several for-profit and not-for-profit entities currently engage in CER to varying degrees. However, these efforts are modest when compared to the $2 trillion annually spent on health care. Although estimates are difficult to obtain, the Coalition for Health Service Research reports that of the $31.3 billion spent by the federal government in 2005 for health research, only $1.5 billion (4.8 percent) was devoted to health services research, of which only a small fraction is likely directed to CER. AHRQ received only $15 million for CER as part of the Medicare Modernization Act. Moreover, there is little to no coordination among CER efforts.
A greater investment in CER may yield clinical and/or economic benefits. The clinical rewards would stem from CER’s ability to support efforts to direct clinical practice toward interventions that yield superior outcomes and away from services that provide little or no clinical benefit. In some cases, evidence suggests that greater benefit can be achieved at an incremental cost. In these cases, policy makers and private sector managers must explicitly acknowledge the tradeoff between better health and medical spending when debating potential solutions to our system’s inefficiency. Determining what level of spending should be established for services that lead to improvements in health is one of the most challenging tasks for U.S. decision makers. Several other countries, such as Australia, Great Britain, and Canada, use measures of incremental cost per health outcome gained in their decisions regarding the allocation of healthcare resources. For several political and operational reasons, such an approach is unlikely to be adopted in the United States in the near future.
Since the over-utilization of unnecessary or unproven medical services is rampant in the United States, economic benefits may accrue if CER is able to reduce utilization of these services. CER may also produce cost savings by increasing the use of those services where the clinical benefits are sufficient to reduce spending on adverse outcomes that may have otherwise occurred. There is also potential to enhance efficiency through the identification of clinical approaches that achieve the same clinical outcomes with
the utilization of less costly interventions (e.g., laparoscopic versus open cholecystectomy).
This paper examines the potential for CER to enhance the efficiency of the healthcare system. Because CER is merely a tool, the impact of CER ultimately depends on how it is used—whether it is used to inform decision makers as part of efforts to improve the way markets function or else is implemented in specific clinical areas such as guideline development or in a broader application such as health benefit design. No matter how CER is utilized, it is imperative that outputs from CER address both the clinical and financial effects, since in some instances improvements in health will require incremental expenditures.
Since there is little argument that the potential clinical and economic impact of any healthcare reform proposal will depend on the knowledge base that guides practice, such an effort is worthwhile regardless of the approach taken to reform our inefficient system. For example, consumer-driven healthcare initiatives require patients to know about the clinical and economic ramifications of their medical care choices. Centralized delivery systems require administrators to understand the effects of different clinical options to allocate resources optimally. Other initiatives, such as payment reform or pay-for-performance programs, also require a better understanding of the merits of alternative clinical strategies. Even investments in infrastructure, such as information technology, cannot achieve their full potential without the detailed clinical and economic knowledge provided by CER.
1. Inefficiency in the System
The potential for CER to enhance efficiency depends on the underlying amount of inefficiency in the current healthcare system. Inefficiency arises when clinical goals could be more efficiently achieved by use of other services or by no services at all. CER inherently addresses the inappropriate use of medical services. Three types of inappropriate use are typically discussed. First, overuse is the delivery of services that provide no (or minimal) clinical value. In certain cases of overuse (e.g., the use of antibiotic therapy for a viral infection), delivery of these services may result in adverse clinical events or a further increase in spending above the costs of the service itself (e.g., patient-level side effects due to allergic reactions as well as societal costs of future antibiotic resistance). Elimination of this type of inappropriate overuse represents the most direct way for CER to achieve cost savings.
A second form of inefficiency, misuse, is the use of certain effective services in situations where they are not clinically indicated. This pertains to the aforementioned point that the value of specific medical services differs
substantially depending to whom and when they are delivered, and we will consider this a subset of overuse.
Third, underuse represents the failure to deliver appropriate services that attain clinical benefits. Elimination of underuse may not save money in most instances, but still may be perceived as an efficient use of expenditures. However, if the provision of appropriate services can prevent clinical deterioration that would result in greater spending over an extended period (e.g., use of certain medications after a heart attack), savings may be possible.
a. Overuse Overuse in the American healthcare system takes many forms. In some cases, it is the use of an expensive treatment choice within any given treatment strategy. This may be the use of an expensive medication when a cheaper medication is available, or an expensive imaging procedure when a less expensive alternative could be used. A notable example of this behavior was the widespread use of the more expensive cyclooxygenase-2 selective non-steroidal anti-inflammatory drug (COX-2 NSAID) for pain relief in patients without risk for gastrointestinal complications. Despite several studies demonstrating no advantages in terms of pain relief when compared to older, less expensive agents, this novel class of agents became the treatment of choice. Similarly, the use of more expensive angiotensin receptor blockers (ARBs) to treat hypertension might not be indicated in patients for whom angiotensin converting enzyme (ACE) inhibitors are effective.
In other cases, overuse occurs when a more expensive treatment strategy is chosen when a less expensive strategy exists or when a procedure or test is not necessary. For example, in some cases a surgical procedure is more expensive than an alternative pharmacologic intervention (e.g., fundoplication versus proton pump inhibitor therapy for gastroesophageal reflux disease). In other cases, the reverse may be true. In some cases, the use of a diagnostic test could reduce overall costs (e.g., T-wave alternans testing prior to implantable coronary defibrillator insertion), and in other cases diagnostic tests may be unnecessary (e.g., upper gastrointestinal endoscopy in suspected ulcer disease).
In analyzing overuse, it is important to recognize that in some cases waste occurs because the same outcome can be achieved with less expense. In other cases, the more expensive approach provides marginally better outcomes at a much higher cost, in which case the more expensive choice is deemed not cost effective.
Assessing the appropriate strategy is complex. The effects of a treatment strategy may accrue over a long period of time, and proper analysis will incorporate the costs of complications at a later date. Strategies that seem less expensive at first assessment may in fact be more expensive if they entail more complications over a lifetime. Moreover, costs beyond those of
the medical care system, such as effects on worker productivity, should be included.
A second complexity arises because patients are heterogeneous. The efficient choice for one patient may not be efficient for another. For example, the use of COX-2 selective NSAIDs may be appropriate for those patients in need of pain relief who are also at high risk of gastrointestinal complications of traditional NSAIDs (despite the possibility of increased risk of coronary adverse events), whereas this same class of medication is likely to provide no better (and perhaps worse) health outcomes at a higher cost for patients at low gastrointestinal risk but high risk for cardiovascular complications. Often, procedures appropriate in some patient groups diffuse into others, generating overuse due largely to this heterogeneity of risk and benefit.
One major challenge for CER is to assess how to get the right service to the right patient at the right time. A similar critical challenge for the health services research community is to devise systems to limit use to the set of patients for whom the service provides value, thereby eliminating overuse. The fundamental tenet of clinical medicine is primum non nocere, “first do no harm.” In today’s complex environment this principle should extend beyond the clinician-patient relationship to include health systems operations.
There are many estimates of waste in the American healthcare system associated with the excessive use of medical interventions. These are documented by several broad categories of literature.
i. Geographic variation The literature documents variations in use across geographic regions, even after controlling for clinical differences. For example, small area analysis, presented in the Dartmouth Atlas of Health Care, compares the use of resources, practice patterns, and spending levels across 306 hospital referral regions to draw inferences about the quality and cost of care provided . Such studies have revealed discrepancies in care for conditions such as hip fracture, colorectal cancer, and acute myocardial infarction.
Fisher et al. looked at end-of-life care spending to examine costs and outcomes. They found that residents of high-spending regions received 60 percent more care but did not have better quality of care or outcomes . Each 10 percent increase in regional end-of-life spending was associated with a 0.3-0.12 percent increase in risk for death, depending on the original condition . Similarly, cardiac bypass surgery rates exhibited about a fourfold range of variation. These rates were strongly correlated with the numbers of per capita cardiac catheterization labs in the regions but not with illness rates as measured by the incidence of heart attacks in the region .
In another study, mortality rates and quality-of-life measures were compared for patients undergoing coronary angiography in Texas, where the utilization of the procedure is high (45 percent), and for similar patients in New York, where utilization is low (30 percent). After adjusting for case mix differences, the researchers found no health advantages associated with greater utilization, suggesting that savings associated with reduced utilization of the procedure in Texas could be achieved with no deleterious clinical consequences .
One estimate suggests that, in aggregate, only 27 percent of the weighted variation in Medicare spending across regions can be explained by population illness levels . If spending levels in all regions were made to match those in the lowest decile (age-, sex-, and race-adjusted), then Medicare could see savings of up to $40 billion in 1996 dollars .
ii. Inappropriate use The second body of research that addresses waste in the system attempts to directly measure how frequently certain medical services are delivered for medically inappropriate indications. Results from this literature often demonstrate high levels of inappropriate use. For example, a 1993 study of members of seven managed care organizations found that about 16 percent of hysterectomies performed were judged to have been clinically inappropriate, and 25 percent of the patients underwent hysterectomy for uncertain indications . A more recent study (in 2000) on hysterectomies found more dramatic results. Among hysterectomies performed in a capitated medical group in Southern California, 70 percent of cases were judged to have been inappropriate, according to RAND appropriateness criteria. Of the 497 women studied, 71 had hysterectomies for conditions covered by three recent ACOG criteria sets. The recommendation for hysterectomy was judged inappropriate for 53 percent of that subset by the RAND criteria and for 76 percent according to the ACOG criteria .
In other cases, the rates of inappropriate use are relatively low, but there is a wide range of situations in which appropriateness is uncertain, which demonstrates the need for a stronger evidence base. For example, in one study, 4 percent of coronary angiographies performed at 15 hospitals in New York State were rated inappropriate; another 20 percent were rated uncertain. The rate of inappropriate use varied from 0 percent to 9 percent among hospitals, but the difference was not significant . In another study, 4 percent of percutaneous transluminal coronary angioplasty (PTCA) performed at 15 hospitals in New York State were rated inappropriate; another 38 percent were rated uncertain. The inappropriate rate varied from 1 percent to 9 percent by hospital, the uncertain rate from 26 percent to 50 percent .
Trends toward inappropriate and uncertain use appear in other clini-
cal areas as well. Reviewing cases of new-onset chest pain not due to myocardial infarction at one of five Los Angeles-area hospital emergency departments revealed that 7 percent of those who received some form of diagnostic cardiac testing had tests that were judged to be inappropriate. A literature review on cases of metastatic renal cell cancer (MRCC) rated 46.9 percent of treatments as inappropriate and 25.8 percent as uncertain . A review of Medicare patients in three geographic areas revealed that 32 percent of the sample had carotid endarterectomy for inappropriate reasons, and 32 percent for uncertain reasons . Seventeen percent of diagnostic upper gastrointestinal endoscopy procedures for Medicare patients were performed for inappropriate indications, and 11 percent were performed for uncertain indications . In cases of hospital use, 23 percent of admissions were judged to be inappropriate and an additional 17 percent could have been avoided by the use of ambulatory surgery .
These studies often examine a specific intervention (e.g., upper gastrointestinal endoscopy or percutaneous coronary angioplasty) and evaluate the usefulness in a number of clinical indications. Most of the appropriateness research focuses on high unit cost services. However, significant expenditures associated with overuse may accrue from inappropriate utilization of low unit cost services if they are used in sufficient volume (e.g., routine blood testing, imaging procedures). Moreover, many of the studies cited above are based on data from the 1980s. The more recent small area variations literature suggests that substantial inappropriateness likely still exists, but much more work is needed in the area if we are to better understand, and address, the inefficiencies in the system.
These findings of substantial variation in practice patterns and often large rates of inappropriate use highlight the fact that the merit of a specific medical intervention depends on the precise reason for use. Thus, in most situations, detailed patient-specific information is required before reporting whether the use of a drug, test, or device is worthwhile.
It is important to recognize that one cannot say that a particular medical service is always appropriate or always inappropriate. Consider an example in the area of diagnostic imaging: radionuclide cardiovascular imaging (RNI). This is but one type of diagnostic imaging, but understanding the appropriateness of imaging as a whole is crucially important. Diagnostic imaging services reimbursed under Medicare’s physician fee schedule have grown more rapidly than any other type of physician service. Between 2000 and 2005, spending doubled from $6.6 billion to $13.7 billion . In 2005 the American College of Cardiology Foundation convened a technical panel to assess the appropriateness of RNI for 52 indications . Of the 52 indications, 13 were deemed inappropriate, 27 appropriate, and for 12 the appropriateness was uncertain. Moreover, there was not even consensus on all of the indications for which RNI was deemed appropriate.
For example, for 6 of the 27 indications deemed appropriate, there was strong disagreement among the panelists about that designation. Much more research is needed to reduce the level of clinical uncertainty and move the system toward efficient practice patterns.
However, CER will not be sufficient to eliminate overuse. Even when identified, system factors and the complexities of care limit the ability of the system to eliminate the waste. Research on these system factors, including patient- and system-oriented interventions such as benefit design and clinician/hospital reimbursement, will be needed to complement CER and to allow development of the systems needed to realize the potential offered by CER.
b. Underuse Paradoxically, while overuse in the healthcare system is common, underuse of medical services rigorously determined to provide substantial clinical benefit is also widespread. While the small area variation discussion commonly focuses on overuse, similar aggregate-level outcomes in high-expenditure areas and low-expenditure areas imply that some of the small area variation may be due to underuse. For example, among patients with heart attacks who were considered “ideal candidates” for beta-blockers, those who actually received the needed drug ranged from 5 percent to 92 percent of patients among the 306 Dartmouth Atlas Hospital Referral Regions (HRRs) .
A substantial portion of underuse reflects the failure of individuals or their physicians to use preventive services or to manage their chronic illnesses as the scientific evidence would recommend. CER is needed to improve our ability to identify when variation represents underuse and when it represents overuse so that the system can respond appropriately. However, as with overuse, CER will not be sufficient to eliminate underuse. While the clinician-patient relationship plays a critical role in this shortcoming, systemic effects such as access to care, benefit design, and ability to pay are also likely contributors, and more research examining these factors will be needed to improve the ability of the system to integrate CER findings into practice.
2. Effects of Inefficiency on Key Stakeholders
Inefficiency in the healthcare system, particularly that which leads to unnecessary expenditures, affects all stakeholders. Both overuse and underuse reduce the value of the resources devoted to the healthcare system. The enormous incremental costs associated with this inefficiency are borne throughout the system.
a. Individuals Individuals, whether they use the system or not, pay for these inefficiencies in several ways and are unmistakably worse off. First, in some
cases, individuals pay out of pocket for services (e.g., total body imaging scans) that provide little value in terms of clinical outcomes. Second, the financial costs associated with waste are reflected in higher healthcare premiums. These are paid by workers either directly or, because higher healthcare costs lead employers to pay lower wages, indirectly . Third, higher costs for public programs are financed by taxpayers. The costs of the largest public program, Medicare, rose 8.9 percent to $342.0 billion between 2004 and 2005 . Furthermore, projections suggest that Medicare will grow at an annual rate of over 9 percent between 2005 and 2015 . The growth of Medicare spending will represent a serious burden for taxpayers and a significant challenge for policy makers. It is well established that the tradeoff between access to medical care and how to pay for it is a complex and extremely political issue.
Fourth, high healthcare costs are also associated with declining rates of health insurance coverage . To the extent that greater waste leads to fewer covered individuals, those that are un- and under-insured must bear greater financial risk and suffer the consequences of diminished access to valuable care in the event that such care is needed.
Finally, inefficiency generates additional adverse consequences for patients already engaged in the system. Specifically, the over-consumption of care often entails clinical risk as well as financial costs. Over the past decade, the “patient safety” movement has brought to light the extent of the clinical and economic ramifications of avoidable medical errors. For example, hospital-acquired infections are estimated to be responsible for between $3.5 billion and $5.7 billion in excess healthcare costs each year [23, 24]. Under-utilization also generates suboptimal clinical outcomes as patients forego utilization of important services.
b. Employers The clinical and financial effects of inefficient care delivery on other stakeholders are more complex. To the extent that employers bear a large fraction of the costs associated with inefficiency, they are adversely affected. As mentioned above, standard economic models supported by empirical evidence suggest that, over time, employers shift the costs of higher healthcare spending to workers in the form of lower wages. However, in the short run, employers (or the shareholders) may bear some of the costs of inefficiency. Moreover, the ability to shift cost to workers is limited for retiree expenses, suggesting that shareholders will bear the costs of inefficiency for this population of workers. Employers may also bear some of the administrative costs associated with managing healthcare benefits in an environment of rising costs and considerable inefficiency.
c. Health insurers The fiscal implications of inefficiency for insurers are also complicated. To the extent that cost increases can be anticipated, they may
be included in premiums. However, as premiums escalate, the demand for coverage may be dampened, suggesting that, on balance, insurers will find it challenging to remain profitable in a rising cost environment over the long run. Yet with the challenge comes opportunity. If insurers can develop ways to address the problems of inefficiency in the healthcare system, substantial profit opportunities may arise.
d. Providers of healthcare services Providers of healthcare services—especially those whose income is related to productivity, not quality of care—may be one stakeholder group that benefits from inefficiency. Since one group’s expense is another’s revenue, the payments for unnecessary interventions are income for the providers of those services. Thus, while no physician or hospital may intentionally, or even knowingly, provide unnecessary services, they likely reap some financial gain from the services delivered, necessary or not. The magnitude of this effect for particular providers depends on the extent to which they deliver unnecessary care. Providers of necessary care would not be adversely affected by reductions in the use of unnecessary services.
Reductions in the use of unnecessary care may offer indirect benefit to providers in the long term. Specifically, higher costs lead to fewer people with coverage. This may place a burden on providers who are increasingly called on to provide uncompensated care. Providers may also benefit from any reductions in inefficient care because they may find this type of cost containment preferable to other approaches (such as fee reductions).
e. Manufacturers The impact of inefficiency (and efforts to reduce inefficiency) on manufacturers is much the same as on the providers of those services. Any reduction in utilization may be a reduction in revenue, but the effects will target low-value or unnecessary services. Manufacturers that have the potential to make important clinical advances can thrive in a low-waste environment. Moreover, relative to other cost containment efforts that may impact manufacturers, efforts to reduce unnecessary use of certain medical products may be preferable.
3. Uses of CER
Discussions about CER frequently focus on the use of these evaluations to assist in development of practice guidelines or in coverage and payment decisions. While CER could be used in these specific endeavors, CER is needed for more far-reaching efforts to improve the efficiency of the healthcare system. The critical nature of a comprehensive CER agenda arises because of the lack of controlled assessments of available therapeutic options and the substantial amount of patient heterogeneity that exists. Waste generally arises when services that are valuable in some clinical situ-
ations are applied to other indications. CER is an essential tool to determine which intervention should be delivered to which person and in what clinical circumstance.
By facilitating improved targeting of both the clinical intervention and the specific patient population, the information provided by CER can benefit key stakeholders, particularly patients and payers. Specifically, by reducing the uncertainty about which treatment course is most appropriate, CER can decrease the frequency that patients receive inappropriate care, reducing costs and the potential for harmful medical errors. Similarly, CER can facilitate efforts to develop coverage policy and design value-based insurance packages, which should enhance the return on healthcare expenditures made by payers—private or public . Taking the perspective of the provider, the effects of CER on utilization will depend on both the nature of the product and the incentives in place to use the service. If coverage and reimbursement levels reflect the findings of CER (i.e., payment based on clinical effects, not exclusively on production costs), providers and manufactures of high-value services should find that the CER increases their market share. However, the demand for low-value services will likely (appropriately) decline. Given that the burden of proof necessary to demonstrate value in the marketplace may intensify, so might the costs to perform the requisite CER studies.
A particular concern for providers is that cost containment efforts designed to eliminate the use of unnecessary services often inadvertently lead to restrictions on the provision of needed care. In almost all of the studies that report the appropriate indications for the use of a specified intervention, the appropriateness is “uncertain” in a significant portion of situations. Recall that there are few instances where the use of a specific drug, diagnostic test, or procedure is always appropriate or inappropriate. This underscores the need for a CER agenda that is able to measure health and economic impact on a granular level that will ultimately target those specific circumstances when certain interventions should and should not be used.
While the evidence examines both under- and overuse of selected medical services, one cannot accurately predict the net effect of a more efficient system on expenditures. This is related to the tradeoffs of how a better evidence base drives the increased use of more valuable services (and likely increases expenditures) and slows the utilization of low-value interventions (and decreases spending).
Individual CER studies may not always suggest that the least expensive course of action is the appropriate course of action—recall that “the most expensive therapy is the one that doesn’t work.” However, medical culture tends to be driven toward the adoption of new, expensive services, and cost growth has widely been attributed to the development and diffusion of new
medical services [25-27]. Therefore, on balance, we would expect that CER would tend to dampen spending to a level below that which would otherwise occur, because the ”adopt everything for everyone” mentality would be replaced with an “adopt when appropriate” paradigm. For example, a 2006 study examined whether some stable, high-risk patients with persistent total occlusion of the infarct-related coronary artery should undergo percutaneous coronary intervention (PCI) in addition to receiving optimal medical therapy . Although use of this procedure in such cases was not universal, the authors reported an inclination among physicians toward its use. In this case, a randomized trial demonstrated that PCI did not improve clinical outcomes, suggesting that resources could be saved by foregoing the procedure. The trend would likely have been toward greater use, and the CER-suggested lower use was medically appropriate.
Since the literature on diffusion of medical technology clearly shows a preference among U.S. clinicians to use new interventions before definitive clinical data are available, one can safely assume that the clinical data provided by a CER agenda will improve the quality of care. However, it should not be assumed that the completion and implementation of a CER agenda will save money in the short term. The short-term financial consequences will depend on how CER is used and on whether the savings incurred to lower rates of use of low-value interventions will offset the added expenses of the increased use of higher-value services.
While enhancing the health of Americans is a noble goal, we acknowledge that cost containment is an integral and inevitable part of the future healthcare policy. Without a strong investment in CER, patients and providers are more likely to face unintended “across the board” restrictions on the provision of valuable care because of the fiscal pressures that are being imposed on public and private health care payers. Whether these are manifested by fewer insured individuals or by the underinsurance of those with some type of benefits, CER provides the knowledge base by which providers of high-value services can advocate their continued use, using accepted scientific approaches to make their case. The findings of research that directly compares the pros and cons of available treatment options from numerous perspectives will be important for clinical practice, regardless of the cost containment/benefit reform approaches being considered. Cost containment efforts that rely on an improved evidence base are likely preferable to current efforts to drive all practice toward those of the lowest cost. Findings from CER should be used to better target, not to limit, care.
The exact mechanisms by which CER will lead to enhanced efficiency will vary based on the level of detail of the data generated by the studies and the ability of the system to implement the findings in everyday practice. On the quality improvement side, similar challenges have been identified in studies examining the suboptimal uptake of evidence-based practice
guidelines. From the financial perspective, cost-sharing approaches aim to control spending by making patients pay more at the point of service. Most efforts to raise patient out-of-pocket costs have resulted in higher costs across all services (with the possible exception of some preventive health services). It has been demonstrated that financial disincentives are often placed at the patient level, making adherence with evidence-based care difficult. Yet when faced with higher costs, patients often make poor clinical decisions, which in fact could, in some cases, lead to greater overall costs. Thus, the alignment of clinical and financial incentives is a necessary component to ensure the attainment of an efficient delivery system. The status quo has been unable to align quality improvement and cost containment initiatives. In fact, in some instances they actually compete with one another, contributing directly to ineffiency .
Such an alignment of incentives is possible in the setting of improved clinical evidence—driven by CER—and health benefit reform. Value-based insurance design (VBID) represents a “clinically sensitive, fiscally responsible” approach that advocates keeping patient out-of-pocket payments low on high-value services and raising them on services of no or marginal clinical value. Similar processes can be developed for clinician payment (e.g., payment based on quality of care delivered, not productivity). Implementation of such a scheme, in any form, would require greater CER since the relative value of services would be based directly on the findings. The advantages of such an alignment of clinical and economic incentives are obvious when compared to the current approach of untargeted “across the board” cost-sharing schemes, where the rates of both non-essential and essential services are negatively affected by higher out-of-pocket rates. By using incentives to encourage the use of high-value services and discouraging low-value ones, VBID has the potential to achieve marked increases in the efficiency of the healthcare system.
Supply-side-oriented healthcare reform approaches could also benefit from added investment in and coordination of CER. Certainly, coverage policy and clinical guidelines require such knowledge. But other initiatives, such as provider education, disease management, or pay-for-performance programs, all require an understanding of which services provide value in which settings and how quality and cost metrics can be designed in a clinically meaningful way.
Healthcare cost growth has placed a growing strain on our healthcare financing system. Although there is no consensus about how we can address the healthcare cost issue, most stakeholders would probably agree that the resources devoted to health care must be allocated more efficiently. This will entail being able to identify situations when more resources are necessary to
overcome the problem of underuse of highly valued services that improve health, as well as when money is being wasted on interventions that do not improve health, or worse, actually produce adverse consequences.
Regardless of the reform approach considered—market-based health savings accounts or a system administered through a single source—enhanced efficiency will require more detailed knowledge about the relative effectiveness of different interventions in specific clinical indications. All vested stakeholders should encourage investment in an infrastructure that prioritizes and undertakes investigations that yield practical information on which services to provide to which patients and when. Our healthcare system is too complex and too large to be guided without an appropriate knowledge base. Moreover, because innovation in the healthcare sector is substantial, investment in an infrastructure that would allow the assessment of the clinical and economic impact of new and existing diagnostic and treatment modalities is essential.
Creating this infrastructure will require a substantial investment. For those who consider the upfront investment necessary to create such an infrastructure to be unaffordable, it is imperative to contemplate the costs of the status quo that propagate tremendous inefficiency.
1. Schoen, C., et al. 2006. US health system performance: A national scorecard. Health Affairs Web Exclusive, W457-475.
2. Kaufman, J. S., et al. 1998, Subcutaneous compared with intravenous epoetin in patients receiving hemodialysis. Department of Veterans Affairs Cooperative Study Group on Erythropoietin in Hemodialysis Patients. New England Journal of Medicine 339(9):578-583.
3. Fisher, E. S., and J. E. Wennberg. 2003. Health care quality, geographic variations, and the challenge of supply-sensitive care. Perspectives in Biology and Medicine 46(1):69-79.
4. Fisher, E. S., et al. 2003. The implications of regional variations in Medicare spending. Part 1: The content, quality, and accessibility of care. Annals of Internal Medicine 138(4):273-287.
5. Fisher, E. S., et al. 2003. The implications of regional variations in Medicare spending. Part 2: health outcomes and satisfaction with care. Annals of Internal Medicine 138(4):288-298.
6. Wennberg, J. E., E. S. Fisher, and J. S. Skinner. 2002. Geography and the debate over Medicare reform. Health Affairs Suppl Web Exclusives:W96-114.
7. Guadagnoli, E., et al. 1995 Variation in the use of cardiac procedures after acute myocardial infarction. New England Journal of Medicine 333(9):573-578.
8. Wennberg, J. E., and M. M. Cooper, eds. 1999. The Dartmouth Atlas of Health Care. Chicago: American Hospital Association.
9. Bernstein, S. J., et al. 1993. The appropriateness of hysterectomy. A comparison of care in seven health plans. Health Maintenance Organization Quality of Care Consortium. JAMA 269(18):2398-2402.
10. Broder, M. S., et al. 2000. The appropriateness of recommendations for hysterectomy. Obstetrics and Gynecology 95(2):199-205.
11. Bernstein, S. J., et al. 1993. The appropriateness of use of coronary angiography in Ne York State. JAMA 269(6):766-769.
12. Hilborne, L. H., et al. 1993. The appropriateness of use of percutaneous transluminal coronary angioplasty in New York State. JAMA 269(6):761-765.
13. Halbert, R. J., et al. 2006. Treatment of patients with metastatic renal cell cancer: A RAND Appropriateness Panel. Cancer 107(10):2375-2383.
14. Winslow, C. M., et al. 1988. The appropriateness of carotid endarterectomy. New England Journal of Medicine 318(12):721-727.
15. Kahn, K. L., et al. 1988. The use and misuse of upper gastrointestinal endoscopy. Annals of Internal Medicine 109(8):664-670.
16. Siu, A. L., et al. 1986. Inappropriate use of hospitals in a randomized trial of health insurance plans. New England Journal of Medicine 315(20):1259-1266.
17. Kuhn, H. 2006. Testimony on Payment for Imaging Services under the Medicare Physician Fee Schedule. July 18. Online. Available at http://www.hhs.gov/asl/testify/t060718.html. Accessed January 22, 2007.
18. Brindis, R. G., et al. 2005. ACCF/ASNC appropriateness criteria for single-photon emission computed tomography myocardial perfusion imaging (SPECT MPI): A report of the American College of Cardiology Foundation Quality Strategic Directions Committee Appropriateness Criteria Working Group and the American Society of Nuclear Cardiology endorsed by the American Heart Association. Journal of the American College of Cardiology 46(8):1587-1605.
19. Gruber, J. 1994. The incidence of mandated maternity benefits. American Economic Review 84(3):622-641.
20. CMS Office of Public Affairs. 2007. Press Release: CMS Releases U.S. Health Spending Estimates Through 2005. January 9. Online. Available at http://www.cms.hhs.gov/apps/media/press/release.asp?Counter=2069&intNumPerPage=10&checkDate=&checkKey=&srchType=&numDays=3500&srchOpt=0&srchData=&keywordType=All&chkNewsType=1%2C+2%2C+3%2C+4%2C+5&intPage=&showAll=&pYear=&year=&desc=&cboOrder=date. Accessed January 22, 2007.
21. Borger, C., et al. 2006. Health spending projections through 2015: Changes on the horizon. Health Affairs 25(2):w61-73.
22. Chernew, M., D. M. Cutler, and P. S. Keenan. 2005. Increasing health insurance costs and the decline in insurance coverage. Health Services Research 40(4):1021-1039.
23. Burke, J. P. 2003. Infection control—A problem for patient safety. New England Journal of Medicine 348(7):651-656.
24. Hollenbeak, C. S. 006. Factors associated with risk of surgical wound infections. American Journal of Medical Quality 21(6):29S-34S.
25. Chernew, M. E., et al. 1998. Managed care, medical technology, and health care cost growth: A review of the evidence. Medical Care Research and Review 55(3):259-288; discussion 89-97.
26. Cutler, D. M. 1995. Technology, Health Costs, and the NIH. Cambridge, MA: Harvard University and the National Bureau of Economic Research.
27. Newhouse, J. P. 1992. Medical care costs: How much welfare loss? Journal of Economic Perspectives 6(3):3-21.
28. Hochman, J. S., et al. 2006. Coronary intervention for persistent occlusion after myocardial infarction. New England Journal of Medicine 355(23):2395-2407.
29. Chernew, M. E., A. B. Rosen, and A. M. Fendrick. 2006. Rising out-of-pocket costs in disease management programs. American Journal of Managed Care 12(3):150-154.