Leadership Commitments to Improve Value in Health Care: Finding Common Ground: Workshop Summary (2009)

Coordinator

Peter Juhn, Johnson & Johnson

Other Contributors

Patricia Adams, National Pharmaceutical Council; Pat Anderson, Stryker; Marc Berger, Lilly; Catherine Bonuccelli, AstraZeneca; Spencer Borden, Health Care Systems, Johnson & Johnson; Linda Carter, Global Regulatory Affairs, Johnson & Johnson; Christopher M. Dezii, Bristol-Myers Squibb; Dave Domann, Ortho McNeil Janssen, Johnson & Johnson; Mary Erslon, Covidien (formerly Tyco Healthcare); Barry Gershon, Wyeth; Kathryn Gleason, National Pharmaceutical Council; Page Kranbuhl, Stryker; Jerry McAteer, Siemens Medical Solutions Diagnostics; Newell McElwee, Pfizer; Scott McKenzie, Ortho Biotech, Johnson & Johnson; Gary Persinger, National Pharmaceutical Council; Wayne Rosenkrans, AstraZeneca; Lisa Saake, Covidien (formerly Tyco Healthcare); Phil Sarocco, Boston Scientific; Hemal Shah, Boehringer-Ingelheim; David Sugano, Schering-Plough; Steve Teutsch, Merck; Karen Williams, National Pharmaceutical Council

The companies in the healthcare products industry represent a unique sector of health care focused on the development and implementation of innovative medical products. The pharmaceutical, medical device, and diagnostic industries have contributed technologies that increase survival and decrease disease-associated morbidities and mortalities. The greater life expectancies and improved quality of life that patients with, for example, cardiac disease, diabetes, and cancer experience can in many ways be credited to improved medical diagnostic technologies and improved therapies: from the improved ability to detect tumors by the use of new imaging procedures to the increased use of cholesterol drugs, blood thinners,

and new cancer medicines; new home-based therapeutic devices, such as diabetes monitors and home oxygen therapy; and the improvements in quality of life achieved with orthopedic implants.

The medical device and diagnostic portion of the industry includes more than 20,000 companies worldwide, most with an average of 50 employees or fewer, and produce more than 80,000 brands and models of medical devices for the U.S. market. Medical technology innovation typically consists of incremental improvements to existing technologies; therefore, the product life cycles in this sector range from about 18 months to 2 years (Advanced Medical Technology Association, 2007). Other medical technology products, including those requiring large capital investments, long-term clinical data, or physician adoption for market penetration, have longer life cycles. Consequently, follower competitors typically capture the benefits from the innovator in the medical device and diagnostic market.

In contrast, the biopharmaceutical portion of the industry is much more consolidated, with some 200 pharmaceutical companies, 400 publicly traded biotechnology companies, and 1,400 privately held biotechnology companies in existence worldwide. The pharmaceutical industry introduces 25 to 30 new innovative products each year, on average, and has some 2,000 products in development that may be useful in all areas of therapy. Biotechnology has created more than 200 new therapies and vaccines and has some 400 products in active development, including products for the treatment of cancer, diabetes, human immunodeficiency virus infection/AIDS, and autoimmune disorders. The discovery and development process for biopharmaceuticals takes an average of 15 years and involves sequential steps, from discovery to preclinical animal tests and human studies. Over the course of drug development, the product attrition rate is high, and the cost of bringing an individual drug to market has been estimated to be more than $800 million (Pharmaceutical Research and Manufacturers of America, 2005a), with new data from the Tufts Center for Drug Development suggesting that the cost for the development of a new biopharmaceutical may top $1.2 billion.

Although these manufacturers produce a heterogeneous assortment of products, each has core capabilities in the design and implementation of programs that produce and disseminate evidence about the safety and efficacy of their products to patients and healthcare providers. Given the considerable requirements for regulatory and market information from this sector, it is likely that it collectively has more experience in evidence development and dissemination than any other healthcare sector.

The financial and resource investments of pharmaceutical and device manufacturers in evidence development and dissemination are substantial. In 2005, the biopharmaceutical industry spent more than $51 billion on new product development (research and development [R&D]); it spent

FIGURE 8-1 Financial and resource investments of pharmaceutical and device manufacturers for the development of new therapies.

SOURCES: DiMasi et al. (2003); Pharmaceutical Research and Manufacturers of America (2005b,c).

about 40 percent of that amount on preapproval clinical trials and considerably less for postmarketing assessment (Pharmaceutical Research and Manufacturers of America, 2005b). The financial and resource investments required for pharmaceutical development are shown in Figure 8-1.

The development of evidence required for regulatory approval varies by the type of product and the disease. Medical device and diagnostic manufacturers supply data to the Food and Drug Administration (FDA) for premarketing approval applications (PMAs) or 510(k) premarketing notifications (as required by section 510(k) of the Federal Food, Drug, and Cosmetic Act), depending on the specific product under evaluation. Pharmaceutical and biotechnology manufacturers provide data in new drug applications or biologics license applications.

The diversity of medical devices and diagnostics has led to a risk-based classification system, and that system uses established standards to evaluate device safety and effectiveness distinct from those that FDA uses to evaluate drug safety and effectiveness. The system calibrates regulatory controls to

the risk that specific devices pose, and only a small percentage of devices—those that pose the greatest risk—require clinical studies for PMAs. Between 50 and 80 high-risk (i.e., Class III) devices receive approval under the PMA review process annually. Most manufacturers of medical devices follow the 510(k) premarketing notification process for substantially equivalent technologies. According to FDA, more than 4,000 new, low-risk (i.e., Class I) devices that are exempt from FDA premarketing review are marketed each year; about 3,500 medium-risk (i.e., Class II) products are reviewed and approved for marketing by FDA under the 510(k) premarketing notification process, with about 8 percent of those products subject to special controls that require clinical data. Postmarketing surveillance studies are an example of special controls. Finally, investigational device exemption allows an investigational device or investigational diagnostic to be used in a clinical study to support a PMA or 510(k) premarketing notification process.

The evidentiary standard for devices is less burdensome than that for drugs because of the medical technology innovation process and its shorter cycle time. Consequently, the amount of evidence required for device evaluation is inherently different from that required for marketing applications for drugs. Also, the type of evidence permitted to substantiate the efficacy of a device is much broader than the type of evidence permitted to substantiate the efficacy of a drug. The evidentiary standard for the approval of biopharmaceuticals, however, is significant. Marketing approval requires randomized controlled trials, starting with dosing and safety studies, typically conducted with healthy volunteers (Phase I studies), and progressing to increasingly larger and logistically more complex studies with patients with the disease to demonstrate safety and efficacy (Phase II and Phase III studies). Biopharmaceutical companies must routinely make go–no go investment decisions throughout the average 15-year development cycle on the basis of the estimated clinical effectiveness and safety of the product, expected regulatory and coverage and reimbursement requirements at launch, the probability of technical and commercial success, the place of the product in expected medical practice, and the net present value of the product.

The growing need for patients, providers, and payers to evaluate treatment options and the financial implications of these evaluations are resulting in increasing demands for evidence of real-world comparative effectiveness and safety for treatment- and coverage-related decision making. However, these new research questions are fundamentally different from the research questions that regulatory agencies ask when they make marketing approval decisions (Lomas et al., 2005) and add incremental costs to the existing

expense of development that the manufacturers have already incurred. In addition to these different types of questions, the larger populations required to power comparative studies adequately also add to the cost. All of these costs serve to lower the net present value of products in development. This lower net present value, in turn, can result in more no go decisions in the development process and higher product attrition rates because of reduced commercial viability.

If the responsibility for the conduct and payment for the development of real-world evidence lies with the industry, then industry will need to develop a process to incorporate the costs and benefits of conducting these studies into early-phase development decisions. That process will, by definition, be inexact. It is impossible to completely anticipate future research needs, because so many research decisions are based on the results of interim research and an evolving understanding of the underlying science, which may not be known for some years. Nevertheless, much of the uncertainty could be reduced if standards for the types of studies that represent appropriate evidence and the types of findings that would allow appropriate third-party coverage were created through broad stakeholder consensus. Such standards will ensure the development of the best-quality evidence, lower the uncertainty for that element in drug development, and increase the ability of manufacturers to continue to develop new products.

Even with the increased predictability that coverage-related decision-making standards foster and the considerable effort being expended on increasing R&D productivity, these incremental costs can become unsustainable and can significantly hinder innovation. A lack of market valuation of products that show incremental improvements or the valuation of only breakthrough innovations can create a greater perceived financial risk in product development when the considerable costs are taken into account and may result in the introduction of fewer products. The introduction of fewer products also reduces the therapeutic choice options that providers have to overcome patient variability in response to therapy and reduces price competition.

For medical devices and diagnostics, the granularity of the evidence that payers need is getting tougher to obtain; that is, meeting the demand for answers from studies with smaller subpopulations or population segments requires more time and more resources. There remains a lack of clarity of how payers’ reimbursement decision-making processes integrate the evidence that healthcare manufacturers generate. Finally, use of the “gold standard” type of clinical study, randomized clinical trials, is not always feasible for medical devices and diagnostics, for multiple inherent reasons. Thus, it is essential that the evolution of a consensus on the value and limitations of studies be explored by using current methodological standards.

These unintended consequences of the development and use of addi-

tional evidence for coverage-related decision making should be considered as the evidence requirements for real-world effectiveness and safety develop, as this may limit the ability to fully deliver on the promise of innovation. Members of the healthcare product sector recommend that informed members of the academic economic community evaluate the potential impact of these new evidentiary standards on the economics of innovation and consequent patient care to help guide payers and the government as they make policy decisions.

Manufacturers generate evidence both to inform internal investment decisions regarding the market viability of emerging products and to inform stakeholders (e.g., regulatory agencies, payers, healthcare professionals, patients, and employers) about the benefits and risks of new and existing treatments. The priority given to the collection of various types of data for each product varies over time. For example, preapproval studies generally focus on dose finding, identification of the target population, and the demonstration of safety and efficacy, whereas postapproval studies may be directed toward the evaluation of comparative effectiveness, cost-effectiveness, patient preference, long-term safety, and patient adherence and may include observational studies. The unique perspectives of various stakeholders can result in a potentially broad range of questions to be addressed about the specific product or disease state. Furthermore, the type of evidence required will vary by desired treatment outcome (e.g., whether curative, preventative, or palliative treatment is needed), the time course of the underlying disease (e.g., whether an acute or a chronic intervention is needed), and whether the treatment alternatives for a given condition already exist or are in development. In the past few decades, the development of evidence in support of the first launch of a product has extended beyond traditional placebo-controlled efficacy and safety trials to include information from studies with certain subpopulations and information on comparative efficacy, quality of life, outcomes, and cost-effectiveness.

After FDA approval, manufacturers provide evidence and generate additional evidence to help providers and patients make appropriate treatment choices and to help payers facilitate their coverage-related decision making through the coverage and payment processes. This may involve the need for studies focused on real-world utilization and outcomes to supplement the findings of clinical trials that have established the defined populations, dosing, and treatment durations. Most recently, stakeholders have expressed a strong interest in information on real-world comparative effectiveness, resulting in requests for additional evidence, including the systematic collection of data from and analysis of the scientific literature,

analyses of the data in clinical and claims databases, and the collection of results from studies of specific technologies. Many manufacturers have devoted significant resources to this type of evidence development for internal purposes, scientific meeting presentations, and informing or partnering with various groups developing similar types of evidence, as discussed below. Thus, healthcare manufacturers play a key role in the development of evidence. Their investments in evidence development for pre- and post-marketing approval are substantial, with significant risk, and require long-term investments for the development of pharmaceuticals.

The industry also recognizes that in addition to the sector’s role in evidence development, several other public and private organizations also contribute to the development and synthesis of evidence about the products that the industry produces. These include the Cochrane Collaboration (Cochrane Collaboration, 2007); the Oregon Drug Effectiveness Review Project process (Oregon Health and Science University, 2007); the Agency for Healthcare Research and Quality (AHRQ) Comparative Effectiveness Program Centers (Agency for Healthcare Research and Quality, 2007b); and various private health technology assessment groups, including the Blue Cross Blue Shield Technology Center (BlueCross BlueShield Association, 2007) and the ECRI Institute (formerly Emergency Care Research Institute) (ECRI Institute, 2007). These organizations use a number of methodologies from evidence-based medicine, including systematic reviews; large observational studies with administrative data (e.g., data from commercial data vendors, like Ingenix and Premier, and health plans, like Kaiser, Partners, Aetna, and others); and increasingly, studies done with data from systems containing integrated health information, such as the systems of regional health information organizations and large electronic health record systems (e.g., the Cleveland Clinic, the Mayo Clinic, and Harvard Pilgrim).

It is important to recognize the dynamic aspect of product development and the associated development of evidence. There is uncertainty at every stage of product development, with significant but decreasing rates of attrition of compounds from Phase I through Phase III. At each stage, evidence development, both pre- and postapproval, builds on previous results and new understanding of the underlying science. It can therefore be impossible to anticipate the total research required during the life of a product because it is impossible to anticipate the results of research conducted at each step of the product’s life. Although healthcare payers prefer data from comparative trials (i.e., trials that compare active products) because of the type of evidence that they provide, comparative trials introduce substantial additional costs as well as risks, especially when they are carried out before approval. In addition, without a consensus on the value and limitations of studies conducted by using current methods, data from these studies become difficult to interpret for coverage decisions. As such, the involve-

ment of all stakeholders in choosing priority questions and conducting research into new methods is needed.

After the introduction of a new medicine into the market, providers go through a learning curve on appropriate product use. Appropriate product use is informed by the data that have been generated before the launch of the product, the information on the product label, data from supplemental research and case report studies, information obtained through informal and formal medical education, and most importantly, the personal experiences of the providers themselves. The use of the product beyond the product indication can be common and sometimes evolves into a recommended or best practice determined on the basis of that fact that use of the product as treatment for a reason other than its original indication has become well established within the medical community, even in the absence of a formal labeled indication (i.e., off-label use).

The learning curve associated with provider use of medical device and diagnostic technologies may be longer than that associated with provider use of drugs. Furthermore, the iterative improvements that mark device and diagnostic technology innovation tend to parallel increases in the skill levels of providers, so that outcomes depend on both product performance and practitioner expertise. Innovations in medical device and diagnostic technologies are not restricted to the premarketing phase of their development. Instead, actual practitioner use of devices in clinical practice typically spurs additional refinements and improvements. Clinical adoption thus serves as the beginning of an iterative process of feedback from medical practitioners, device redesign, use, and more feedback. Furthermore, in addition to technological refinements, these medical practitioners may use medical devices for reasons other than their original intended uses.

The proper synthesis and interpretation of a body of clinical evidence requires two critical but very different skill sets. Ideally, teams with collective expertise in the specific domain area, in methods of synthesis, and in the analysis of many types of clinical evidence should carry out this exercise; and they should use structured and reproducible techniques to carry out the exercise. The use of such a multidisciplinary team approach avoids the performance of evaluations by a clinical expert who tends to review and interpret the clinical literature from his or her personal perspective or by a nonclinical technical expert who might use structured methods to essentially filter out all evidence that does not conform to a predefined set of strict criteria that do not require clinical judgment. Both extremes are, of course, wrong, but in different settings they are both called “evidence-based medicine.”

The challenge of this new age of health technology assessment is to find

processes that are not specific either to clinical experts, who focus on evidence that supports their personal experience and views, or to methodological technicians, who focus only on the evidence that supports their preferences for the types of studies and data reporting that they are most comfortable in reviewing. Even the National Institute for Health and Clinical Excellence (2007), which probably considers the broadest scope of clinical evidence in its reviews, tends to depend more on nonclinical reviewers than on domain experts in its interpretation of clinical effectiveness. New evidence-based medicine and health technology assessment procedures must, from the beginning of the process, create teams of clinical experts and methodological experts who work hand in hand on the challenging task of assessing all potentially useful evidence for the comparison of technologies and then structuring the analysis to ensure that other groups may reproduce the synthesis and interpretation of the evidence. Ultimately, because of the various approaches used to interpret evidence, it is critical that this decision-making process be open to appeal through an independent, transparent, and facile process.

Healthcare product manufacturers disseminate evidence about their products strictly within a clearly defined regulatory framework that aligns with the approved product labeling. Labeled product information is proactively disseminated by multiple routes, including scientific presentations and publications, personal selling, prescriber advertising, product labels, speaker programs, product exhibits at conferences and events, and more recently, tightly controlled direct-to-consumer marketing. It is noteworthy that when new evidence outside the approved labeling information becomes available, product manufacturers cannot, because of regulatory restrictions, proactively disseminate or initiate discussions about these new data. They can, however, use these data in response to specific inquiries by providers and payers and can publish the findings of industry-sponsored clinical trials at scientific meetings and in peer-reviewed journals. Information dissemination from sources other than the industry are not subject to these regulatory restrictions, and evidence can therefore be disseminated through the broadest possible means, including continuing medical education classes, academic forums, the development of guidelines, published case studies, Internet chat rooms and blogs, pharmacist brochures, and health information websites.

The translation of new evidence into improved healthcare outcomes continues to be a primary goal for healthcare stakeholders. Manufacturers rely on the routes of evidence dissemination outlined above to drive the

application of evidence for their products. However, an awareness of new evidence does not automatically translate into new behaviors for patients or physicians. There are a myriad of reasons for this disconnect between knowledge and action, including the nearly overwhelming volume of new evidence available to physicians, patient nonadherence, gaps in the process of care, and a lack of decision support tools. The application of scientific evidence to medical decisions (evidence-based decision making) is complex. Factors such as baseline risk, variations in treatment response, susceptibility to adverse events, and patient preferences should be taken into consideration; but often they are not (Kravitz et al., 2004). Patient and healthcare system understanding of the applicable and actionable evidence could greatly enhance optimal product use and patient outcomes. In addition, the incorporation of this actionable evidence into direct patient care can be facilitated through systems that enable the use of health information technology, including electronic prescribing and decision support tools in electronic medical records. Accelerating the use of health information technology in healthcare will provide unparalleled opportunities to bring evidence-based information to the point of patient care and decision making.

A major underlying problem in the healthcare system continues to be that much of the important evidence about the most effective ways to treat patients that already exists is not embedded into clinical practice. In other words, healthcare providers are not using the best evidence available to make patient treatment decisions. Recent studies show that only a little more than one-half (about 55 percent) of adult patients receive the recommended care during a given encounter with the healthcare system (McGlynn et al., 2003). National medical societies are responsible for the creation of clinical practice guidelines and establishing these standards of care. Guidelines are created through the use of the evidence available from randomized controlled trials (much of it from the industry) and other forms of evidence, including medical consensus, when more rigorous evidence is not available. The best evidence available from clinical practice guidelines is disseminated to healthcare professionals by publication in medical journals and through continuing medical education and other postgraduate coursework, including relicensure and specialty recertification examinations. The evidence base for effectively treating patients, however, is constantly evolving, and best practices change over time. There are multiple opportunities for physicians and other healthcare professionals to keep abreast of new and revised guidelines and other advances and for their integration into clinical care; but there are also considerable barriers, and the diffusion of new evidence into medical practice appears to be slow and incomplete.

The balance between population-based evidence and the evidence needs for an episode of care, that is, the interaction between the individual patient and the physician, is critical. Evaluations should consider the evidence on

variations in individual responses to a particular treatment to ensure that an adequate variety of treatment choices is available to meet the needs of individual patients.

The application of evidence has been supported in recent years through healthcare quality improvement initiatives in which evidence-based care is evaluated and promoted through quality indicator measurement and pay-for-performance programs. These programs are designed to improve the practice of evidence-based care through transparent quality measurement and reporting. Programs such as the Health Plan Employer Data and Information Set offered through the National Committee for Quality Assurance,¹ Ambulatory Quality Alliance,² and Hospital Quality Alliance³ have created evidence-based quality measures that are increasingly applied in the institutional and ambulatory environments aligned with pay-for-performance programs directed by the organizations LeapFrog⁴ and Bridges to Excellence.⁵ It will be important to provide feedback to healthcare providers and payers on the quality of care that they provide and to provide incentives to practice evidence-based care. Additionally, those stakeholders responsible for generating evidence for medical practice and the use of medications must receive feedback on the types of evidence to be generated for future practice and the safe and appropriate use of medications.

Although the dissemination of evidence to health policy decision makers and healthcare providers for patient care decisions is critical, the availability of evidence for consumers is also necessary to support consumers’ increasing role in making decisions about their health management. Various organizations, including WebMD,⁶ Harvard Medical School,⁷ the Mayo Clinic,⁸ and others, have created consumer medical knowledge services available through the Internet, public health campaigns, the popular press, and broadcast consumer advertising.

Finally, the promotion of a learning environment in healthcare practice in which the best available evidence is applied and quality and outcomes are measured will be critical to advancing evidence-based care. However, until effective mechanisms to ensure the appropriate application of evidence to health care are defined, the development of new evidence will not result in meaningful improvements in patient care. General principles for apply-

ing evidence are lacking. The industry needs stable processes that lead to predictable outcomes, especially in the area of coverage and reimbursement decisions, to ensure its ability to deliver innovative products.

Several factors continue to influence the healthcare product industry’s ability to ensure the appropriate and safe use of their products to improve overall patient outcomes:

• With the availability of a range of new technologies and a deeper understanding of the molecular and genetic bases of disease, the science of drug development has become even more complex.

• When the industry successfully develops a new product, the internally developed evidence is sometimes not as well accepted as evidence developed by other groups (e.g., academia and government).

• Although the need for additional sources of evidence and types of evidence is acknowledged, there is confusion and inconsistency regarding the standards and methodologies that should be used to obtain these data and evidence and the relative importance of these data in creating clinical guidance.

• The application of evidence in coverage decisions is inconsistent. Decisions should be based on all of the available evidence (albeit appropriately weighted), without arbitrary rules placed on the inclusion or exclusion of certain types of evidence.

• There is very little evidence on nonpharmaceutical interventions available, making the comparison of new products in this area with the available alternative interventions very difficult.

The application of evidence in medical care is challenged in many ways. Physicians and other providers are overwhelmed with the vast amount of medical evidence constantly published and disseminated through publication in the medical literature. Additionally, clinical practice guidelines are often slow in adopting new evidence-based practices, and the dissemination of new guidelines and their recognition by healthcare providers are often delayed.

Without rapid access to new evidence at the point of care, physicians will continue to rely on the opinions of their colleagues and their own practice experience in patient care decision making. Unfortunately, current decision support methods focus on the enforcement of payment policies and the utilization of care rather than on support of the differential diag-

nosis and the treatment decisions that they make for their patients. Absent objective clinical evidence at the point of care, doctors may continue to overestimate the quality of care that they are providing. However, some evidence suggests that physicians are likely to change their clinical practices when they are provided with credible, actionable information about how the care that their patients receive compares with the recommended or best-practice care.

The move toward measuring medical outcomes, with healthcare providers receiving constant feedback on the quality of care that they provide (with incentives), will be key to improving the percentage of evidence-based care provided. Methods that provide up-to-date evidence through the use of decision support tools and methods that monitor patient outcomes in electronic medical records hold much promise for linking positive assessments to the rapid implementation of new clinical practice standards and the appropriate adoption of new technologies.

The large amount of patient-directed information in the marketplace can be confusing and misleading to many consumers unless it is actively filtered by well-trained healthcare professionals. Credible sources of consumer information need to be identified, and the information needs to be communicated in a form that allows patients to make educated decisions with their healthcare providers on the anticipated benefits and potential risks of a specific treatment.

In summary, the healthcare product industry has a rich experience base and competency in the development and the dissemination of evidence about their products. However, the majority of evidence development in this sector is driven by the product learning curve and the regulatory requirements for approval. Evidence dissemination, a core capability for the sector, is limited to evidence consistent with the product label. Application of the evidence is essential for the safe and appropriate use of effective new interventions.

The tasks of generating and helping to translate evidence into practice have long been core activities in pharmaceutical and medical device companies. However, the industry’s approaches to these activities are evolving

as it is asked to demonstrate the value of its products in a competitive, resource-constrained healthcare system. It is difficult to respond to this evolution, however, because there is little agreement about the right kinds of evidence and how the data should be generated. There are also barriers to the effective interpretation of the data and translation of the data into relevant, actionable, and patient-specific clinical information.

We believe that there is great value in evidence-based medical practice, but we are aware that developing and implementing systems that create and use such evidence is a very complicated process, and one that should be designed with careful thought and consideration. As key generators of evidence for the healthcare system, it is our view that we must consider all stakeholder perspectives in coming to consensus on what kinds of evidence are truly required for good decision making by regulators, payers, physicians, patients, and others. We also see a need for approaches to educating decision makers to help them understand and judge evidence in an objective manner and to place that information into the context of other medical procedures and tests. We advocate for decision support tools that will help with diagnosis, therapy selection, therapy adherence, and benefit design. This section will also include our assessment of some of the challenges associated with new standards for evidence, the barriers to overcoming them, and some suggested ways forward.

To attain the vision of achieving a fully evidence-based healthcare system by 2020, the healthcare product sector can make significant contributions to a number of issues in the area of evidence development:

• Explore standards for evidence development. There is no consensus on the kinds of evidence that are best suited to guide various kinds of healthcare decisions. Although the randomized controlled trial has been considered a “gold standard” for health policy decision makers, other types of evidence may be more relevant for clinical decisions at the patient care level. Furthermore, there is little agreement about how outcomes metrics, comparators, and study designs should be standardized. This can reduce the reliability of the design and conduct of clinical studies and the application of their findings. Research must be prioritized to address questions judged to be the most valuable by all healthcare stakeholders.

• Generate evidence that incorporates individual patient needs in the context of healthcare systems decision making. To meet the needs of regulators, payers, healthcare professionals, and patients, a balance must be struck between generating broad-based evidence of safety

and efficacy in studies with a controlled population and generating evidence of high relevance to particular individuals and subgroups in the real-world setting. In addition, variations in clinical responses and patient preferences need to be acknowledged and considered.

• Develop new methodologies and standards for application of the evidence. Randomized controlled trials comparing one treatment modality with another are very costly and require significant investments by product developers, patients, and providers. Alternative methodologies and standards for their use, including large and robust observational studies that use validated sources, such as electronic health record systems, should be explored. These new approaches can also be used to evaluate other (nonpharmaceutical) products and services, such as medical and surgical procedures, behavioral interventions, and nutritional supplements. The establishment of standards for methodologies other than the randomized controlled trial, including systematic reviews, should also be explored. Such studies will need to address issues of potential confounding because of the selection of patients receiving particular therapies based on characteristics (sometimes unmeasured) that are related to outcomes. In the absence of high-quality comparative information, methods for capitalizing on other information, for example, genomics, need to be developed to facilitate a scientific process for identifying the most appropriate management for patients.

The data required for regulatory review and approval by FDA are often not sufficient to meet all of the stated needs of insurers, patients, and other stakeholders; and those data rarely answer questions of effectiveness apart from efficacy. A clear and harmonized set of standards for evidence development that are consistently and appropriately applied will facilitate clear expectations for the quality of that evidence and broader agreement about the conclusions drawn from it. Despite the availability of guidelines from many different organizations (e.g., specialty and primary care professional organizations and federal agencies) and methods for the evaluation of utilization and outcome, no clear standards for guidelines on providing means for providers and consumers to understand the labyrinth of medical information and best recommendations for current care have been defined. These standards for evidence development should reflect input from a wide variety of stakeholders.

The healthcare product sector can provide insight into the evidence that is of greatest importance to patients, providers, and payers in making

treatment choices and can apply its expertise with research design standards to these additional studies.

• The healthcare product sector’s long experience with and considerable capacity for developing evidence for regulatory authorities has created a deep understanding of methods for the design and implementation of clinical trials and the strengths and weaknesses of such trials in meeting those evidentiary standards.

• The healthcare product sector also has experience in developing and implementing appropriate standards for the development of medical evidence and can define standards to support the development of evidence on the basis of other measures of product effectiveness with limited bias and error.

• Finally, the sector understands how patients and providers make decisions, including what kinds of information that they find important and how that information can be effectively communicated.

Even the highest-quality comparator trials often have rigid entry criteria and, consequently, have restricted and uniform patient populations that may limit the applicability of the findings of the trial to larger patient populations.

Depending on the available resources, the industry can contribute to the following:

• ensuring that the evidence is relevant to a broad range of patients and to specific populations,

• developing data with different patient populations who experience realistic follow-up consistent with that which today’s healthcare system provides, and

• conducting longer-term evaluations of patients experiencing concomitant medical conditions and health interventions.

Although this is an important endeavor, there are trade-offs between developing evidence and supporting the healthcare product industry’s continued ability to bring innovative products to market. This is particularly true given that the evidence required for product approval often differs from the evidence that payers, healthcare professionals, and other decision makers are now requesting. Indeed, the creation of customized clinical programs for each different type of healthcare organization, institution, or payer that answer questions relevant to every patient subpopulation is

neither realistic nor affordable. There is likely to be a continued struggle to find the right balance, as will be discussed below in the “Challenges” section.

The goal to create the best evidence for product use will always demand high standards for the creation of unbiased evidence on the basis of rigorous scientific methods and quality. This goal will continue to be the standard for regulatory authorization throughout the world. The performance of randomized placebo-controlled trials that control for bias, confounding factors, and systematic error is the core requirement for the demonstration of safety and efficacy for regulatory approval.

Observational data evaluations are important additional tools that help provide an understanding of treatment practice and that support the benefits and safety of a product. The data sources underpinning these evaluations hold some promise; however, many observational studies (particularly those used primarily for administrative claims) have inherent limitations, including incomplete information about the total medical care experience (e.g., patient follow-up) and questionable accuracy, quality, and validity. The following are considerations in the use of observational data and other methodologies:

• The validity of data sources needs to be confirmed and reported to improve quality and accuracy. The healthcare product sector can contribute to the setting of standards for the quality of the data sources used for observational studies to ensure quality output. These standards can build on the work already completed by AHRQ on standards for registries and will need to be established in partnership with the other healthcare sectors.

• The use of observational data requires adjustment for confounding, missing data, and possible systematic bias. Techniques that allow such adjustments to be made exist, but they need to be applied in a rigorous fashion with transparency and with clarity about the assumptions that have been made to allow confirmation through scientific studies. The sector can contribute to the generation of standards for the conduct of observational studies along the lines of the good clinical practice level standards currently used for randomized controlled trials.

• The published literature describes a host of other methodologies. These include simple or practical clinical trials, real-world studies and the application of various technical procedures, such as predictive modeling and Bayesian analysis. The healthcare product sector

can contribute to the further development and eventual use of these evolving methodologies through the use of its already established infrastructure for clinical product development.

Little direct work has been done to understand the therapeutic responses of subsets of patients or to describe (and predict) individual variations in response to therapy. However, as scientific knowledge on the molecular nature of disease states has evolved, there has emerged a greater opportunity to define and study responses to therapy with narrower, more clearly defined patient populations. These approaches to development and commercialization, known as personalized medicine or stratified medicine, challenge the standards of traditional business economics but have the potential to make evidence-based medicine more patient-centric. Nevertheless, challenging statistical issues arise when subgroups of patients who are likely (or unlikely) to respond or have an adverse reaction to a particular therapy are identified.

The research community, healthcare systems, and patients have invested tremendous amounts of time and effort in the development of evidence. Moving toward the 2020 vision of having 90 percent of the health care provided be evidence based by 2020 will require unprecedented cooperation among the sectors if this process is to be efficient and still create meaningful evidence to guide patient-centric decision making.

The responsibility for ensuring integrity in healthcare decisions is broadly distributed in U.S. society. Without a shared understanding of how this evidence is being interpreted, the development of new evidence may result in little or no benefit to patient care. A dialogue is needed to establish principles governing how evidence is to be integrated into healthcare coverage decisions. This has immediate import when these coverage decisions include the denial of access to medicines to disenfranchised populations or the swift introduction of a new technology for which early evidence shows that it provides superior benefit. It is also important over the long term as the requirements of ongoing medical innovation are considered. This process, designed to balance societal and individual needs (which can potentially conflict), must be governed by transparency and full disclosure; in addition, it must be informed by the views of all the stakeholders. The following opportunities should be considered in the interpretation of evidence:

• Syntheses and interpretations of evidence should involve clinical and methodological experts, consider all potentially useful evidence, and use methods that are reproducible by others.

• The healthcare product industry can become more active in informing clinicians, payers, and patients about the proper interpretation and the limitations of various types of evidence. This would include educating the public, policy makers, and physicians about the uncertainties related to making decisions based on the statistical outcomes of studies with (potentially) nonrepresentative populations.

• The industry can assist with the development of best practice standards for evidence integration to address the complexity of clinical decisions and evaluating the trade-offs of different study designs.

• All stakeholders can participate in the initiation of a research agenda to conduct research on the interpretation and application of evidence in healthcare decisions and the actual practice of medicine.

• All stakeholders can also participate in an exploration of the importance of patient-consumer inclusion in the development, translation, and dissemination of evidence for healthcare decision making.

The rate of translation of the available evidence into clinical practice is slow, and the translation of evidence is often challenged by gaps in the evidence. New evidence does not always translate into new behaviors among physicians and patients, in part because of the volume of new evidence, patient nonadherence, gaps in care, and a lack of decision support tools. The use of evidence in health policy decision making, coverage and reimbursement decisions, and patient care decisions requires different approaches to evaluating the evidence. All stakeholders must have a better understanding of the means of application of the evidence if they are to incorporate the evidence into their own decision-making processes. Finally, the practice of evidence-based medicine must allow the diffusion of innovation in medical practice.

There are important opportunities to improve the application of evidence in medical practice and decision making that focus on the creation of the specific evidence required for decision making, effective communication of the clinical action that is needed, and systems that make the information available and easy to implement at the point of care as well as for policy and population management. These opportunities include:

• Development and implementation of a research agenda to improve the creation and the translation of evidence-based guidelines into clinical practice. There is a need to identify the areas in which real-

world evidence on the therapeutic use of medications needs to be generated to support the data needs of medical societies responsible for the creation of clinical practice guidelines. Again, this process should consider a wide variety of evidence, scientific reviews of the effectiveness of a product, consultations with many stakeholders, and the specific needs of the populations to be served (and should perhaps include benefits that are not included as part of most effectiveness evaluations, e.g., improved adherence or improved tolerability). These discussions will also need to address potential conflicts of interest. In addition, it is important that these guidelines be updated in a timely fashion so that clinical practice is not locked into being based on earlier standards, as the performance metrics may badly lag behind the state of the art.

• Development of general principles for applying evidence to foster predictable decisions, such as for coverage and reimbursement. The various healthcare system stakeholders should partner with payers to develop processes for setting coverage and payment policies that are seen to be open, transparent, and trustworthy in their consideration of a wide variety of evidence, including a scientific review of the effectiveness of a product, consultation with many stakeholders, and the specific needs of the populations served.

• Support for educational initiatives for physicians and other providers in applying evidence to patient care decisions, including consumer-based decision making. The funders of medical education should partner with continuing education providers to focus education on evidence-based care. Pharmaceutical industry promotions can focus on areas that support the recommendations of clinical practice guidelines and evidence-based care. Collaborations with healthcare systems, academia, and health information technology organizations will be required to accelerate the use of health information technology in health care and to bring evidence-based information to point-of-care decision making. Additionally, methods of communicating evidence to consumers must be explored to better assist consumers with their healthcare decision making.

• Promotion of a learning environment in healthcare practice. The various healthcare sectors can work collaboratively to promote a learning environment in healthcare practice in which healthcare practitioners apply the best available evidence, measure quality and outcomes, and recirculate the new evidence so generated to inform the practice of care. The healthcare manufacturing sector can support this learning environment by supporting quality improvement programs that measure medication use against defined quality indi-

cators (that go beyond mere process) and informing healthcare plans and providers to ensure the appropriate use of medications and the quality of care provided by medications.

• Promotion of national quality improvement of medication use. A partnership with national quality improvement groups and quality improvement personnel in health plans can be created to obtain agreement on a strategy for measuring performance at the health plan, physician, and pharmacist levels; to collect and aggregate data in the least burdensome way; and to report meaningful information to consumers, physicians, and stakeholders to inform their choices and improve outcomes. This may include the development of performance measures for medication use, such as those endorsed by the National Quality Forum, and the design and approach to the measurement and reporting of results.

The healthcare product industry can help advance the appropriate use of evidence; however, no one sector can move very far forward alone. With that in mind, some ongoing challenges remain as the various healthcare sectors try to align their efforts toward achievement of a common goal while focusing on areas in which the greatest gains in the quality of patient care and the efficiency of the healthcare system can be achieved.

The entire spectrum of healthcare delivery and overall treatment must be considered in order to achieve the greatest efficiency and impact. A narrow focus on any single portion of healthcare delivery, “because it’s where the data is,” will fail to produce the greatest savings or impact on quality of care and is likely to hamper innovation in that sector.

Rather than a narrow focus, there needs to be an explicit process to prioritize these expanding areas of research. The goal of the prioritization should be to identify the areas of greatest improvements in quality and impact on the total healthcare system and should ensure that the most valuable questions are being considered from the perspective of all healthcare stakeholders. The goals defined in Section 1013, Priority Topics for Research, of the Medicare Prescription Drug, Improvement, and Modernization Act of 2003 offer an opportunity but also highlight the ongoing uncertainty of how the gaps in research that can inform decision making and analysis can be identified and addressed. The current input process that

allows stakeholders to question developments in the AHRQ Comparative Effectiveness Program provides a good starting point in this regard.

If there is no standardization in research, in particular, guidelines for methodology selection and conduct of the chosen study, it will be impossible to prioritize resources and build a credible, widely trusted process for developing and appropriately using new forms of evidence.

Different stakeholders have a wide range of needs for various types of evidence as well as a similar range in how they interpret evidence. For example, it appears that payer organizations use the Academy of Managed Care Pharmacists standard dossiers of evidence in a variety of ways, with some requiring more and different kinds of evidence and others making little use of any evidence that a manufacturer provides. Given resource limitations, it will not be possible for the healthcare product sector to generate an evidence base that will meet a limitless range of needs. The harmonization of evidence standards will facilitate the understanding and use of data. Although a single set of standards will not likely be applied to every study and decision, appropriate and predictable variations in standards should be allowed for different purposes. Nevertheless, those standards should be consistent with one another and not require expensive, nonproductive parallel efforts, for example, one set of requirements for WellPoint, another for Aetna, and yet another for the Centers for Medicare and Medicaid Services. Decisions about how these issues should be addressed must reflect input from the full spectrum of stakeholders, each of which needs to consider the importance of the input from the other stakeholders.

Once evidence is generated, a more comprehensive standard is needed to ensure that the evidence is appropriately applied. There is a lack of transparency in how interpreters of evidence (including physicians, payers, patients, and policy makers) use the evidence.

Transparency in this process is necessary to ensure that it is widely accepted and that the decisions are subject to quality checks through public scrutiny. In addition, the establishment of principles governing how evidence is integrated into coverage decisions will facilitate this process. The interpreters of the evidence also have various competencies and needs that they bring to understanding evidence, further complicating what evidence will bring to improving value. Finally, the development of appropriate methodologies for communicating comparative effectiveness research to other stakeholders in the healthcare system, especially patients, is a chal-

lenge (e.g., the current activities at AHRQ [Agency for Healthcare Research and Quality, 2007a] and Consumers Union Best Buy Drugs [Consumer Reports Health, 2007]) and represents an area in which the healthcare product sector can make a significant contribution.

The use of health information technology has a great potential to make health care more transparent and evidence based and to provide clinicians with up-to-date decision-making support when treating their patients. In addition, the use of electronic data sources could accelerate the development of new evidence in real-world settings.

The establishment of a viable, interoperable health information technology infrastructure may, in fact, be a prerequisite for implementation of the learning healthcare system envisioned in the 2020 vision. There are enormous challenges to the implementation of a national health information technology architecture, and addressing those challenges is outside the scope of this chapter. Nonetheless, solutions to those challenges may be critical.

The healthcare system of 2020 will encounter entirely new challenges as a result of the targeting of therapy for patients for whom a genuine benefit or the avoidance of harm can be achieved on the basis of molecular diagnostic testing.

Assessment of the clinical validity, reproducibility, and utility of molecular diagnostic tests, in conjunction with the demonstration of the effectiveness of an associated treatments, poses new challenges in the development of evidence but also holds great promise for making evidence-based medicine more patient-centric. By 2020 the healthcare system will likely be dealing with not only therapies for existing conditions but also preventive therapies and associated diagnostics to effectively avoid (or delay) the onset or escalation of disease. Standards of evidence for such preventive approaches have not yet been considered.

The nation’s capacity for clinical research is currently inadequate to provide the information needed for regulatory approval and effectiveness research goals. Randomized controlled trials, on which the healthcare system currently depends, take too much time and are too expensive, and their findings are not always generalizable.

A key to advancing progress in meeting the near-term need for expanded requirements to generate evidence is the development of a sustainable evidence development and evidence application capacity for the private and public sectors involved in these efforts. The way forward will require collaboration among the various healthcare sectors to carefully consider research priorities and methods and the policies involved in decision making and the application of evidence to achieve sound medical care.

The economics of innovation in product development are such that the additional expense to the industry of producing evidence beyond that currently required by regulatory agencies is not sustainable from a financial perspective, especially if payers do not value the incremental benefits of new products. If the standards for evidence development are not realistic, the incentives provided to create genuine innovation could be substantially reduced, particularly for innovations for the treatment of uncommon conditions. This, in turn, could change the economics of innovation and affect the business models.

Strengthening the national capacity for additional effectiveness research will need to be addressed through the establishment of definitions for evidence development standards for different types of decisions. Ultimately, the process of generating and applying the best evidence will be the natural and seamless components of medical care itself.

Several concepts embedded in the healthcare product sector’s recommendations for collaboration will contribute in the near term to efforts to create this capacity to generate and apply evidence:

• All stakeholders should be involved in establishing research priorities and setting standards for evidence development.

• Transparency and consistency (of standards) in processes, data requirements, methods of assessment and interpretation, and criteria for decision making are needed.

• Health policy decisions should be based on all of the available evidence (albeit, appropriately weighted), without arbitrary rules on the inclusion or exclusion of certain types of evidence. Several research designs should be explored to determine whether the findings of that research are adequate for healthcare decision making, including practical clinical trials, clinical registries, observational studies, and model development.

• A balance between population-based evidence and the evidence needs for an episode of care (i.e., the individual’s interaction with his or her physician) is critical. Evaluations should consider the evidence on variations in individual responses to a treatment to ensure that an adequate variety of treatment choices are available to meet individual patient needs.

• Until (and if) consensus is reached on standards of evidence interpretation, the interpretation of evidence should be open to appeal through an independent, transparent, and facile process.

• The assessment of the safety and efficacy of a treatment should be separate from considerations of the clinical effectiveness of a treatment.

• The process of health technology assessment should be linked to implementation of the health technology in healthcare systems such that positive assessments then lead to the rapid implementation of decisions and the appropriate adoption of new technologies.

The healthcare product industry, as the supplier of healthcare products, can add far more value to healthcare delivery and the appropriate use of medications and devices than has yet been realized. The sector has demonstrated broad experience in the development of evidence on the safety and efficacy of medications and devices and has been involved in the promotion of the safe and effective use of therapeutics.

The tasks of developing and translating evidence into practice are the core capabilities of the industry. The scientists, clinicians, and technologists that the sector employs have broad experience and considerable knowledge in designing randomized controlled trials and are gaining experience in the conduct of practical clinical trials, observational studies, and registries. Additionally, many industry employees have expertise in statistical analysis, database aggregation and synthesis, and the communication of results to healthcare professionals and patients. It is important to recognize the dynamic nature of evidence development, beginning with the conduct of randomized controlled trials for regulatory purposes and continuing through the collection of data from real-world experiences in the postmarketing phase. Collaboration across this continuum will be a future requirement as the evidence base for the safe and effective use of a therapeutic product is established and monitored.

Unfortunately, a general misalignment of incentives in health care has promoted inefficiencies that have led to a lack of trust between the industry, the public, and the other healthcare sectors, including payers and providers. Although the industry is always concerned for patient health, competition within the industry has often relied not only on the productivity of its R&D pipelines but also on marketing. At the same time, the reputation of the industry has fallen on the basis of public concerns about the accuracy and the transparency of the evidence that the industry provides and the way in which companies employ evidence when they market their products to physicians and consumers.

The industry has made important strides in establishing healthcare compliance guidelines and a code of ethics in the promotion of products to physicians and consumers. Additionally, the transparency of clinical trials data has evolved with the implementation of the Clinical-Trials.Gov website and participation by the industry in communicating the results of clinical trials on the safety and efficacy of medications and devices.

In the past, healthcare sectors have missed opportunities to add value in healthcare delivery because of a lack of cooperation and partnering. In supporting the emerging evidence-based healthcare model, collaboration with the healthcare products industry, which is designed to improve overall patient value by using the right therapeutic products at the right time for the right patient, will be a big step forward to promoting evidence-based quality care.

Collaboration among Roundtable members will be critical in achieving the goal that, by the year 2020, 90 percent of clinical decisions will be supported by accurate, timely, and up-to-date clinical information, and will reflect the best available evidence.

The nature of the barriers and possible solutions and priorities for action will be addressed by open discussions that focus on key areas of collaboration and a program of activities to address them. The priority areas for collaboration are described in the next section.

• The development of standards of evidence for product approval, health policy decision making, and patient care decisions will allow consensus on the types of evidence that are best suited to inform various kinds of healthcare decisions. The role of healthcare product developers in achieving this goal will be to participate with other healthcare sectors, in particular, patients, healthcare delivery organizations, clinical research and evaluators, insurers, and regulators, in discussing the total cost of care and the overall value derived from greater research on evidence-based medicine. These discussions will result in a more prioritized approach to evidence-based research.

• To develop evidence that incorporates individual patient needs, a balance between generating broad-based evidence for safety and efficacy in a controlled population and generating evidence with a

high degree of relevance to particular individuals and subgroups in the real world is required. To help reach this goal, healthcare product developers and partners from healthcare delivery organizations, clinical researchers, and the insurer sector should engage in discussions on evidence requirements. These discussions should focus on the healthcare policy point of view and that of the consumer-patient.

• Collaborative dialogue is needed to establish principles governing how evidence is integrated into coverage decisions, especially when these decisions include the denial of access to medicines by disenfranchised populations or the swift introduction of a new technology with evidence of a superior benefit. Proposed sector partners include consumer-patient groups, healthcare delivery organizations, clinical investigators and evaluators, and insurers. From these discussions, healthcare product developers can become proactive, informing clinicians, payers, and patients about the proper interpretation and the limitations of the evidence generated.

• Education about the uncertainties of decision making by the use of evidence from studies conducted with nonrepresentative populations is another area for collaborative work. Healthcare product developers can educate the public, policy makers, and physicians about the residual uncertainties that any individual making decisions on the basis of statistical outcomes from studies conducted with potentially nonrepresentative populations may have. The partners needed in this effort include patients-consumers, healthcare delivery organizations, clinical investigators-evaluators, and insurers.

• The development of best practice standards for evidence interpretation is needed to inform practice, measure quality, and improve how evidence is integrated into coverage decisions. In partnership with consumers-patients, healthcare delivery organizations, clinical investigators-evaluators, and insurers, healthcare product developers can assist with the development of best practice standards for evidence integration by initiating a transparent research agenda on the basis of the interpretation and application of the evidence.

• Understanding the proper role of evidence in healthcare decision making at the patient care level versus the proper role of evidence at the health policy level requires collaborative work by product developers, consumers-patients, healthcare delivery organizations, clinical investigators, and insurers. The industry can contribute by

engaging in a research agenda around policy-level decision-making and its impact on patient-level care.

• The development of a process for setting coverage and payment policies that are open, transparent, and trustworthy as a result of the consideration of a wide range of relevant evidence can be achieved through industry collaboration with healthcare professionals, healthcare delivery organizations, and insurers. Work might include the development of appropriate policies, and product developers could provide relevant research, whenever applicable.

• The development and implementation of an agenda for research that provides real-world data about medications and the specific needs of populations that inform the creation of clinical practice guidelines could be performed in collaboration with healthcare professionals, researchers, healthcare delivery organizations, and insurers. The developers of healthcare products could gain a better understanding of the real-world data requirements of the developers of clinical practice guidelines.

• Refining methods of communicating evidence to consumers to assist them with their decision making is another opportunity for collaboration. Relevant partners would include healthcare professionals and consumers-patients. The industry can contribute knowledge about consumer behaviors relevant to medication use.

• The development and implementation of an agenda of research on the systems changes and behavioral approaches needed to improve the translation of evidence-based guidelines into clinical practice and factors influencing adherence to regimens is another important area for improving the application of evidence. To collaborative discussions and work involving healthcare professionals, healthcare delivery organizations, and insurers, healthcare product developers can contribute knowledge about the behaviors affecting adherence to guidelines and standards of medical practice.

• Healthcare product developers are also positioned to help support quality improvement programs that measure medication use against defined quality indicators and inform health plans and healthcare providers to ensure the appropriate use of medications and the quality of care provided by medications. Overall, this will require partnerships with national quality improvement groups and quality improvement personnel in health plans and healthcare systems to develop quality measures for medical practice that reinforce the appropriate use of evidence. The industry can engage with quality

measure developers to focus on key medication use indicators of importance in guiding medication use. Additionally, measurement of medication use indicators in different types of studies can be addressed. Work might include collaboration with patients and consumer groups, healthcare delivery organizations, and insurers.

Advanced Medical Technology Association. 2007. Medical technology innovation process. http://www.advamed.org/MemberPortal/About/NewsRoom/MediaKits/medicaltechnologyinnovation.htm (accessed May 12, 2008).

Agency for Healthcare Research and Quality. 2007a. Effective health care home. http://effectivehealthcare.ahrq.gov/ (accessed May 12, 2008).

———. 2007b. Evidence-based practice program. http://www.ahrq.gov/clinic/epcix.htm (accessed May 12, 2008).

BlueCross BlueShield Association. 2007. Technology evaluation center. http://www.bcbs.com/betterknowledge/tec/ (accessed May 12, 2008).

Cochrane Collaboration. 2007. The Cochrane Collaboration. http://www.cochrane.org/ (accessed May 12, 2008).

Consumer Reports Health. 2007. Best buy drugs. http://www.crbestbuydrugs.org/ (accessed May 12, 2008).

DiMasi, J. A., R. W. Hansen, and H. G. Grabowski. 2003. The price of innovation: New estimates of drug development costs. Journal of Health Economics 22:151-185.

ECRI Institute. 2007. ECRI Institute website. http://www.ecri.org/Pages/default.aspx (accessed May 12, 2008).

Kravitz, R. L., N. Duan, and J. Braslow. 2004. Evidence-based medicine, heterogeneity of treatment effects, and the trouble with averages. Milbank Quarterly 82(4):661-687.

Lomas, T., T. Culyer, and C. McCutcheon. 2005. Final report: Conceptualizing and combining evidence for health system guidance. Ottawa, Ontario, Canada: Canadian Health Services Research Foundation.

McGlynn, E., S. Asch, J. Adams, J. Keesey, J. Hicks, A. DeCristofaro, and E. Kerr. 2003. The quality of health care delivered to adults in the United States. New England Journal of Medicine 348(26):2635-2645.

National Institute for Health and Clinical Excellence. 2007. National Institute for Health and Clinical Excellence, National Health Service of the United Kingdom website. http://www.nice.org.uk/ (accessed May 12, 2008).

Oregon Health and Science University. 2007. Drug effectiveness review project. http://www.ohsu.edu/drugeffectiveness/ (accessed May 12, 2008).

Pharmaceutical Research and Manufacturers of America. 2005a. The cost of innovation. PhRMA member companies. In Pharmaceutical industry profile 2007, p. 5. http://www.phrma.org/files/Profile%202007.pdf (accessed May 12, 2008).

———. 2005b. R&D by function, PhRMA member companies, 2005. In Pharmaceutical industry profile 2007, Appendix, Table 5, p. 45. http://www.phrma.org/files/Profile%202007.pdf (accessed May 12, 2008).

———. 2005c. The R&D process: Long, complex and costly. PhRMA member companies, 2005. In Pharmaceutical industry profile 2007, p. 6. http://www.phrma.org/files/Profile%202007.pdf (accessed May 12, 2008).

This page intentionally left blank.