Overcoming Barriers for Evidence Generation

Key Points Raised by Speakers

  • Clinical research needs to balance validity with feasibility and timeliness.

  • Establishing partnerships and sharing risk among the public sec­tor, payers, and industry will allow for robust development of diagnostic tests with diverse clinical focuses.

  • Increased dialogue among stakeholders at various points in the development process could help provide alignment around needed evidence.

  • Placing a higher value on diagnostic test and marker develop­ment will create incentives to produce the type of evidence needed for decision making. Evidence Generation


The purpose of comparative effectiveness research is to provide patients, clinicians, and payers with information that is useful in making treatment and coverage decisions. Many, if not most, comparative effectiveness studies will require a conscious decision to sacrifice internal validity in order to increase generalizability, relevance, feasibility, and timeliness, said session moderator Sean Tunis of the Center for Medical Technology Policy. However, this is not something researchers are generally comfortable doing.

The National Academies | 500 Fifth St. N.W. | Washington, D.C. 20001
Copyright © National Academy of Sciences. All rights reserved.
Terms of Use and Privacy Statement

Below are the first 10 and last 10 pages of uncorrected machine-read text (when available) of this chapter, followed by the top 30 algorithmically extracted key phrases from the chapter as a whole.
Intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text on the opening pages of each chapter. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.

Do not use for reproduction, copying, pasting, or reading; exclusively for search engines.

OCR for page 43
4 Overcoming Barriers for Evidence Generation Key Points Raised by Speakers • Clinical research needs to balance validity with feasibility and timeliness. • Establishing partnerships and sharing risk among the public sec- tor, payers, and industry will allow for robust development of diagnostic tests with diverse clinical focuses. • Increased dialogue among stakeholders at various points in the development process could help provide alignment around needed evidence. • Placing a higher value on diagnostic test and marker develop- ment will create incentives to produce the type of evidence needed for decision making. BALANCING STAKEHOLDER NEEDS The purpose of comparative effectiveness research is to provide patients, clinicians, and payers with information that is useful in making treatment and coverage decisions. Many, if not most, comparative effectiveness stud- ies will require a conscious decision to sacrifice internal validity in order to increase generalizability, relevance, feasibility, and timeliness, said ses- sion moderator Sean Tunis of the Center for Medical Technology Policy. However, this is not something researchers are generally comfortable doing. 43

OCR for page 43
44 GENOMIC DIAGNOSTIC TEST DEVELOPMENT The frameworks that have been discussed (e.g., EGAPP, TEC) are designed to maximize internal validity but are not optimal with regard to feasibil- ity or timeliness, which are important to the diagnostics industry and to patients. There is a need for methodologies to evaluate clinical utility that can achieve an acceptable balance of these elements, Tunis said, noting that the correct balance of validity with feasibility and timeliness is not solely a methodology issue. There is also a social judgment that must be made collectively by all stakeholders regarding the acceptable level of uncertainty. A participant said that the goal in developing the Oncotype DX test for breast cancer, as well as with tests currently in development for colon cancer and prostate cancer, was to gather evidence that would be persuasive to both clinicians and to payers. Performing RCTs for diagnostics is not a necessity, and the length of time they take to produce outcomes data would render the test obsolete. Payer support is also required to ensure patient access. The participant identified key questions in balancing stakeholder needs, including What are the risks and ramifications of being wrong? and, How comfortable are we with those risks? The further that studies deviate from the principles of the RCTs, the more that certainty declines. To move rapid evaluation forward, new data must be evaluated systematically as they emerge, and decision makers must be willing to stop coverage when it becomes clear that a product does not work as originally thought. It was noted that this is what already happens in many systems, such as the Ontario experience that was presented.1 PUBLIC-PRIVATE PARTNERSHIPS One organizational model that can help address issues of fund- ing, knowledge generation, and social change in the area of data shar- ing is a public-private, pre-competitive research partnership, said Aled Edwards of the Structural Genomics Consortium. Pre-competitive research is knowledge-generating research where data is openly shared and not encumbered by any restrictions on its use. For genetic tests, current pre- competitive research is focused on generating hypotheses. The Structural Genomics Consortium, founded in 2004, has 250 sci- entists working in three laboratories located at the University of Toronto, the University of Oxford, and the Karolinska Institute. Initially focused on studying the three-dimensional protein structure of drug targets, the consortium is now also working on pre-competitive medicinal chemistry. Thirty medicinal chemists from industry partners (including GlaxoSmith- Kline, Novartis, Eli Lilly and Company, and Merck) are generating new 1 Discussed by Levin in Chapter 3.

OCR for page 43
45 OVERCOMING BARRIERS FOR EVIDENCE GENERATION molecular entities that they are not patenting but rather placing into the public domain without restriction. The pharmaceutical industry, Edwards explained, has numerous potential drug candidates but can only devote resources to fully pursue the very top candidates. By pooling resources, it is possible to assess many more potential drugs and targets. Edwards said that there is now some industry and government interest in funding clini- cal proof-of-concept trials of public-domain compounds. Data from such trials would also be placed into the public domain without restriction. He suggested that the organizational structure that will be developed to carry out these trials could be used to superimpose genomics studies on the proof- of-concept trials so that they can be carried out at the same time. Edwards said that the only way to develop a truly robust pipeline of genomic diagnostic tests flowing into the clinic will be to share the risk between the public sector, payers, and industry. The Structural Genomics Consortium was begun by stakeholders, including the pharmaceutical com- panies, the Wellcome Trust, and the Canadian government, declaring a cer- tain area of scientific research as pre-competitive. The consortium was then given clear milestones and deliverables to reach. Academics were willing to participate because of the no-patent policy, and industry was willing to par- ticipate because it could have full access to the information while only put- ting in a small percentage of the funding. Edwards suggested that much of the early-stage discovery in genetic tests should be done pre-competitively and that failure should be anticipated. This will allow stratification of tests so that not everyone is focusing on the perceived high-value targets and mostly overlooking other potentially important analytes. As one participant pointed out, the development of partnership frame- works to enable biomarker discovery and development was the subject of a July 2010, Institute of Medicine roundtable activity (IOM, 2011). At that workshop it was noted that the pharmaceutical industry has long been col- lecting biological specimens from clinical trials and has allowed a number of entities access to those specimens—and to the associated data—for the purposes of developing novel biomarkers and eventually tying them to drug development programs or developing them as stand-alone diagnostics. Several public–private partnerships were given as examples, including the Genomic Applications in Practice and Prevention Network (GAPPNet), Sage Bionetworks, and the biomarkers consortium that is coordinated by the Foundation for the NIH. Pharmaceutical companies left that July work- shop willing to enter into collaborations to share data and biospecimens because they understood that they will recoup greater value by partnering than by retaining the information individually. Another point made at the July workshop was that biospecimens collected by the various partners do not necessarily need to be submitted to a central location. They can remain

OCR for page 43
46 GENOMIC DIAGNOSTIC TEST DEVELOPMENT locally housed at their source as long as they are indexed in some central way so they can be located. THE EVIDENTIARY BAR FOR CLINICAL UTILITY Different stakeholders have somewhat different definitions for “evi- dence of clinical utility.” Robert Epstein of the Medco Research Institute suggested that dialogue between payers and regulators is needed to establish some consistency. Is a difference in health outcomes necessary, and how is that defined? Can a surrogate outcome be assessed, or is a hard outcome necessary? What type of study design is necessary to answer questions of clinical utility? Epstein noted that pharmaceutical product sponsors have an end-of-Phase II meeting with FDA to ensure that they are collect- ing the necessary data in the appropriate way as they move to the next phase. However, there are no similar meetings with the payers and other stakeholders who ultimately review and help promulgate the use of new technologies. Epstein suggested that it could be very useful to hold a simi- lar mid-development meeting with other stakeholders before the product reaches the market. It is also important to evaluate the criteria for what constitutes ade- quate evidence, Epstein said. There are other ways to get data besides RCTs. The computing power and biostatistical expertise available today can reveal many details about a population that were not possible in the past. Epstein urged that “we should begin to look at our criteria and ask ourselves [if] we can improve on them.” A participant noted that there is much focus on the clinical utility of genomic tests because people believe treatment decisions are based on the results of the tests. But if this is the case, the participant continued, then should not clinical utility be established for all diagnostic tests, given that they all affect patient decision making? Eric Larson of the Group Health Research Institute added that it is often assumed that every new test adds value because it provides information. Genetic testing is not different from other diagnostic testing, and the current focus on genetic tests may provide an opportunity to reframe the overall diagnostics process and highlight the need for the same rigor in developing and evaluating diagnostic tests that are used in developing and evaluating therapeutic products. DATA SOURCES Clinical Trials Conducting a clinical trial for a marketed product, as is done in cover- age with evidence development, can be challenging. Citing prostate-specific

OCR for page 43
47 OVERCOMING BARRIERS FOR EVIDENCE GENERATION antigen (PSA) screening for prostate cancer as an example, Dan Hayes said that so many men were already being screened that it was hard to identify participants for a randomized trial. Another difficulty is that many genetic disorders are relatively rare and, once a marker has been discovered, there are simply not enough patients needed to conduct sufficiently large trials. Many diagnostic companies are not likely to survive in the current industry environment if they are mandated to conduct prospective trials, said Hayes. The pharmaceutical industry can better afford clinical trials, as the payoff for a new drug can be quite significant compared to research and development costs. Hayes suggested that if markers were more highly valued for their roles in preventing unnecessary treatment of patients who are unlikely to benefit and in identifying those patients who will benefit, it would create an incentive and a revenue stream for diagnostic device developers to carry out prospective trials. Many participants concurred that there is no real incentive to conduct trials on genetic and genomic tests that are not directly tied to a treatment. A participant suggested that perhaps an “Orphan Diagnostic Act” is needed to provide incentives for such tests in the same way that the Orphan Drug Act has provided incentives to develop drugs for rare diseases. RCTs are artificial relative to real world use of products. Trials have exclusion and inclusion criteria, for example, and they limit or control the concomitant use of other drugs. In practice, anyone can use the product, including off-label use for non-tested indications. One approach that was suggested would be to combine phase III and phase IV studies, extending the traditional phase III trial to assess longer term outcomes in the post- marketing (phase IV) stage. A participant clarified that the RCT has not really been the standard for diagnostics, which more typically come into practice through technol- ogy assessment on chains of evidence. Except for screening tests, Larson added, there are very few published RCTs of diagnostics. It is important to conduct such trials, which are very different from the randomized trials car- ried out for drugs, in order to make sure that the tests under consideration are actually adding value. A participant noted that, in reality, genomic medicine is not individually personalized; rather, patients are treated based on the cohort they best fit into. This should guide how evidence is generated. Evaluating the patients and the characteristics of the test and applying it to the proper populations will drive evidence and use more than an RCT. Coverage with Evidence Development The health system underinvests in diagnostics, and the coverage with evidence development approach is one way of subsidizing the development

OCR for page 43
48 GENOMIC DIAGNOSTIC TEST DEVELOPMENT of potentially high-value diagnostics, Tunis said. There are promising tech- nologies that might merit subsidy while they are being further evaluated in order to demonstrate clinical utility. A participant added that there are, however, inherent conflicts between, on the one hand, the purposes of an insurance system and its obligations to its beneficiaries and, on the other hand, what it takes to conduct certain types of trials. There are also issues with covering a product provisionally and then withdrawing coverage. Pro- viders may have made very significant capital investments in equipment and are not likely to give up on these investments easily. In addition, anything that adds to administrative costs is a problem and this will become even more true as health reforms and more stringent standards are implemented. Another issue is that differences in copayments can affect the randomiza- tion and blinding processes. Although observational data can be collected feasibly, there are many challenges to superimposing a clinical research structure onto an insurance structure. People are often not willing to vol- unteer for random selection. One approach that was suggested during the workshop would be to have payers contribute to a pool that supports trials. Employers who purchase coverage for their employees may also be willing to collaborate in and support prospective research, Epstein said. It was suggested by one participant that the successes and limitations of the coverage with evidence development approach up to now should be evaluated. The longest history of coverage with evidence development in the United States has been with the Medicare system, and it is important to know how successful this has been. Provisional Approval While insurers may approve provisional coverage pending the collec- tion of further evidence, a participant from the FDA explained that the agency does not have the legal authority to provide provisional marketing approval for products. Devices must be demonstrated to be safe and effec- tive or else to be substantially equivalent to (i.e., as safe and effective as) an already marketed device. There is no way that provisional device approval could be done in the United States without changes to the existing device law. In Canada, Levin explained, most of the tests that are provided as an insured service need to have been previously approved, by either Health Canada or by the provincial jurisdiction, in order to be regarded as medi- cally necessary; the Centers for Medicare and Medicaid Services play a similar role in the United States. In Canada, if a genomic test is to be used for targeted therapy, that fact is stated in the licensing approval provided by Health Canada. A workshop participant added that because the health system and insurance companies in Canada are linked, they can fund and

OCR for page 43
49 OVERCOMING BARRIERS FOR EVIDENCE GENERATION run very large, system-wide trials to develop evidence of utility. This is not possible in the current U.S. health-care system. Biobanks and Retrospective Studies While there was much discussion about RCTs, Epstein reiterated that there are other data sources worth considering which may provide a faster way to gather evidence. He suggested, for example, leveraging the data held by closed health-care systems, which have biobanks containing hun- dreds of thousands of DNA samples from their members. These samples are matched with the patients’ electronic health records, which are in turn matched to claims data. Gregory Germino of the National Institute of Diabetes and Digestive and Kidney Diseases (NIDDK) agreed with others that a prospective RCT looking at different biomarkers is the gold standard for evidence, but the challenge, he said, is knowing which markers to assess at any given time, and the choice of markers may change over time. To help address this, NIDDK mandates that samples from all large-scale NIDDK-funded studies be turned over to the institute at the end of the study. These samples then become publicly available. Since 2003 NIDDK has collected approximately 73,000 independent DNA samples (with or without cell lines, depending on the nature of the samples) and over 4 million different biospecimens, with the intent of facilitating prospective–retrospective controlled trials. There are no intellectual property issues, and innovators are free to market products that they discover and develop using these samples. Germino added that having a centralized, quality-controlled, and quality-assured repository helps ensure the stability of the samples. NIDDK has found that the research community, after some initial trepidation, has really embraced this repository system. There are some challenges to conducting studies using biobank sam- ples, Germino noted. Studies may not have been large enough, or may not have enough samples of any given subset, to assess a marker of interest with enough statistical power to draw a strong conclusion based on the study set. A second issue is cost. Two to five percent of the NIDDK clinical trial budget is devoted to the repositories, Germino said, and, while this may seem small, it is actually a substantial draw on the budget. This may also be a factor, he suggested, in why many other institutes have not incorporated biobanks into their study designs. A question was asked about how applicable the NIDDK biobank approach is to other NIH institutes. Germino responded that the National Heart, Lung, and Blood Institute (NHLBI) has begun a data and biospeci- men repository, but there is currently no mechanism to implement a trans- NIH repository.

OCR for page 43
50 GENOMIC DIAGNOSTIC TEST DEVELOPMENT Not knowing where the biobanks are is certainly a barrier to progress, Germino said. Even within NIH, institutes may not know what sample repositories other institutes have. He suggested that ClinicalTrials.gov should require trial sponsors to identify biobank samples that are linked to their clinical study when they register a trial. A participant noted that the NCI does not own the specimens from its cooperative trials and has faced some challenges in gaining access to some of the samples. Each cooperative trial group stewards its own specimen bank with NCI oversight, Hayes responded. All specimens from one trial go to a single bank. Requests for sample access are evaluated by peer-review committees composed of study investigators, and there are standardized policies and procedures on how to collect and store specimens. A partici- pant noted that, historically, NCI has provided little support for the col- lection, maintenance, and distribution of specimens. This expense is often overlooked. Once a request for samples from an outside group is approved, there is no support provided for retrieving all of the specimens, packing them up, and assembling the annotation, data, and statistics. The potential treasure in biobanking clinical trial samples will not be fully realized unless there is sufficient support for acquiring it. Larson supported merging biobanks with electronic medical records and cited as an example the Electronic Medical Records and Genomics (eMERGE) network organized by the National Human Genome Research Institute (NHGRI), which was designed to facilitate genome-wide associa- tion studies (GWASs) in participants from whom phenotypic and environ- mental data are available through electronic medical records. The HMO Research Network also has a number of sites that have biobanks, such as the Marshfield Clinic in Wisconsin, which has a large population-based biobank. Larson mentioned an inventory of the network biobanks that was expected to be available in December 2010. A number of participants pointed out that the samples in repositories are precious resources. The model is collaborative while the studies are ongoing and more custodial after they are finished, with precautions in place to allow for the generation of useful information. Germino suggested that one approach to stewardship of specimens is to issue a program announcement and have researchers submit applications to request the samples. A study section would review the applications to ensure that the questions being asked can be addressed with the study design proposed and with the number of samples being requested. Consent When done prospectively, it is possible to secure consent for poten- tial future studies that have not yet been envisioned. There are, however,

OCR for page 43
51 OVERCOMING BARRIERS FOR EVIDENCE GENERATION numerous archival specimens for which consent was not obtained, and a question was raised regarding retrospectively obtaining consent. Germino said that some sample sets that NIDDK now has from outside institutions were not originally collected with the intention to later transfer the samples to NIDDK for safekeeping and distribution. In those cases, the investigators met with their institutional review boards to discuss whether the original consents would allow subsequent use by NIDDK. These speci- mens have had identifiers removed so that they are anonymous, and while they are linked to the clinical outcomes data, there are no Health Insurance Portability and Accountability Act identifiers. In every case so far there have been no barriers to having the samples transferred to NIDDK, but Germino noted that this is not the universal experience. Larson added that work from the eMERGE project has found that people who consent to research are likely to reconsent when asked. Forming a partnership with the popula- tion that is contributing samples is essential, he said, and “goes a long way to solving these issues around depositing data in large public data sets.” The Learning Health-Care System Participants agreed that there is a disconnect between the health system enterprise and the research and industry enterprise. Larson endorsed the Institute of Medicine concept of the “learning health-care system,” which incorporates the generation and application of evidence into the patient care system itself.2 Conducting research within existing clinical systems, he said, would presumably be cheaper because the data are already being col- lected. Such data collected within the context of care are also more likely to be generalizable and applicable to care. Observational Studies Epstein drew attention to the Bradford Hill Criteria used in epide- miology to help establish causal relationships, and he suggested that this approach might be helpful in making the field comfortable with the weight of genomic evidence from multiple observational studies in the absence of a RCT. 2 See http://www.iom.edu/Activities/Quality/LearningHealthCare.aspx for further informa- tion about an ongoing IOM consensus study on the learning healthcare system. See also The Learning Healthcare System: Workshop Summary available at http://www.nap.edu/catalog. php record_id=11903.

OCR for page 43