National Academies Press: OpenBook

Generating Evidence for Genomic Diagnostic Test Development: Workshop Summary (2011)

Chapter: 4 Overcoming Barriers for Evidence Generation

« Previous: 3 Approaches to Evidence Generation
Suggested Citation:"4 Overcoming Barriers for Evidence Generation." Institute of Medicine. 2011. Generating Evidence for Genomic Diagnostic Test Development: Workshop Summary. Washington, DC: The National Academies Press. doi: 10.17226/13133.
×

4
Overcoming Barriers for Evidence Generation

Key Points Raised by Speakers

  • Clinical research needs to balance validity with feasibility and timeliness.

  • Establishing partnerships and sharing risk among the public sec­tor, payers, and industry will allow for robust development of diagnostic tests with diverse clinical focuses.

  • Increased dialogue among stakeholders at various points in the development process could help provide alignment around needed evidence.

  • Placing a higher value on diagnostic test and marker develop­ment will create incentives to produce the type of evidence needed for decision making. Evidence Generation

BALANCING STAKEHOLDER NEEDS

The purpose of comparative effectiveness research is to provide patients, clinicians, and payers with information that is useful in making treatment and coverage decisions. Many, if not most, comparative effectiveness studies will require a conscious decision to sacrifice internal validity in order to increase generalizability, relevance, feasibility, and timeliness, said session moderator Sean Tunis of the Center for Medical Technology Policy. However, this is not something researchers are generally comfortable doing.

Suggested Citation:"4 Overcoming Barriers for Evidence Generation." Institute of Medicine. 2011. Generating Evidence for Genomic Diagnostic Test Development: Workshop Summary. Washington, DC: The National Academies Press. doi: 10.17226/13133.
×

The frameworks that have been discussed (e.g., EGAPP, TEC) are designed to maximize internal validity but are not optimal with regard to feasibility or timeliness, which are important to the diagnostics industry and to patients. There is a need for methodologies to evaluate clinical utility that can achieve an acceptable balance of these elements, Tunis said, noting that the correct balance of validity with feasibility and timeliness is not solely a methodology issue. There is also a social judgment that must be made collectively by all stakeholders regarding the acceptable level of uncertainty.

A participant said that the goal in developing the Oncotype DX test for breast cancer, as well as with tests currently in development for colon cancer and prostate cancer, was to gather evidence that would be persuasive to both clinicians and to payers. Performing RCTs for diagnostics is not a necessity, and the length of time they take to produce outcomes data would render the test obsolete. Payer support is also required to ensure patient access.

The participant identified key questions in balancing stakeholder needs, including What are the risks and ramifications of being wrong? and, How comfortable are we with those risks? The further that studies deviate from the principles of the RCTs, the more that certainty declines. To move rapid evaluation forward, new data must be evaluated systematically as they emerge, and decision makers must be willing to stop coverage when it becomes clear that a product does not work as originally thought. It was noted that this is what already happens in many systems, such as the Ontario experience that was presented.1

PUBLIC-PRIVATE PARTNERSHIPS

One organizational model that can help address issues of funding, knowledge generation, and social change in the area of data sharing is a public-private, pre-competitive research partnership, said Aled Edwards of the Structural Genomics Consortium. Pre-competitive research is knowledge-generating research where data is openly shared and not encumbered by any restrictions on its use. For genetic tests, current precompetitive research is focused on generating hypotheses.

The Structural Genomics Consortium, founded in 2004, has 250 scientists working in three laboratories located at the University of Toronto, the University of Oxford, and the Karolinska Institute. Initially focused on studying the three-dimensional protein structure of drug targets, the consortium is now also working on pre-competitive medicinal chemistry. Thirty medicinal chemists from industry partners (including GlaxoSmithKline, Novartis, Eli Lilly and Company, and Merck) are generating new

1

Discussed by Levin in Chapter 3.

Suggested Citation:"4 Overcoming Barriers for Evidence Generation." Institute of Medicine. 2011. Generating Evidence for Genomic Diagnostic Test Development: Workshop Summary. Washington, DC: The National Academies Press. doi: 10.17226/13133.
×

molecular entities that they are not patenting but rather placing into the public domain without restriction. The pharmaceutical industry, Edwards explained, has numerous potential drug candidates but can only devote resources to fully pursue the very top candidates. By pooling resources, it is possible to assess many more potential drugs and targets. Edwards said that there is now some industry and government interest in funding clinical proof-of-concept trials of public-domain compounds. Data from such trials would also be placed into the public domain without restriction. He suggested that the organizational structure that will be developed to carry out these trials could be used to superimpose genomics studies on the proof-of-concept trials so that they can be carried out at the same time.

Edwards said that the only way to develop a truly robust pipeline of genomic diagnostic tests flowing into the clinic will be to share the risk between the public sector, payers, and industry. The Structural Genomics Consortium was begun by stakeholders, including the pharmaceutical companies, the Wellcome Trust, and the Canadian government, declaring a certain area of scientific research as pre-competitive. The consortium was then given clear milestones and deliverables to reach. Academics were willing to participate because of the no-patent policy, and industry was willing to participate because it could have full access to the information while only putting in a small percentage of the funding. Edwards suggested that much of the early-stage discovery in genetic tests should be done pre-competitively and that failure should be anticipated. This will allow stratification of tests so that not everyone is focusing on the perceived high-value targets and mostly overlooking other potentially important analytes.

As one participant pointed out, the development of partnership frameworks to enable biomarker discovery and development was the subject of a July 2010, Institute of Medicine roundtable activity (IOM, 2011). At that workshop it was noted that the pharmaceutical industry has long been collecting biological specimens from clinical trials and has allowed a number of entities access to those specimens—and to the associated data—for the purposes of developing novel biomarkers and eventually tying them to drug development programs or developing them as stand-alone diagnostics. Several public–private partnerships were given as examples, including the Genomic Applications in Practice and Prevention Network (GAPPNet), Sage Bionetworks, and the biomarkers consortium that is coordinated by the Foundation for the NIH. Pharmaceutical companies left that July workshop willing to enter into collaborations to share data and biospecimens because they understood that they will recoup greater value by partnering than by retaining the information individually. Another point made at the July workshop was that biospecimens collected by the various partners do not necessarily need to be submitted to a central location. They can remain

Suggested Citation:"4 Overcoming Barriers for Evidence Generation." Institute of Medicine. 2011. Generating Evidence for Genomic Diagnostic Test Development: Workshop Summary. Washington, DC: The National Academies Press. doi: 10.17226/13133.
×

locally housed at their source as long as they are indexed in some central way so they can be located.

THE EVIDENTIARY BAR FOR CLINICAL UTILITY

Different stakeholders have somewhat different definitions for “evidence of clinical utility.” Robert Epstein of the Medco Research Institute suggested that dialogue between payers and regulators is needed to establish some consistency. Is a difference in health outcomes necessary, and how is that defined? Can a surrogate outcome be assessed, or is a hard outcome necessary? What type of study design is necessary to answer questions of clinical utility? Epstein noted that pharmaceutical product sponsors have an end-of-Phase II meeting with FDA to ensure that they are collecting the necessary data in the appropriate way as they move to the next phase. However, there are no similar meetings with the payers and other stakeholders who ultimately review and help promulgate the use of new technologies. Epstein suggested that it could be very useful to hold a similar mid-development meeting with other stakeholders before the product reaches the market.

It is also important to evaluate the criteria for what constitutes adequate evidence, Epstein said. There are other ways to get data besides RCTs. The computing power and biostatistical expertise available today can reveal many details about a population that were not possible in the past. Epstein urged that “we should begin to look at our criteria and ask ourselves [if] we can improve on them.”

A participant noted that there is much focus on the clinical utility of genomic tests because people believe treatment decisions are based on the results of the tests. But if this is the case, the participant continued, then should not clinical utility be established for all diagnostic tests, given that they all affect patient decision making? Eric Larson of the Group Health Research Institute added that it is often assumed that every new test adds value because it provides information. Genetic testing is not different from other diagnostic testing, and the current focus on genetic tests may provide an opportunity to reframe the overall diagnostics process and highlight the need for the same rigor in developing and evaluating diagnostic tests that are used in developing and evaluating therapeutic products.

DATA SOURCES

Clinical Trials

Conducting a clinical trial for a marketed product, as is done in coverage with evidence development, can be challenging. Citing prostate-specific

Suggested Citation:"4 Overcoming Barriers for Evidence Generation." Institute of Medicine. 2011. Generating Evidence for Genomic Diagnostic Test Development: Workshop Summary. Washington, DC: The National Academies Press. doi: 10.17226/13133.
×

antigen (PSA) screening for prostate cancer as an example, Dan Hayes said that so many men were already being screened that it was hard to identify participants for a randomized trial. Another difficulty is that many genetic disorders are relatively rare and, once a marker has been discovered, there are simply not enough patients needed to conduct sufficiently large trials.

Many diagnostic companies are not likely to survive in the current industry environment if they are mandated to conduct prospective trials, said Hayes. The pharmaceutical industry can better afford clinical trials, as the payoff for a new drug can be quite significant compared to research and development costs. Hayes suggested that if markers were more highly valued for their roles in preventing unnecessary treatment of patients who are unlikely to benefit and in identifying those patients who will benefit, it would create an incentive and a revenue stream for diagnostic device developers to carry out prospective trials. Many participants concurred that there is no real incentive to conduct trials on genetic and genomic tests that are not directly tied to a treatment. A participant suggested that perhaps an “Orphan Diagnostic Act” is needed to provide incentives for such tests in the same way that the Orphan Drug Act has provided incentives to develop drugs for rare diseases.

RCTs are artificial relative to real world use of products. Trials have exclusion and inclusion criteria, for example, and they limit or control the concomitant use of other drugs. In practice, anyone can use the product, including off-label use for non-tested indications. One approach that was suggested would be to combine phase III and phase IV studies, extending the traditional phase III trial to assess longer term outcomes in the postmarketing (phase IV) stage.

A participant clarified that the RCT has not really been the standard for diagnostics, which more typically come into practice through technology assessment on chains of evidence. Except for screening tests, Larson added, there are very few published RCTs of diagnostics. It is important to conduct such trials, which are very different from the randomized trials carried out for drugs, in order to make sure that the tests under consideration are actually adding value.

A participant noted that, in reality, genomic medicine is not individually personalized; rather, patients are treated based on the cohort they best fit into. This should guide how evidence is generated. Evaluating the patients and the characteristics of the test and applying it to the proper populations will drive evidence and use more than an RCT.

Coverage with Evidence Development

The health system underinvests in diagnostics, and the coverage with evidence development approach is one way of subsidizing the development

Suggested Citation:"4 Overcoming Barriers for Evidence Generation." Institute of Medicine. 2011. Generating Evidence for Genomic Diagnostic Test Development: Workshop Summary. Washington, DC: The National Academies Press. doi: 10.17226/13133.
×

of potentially high-value diagnostics, Tunis said. There are promising technologies that might merit subsidy while they are being further evaluated in order to demonstrate clinical utility. A participant added that there are, however, inherent conflicts between, on the one hand, the purposes of an insurance system and its obligations to its beneficiaries and, on the other hand, what it takes to conduct certain types of trials. There are also issues with covering a product provisionally and then withdrawing coverage. Providers may have made very significant capital investments in equipment and are not likely to give up on these investments easily. In addition, anything that adds to administrative costs is a problem and this will become even more true as health reforms and more stringent standards are implemented. Another issue is that differences in copayments can affect the randomization and blinding processes. Although observational data can be collected feasibly, there are many challenges to superimposing a clinical research structure onto an insurance structure. People are often not willing to volunteer for random selection. One approach that was suggested during the workshop would be to have payers contribute to a pool that supports trials. Employers who purchase coverage for their employees may also be willing to collaborate in and support prospective research, Epstein said.

It was suggested by one participant that the successes and limitations of the coverage with evidence development approach up to now should be evaluated. The longest history of coverage with evidence development in the United States has been with the Medicare system, and it is important to know how successful this has been.

Provisional Approval

While insurers may approve provisional coverage pending the collection of further evidence, a participant from the FDA explained that the agency does not have the legal authority to provide provisional marketing approval for products. Devices must be demonstrated to be safe and effective or else to be substantially equivalent to (i.e., as safe and effective as) an already marketed device. There is no way that provisional device approval could be done in the United States without changes to the existing device law.

In Canada, Levin explained, most of the tests that are provided as an insured service need to have been previously approved, by either Health Canada or by the provincial jurisdiction, in order to be regarded as medically necessary; the Centers for Medicare and Medicaid Services play a similar role in the United States. In Canada, if a genomic test is to be used for targeted therapy, that fact is stated in the licensing approval provided by Health Canada. A workshop participant added that because the health system and insurance companies in Canada are linked, they can fund and

Suggested Citation:"4 Overcoming Barriers for Evidence Generation." Institute of Medicine. 2011. Generating Evidence for Genomic Diagnostic Test Development: Workshop Summary. Washington, DC: The National Academies Press. doi: 10.17226/13133.
×

run very large, system-wide trials to develop evidence of utility. This is not possible in the current U.S. health-care system.

Biobanks and Retrospective Studies

While there was much discussion about RCTs, Epstein reiterated that there are other data sources worth considering which may provide a faster way to gather evidence. He suggested, for example, leveraging the data held by closed health-care systems, which have biobanks containing hundreds of thousands of DNA samples from their members. These samples are matched with the patients’ electronic health records, which are in turn matched to claims data.

Gregory Germino of the National Institute of Diabetes and Digestive and Kidney Diseases (NIDDK) agreed with others that a prospective RCT looking at different biomarkers is the gold standard for evidence, but the challenge, he said, is knowing which markers to assess at any given time, and the choice of markers may change over time. To help address this, NIDDK mandates that samples from all large-scale NIDDK-funded studies be turned over to the institute at the end of the study. These samples then become publicly available. Since 2003 NIDDK has collected approximately 73,000 independent DNA samples (with or without cell lines, depending on the nature of the samples) and over 4 million different biospecimens, with the intent of facilitating prospective–retrospective controlled trials. There are no intellectual property issues, and innovators are free to market products that they discover and develop using these samples. Germino added that having a centralized, quality-controlled, and quality-assured repository helps ensure the stability of the samples. NIDDK has found that the research community, after some initial trepidation, has really embraced this repository system.

There are some challenges to conducting studies using biobank samples, Germino noted. Studies may not have been large enough, or may not have enough samples of any given subset, to assess a marker of interest with enough statistical power to draw a strong conclusion based on the study set. A second issue is cost. Two to five percent of the NIDDK clinical trial budget is devoted to the repositories, Germino said, and, while this may seem small, it is actually a substantial draw on the budget. This may also be a factor, he suggested, in why many other institutes have not incorporated biobanks into their study designs.

A question was asked about how applicable the NIDDK biobank approach is to other NIH institutes. Germino responded that the National Heart, Lung, and Blood Institute (NHLBI) has begun a data and biospecimen repository, but there is currently no mechanism to implement a trans-NIH repository.

Suggested Citation:"4 Overcoming Barriers for Evidence Generation." Institute of Medicine. 2011. Generating Evidence for Genomic Diagnostic Test Development: Workshop Summary. Washington, DC: The National Academies Press. doi: 10.17226/13133.
×

Not knowing where the biobanks are is certainly a barrier to progress, Germino said. Even within NIH, institutes may not know what sample repositories other institutes have. He suggested that ClinicalTrials.gov should require trial sponsors to identify biobank samples that are linked to their clinical study when they register a trial.

A participant noted that the NCI does not own the specimens from its cooperative trials and has faced some challenges in gaining access to some of the samples. Each cooperative trial group stewards its own specimen bank with NCI oversight, Hayes responded. All specimens from one trial go to a single bank. Requests for sample access are evaluated by peer-review committees composed of study investigators, and there are standardized policies and procedures on how to collect and store specimens. A participant noted that, historically, NCI has provided little support for the collection, maintenance, and distribution of specimens. This expense is often overlooked. Once a request for samples from an outside group is approved, there is no support provided for retrieving all of the specimens, packing them up, and assembling the annotation, data, and statistics. The potential treasure in biobanking clinical trial samples will not be fully realized unless there is sufficient support for acquiring it.

Larson supported merging biobanks with electronic medical records and cited as an example the Electronic Medical Records and Genomics (eMERGE) network organized by the National Human Genome Research Institute (NHGRI), which was designed to facilitate genome-wide association studies (GWASs) in participants from whom phenotypic and environmental data are available through electronic medical records. The HMO Research Network also has a number of sites that have biobanks, such as the Marshfield Clinic in Wisconsin, which has a large population-based biobank. Larson mentioned an inventory of the network biobanks that was expected to be available in December 2010.

A number of participants pointed out that the samples in repositories are precious resources. The model is collaborative while the studies are ongoing and more custodial after they are finished, with precautions in place to allow for the generation of useful information. Germino suggested that one approach to stewardship of specimens is to issue a program announcement and have researchers submit applications to request the samples. A study section would review the applications to ensure that the questions being asked can be addressed with the study design proposed and with the number of samples being requested.

Consent

When done prospectively, it is possible to secure consent for potential future studies that have not yet been envisioned. There are, however,

Suggested Citation:"4 Overcoming Barriers for Evidence Generation." Institute of Medicine. 2011. Generating Evidence for Genomic Diagnostic Test Development: Workshop Summary. Washington, DC: The National Academies Press. doi: 10.17226/13133.
×

numerous archival specimens for which consent was not obtained, and a question was raised regarding retrospectively obtaining consent.

Germino said that some sample sets that NIDDK now has from outside institutions were not originally collected with the intention to later transfer the samples to NIDDK for safekeeping and distribution. In those cases, the investigators met with their institutional review boards to discuss whether the original consents would allow subsequent use by NIDDK. These specimens have had identifiers removed so that they are anonymous, and while they are linked to the clinical outcomes data, there are no Health Insurance Portability and Accountability Act identifiers. In every case so far there have been no barriers to having the samples transferred to NIDDK, but Germino noted that this is not the universal experience. Larson added that work from the eMERGE project has found that people who consent to research are likely to reconsent when asked. Forming a partnership with the population that is contributing samples is essential, he said, and “goes a long way to solving these issues around depositing data in large public data sets.”

The Learning Health-Care System

Participants agreed that there is a disconnect between the health system enterprise and the research and industry enterprise. Larson endorsed the Institute of Medicine concept of the “learning health-care system,” which incorporates the generation and application of evidence into the patient care system itself.2 Conducting research within existing clinical systems, he said, would presumably be cheaper because the data are already being collected. Such data collected within the context of care are also more likely to be generalizable and applicable to care.

Observational Studies

Epstein drew attention to the Bradford Hill Criteria used in epidemiology to help establish causal relationships, and he suggested that this approach might be helpful in making the field comfortable with the weight of genomic evidence from multiple observational studies in the absence of a RCT.

2

See http://www.iom.edu/Activities/Quality/LearningHealthCare.aspx for further information about an ongoing IOM consensus study on the learning healthcare system. See also The Learning Healthcare System: Workshop Summary available at http://www.nap.edu/catalog.php record_id=11903.

Suggested Citation:"4 Overcoming Barriers for Evidence Generation." Institute of Medicine. 2011. Generating Evidence for Genomic Diagnostic Test Development: Workshop Summary. Washington, DC: The National Academies Press. doi: 10.17226/13133.
×

This page intentionally left blank.

Suggested Citation:"4 Overcoming Barriers for Evidence Generation." Institute of Medicine. 2011. Generating Evidence for Genomic Diagnostic Test Development: Workshop Summary. Washington, DC: The National Academies Press. doi: 10.17226/13133.
×
Page 43
Suggested Citation:"4 Overcoming Barriers for Evidence Generation." Institute of Medicine. 2011. Generating Evidence for Genomic Diagnostic Test Development: Workshop Summary. Washington, DC: The National Academies Press. doi: 10.17226/13133.
×
Page 44
Suggested Citation:"4 Overcoming Barriers for Evidence Generation." Institute of Medicine. 2011. Generating Evidence for Genomic Diagnostic Test Development: Workshop Summary. Washington, DC: The National Academies Press. doi: 10.17226/13133.
×
Page 45
Suggested Citation:"4 Overcoming Barriers for Evidence Generation." Institute of Medicine. 2011. Generating Evidence for Genomic Diagnostic Test Development: Workshop Summary. Washington, DC: The National Academies Press. doi: 10.17226/13133.
×
Page 46
Suggested Citation:"4 Overcoming Barriers for Evidence Generation." Institute of Medicine. 2011. Generating Evidence for Genomic Diagnostic Test Development: Workshop Summary. Washington, DC: The National Academies Press. doi: 10.17226/13133.
×
Page 47
Suggested Citation:"4 Overcoming Barriers for Evidence Generation." Institute of Medicine. 2011. Generating Evidence for Genomic Diagnostic Test Development: Workshop Summary. Washington, DC: The National Academies Press. doi: 10.17226/13133.
×
Page 48
Suggested Citation:"4 Overcoming Barriers for Evidence Generation." Institute of Medicine. 2011. Generating Evidence for Genomic Diagnostic Test Development: Workshop Summary. Washington, DC: The National Academies Press. doi: 10.17226/13133.
×
Page 49
Suggested Citation:"4 Overcoming Barriers for Evidence Generation." Institute of Medicine. 2011. Generating Evidence for Genomic Diagnostic Test Development: Workshop Summary. Washington, DC: The National Academies Press. doi: 10.17226/13133.
×
Page 50
Suggested Citation:"4 Overcoming Barriers for Evidence Generation." Institute of Medicine. 2011. Generating Evidence for Genomic Diagnostic Test Development: Workshop Summary. Washington, DC: The National Academies Press. doi: 10.17226/13133.
×
Page 51
Suggested Citation:"4 Overcoming Barriers for Evidence Generation." Institute of Medicine. 2011. Generating Evidence for Genomic Diagnostic Test Development: Workshop Summary. Washington, DC: The National Academies Press. doi: 10.17226/13133.
×
Page 52
Next: 5 Considerations Moving Forward »
Generating Evidence for Genomic Diagnostic Test Development: Workshop Summary Get This Book
×
Buy Paperback | $29.00 Buy Ebook | $23.99
MyNAP members save 10% online.
Login or Register to save!
Download Free PDF

Ten years after the sequencing of the human genome, scientists have developed genetic tests that can predict a person's response to certain drugs, estimate the risk of developing Alzheimer's disease, and make other predictions based on known links between genes and diseases. However, genetic tests have yet to become a routine part of medical care, in part because there is not enough evidence to show they help improve patients' health.

The Institute of Medicine (IOM) held a workshop to explore how researchers can gather better evidence more efficiently on the clinical utility of genetic tests. Generating Evidence for Genomic Diagnostic Test Development compares the evidence that is required for decisions regarding clearance, use, and reimbursement, to the evidence that is currently generated. The report also addresses innovative and efficient ways to generate high-quality evidence, as well as barriers to generating this evidence.

Generating Evidence for Genomic Diagnostic Test Development contains information that will be of great value to regulators and policymakers, payers, health-care providers, researchers, funders, and evidence-based review groups.

  1. ×

    Welcome to OpenBook!

    You're looking at OpenBook, NAP.edu's online reading room since 1999. Based on feedback from you, our users, we've made some improvements that make it easier than ever to read thousands of publications on our website.

    Do you want to take a quick tour of the OpenBook's features?

    No Thanks Take a Tour »
  2. ×

    Show this book's table of contents, where you can jump to any chapter by name.

    « Back Next »
  3. ×

    ...or use these buttons to go back to the previous chapter or skip to the next one.

    « Back Next »
  4. ×

    Jump up to the previous page or down to the next one. Also, you can type in a page number and press Enter to go directly to that page in the book.

    « Back Next »
  5. ×

    Switch between the Original Pages, where you can read the report as it appeared in print, and Text Pages for the web version, where you can highlight and search the text.

    « Back Next »
  6. ×

    To search the entire text of this book, type in your search term here and press Enter.

    « Back Next »
  7. ×

    Share a link to this book page on your preferred social network or via email.

    « Back Next »
  8. ×

    View our suggested citation for this chapter.

    « Back Next »
  9. ×

    Ready to take your reading offline? Click here to buy this book in print or download it as a free PDF, if available.

    « Back Next »
Stay Connected!