|
||||||||||||||||
|
||||||||||||||||
The following HTML text is provided to enhance online readability. Many aspects of typography translate only awkwardly to HTML. Please use the page image as the authoritative form to ensure accuracy. 2
|
||||||||||||||||
 |
 |
Page 3 |
 |
 |
 |
|
| ||||||||||||||||||||||||||||||||
|
|
|||||||||||||||||||||||||||||||
Below are the first 10 and last 10 pages of uncorrected machine-read text (when available) of this chapter, followed by the top 30 algorithmically extracted key phrases from the chapter as a whole.
Intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text on the opening pages of each chapter.
Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.
Do not use for reproduction, copying, pasting, or reading; exclusively for search engines.
OCR for page 3
Systems for Research and Evaluation for Translating Genome-Based Discoveries for Health: Workshop Summary
2
Generating Evidence for Decision Making
DOES THE TYPE OF DECISION BEING MADE INFLUENCE THE EVIDENCE NEEDED?
Steven Teutsch, M.D., M.P.H.
County of Los Angeles Department of Public Health
Decisions affecting health care must be acceptable and legitimate to the people they will affect, Teutsch began. The legitimization of health policy decisions requires prospective agreement about the evidentiary standards that will be used. This is a deliberative and inclusive process to develop an understanding of the different types of decisions to be made, and the nature and importance of the evidence that is appropriate for each. There is no simple formula or prescription for decision making. Each decision is based not only on the evidence, but also the context in which each decision is being made. Transparency of the process is also important, so that it is clear what information was used in making the decision.
Evidentiary Threshold
The translational process can be viewed as moving from gene discovery to application in a health context, to health practice, and finally to understanding the health impact (Figure 2-1). The critical step in translation is the development of an evidence-based guideline that allows the technology to move from research into clinical or public health practice. A key question
OCR for page 4
Systems for Research and Evaluation for Translating Genome-Based Discoveries for Health: Workshop Summary
FIGURE 2-1 The translational process.
SOURCE: Teutsch, 2009.
in developing guidelines, Teutsch said, is how high the evidence bar should be. By employing a lower threshold, technologies can move more rapidly from research into practice. The consequences are that less information is available on the clinical validity of the technology, and almost no information is available about the clinical use. This lack of information can lead to negative insurance coverage decisions. There is the potential for increased harms because less is known about the technology, but also the potential for increased benefits by providing the technology sooner to those who may need it. Requiring a lower evidentiary bar means a greater dependence on models and expert opinion. Because technologies can enter practice more easily, a lower bar might stimulate innovation, thereby making more technologies available.
If the evidentiary bar is high, more will be known about the validity and utility of the technology, and payers can make better decisions about reimbursement. On the other hand, a higher threshold for evidence makes moving technologies into practice more difficult, which can potentially lower the incentive for innovation. More is known about the technology, resulting in a diminished potential for harms, but it will take a longer time to bring the product to those who can benefit from it.
When making an evidence-based decision, several questions must be answered:
What decision must be made?
How does the nature of that decision affect the evidentiary standards that should be applied?
What are the relevant contextual issues?
OCR for page 5
Systems for Research and Evaluation for Translating Genome-Based Discoveries for Health: Workshop Summary
How will information (both scientific and contextual) be integrated and applied?
What processes are needed to legitimize the decision process?
There is a dynamic relationship between evidence-based decision making and evidence review and synthesis (Figure 2-2). Decisions may pertain to regulation, coverage, guidelines, quality improvement metrics (e.g., pay-for-performance), or individual care decisions made by a clinician and/or patient. The decision maker should first frame the key questions to be answered and determine the level of rigor required. Then evidence reviewers should synthesize data from studies as well as desired economic information. With quantitative scientific evidence in hand, the decision makers should also consider budget constraints, values and preferences, equity issues, acceptability, and other contextual issues before making a decision.
Quantitative Information for Decision Making
Quantitative information needed for decision making includes data on effectiveness, such as the level of certainty there will be an impact, and the magnitude of the effect, or net benefit. Cost and cost-effectiveness data are
FIGURE 2-2 Dynamic relationship between evidence review and synthesis and evidence-based decision making.
SOURCE: Teutsch and Berger, 2005.
OCR for page 6
Systems for Research and Evaluation for Translating Genome-Based Discoveries for Health: Workshop Summary
FIGURE 2-3 Comparative clinical effectiveness matrix.
SOURCE: Developed by the America’s Health Insurance Plans (AHIP) Evidence Based Medicine Roadmap Group, Personal communication, S. Pearson, Institute for Clinical and Economic Review (ICER), July 9, 2009.
also important, as are any data regarding how the new technology compares to existing alternatives. Clinical effectiveness and cost effectiveness are usually assessed in relationship to therapeutic or diagnostic alternatives.
A matrix, such as the one under development by America’s Health Insurance Plans, can be useful to help payers compare two technologies with regard to net benefit and certainty (Figure 2-3). Technologies that have large net benefit and high certainty would be good candidates for coverage. On the other hand, products with limited or low certainty and equal net benefit are not ready for broad use. Some will have incremental benefits, but high certainty, and others will have new technology that is unproven, but has potential. Different insurance groups are likely to make different coverage decisions. Payers should be able to articulate what their criteria are, or how high the evidentiary bar is going to be, so a technology developer can decide whether to invest in developing the technology.
The key effectiveness questions relate to the following:
OCR for page 7
Systems for Research and Evaluation for Translating Genome-Based Discoveries for Health: Workshop Summary
Efficacy: Can the technology work in controlled conditions?
Harms: What are the possible harms?
Effectiveness: Does it work in practice?
Trade-offs: What is the balance of harms and benefits?
Comparative effectiveness: Does it work better than alternatives currently in use?
Subpopulations: Are there specific groups for whom it is likely to be a technology of choice?
As one example of a framework to determine how high the evidentiary bar should be for clinical management decisions, Teutsch cited the work of Djulbegovic and colleagues (2005) on cancer. The framework lays out proposed evidentiary standards for clinical applications as a function of treatment goals and acceptable regret. Considering the various goals of treatment—including cancer prevention in healthy individuals, palliative therapies, procedures that offer incremental improvement in terms of survival, or curative measures—how much certainty is needed before a technology should be used? How much regret will there be if the technology used is ineffective or even harmful?
In the prevention arena, Teutsch said, the evidentiary bar is very high because the interventions are being delivered to people who are otherwise healthy. The Evaluation of Genomic Applications in Practice and Prevention (EGAPP) working group, established by the Centers for Disease Control and Prevention, recently published its methods for evidence-based evaluation of genetic tests (Teutsch, 2009). Genome-based products first were categorized by application: diagnostic, screening, risk assessment and susceptibility, prognostic, or predicting therapeutic response. EGAPP then established the criteria that would be used when assessing clinical validity and utility issues (Table 2-1).
One approach to answering the quantitative questions is the ACCE model for evaluating data on emerging genetic tests. The model breaks down the information needed into four main areas (from which the name is derived): Analytic validity, Clinical validity, Clinical utility, and Ethical, legal, and social implications (Haddow and Palomaki, 2004). At the center of the circle in Figure 2-4 is the disorder to which the genetic test will be applied, and the setting in which the testing will be done. From there, an analytic framework is constructed by answering more than 40 targeted questions in each of the 4 areas.
EGAPP has been working within the ACCE framework to articulate the evidentiary standards that could or should be applied to evaluation of genetic tests. Table 2-2 presents a hierarchy of data sources and study designs for the analytic validity, clinical validity, and clinical utility compo-
OCR for page 8
Systems for Research and Evaluation for Translating Genome-Based Discoveries for Health: Workshop Summary
TABLE 2-1 Categories of Genetic Test Applications and Some Characteristics of How Clinical Validity and Utility Are Assessed
Application
Clinical Validity
Clinical Utility
Diagnosis
Association with disorder
Improved clinical outcomes
Usefulness for decision making
End of diagnostic odyssey
Disease screening
Association with disorder
Improved health outcome
Usefulness for decision making
Risk assessment/susceptibility
Association with future disorder
Improved health outcomes
Prognosis of diagnosed disease
Association with natural history
Improved health outcomes, or outcomes of value to patients, based on changes in patient management
Predicting treatment response
Association with a state that relates to drug efficacy or Adverse Drug Experiences
Improved health outcomes or adherence based on drug selection or dosage
SOURCE: Adapted from Teutsch et al., 2009.
FIGURE 2-4 The ACCE method for multidisciplinary evaluation of genetic tests.
SOURCE: CDC, 2007.
OCR for page 9
Systems for Research and Evaluation for Translating Genome-Based Discoveries for Health: Workshop Summary
TABLE 2-2 Hierarchies of Data Sources and Study Designs for the Components of Evaluation
Level
Analytic Validity
Clinical Validity
Clinical Utility
1
Collaborative study
Summary data from well-designed external proficiency testing
Well-designed longitudinal cohort studies
Validated clinical decision rule
Meta-analysis of RCTs
2
Other proficiency testing
Well-designed peer-reviewed studies
Expert panel reviewed
FDA summaries
Well-designed case-control studies
A single RCT
3
Less well-designed peer-reviewed studies
Lower quality case-control and cross-sectional studies
Unvalidated clinical decision rule
Controlled trial without randomization
Cohort or case-control study
4
Other research, clinical laboratory or manufacturer data
Studies on performance of the same basic methodology
Case series
Other research, clinical laboratory or manufacturer data
Consensus guidelines
Expert opinion
Case series
Other studies, clinical laboratory or manufacturer data
Consensus guidelines
Expert opinion
SOURCE: Teutsch, 2009.
nents of evaluation. Looking at clinical utility, for example, meta-analysis of randomized controlled trials (RCTs) would be the strongest form of evidence. A good single RCT may be adequate, but less strong. The list then covers other study designs that are progressively less desirable, such as controlled trials that are not randomized, or cohort studies, with case series or expert opinion being the least desirable form of evidence.
Contextual Information for Decision Making
Numerous contextual issues can inform the decision to introduce a test into practice. Clinical applications differ widely, and it is important to consider the severity of the condition, subgroup differences, the availability of alternatives, the severity and frequency of harms, and the risk of overuse or inappropriate use of the test. Economics is also considered from
OCR for page 10
Systems for Research and Evaluation for Translating Genome-Based Discoveries for Health: Workshop Summary
a contextual perspective. Many decision makers are interested not only in cost-effectiveness, but also budget impact, budget constraints, and value. Legal and ethical considerations include federal and state regulatory constraints, as well as issues of precedent, and regret as a result of introducing or not introducing a test. Feasibility of the test in question refers to the current level of use, the infrastructure required to use the test properly, and the acceptability of the test to all partners and stakeholders, particularly patients. Decisions should be made in the context of the preferences and values of those who are going to be affected by the decision. Finally, there are administrative issues, such as options for targeting or limiting the use of the test to patients who would benefit most, and how to consider possible further evidence.
Decision-Factor Matrix
In the end, Teutsch said, a systematic process is needed to ensure fairness and reasonableness in decision making. This process includes: clear “rules of the road” for the technology developers, patient advocacy groups, and others; a deliberative process incorporating both quantitative and qualitative or contextual information; transparency; and an appeals processes so that when other issues arise, they can be addressed, and the decision changed where appropriate.
Teutsch presented a draft of a decision matrix, plotting different decisions that are likely to be made for any test or technology against a set of quantitative and qualitative information that might need to be generated. His example (Figure 2-5) suggests that a regulator may be primarily interested in efficacy, safety, and the legal and ethical constraints. These aspects, however, would be less likely to impact individual decisions. Rather, effectiveness, as well as cost, may be of great interest in practice. Each type of user will have important criteria, some secondary considerations, and other information that may not be directly relevant. The important point, Teutsch said, is that different decision makers require different kinds of information, and it is important to be able to generate that information for them.
In refining the approach to standards of evidence, Teutsch said in conclusion, it will be important to rethink the hierarchy of evidence in terms of the many different applications and new types of evidence. When is it appropriate to use predictive modeling, for example? Another critical issue is how research efforts are aligned with application needs. The evolving role of observational data must be accommodated, and appropriate methods must be used to make better decisions when the evidence is insufficient.
OCR for page 11
Systems for Research and Evaluation for Translating Genome-Based Discoveries for Health: Workshop Summary
FIGURE 2-5 Example of a hypothetical decision-factor matrix.
* Administrative feasibility of management, e.g., limiting coverage to people who meet specific criteria.
Legend:
White: primary consideration.
Light grey: secondary consideration.
Dark grey: minor or no consideration.
SOURCE: Teutsch, 2009.
DISCUSSION
Wylie Burke, M.D., Ph.D.
Moderator
A question was asked as to whether the appeals process mentioned by Teutsch would address passive challenges, such as a need for change identified as a result of horizon scanning, as well as active challenges. Teutsch responded that there may be information that was not taken into consideration in the original decision, and the appeals processes can help address that issue. But in general, one should be proactive about the information generation process. In trial design, for example, it is important to ensure representation from the appropriate groups, and that may require participation of the affected groups in the development of the study.
A participant noted that the methodology outlined focuses on the test or the technology itself, and asked if the questions would change when the
OCR for page 12
Systems for Research and Evaluation for Translating Genome-Based Discoveries for Health: Workshop Summary
focus was on whether or not to screen for a condition. Teutsch responded that one needs to have a specific clinical scenario in mind, and that assessments should not be done in the abstract.
Another participant expressed concern about the decision matrixes considering low efficacy and harm as if they were similar in impact, and suggested that a distinction be made. Teutsch said the vocabulary varies, but in his perspective, efficacy refers to benefits, and effectiveness refers to the balance of the benefits and potential harms. On some occasions, risk of substantial harm may be acceptable because of the potential for substantial benefits, while at other times the equation will be different. He agreed there is a need to be clear about whether one is talking about benefits or harms, and to whom they accrue.