5

An Evidence Framework for Genetic Testing

The committee explored issues regarding genetic testing, including various methods of assessing genetic tests and evidence supporting the use of genetic tests. In this chapter, the committee integrates that information and presents its framework for decision making. Specifically, the committee describes a decision framework for the Department of Defense (DoD) to use when making decisions about genetic tests. The framework includes a definition of the topic being considered and the application of a triage process to evaluate whether a test is worthwhile and whether an expedited provisional decision can be made. For topics that need to be evaluated in more detail, the committee describes an additional process that would guide DoD in conducting an evidence review. Such a review would consist of using an evaluation structure to identify evidentiary requirements suitable to the outcome of interest and assessing the available evidence. Insofar as possible, the committee considers the distinctions between individual tests versus large gene panels and genome-scale testing, and provides several considerations on how a decision framework could account for these differences.

The following definitions will be used in this chapter in referring to processes that lead to decisions about the use of genetic tests in a clinical setting:

• Decision framework: an overall plan for an evaluator to use in determining which tests should be used in evidence-based clinical care
• Evaluation structure: the process of determining the type of evidence that needs to be evaluated, given the desired clinical outcome of a genetic test
• Evidence assessment: the process of evidence review and analysis to determine the level of evidence that is available to support the use of a genetic test, depending on the desired outcome

The purpose of a decision framework is to assist stakeholders (clinicians, patients, payers, public health officials, policy makers, and others) in making decisions about the balance of benefits and harms associated with using a test, whether genetic or otherwise; it defines a process by which such deliberations can be carried out and outlines the procedures used to examine evidence related to the use of a genetic test in a specific clinical scenario. An evaluation structure can be used to guide the decision-making process by framing different outcomes and establishing the evidentiary requirements for use of a genetic test with a chosen outcome in mind (see Chapter 3). And an evidence assessment process guides a reviewer to particular evidence types,

according to the outcome of interest, and helps in the evaluation of the available evidence (see Chapter 4). The remainder of this chapter provides the details of the committee’s recommended framework for DoD.

OVERVIEW OF THE PROPOSED FRAMEWORK

After reviewing different methods that have been developed for the evaluation of medical tests, some of which were specifically intended for use with genetic tests, the committee developed a hybrid system that incorporates elements of several of the methods described in Chapter 3. The system that the committee developed—its decision framework—consists of seven components depicted in Figure 5-1, and described in the text that follows. The framework is not linear nor is it meant to be a formal algorithm, but rather to provide general guidance to DoD.

conclusions a user might reach at certain points in the process. Dotted lines show how information gathered through use of the framework can inform other steps in the process.

Component 1. Genetic Test Scenario

As a first step, a “genetic test scenario” must be systematically defined to enable relevant information to be identified. The scenario should include the test being performed, the population in which testing is considered appropriate, the purpose of the test in that population (clinical scenario), the outcomes of interest, and comparable alternative methods to accomplish those tasks. Different decisions might be made regarding the use of a particular genetic test, depending on the population and the purpose of testing. However, defining the test can be challenging, given the rapid expansion of molecular genetic tests that are now offered, which range from analysis of a single genetic variant to the whole genome sequence. Tests that evaluate the same gene might use different techniques with distinct performance characteristics; in some cases, different modalities (such as gene sequencing and analysis of large deletions or duplications) are coupled in an effort to evaluate the gene or genes included in the test comprehensively. Definition of the test under consideration, therefore, should include the target being evaluated as well as the method or methods used to detect clinically relevant variation.

Within any given genetic test scenario, the population being tested and the purpose of the test are often considered together as a “clinical scenario.” In many cases, a particular genetic test could be considered for use in one population for one purpose and in a different population for another purpose. The performance of the test and the decision about whether testing is worthwhile might vary dramatically, depending on the clinical scenario. In general, the purpose of genetic testing in a particular clinical scenario falls into one of several categories (see Chapter 2).

Outcomes of interest follow directly from the purpose of a test but might depend on the population being tested. For example, if the purpose of a test is risk assessment or screening of asymptomatic people, the outcomes of interest might be related to health, such as improved morbidity and reduced mortality in the population. Some genetic tests might be ordered to support or inform management decisions specific to drug therapy, in which the outcomes are likely to include both clinical and economic measures of improved health care. If, however, the purpose of a test is to obtain a molecular diagnosis in a patient population in which genetic conditions are suspected, the outcomes of interest might be related to the diagnostic yield¹ of the test and the effect on a physician’s diagnostic thinking.

If a decision has already been made about the use of one genetic test in a particular clinical scenario (considering the population and the purpose of testing), decisions about the use of other genetic tests in the same test scenario, or decisions about the use of the same test in a different clinical scenario, could be streamlined.

Component 2 (A, B, C). Prioritization of Topics to Be Evaluated and Triage Process

The evaluator must prioritize the various genetic test scenarios to determine which ones to address. This determination could be made based on a number of different criteria (e.g.,

___________________

¹ Diagnostic yield is the percentage of patients with a particular phenotype who are found to have a mutation in the gene, exome, or genome tested.

volume of test requests, unit cost of requested tests, stakeholder assessments, test development planning) depending on DoD’s priorities.

The evaluator then carries out an initial triage step (Component 2) to determine if a rapid decision can be made based on existing information. This triage step actually represents three sequential questions (see Box 5-1). First, the evaluator determines whether the purpose of the test is “worthwhile” (Component 2A). This determination could be based on whether DoD already offers coverage of other genetic or nongenetic tests used for the same purpose. If the answer is “NO” then a decision can be rendered without further evidence review. If the answer is “YES” (because of similar existing coverage decisions) or “MAYBE” (depending on the characteristics of the test or completeness of the knowledge base), the evaluator performs a streamlined “rapid review” of the available evidence for the genetic test scenario (Component 2B), such as by looking for practice guidelines, systematic evidence-based reviews, or other authoritative expert reviews (if available).

Systematic reviews can help DoD understand the depth and quality of the evidence base for a test quickly to aid in triage decisions. Because the scope, quality, and timeliness of existing systematic reviews vary with their purpose and intended audience, they should be considered carefully for applicability to the test, the specific scenario, and as to whether the quality is acceptable. If reviews are deemed sufficiently credible and reliable, demonstrating clinical utility might not require a new systematic review. Alternatively, DoD could utilize information from previous decisions regarding the same or similar genetic test scenarios or by using a list of standard questions like those posed by the analytical validity, clinical validity, clinical utility, and associated ethical, legal, and social implications (ACCE) or Genetic testing Evidence Tracking Tool approaches. The McMaster framework also provides a set of useful questions to be addressed in the rapid review. Although there is insufficient evidence to recommend a single approach, DoD might consider the similarities among those approaches when making decisions regarding coverage for TRICARE. In both systems, a key question is whether the test is worthwhile or valued on the basis of the purpose of the test and in comparison with outcomes that would be obtained without the test. DoD should triage the evaluation process for any given genetic test according to whether nongenetic tests with the same or similar purpose (such as prediction, diagnosis, or reproductive) are covered, and therefore at least potentially worthwhile if other criteria are met. If a test is deemed not to be worthwhile, no further evaluation of evidence should be necessary, and DoD can determine that this particular use of genetic testing and presumably other technologies that accomplish the same purpose are not part of the covered benefit. The committee believes that DoD should aspire to transparency and accountability to its stakeholders with all such decisions, providing justification where indicated.

If the rapid review reveals sufficient evidence (YES), or areas of missing evidence are deemed non-essential (MAYBE), the evaluator will need to consider contextual issues (Component 2C). Those might include social issues, net benefits and harms, and aggregate costs. The balance of the issues, that is, whether a test is worthwhile, the evidence, and contextual factors should inform a triage decision. Those considerations, along with the rapid evidence review, will be brought to the decision process (Component 4). If the rapid review does not reveal sufficient evidence (NO), then a formal evidence-based review process (Component 3) would be appropriate to systematically evaluate the evidence to facilitate a decision process (Component 4) as well as identify evidence gaps (Component 7).

Component 3. Evidence-Review Process

A formal evidence review process is needed for topics that require additional evaluation. The first stage of the process is establishing the value proposition for the use of the genetic test in a particular clinical scenario. That involves identifying the purpose, important outcomes of testing, and any relevant comparators, such as other genetic or medical tests already used in clinical care for the same purpose. According to Frueh and Quinn (2014) the value proposition that a test will generate a more accurate prognosis is highly actionable in some settings, but it is statistically abstract and therefore hard to visualize in clinically meaningful terms. For example, when a prognostic test will be used to deselect lower-risk patients from surgery or from the need for chemotherapy, decision makers might require strong evidence that indicates that the deselection is safe and in the patient’s best interest. A predictive test to deselect a cancer patient from a targeted therapy will be viewed more cautiously when no other therapy is available.

Thus, with a clear scope and a systematic approach to evidence assessment to provide guidance regarding the types of evidence that are needed, such as the modified Frybeck–Thornbury approach, a systematic evidence assessment should be conducted. Other criteria for evaluation have been articulated (Giacomini et al., 2003; Frueh and Quinn, 2014). Decision makers should set cutoffs for each of the evaluated criteria, which are required to make “a clear delineation of what is ‘good enough’ from what is not” (Giacomini et al., 2003).

There is ample guidance on how to conduct systematic reviews of evidence from medical tests and interventions (see Guyatt et al., 2011, 2013; Higgins and Green, 2011; IOM, 2011a; AHRQ, 2012; NICE, 2013; Murad et al., 2014; USPSTF, 2015). There is also guidance specific to the conduct of systematic reviews and evidence assessments for genetic tests, for example, the HuGE [Human Genome Epidemiologic] Review Handbook (Little and Higgins, 2006; Gwinn et al., 2014).

Systematic reviews are useful because inferences from single studies might be unrepresentative of the larger body of evidence and might generate misleading conclusions (Murad and Montori, 2013). Meta-analyses of existing data on outcomes should be undertaken as an accompaniment to systematic reviews where feasible. Meta-analysis of a field of evidence is able to use a larger sample and more events than any individual study, increases precision and reliability of effect-size estimates, and provides an opportunity to explore reasons for inconsistency between studies and to rate the risk of bias contributed by single studies (Murad et al., 2014).

If the triage process suggests a potential health benefit associated with genetic test scenarios that are especially promising for clinical utility but lack comprehensive evidence, or those with strong stakeholder interest but without direct evidence of clinical utility, DoD could consider allowing such genetic tests to be used in a controlled manner and would be dependent on longitudinal follow-up of outcomes. That option would help to address evidence gaps (Component 7) and would be expected to lead to re-evaluation of the decision (Component 6).

Component 4. Decision Process

After the formal evidence-assessment process, DoD will need to establish a decision process for evaluating the results of the evidence review and determining whether the available evidence reaches a threshold appropriate for the genetic test scenario in question. The decision process will incorporate the preceding evidence review (Component 3) and contextual factors such as social issues, potential harms, or benefit/cost considerations and result in either a “YES” or “NO” decision regarding coverage, which should then be documented transparently in a decision repository (Component 5).

Even when evidence is strong and clear, the judgment of whether benefits outweigh harms is subjective. It has been noted that decision makers, with identical data, can reach different conclusions based on their values. Decision makers commonly consider clinical experience, expert opinion, and personal judgments regarding potential harms of the test versus not having the test. Such considerations color the evidence and resulting recommendations (IOM, 2011c).

In addition to questions about whether the available evidence supports the use of a genetic test in a given clinical scenario, the process is likely to include economic considerations. In that regard, it might be important to consider multiple stakeholder perspectives in the determination and the differences among them. Input from stakeholders might affect the relative weight of net benefit and harms. Ultimately, demand for genetic tests, unit costs, and expert consensus on clinical value will be considered by decision makers.

It has been suggested that thresholds for making decisions might be further informed by three approaches. These include abstract principles set by stakeholder values and require careful justification; precedents set by earlier decisions regarding genetic tests and the reasons for making them; and precedents set elsewhere in the health system (Giacomini et al., 2003).

Thus, because the process of determining what constitutes “sufficient” evidence is subjective and based on value judgments, DoD must set its own standards, preferably in a clear and consistent way, to reflect its values and needs, with transparency and accountability to its stakeholders. Approaches for making clinical decisions, such as clinical-practice guidelines, that are based on systematic evidence reviews are well documented (e.g., Guyatt et al., 2008; IOM, 2011c; Schunemann et al., 2016).

Component 5. Decision Repository

A repository of decisions about individual genetic test scenarios will foster the “institutional memory” of the process and facilitate future reviews. If appropriately structured, the repository will allow reviewers to evaluate new scenarios rapidly in light of previous decisions that have been made for similar tests, populations, purposes, and outcomes of interest and for comparable methods. A decision repository also serves as a record of the value judgments made about whether particular genetic test scenarios are deemed worthwhile so that stakeholders can understand the decisions and DoD can ensure consistency among decisions.

The McMaster framework, for example, allows for the evaluation of panels reasonably well. Newer tests based on sequencing panels of genes, or even the entire genome, could be viewed as collections of individual tests, in which there may be substantial evidence for some genes and less evidence for others. In these cases, a pragmatic decision about coverage of a panel or genome-scale test might include consideration of both the evidence supporting individual genes relevant to a particular clinical scenario and the efficiency of evaluation (cost per test). The repository will tell them whether prior decisions have been made for any of the genes on the panel and if coverage has already been decided for one or more of those genes in the same clinical scenario. For genetic test scenarios that undergo a formal evidence review, the details of the process and the evidentiary thresholds required by DoD will provide transparency to the process and allow identification of evidence gaps that can be addressed through research.

Component 6. Process for Periodic Review and Revision

Given the rapid development of technology and research in genomic medicine, it is critical to develop a mechanism for periodic reassessment of decisions in light of new data. The extent to which the mechanism can be enacted might depend somewhat on the organization of the decision-making process. If designed properly, the decision repository could facilitate a systematic process by including time stamps and automated prompts for re-review of decisions on a schedule or provide a process for stakeholders to request revised decisions.

Component 7. Identification of Evidence Gaps

As noted in Chapter 4, the evidence base for genetic testing is limited for several reasons, including the paucity of the types of research studies on which evidence-based medical decisions depend, exacerbated by the accelerated pace of development of new tests, discovery of variants, and technologic discoveries. RCTs to establish clinical utility are expensive and take a long time, so decisions are often based on lower levels of evidence. Therefore it is not surprising that specific gaps will be identified in the course of review in the decision-making process.

If the available evidence suggests a potential health benefit associated with a genetic test, especially for tests related to high-priority health conditions, DoD might consider a process for

evidence development. Some genetic test scenarios might benefit from traditional research studies to address evidence gaps. For example, meta-analyses of cohort studies might indicate that a specific genetic test has a high predictive value for development of a cancer of major incident morbidity and mortality in the military and civilian populations. However, for that hypothetical test, there have been no prospective trials that have randomized people to genetic testing or no testing to evaluate effectiveness in improving early detection and treatment and reducing mortality. The evidence base on genetic testing would be substantially enhanced, in this case, by supporting a prospective trial designed to establish clinical utility if promising meta-analyzed retrospective results were confirmed prospectively. Evidence-gap analysis should be conducted on a regular basis as part of a process of continual quality improvement that uses a clinical implementation science process (David et al., 2015). Evidence gaps identified should inform research priorities, such as those suggested in Chapter 4.

RECOMMENDATION

In its review of frameworks for decision making, the committee noted that a framework should include a clear definition of the topic being considered and a triage process to evaluate whether the purpose of a test is worthwhile and an expedited provisional decision can be made. However, for topics that need to be evaluated in more detail, the committee provides an additional process to guide DoD in conducting an evidence review. Such a review would consist of using an evaluation structure to identify evidentiary requirements suitable for the outcome of interest and assessing the evidence that is available. Recognizing the rapid advances in genetic testing and evolving evidence regarding the role of genetics in different health conditions, the decision framework includes periodic review and revision of decisions. This process could be triggered automatically after a certain period of time or be prompted by stakeholder requests. As noted in the discussion, the systematic review process will lead to the identification of evidence gaps that should be tracked.

The committee recommends a decision framework based on the numerous frameworks it has reviewed and adapts the best of those for its intended purpose. The framework is neither linear nor is it meant to be a formal algorithm, but rather to provide general guidance to the DoD. It begins with a clear definition of the topic being considered and a triage process to evaluate whether the purpose of a test is worthwhile and an expedited provisional decision can be made. For topics that need to be evaluated in more detail, the committee provides guidance for conducting an evidence review.

1. Define genetic test scenarios on the basis of the clinical setting, the purpose of the test, the population, the outcomes of interest, and comparable alternative methods.
2. For each genetic test scenario, conduct an initial structured assessment to determine whether the test should be covered, denied, or subject to additional evaluation.
3. Conduct or support evidence-based systematic reviews for genetic test scenarios that require additional evaluation.

4. Conduct or support a structured process to produce clinical guidance for a genetic test scenario.
5. Publicly share resulting decisions and justification about evaluated genetic test scenarios, and retain decisions in a repository.
6. Implement timely review and revision of decisions on the basis of new data.
7. Identify evidence gaps to be addressed by research.

This page intentionally left blank.