Assessing Genomic Sequencing Information for Health Care Decision Making: Workshop Summary (2014)

Below is the uncorrected machine-read text of this chapter, intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text of each book. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.

2 How Evidence Is Gathered and Evaluated Important Points Highlighted by the Individual Speakers • While targeted mutation and gene panel testing are more techni- cally complete than exome sequencing, exome information can be useful for carrier screening, disease diagnosis, and phar- macogenomic testing. • Rare variant databases and other databases that include both genotype and phenotype information can be valuable shared re- sources for clinicians and researchers to aid in disease diagno- sis, gene–phenotype associations, and drug development. • A transparent, reproducible, evidence-based method for deter- mining variant actionability is helpful when individual experts have different opinions about what variant information should be returned to patients. • The actionability of genomic findings depends on the clinical context, such as whether testing is done before conception, pre- natally, for newborn screening, during childhood, or for screen- ing, diagnostic, or monitoring reasons. • “Binning” the genome on the basis of clinical validity and clini- cal utility and other staged approaches can facilitate pre-test in- formed consent, analysis, and post-test return of results. • Given how resource-intensive the process of evaluating variant ev- idence is, a collective effort using a standardized assessment ap- proach and shared variant databases would be helpful in leading to more efficient variant curation. • Improving the communication between testing laboratories and clinics would make it possible to update genotype–phenotype information as new data are collected. • Technical issues—from gene coverage during data collection to bioinformatics interpretation of the data—vary and can impose limits on the information that can be derived from whole-exome or whole-genome sequencing unless they are standardized by the genomics community. 5 

6 GENOME SEQUENCE INFORMATION IN HEALTH CARE DECISIONS During the workshop a variety of experts in academia and the private sector described current research and clinical perspectives concerning the ways in which genomic data are being generated and linked to human diseases and applied to the practice of medicine. The topics covered dur- ing the presentations and discussions included the sources of genomic data, various processes such as “binning” genomic findings into catego- ries with different degrees of actionability, systematic approaches to evaluating gene–phenotype associations, and a collaboration to create a curated resource that can help standardize the interpretation of genetic variation. Other topics addressed were the gathering, assessment, and evaluation of evidence for use in next-generation sequencing in cancer genomics; how new information is reviewed in the context of existing information; and how variant information can be shared more widely. GATHERING DATA In recent years, many gene panels have been introduced into the clin- ical setting, noted Madhuri Hegde, professor of human genetics and ex- ecutive director of the Emory Genetics Laboratory. The targeted mutation and gene sequencing panels are technically complete in that they cover all the exons of a gene and the entire mutation spectrum of a gene, including point mutations, insertions–deletions, copy number vari- ability, and deep intronic pathogenic changes. By contrast, while exome sequencing covers more overall genes, the majority of the genes covered by exome sequencing are not clinically relevant, and for those genes that are clinically relevant, exome sequencing may not have complete cover- age of all exons and may not cover the full spectrum of mutations, Hegde said. Despite this incompleteness, however, exome sequencing can still collect evidence important in assigning genes to a disorder, and it can be useful in yielding information relevant to carrier screening and phar- macogenetic markers. There is a “critical need in our community to establish what is the [essential] amount of data [for including] a gene in a genetic test,” said Heidi Rehm, director of the Laboratory for Molecular Medicine at the Partners Healthcare Center for Personalized Genetic Medicine and assis- tant professor of pathology at Harvard Medical School. Many of the gene panels being offered today have a highly variable number of genes for the same indication, partly because of different evalu- ations of the evidence for a gene–phenotype association (Rehm, 2013). 

HOW EVIDENCE IS GATHERED AND EVALUATED 7 Even in the case of a targeted panel where phenotypic information can be gained from the results, complementary assays often need to be included with the gene panel, Hegde noted. For example, with the gene panel for short stature, methylation-based assays are necessary. Whether a gene panel works in a clinical setting therefore “depends on which dis- order you are looking at,” Hegde said. Exome sequencing can be used for either clinical or research purpos- es, though recently the boundaries between the two have been blurring. In Hegde’s laboratory, exome data are divided according to why the se- quencing is being done. For new disease presentations the diagnostic yield, or likelihood that the test will provide enough information to make an appropriate diagnosis, ranges roughly from 30 percent to 40 percent, depending on which laboratory is reporting and what kinds of cases are considered, Hegde said. When writing clinical reports, she said, it is crit- ical to sorting the data into categories of what can be interpreted in the clinic and what is clinically actionable (see Box 2-1). BOX 2-1 Contextual Usage of Clinical Actionability, Validity, and Utility • Clinical actionability: in the context of incidental findings or in an asymptomatic individual, the degree to which an interven- tion exists that can mitigate harm before a clinical diagnosis is made. • Clinical validity: the accuracy and reliability of a variant for identifying or predicting an event with biological or medical significance in an asymptomatic individual. • Clinical utility: the usefulness of information in clinical deci- sion making and in improving health outcomes. 

8 GENOME SEQUENCE INFORMATION IN HEALTH CARE DECISIONS Genomic Sequencing in Oncology In the past, oncologists have based treatment largely on traditional immunochemistry, pathology, and, more generally, anatomical staging, said Mark Robson, attending physician of the clinical genetics and breast cancer medicine services in the Department of Medicine at Memorial Sloan–Kettering Cancer Center. Now next-generation sequencing is creating a massive experiment in whether knowing the pattern of genomic aberra- tions will allow therapies to be targeted more effectively. “Although every- body is very enthusiastic about it,” Robson said, “whether or not we are going to be able to achieve better outcomes on a global scale throughout the cancer population still remains to be seen.” Most cancer centers are using targeted assays rather than whole- exome or whole-genome sequencing to look at a variable number of genes that have been selected according to an a priori rationale for in- volvement in the oncogenetic or oncologic process. For example, the Integrated Mutation Profiling of Actionable Cancer Targets (IMPACT) panel probes for biologically or clinically relevant cancer genes (Wagle et al., 2012). Many of the genes are linked to cancer only through somat- ic mutations, but most of the germline predisposition syndrome genes are included on these panels as well, because many of them are also involved in carcinogenesis in nonhereditary contexts, Robson said. In the clinical context, mutational profiling is used for variants that are clearly linked to response to a U.S. Food and Drug Administration (FDA)-approved drug, that define clinical trial eligibility, or that are plausibly predictive of response to an approved drug which might not otherwise have been chosen. Variants linked to response to an approved drug have already been defined through the companion diagnostic mech- anism, though the companion diagnostic development process can be extremely complicated (IOM, 2014; McCormack et al., 2014). Similarly, variants used to define clinical trial eligibility have generally already been defined. The more challenging area involves variants that are potentially pre- dictive of response to an already approved drug. “In other words,” Robson said, “you send the test off to [a] commercial entity, get back a series of variations, and now you pull [a drug] off the shelf and use it.” This de- termination depends on such factors as whether the variant is germline or somatic, whether the link is biologically plausible, the prevalence of the allele for somatic mutations, whether the primary tumor or metastatic 

HOW EVIDENCE IS GATHERED AND EVALUATED 9 disease has been analyzed, and whether a drug or a combination of drugs is available to use. The optimal interpretation of a somatic sequence requires the se- quencing of normal tissues, Robson said. Sorting out driver and passen- ger mutations can be very difficult, but finding that something is present in a tumor and not present in the germline is at least an initial piece of evidence that could be relevant to the cancer process. However, if there is a germline alteration, it may not be seen when comparing the two se- quences, as many algorithms subtract germline from somatic variants found during sequencing (Bombard et al., 2013). Using next-generation sequencing techniques to generate data and compare germline and so- matic mutations has also shown promise for identifying variants that are associated with susceptibility to cancer (Stadler et al., 2014). SOURCES OF DATA Databases for Genomic Case Reports Databases could be useful repositories for finding information about genes with weak disease associations or with unknown significance. For example, Rehm told of how a patient with the rare disease distal ar- throgryposis type 5, a condition related to congenital joint contracture, underwent genome sequencing even though at the time the disorder had no known genetic etiology. Because this patient had unaffected parents, a de novo cause of disease was suspected, Rehm said. Sequencing the genomes of the patient and the parents revealed two such de novo variants, one of which was quickly ruled out as a common loss-of-function mutation in that population. The remaining variant was a candidate, but there was no evidence to indicate it was causative of the disease, because everyone has de novo variants that are not necessarily related to a phenotype. Rehm and her colleagues contacted a researcher who studied the PIEZO2 protein, the product of the gene in which the variant appeared, and in this way they learned about a second family with a mutation in the same gene who had the same phenotype. The in- teraction “gave us enough evidence to claim a true causal association with this gene and that phenotype,” Rehm said (Coste et al., 2013). One cannot expect serendipity to produce such findings too often, so Rehm and her colleagues are working to establish a database to house genomic cases. Various groups have contributed exome and genome data 

10 GENOME SEQUENCE INFORMATION IN HEALTH CARE DECISIONS along with phenotype information to a database that Rehm has devel- oped. The data will be searchable and structured in a way that will allow for the identification of genetic commonalities among phenotypes. This is, she said, “a more robust, international approach to solving these very rare cases in both a clinical testing arena as well as a research context.” ClinVar and ClinGen The ClinVar variant database is designed to provide a freely accessi- ble, public archive of reports of the relationships between human genetic variations and phenotypes (Landrum et al., 2014). All of the information being generated in Rehm’s laboratory is also being submitted to ClinVar so that the community can benefit from that information. “By putting a lot of this data that we come across [from] clinical testing and research testing into a common environment,” Rehm said, “that then provides a list of variants that either a researcher or a pharmaceutical company could . . . study. If they don’t know what variants are out there, there is no project to be done.” The individual efforts of institutions to gather and evaluate evidence can be scaled to benefit the larger genomics communi- ty through databases such as ClinVar, Rehm said. Data, including benign variant assessments, are deposited here for sharing it more broadly. The Clinical Genome Resource, or ClinGen, is a collaboration among research groups dedicated to combining research data with data from clinical tests as well as expert curation to determine which genetic variants are most relevant to patient care (NIH, 2013). As part of this effort, the research groups are examining the standards and processes for evaluating genes and variants and genetic disorders in order to move to- ward more standardized procedures, said Jonathan Berg, assistant profes- sor in the Department of Genetics at the University of North Carolina at Chapel Hill. ClinGen starts with the variants, Berg said, so the first step in the ef- fort has been to encourage laboratories to submit data to the project. The next step is to gather phenotypic information about patients in whom the variants are found, along with evidence from the laboratory indicating whether a variant is pathogenic or benign or if there is not enough evi- dence to be certain. The final step is to understand the clinical validity of gene–phenotype associations, which will provide a standardized frame- work for curating these associations. “If we can bring that information all together with standardized language and using the same vocabularies to describe what we’re talking about, then we will have a computational 

HOW EVIDENCE IS GATHERED AND EVALUATED 11 resource that can be mined for clinical validity and the associations of these variants to disease,” he said. ClinVar is part of the ClinGen collaboration, and together these re- sources will have a number of valuable uses, Rehm said. For example, they could enable the community to define what the best assays are for assessing a particular gene or disease model. “When you come up with a variant, you can turn toward the appropriate assay . . . and know where you could get it done,” she said. THE ELEMENTS OF ACTIONABILITY Clinical actionability (see Box 2-1) requires both technical accuracy and interpretive accuracy, which together produce high specificity in terms of predictive value. It is important, Berg said, that such an inter- vention not impose undue hazards to an individual, whether psychoso- cial, medical, or financial. Because individual expert opinions vary considerably, there is a need for a transparent, reproducible, evidence-based method for determining whether an identified variant is clinically relevant, Berg said. Thus Berg and his colleagues have divided the concept of actionability into several specific elements that give a semi-quantitative assessment of actionabil- ity for every gene–phenotype pair: • Severity of a disease, which is typically the most severe possible outcome • Likelihood of a severe outcome • Effectiveness of an intervention to mitigate the severe outcome • Acceptability of the intervention, with consideration given to all the hazards of the intervention • State of the knowledge base, including knowledge about the gene–phenotype association, disease manifestations, and inter- ventions Each of the 5 elements receives a score from 0 to 3, for a total score of between 0 and 15. Thresholds can be set for dividing variants into bins indicating whether the variants have clinical utility or clinical validity or the clinical implications are unknown (Berg et al., 2011). (More details are provided later in this chapter in the subsection labeled “Binning the Genome.”) Different users could set the thresholds in different places, which provides the system with a measure of flexibility. “It balances the 

12 GENOME SEQUENCE INFORMATION IN HEALTH CARE DECISIONS benefits of the information versus the harms of the information, the pa- ternalism of the physician’s duty to warn versus not doing any harm, and patient preferences for their right to know and not to know,” Berg said. In addition to being flexible, the advantages of this system are that it is transparent and less subjective than expert opinion, with a clearly de- fined evidence base, Berg said. Furthermore, some of the workload can be crowd sourced—for example, in the analysis of the consistency or variability of scores. Different end users can use the information in vari- ous ways, weighing the parameters depending on the scenario of interest to the particular user (for example, research, diagnostic testing, healthy adults, or newborn screening). Finally, scoring can be revisited as new information becomes available. This system could be useful in the context of other efforts, such as the return of incidental findings. For example, when Berg and colleagues used Berg’s system to compare 200 genes sorted into bins with a recent list of variants in 56 genes for which the American College of Medical Genetics and Genomics (ACMG) recommends returning information to individuals,1 they found variability in what different groups consider ac- tionable (Green et al., 2013) (see Figure 2-1). The spectrum of actionabil- ity raises the question of whether the threshold has been set too low for the ACMG list because, for example, a number of genes on that list score only between 7 and 10 using Berg’s methodology. As Robert Green, director of the Genomes to People Research Pro- gram in Translational Genomics and Health Outcomes in the Division of Genetics at Brigham and Women’s Hospital and Harvard Medical School, observed, thousands of genomes were being sequenced, and phy- sicians were becoming uncomfortable with the idea that potentially life- saving information discovered in sequencing data was not being report- ed. The ACMG recommendations were crafted to address this issue. The recommendations propose reporting specific mutations found in those 56 genes to physicians regardless of the indication for which the clinical sequencing was ordered. With the information in hand, physicians are able to decide what to do with it while taking patient preferences into account. “You can have a __________________________ 1 Following much discussion over the ACMG Genome Sequencing Return of Results guidelines issued in March 2013, ACMG has since updated their recommendations to include an “opt-out” option for patients undergoing whole exome or whole genome se- quencing. For more information, see ACMG Updates Recommendation on “Opt Out” for Genome Sequencing Return of Results, https://www.acmg.net.docs/Release_ACMGUpdates Recommendations_final.pdf (accessed June 11, 2014). 

HOW EVIDENCE IS GATHERED AND EVALUATED 13 FIGURE 2-1 Application of Berg’s binning metric to genetic variants demon- strates variability in which variants different groups would consider actionable. NOTE: ACMG = American College of Medical Genetics and Genomics; HFE = hemochromatosis gene SOURCE: Jonathan Berg, IOM workshop presentation, February 3, 2014. very clear conversation with a patient about what they do not want to hear about, and you can respect that,” Green said. Berg asked whether some genes not included in the ACMG list, such as those involved in hemochromatosis, for example, should be considered for addition be- cause of their high scores on the metric he developed. The Medical Exome Project Hegde’s group has taken an approach to qualifying evidence that is different from Berg’s. The production of a medical exome—the subset of a human genome consisting of the more than 4,000 genes that have been identified as clinically relevant and that can be adequately covered—will require evidence about each gene and a technically complete assay, Hegde said. To do this, Hegde’s laboratory has collaborated with the Children’s Hospital of Philadelphia and Partners HealthCare Laboratory 

14 GENOME SEQUENCE INFORMATION IN HEALTH CARE DECISIONS for Molecular Medicine to create the Medical Exome Project, a “highly curated gene resource and a technically optimized assay to provide a stepping stone for standardizing the interpretation of genetic variation.” The goal of the project is to develop a “medically enhanced exome” cap- ture kit that covers all clinically significant genes so that when physi- cians are trying to diagnose a patient, they will have confidence that the known clinically relevant genes have complete coverage. Achievements to date have included increasing the coverage of known relevant cardio- myopathy genes from 85 percent to close to 99 percent. The members of the project have defined the medical exome, Hegde said, by starting with all genes that have possible or proven disease associations, then curating to eliminate false-positive disease association claims, and doing iterative curation to remain current. The Medical Exome Project has worked closely with the ClinGen project to set up a four-tier classification scheme for genes (see Table 2-1). It also went through a pilot curation phase that found many incorrect gene–phenotype associations. This is a time-consuming process; it takes about 5 hours per gene with at least 2 people researching and curating the gene data. With ap- proximately 4,000 clinically relevant genes, Hegde said, “it is going to take a tremendous amount of [curation] time,” with many of the genes eventually being discarded because of a lack of evidence. TABLE 2-1 Proposed Gene Classification Criteria Evidence Level Description Criteria 0 Gene of undetermined Undetermined: No reported evidence (no studies available) or Unlikely: Evidence arguing against unlikely significance role in disease 1 Gene of “uncertain signif- Single or few studies, variants, and icance” (studies available families reported AND segregation not but insufficient to draw established OR no human studies re- conclusions) ported but strong animal model data with relevance to human disease 2 Probably disease associated Single or few studies, variants, and families reported AND limited segrega- tion observed 3 Definitely an established Multiple studies, variants, and families disease gene reported AND significant segregation and or strong functional evidence SOURCE: Madhuri Hegde, IOM workshop presentation on February 3, 2014. 

HOW EVIDENCE IS GATHERED AND EVALUATED 15 The Medical Exome Project is working on standardizing assays that will be publicly available for assessing variants, Hegde said. Through the Jain Foundation,2 Hegde and her colleagues have been assessing the bio- logical significance of the variants of unknown significance of dysferlin, a protein involved in muscular dystrophies. By working with the Jain Foundation to acquire clinical data from patients, Hegde’s group is gen- erating information about the variants, which will be submitted to Clin- Var, Hegde said. ACTIONABILITY DETERMINATION Actionability depends on the clinical context in which a genetic test is performed, said Katrina Goddard, senior investigator with the Kaiser Permanente Northwest Center for Health Research, in agreement with Berg. For example, actionability can be different depending on whether testing is done for the purposes of prenatal testing or newborn screening versus being performed during adulthood for disease screening (or pre- conception carrier testing), diagnostic, or monitoring reasons. In the EGAPP working group with which Goddard has been in- volved, genes and conditions related to adult screening and predictive testing were proposed for full evidence review and evaluation based on the recommendations of subject matter experts or on the priorities of funding agencies. Topics then were selected for full review and evalua- tion based on the availability of evidence and other criteria. Actionability was defined for adult incidental findings on the basis of the following three questions, Goddard said: •Is there a practice guideline or systematic review for the genetic condition? • Does the practice guideline or systematic review indicate that the result is actionable in one or more of the following ways? o Patient management o Surveillance or screening o Family management o Circumstances to avoid • Is the result actionable in an undiagnosed adult with the genetic condition? _________________________ 2 Jain Foundation, http://www.jain-foundation.org (accessed April 22, 2014). 

16 GENOME SEQUENCE INFORMATION IN HEALTH CARE DECISIONS The group also decided that some areas were not actionable, such as in- cidental findings that are not related to the indication for testing at the end of a “diagnostic odyssey” where a patient or family has been search- ing for the explanation of a phenotype, findings that are not relevant for all patients in the EGAPP clinical scenario for reproductive decision making, and findings related to personal utility because they may not be actionable in a clinical context. While results may not be clinically actionable, Berg said that just finding a molecular explanation for a patient’s previously unexplained symptoms and ending a diagnostic odyssey can have significant personal utility for the patient and his or her family. The information provided in a report to the patient from a test for such a case may be helpful but the report would not necessarily contain information about variants of uncer- tain significance. For general clinical use, genomic incidental or second- ary findings would not be considered to be part of the routine report. While variants that have sufficient clinical actionability should be part of a routine clinical report, consistent with the ACMG recommendations, other classes of conditions that are clinically valid but have insufficient clinical actionability would be subject to more careful consideration on the part of the patient and clinician about whether a patient would prefer to be given such information, Berg said. Genomic testing at Washington University uses a definition of ac- tionability with components that are very similar to those described by Goddard in that practice guidelines for the genetic condition exist and that professional society practice guidelines recommend action for the purposes of patient management, surveillance or screening, family man- agement, and circumstances to avoid. However, Shashikant Kulkarni, director of cytogenomics and molecular pathology at the Washington University School of Medicine, added that actionability also implies that medical interventions based on new results are effective and that actions are acceptable to the individual in terms of burdens or risks. Actionability in Oncology At Memorial Sloan–Kettering Cancer Center, a consensus-based ap- proach is taken for reviewing potential actionability for genomic find- ings. As Robson explained, a multidisciplinary panel of individuals with expertise in basic science, drug development, clinical trial design, assay development and interpretation, and computational biology and biostatis- tics does a case-by-case evaluation of the evidence. For a tumor specific 

HOW EVIDENCE IS GATHERED AND EVALUATED 17 driver mutation, actionability relates to whether a targeted therapeutic is indicated and available while for a deleterious germline mutation, action- ability relates to mutation penetrance and the efficacy of available pre- ventive medical interventions. Incidental germline findings from tumor profiling are reported only in discovery studies with approval from the institutional review board (Yang et al., 2013) and only with proper consent. A multidisciplinary panel within the institutional review board (IRB) reviews the findings and decides whether to initiate the process of contacting the physician and patient. The actionability criteria remain “a bit fluid,” Robson said, and discussions have centered on what level of risk justifies contact. Very high penetrance predispositions, such as those associated with BRCA1 and BRCA2, are relatively straightforward to make decisions about. But variants that confer more modest risks are more problematic. For example, a test result may not be directly relevant for the person who was tested but could be relevant for family members. This remains an issue even with knowledge gained from sequencing specimens from pa- tients who are deceased. Reaching out to family members in such cir- cumstances can be difficult, but this information can be extremely relevant to their health. Today these decisions are being worked out largely on a case-by-case basis, Robson said. METHODS FOR ASSESSING GENETIC VARIANTS The vast majority of genetic variants have no known clinical rele- vance. The challenge, Berg said, is therefore to parse through variants to determine which ones can be used to inform clinical decisions. This pro- cess requires setting a high bar for which variants from a genome-scale test to report; otherwise, reporting variants with unknown clinical validi- ty (see Box 2-1) or unknown implications for the asymptomatic patient’s health could potentially have negative impacts, such as patient concern about a test result or unnecessary medical costs for testing that may not be clinically useful. Different people may hold different views on the benefits and risks of obtaining genetic information, so individual prefer- ences factor into decisions on whether to return results to patients. Vari- ants reported to physicians and to patients need to be those that can be incorporated into clinical care in an evidence-based fashion, Berg said. 

18 GENOME SEQUENCE INFORMATION IN HEALTH CARE DECISIONS Binning the Genome Berg and colleagues have developed an a priori structured frame- work for handling genomic findings that they described as “binning” the genome (Berg et al., 2011). The framework is organized according to the concepts of clinical validity and clinical utility (see Box 2-1), and the binning is intended to facilitate pre-test informed consent, analysis, and post-test return of results. Use of the framework makes it possible to avoid “one-off” decisions that may not be consistent from one patient to the next. “Ideally,” Berg said, “we should know what we’re going to do with different classes of variants up front, so that when we are analyzing the data and we come across something, we know how we will handle it.” The first step of the binning process is to categorize gene–phenotype pairs into bins according to the clinical actionability elements that Berg described earlier as well as to the risk for psychosocial harm. The second step defines the types of variants that should be reported. For example, known pathogenic variants are reportable, while likely pathogenic variants require further scrutiny before reporting. The third step is to sort an indi- vidual’s variants computationally into predetermined bins. Only variants that meet defined bin criteria are reviewed and reported, and new evi- dence triggers new determinations of how a variant is binned, Berg said. Using this framework, Berg and his colleagues developed three bins (see Figure 2-2). Bin 1 includes variants that meet clinical utility criteria based on the medical literature and are therefore defined as medically actionable; examples include variants that are known or presumed to be deleterious. Bin 2 includes variants that have clinical validity but not clinical utility. Because of the lack of evidence for clinical action, the return of results to patients for these variants will depend on the individ- ual patient’s interest in receiving the information balanced with any un- due stress that may come with learning of the information. The amount of distress that the results could bring to patients is considered by divid- ing Bin 2 into low-, medium-, and high-risk information. Finally, Bin 3 includes all of the variants that have unknown clinical relevance related to phenotype, outcome, or clinical intervention. 

HOW EVIDENCE IS GATHERED AND EVALUATED 19 FIGURE 2-2 Genetic variants can be sorted into three bins depending on the level of clinical utility. NOTE: APOE = apolipoprotein E gene; GWAS = genome-wide association study; Long QT = Long QT syndrome; PGx = pharmacogenomics. SOURCE: Jonathan Berg, IOM workshop presentation, February 3, 2014. Systematic Evidence Gathering and Actionability Determination The recognition of weaknesses in gene–phenotype associations has led those in Rehm’s laboratory to take a more systematic approach to evaluating and scoring the evidence. The approach divides gene– phenotype associations into the following categories: definitive, likely, weak, uncertain or unknown, and no association. Because the numbering systems currently in use vary and can cause confusion, some groups have moved away from labeling these or similar categories with numbers. Although it will take time and significant effort, it is important for those in the field to come to a consensus on a standard system for labeling var- iants, Rehm said. Genes in Rehm’s first category—“definitive”—are in- cluded in predictive tests and can be returned as incidental findings, while genes in the first two categories—“definitive” and “likely”—are included in diagnostic panels where the patient already has a phenotype. Rehm and her colleagues comprise one of three groups that are working together to define the content for newborn genomic screen- ing. The groups have been using the same categorical gene–phenotype association-based approach described earlier, but they are structuring the 

20 GENOME SEQUENCE INFORMATION IN HEALTH CARE DECISIONS data for making decisions about what should be returned to patients with respect to the age of onset of the disease, the inheritance pattern, pene- trance, the phenotype category, and the availability of a clinical test. More than 600 genes have been evaluated, with approximately 3,000 to go, Rehm said. “We hope that by structuring this data, it will allow groups to make cutoffs and decisions about what we think should be returned to individuals.” As part of the Clinical Sequencing Exploratory Research (CSER) consortium,3 Goddard said, the NextGen project is integrating whole- genome sequencing into preconception carrier status testing and evaluat- ing the downstream costs and use versus those of the current standard of care. Through expert analysis, surveys, and focus groups, the project is gathering information from participants about whether they want to re- ceive results for preconception carrier status screening in various health categories (see Table 2-2). The hope is to gain a better understanding of TABLE 2-2 Actionability Categories for Pre-Conception Carrier Status Screening Category Description Shortened Most children do not live past early childhood, even with lifespan medical intervention. Serious Most children will have medical problems that require regular medical visits, daily medications, carefully moni- tored diets, or surgeries; or will have serious problems with learning, vision, hearing, or mobility. Children may have shortened lifespans into early childhood. Mild/moderate Most children will have medical problems that require occasional extra medical visits, occasional medications, a slightly modified diet, or surgery; or will have mild prob- lems with learning, vision, hearing, or mobility. Unpredictable It is difficult to predict the outcome for many children with these conditions. Some children will have more serious versions but others will have a more mild version or no problems at all. Adult onset Few have any symptoms as children, but medical, behav- ioral, vision, or hearing problems may begin as adults. SOURCE: Katrina Goddard, IOM workshop presentation, February 3, 2014. ___________________________________ 3 More information is available at https://cser-consortium.org (accessed May 16, 2014). 

HOW EVIDENCE IS GATHERED AND EVALUATED 21 what types of carrier status results patients will be interested in receiving in the future, Goddard said. Three-Stage Evaluation Process The evaluation process used to determine which results to return to patients for projects such as NextGen consists of three stages, Goddard said. The first stage is a preliminary assessment to determine whether sufficient information is available to do a full review. In this stage, the actionability concepts described earlier as well as variant penetrance and whether the condition is a significant and important health problem are considered. If the condition does not meet one of these criteria, a full re- view will not be undertaken. The objective of this stage is to provide a rapid mechanism for determining which conditions do not have sufficient information to warrant further evaluation. In the second stage, an evidence-based process for each specific gene–phenotype pair is documented in a summary report. Reproducible search methods are used to identify studies and data, which are restricted to systematic reviews, evidence-based practice guidelines, or expert consensus- based practice guidelines. Each gene–phenotype pair is summarized in about two pages, with a goal of keeping the summaries brief, transparent, and reproducible. “This is not a comprehensive method, and we are aware of that, but that was [a] pragmatic choice,” Goddard said. To as- sess the data, it is sorted into evidence tiers (see Box 2-2) to address ex- pected disagreement among sources and to signal the overall quality of sources. Quality ratings are used as tie-breakers for conflicting evidence at the same tier. In Stage 3 the summary produced in Stage 2 is used by a decision-making group—whether EGAPP or another group—to make recommendations. BOX 2-2 Tiers of Evidence During the data assessment stage, Goddard said, information is categorized into tiers of evidence to classify the source of data and its quality. Those tiers are: • First Tier: Evidence from a systematic review, meta-analysis, or clinical practice guideline based on a systematic reviewa of the objectives, methods, findings, and other criteria. 

22 GENOME SEQUENCE INFORMATION IN HEALTH CARE DECISIONS • Second Tier: Evidence from clinical practice guidelines or broad-based expert consensus with some level of evidence review, but using unclear methods or using sources that were not systematically identified. • Third Tier: Evidence from another source with non-systematic review of evidence (e.g., GeneTest Reviews, OrphaNet, Clini- cal Utility Gene Cards, and the opinion of fewer than five experts), with additional primary literature cited. • Fourth Tier: Evidence from another source with non-systematic review of evidence (e.g., GeneTest Reviews, OrphaNet, Clini- cal Utility Gene Cards, and the opinion of less than five experts) lacking citation of primary data sources. ______________________ a Systematic review according to the Cochrane Handbook. For more information, see http://handbook.cochrane.org (accessed May 6, 2014). Methods for Variant Annotation in Cancer In 2011 the Washington University School of Medicine began offer- ing next-generation sequencing in addition to the other genomic tests it performs. Because of the school’s particular expertise, it focused on can- cer genomics. Curating genomic variants has proven to be a huge task, Kulkarni said. “If the germline is that difficult, consider how difficult cancer variation data curation could be.” A bioinformatics team is needed to analyze the sequencing data after it is generated, Kulkarni said. Even in the case with a 42-cancer gene panel, there is too much information to process manually, so a software system was designed to perform base calling, alignment, variant calling, and genome annotation in a semi-automated way. In the first phase, cus- tom scripted software programs facilitate an automated step in which the data are compared against publicly available gene information and clini- cal and mutation terms. Criteria for the searches are set such that relevant papers must contain human data and one or more mutations and must describe a clinical outcome. Where there are commonalities in these three areas, an annotation worthiness score is generated for each variant, and the information is deposited into a searchable spreadsheet for the next phase. Following this automated process, an external group of six annota- tors reviews the data over several months. A second evaluation is con- ducted by the clinical fellows and attending physicians at Washington 

HOW EVIDENCE IS GATHERED AND EVALUATED 23 University School of Medicine, Kulkarni said. Variants are classified into five levels, which are based on the ACMG guidelines: • Level 1—Predictive or prognostic in tumor type (includes inher- ited cancer susceptibility variants). • Level 2—Predictive or prognostic in another tumor type or types. • Level 3—Reported in cancer or other disease. • Level 4—Variant of unknown significance. • Level 5—Known polymorphism. The data are then made available on a wiki-based user interface where other clinical fellows and attending physicians could review and modify information about the variants. Presentation of the results sorts the vari- ants by level, with an interpretation of the role of the variants and refer- ences to the medical literature. The resulting report provides information about the variant and related data, as well as the ability to examine each step of the variant annotation filtering process. During monthly meetings, new evidence is collectively reviewed. “This is a very comprehensive effort,” Kulkarni said, “and it’s ongoing because there is a lot of new information coming out . . . on these cancer genes.” Since March 2012 about 1,500 clinical tests have been ordered, not including those from clinical trials, Kulkarni said. The tumor types tested cover a broad range, including brain, colorectal, lung, pancreatic, and sarcomas and the initial findings suggest that about 45 percent of se- quenced cases have specific actionable mutations in targetable genes, including EGFR, KIT, KRAS, and PIK3CA, he said. Challenges Workshop participants described a number of challenges for the fu- ture. For example, more than 50 million genetic variants have been found in the human genome, Rehm said, with many of them unique to individ- uals, and misinterpretation of these variants can affect clinical care and study outcomes. The vast majority of the variants seen in clinical testing and research studies are rare, which makes it difficult to generate sufficient evidence to make a claim. For example, Rehm said, diagnostic testing of 15,000 probands for a variety of hereditary disorders and for somatic cancer re- vealed about 1,600 variants reported as either pathogenic or likely patho- 

24 GENOME SEQUENCE INFORMATION IN HEALTH CARE DECISIONS genic; in this case, 68 percent of those variants were seen only once, and 96 percent of the variants were seen fewer than 10 times. Based on test- ing conducted in Rehm’s laboratory, about one-third of the variants are categorized as having uncertain significance. “Our community will need to develop better approaches to evaluating these variants and their im- pact,” she said. The challenge is even greater for the return of incidental findings from exome or genomic sequencing. Rehm cited data from the MedSeq project that indicated that each patient in the study had 20 to 40 variants that had been published as disease causing or as pathogenic (Vassy et al., 2014). However, when these variants were reviewed with strict criteria for pathogenicity, 97 percent were excluded, most of them being uncer- tain, and many having clear evidence for being benign. Similarly, 30 to 50 variants were linked with loss of function in disease-associated genes in a MedSeq study, but strict review of the evidence determined that 94 percent of these variants did not meet the criteria for pathogenicity. “This is a lot of work with a low yield as we look through patients’ genomes,” Rehm said. The group is now working to develop better guidance on the inter- pretation of sequence variants, and a draft of that guidance is expected to be ready for the community in the coming months, Rehm said. A further complication, she noted, is that the evidence constantly changes, so that new information needs to be returned to patients tested in the past. Berg agreed, noting that information will need to be updated over time. A sys- tem developed by Partners HealthCare, which features e-mails sent to physicians with new information that can be inserted into patients’ elec- tronic health records (EHRs), has been “very effective,” Rehm said. Future challenges include continuing to seek the right balance be- tween brevity and comprehensiveness of the assessments and determin- ing whether to relate variants on genes to conditions, Goddard said. There are certain challenges that come with Kulkarni’s approach to assessing variants for clinical use. It would be possible to scale the sys- tem to a large number of genes and variants, Kulkarni said, although it requires a large amount of upfront work. For example, a team of six annotators worked for 6 months on the initial 28-gene-variant curation, sort- ing out the variants based on given criteria. Additionally, both algorithmic and knowledge-based variant curation methods are necessary for clinical interpretation, and annotating and keeping up with variant management is expensive. As a result, there is an urgent need for implementing uni- versal standards and a variant resource database, he said. Fortunately, 

HOW EVIDENCE IS GATHERED AND EVALUATED 25 this work does not all have to be done by one organization, but rather can be done collaboratively. Kulkarni observed that ethnicity is considered in the review of anno- tations, but more data need to be generated from different racial and eth- nic groups. He also noted that data is available through the International Consortium of Cancer Genomics, which has data from different popula- tion groups, though in this area, too, more information from different groups is needed. Goddard identified concerns over extrapolating data from high-risk populations to the general population and considering the variation among individuals. For example, X-linked conditions are more relevant for males, and conditions with variable penetrance have differ- ent risks depending on the strength of family history. UPDATING EVIDENCE New data is often produced that can trigger the reinterpretation of a variant–disease link, but a key question is how those data should be communicated broadly so that clinicians and laboratories are working with current information. Perhaps, Berg said, this needs to be an ongoing process such that new evidence would initiate new reviews through a system that could be triggered by inquiries from physicians, researchers, or patients. The reclassification strategy developed at Emory University was designed so that clinicians could generate or point to evidence that a gene is probably not related to a disease, Hegde said. “The labs cannot do it on their own,” she said. “The number of variants of unknown sig- nificance has grown so much [that] you can imagine how much time it takes for the lab to go through all those variants and reclassify them. It is a huge help for us if the clinicians actually approach us.” The ACMG guidelines require that a testing laboratory make an ef- fort to contact physicians who previously tested patients in the event that new information changes the initial clinical interpretation of the se- quence variant, Hegde said. To fulfill this guideline, the Emory Genetics Laboratory has set up a Web-based system to release updates on all the variants seen and analyzed by the laboratory (Bean et al., 2013). As vari- ants are reclassified, the system automatically scans the internal database and identifies previously affected patients. It then sends an alert to labor- atory directors and issues amended reports. In addition, the system re- classifies variants based on outside requests, with the information then returned to clinical targets. 

26 GENOME SEQUENCE INFORMATION IN HEALTH CARE DECISIONS Asking physicians to provide clinical data can be challenging be- cause they are often too busy to fill out data forms, Rehm said. Neverthe- less, these data can be extremely important—for example, when an affected patient in a family tests negative for a variant. “As a community, we need to underscore the importance of the dynamic relationship be- tween the lab and the physician if we hope to improve our understanding of genomic variation,” Rehm said. Berg agreed, noting that the paper- work needed to report on a variant can seem particularly extensive in the context of a pressured clinic. “There needs to be better mechanisms for physicians to be able to supply the phenotypic information to the labs in a structured format,” he said, “because I think that adds to the specificity of the analysis.” There are also challenges of how much clinical infor- mation can be shared because of Health Insurance Portability and Ac- countability Act (HIPAA) issues. Data Quality Related to the issue of updating variant information is a concern over the reproducibility of data in the literature. When published studies do not include complete procedures or the primary data are not accessible, it makes evaluating the quality of the data difficult. Rehm noted that the ClinGen project and the ClinVar database are creating a structured mechanism that requires the authors of a paper to make the raw data available so that their results can be verified and extended in a transpar- ent way. But published associations that are inaccurate remain an unre- solved problem, because the act of publishing data is often misconstrued as providing a quality piece of evidence in and of itself. Similarly, Hegde observed that important data may not be available in a database, may be out of date, or may be expensive to access. ClinVar will be important for that reason because it will be a free and open database. Other efforts, such as the Global Alliance for Genomics and Health, are also working on ways to responsibly share genomic and clinical information across groups (Callaway, 2014). Standardized methods for rating quality in the field of genomics do not yet exist, Goddard said. Berg, too, pointed out that “there is really no specific definition of what constitutes a proven gene–phenotype association. There are certainly genes that we know because the evidence is compelling and overwhelming, but then you get to the many gene–phenotype associa- tions based on a couple of case reports or a handful of families, and there are no specific guidelines to say this is where you draw the line.” 

HOW EVIDENCE IS GATHERED AND EVALUATED 27 There is also the issue of an evidence gap when there is no synthe- sized evidence in the literature to rely on for evaluating gene–phenotype pairs. When this occurs, we need to prioritize our reviews, Goddard said, so that researchers and clinicians do not spend too much time addressing gene–phenotype associations that have no systematic review or practice guideline that can be referred to. Kulkarni added that an ideal situation dealing with these evidence gaps would be for researchers to have sufficient tools to model and as- sess disease progression, clonal evolution, and response to therapies. Maybe in 5 to 10 years, he said, we will have a Clinical Laboratory Im- provement Amendments–certified mouse facility for treating primary, secondary, and metastatic tumors from the same patient and then report- ing the results back to the clinic. One collaborative approach to assessing data has been taken by the Evidence-based Network for the Interpretation of Germline Mutant Al- leles (ENIGMA) Consortium,4 an international consortium that is taking a multidisciplinary, multi-institutional approach to understanding the involvement of all variants of uncertain significance in BRCA1 and BRCA2 that may be related to breast and ovarian cancers, Robson said. The members of this organization have pooled their clinical expertise, their data, and their laboratory capabilities to resolve issues of evidence involving breast cancer predisposition genes. TECHNICAL ISSUES There are also technical issues with DNA sequencing that make interpretation of evidence difficult. Rehm said out that no whole-genome sequencing effort today is complete in covering all regions of the genome or detecting all types of variation including substitutions, insertion– deletions, copy number variations, structural variants, and gene fusions. In addition, the bioinformatics techniques applied to raw sequence data in different labs can produce differences in sequences. Finally, the interpre- tive process can vary from laboratory to laboratory, resulting in different __________________________ 4 Evidence-based Network for the Interpretation of Germline Mutant Alleles Consorti- um, http://enigmaconsortium.org (accessed April 22, 2014). 

28 GENOME SEQUENCE INFORMATION IN HEALTH CARE DECISIONS interpretations of results due to variation in data filtering, alignment, var- iant calling, quality thresholds, and annotation. “There are so many levels today that are non-standardized and distinct that you will often get different results for different reasons,” Rehm said. “Those are all aspects of this pro- cess that we, as a community . . . need to address.” On the subject of assessing the comparability of genome sequencing results from different laboratories, Hegde observed that the College of American Pathologists has a next-generation sequencing committee that is working on a database for proficiency testing and cross-validation of variant detection. Select variants will be confirmed by Sanger sequenc- ing, and then the data will be shared more broadly. “This is very im- portant because, as we just heard, these platform differences are significant,” Hegde said. Rehm added that the ClinVar database will help in this regard because it will provide transparency where interpretations differ. When data are submitted to ClinVar, a quality control report is generated that describes where the submitter’s variant interpretations differ from those that are already in the system. There is also a disadvantage for clinical laboratories in that they gen- erally cannot do functional assays for the biological relevance of variants they observe, Hegde said. Instead, they need to connect with a research laboratory to work on a particular gene or disease. But researchers who have worked on a gene or disease in the past may no longer have the funding to do more research in that area. “The question of how you can do a biological relevance assay is a big one,” Hegde said. Berg agreed, adding that there is an opportunity here for researchers who have robust assays and who can reproducibly separate benign from pathogenic vari- ants in order to overcome these types of technical challenges. However, resources would be necessary to complete the work, he said. Berg acknowledged the complexity related to genetic variants. Vari- ants are interpreted as being pathogenic or benign with respect to particu- lar disorders, but a given gene can be involved in multiple disorders. “Figuring out how you define pathogenicity means you have to explicitly link the pathogenicity assertion to the gene and the disease that is related to it,” Berg said. Additionally, curation of variants of unknown signifi- cance can be more than a technical challenge because the level of under- standing of the basic biology of the gene, protein, and pathway involved can influence how variants of unknown significance are classified. 

HOW EVIDENCE IS GATHERED AND EVALUATED 29 Electronic Health Records Genomic information needs to be available in some way, but various barriers exist to making it available through an EHR, such as the capacity of the record, Berg said. Rehm noted that the issue has been discussed in the past and that the thinking has been that the information should not be in an EHR. Only a percentage of the variants in genomic data have been rigorously confirmed, and these can be interpreted and put in a report that goes in the EHR. “The consensus right now is you can update [in- formation] that you’ve already put there,” she said, “but we are not ready to expose the 5 million variants in a genome to an environment where a clinician queries that, finds a variant, and goes and treats a patient when in fact that was an incorrect call.” This view may change as the technol- ogy advances, she added, “but we are not there yet.”