3

IRIS Assessment Development Plans: Evidence Evaluation, Systematic Review, and Meta-analysis

EVIDENCE EVALUATION

The US Environmental Protection Agency (EPA) draft plans for the inorganic arsenic assessment indicate that a broad literature search and screening process will be used to identify health effects that have been studied in relation to arsenic (EPA 2013c), and a preliminary draft of these efforts was provided to the committee (EPA 2013d). The scientific literature will be organized into summary tables by health-effect category to get an overview of the types and numbers of studies available for each health effect. Evidence tables will be used to provide more specific information on exposures, outcomes, and evaluation methods. Overall, the committee found that the draft plans and example tables captured the salient categories of information with respect to epidemiologic studies. However, no descriptions or examples were provided for organizing the evidence from animal and in vitro studies. Such information will be particularly important for mode-of-action analyses (discussed in Chapter 6).

Evidence Tables for Animal and In Vitro Studies

Understanding exposure data for each health outcome across the dose range is important and is an acknowledged difficulty EPA will face with the existing arsenic database. The draft plans for the inorganic arsenic assessment describes Evidence Tables by Health Effect Category that record exposure and outcome information for human studies. Evidence tables for animal and in vitro studies are also needed for mode-of-action analyses, described in Chapter 6 of this report. Information for the mode-of-action analyses will come in part from studies to be entered in the evidence tables, and the committee recommends that EPA design these tables, or additional tables, early in its evaluation process, to avoid unnecessary duplication of effort in evaluating the literature and populating the tables.

Evidence tables for animal and in vitro studies are not described in EPA’s draft work plan. Information recorded should include details of dosing or exposure (such as form of arsenic, concentration or dose, route of exposure, and duration of exposure); the species and strain of animal, tissue or cell line; and all end points measured. For mechanistic studies, end points related to mechanisms of effects or mode of action should be recorded for each exposure reported. For all studies, observations of morbidity and mortality or biomarkers of cell death or impaired cellular integrity should be recorded. Such information can facilitate comparing dose-response relationships among different routes of exposure, cell lines, and especially animal models, given the marked species differences in arsenic kinetics and toxicity discussed in Chapter 2 and in the committee’s workshop.

Analysis of Gene–Expression and Genomic Data

The dataset for arsenic includes genomic data, such as gene expression data, variability based on DNA polymorphisms, and differences in epigenetic control of expression. Some of the studies have used



The National Academies | 500 Fifth St. N.W. | Washington, D.C. 20001
Copyright © National Academy of Sciences. All rights reserved.
Terms of Use and Privacy Statement



Below are the first 10 and last 10 pages of uncorrected machine-read text (when available) of this chapter, followed by the top 30 algorithmically extracted key phrases from the chapter as a whole.
Intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text on the opening pages of each chapter. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.

Do not use for reproduction, copying, pasting, or reading; exclusively for search engines.

OCR for page 18
3 IRIS Assessment Development Plans: Evidence Evaluation, Systematic Review, and Meta-analysis EVIDENCE EVALUATION The US Environmental Protection Agency (EPA) draft plans for the inorganic arsenic assessment indicate that a broad literature search and screening process will be used to identify health effects that have been studied in relation to arsenic (EPA 2013c), and a preliminary draft of these efforts was provid- ed to the committee (EPA 2013d). The scientific literature will be organized into summary tables by health-effect category to get an overview of the types and numbers of studies available for each health effect. Evidence tables will be used to provide more specific information on exposures, outcomes, and evaluation methods. Overall, the committee found that the draft plans and example tables captured the salient categories of information with respect to epidemiologic studies. However, no descriptions or ex- amples were provided for organizing the evidence from animal and in vitro studies. Such information will be particularly important for mode-of-action analyses (discussed in Chapter 6). Evidence Tables for Animal and In Vitro Studies Understanding exposure data for each health outcome across the dose range is important and is an acknowledged difficulty EPA will face with the existing arsenic database. The draft plans for the inorganic arsenic assessment describes Evidence Tables by Health Effect Category that record exposure and outcome information for human studies. Evidence tables for animal and in vitro studies are also needed for mode-of- action analyses, described in Chapter 6 of this report. Information for the mode-of-action analyses will come in part from studies to be entered in the evidence tables, and the committee recommends that EPA design these tables, or additional tables, early in its evaluation process, to avoid unnecessary duplication of effort in evaluating the literature and populating the tables. Evidence tables for animal and in vitro studies are not described in EPA’s draft work plan. Infor- mation recorded should include details of dosing or exposure (such as form of arsenic, concentration or dose, route of exposure, and duration of exposure); the species and strain of animal, tissue or cell line; and all end points measured. For mechanistic studies, end points related to mechanisms of effects or mode of action should be recorded for each exposure reported. For all studies, observations of morbidity and mor- tality or biomarkers of cell death or impaired cellular integrity should be recorded. Such information can facilitate comparing dose-response relationships among different routes of exposure, cell lines, and espe- cially animal models, given the marked species differences in arsenic kinetics and toxicity discussed in Chapter 2 and in the committee’s workshop. Analysis of Gene–Expression and Genomic Data The dataset for arsenic includes genomic data, such as gene expression data, variability based on DNA polymorphisms, and differences in epigenetic control of expression. Some of the studies have used 18

OCR for page 18
IRIS Assessment Development Plans 19 high-throughput technologies. Microarray technology is typically used for assessing gene expression and specialized arrays are available for detecting polymorphic and isoform variation. Gene expression meas- ured by next–generation sequencing (NGS), often referred to as RNA-seq, is becoming more affordable and therefore its use is increasing. RNA-seq has a higher dynamic range than microarray and allows de- tection of polymorphisms and splice variants. NGS can be used to study epigenetic regulation of DNA expression. In reviewing studies reporting microarray or NGS data, EPA should consider the strengths and limitations inherent to the different high-throughput methods. In evaluating gene–expression and genomic data, EPA should give close attention to the data pro- cessing steps and how they influence the results. For both technologies, raw expression values, extracted from image files, need to be adjusted so that the expression values reflect the abundance of genes in the sample. This preprocessing of microarray data has been reviewed by Allison et al. (2006)and Bolstrad et al. (2006). Various techniques for normalizing the expression values among microarray chips in an exper- iment have been developed and have been subjected to comparisons by statisticians and bioinformaticians (e.g., Qin et al. 2013). Hansen et al. (2012) discuss RNA-seq preprocessing. EPA’s review could also as- sess how the alignment of the reads to the reference genome would influence gene expression results; this is discussed by Garber et al. (2011). Because batch effects, such as daily variations in ozone effects on dye and drift of scanner performance or between sequencer flow cells, are not eliminated by normaliza- tion, EPA should consider whether the statistical analysis considers batch effects appropriately. Both technologies measure the expression of tens of thousands of genes simultaneously. When there are tens of thousands of comparisons and a paucity of replicates relative to the number of comparisons, sophisticated techniques are needed to appropriately apportion variance among fixed effects and random effects and to control the type-1 error rate for the multiple comparisons. In addition, expressions of indi- vidual genes are not necessarily independent of each other; that is, a given gene can increase the expres- sion of some genes and decrease the expression of others if they belong to the same biological pathway. Thus, analysis of gene networks are now commonly used. In general, classical procedures for controlling overall error are over strict and tests of false discovery rate (Benjamini and Hochberg 1995) are often used in identifying differentially expressed genes. Other approaches use information from all genes on the chip to calculate moderated t-statistics. Two widely used methods are linear models for microarray data (limma, a Bayesian method, Smyth 2004) and statistical analysis of microarrays (SAM, a permutation method, Tusher et al. 2001). These methods have been adapted for NGS and were compared by Sonenson and Delorenzi (2013). EPA should review the statistical analyses used, ensure that they are sufficient to support conclusions reached, and note in the evidence tables details of the statistical analysis and limita- tions on the conclusions. Many scientists confirm findings from high-throughput methods using other techniques, such as polymerase-chain-reaction or protein-based measurements. For conclusions based solely on expression results, the committee recommends a detailed analysis of the data supporting the conclusion. This should include thorough review of the effectiveness of the preprocessing and the statistical analysis and will re- quire access to the raw data. SYSTEMATIC REVIEW AND META-ANALYSIS EPA’s draft plans indicate that the hazard-identification process will generate questions specific to the toxicologic review, and the questions will be addressed in systematic reviews. EPA expects questions to be generated about aspects related to adverse-outcome pathways and dose–response characterization. For example, EPA indicates that systematic reviews of dose–response analyses will be used to identify chemical and nonchemical stressors that might contribute to the health outcome under consideration and to identify underlying disease processes that might be added to or increased by exposure to inorganic ar- senic. Those uses were recommended by the National Research Council committee that wrote Science and Decisions (NRC 2009) and were followed up on in Chapter 7 of Review of the Environmental Protec- tion Agency’s Draft IRIS Assessment of Formaldehyde (NRC 2011).

OCR for page 18
20 Critical Aspects of EPA’s IRIS Assessment of Inorganic Arsenic Efforts are under way to develop EPA guidance on performing systematic reviews. EPA held a workshop on August 26, 2013, that was devoted to that topic. Another National Research Council com- mittee is evaluating such improvements of the IRIS process and is expected to issue a report in the first quarter of 2014. In light of those activities, the present committee focused its efforts on identifying issues specific to inorganic arsenic in the performance of systematic reviews rather than on specific procedural aspects of such reviews. Systematic reviews and meta-analyses are widely used in medicine, epidemiology, and public health to evaluate and synthesize scientific evidence objectively (Egger et al. 2001). Systematic reviews comprehen- sively identify and evaluate the available evidence with a transparent process. They are used to reach evi- dence-based conclusions, to clarify the need for additional research, and potentially to summarize the evi- dence quantitatively with meta-analysis (Egger et al. 2001; Porta 2008; Rooney 2013). Meta-analysis is a statistical technique that can be used to pool evidence from systematic reviews if various conditions are met. To minimize uncertainty in original research, systematic reviews follow objective criteria for collecting and evaluating evidence in a comprehensive manner. A priori decisions and a predefined protocol are critical during the systematic review process (Berlin and Colditz 1999; Dickersin 2002); the protocol should de- scribe the following steps: the research question, the search strategy and data sources, the study inclusion and exclusion criteria, the data to be abstracted and derived from the original studies (such as sample size, exposure and outcome assessment methods, and confounders evaluated), the criteria and methods for pool- ing effect estimates and measures of variability among studies. Systematic reviews and meta-analyses need to be replicable; other investigators following the same steps should be able to identify the same articles, abstract the same data, and reach similar conclusions. Originally developed in the field of clinical trials, sys- tematic reviews and meta-analyses are commonly used to summarize epidemiologic research, including risk factors for a disease and dose–response relationships, but can also be used to summarize disease mecha- nisms (Dickersin 2002). The use of systematic reviews to evaluate experimental evidence is increasing, and some aspects are still in development. For instance, there is no clear best practice for evaluating the risk of bias in in vitro studies (Rooney 2013). Chapter 7 describes in more detail the role of meta-analyses in the evaluation of dose–response relationships and how they can be used for developing risk-based estimates. The role of systematic reviews and meta-analyses in the regulatory process for drugs, foods, and medical devices has long been recognized (Berlin and Colditz 1999), and their relevance for environmental policy, including both epidemiologic and experimental research, is increasing (NRC 2009, 2011). The Office of Health Assessment and Translation of the National Toxicology Program has developed an initiative to in- corporate systematic reviews and evidence integration for literature-based environmental health assessment (Rooney 2013). Given the large number of studies, evaluating the epidemiologic and experimental literature on arse- nic health effects can be a daunting task. To guide the process and ensure high quality, it is critical to es- tablish clearly the research questions that need to be answered. Examples of research questions about epi- demiologic and experimental data, using cardiovascular disease as the health outcome, are shown in Box 4. EPA plans to include systematic review principles in performing its Integrated Risk Information Sys- tem (IRIS) assessment of inorganic arsenic will be a welcome improvement. EPA has some experience with performing systematic reviews and meta-analyses for IRIS assess- ments of other chemicals, such as trichloroethylene (EPA 2011). Lessons learned in performing those anal- yses might guide the systematic reviews and meta-analyses for inorganic arsenic. Also, a number of system- atic reviews have summarized the association between arsenic and several health end points, including cancer (Celik et al. 2008), cardiovascular disease (Navas-Acien et al. 2005; C.H. Wang et al. 2007; Moon et al. 2012), diabetes (Navas-Acien et al. 2006; Maull et al. 2012), hypertension (Abhyankar et al. 2012), and neurodevelopment (Rodriguez-Barranco et al. 2013). Only a few of those reviews have attempted to gener- ate pooled estimates of the association between arsenic and disease (Navas-Acien et al. 2006; Moon et al. 2012; Rodriguez-Barranco et al. 2013). Some of the systematic reviews of arsenic and health end points can be useful starting points, including their search strategies, table format, data collection, and quality evalua- tion forms. It is likely, however, that none of them will be sufficient for EPA’s purposes. First, the article

OCR for page 18
IRIS Assessment Development Plans 21 BOX 4 Examples of Research Questions about Epidemiologic and Experimental Data Examples of research questions that can be used to guide systematic reviews for the evaluation of arsenic-related health effects with overall clinical cardiovascular-disease mortality as the effect. They in- clude questions related to hazard identification, dose–response analysis, and mode-of-action evaluation. Once the research questions have been established, a protocol should be developed to ensure that all relevant articles answering each of the questions are identified. Although the questions were developed by using cardiovascular disease as an example, other health end points and mechanisms could have been selected. 1. Is arsenic exposure, measured before the development of cardiovascular disease at the individual level, associated with overall clinical cardiovascular disease mortality in populations exposed to arsenic in drinking water at less than 100 µg/L? 2. What is the dose–response relationship between arsenic and cardiovascular-disease mortality throughout the range of arsenic exposure that is relevant for human populations (for example, arsenic in drinking water ranging from less than 10 µg/L to over 100 µg/L)? 3. What is the dose–response relationship between arsenic exposure and mechanisms of generation of reactive oxidant species or inflammation in relevant animal models and cellular systems that can be used to evaluate the mode of action of arsenic-related cardiovascular disease? search will need to be updated. Second, the reviews categorized the studies into two groups (studies with arsenic concentrations less than or greater than 100 µg/L). However, a few studies conducted in populations exposed to high arsenic concentrations in drinking water had some categories at concentrations below 100 µg/L, and the results for those categories can be included in the analyses at low to moderate arsenic expo- sure. General limitations of the conduct and interpretation of meta-analyses for arsenic health effects have been the substantial heterogeneity among studies, methodologic limitations in outcome and exposure as- sessment, the use of different biomarkers, the temporality of an association, adjustment for relevant con- founders, and the potential for publication bias. Heterogeneity, however, is relatively common in observa- tional studies; it should be evaluated but should not necessarily impede data-pooling. Sources of heterogeneity could be evaluated with meta-regression and analysis of influential studies. Publication bias is a concern in the conduct of systematic reviews, and different strategies are available to evaluate it, in- cluding the use of funnel plots. The increasing number of published prospective studies that have adequate exposure and outcome assessment (for example, see section “Cardiovascular Disease” in Chapter 4) provides an opportunity for EPA to apply meta-analysis techniques to estimate the association between arsenic and relevant health end points. In selecting the studies to be included in the meta-analyses, EPA should systematically search for studies with individual measures of arsenic exposure (preferably with biomarker measurements), measurement of arsenic that precedes the outcome, and low to moderate exposure to inorganic arsenic (less than 100 µg/L in drinking water). Meta-analysis can then be performed if there are three or more peer reviewed studies available on the outcome of interest. For dose-response meta-analysis, studies will need to have at least three or more doses tested. As part of the systematic review process, study quality (risk of bias) needs to be evaluated using established guidelines for epidemiologic studies (Viswanathan et al. 2012; Rooney 2013). Elements in the evaluation of study quality include the assessment of study outcome based on standardized definitions, participation rate, adjustment of association for relevant con- founders, and other considerations that will depend on study design. Meta-analyses of published studies are sometimes limited by what is available in the aggregated da- ta. Individual-level meta-analyses—meta-analyses that use raw data from the original studies (Riley et al. 2010)—provide an opportunity to answer specific questions and to use common definitions across stud- ies. Major limitations of individual-level meta-analyses, however, are the requirement of active participa- tion by the original investigators, of standardization across studies, and of additional resources and fund-

OCR for page 18
22 Critical Aspects of EPA’s IRIS Assessment of Inorganic Arsenic ing to conduct the studies. For the purpose of the IRIS assessment, given limited resources and the num- ber and quality of published studies of some health end points associated with low to moderate arsenic exposure, conducting meta-analyses of aggregated data from published studies is an appropriate alterna- tive to collecting and analyzing raw data for individual-level meta-analysis.