National Academies Press: OpenBook

Critical Aspects of EPA's IRIS Assessment of Inorganic Arsenic: Interim Report (2013)

Chapter: 3 IRIS Assessment Development Plans: Evidence Evaluation, Systematic Review, and Meta-analysis

« Previous: 2 Exposure Considerations
Suggested Citation:"3 IRIS Assessment Development Plans: Evidence Evaluation, Systematic Review, and Meta-analysis." National Research Council. 2013. Critical Aspects of EPA's IRIS Assessment of Inorganic Arsenic: Interim Report. Washington, DC: The National Academies Press. doi: 10.17226/18594.
×

3

IRIS Assessment Development Plans: Evidence Evaluation, Systematic Review, and Meta-analysis

EVIDENCE EVALUATION

The US Environmental Protection Agency (EPA) draft plans for the inorganic arsenic assessment indicate that a broad literature search and screening process will be used to identify health effects that have been studied in relation to arsenic (EPA 2013c), and a preliminary draft of these efforts was provided to the committee (EPA 2013d). The scientific literature will be organized into summary tables by health-effect category to get an overview of the types and numbers of studies available for each health effect. Evidence tables will be used to provide more specific information on exposures, outcomes, and evaluation methods. Overall, the committee found that the draft plans and example tables captured the salient categories of information with respect to epidemiologic studies. However, no descriptions or examples were provided for organizing the evidence from animal and in vitro studies. Such information will be particularly important for mode-of-action analyses (discussed in Chapter 6).

Evidence Tables for Animal and In Vitro Studies

Understanding exposure data for each health outcome across the dose range is important and is an acknowledged difficulty EPA will face with the existing arsenic database. The draft plans for the inorganic arsenic assessment describes Evidence Tables by Health Effect Category that record exposure and outcome information for human studies. Evidence tables for animal and in vitro studies are also needed for mode-of-action analyses, described in Chapter 6 of this report. Information for the mode-of-action analyses will come in part from studies to be entered in the evidence tables, and the committee recommends that EPA design these tables, or additional tables, early in its evaluation process, to avoid unnecessary duplication of effort in evaluating the literature and populating the tables.

Evidence tables for animal and in vitro studies are not described in EPA’s draft work plan. Information recorded should include details of dosing or exposure (such as form of arsenic, concentration or dose, route of exposure, and duration of exposure); the species and strain of animal, tissue or cell line; and all end points measured. For mechanistic studies, end points related to mechanisms of effects or mode of action should be recorded for each exposure reported. For all studies, observations of morbidity and mortality or biomarkers of cell death or impaired cellular integrity should be recorded. Such information can facilitate comparing dose-response relationships among different routes of exposure, cell lines, and especially animal models, given the marked species differences in arsenic kinetics and toxicity discussed in Chapter 2 and in the committee’s workshop.

Analysis of Gene–Expression and Genomic Data

The dataset for arsenic includes genomic data, such as gene expression data, variability based on DNA polymorphisms, and differences in epigenetic control of expression. Some of the studies have used

Suggested Citation:"3 IRIS Assessment Development Plans: Evidence Evaluation, Systematic Review, and Meta-analysis." National Research Council. 2013. Critical Aspects of EPA's IRIS Assessment of Inorganic Arsenic: Interim Report. Washington, DC: The National Academies Press. doi: 10.17226/18594.
×

high-throughput technologies. Microarray technology is typically used for assessing gene expression and specialized arrays are available for detecting polymorphic and isoform variation. Gene expression measured by next–generation sequencing (NGS), often referred to as RNA-seq, is becoming more affordable and therefore its use is increasing. RNA-seq has a higher dynamic range than microarray and allows detection of polymorphisms and splice variants. NGS can be used to study epigenetic regulation of DNA expression. In reviewing studies reporting microarray or NGS data, EPA should consider the strengths and limitations inherent to the different high-throughput methods.

In evaluating gene–expression and genomic data, EPA should give close attention to the data processing steps and how they influence the results. For both technologies, raw expression values, extracted from image files, need to be adjusted so that the expression values reflect the abundance of genes in the sample. This preprocessing of microarray data has been reviewed by Allison et al. (2006)and Bolstrad et al. (2006). Various techniques for normalizing the expression values among microarray chips in an experiment have been developed and have been subjected to comparisons by statisticians and bioinformaticians (e.g., Qin et al. 2013). Hansen et al. (2012) discuss RNA-seq preprocessing. EPA’s review could also assess how the alignment of the reads to the reference genome would influence gene expression results; this is discussed by Garber et al. (2011). Because batch effects, such as daily variations in ozone effects on dye and drift of scanner performance or between sequencer flow cells, are not eliminated by normalization, EPA should consider whether the statistical analysis considers batch effects appropriately.

Both technologies measure the expression of tens of thousands of genes simultaneously. When there are tens of thousands of comparisons and a paucity of replicates relative to the number of comparisons, sophisticated techniques are needed to appropriately apportion variance among fixed effects and random effects and to control the type-1 error rate for the multiple comparisons. In addition, expressions of individual genes are not necessarily independent of each other; that is, a given gene can increase the expression of some genes and decrease the expression of others if they belong to the same biological pathway. Thus, analysis of gene networks are now commonly used. In general, classical procedures for controlling overall error are over strict and tests of false discovery rate (Benjamini and Hochberg 1995) are often used in identifying differentially expressed genes. Other approaches use information from all genes on the chip to calculate moderated t-statistics. Two widely used methods are linear models for microarray data (limma, a Bayesian method, Smyth 2004) and statistical analysis of microarrays (SAM, a permutation method, Tusher et al. 2001). These methods have been adapted for NGS and were compared by Sonenson and Delorenzi (2013). EPA should review the statistical analyses used, ensure that they are sufficient to support conclusions reached, and note in the evidence tables details of the statistical analysis and limitations on the conclusions.

Many scientists confirm findings from high-throughput methods using other techniques, such as polymerase-chain-reaction or protein-based measurements. For conclusions based solely on expression results, the committee recommends a detailed analysis of the data supporting the conclusion. This should include thorough review of the effectiveness of the preprocessing and the statistical analysis and will require access to the raw data.

SYSTEMATIC REVIEW AND META-ANALYSIS

EPA’s draft plans indicate that the hazard-identification process will generate questions specific to the toxicologic review, and the questions will be addressed in systematic reviews. EPA expects questions to be generated about aspects related to adverse-outcome pathways and dose–response characterization. For example, EPA indicates that systematic reviews of dose–response analyses will be used to identify chemical and nonchemical stressors that might contribute to the health outcome under consideration and to identify underlying disease processes that might be added to or increased by exposure to inorganic arsenic. Those uses were recommended by the National Research Council committee that wrote Science and Decisions (NRC 2009) and were followed up on in Chapter 7 of Review of the Environmental Protection Agency’s Draft IRIS Assessment of Formaldehyde (NRC 2011).

Suggested Citation:"3 IRIS Assessment Development Plans: Evidence Evaluation, Systematic Review, and Meta-analysis." National Research Council. 2013. Critical Aspects of EPA's IRIS Assessment of Inorganic Arsenic: Interim Report. Washington, DC: The National Academies Press. doi: 10.17226/18594.
×

Efforts are under way to develop EPA guidance on performing systematic reviews. EPA held a workshop on August 26, 2013, that was devoted to that topic. Another National Research Council committee is evaluating such improvements of the IRIS process and is expected to issue a report in the first quarter of 2014. In light of those activities, the present committee focused its efforts on identifying issues specific to inorganic arsenic in the performance of systematic reviews rather than on specific procedural aspects of such reviews.

Systematic reviews and meta-analyses are widely used in medicine, epidemiology, and public health to evaluate and synthesize scientific evidence objectively (Egger et al. 2001). Systematic reviews comprehensively identify and evaluate the available evidence with a transparent process. They are used to reach evidence-based conclusions, to clarify the need for additional research, and potentially to summarize the evidence quantitatively with meta-analysis (Egger et al. 2001; Porta 2008; Rooney 2013). Meta-analysis is a statistical technique that can be used to pool evidence from systematic reviews if various conditions are met. To minimize uncertainty in original research, systematic reviews follow objective criteria for collecting and evaluating evidence in a comprehensive manner. A priori decisions and a predefined protocol are critical during the systematic review process (Berlin and Colditz 1999; Dickersin 2002); the protocol should describe the following steps: the research question, the search strategy and data sources, the study inclusion and exclusion criteria, the data to be abstracted and derived from the original studies (such as sample size, exposure and outcome assessment methods, and confounders evaluated), the criteria and methods for pooling effect estimates and measures of variability among studies. Systematic reviews and meta-analyses need to be replicable; other investigators following the same steps should be able to identify the same articles, abstract the same data, and reach similar conclusions. Originally developed in the field of clinical trials, systematic reviews and meta-analyses are commonly used to summarize epidemiologic research, including risk factors for a disease and dose–response relationships, but can also be used to summarize disease mechanisms (Dickersin 2002). The use of systematic reviews to evaluate experimental evidence is increasing, and some aspects are still in development. For instance, there is no clear best practice for evaluating the risk of bias in in vitro studies (Rooney 2013). Chapter 7 describes in more detail the role of meta-analyses in the evaluation of dose–response relationships and how they can be used for developing risk-based estimates. The role of systematic reviews and meta-analyses in the regulatory process for drugs, foods, and medical devices has long been recognized (Berlin and Colditz 1999), and their relevance for environmental policy, including both epidemiologic and experimental research, is increasing (NRC 2009, 2011). The Office of Health Assessment and Translation of the National Toxicology Program has developed an initiative to incorporate systematic reviews and evidence integration for literature-based environmental health assessment (Rooney 2013).

Given the large number of studies, evaluating the epidemiologic and experimental literature on arsenic health effects can be a daunting task. To guide the process and ensure high quality, it is critical to establish clearly the research questions that need to be answered. Examples of research questions about epidemiologic and experimental data, using cardiovascular disease as the health outcome, are shown in Box 4. EPA plans to include systematic review principles in performing its Integrated Risk Information System (IRIS) assessment of inorganic arsenic will be a welcome improvement.

EPA has some experience with performing systematic reviews and meta-analyses for IRIS assessments of other chemicals, such as trichloroethylene (EPA 2011). Lessons learned in performing those analyses might guide the systematic reviews and meta-analyses for inorganic arsenic. Also, a number of systematic reviews have summarized the association between arsenic and several health end points, including cancer (Celik et al. 2008), cardiovascular disease (Navas-Acien et al. 2005; C.H. Wang et al. 2007; Moon et al. 2012), diabetes (Navas-Acien et al. 2006; Maull et al. 2012), hypertension (Abhyankar et al. 2012), and neurodevelopment (Rodriguez-Barranco et al. 2013). Only a few of those reviews have attempted to generate pooled estimates of the association between arsenic and disease (Navas-Acien et al. 2006; Moon et al. 2012; Rodriguez-Barranco et al. 2013). Some of the systematic reviews of arsenic and health end points can be useful starting points, including their search strategies, table format, data collection, and quality evaluation forms. It is likely, however, that none of them will be sufficient for EPA’s purposes. First, the article

Suggested Citation:"3 IRIS Assessment Development Plans: Evidence Evaluation, Systematic Review, and Meta-analysis." National Research Council. 2013. Critical Aspects of EPA's IRIS Assessment of Inorganic Arsenic: Interim Report. Washington, DC: The National Academies Press. doi: 10.17226/18594.
×

BOX 4 Examples of Research Questions about Epidemiologic and Experimental Data

Examples of research questions that can be used to guide systematic reviews for the evaluation of arsenic-related health effects with overall clinical cardiovascular-disease mortality as the effect. They include questions related to hazard identification, dose–response analysis, and mode-of-action evaluation. Once the research questions have been established, a protocol should be developed to ensure that all relevant articles answering each of the questions are identified. Although the questions were developed by using cardiovascular disease as an example, other health end points and mechanisms could have been selected.

1. Is arsenic exposure, measured before the development of cardiovascular disease at the individual level, associated with overall clinical cardiovascular disease mortality in populations exposed to arsenic in drinking water at less than 100 μg/L?

2. What is the dose–response relationship between arsenic and cardiovascular-disease mortality throughout the range of arsenic exposure that is relevant for human populations (for example, arsenic in drinking water ranging from less than 10 μg/L to over 100 μg/L)?

3. What is the dose–response relationship between arsenic exposure and mechanisms of generation of reactive oxidant species or inflammation in relevant animal models and cellular systems that can be used to evaluate the mode of action of arsenic-related cardiovascular disease?

search will need to be updated. Second, the reviews categorized the studies into two groups (studies with arsenic concentrations less than or greater than 100 μg/L). However, a few studies conducted in populations exposed to high arsenic concentrations in drinking water had some categories at concentrations below 100 μg/L, and the results for those categories can be included in the analyses at low to moderate arsenic exposure.

General limitations of the conduct and interpretation of meta-analyses for arsenic health effects have been the substantial heterogeneity among studies, methodologic limitations in outcome and exposure assessment, the use of different biomarkers, the temporality of an association, adjustment for relevant confounders, and the potential for publication bias. Heterogeneity, however, is relatively common in observational studies; it should be evaluated but should not necessarily impede data-pooling. Sources of heterogeneity could be evaluated with meta-regression and analysis of influential studies. Publication bias is a concern in the conduct of systematic reviews, and different strategies are available to evaluate it, including the use of funnel plots.

The increasing number of published prospective studies that have adequate exposure and outcome assessment (for example, see section “Cardiovascular Disease” in Chapter 4) provides an opportunity for EPA to apply meta-analysis techniques to estimate the association between arsenic and relevant health end points. In selecting the studies to be included in the meta-analyses, EPA should systematically search for studies with individual measures of arsenic exposure (preferably with biomarker measurements), measurement of arsenic that precedes the outcome, and low to moderate exposure to inorganic arsenic (less than 100 μg/L in drinking water). Meta-analysis can then be performed if there are three or more peer reviewed studies available on the outcome of interest. For dose-response meta-analysis, studies will need to have at least three or more doses tested. As part of the systematic review process, study quality (risk of bias) needs to be evaluated using established guidelines for epidemiologic studies (Viswanathan et al. 2012; Rooney 2013). Elements in the evaluation of study quality include the assessment of study outcome based on standardized definitions, participation rate, adjustment of association for relevant confounders, and other considerations that will depend on study design.

Meta-analyses of published studies are sometimes limited by what is available in the aggregated data. Individual-level meta-analyses—meta-analyses that use raw data from the original studies (Riley et al. 2010)—provide an opportunity to answer specific questions and to use common definitions across studies. Major limitations of individual-level meta-analyses, however, are the requirement of active participation by the original investigators, of standardization across studies, and of additional resources and fund-

Suggested Citation:"3 IRIS Assessment Development Plans: Evidence Evaluation, Systematic Review, and Meta-analysis." National Research Council. 2013. Critical Aspects of EPA's IRIS Assessment of Inorganic Arsenic: Interim Report. Washington, DC: The National Academies Press. doi: 10.17226/18594.
×

ing to conduct the studies. For the purpose of the IRIS assessment, given limited resources and the number and quality of published studies of some health end points associated with low to moderate arsenic exposure, conducting meta-analyses of aggregated data from published studies is an appropriate alternative to collecting and analyzing raw data for individual-level meta-analysis.

Suggested Citation:"3 IRIS Assessment Development Plans: Evidence Evaluation, Systematic Review, and Meta-analysis." National Research Council. 2013. Critical Aspects of EPA's IRIS Assessment of Inorganic Arsenic: Interim Report. Washington, DC: The National Academies Press. doi: 10.17226/18594.
×
Page 18
Suggested Citation:"3 IRIS Assessment Development Plans: Evidence Evaluation, Systematic Review, and Meta-analysis." National Research Council. 2013. Critical Aspects of EPA's IRIS Assessment of Inorganic Arsenic: Interim Report. Washington, DC: The National Academies Press. doi: 10.17226/18594.
×
Page 19
Suggested Citation:"3 IRIS Assessment Development Plans: Evidence Evaluation, Systematic Review, and Meta-analysis." National Research Council. 2013. Critical Aspects of EPA's IRIS Assessment of Inorganic Arsenic: Interim Report. Washington, DC: The National Academies Press. doi: 10.17226/18594.
×
Page 20
Suggested Citation:"3 IRIS Assessment Development Plans: Evidence Evaluation, Systematic Review, and Meta-analysis." National Research Council. 2013. Critical Aspects of EPA's IRIS Assessment of Inorganic Arsenic: Interim Report. Washington, DC: The National Academies Press. doi: 10.17226/18594.
×
Page 21
Suggested Citation:"3 IRIS Assessment Development Plans: Evidence Evaluation, Systematic Review, and Meta-analysis." National Research Council. 2013. Critical Aspects of EPA's IRIS Assessment of Inorganic Arsenic: Interim Report. Washington, DC: The National Academies Press. doi: 10.17226/18594.
×
Page 22
Next: 4 Hazard Identification »
Critical Aspects of EPA's IRIS Assessment of Inorganic Arsenic: Interim Report Get This Book
×
Buy Paperback | $39.00 Buy Ebook | $31.99
MyNAP members save 10% online.
Login or Register to save!
Download Free PDF

The US Environmental Protection Agency (EPA) Integrated Risk Information System (IRIS) program develops toxicologic assessments of environmental contaminants. IRIS assessments provide hazard identification and dose-response assessment information. The information is then used in conjunction with exposure information to characterize risks to public health and may be used in risk-based decisionmaking, in regulatory actions, and for other risk-management purposes. Since the middle 1990s, EPA has been in the process of updating the IRIS assessment of inorganic arsenic. In response to a congressional mandate for an independent review of the IRIS assessment of inorganic arsenic, EPA requested that the National Research Council convene a committee to conduct a two-phase study. Critical Aspects of EPA's IRIS Assessment of Inorganic Arsenic is the report of the first phase of that study. This report evaluates critical scientific issues in assessing cancer and noncancer effects of oral exposure to inorganic arsenic and offers recommendations on how the issues could be addressed in EPA's IRIS assessment.

  1. ×

    Welcome to OpenBook!

    You're looking at OpenBook, NAP.edu's online reading room since 1999. Based on feedback from you, our users, we've made some improvements that make it easier than ever to read thousands of publications on our website.

    Do you want to take a quick tour of the OpenBook's features?

    No Thanks Take a Tour »
  2. ×

    Show this book's table of contents, where you can jump to any chapter by name.

    « Back Next »
  3. ×

    ...or use these buttons to go back to the previous chapter or skip to the next one.

    « Back Next »
  4. ×

    Jump up to the previous page or down to the next one. Also, you can type in a page number and press Enter to go directly to that page in the book.

    « Back Next »
  5. ×

    Switch between the Original Pages, where you can read the report as it appeared in print, and Text Pages for the web version, where you can highlight and search the text.

    « Back Next »
  6. ×

    To search the entire text of this book, type in your search term here and press Enter.

    « Back Next »
  7. ×

    Share a link to this book page on your preferred social network or via email.

    « Back Next »
  8. ×

    View our suggested citation for this chapter.

    « Back Next »
  9. ×

    Ready to take your reading offline? Click here to buy this book in print or download it as a free PDF, if available.

    « Back Next »
Stay Connected!