National Academies Press: OpenBook

Guiding Principles for Developing Dietary Reference Intakes Based on Chronic Disease (2017)

Chapter: 6 Evidence Review: Judging the Evidence for Causal Relationships

« Previous: 5 Measuring Chronic Disease Outcomes
Suggested Citation:"6 Evidence Review: Judging the Evidence for Causal Relationships." National Academies of Sciences, Engineering, and Medicine. 2017. Guiding Principles for Developing Dietary Reference Intakes Based on Chronic Disease. Washington, DC: The National Academies Press. doi: 10.17226/24828.
×

6

Evidence Review: Judging the Evidence for Causal Relationships

Previous chapters illustrate the current process and framework (see Chapter 1, Figure 1-2) as well as the relevant activities to date (see Chapter 2) for developing Dietary Reference Intakes (DRIs) based on indicators of adequacy, toxicity, or chronic disease. Although the current DRI process considers all three indicators, this committee has been asked to provide recommendations and guiding principles for developing chronic disease DRIs, specifically. As mentioned in Chapter 1, a key activity relevant to chronic disease endpoints within the purview of DRI committees is evaluating the certainty of the evidence with regard to two main questions: (1) What are acceptable levels of confidence that the relationship between an NOFS and a chronic disease is causal? and (2) If a causal relationship exists, what are acceptable levels of confidence in the intake-response relationship data, what are approaches for identifying and characterizing such relationship, and if appropriate, to recommend chronic disease DRIs? This chapter discusses the methodological tools that can be used to answer the first question (does a causal relationship exist?). The chapter starts with an overview of the conduct of a systematic review, as a modern tool to answer scientific questions and to provide context for the reminder of the chapter. It also provides an overview of the systems and tools that are often used to evaluate scientific evidence. Finally, it discusses in more depth the strengths and cautions in applying one of those systems, Grading of Recommendations, Assessment, Development and Evaluation (GRADE), in determining causal associations between nutrients or other food substances (NOFSs) and chronic diseases for DRI development. Resources for more detailed application of the procedures are provided. At the end of the chapter, the

Suggested Citation:"6 Evidence Review: Judging the Evidence for Causal Relationships." National Academies of Sciences, Engineering, and Medicine. 2017. Guiding Principles for Developing Dietary Reference Intakes Based on Chronic Disease. Washington, DC: The National Academies Press. doi: 10.17226/24828.
×

guiding principles as foundations for a scientifically credible chronic disease DRI process are listed. A list of key terms and their definitions is in Box 6-1.

Throughout the report and particularly within this chapter, the committee makes a clear distinction between the task of the systematic review team, which conducts the systematic review, and the task of the DRI committee, which reviews the totality of evidence and recommends chronic

Suggested Citation:"6 Evidence Review: Judging the Evidence for Causal Relationships." National Academies of Sciences, Engineering, and Medicine. 2017. Guiding Principles for Developing Dietary Reference Intakes Based on Chronic Disease. Washington, DC: The National Academies Press. doi: 10.17226/24828.
×

disease DRIs, if appropriate. A systematic review is a scientific investigation that focuses on a specific question and that uses explicit, planned scientific methods to identify, select, assess, and summarize the findings of similar but separate studies. It may or may not include a quantitative synthesis (meta-analysis) of the results from separate studies (IOM, 2011b). A systematic review is typically conducted by a group of experts in the process itself

Suggested Citation:"6 Evidence Review: Judging the Evidence for Causal Relationships." National Academies of Sciences, Engineering, and Medicine. 2017. Guiding Principles for Developing Dietary Reference Intakes Based on Chronic Disease. Washington, DC: The National Academies Press. doi: 10.17226/24828.
×
Suggested Citation:"6 Evidence Review: Judging the Evidence for Causal Relationships." National Academies of Sciences, Engineering, and Medicine. 2017. Guiding Principles for Developing Dietary Reference Intakes Based on Chronic Disease. Washington, DC: The National Academies Press. doi: 10.17226/24828.
×
Suggested Citation:"6 Evidence Review: Judging the Evidence for Causal Relationships." National Academies of Sciences, Engineering, and Medicine. 2017. Guiding Principles for Developing Dietary Reference Intakes Based on Chronic Disease. Washington, DC: The National Academies Press. doi: 10.17226/24828.
×

that includes subject matter experts (e.g., a systematic review team) and in consultation with external subject matter experts (e.g., a technical expert panel). In contrast, the DRI committee’s review consists of drawing conclusions about the totality of the findings. The guidelines and recommendations in this chapter are envisioned to be followed in the context of the current process, where DRI committees do not conduct systematic reviews themselves but are the recipients of such reviews. Based on the systematic reviews, the DRI committee is tasked with the review of the totality of the evidence, including evidence about quantitative intake-response relationships described in Chapter 7, and, if appropriate, with recommending a DRI. It is further assumed that the initial and any subsequent systematic reviews and protocols are conducted independently of the DRI committee. The first use of this type of systematic review and evidence review process for DRI purposes was reflected in the development of the 2011 update of the DRIs for calcium and vitamin D, which included consideration of chronic disease endpoints (Brannon et al., 2014; IOM, 2011a) (see the vitamin D case example in Chapter 3).

This chapter particularly considers the nutrition-specific issues outlined in Chapter 3. It draws on other sections of the report that discuss methods to measure dietary intake (see Chapter 4), and disease outcomes (see Chapter 5) and leads up to the discussion, in Chapter 7, of approaches to specifying intake-response relationships when causation has been established with sufficient certainty (i.e., the second main question). In addition, although it is not the task of this report to provide direction in moving to guidelines and policies, in various places the committee offers comments on how the conduct of systematic reviews can and should anticipate the guideline decisions that follow them as best as possible, in order to facilitate that process.

OVERVIEW OF THE SYSTEMATIC REVIEW PROCESS

In the context of setting DRIs, the systematic review process is an essential activity before the totality of the evidence is reviewed by the DRI committee. Figure 6-1 is an overview of an ideal systematic review as conceptualized in the IOM report Finding What Works in Health Care: Standards for Systematic Reviews (IOM, 2011b). Depending on the context for a systematic review and the group that conducts it, the specific nature of the activities may vary. This section includes a description of some of the activities highlighted as especially important in the context of developing DRIs. These include formulating the questions, registering the systematic reviews and adopting an appropriate format for reporting methods and results, and anticipating additional information needs beyond the systematic review results.

Suggested Citation:"6 Evidence Review: Judging the Evidence for Causal Relationships." National Academies of Sciences, Engineering, and Medicine. 2017. Guiding Principles for Developing Dietary Reference Intakes Based on Chronic Disease. Washington, DC: The National Academies Press. doi: 10.17226/24828.
×

Problem Formulation, Scoping and Developing a Systematic Review Protocol

The initial selection of particular NOFSs and health outcomes is within the purview of the federal agencies (see Figure 1-2), which prioritize NOFSs and health outcomes for study based on public health priorities and other factors. Whatever the choices, each one implies some tentative perspective that a scientific basis and a literature exist to proceed. Therefore, once the general scientific questions that form the basis for the systematic evidence review have been formulated, an important next preliminary step is to sample the available literature, to determine that it is sufficiently robust to make the review useful for each of the central scientific questions identified. Often, a preliminary literature review may be available, including previously conducted systematic reviews and meta-analyses. If not, a preliminary “scoping” review should be conducted (see, e.g., Brannon et al., 2016). If the studies are clearly insufficient, then setting a DRI is not scientifically justifiable. In addition to assessing the literature’s breadth, depth, and relevance, the scoping exercise could have other potential dividends, such as determining the need for translation services and identifying additional keywords to aid in searching. This step is different from conducting the comprehensive literature search. Scoping may also identify systematic reviews, including meta-analyses already published on nutrition-disease topics. These previously published systematic reviews can help target relevant literature and also can be incorporated as part of the evidence review. Instruments are available to assess systematic review quality (e.g., AMSTAR [Pieper et al., 2014; Shea et al., 2007, 2009] and risk of bias [ROBIS] [Whiting et al., 2013, 2016]). It may be possible to add recent research reports to existing systematic reviews for reasons of efficiency if the search is comprehensive up to a particular date and all other quality criteria are met (Chung et al., 2012; Garner et al., 2016; Shekelle et al., 2009).

Any comprehensive systematic review can have multiple scientific questions (e.g., multiple dimensions of efficacy and safety of an NOFS-focused intervention, including nutrient-nutrient interactions, varied effects in subpopulations defined by age or other characteristics, and effects on multiple disease outcomes or surrogate markers). Each question should be specific and clearly defined, and narrow enough to be aligned with the evidence base under review. Framing questions as specific scientific hypotheses is essential.

The structure for questions to guide systematic and evidence reviews involves detailed and explicit characterization of the study level and design details, including target population, candidate interventions, and outcomes important to individuals. The PICO (population, intervention, comparator,

Suggested Citation:"6 Evidence Review: Judging the Evidence for Causal Relationships." National Academies of Sciences, Engineering, and Medicine. 2017. Guiding Principles for Developing Dietary Reference Intakes Based on Chronic Disease. Washington, DC: The National Academies Press. doi: 10.17226/24828.
×
Image
FIGURE 6-1 Standards for systematic reviews.
SOURCE: IOM, 2011b.
Suggested Citation:"6 Evidence Review: Judging the Evidence for Causal Relationships." National Academies of Sciences, Engineering, and Medicine. 2017. Guiding Principles for Developing Dietary Reference Intakes Based on Chronic Disease. Washington, DC: The National Academies Press. doi: 10.17226/24828.
×
Image
Suggested Citation:"6 Evidence Review: Judging the Evidence for Causal Relationships." National Academies of Sciences, Engineering, and Medicine. 2017. Guiding Principles for Developing Dietary Reference Intakes Based on Chronic Disease. Washington, DC: The National Academies Press. doi: 10.17226/24828.
×

and outcome) structure1 is considered helpful in bringing focus and order to the questions at hand (Guyatt et al., 2011b). In the context of chronic disease DRIs, key questions would be formulated, and the parameters of interest to the sponsoring agency or agencies in consultation with a technical expert panel would be specified using the PICO framework. The vitamin D case example in Chapter 3 demonstrates how a map of the various biological pathways of interest can be created as an analytic framework to guide the process of framing the key questions (see Figure 3-1). The intervention and comparator in these questions are implicit, i.e., effects of interest are not specified with respect to a given level of vitamin D intake. Data abstraction methods that identify the P, I, C, and O elements for each study are included in the systematic review protocol. As noted in Chapter 3, because everyone in the population consumes NOFSs at some level, for the purpose of establishing a causal relationship, some questions may be framed as an increase or decrease relative to existing consumption patterns rather than above or below a certain threshold.

As another example, formulated questions for systematic reviews to support the World Health Organization (WHO) guideline on free sugars2 were as follows: “What is the effect of a decrease or increase in free sugars in adults and children?” and “What is the effect of restricting intake of free sugars to below 10 percent of total energy intake?”—accompanied by a definition of the term “free sugars” (WHO, 2015). Annex 6-1 shows the full detail of the free sugar questions in a PICO table format that specifies population considerations, exposure definitions, comparisons, confounders, effect modifiers and intermediates, outcomes, and time frames (WHO, 2015). Evidence on these questions about sugar was gathered through several systematic reviews (peer reviewed and published) in order to inform the eventual guideline. The questions in Box 6-2, also drawn from systematic reviews to support WHO guideline development, illustrate more complex NOFS-chronic disease questions using the general PICO approach (Brouwer, 2016; Mensink, 2016). As with the free sugar questions, these questions frame “interventions” about intake level in terms of increases or decreases, sometimes related to specific thresholds in association with particular outcomes—in this case blood lipids as surrogates for noncommunicable disease risk. Other questions address effects of different types

___________________

1 Some groups use PICOTS: population, intervention, comparators, outcomes, timing, and setting.

2 Free sugars is defined by the United Nations World Health Organization and the Food and Agriculture Organization as “all monosaccharides and disaccharides added to foods by the manufacturer, cook, or consumer, plus sugars naturally present in honey, syrups, and fruit juices” (WHO, 2015, p. 1). In the United States, the term “free sugars” is generally equivalent to “added sugars,” the term used by the 2015-2020 Dietary Guidelines for Americans.

Suggested Citation:"6 Evidence Review: Judging the Evidence for Causal Relationships." National Academies of Sciences, Engineering, and Medicine. 2017. Guiding Principles for Developing Dietary Reference Intakes Based on Chronic Disease. Washington, DC: The National Academies Press. doi: 10.17226/24828.
×
Suggested Citation:"6 Evidence Review: Judging the Evidence for Causal Relationships." National Academies of Sciences, Engineering, and Medicine. 2017. Guiding Principles for Developing Dietary Reference Intakes Based on Chronic Disease. Washington, DC: The National Academies Press. doi: 10.17226/24828.
×

of saturated or trans fat and substitution or replacement effects that are relevant to food manufacturing.

Systematic Review Registration and Report Format

Once the systematic review team is assembled and as the systematic review process begins, the committee strongly suggests registering the review in advance with the international prospective register of systematic reviews (PROSPERO).3 This registration applies to the initial systematic review and any subsequent reviews needed. Registering the review in advance with the PROSPERO system is mandatory for the Agency for Healthcare Research and Quality (AHRQ) Evidence-based Practice Center (EPC) program, which currently leads the systematic reviews for setting DRIs and has value in several ways, such as avoiding duplication of effort and promoting transparency. In planning the report, the team should follow the Preferred Reporting Items for Systematic Reviews and Meta-Analyses for Protocols (PRISMA-P) approach,4 which specifies structure and completeness for reporting systematic reviews and meta-analyses.

Anticipating Additional Literature Needs When Planning Systematic Reviews

When planning a systematic review, anticipating downstream needs for literature searches and guideline creation can be important, particularly if the systematic review cannot be repeated or re-opened after completion (e.g., due to financial or other constraints). Examples of such activities

___________________

3 See https://www.crd.york.ac.uk/PROSPERO.

4 See http://www.prisma-statement.org.

Suggested Citation:"6 Evidence Review: Judging the Evidence for Causal Relationships." National Academies of Sciences, Engineering, and Medicine. 2017. Guiding Principles for Developing Dietary Reference Intakes Based on Chronic Disease. Washington, DC: The National Academies Press. doi: 10.17226/24828.
×

include collecting literature on validation of potential surrogate markers or biomarkers of intake, and the general health characteristics or diets of the target populations of interest, identifying disease risks in those groups of special interest, searching for published computational or other public health decision models that could aid in recommending the DRI, identifying studies of adverse effects of selected NOFS interventions, and cataloging relevant literature on population interventions. These tasks will be accomplished by the systematic review team as it develops its search strategy.

The committee also notes that mechanistic studies, whether in humans, animals, or various in vitro systems, may be important to generating hypotheses regarding the ability of an NOFS to affect the risk of developing a chronic disease. The World Cancer Research Fund Expert Panel on Diet, Nutrition, and Cancer Prevention, for example, used a hierarchy of robustness based on categorization of human and animal experimental findings into three classes of which only one class was thought to be useful for making judgments about diet and cancer risk (WCRF/AICR, 2007). Although it is not generally practical to include mechanistic studies in systematic reviews of the effect of NOFSs on health, mechanistic studies can contribute to addressing the study question by providing additional information. Thus, it would be important to anticipate mechanistic research questions at the start of the systematic review process. Box 6-3 lists some of the important information that can be gathered from mechanistic studies.

Forming the Systematic Review Team, Technical Expert Panel, and DRI Committee

Developing chronic disease DRIs depends on the contributions of many stakeholders, including the federal agencies, groups of experts, and the public. Within the current process, three groups of experts—the systematic review team, the technical expert panel, and the DRI committee—are involved in different capacities and time frames. Briefly, the systematic review team conducts the systematic review in consultation with the technical expert panel, and the DRI committee reviews the totality of the evidence and recommends chronic disease DRIs, if appropriate. Formation of these three groups necessitates a formal process. Although no dictates on group structure and the disciplines represented in any of the three groups have been established, experience suggests that several types of expertise are of great value. While the systematic review team includes mainly methodologists, the technical expert panel would include relevant content experts in nutrition science, toxicology, scientific study design and analysis, public health, biostatistics, nutrition epidemiology and chronic disease epidemiology, and disease pathogenesis. In terms of the DRI committee, disciplines include content experts for the main themes of the review, methodologists

Suggested Citation:"6 Evidence Review: Judging the Evidence for Causal Relationships." National Academies of Sciences, Engineering, and Medicine. 2017. Guiding Principles for Developing Dietary Reference Intakes Based on Chronic Disease. Washington, DC: The National Academies Press. doi: 10.17226/24828.
×

experienced with the evidence review approach, biostatisticians, and persons with experience in guideline construction and policy applications. The size of the group should be consonant with the breadth of review topics, and available resources.

To maintain objectivity, scientific rigor, and acceptability of a systematic review, it is important to recognize and manage potential bias and conflict of interests of the individuals conducting or providing input into the systematic review (IOM, 2011b). AHRQ, which is the agency that in the current DRI process has been responsible for administering the systematic review for DRI development, has policies to reduce the risk of bias from financial and non-financial interests. This requirement was also highlighted in a recent report from the National Academies of Sciences, Engineering, and Medicine (the National Academies) in the context of advisory panels (NASEM, 2017). Specifically, the report made recommendations on the selection process, including disclosure of how conflict of interest and biases are identified and managed, of members of the Dietary Guidelines Advisory

Suggested Citation:"6 Evidence Review: Judging the Evidence for Causal Relationships." National Academies of Sciences, Engineering, and Medicine. 2017. Guiding Principles for Developing Dietary Reference Intakes Based on Chronic Disease. Washington, DC: The National Academies Press. doi: 10.17226/24828.
×

Committee.5 The National Academies, which has convened all DRI committees in the past, has policies for assessing potential conflicts of interest among committee members.6 To promote transparency and enhance public and stakeholder assurances of committees, the National Academies publishes the biographies of individual members for public review and comment.

Regardless of the subject matter and the disciplines required, it is important to establish a mechanism for communication among the three groups that will ensure a moderate level of communication while also protecting against inappropriate influence on the systematic review methods.

SYSTEMS OF EVIDENCE SYNTHESIS AND THEIR APPLICATIONS

The committee concluded that addressing diet-chronic disease relationships requires a credible, established, structured system for systematic review and review of the totality of the evidence, for all important topics that may be translated into population policy, and that this applies where “diet” refers to specific NOFSs, as is the focus of this DRI-oriented report, as well as when applied to foods or entire dietary patterns. The guiding principles discussed throughout the chapter reflect the fact that systematic reviews and evidence reviews in general are well-developed scientific activities for informing policy and practice. The several different systematic review and evidence review systems that have been applied to nutrition have many features in common. Requirements for such systems include:

  • A record of robust experience and precedent to approach important scientific issues, including those involving questions about health effects of NOFS consumption,
  • Well-structured protocols that can accommodate multiple major and ancillary scientific issues,
  • Sufficient clarity and transparency to be useful for evidence review and policymaking groups,
  • Sufficient detail to be replicable and to withstand expert peer review,
  • The ability to cogently address causal pathways, which can potentially be used to develop policy recommendations, and
  • A core set of methods stable enough to be applied over time but sufficiently flexible to be sensitive to the important details for different content areas. The methods should improve and evolve with thoughtful input from content experts and the user communities

___________________

5 The Dietary Guidelines Advisory Committee independently evaluates the scientific evidence and makes recommendations about how the Dietary Guidelines for Americans could be revised.

6 See www.nationalacademies.org/coi.

Suggested Citation:"6 Evidence Review: Judging the Evidence for Causal Relationships." National Academies of Sciences, Engineering, and Medicine. 2017. Guiding Principles for Developing Dietary Reference Intakes Based on Chronic Disease. Washington, DC: The National Academies Press. doi: 10.17226/24828.
×

and with sufficient documentation so that changes can be understood and carried forth.

The committee acknowledges that no system will meet all needs and contingencies equally, and that all systems have strengths and weaknesses. The committee identified and reviewed key features of a variety of available evidence review and systematic review systems and instruments, contained in Annex 6-2, in sufficient detail to confirm that they have much in common conceptually although they vary in intent and comprehensiveness. The Annex includes systems or instruments intended only for assessing specific elements of primary empirical research reports, such as “risk-of-bias” tools, or instruments to evaluate the quality of reporting and completed systematic reviews. As others have concluded (Dwyer et al., 2016), all of these systems and instruments were deemed to have important value and apply similar principles to the review of the evidence. Some instruments for reviewing evidence are intended only for RCTs while others apply to both trials and observational studies. Some systems are used for a variety of issues related to primary and secondary prevention issues, including nutrition-related issues (AHRQ, 2014; USPSTF, 2015). Another system is used for assessing toxic environmental exposures (NTP, 2015) and others have been adapted for specific nutrition-related uses (NEL, 2015; WCRF/AICR, 2007). Still other systems are designed to evaluate the quality of reporting of completed systematic reviews and meta-analyses (e.g., AMSTAR [Pieper et al., 2014; Shea et al., 2007, 2009]).

THE “GRADE” (GRADING OF RECOMMENDATIONS, ASSESSMENT, DEVELOPMENT AND EVALUATION) SYSTEM: DEFINITION AND JUSTIFICATION

After reviewing the systems in Annex 6-2 and based on the knowledge and experience of committee members about what would meet the needs of DRI committees and users of DRI reports, the committee adopted the GRADE system as the reference point for the evidence reviews relating to NOFS-chronic disease considerations.

The GRADE system is an approach for rating certainty in the evidence in systematic reviews and guidelines. The system is designed for examining alternative management strategies or interventions, which may include no intervention or current best management. Although originally developed in the context of health care (diagnosis, screening, prevention, and therapy), it can be (and has been) applied to public health questions. In the particular case of evaluating the associations of NOFSs with chronic disease outcomes, GRADE will be used to assess the certainty in the association between an NOFS and benefit and harm, which implies causality. In addi-

Suggested Citation:"6 Evidence Review: Judging the Evidence for Causal Relationships." National Academies of Sciences, Engineering, and Medicine. 2017. Guiding Principles for Developing Dietary Reference Intakes Based on Chronic Disease. Washington, DC: The National Academies Press. doi: 10.17226/24828.
×

tion to rating the certainty of a body of evidence, the GRADE approach includes tools for grading the strength of recommendations (see also Chapter 7). The rationale for the selection of GRADE is presented below.

Meets Criteria for an Appropriate Evidence Review Tool

The GRADE system meets the requirements for an established system of guidance for evidence synthesis. It has been used extensively in many scientific domains, and comprehensive documentation and precedent exist to address many of the issues relevant to NOFS-chronic disease questions. A particular advantage of the GRADE system is that it provides a comprehensive and clearly structured taxonomy for rating the certainty of evidence and extensive, specific guidance for making the ratings. In addition, GRADE working subgroups continue to address specific methodological or subject matter issues as needed. Publications for using GRADE include a six-part series of papers in the British Medical Journal (Guyatt et al., 2008a,b,c,d; Jaeschke et al., 2008; Schunemann et al., 2008), a large series of papers in the Journal of Clinical Epidemiology (Andrews et al., 2013a,b; Balshem et al., 2011; Guyatt et al., 2011a,b,c,d,e,f,g,h), and two additional papers focusing on the framework for translating evidence to guideline decisions (Alsonso-Coello et al., 2016a,b). The online GRADE Handbook (Schunemann et al., 2013) summarizes the guidance in these articles.

As long as the fundamental tenets are followed and a clear rationale for specific decisions stated, GRADE can accommodate particular perspectives. Indeed, the merit of this system is not that all members of a group using it will ultimately agree, but that it provides a structured, explicit, and transparent system for making decisions. Therefore, the GRADE working group strongly discourages modifications to the approach itself and recommends adherence to the currently available methods and applications. However, the GRADE system is neither rigid nor static. GRADE continues to evolve, and the GRADE working group is open to participation and collaboration from those facing challenges in applying the system, such as for informing nutritional guidelines in the context of the many unique conceptual and methodological issues outlined in Chapter 3.

Is in Wide Use and Applicable to Nutrition Policy Questions

The GRADE system provides an approach to rating the certainty of evidence for the outcomes relevant to optimizing health, either in terms of personal habits and lifestyle, or in decisions about health care interventions. It has been adopted by more than 100 organizations including WHO (2014), the Cochrane Collaboration, the American College of Physicians, the National Institute for Health and Care Excellence (NICE), and

Suggested Citation:"6 Evidence Review: Judging the Evidence for Causal Relationships." National Academies of Sciences, Engineering, and Medicine. 2017. Guiding Principles for Developing Dietary Reference Intakes Based on Chronic Disease. Washington, DC: The National Academies Press. doi: 10.17226/24828.
×

UpToDate®), increasing its use and familiarity. GRADE has been applied in more than 100 countries. This facilitates the ability to compare reviews developed by diverse users. GRADE has been applied to nutrition-related systematic reviews (Garcia-Larsen et al., 2016; Mayhew et al., 2016; Naude et al., 2014; Pennant et al., 2015; Santesso et al., 2012) and is the method generally used by WHO to support guideline development, including for nutrition topics.

Has Criteria for Assessing Strength of Recommendations

The GRADE working group has seriously considered the issues of policy guidelines creation after evidence review, though the latter is not the task of this report. GRADE separates questions about confidence in estimates of effect in studies from questions about the strength of recommendations. This is critical for preserving the integrity of the judgments made about causal associations. Strong recommendations are far more likely when evidence is of high or moderate certainty and far less likely when evidence is of low or very low certainty. Nevertheless, when desirable and undesirable consequences and outcomes are closely balanced, it is possible to have a weak recommendation in the context of high certainty of the evidence. Similarly, GRADE has identified five paradigmatic situations in which strong recommendations are warranted in the face of low certainty evidence (Andrews et al., 2013b).

APPLYING THE GRADE APPROACH FOR SYNTHESIZING EVIDENCE ABOUT THE RELATIONSHIP BETWEEN AN NOFS AND CHRONIC DISEASE

This section is not intended to provide specific methods or protocols, which are published in peer-reviewed journals and the online handbook (Schunemann et al., 2013). The section describes how the GRADE system (see Figure 6-2 for an overview) is applied to evidence review in general, with particular suggestions and examples for application to the development of NOFS-chronic disease DRIs. In particular, this section addresses four critical and challenging issues in conducting an evidence review and determining the certainty of the evidence using the GRADE system: (1) selecting and ranking the outcomes, (2) addressing multiple comparators, (3) developing evidence summaries, and (4) determining the certainty of the evidence for each outcome. Published information that provides additional detail is highlighted.

It is envisioned that future DRI committees will review the evidence synthesis provided to them by applying the GRADE approach. Through that process, the committees will make judgments about causality and,

Suggested Citation:"6 Evidence Review: Judging the Evidence for Causal Relationships." National Academies of Sciences, Engineering, and Medicine. 2017. Guiding Principles for Developing Dietary Reference Intakes Based on Chronic Disease. Washington, DC: The National Academies Press. doi: 10.17226/24828.
×

where indicated, intake-response. Thus, although some of the activities shown in Figure 6-2 and described above under the “overview of systematic review” are not conducted by DRI committees, they are essential to ensuring that DRIs are established with the best possible evidence. For example, as Figure 6-2 illustrates and as described earlier in this chapter, in the GRADE approach, a systematic review of the relevant scientific literature by the systematic review team precedes the work of the DRI committees to evaluate the totality of the evidence. The following discussion assumes that the systematic review and subsequent evidence reviews by the DRI committee are largely independent and that these processes precede and are separate from the development of guidelines, as described in Chapter 1 and above.7

Selecting and Ranking Health and Disease Outcomes

GRADE emphasizes the importance of identifying all outcomes that are meaningful to the population to whom a recommendation or guideline will be applied. For example, a particular NOFS may have causal relationships with several disease outcomes, beneficial or harmful. For risk assessment purposes, effects on physical and mental health are of primary importance. Effects on individual well-being (quality of life) may be considered during the risk management process, but these general effects are more relevant to consuming foods or following dietary patterns than to the intake of specific NOFSs.

Selecting the chronic disease outcomes to be considered as the basis for a DRI is a two-step process. First, all important chronic disease outcomes are identified during the formulation of the PICO questions. For the WHO free sugars guideline (WHO, 2015), several indicators were identified for body weight or body adiposity outcomes, and the outcome of dental caries was specifically defined as not including dental erosion (see Annex 6-1 for free sugar guidelines PICO specifications). In this step, designating and recording other secondary outcomes from relevant scientific reports is likely to be valuable, even if not originally part of the review’s goal. Defining longer-term health outcomes from a nutritional intervention will be central for developing chronic disease DRIs. If available, general health outcomes, such as all-cause mortality or disability rates, may inform summative population effects of the NOFS under consideration, both positive and adverse. The inclusion of such broad outcomes is of particular importance for NOFS

___________________

7 In other contexts, these activities may be performed as a continuum in which the same committee acts as or liaises to the technical advisory group before the conduct of systematic reviews, receives the systematic review, proceeds with evidence review, and then develops policy recommendations.

Suggested Citation:"6 Evidence Review: Judging the Evidence for Causal Relationships." National Academies of Sciences, Engineering, and Medicine. 2017. Guiding Principles for Developing Dietary Reference Intakes Based on Chronic Disease. Washington, DC: The National Academies Press. doi: 10.17226/24828.
×
Image
FIGURE 6-2 Schematic view of GRADE process for developing recommendations. In contrast to the GRADE approach, DRI committees will make DRI recommendations based on the balance of risk and benefits of health outcomes alone and will not consider values and preferences or resource use, which are within the purview of policy considerations.
NOTE: OC = outcomes; PICO = population, intervention, comparator, and outcome; RCT = randomized controlled trial; S = studies.
* Also labeled “conditional” or “discretionary.”
SOURCE: Reprinted from Guyatt, G., A. D. Oxman, E. A. Akl, R. Kunz, G. Vist, J. Brozek, S. Norris, Y. Falck-Ytter, P. Glasziou, H. DeBeer, R. Jaeschke, D. Rind, J. Meerpohl, P. Dahm, and H. J. Schunemann. 2011. GRADE guidelines: 1. Introduction—GRADE evidence profiles and summary of findings tables. J Clin Epidemiol 64(4):383-394, with permission from Elsevier.
Suggested Citation:"6 Evidence Review: Judging the Evidence for Causal Relationships." National Academies of Sciences, Engineering, and Medicine. 2017. Guiding Principles for Developing Dietary Reference Intakes Based on Chronic Disease. Washington, DC: The National Academies Press. doi: 10.17226/24828.
×

questions because so many different outcomes may be affected, directly or indirectly. However, because of the many other NOFSs and other factors (e.g., other behaviors, medication use) that may be in the same pathway, attributing total mortality to a specific NOFS is challenging. The systematic review protocols also may include relevant surrogate markers (see Chapter 5 and the discussion of indirectness in this chapter). As mentioned in Chapter 5, conceptually, surrogate markers have no importance in themselves unless they are evaluated as “qualified” to serve as surrogates for the important policy relevant outcome. The protocol also should specify the time frame to which the outcomes apply (e.g., the interval between intervention and expected effects, such as 1 year, 5 years, or a lifetime).

Dealing with Multiple Comparators

The structure of the PICO questions, i.e., specifying one comparator for a given intervention, requires modification when more than two alternative courses of action exist. Under these circumstances, all potential courses of action can be characterized as alternative interventions in a series of questions in which the intervention is compared with each relevant alternative, or as a single intervention with multiple comparators by group (possible alternatives within a single question). For example, one could ask in one question “what is the effect of replacing saturated fat with monounsaturated fatty acids or various types of polyunsaturated fatty acids on cardiovascular disease outcomes” but extract data to support examination of each type of replacement separately. However, when multiple alternatives exist, it is unlikely that the primary literature will have compared each alternative to every other alternative. In this case, inferences regarding the relative merits of the alternatives require indirect comparisons. A novel statistical methodology for simultaneously considering all direct and indirect comparisons, network meta-analysis, is now available for dealing with such situations. GRADE has developed guidance for rating certainty of evidence in the context of network meta-analysis (Puhan et al., 2014).

Developing Evidence Summaries

Identifying, collecting, and evaluating published scientific studies (i.e., the evidence) is described in detail in the GRADE handbook (Schunemann et al., 2013). Once the studies have been identified, and the evidence evaluated, GRADE suggests aggregating the findings into two types of evidence summary tables: evidence profiles and summary-of-findings tables (Guyatt et al., 2011a). Although this task would be conducted by the systematic review team, not the DRI committee, it is an essential task that allows the committee to assess the totality of the evidence. The DRI committee would

Suggested Citation:"6 Evidence Review: Judging the Evidence for Causal Relationships." National Academies of Sciences, Engineering, and Medicine. 2017. Guiding Principles for Developing Dietary Reference Intakes Based on Chronic Disease. Washington, DC: The National Academies Press. doi: 10.17226/24828.
×

review the evidence tables in order to make their own decisions regarding the certainty of the evidence for each outcome. Any discrepancies should be clearly documented and justified. As mentioned above, sufficient interaction between the DRI committee and the systematic review team and technical expert panel will ensure a more efficient process because any differences in judgments would have been discussed before the DRI committee receiving the final systematic review.

For binary or other categorical outcomes, evidence profiles include, at a minimum, a list of all outcomes, the numbers of participants and studies addressing these outcomes, the fundamental study designs (RCTs or observational studies), and measures of both the relative and absolute effect of the intervention compared to control. For continuous outcomes, evidence profiles differ only in the presentation of the effects. Possible presentation of continuous outcomes includes natural units and a number of transformations—standardized mean differences, minimal important difference units, and ratio of means, and binary outcomes—which will then be presented as relative and absolute effects.

RCTs start with a high certainty of evidence. Following the GRADE approach (see below under Factors Determining the Certainty of Evidence), the systematic review team will specify judgments (i.e., not serious, serious, or very serious) about specific factors (i.e., risk of bias, imprecision, inconsistency, indirectness, and publication bias) that may warrant rating down the certainty of evidence about specific outcomes from high to moderate (or low) and influence the judgment about the overall certainty of evidence. These factors, which also apply to observational studies, are discussed in the following section.

Ideally, for observational studies, the evidence profiles from the systematic review team will document each reason for rating down, and may also document factors for rating up, such as the presence of large effect sizes, dose-response gradients, and coverage of all plausible biases (see the following section for more detail). These can reinforce rather than undermine inferences regarding the effects of NOFS interventions. For each outcome and factor, the specific judgment would vary. Furthermore, the overall certainty rating would vary for each outcome.

Summary-of-findings tables are identical to evidence profiles, with one exception: they do not document judgments for each factor (rating up or down) but only the final overall certainty rating and specification of the domains that were rated down or up.

In practice, full documentation of the evidence reviewed can require multiple tables in order to characterize all important details, usually published as supplemental material in conjunction with the systematic review itself. For example, the systematic review of prospective studies of associations of saturated fat and trans fat with noncommunicable diseases as part

Suggested Citation:"6 Evidence Review: Judging the Evidence for Causal Relationships." National Academies of Sciences, Engineering, and Medicine. 2017. Guiding Principles for Developing Dietary Reference Intakes Based on Chronic Disease. Washington, DC: The National Academies Press. doi: 10.17226/24828.
×

of a WHO guideline development process (de Souza et al., 2015) required six appendixes in order to provide details of the search strategy, supplemental tables to describe characteristics of studies reviewed and included in the GRADE summary (20 tables), characteristics of studies reviewed but not included in the GRADE summary, supplemental figures (68 figures), and two final GRADE summary tables, one each for saturated fat and trans fat and including all outcomes.

Figure 6-3 lists the 20 summary tables from the de Souza et al. review (de Souza et al., 2015). It shows that tables may address methodological issues such as the accuracy of measurements (eTable 1) and reported dietary intakes (eTable 20) in addition to findings grouped by study design or analytic approach, exposure, and outcomes. These tables are typically documented with footnotes to clarify definitions, analytic issues, and reasons for rating up or down, as appropriate (Santesso et al., 2016). The GRADE summary tables for saturated fat and trans fat were annotated with 40 and 86 footnotes, respectively. The format for GRADE table entries is shown in Figure 6-4, using the example for trans fat.

Once the systematic review team submits to the DRI committee its systematic review with evidence profiles and summary-of-findings tables for all outcomes of interest, positive and adverse, the committee evaluates the evidence presented, notes and justifies agreements and disagreements, and evaluates the importance of the health outcomes. The committee first makes a decision on the certainty of the evidence with regard to a causal relationship for each outcome they view as important. If such a relationship exists, then the DRI committee characterizes the certainty of the intake-response relationships and considers benefits and harms and their relative importance, as explained in Chapter 7.

Factors Determining the Certainty of Evidence

Initial Rating Based on Study Design: RCTs Versus Observational Studies

GRADE initially rates the certainty of the evidence as high, moderate, low, or very low with the possibility of changing the rating—up or down—based on the factors described below (i.e., risk of bias, imprecision, inconsistency, indirectness, and publication bias). Because randomization—when concealed, practiced with an intention-to-treat approach to the analysis, and with a sufficiently large sample size—deals with the problem of prognostic balance between intervention and control groups, RCTs begin with a high certainty of evidence. However, as discussed in Chapter 3, many NOFS-chronic disease questions will not have been addressed through RCTs for reasons of feasibility or cost. Also, the trials that do exist might have limited value for several reasons related to the form or dose of the

Suggested Citation:"6 Evidence Review: Judging the Evidence for Causal Relationships." National Academies of Sciences, Engineering, and Medicine. 2017. Guiding Principles for Developing Dietary Reference Intakes Based on Chronic Disease. Washington, DC: The National Academies Press. doi: 10.17226/24828.
×
Image
FIGURE 6-3 Example of detailed supplemental information accompanying a published systematic review to support evaluation of the evidence in GRADE summary tables.
NOTE: RR = relative risk; TFA = trans fatty acid.
SOURCE: de Souza et al., 2015, Appendix 2. Reprinted with permission under the Creative Commons Attribution Non Commercial (CC BY-NC 4.0) license: http://creativecommons.org/licenses/by-nc/4.0 (accessed July 15, 2017).
Suggested Citation:"6 Evidence Review: Judging the Evidence for Causal Relationships." National Academies of Sciences, Engineering, and Medicine. 2017. Guiding Principles for Developing Dietary Reference Intakes Based on Chronic Disease. Washington, DC: The National Academies Press. doi: 10.17226/24828.
×

NOFS, study duration, study population, or other factors. In the GRADE approach, RCTs may be rated down for these and other reasons as the following sections describe.

Chapter 3 also discusses the fact that the relevant evidence on NOFSs and chronic disease risk is mostly observational, with large cohort studies playing a central role. In the GRADE system, conclusions based solely on observational studies are judged as having low certainty that a causal relationship is present because of concerns about residual confounding (i.e., confounding that remains after statistical adjustment), bias, and the accuracy of dietary exposure assessment. For observational studies, the certainty that the relationship is causal may be rated up to moderate under specific circumstances. Observational studies also provide information that is used in assessing intake-response relationship and in risk management. However, as noted in Chapter 3, observational studies that have a retrospective case-control or cross-sectional design may be excluded from eligibility for inclusion in systematic reviews because of their lower suitability for supporting judgments about causal relationships, in comparison to prospective studies. If included, this type of evidence would almost always be rated as low or very low quality, for reasons discussed earlier.

Addressing potential confounding typically requires appropriate use of individual-level data capable of characterizing both chronic disease risk and exposure propensity in the study population. Also, when dietary intake measurement depends on self-report, which is affected by individual characteristics, the study exposure assessment would need to have been shown to correlate strongly with a suitable objective measure of exposure (e.g., exposure biomarker) (see also Chapter 4). In the above-cited systematic review of trans fatty acids and noncommunicable disease risk, the authors used a three-tiered rating system to classify the quality of exposure assessment and applied this during their rating of study quality (de Souza et al., 2015, Appendix 2, eTable1).

Observational studies that address both sources of bias (confounding and exposure assessment) in a convincing way, and that report clear intake and chronic disease relationships, can potentially contribute usefully to establish chronic disease DRIs. However, these studies remain as low certainty evidence unless (1) the effect on the outcome is large (relative effect of two or more8), (2) a dose-response gradient is present, or (3) all important biases would either diminish an effect that is present or create an effect where none is observed. In Figure 6-4, the observation of a dose response led to rating up, by one level (from low to moderate) for the body of evidence on associations of both coronary heart disease (CHD) mortality and total

___________________

8 For continuous variables, a statistical method of conversion to binary variables should be used.

Suggested Citation:"6 Evidence Review: Judging the Evidence for Causal Relationships." National Academies of Sciences, Engineering, and Medicine. 2017. Guiding Principles for Developing Dietary Reference Intakes Based on Chronic Disease. Washington, DC: The National Academies Press. doi: 10.17226/24828.
×
FIGURE 6-4 Example of GRADE summary evidence table entries (prospective cohort studies of trans fatty acids and health outcomes).
NOTES: Table entries for associations with ruminant trans fatty acids are not shown. The 86 footnotes annotating this table are also not shown.
SOURCE: de Souza et al., 2015. Excerpt from Appendix 6. Reprinted with permission under the Creative Commons Attribution Non Commercial (CC BY-NC 4.0) license: http://creativecommons.org/licenses/by-nc/4.0 (accessed July 15, 2017).

CHD events with intake of total trans fatty acids (based on table footnotes 15 and 22 in the source; not shown in the figure). In contrast, all evidence relating total trans fatty acids to other outcomes and industrial trans fatty acids to CHD outcomes was rated down from low to very low due to one or more of the following certainty factors: serious risk of bias, serious inconsistency, serious imprecision (footnotes 29, 36, 43, 50, and 57 in the source; not shown in the figure). The nature of these factors, which are especially critical with respect to observational studies, is discussed in the next sections. Bodies of evidence comprising observational studies that are subject to being rated down on the basis of serious limitations would raise substantial uncertainty with respect to supporting causal relationships.

Risk of Bias of Individual Studies (Internal Validity and Study Limitations)

One of the factors considered in assessing the certainty of a body of evidence is the risk of bias of the individual studies. Systematic reviews are expected to critically appraise individual studies and evaluate the totality of evidence for each outcome in terms of the risk of bias (i.e., not serious, serious, or very serious). Key features to consider in risk of bias assessment for RCTs are concealment of randomization, blinding, co-intervention,9 loss to follow-up, sufficient experimental contrast, and conduct of an intention-to-treat analysis. Although GRADE does not mandate the use of particular instruments to assess risk of bias in RCTs, guidance is offered. One suggestion is to use a modified version of the Cochrane Risk of Bias instrument (Guyatt and Busse, 2017).

For observational studies, important features to consider in risk of bias assessment are selection of exposed and unexposed individuals from different cohorts, flawed measurement of exposure and outcome (see Chapters 4

___________________

9 Interventions other than the treatment under study that are applied differently to the treatment and control groups.

Suggested Citation:"6 Evidence Review: Judging the Evidence for Causal Relationships." National Academies of Sciences, Engineering, and Medicine. 2017. Guiding Principles for Developing Dietary Reference Intakes Based on Chronic Disease. Washington, DC: The National Academies Press. doi: 10.17226/24828.
×
Image
Suggested Citation:"6 Evidence Review: Judging the Evidence for Causal Relationships." National Academies of Sciences, Engineering, and Medicine. 2017. Guiding Principles for Developing Dietary Reference Intakes Based on Chronic Disease. Washington, DC: The National Academies Press. doi: 10.17226/24828.
×

and 5), failure to accurately measure all important prognostic factors and to adjust for these factors in the analysis, and loss to follow-up. Because of the diversity in study designs, the availability of a universal tool to assess risk of bias for all study designs is unlikely. It is also not within the study scope or feasible for this committee to delineate an exhaustive list of questions related to risk of bias assessment. The CLARITY Group at McMaster University offers candidate instruments for measuring risk of bias within observational studies (Busse and Guyatt, 2017a,b; Tikkinen et al., 2017a,b). Also, it is expected that DRI committees will include sufficient expertise in epidemiological study designs and their potential limitations, to be able to appropriately assess the risk of bias of the individual studies. As discussed below, the field of nutrition has no well-accepted risk-of-bias tools. There are no tools that prompt for evaluation of certain methodological features that are encountered with respect to assessing causal associations of NOFSs or other aspects of diet with chronic diseases (Chung, 2017). Current efforts are being directed at formally modifying risk-of-bias tools with the addition of questions that are relevant for nutrition (see also Chapter 3 for unique characteristics of nutrition research). Some nutrition questions are topic dependent, such as those related to 25(OH)D assay methods, while others are more generic, such as compliance issue for RCTs and measurement errors/biases in dietary assessment methods (see also Chapter 4). Because intake-response relationships are very important for ultimately developing DRIs, any biases in dietary assessment methods need to be considered in a risk-of-bias assessment by the addition of questions relevant to nutrition (see Chapter 4).

For RCTs involving NOFS exposures from foods, blinding of the intervention exposure is often not possible. The exception is in some controlled feeding studies, but these are usually of short duration and dependent on intermediates as outcome variables. Also, unlike trials of pharmaceuticals, the control group is always exposed to some level or form of the NOFS through intake of foods even in most trials of NOFSs given as a pill supplements. In such cases, unless the baseline exposure of subjects is considered in the trial, the real question being examined in nutrition RCTs involves the health benefits or risks of higher versus lower levels of intake, rather than specific exposure levels of the NOFS. For RCTs, therefore, a question related to whether baseline exposure has been measured and considered would be necessary in order to assess the certainty of the evidence.

In the current absence of a validated, nutrition-specific tool, the Cochrane risk-of-bias tool for RCTs10 and Newcastle-Ottawa Scale11 for cohort studies, supplemented with items specific to nutrition have been used

___________________

10 See http://methods.cochrane.org/bias/assessing-risk-bias-included-studies.

11 See http://www.ohri.ca/programs/clinical_epidemiology/oxford.asp.

Suggested Citation:"6 Evidence Review: Judging the Evidence for Causal Relationships." National Academies of Sciences, Engineering, and Medicine. 2017. Guiding Principles for Developing Dietary Reference Intakes Based on Chronic Disease. Washington, DC: The National Academies Press. doi: 10.17226/24828.
×

in the field of nutrition. Another risk-of-bias tool used in nutrition is the Nutrition Evidence Library Bias Assessment Tool (NEL BAT).12 This tool is based on Cochrane risk-of-bias domains (Selection Bias, Performance Bias, Detection Bias, Attrition Bias) and it is tailored by study design, with different sets of questions applying to RCTs (14 questions), non-RCTs (14 questions), and observational studies (12 questions). However, the tool has no nutrition-specific items.

Inconsistency of Results

Inconsistency is defined as heterogeneity or variability of study results. Issues to consider regarding consistency of results are the extent to which point estimates are similar, the extent to which confidence intervals overlap, and statistical analyses, including I2, a measure of variability, and tests for heterogeneity (Guyatt et al., 2011f). Consistency of outcomes may depend on matters related to the design and conduct of the systematic review itself, such as the sharpness of study question focus, and the inclusion/exclusion criteria for studies included in the analysis, such as the precision of the outcome assessment.

When results have large inconsistencies, effect modifiers can be explored. Subgroup analyses may be misleading, and methodologists have developed criteria to assess the credibility of such analyses. These criteria include whether evidence comes from comparison within or between studies; whether chance can explain the apparent subgroup differences; whether the study questions were developed a priori, deriving from among a small number of hypotheses; and whether the expected direction of the apparent effect was accurately specified in advance (Sun et al., 2010).

Indirectness of Evidence

The available evidence may not provide an ideal match with the question of interest. This occurs, for example, when studies are not available for certain age groups or particular ethnic groups with respect to exposures or outcomes of interest. For example, the WHO systematic review on free sugars and dental caries identified only studies of children, suggesting that conclusions about adults would be indirect and therefore subject to being rated down on this basis. However, in this particular case, a judgment of serious indirectness did not occur, with the following justification:

Data were not downgraded for indirectness though all 8 cohort studies were conducted in children. Aetiology of dental caries is the same in

___________________

12 See http://www.nel.gov/topic.cfm?cat=3384.

Suggested Citation:"6 Evidence Review: Judging the Evidence for Causal Relationships." National Academies of Sciences, Engineering, and Medicine. 2017. Guiding Principles for Developing Dietary Reference Intakes Based on Chronic Disease. Washington, DC: The National Academies Press. doi: 10.17226/24828.
×

children and adults (though enamel of the primary dentition is softer and more vulnerable to demineralization by plaque acid). There were no cohort studies in adults; however, data from 5 out of 5 studies of other study design in adults included in the systematic review detected a statistically significant positive relationshzip between dietary free sugars and levels of dental caries. Dental caries is progressive and tracks from childhood (permanent dentition) to adulthood. 4/8 cohort studies include permanent dentition. Studies were conducted in worldwide populations—Argentina, Brazil, S. Africa, Finland, Sweden, England, USA. (Moynihan and Kelly, 2014, Supplemental data 4, Table S-3, Footnote 4)

The evidence reviewed by Te Morenga et al. relating intake of free sugars to body weight outcomes raised the potential for another type of indirectness based on short study duration (Te Morenga et al., 2012). The RCTs that addressed the impact of free sugar consumption on weight were all relatively short term. This could be considered only indirect evidence for what would happen over a more extended period. However, the evidence was not rated down for serious indirectness on the basis that, “These short-term studies were of sufficient duration to detect a change on the outcome of interest and, thus, the data provide relevant evidence for the association between increased free sugars intakes and weight gain” (Te Morenga et al., 2012, supplement 2, footnote 4).

Other types of indirectness are situations in which available studies of individual NOFSs do not reflect dietary patterns of a population of interest or when only surrogate markers rather than the disease outcomes of interest have been studied. The comment in the above quotation from Moynihan and Kelly about the range of populations studied addresses the breadth of population coverage. In the case of fatty acids, evidence from blood lipids, a possible surrogate marker, was used to facilitate judgments about certain questions, but the overall evidence base included data on mortality and other events.

Studies also may provide only indirect comparisons in other ways. The effect of Diet A versus Diet B may be of interest, but no direct comparisons are available. Yet, if both Diet A and Diet B have been compared to Diet C, this allows the possibility of an indirect comparison between A and B. Such indirect comparisons are at greater risk of producing misleading results than are direct comparisons. However, this linking approach may be useful in complementing information not available in direct comparisons.

Imprecision

In the context of nutritional studies, examination of the boundaries of the confidence interval for a given overall effect informs the decision

Suggested Citation:"6 Evidence Review: Judging the Evidence for Causal Relationships." National Academies of Sciences, Engineering, and Medicine. 2017. Guiding Principles for Developing Dietary Reference Intakes Based on Chronic Disease. Washington, DC: The National Academies Press. doi: 10.17226/24828.
×

regarding precision. Imprecision is the random error in studies that results in wide 95 percent confidence intervals around the estimate of the effect. This differs from inconsistency, which refers to the consistency across individual studies. For each outcome, the authors of each study, or ideally the DRI committee, evaluate the decision regarding whether a proposed intervention or comparator is preferable by selecting estimates at both ends of the confidence interval, to see whether these alternatives would change the interpretation. If it would, then one rates down the evidence for imprecision. For instance, in Figure 6-4, evidence from cohort studies of total trans fatty acid on ischemic stroke was rated down for imprecision because of 95 percent confidence intervals that suggested benefit at the lower bound and exceeded the threshold for harm at the upper bound (de Souza et al., 2015, appendix 6, footnote 27).

This guidance has one exception. If a study or studies report a large or very large effect with a relatively small number of events, experience has shown that the results do not stand the test of time. Therefore, under these circumstances, even if the confidence interval standard suggests satisfactory precision, the study certainty may be rated down on the grounds of imprecision.

If a DRI committee wished to be quantitative about the decisions to rate down a body of evidence under these circumstances, it could specify a magnitude of effect (typically a modest relative risk reduction (RRR), for example ≤30 percent) and calculate the sample size required, in a single study, to have adequate power (80 or 90 percent) with the usual Type 1 error threshold of 0.05 and assume the baseline risk in the studies available. The sample size that results from this exercise is labeled the optimal information size (OIS). If the total number of individuals in a systematic review is less than the number of individuals generated by a conventional sample size calculation for a single adequately powered trial, or the OIS, the committee should consider rating down for imprecision (Schunemann et al., 2013). Many online calculators are available for sample size calculation.13 Alternatively, Table 6-1 provides a guide for what happens, given the number of events observed in the studies, when this exercise is conducted with RRRs of 20 to 30 percent. For example, if an RRR of ≤30 is chosen when the total number of events is 100 or less, the thresholds for precision will most likely not be met (Guyatt et al., 2011e; Pogue and Yusef, 1997).

Publication Bias

Publication bias is defined as the systematic under-estimation or an over-estimation of the underlying beneficial or harmful effect due to

___________________

13 See http://www.stat.ubc.ca/~rollin/stats/ssize/b2.html.

Suggested Citation:"6 Evidence Review: Judging the Evidence for Causal Relationships." National Academies of Sciences, Engineering, and Medicine. 2017. Guiding Principles for Developing Dietary Reference Intakes Based on Chronic Disease. Washington, DC: The National Academies Press. doi: 10.17226/24828.
×

TABLE 6-1 Optimal Information Size Implications When an Adequate Power and Magnitude of Effect Is Desired

Total Number of Events Relative Risk Reduction (%) Implications for Meeting Optimal Information Size Threshold
≤100 ≤30 Will almost never meet threshold whatever control event rate
200 30 Will meet threshold for control group risks of ~25 percent or greater
200 25 Will meet threshold for control group risks of ~50 percent or greater
200 20 Will meet threshold only for control group risks of ~80 percent or greater
300 ≥30 Will meet threshold
300 25 Will meet threshold for control group risks of ~25 percent or greater
≥400 20 Will meet threshold for control group risks of ~40 percent or greater

SOURCE: Adapted from Guyatt et al., 2011e.

the selective publication of studies. In the GRADE approach, options include rating down when the evidence comes from a number of small industry-funded trials, creating and evaluating a funnel plot, and using one of a number of statistical tests that evaluate the asymmetry of the data (small studies showing larger effects than the larger studies) (Guyatt et al., 2011d). The methods guide on grading the strength of a body of evidence by AHRQ describes a conceptual framework and recommends both qualitative and quantitative methods, such as the funnel plot, to assess publication bias. Although both of these groups offer important guidance, they also recognize that measuring publications bias is challenging (AHRQ, 2014; Guyatt et al., 2011d) and that there are no completely satisfactory strategies. This is particularly the case when the body of evidence is from observational studies because assessors might not know if the published observational studies are representative of the studies conducted due to the lack of published or registered protocols. With this in mind, the terms GRADE suggests using are “undetected” and “strongly suspected” publication bias. In the examples in Figure 6-4, publication bias was not assessed formally due to too few studies.

Suggested Citation:"6 Evidence Review: Judging the Evidence for Causal Relationships." National Academies of Sciences, Engineering, and Medicine. 2017. Guiding Principles for Developing Dietary Reference Intakes Based on Chronic Disease. Washington, DC: The National Academies Press. doi: 10.17226/24828.
×

Factors That Can Increase the Certainty of the Evidence

Large Magnitude of an Effect

Rating up for large effect size is typically applied in the context of observational studies. GRADE’s rule of thumb is that evidence may be rated up one level if the odds ratio or relative risk is >2.0 or <0.5 or two levels if the relative risk is >10 or <0.1. The WHO free sugars review (Te Morenga et al., 2012) rated up certainty in evidence based on a large effect of sugar consumption on dental caries. Although large effects are rarely seen with any nutrition intervention unless individuals are deficient in a particular nutrient, the committee concluded that uprating the confidence in the body of evidence of observational studies requires a large effect in the health outcome. This requirement provides higher assurance that the association could not be due to residual confounding.

Intake-Response

Another reason for rating up evidence certainty is observing an intake-response gradient, in this case between the NOFS and the health endpoint of interest. An intake-response gradient implies that the magnitude of the outcome increases in a linear or non-linear fashion with increasing magnitude of exposure and the question being answered would be: “What is the level of confidence that an intake-response relationship gradient exists?” In contrast, Chapter 7 addresses not only whether an intake-response relationship exists but also how to characterize it quantitatively.

The type of evidence needed for rating up the certainty of the evidence based on an intake-response relationship is less stringent than that needed to characterize the relationship quantitatively. For instance, intake levels may be categorical (low, medium, high), so that while evidence of an increase in response with intake exists, a quantitative relationship cannot be determined. Alternatively, one might have high confidence that the “slope” of a linear relationship is greater than zero, but because of potential bias toward the null, confidence is lower as to what the value of the slope might be. In attempting to make this distinction, this report reserves the term quantitative intake-response for a relationship that can be characterized quantitatively.

Effect of Plausible Residual Confounding

Very rarely, evidence certainty may be rated up when all plausible confounding factors would minimize an observed effect, or create an effect when an effect has not been observed. A systematic review that reported

Suggested Citation:"6 Evidence Review: Judging the Evidence for Causal Relationships." National Academies of Sciences, Engineering, and Medicine. 2017. Guiding Principles for Developing Dietary Reference Intakes Based on Chronic Disease. Washington, DC: The National Academies Press. doi: 10.17226/24828.
×

higher death rates in private for-profit versus private not-for-profit hospitals provides an example of the phenomenon of plausible confounding (Guyatt et al., 2011h). Plausible biases included patients in the not-for-profit hospitals being sicker and higher numbers of patients with excellent private insurance coverage in the for-profit hospitals, both of which would be expected to result in better outcomes for private for-profit hospital patients. Because these biases would diminish the observed effect, even if they were not accounted for in the analysis, the circumstances surrounding this confounder might lead the evidence panel to consider the evidence from observational studies as moderate rather than low certainty (Guyatt et al., 2011h).

Rating the Overall Certainty of Evidence

Within the GRADE approach, the evidence is rated by outcome and study design. That is, observational studies are rated separately from RCTs for each outcome. Within evidence from observational studies, those with different designs may also be rated separately. This is illustrated in the list of supplemental tables from a systematic review of observational studies of fatty acids and noncommunicable diseases in Figure 6-3. Typically, certainty of evidence differs across outcomes. For each outcome, when evidence from both RCTs and observational studies is available, the focus should be on the higher certainty evidence, and the overall rating of certainty should be that of the higher certainty. With regard to the overall certainty of evidence, GRADE specifies that it should be designated as the lowest detected among the critical outcomes.

ACCEPTABLE LEVEL OF EVIDENCE THAT THE RELATIONSHIP IS CAUSAL

As already mentioned, once it receives the systematic review with evidence profiles and summary-of-findings tables for all outcomes of interest, positive and adverse, the DRI committee will evaluate the evidence presented, note and justify agreements and disagreements, and evaluate the importance of the health outcomes. The DRI committee first makes a decision on the certainty of the evidence with regard to a causal relationship for each outcome it views as important. If such a relationship exists, then the DRI committee will characterize the certainty of the intake-response relationships, consider benefits and harms and their relative importance, and recommend chronic disease DRIs levels or ranges when appropriate, as explained in Chapter 7.

Determining causation between exposures and outcomes has been a longstanding philosophical as well as practical challenge in population

Suggested Citation:"6 Evidence Review: Judging the Evidence for Causal Relationships." National Academies of Sciences, Engineering, and Medicine. 2017. Guiding Principles for Developing Dietary Reference Intakes Based on Chronic Disease. Washington, DC: The National Academies Press. doi: 10.17226/24828.
×

research as well as for science in general. As discussed above, RCTs, as true experiments, provide the best overall evidence of causation, but the actual certainty in the evidence depends on the certainty of those trials. Other criteria and systems for approaching causation exist, such as the Bradford Hill criteria (Hill, 1965). Parascandola and Weed reviewed the epidemiological literature on causation and found multiple definitions (Parascandola and Weed, 2001).

The Options Report (Yetley et al., 2017) had one set of options that directly addressed causation. Four options were presented, shown in Box 6-4, relating to the levels of confidence that an association between an NOFS exposure and an outcome is causal. All the options portrayed are feasible approaches to defining acceptable levels. The GRADE approach addresses the issue of whether the association between an exposure and an outcome is causal (Schunemann et al., 2013). After considering these approaches in detail and based on the committee’s evaluation of the GRADE approach, the recommendation is that to accept the likelihood of a causal relationship

Suggested Citation:"6 Evidence Review: Judging the Evidence for Causal Relationships." National Academies of Sciences, Engineering, and Medicine. 2017. Guiding Principles for Developing Dietary Reference Intakes Based on Chronic Disease. Washington, DC: The National Academies Press. doi: 10.17226/24828.
×

between an NOFS and a chronic disease outcome, a moderate rating of the relevant overall evidence base derived using GRADE criteria is needed. Lesser levels of confidence in causation should not lead to actionable recommendations or, in the case here, the development of chronic disease DRIs. Presumably, to move forward with developing chronic disease DRIs, determination of an intake-response in data from observational studies as part of the evidence rating also would be needed.

The GRADE system does not specify either a minimum number of studies or participants (although it does provide guidance for minimum number of participants to avoid rating down certainty for imprecision) or the characteristics of the study design (e.g., observational studies have established the causal relationship of smoking and lung cancer) necessary for determining that a relationship has at least moderate certainty, and

Suggested Citation:"6 Evidence Review: Judging the Evidence for Causal Relationships." National Academies of Sciences, Engineering, and Medicine. 2017. Guiding Principles for Developing Dietary Reference Intakes Based on Chronic Disease. Washington, DC: The National Academies Press. doi: 10.17226/24828.
×

therefore is likely to be causal. In the same manner, the DRI committees will apply their judgment relative to this matter.

GUIDING PRINCIPLES AS FOUNDATION FOR A CHRONIC DISEASE DRI PROCESS

Guiding principles suitable for the evidence review process are summarized in Box 6-5. These principles reflect the fact that evidence review in general is well developed as a scientific activity for informing policy and practice. The guiding principles are the foundation of a rigorous scientific process for setting chronic disease DRIs.

Suggested Citation:"6 Evidence Review: Judging the Evidence for Causal Relationships." National Academies of Sciences, Engineering, and Medicine. 2017. Guiding Principles for Developing Dietary Reference Intakes Based on Chronic Disease. Washington, DC: The National Academies Press. doi: 10.17226/24828.
×
Suggested Citation:"6 Evidence Review: Judging the Evidence for Causal Relationships." National Academies of Sciences, Engineering, and Medicine. 2017. Guiding Principles for Developing Dietary Reference Intakes Based on Chronic Disease. Washington, DC: The National Academies Press. doi: 10.17226/24828.
×

REFERENCES

AHRQ (Agency for Healthcare Research and Quality). 2014. Methods guide for effectiveness and comparative effectiveness reviews. AHRQ Publication No. 10(14)-EHC063-EF. Rockville, MD: AHRQ.

Alonso-Coello, P., H. J. Schunemann, J. Moberg, R. Brignardello-Petersen, E. A. Akl, M. Davoli, S. Treweek, R. A. Mustafa, G. Rada, S. Rosenbaum, A. Morelli, G. H. Guyatt, A. D. Oxman, and Grade Working Group. 2016a. GRADE Evidence to Decision (EtD) frameworks: A systematic and transparent approach to making well informed healthcare choices. 1: Introduction. BMJ 353:i2016.

Alonso-Coello, P., A. D. Oxman, J. Moberg, R. Brignardello-Petersen, E. A. Akl, M. Davoli, S. Treweek, R. A. Mustafa, P. O. Vandvik, J. Meerpohl, G. H. Guyatt, H. J. Schunemann, and Grade Working Group. 2016b. GRADE Evidence to Decision (EtD) frameworks: A systematic and transparent approach to making well informed healthcare choices. 2: Clinical practice guidelines. BMJ 353:i2089.

Andrews, J., G. Guyatt, A. D. Oxman, P. Alderson, P. Dahm, Y. Falck-Ytter, M. Nasser, J. Meerpohl, P. N. Post, R. Kunz, J. Brozek, G. Vist, D. Rind, E. A. Akl, and H. J. Schunemann. 2013a. GRADE guidelines: 14. Going from evidence to recommendations: The significance and presentation of recommendations. J Clin Epidemiol 66(7):719-725.

Andrews, J. C., H. J. Schunemann, A. D. Oxman, K. Pottie, J. J. Meerpohl, P. A. Coello, D. Rind, V. M. Montori, J. P. Brito, S. Norris, M. Elbarbary, P. Post, M. Nasser, V. Shukla, R. Jaeschke, J. Brozek, B. Djulbegovic, and G. Guyatt. 2013b. GRADE guidelines: 15. Going from evidence to recommendation—Determinants of a recommendation’s direction and strength. J Clin Epidemiol 66(7):726-735.

Balshem, H., M. Helfand, H. J. Schunemann, A. D. Oxman, R. Kunz, J. Brozek, G. E. Vist, Y. Falck-Ytter, J. Meerpohl, S. Norris, and G. H. Guyatt. 2011. GRADE guidelines: 3. Rating the quality of evidence. J Clin Epidemiol 64(4):401-406.

Brannon, P. M., C. L. Taylor, and P. M. Coates. 2014. Use and applications of systematic reviews in public health nutrition. Annu Rev Nutr 34:401-419.

Brannon, P. M., C. M. Weaver, C. A. Anderson, S. M. Donovan, S. P. Murphy, and A. L. Yaktine. 2016. Scanning for new evidence to prioritize updates to the Dietary Reference Intakes: Case studies for thiamin and phosphorus. Am J Clin Nutr 104(5):1366-1377.

Brouwer, I. A. 2016. Effect of trans-fatty acid intake on blood lipids and lipoproteins: A systematic review and meta-regression analysis. Geneva, Switzerland: World Health Organization.

Busse, J. W., and G. Guyatt. 2017a. Tool to assess risk of bias in case control studies. https://www.evidencepartners.com/resources/methodological-resources (accessed May 16, 2017).

Busse, J. W., and G. Guyatt. 2017b. Tool to assess risk of bias in cohort studies. https://www.evidencepartners.com/resources/methodological-resources (accessed May 16, 2017).

Chung, M. 2017. Overview and comparisons of risk of bias and strength of evidence assessment tools: Opportunities and challenges of application in developing DRIs. Presented at the Workshop of the Committee on Development of Guiding Principles for the Inclusion of Chronic Disease Endpoints in Future Dietary Reference Intakes, January 9, 2017, Washington, DC.

Chung, M., S. J. Newberry, M. T. Ansari, W. W. Yu, H. Wu, J. Lee, M. Suttorp, J. M. Gaylor, A. Motala, D. Moher, E. M. Balk, and P. G. Shekelle. 2012. Two methods provide similar signals for the need to update systematic reviews. J Clin Epidemiol 65(6):660-668.

Suggested Citation:"6 Evidence Review: Judging the Evidence for Causal Relationships." National Academies of Sciences, Engineering, and Medicine. 2017. Guiding Principles for Developing Dietary Reference Intakes Based on Chronic Disease. Washington, DC: The National Academies Press. doi: 10.17226/24828.
×

de Souza, R. J., A. Mente, A. Maroleanu, A. I. Cozma, V. Ha, T. Kishibe, E. Uleryk, P. Budylowski, H. Schunemann, J. Beyene, and S. S. Anand. 2015. Intake of saturated and trans unsaturated fatty acids and risk of all cause mortality, cardiovascular disease, and type 2 diabetes: Systematic review and meta-analysis of observational studies. BMJ 351:h3978.

Dwyer, J. T., K. H. Rubin, K. L. Fritsche, T. L. Psota, D. J. Liska, W. S. Harris, S. J. Montain, and B. J. Lyle. 2016. Creating the future of evidence-based nutrition recommendations: Case studies from lipid research. Adv Nutr 7(4):747-755.

Garcia-Larsen, V., S. R. Del Giacco, A. Moreira, M. Bonini, D. Charles, T. Reeves, K. H. Carlsen, T. Haahtela, S. Bonini, J. Fonseca, I. Agache, N. G. Papadopoulos, and L. Delgado. 2016. Asthma and dietary intake: An overview of systematic reviews. Allergy 71(4):433-442.

Garner, P., S. Hopewell, J. Chandler, H. MacLehose, H. J. Schunemann, E. A. Akl, J. Beyene, S. Chang, R. Churchill, K. Dearness, G. Guyatt, C. Lefebvre, B. Liles, R. Marshall, L. Martinez Garcia, C. Mavergames, M. Nasser, A. Qaseem, M. Sampson, K. Soares-Weiser, Y. Takwoingi, L. Thabane, M. Trivella, P. Tugwell, E. Welsh, E. C. Wilson, H. J. Schunemann, and Panel for Updating Guidance for Systematic Reviews. 2016. When and how to update systematic reviews: Consensus and checklist. BMJ 354:i3507.

Guyatt, G., and J. W. Busse. 2017. Modification of Cochrane tool to assess risk of bias in randomized trials. https://www.evidencepartners.com/resources/methodological-resources (accessed May 12, 2017).

Guyatt, G. H., A. D. Oxman, R. Kunz, Y. Falck-Ytter, G. E. Vist, A. Liberati, H. J. Schunemann, and Grade Working Group. 2008a. Going from evidence to recommendations. BMJ 336(7652):1049-1051.

Guyatt, G. H., A. D. Oxman, G. E. Vist, R. Kunz, Y. Falck-Ytter, P. Alonso-Coello, H. J. Schunemann, and Grade Working Group. 2008b. GRADE: An emerging consensus on rating quality of evidence and strength of recommendations. BMJ 336(7650):924-926.

Guyatt, G. H., A. D. Oxman, R. Kunz, R. Jaeschke, M. Helfand, A. Liberati, G. E. Vist, H. J. Schunemann, and Grade Working Group. 2008c. Incorporating considerations of resources use into grading recommendations. BMJ 336(7654):1170-1173.

Guyatt, G. H., A. D. Oxman, R. Kunz, G. E. Vist, Y. Falck-Ytter, H. J. Schunemann, and Grade Working Group. 2008d. What is “quality of evidence” and why is it important to clinicians? BMJ 336(7651):995-998.

Guyatt, G., A. D. Oxman, E. A. Akl, R. Kunz, G. Vist, J. Brozek, S. Norris, Y. Falck-Ytter, P. Glasziou, H. DeBeer, R. Jaeschke, D. Rind, J. Meerpohl, P. Dahm, and H. J. Schunemann. 2011a. GRADE guidelines: 1. Introduction-GRADE evidence profiles and summary of findings tables. J Clin Epidemiol 64(4):383-394.

Guyatt, G. H., A. D. Oxman, R. Kunz, D. Atkins, J. Brozek, G. Vist, P. Alderson, P. Glasziou, Y. Falck-Ytter, and H. J. Schunemann. 2011b. GRADE guidelines: 2. Framing the question and deciding on important outcomes. J Clin Epidemiol 64(4):395-400.

Guyatt, G. H., A. D. Oxman, G. Vist, R. Kunz, J. Brozek, P. Alonso-Coello, V. Montori, E. A. Akl, B. Djulbegovic, Y. Falck-Ytter, S. L. Norris, J. W. Williams, Jr., D. Atkins, J. Meerpohl, and H. J. Schunemann. 2011c. GRADE guidelines: 4. Rating the quality of evidence—Study limitations (risk of bias). J Clin Epidemiol 64(4):407-415.

Guyatt, G. H., A. D. Oxman, V. Montori, G. Vist, R. Kunz, J. Brozek, P. Alonso-Coello, B. Djulbegovic, D. Atkins, Y. Falck-Ytter, J. W. Williams, Jr., J. Meerpohl, S. L. Norris, E. A. Akl, and H. J. Schunemann. 2011d. GRADE guidelines: 5. Rating the quality of evidence—Publication bias. J Clin Epidemiol 64(12):1277-1282.

Suggested Citation:"6 Evidence Review: Judging the Evidence for Causal Relationships." National Academies of Sciences, Engineering, and Medicine. 2017. Guiding Principles for Developing Dietary Reference Intakes Based on Chronic Disease. Washington, DC: The National Academies Press. doi: 10.17226/24828.
×

Guyatt, G. H., A. D. Oxman, R. Kunz, J. Brozek, P. Alonso-Coello, D. Rind, P. J. Devereaux, V. M. Montori, B. Freyschuss, G. Vist, R. Jaeschke, J. W. Williams, Jr., M. H. Murad, D. Sinclair, Y. Falck-Ytter, J. Meerpohl, C. Whittington, K. Thorlund, J. Andrews, and H. J. Schunemann. 2011e. GRADE guidelines 6. Rating the quality of evidence—Imprecision. J Clin Epidemiol 64(12):1283-1293.

Guyatt, G. H., A. D. Oxman, R. Kunz, J. Woodcock, J. Brozek, M. Helfand, P. Alonso-Coello, P. Glasziou, R. Jaeschke, E. A. Akl, S. Norris, G. Vist, P. Dahm, V. K. Shukla, J. Higgins, Y. Falck-Ytter, H. J. Schunemann, and Grade Working Group. 2011f. GRADE guidelines: 7. Rating the quality of evidence—Inconsistency. J Clin Epidemiol 64(12):1294-1302.

Guyatt, G. H., A. D. Oxman, R. Kunz, J. Woodcock, J. Brozek, M. Helfand, P. Alonso-Coello, Y. Falck-Ytter, R. Jaeschke, G. Vist, E. A. Akl, P. N. Post, S. Norris, J. Meerpohl, V. K. Shukla, M. Nasser, H. J. Schunemann, and Grade Working Group. 2011g. GRADE guidelines: 8. Rating the quality of evidence—Indirectness. J Clin Epidemiol 64(12):1303-1310.

Guyatt, G. H., A. D. Oxman, S. Sultan, P. Glasziou, E. A. Akl, P. Alonso-Coello, D. Atkins, R. Kunz, J. Brozek, V. Montori, R. Jaeschke, D. Rind, P. Dahm, J. Meerpohl, G. Vist, E. Berliner, S. Norris, Y. Falck-Ytter, M. H. Murad, H. J. Schunemann, and Grade Working Group. 2011h. GRADE guidelines: 9. Rating up the quality of evidence. J Clin Epidemiol 64(12):1311-1316.

Hill, A. B. 1965. The environment and disease: Association or causation? Proc R Soc Med 58:295-300.

IOM (Institute of Medicine). 2011a. Dietary Reference Intakes for calcium and vitamin D. Washington, DC: The National Academies Press.

IOM. 2011b. Finding what works in health care: Standards for systematic reviews. Washington, DC: The National Academies Press.

Jaeschke, R., G. H. Guyatt, P. Dellinger, H. Schunemann, M. M. Levy, R. Kunz, S. Norris, J. Bion, and Grade Working Group. 2008. Use of GRADE grid to reach decisions on clinical practice guidelines when consensus is elusive. BMJ 337:a744.

Mayhew, A. J., R. J. de Souza, D. Meyre, S. S. Anand, and A. Mente. 2016. A systematic review and meta-analysis of nut consumption and incident risk of CVD and all-cause mortality. Br J Nutr 115(2):212-225.

Mensink, R. P. 2016. Effects of saturated fatty acids on serum lipids and lipoproteins: A systematic review and regression analysis. Geneva, Switzerland: World Health Organization.

Moynihan, P. J., and S. A. Kelly. 2014. Effect on caries of restricting sugars intake: Systematic review to inform WHO guidelines. J Dent Res 93(1):8-18.

NASEM (National Academies of Sciences, Engineering, and Medicine). 2017. Optimizing the process for establishing the Dietary Guidelines for Americans: The selection process. Washington, DC: The National Academies Press.

Naude, C. E., A. Schoonees, M. Senekal, T. Young, P. Garner, and J. Volmink. 2014. Low carbohydrate versus isoenergetic balanced diets for reducing weight and cardiovascular risk: A systematic review and meta-analysis. PLOS ONE 9(7):e100652.

NEL (Nutrition Evidence Library). 2015. Nutrition Evidence Library—2015 Dietary Guidelines Advisory Committee systematic reviews. https://www.cnpp.usda.gov/nutrition-evidence-library-2015-dietary-guidelines-advisory-committee-systematic-reviews (accessed April 28, 2017).

NTP (National Toxicology Program). 2015. Handbook for conducting a literature-based health assessment using OHAT approach for systematic review and evidence integration. https://ntp.niehs.nih.gov/pubhealth/hat/noms/index-2.html (accessed July 16, 2017).

Parascandola, M., and D. L. Weed. 2001. Causation in epidemiology. J Epidemiol Community Health 55(12):905-912.

Suggested Citation:"6 Evidence Review: Judging the Evidence for Causal Relationships." National Academies of Sciences, Engineering, and Medicine. 2017. Guiding Principles for Developing Dietary Reference Intakes Based on Chronic Disease. Washington, DC: The National Academies Press. doi: 10.17226/24828.
×

Pennant, M., M. Steur, C. Moore, A. Butterworth, and L. Johnson. 2015. Comparative validity of vitamin C and carotenoids as indicators of fruit and vegetable intake: A systematic review and meta-analysis of randomised controlled trials. Br J Nutr 114(9):1331-1340.

Pieper, D., T. Mathes, and M. Eikermann. 2014. Can AMSTAR also be applied to systematic reviews of non-randomized studies? BMC Res Notes 7:609.

Pogue, J. M., and S. Yusuf. 1997. Cumulating evidence from randomized trials: Utilizing sequential monitoring boundaries for cumulative meta-analysis. Control Clin Trials 18(6):580-593; discussion 661-666.

Puhan, M. A., H. J. Schunemann, M. H. Murad, T. Li, R. Brignardello-Petersen, J. A. Singh, A. G. Kessels, G. H. Guyatt, and Grade Working Group. 2014. A GRADE Working Group approach for rating the quality of treatment effect estimates from network meta-analysis. BMJ 349:g5630.

Santesso, N., E. A. Akl, M. Bianchi, A. Mente, R. Mustafa, D. Heels-Ansdell, and H. J. Schunemann. 2012. Effects of higher-versus lower-protein diets on health outcomes: A systematic review and meta-analysis. Eur J Clin Nutr 66(7):780-788.

Santesso, N., A. Carrasco-Labra, M. Langendam, R. Brignardello-Petersen, R. A. Mustafa, P. Heus, T. Lasserson, N. Opiyo, I. Kunnamo, D. Sinclair, P. Garner, S. Treweek, D. Tovey, E. A. Akl, P. Tugwell, J. L. Brozek, G. Guyatt, and H. J. Schunemann. 2016. Improving GRADE evidence tables part 3: Detailed guidance for explanatory footnotes supports creating and understanding GRADE certainty in the evidence judgments. J Clin Epidemiol 74:28-39.

Schunemann, H. J., A. D. Oxman, J. Brozek, P. Glasziou, R. Jaeschke, G. E. Vist, J. W. Williams, Jr., R. Kunz, J. Craig, V. M. Montori, P. Bossuyt, G. H. Guyatt, and Grade Working Group. 2008. Grading quality of evidence and strength of recommendations for diagnostic tests and strategies. BMJ 336(7653):1106-1110.

Schunemann, H., J. Brozek, G. Guyatt, and A. Oxman. 2013. Introduction to GRADE Handbook. http://gdt.guidelinedevelopment.org/app/handbook/handbook.html (accessed May 16, 2017).

Shea, B. J., J. M. Grimshaw, G. A. Wells, M. Boers, N. Andersson, C. Hamel, A. C. Porter, P. Tugwell, D. Moher, and L. M. Bouter. 2007. Development of AMSTAR: A measurement tool to assess the methodological quality of systematic reviews. BMC Med Res Methodol 7:10.

Shea, B. J., C. Hamel, G. A. Wells, L. M. Bouter, E. Kristjansson, J. Grimshaw, D. A. Henry, and M. Boers. 2009. AMSTAR is a reliable and valid measurement tool to assess the methodological quality of systematic reviews. J Clin Epidemiol 62(10):1013-1020.

Shekelle, P., S. Newberry, M. Maglione, R. Shanman, B. Johnsen, J. Carter, A. Motala, B. Hulley, Z. Wang, D. Bravata, M. Chen, and J. Grossman. 2009. Assessment of the need to update comparative effectiveness reviews: Report of an initial rapid program assessment (2005-2009), Rockville, MD: Agency for Healthcare Research and Quality. https://www.ncbi.nlm.nih.gov/pubmed/21204320 (accessed July 16, 2017).

Sun, X., M. Briel, S. D. Walter, and G. H. Guyatt. 2010. Is a subgroup effect believable? Updating criteria to evaluate the credibility of subgroup analyses. BMJ 340:c117.

Te Morenga, L., S. Mallard, and J. Mann. 2012. Dietary sugars and body weight: Systematic review and meta-analyses of randomised controlled trials and cohort studies. BMJ 346:e7492.

Tikkinen, K. A., J. W. Busse, and G. Guyatt. 2017a. Tool to assess risk of bias in observational studies of natural history of medical symptoms/conditions in general populations. https://www.evidencepartners.com/resources/methodological-resources (accessed May 16, 2017).

Suggested Citation:"6 Evidence Review: Judging the Evidence for Causal Relationships." National Academies of Sciences, Engineering, and Medicine. 2017. Guiding Principles for Developing Dietary Reference Intakes Based on Chronic Disease. Washington, DC: The National Academies Press. doi: 10.17226/24828.
×

Tikkinen, K. A., R. Cartwright, and G. Guyatt. 2017b. Tool to assess risk of bias in studies of prognosis. https://www.evidencepartners.com/resources/methodological-resources (accessed May 16, 2017).

USPSTF (U.S. Preventive Services Task Force). 2015. Procedure manual. https://www.uspreventiveservicestaskforce.org/Page/Name/methods-and-processes (accessed July 16, 2017).

WCRF/AICR (World Cancer Research Fund/American Institute for Cancer Research). 2007. Food, nutrition, physical activity, and the prevention of cancer: A global perspective. Washington, DC: AICR.

Whiting, P., P. Davies, J. Savovic, D. M. Caldwell, and R. Churchill. 2013. Evidence to inform the development of ROBIS, a new tool to assess the risk of bias in systematic reviews. http://www.bristol.ac.uk/media-library/sites/social-community-medicine/robis/ROBIS%20Report%204_9.pdf (accessed July 16, 2017).

Whiting, P., J. Savovic, J. P. Higgins, D. M. Caldwell, B. C. Reeves, B. Shea, P. Davies, J. Kleijnen, R. Churchill, and ROBIS group. 2016. ROBIS: A new tool to assess risk of bias in systematic reviews was developed. J Clin Epidemiol 69:225-234.

WHO (World Health Organization). 2014. WHO handbook for guideline development. Geneva, Switzerland: World Health Organization.

WHO. 2015. Sugars intake for adults and children. Guideline. Geneva, Switzerland: World Health Organization.

Yetley, E. A., A. J. MacFarlane, L. S. Greene-Finestone, C. Garza, J. D. Ard, S. A. Atkinson, D. M. Bier, A. L. Carriquiry, W. R. Harlan, D. Hattis, J. C. King, D. Krewski, D. L. O’Connor, R. L. Prentice, J. V. Rodricks, and G. A. Wells. 2017. Options for basing Dietary Reference Intakes (DRIs) on chronic disease endpoints: Report from a joint US-/ Canadian-sponsored working group. Am J Clin Nutr 105(1):249S-285S.

Suggested Citation:"6 Evidence Review: Judging the Evidence for Causal Relationships." National Academies of Sciences, Engineering, and Medicine. 2017. Guiding Principles for Developing Dietary Reference Intakes Based on Chronic Disease. Washington, DC: The National Academies Press. doi: 10.17226/24828.
×

ANNEX 6-1
PRIORITY QUESTIONS IN THE FORMAT OF POPULATION, INTERVENTION, CONTROL, AND OUTCOMES (PICO)14 FOR GUIDING THE SYSTEMATIC REVIEWS IN SUPPORT OF THE WORLD HEALTH ORGANIZATION GUIDELINE ON FREE SUGARS INTAKE FOR ADULTS AND CHILDREN

What is the effect of a decrease or increase in free sugars intake in adults and children?

What is the effect of restricting intake of free sugars to below 10 percent of total energy intake?15

Adults
Population Apparently healthy adults in low-, middle-, and high-income countries
  • In each, consider population characteristics, such as age, gender, ethnicity, country/region (urban/rural), socioeconomic status/demographic factors/sanitation, health background, and health status
Intervention/exposure Definitions
  • Total sugars
  • Free sugars
  • Added sugars (sucrose; table sugar; sugars in processed foods)
  • Percent of total energy intake from sugars
  • Consumption of sugar-sweetened beverages
  • Fruit juices
Control Comparison of levels
Continuous or categorical
Adherence to recommendations

___________________

14 Reprinted with permission from WHO (World Health Organization). 2015. Sugars intake for adults and children. Guideline. Geneva, Switzerland: World Health Organization.

15 Less than 10 percent of total energy intake is the existing population nutrient intake goal for free sugars (WHO/FAO, 2003).

Suggested Citation:"6 Evidence Review: Judging the Evidence for Causal Relationships." National Academies of Sciences, Engineering, and Medicine. 2017. Guiding Principles for Developing Dietary Reference Intakes Based on Chronic Disease. Washington, DC: The National Academies Press. doi: 10.17226/24828.
×
Confounders/effect modifiers/intermediates
  • Baseline level of all categories of sugars intake
  • Energy intake
  • Energy expenditure, fitness, and physical activity
  • Consider other interventions in design, dietary and non-dietary (protocol to specify)
  • Consider influence of other aspects of diet/dietary patterns

In cohort studies: unadjusted and adjusted estimates; what adjusted for, how (protocol to specify), and impact

Consider whether artificial sweeteners/milk/other foods are used as control

Intermediates

  • Take into account effect of energy density
Outcome
  • Body weight or fatness gain measured by:
    • weight change, BMI
    • incidence of obesity and overweight
    • body fatness16 and distribution assessed in a variety of ways
  • Dental caries (not erosion)
Time frame
  • For controlled feeding studies where a high proportion of food is directly provided and there is no caloric restriction, outcomes are change in weight or body fatness within a minimum study duration of 8 weeks
  • For studies where the intervention is advisory or shopping type, without caloric restriction, outcomes are obesity incidence, change in weight or body fatness with a minimum study duration of 6 months (26 weeks)

___________________

16 The percentage of fat (i.e., adipose tissue) that a person’s body contains.

Suggested Citation:"6 Evidence Review: Judging the Evidence for Causal Relationships." National Academies of Sciences, Engineering, and Medicine. 2017. Guiding Principles for Developing Dietary Reference Intakes Based on Chronic Disease. Washington, DC: The National Academies Press. doi: 10.17226/24828.
×
Children
Population Apparently healthy children in low-, middle-, and high-income countries
  • In each, consider population characteristics, such as age, gender, ethnicity, country/region (urban/rural), socioeconomic status/demographic factors/sanitation, health background, and health status
Intervention/exposure Definitions
  • Total sugars
  • Free sugars
  • Added sugars (sucrose; table sugar; sugars in processed foods)
  • Percent of total energy intake from sugars
  • Consumption of sugar-sweetened beverages
  • Fruit juices
Control Comparison of levels
Continuous or categorical
Adherence to recommendations
Confounders/effect modifiers/intermediates
  • Baseline level of all categories of sugars intake
  • Energy intake
  • Energy expenditure, fitness, and physical activity
  • Consider other interventions in design, dietary and non-dietary (protocol to specify)
  • Consider influence of other aspects of diet/dietary patterns

In cohort studies: unadjusted and adjusted estimates; what adjusted for, how (protocol to specify), and impact

Consider if artificial sweeteners/milk/other foods are used as control

Intermediates

  • Take into account effect of energy density
Suggested Citation:"6 Evidence Review: Judging the Evidence for Causal Relationships." National Academies of Sciences, Engineering, and Medicine. 2017. Guiding Principles for Developing Dietary Reference Intakes Based on Chronic Disease. Washington, DC: The National Academies Press. doi: 10.17226/24828.
×
Outcome
  • Body weight or fatness gain measured by:
    • weight change, BMI
    • incidence of obesity and overweight
    • body fatness and distribution assessed in a variety of ways
  • Dental caries (not erosion)
Time frame
  • For controlled feeding studies where a high proportion of food is directly provided and there is no caloric restriction, outcomes are change in weight or body fatness, with a minimum study duration of 8 weeks
  • For studies where the intervention is advisory or shopping type, without caloric restriction, outcomes are obesity incidence, change in weight or body fatness, with a minimum study duration of 6 months (26 weeks)

ANNEX 6-1 REFERENCES

WHO (World Health Organization). 2015. Sugars intake for adults and children. Guideline. Geneva, Switzerland: World Health Organization.

WHO/FAO (Food and Agriculture Organization). 2003. Diet, nutrition and the prevention of chronic diseases: Report of a joint WHO/FAO expert consultation. Geneva, Switzerland: World Health Organization.

Suggested Citation:"6 Evidence Review: Judging the Evidence for Causal Relationships." National Academies of Sciences, Engineering, and Medicine. 2017. Guiding Principles for Developing Dietary Reference Intakes Based on Chronic Disease. Washington, DC: The National Academies Press. doi: 10.17226/24828.
×

ANNEX 6-2
ASSESSMENT TOOLS USED IN SYSTEMATIC REVIEWS

Suggested Citation:"6 Evidence Review: Judging the Evidence for Causal Relationships." National Academies of Sciences, Engineering, and Medicine. 2017. Guiding Principles for Developing Dietary Reference Intakes Based on Chronic Disease. Washington, DC: The National Academies Press. doi: 10.17226/24828.
×

TABLE ANNEX 6-2-1 Examples of Tools for Risk of Bias (or Quality) Assessment for Individual Studies Included in the Systematic Reviews

Tool (and Modifications) Purpose/Who Is Using It Risk of Bias Domains Grading Method Notes
Nutrition Evidence Library Bias Assessment Tool (NEL BAT)a To assess the risk of bias of each individual study (any design) included in a systematic review

Used by the 2015 Dietary Guidelines Advisory Committee

  • Selection bias
  • Performance bias
  • Detection bias
  • Attrition bias
The NEL BAT is tailored by study design, with different sets of questions applying to RCTs, non-RCTs, and observational studies.

There are four response options, which are given a score:

  • Yes (0 points)
  • No (2 points)
  • Cannot Determine (1 point)
  • N/A (0 points)

The lower the number of points received, the lower the risk of bias.

Adapted from the Cochrane Bias Methods Group

No nutrition-specific items

Suggested Citation:"6 Evidence Review: Judging the Evidence for Causal Relationships." National Academies of Sciences, Engineering, and Medicine. 2017. Guiding Principles for Developing Dietary Reference Intakes Based on Chronic Disease. Washington, DC: The National Academies Press. doi: 10.17226/24828.
×
Tool (and Modifications) Purpose/Who Is Using It Risk of Bias Domains Grading Method Notes
Cochrane Risk of Bias Toolb To assess the quality of intervention studies for use in the context of a systematic review and elsewhere

It is a domain-based evaluation, in which critical assessments are made separately for different domains.

Used in Cochrane systematic reviews

  • Selection bias
  • Performance bias
  • Detection bias
  • Attrition bias
  • Reporting bias
  • Other bias
This is a two-part tool. Each part contains criteria that reviewers use to formulate their descriptions and judgments.

Part 1: Within each domain entry, reviewers describe what was reported to have happened in the study in sufficient detail to support a judgment about the risk of bias.

Part 2: Reviewers assign a judgement relating to the risk of bias for that entry. This is achieved by assigning a judgement of Low risk of bias, High risk of bias, or Unclear risk of bias.

Used for randomized studies

Supplemented with nutrition-specific items

Suggested Citation:"6 Evidence Review: Judging the Evidence for Causal Relationships." National Academies of Sciences, Engineering, and Medicine. 2017. Guiding Principles for Developing Dietary Reference Intakes Based on Chronic Disease. Washington, DC: The National Academies Press. doi: 10.17226/24828.
×
Risk of Bias in Nonrandomized Studies- of Interventions (ROBINS-I)c To evaluate the risk of bias in the results of nonrandomized intervention studies that compare the health effects of two or more interventions

Developed by members of the Cochrane Bias Methods Group and the Cochrane Nonrandomized Studies Methods Group

  • Bias due to confounding
  • Bias in selection of participants into the study
  • Bias in classification of interventions
  • Bias due to deviations from intended interventions
  • Bias due to missing data
  • Bias in measurement of outcomes
  • Bias in selection of the reported result
Domain-level risk of bias judgements are made, followed by overall judgements about risk of bias.

Response options are:

  1. Low risk of bias
  2. Moderate risk of bias
  3. Serious risk of bias
  4. Critical risk of bias
  5. No information on which to base a judgement about risk of bias
Used for nonrandomized studies

Not inclusive of study characteristics that are relevant for nutrition

Can be outcome specific

Provides a detailed assessment of bias domains

The Newcastle-Ottawa Scale (NOS)d To assess the quality of non-randomized studies (cohort or case-control studies) in meta-analyses and systematic reviews

Used by AHRQ

  1. Selection of the study groups
  2. Comparability of the groups
  3. Ascertainment of either the exposure or outcome of interest for case-control or cohort studies, respectively
A star system in which a study is judged on the three domains. Each domain has one to four questions to answer. A different set of questions is used for cohort studies and case-control studies. Designed for case-control and cohort studies

Supplemented with nutrition-specific items The face/content validity of the NOS has been established.

The NOS has been refined based on experience using it in several projects.

Suggested Citation:"6 Evidence Review: Judging the Evidence for Causal Relationships." National Academies of Sciences, Engineering, and Medicine. 2017. Guiding Principles for Developing Dietary Reference Intakes Based on Chronic Disease. Washington, DC: The National Academies Press. doi: 10.17226/24828.
×
Tool (and Modifications) Purpose/Who Is Using It Risk of Bias Domains Grading Method Notes
National Toxicology Program (NTP) Office of Health Assessment and Translation (OHAT) Risk of Bias Toole To assess internal validity of individual studies (human and nonhuman animal studies) typically considered in environmental health reviews Selection bias
  • Confounding bias
  • Performance bias
  • Attrition/Exclusion bias
  • Detection bias
  • Selective reporting bias
  • Other sources of bias
Each domain has one to three questions, depending on study type.

Response options are:

Definitely Low risk of bias: There is direct evidence of low risk of bias practices.

Probably Low risk of bias: There is indirect evidence of low risk of bias practices, or it is deemed that deviations from low risk of bias practices for these criteria during the study would not appreciably bias results, including consideration of direction and magnitude of bias.

Probably High risk of bias: There is indirect evidence of high risk of bias practices, or there is insufficient information provided about relevant risk of bias practices.

This tool was developed using established guidelines for experimental human studies (randomized clinical trials).
Suggested Citation:"6 Evidence Review: Judging the Evidence for Causal Relationships." National Academies of Sciences, Engineering, and Medicine. 2017. Guiding Principles for Developing Dietary Reference Intakes Based on Chronic Disease. Washington, DC: The National Academies Press. doi: 10.17226/24828.
×
Tool (and Modifications) Purpose/Who Is Using It Risk of Bias Domains Grading Method Notes

Definitely High risk of bias: There is direct evidence of high risk of bias practices.

a See http://www.nel.gov/topic.cfm?cat=3384 (accessed July 24, 2017).

b See http://methods.cochrane.org/bias/assessing-risk-bias-included-studies (accessed July 24, 2017); http://handbook.cochrane.org (accessed July 24, 2017); Higgins et al., 2011.

c See https://sites.google.com/site/riskofbiastool/welcome/home (accessed July 24, 2017); Sterne et al., 2016.

d See http://www.ohri.ca/programs/clinical_epidemiology/oxford.asp (accessed July 24, 2017).

e See https://ntp.niehs.nih.gov/pubhealth/hat/noms/index-2.html (accessed July 24, 2017).

Suggested Citation:"6 Evidence Review: Judging the Evidence for Causal Relationships." National Academies of Sciences, Engineering, and Medicine. 2017. Guiding Principles for Developing Dietary Reference Intakes Based on Chronic Disease. Washington, DC: The National Academies Press. doi: 10.17226/24828.
×

TABLE ANNEX 6-2-2 Tools for Evaluating the Strength (or Quality) of the Body of Evidence*

Tool (and modifications) Purpose/Who Is Using It Elements (or Quality Criteria) Grading Method Notes
Grading of Recommendations, Assessment, Development and Evaluation (GRADE)a

The working group discourages the use of “modified GRADE approaches” that differ from the approach described by the GRADE Working Group.

GRADE is an approach to evaluate the quality of the evidence and the strength of recommendations

Used for clinical practice guideline development

Used to evaluate the evidence in systematic reviews

Recommended approach for use in Cochrane Reviews

An initial confidence is determined based on study design:

RCTs start as High quality evidence. Observational studies start with a Low quality rating.

Studies may be downgraded based on:

  • Study limitations (risk of bias)
  • Inconsistency of results
  • Indirectness of evidence
  • Imprecision
  • Reporting bias

Studies may be upgraded based on:

  • Magnitude of treatment effect
  • Evidence of dose-response relationship
  • If all plausible biases would decrease the magnitude of an apparent treatment effect
The quality of the body of evidence is classified in one of four levels:

High quality: We are very confident that the true effect lies close to that of the estimate of the effect

Moderate quality: We are moderately confident in the effect estimate. The true effect is likely to be close to the estimate of the effect, but there is a possibility that it is substantially different

Low quality: Our confidence in the effect estimate is limited. The true effect may be substantially different from the estimate of the effect

Very low quality: We have very little confidence in the effect estimate. The true effect is likely to be substantially different from the estimate of effect

The strength of the recommendations are graded as strong or weak based on:

  • Quality of evidence
  • Uncertainty about the balance between desirable and undesirable effects
  • Uncertainty or variability in values and preferences
  • Uncertainty about whether the intervention represents a wise use of resources
No nutrition-specific items
Suggested Citation:"6 Evidence Review: Judging the Evidence for Causal Relationships." National Academies of Sciences, Engineering, and Medicine. 2017. Guiding Principles for Developing Dietary Reference Intakes Based on Chronic Disease. Washington, DC: The National Academies Press. doi: 10.17226/24828.
×
Nutri-Gradeb Assess the meta-evidence (from RCTs and cohort studies) of an association or effect between different nutrition factors and outcomes, taking into account nutrition research-specific requirements not considered by other tools Seven items for meta-analyses of RCTs and eight items for meta-analyses of cohort studies:
  • Risk of bias, study quality, and study limitations
  • Precision
  • Heterogeneity
  • Directness
  • Publication bias
  • Funding bias
  • Study design (only for meta-analyses of RCTs)
  • Effect size (only for meta-analyses of cohort studies)
  • Dose-response (only for meta-analyses of cohort studies)
Each item is scored from 0-3 points, depending on the item. A total score is calculated: High (≥8 total points): There is high confidence in the effect estimate, and further research probably will not change the confidence in the effect estimate.

Moderate (6-7.99 total points): There is moderate confidence in the effect estimate; further research could add evidence on the confidence and may change the effect estimate.

Low (4-5.99 total points): There is low confidence in the effect estimate; further research will provide important evidence on the confidence and likely change the effect estimate.

Very low (0-3.99 total points): There is very low confidence in the effect estimate; meta-evidence is very limited and uncertain.

Takes into account nutrition research-specific requirements (e.g., dietary assessment methods and their validation or funding bias) for assessing the meta-evidence

Based on GRADE, but has modified the classification for meta-analyses of RCTs and cohort studies compared with GRADE

Nutrition Evidence Library (NEL) Grading Rubricc To judge the strength of the body of evidence for each conclusion statement in the Dietary Guidelines Advisory Committee (DGAC) report

Used by the 2010 and 2015 DGACs

For each study, researchers carefully evaluated five elements:
  • Quality/Risk of bias (based on NEL BAT)
  • Quantity
  • Consistency
  • Impact
  • Generalizability
A grade for the strength of the body of evidence supporting each conclusion statement was determined for each of the five elements:

Grade I: Strong

Grade II: Moderate

Grade III: Limited

Grade IV: Expert opinion only**

Grade V: Grade not assignable

**Only included in the 2010 DGAC

The grading rubric was developed for the 2010 DGAC, and was modified for the 2015 DGAC

No nutrition-specific items

Suggested Citation:"6 Evidence Review: Judging the Evidence for Causal Relationships." National Academies of Sciences, Engineering, and Medicine. 2017. Guiding Principles for Developing Dietary Reference Intakes Based on Chronic Disease. Washington, DC: The National Academies Press. doi: 10.17226/24828.
×
Tool (and modifications) Purpose/Who Is Using It Elements (or Quality Criteria) Grading Method Notes
AHRQ Methods Guide for Evidence-based Practice Centers (EPC)d To grade the strength of the evidence for each outcome in a systematic review Required domains:
  • Study limitations (risk of bias)
  • Consistency
  • Directness
  • Precision
  • Reporting bias

Additional domains:

  • Dose-response association
  • Strength of association (magnitude of effect)
  • Plausible confounding that would decrease observed effect
Summary tables that rate the strength of the evidence

Overall grade for strength of evidence for an outcome is based on domain scores and a holistic summary recognition of the possible complex interaction among domains:

  • High
  • Moderate
  • Low
  • Insufficient

Strength of evidence grades are first done separately for RCTs and observational studies and then combined into one overall grade.

Conceptually similar to the GRADE approach, but tailored for AHRQ’s Evidence Practice Center reviews

No nutrition-specific items

No explicit instruction on “hierarchy” of the evidence, and not a grading for “strength of the recommendations” because AHRQ systematic reviews do not make clinical recommendations.

The clinical recommendation process is a totally independent process thus not covered in ARHQ methods guide.

U.S. Preventive Services Task Force (USPSTF)e USPSTF provides evidence-based recommendations about clinical preventive services and health promotion.
  1. Assess individual studies:
  • Internal validity of individual studies
  • External validity (applicability) of individual studies (subjects, setting, provider)
Each recommendation is assigned a letter grade (A, B, C, or D or an I statement) based on the strength of the evidence and the balance of benefits and harms of a preventive service: USPSTF conducts its evidence reviews with assistance from AHRQ/EPC.
Suggested Citation:"6 Evidence Review: Judging the Evidence for Causal Relationships." National Academies of Sciences, Engineering, and Medicine. 2017. Guiding Principles for Developing Dietary Reference Intakes Based on Chronic Disease. Washington, DC: The National Academies Press. doi: 10.17226/24828.
×

Used by health care professionals and the American public

  1. Assess the level of evidence at the key question level:

  • Quantitative or qualitative synthesis, depending on number of studies and heterogeneity
  1. Overall summary of evidence considers:

  • Key question
  • Number of studies
  • Summary of findings
  • Consistency/precision
  • Reporting bias
  • Overall study quality
  • Body of evidence limitations
  • Applicability
  • Overall strength of evidence
  1. Make a recommendation:

  • Assess the adequacy of evidence at the key question level
  • Assess the adequacy of evidence at the linkage level
  • Estimate the magnitude of benefit and harm of the preventive service
  • Evaluate the certainty of the evidence for net benefit for the preventive service (High, Moderate, Low)
  • Estimate the magnitude of the net benefit of the preventive service
  • Develop a recommendation grade for the preventive service in the relevant population, based on the above parameters

A: The USPSTF recommends the service. There is high certainty that the net benefit is substantial.

B: The USPSTF recommends the service. There is high certainty that the net benefit is moderate or there is moderate certainty that the net benefit is moderate to substantial.

C: The USPSTF recommends selectively offering or providing this service to individual patients based on professional judgment and patient preferences. There is at least moderate certainty that the net benefit is small.

D: The USPSTF recommends against the service. There is moderate or high certainty that the service has no net benefit or that the harms outweigh the benefits.

I statement: The USPSTF concludes that the current evidence is insufficient to assess the balance of benefits and harms of the service. Evidence is lacking, of poor quality, or conflicting, and the balance of benefits and harms cannot be determined.

Suggested Citation:"6 Evidence Review: Judging the Evidence for Causal Relationships." National Academies of Sciences, Engineering, and Medicine. 2017. Guiding Principles for Developing Dietary Reference Intakes Based on Chronic Disease. Washington, DC: The National Academies Press. doi: 10.17226/24828.
×
Tool (and modifications) Purpose/Who Is Using It Elements (or Quality Criteria) Grading Method Notes
National Toxicology Program (NTP) Office of Health Assessment and Translation (OHAT) Handbookf Framework for systematic review and evidence integration for literature-based health assessments of environmental substances Seven-step process
  1. Formulate problem and develop protocol
  2. Search for and select studies for inclusion
  3. Extract data from studies
  4. Assess quality of individual studies using OHAT risk of bias tool (see above in Table 1)
  5. Synthesize evidence and rate confidence in the body of evidence
  6. Translate confidence ratings into level of evidence of health effects
  7. Integrate evidence to develop hazard identification conclusions

Step 5

An initial confidence is determined based on four criteria related to study design:

  1. Controlled exposure
  2. Exposure occurs before outcome
  3. Outcome assessed at individual level
  4. Comparison group used
Level of confidence in the body of evidence:
  • High Confidence (++++) in the association between exposure to the substance and the outcome. The true effect is highly likely to be reflected in the apparent relationship.
  • Moderate Confidence (+++) in the association between exposure to the substance and the outcome. The true effect may be reflected in the apparent relationship.
  • Low Confidence (++) in the association between exposure to the substance and the outcome. The true effect may be different from the apparent relationship.
  • Very Low Confidence (+) in the association between exposure to the substance and the outcome. The true effect is highly likely to be different from the apparent relationship.

In Step 6, determine the conclusions for the level of evidence of health effect (assess human and animal data separately).

    Confidence rating is based primarily on guidance from the GRADE working group

    Confidence rating accommodates the need to integrate data from multiple evidence streams (human, animal, in vitro) and focus on observational human studies.

    Suggested Citation:"6 Evidence Review: Judging the Evidence for Causal Relationships." National Academies of Sciences, Engineering, and Medicine. 2017. Guiding Principles for Developing Dietary Reference Intakes Based on Chronic Disease. Washington, DC: The National Academies Press. doi: 10.17226/24828.
    ×

    Then, the following factors decrease confidence:

    Risk of bias, unexplained inconsistency, imprecision, publication bias

    Or increase confidence:

    Large magnitude of effect, dose response, residual confounding, consistency

    • High level of evidence: There is high confidence in the body of evidence for an association between exposure to the substance and the health outcome(s).
    • Moderate level of evidence: There is moderate confidence in the body of evidence for an association between exposure to the substance and the health outcome(s).
    • Low level of evidence: There is low confidence in the body of evidence for an association between exposure to the substance and the health outcome(s), or no data are available.
    • Evidence of No Health Effect: There is high confidence in the body of evidence that exposure to the substance is not associated with the health outcome(s).
    • Inadequate evidence: There is insufficient evidence available to assess whether the exposure to the substance is associated with the health outcome(s).
    Suggested Citation:"6 Evidence Review: Judging the Evidence for Causal Relationships." National Academies of Sciences, Engineering, and Medicine. 2017. Guiding Principles for Developing Dietary Reference Intakes Based on Chronic Disease. Washington, DC: The National Academies Press. doi: 10.17226/24828.
    ×
    Tool (and modifications) Purpose/Who Is Using It Elements (or Quality Criteria) Grading Method Notes
    World Cancer Research Fund/American Institute for Cancer Research (WCRF/AICR)g Methodology used by the WCRF/AICR to ascertain causal relationships between food, nutrition, physical activity, and cancer
    • Study type
    • Heterogeneity within or between study types or in different populations
    • Random or systematic error (including confounding, measurement error, and selection bias)
    • Plausible biological gradient (dose-response)
    • Strong and plausible experimental evidence (human studies or animal models)
    Criteria for grading evidence:

    Convincing = Evidence is strong enough to support a judgment of a convincing causal relationship, which justifies goals and recommendations designed to reduce the incidence of cancer.

    Probable = Evidence is strong enough to support a judgment of a probable causal relationship, which would generally justify goals and recommendations designed to reduce the incidence of cancer.

    Limited-suggestive = Evidence that is too limited to permit a probable or convincing causal judgment, but where evidence suggests a direction of effect. The evidence may have methodological flaws, or be limited in amount, but shows a generally consistent direction of effect. This almost always does not justify recommendations designed to reduce the incidence of cancer.

    Limited-no conclusion = Evidence is so limited that no firm conclusion can be made.

    Substantial effect on risk unlikely = Evidence is strong enough to support a judgment that a particular food, nutrition, or physical activity exposure is unlikely to have a substantial causal relation to a cancer outcome.

    Judgments were based on an assessment of the evidence available in the scientific literature, with due consideration given to the advantages and disadvantages of each type of study design and to the quality of individual studies. An inclusive approach was taken that recognized the relative strengths and weaknesses of different types of study, but in which no single type of study design is given pre-eminence.
    Suggested Citation:"6 Evidence Review: Judging the Evidence for Causal Relationships." National Academies of Sciences, Engineering, and Medicine. 2017. Guiding Principles for Developing Dietary Reference Intakes Based on Chronic Disease. Washington, DC: The National Academies Press. doi: 10.17226/24828.
    ×

    Special upgrading factors

    • Presence of a plausible biological gradient (dose-response) in the association. Such a gradient need not be linear or even in the same direction across the different levels of exposure, so long as this can be explained plausibly.
    • A particularly large summary effect size (an odds ratio or relative risk of 2.0 or more, depending on the unit of exposure) after appropriate control for confounders.
    • Evidence from RCTs in humans.
    • Evidence from appropriately controlled experiments demonstrating one or more plausible and specific mechanisms actually operating in humans.
    • Robust and reproducible evidence from experimental studies in appropriate animal models showing that typical human exposures can lead to relevant cancer outcomes.

    * Any grading tool for strength of the body of evidence can be applied to grade the body of evidence of any study design that is defined by the inclusion criteria of the systematic reviews. If a systematic review included both observational studies and randomized controlled trials (RCTs), the strength of the body of evidence is rated as such. If a systematic review only included RCTs, the strength of the body of evidence can only rate the body of RCT evidence.

    a See www.gradeworkinggroup.org (accessed July 24, 2017).

    b See http://www.nutrigrade.net (accessed July 24, 2017); Schwingshackl et al., 2016.

    c See http://www.nel.gov/topic.cfm?cat=3368 (accessed July 24, 2017); https://health.gov/dietaryguidelines/2015-scientific-report/05-methodology.asp (accessed July 24, 2017).

    d See http://effectivehealthcare.ahrq.gov/index.cfm/search-for-guides-reviews-and-reports/?pageaction=displayproduct&productid=318 (accessed July 24, 2017); Berkman et al., 2015.

    e See https://www.uspreventiveservicestaskforce.org/Page/Name/methods-and-processes (accessed July 24, 2017).

    f See https://ntp.niehs.nih.gov/pubhealth/hat/noms/index-2.html (accessed July 24, 2017).

    g WCRF/AICR, 2007.

    Suggested Citation:"6 Evidence Review: Judging the Evidence for Causal Relationships." National Academies of Sciences, Engineering, and Medicine. 2017. Guiding Principles for Developing Dietary Reference Intakes Based on Chronic Disease. Washington, DC: The National Academies Press. doi: 10.17226/24828.
    ×

    TABLE ANNEX 6-2-3 Tools for Evaluating the “Quality” (or Reporting Quality*) of Systematic Reviews

    Tool (and Modifications) Purpose/Who Is Using It Checklist Grading Method Notes
    A Measurement Tool to Assess Systematic Reviews (AMSTAR)a To develop and evaluate the methodological quality of systematic reviews

    To guide the conduct of systematic reviews To aid in teaching about systematic reviews

    Used by professional health care associations and other policy institutions

    Used by the 2015 Dietary Guidelines Advisory Committee

    AMSTAR Checklist (Answer yes/no/can’t answer/not applicable):
    1. Was an a priori design provided?
    2. Was there duplicate study selection and data extraction?
    3. Was a comprehensive literature search performed?
    4. Was the status of publication (i.e., grey literature) used as an inclusion criterion?
    5. Was a list of studies (included and excluded) provided?
    6. Were the characteristics of the included studies provided?
    7. Was the scientific quality of the included studies assessed and documented?
    8. Was the scientific quality of the included studies used appropriately in formulating conclusions?
    9. Were the methods used to combine the findings of studies appropriate?
    A well-done systematic review is one that has addressed all the items on the checklist.

    (Sometimes this checklist is quantified by assigning number values to each response.)

    AMSTAR can be used for systematic reviews of non-RCTs, but revisions and extensions are needed.

    Has good inter-rater agreement, test-retest reliability, and face and construct validity.

    Suggested Citation:"6 Evidence Review: Judging the Evidence for Causal Relationships." National Academies of Sciences, Engineering, and Medicine. 2017. Guiding Principles for Developing Dietary Reference Intakes Based on Chronic Disease. Washington, DC: The National Academies Press. doi: 10.17226/24828.
    ×
    1. Was the likelihood of publication bias assessed?
    2. Was the conflict of interest included?
    Revised Assessment of Multiple Systematic Reviews (R-AMSTAR)b To quantify the quality of systematic reviews Added quantification to the AMSTAR checklist For each of the 11 questions in the AMSTAR checklist, a set of criteria is given and a number value is assigned based on how many criteria are met.

    Each question’s score ranges between 1 and 4, and the R-AMSTAR total score has a range of 11 to 44.

    A total score of 11 signifies that none of the AMSTAR criteria was satisfied. A score of 44 shows that all of the criteria of systematic review excellence were verified in every domain.

    Low R-AMSTAR total scores should lead to prudence on the part of the clinician.

    High R-AMSTAR total scores should impart a certain degree of confidence about the clinical relevance and implications of the findings discussed in the high scoring systematic review.

    Not developed by the AMSTAR group
    Suggested Citation:"6 Evidence Review: Judging the Evidence for Causal Relationships." National Academies of Sciences, Engineering, and Medicine. 2017. Guiding Principles for Developing Dietary Reference Intakes Based on Chronic Disease. Washington, DC: The National Academies Press. doi: 10.17226/24828.
    ×
    Tool (and Modifications) Purpose/Who Is Using It Checklist Grading Method Notes
    Risk of Bias in Systematic Reviews Tool (ROBIS)c To assess risk of bias in systematic reviews

    Target audience is guideline developers, authors of overviews of systematic reviews, and review authors who might want to assess or avoid risk of bias in their reviews.

    3-phase tool

    Phase 1: Assess relevance (optional)

    Phase 2: Identify concerns with the review process (study eligibility criteria; identification and selection of studies; data collection and study appraisal; synthesis and findings)

    Phase 3: Judge risk of bias

    Signaling questions are used in phases 2 and 3. Responses are: yes, probably yes, probably no, no, or no information. Yes = low concern.

    Phase 2: Depending on the responses to the signaling questions, the level of concern about bias associated with each domain is judged as “low,” “high,” or “unclear.” If the answers to all signaling questions for a domain are “yes” or “probably yes,” then level of concern can be judged as low. If any signaling question is answered “no” or “probably no,” potential for concern about bias exists.

    Phase 3: Overall bias is judged based on phase 2 analysis and further signaling questions. Results can be summarized in text or depicted in a table or graph.

    Used for assessing risk of bias in systematic reviews of RCTs systematic reviews of non-RCTs

    * Only ROBIS was developed to assess the risk of bias of a systematic review. AMSTAR is primarily a rating of reporting quality and cannot assess the other potential biases in a systematic review. AMSTAR is generally used as a quality assessment tool for complex systematic reviews that want to include existing systematic reviews to determine whether a systematic review has sufficient quality to be included.

    a See https://amstar.ca/index.php (accessed July 24, 2017); Pieper et al., 2014; Shea et al., 2007, 2009.

    b Kung et al., 2010; Pieper et al., 2015.

    c See http://www.bristol.ac.uk/social-community-medicine/projects/robis (accessed July 24, 2017); Whiting et al., 2013, 2016.

    Suggested Citation:"6 Evidence Review: Judging the Evidence for Causal Relationships." National Academies of Sciences, Engineering, and Medicine. 2017. Guiding Principles for Developing Dietary Reference Intakes Based on Chronic Disease. Washington, DC: The National Academies Press. doi: 10.17226/24828.
    ×

    ANNEX 6-2 REFERENCES

    Berkman, N. D., K. N. Lohr, M. T. Ansari, E. M. Balk, R. Kane, M. McDonagh, S. C. Morton, M. Viswanathan, E. B. Bass, M. Butler, G. Gartlehner, L. Hartling, M. McPheeters, L. C. Morgan, J. Reston, P. Sista, E. Whitlock, and S. Chang. 2015. Grading the strength of a body of evidence when assessing health care interventions: An EPC update. J Clin Epidemiol 68(11):1312-1324.

    Higgins, J. P., D. G. Altman, P. C. Gotzsche, P. Juni, D. Moher, A. D. Oxman, J. Savovic, K. F. Schulz, L. Weeks, J. A. Sterne, G. Cochrane Bias Methods, and G. Cochrane Statistical Methods. 2011. The Cochrane Collaboration’s tool for assessing risk of bias in randomised trials. BMJ 343:d5928.

    Kung, J., F. Chiappelli, O. O. Cajulis, R. Avezova, G. Kossan, L. Chew, and C. A. Maida. 2010. From systematic reviews to clinical recommendations for evidence-based health care: Validation of revised assessment of multiple systematic reviews (R-AMSTAR) for grading of clinical relevance. Open Dent J 4:84-91.

    Pieper, D., T. Mathes, and M. Eikermann. 2014. Can AMSTAR also be applied to systematic reviews of non-randomized studies? BMC Res Notes 7:609.

    Pieper, D., R. B. Buechter, L. Li, B. Prediger, and M. Eikermann. 2015. Systematic review found AMSTAR, but not R(evised)-AMSTAR, to have good measurement properties. J Clin Epidemiol 68(5):574-583.

    Schwingshackl, L., S. Knüppel, C. Schwedhelm, G. Hoffmann, B. Missbach, M. Stelmach-Mardas, S. Dietrich, F. Eichelmann, E. Kontopanteils, K. Iqbal, K. Aleksandrova, S. Lorkowski, M. F. Leitzmann, A. Kroke, and H. Boeing. 2016. Perspective: NutriGrade: A scoring system to assess and judge the meta-evidence of randomized controlled trials and cohort studies in nutrition research. Advances in Nutrition: An International Review Journal 7(6):994-1004.

    Shea, B. J., J. M. Grimshaw, G. A. Wells, M. Boers, N. Andersson, C. Hamel, A. C. Porter, P. Tugwell, D. Moher, and L. M. Bouter. 2007. Development of AMSTAR: A measurement tool to assess the methodological quality of systematic reviews. BMC Med Res Methodol 7:10.

    Shea, B. J., C. Hamel, G. A. Wells, L. M. Bouter, E. Kristjansson, J. Grimshaw, D. A. Henry, and M. Boers. 2009. AMSTAR is a reliable and valid measurement tool to assess the methodological quality of systematic reviews. J Clin Epidemiol 62(10):1013-1020.

    Sterne, J. A., M. A. Hernan, B. C. Reeves, J. Savovic, N. D. Berkman, M. Viswanathan, D. Henry, D. G. Altman, M. T. Ansari, I. Boutron, J. R. Carpenter, A. W. Chan, R. Churchill, J. J. Deeks, A. Hrobjartsson, J. Kirkham, P. Juni, Y. K. Loke, T. D. Pigott, C. R. Ramsay, D. Regidor, H. R. Rothstein, L. Sandhu, P. L. Santaguida, H. J. Schunemann, B. Shea, I. Shrier, P. Tugwell, L. Turner, J. C. Valentine, H. Waddington, E. Waters, G. A. Wells, P. F. Whiting, and J. P. Higgins. 2016. ROBINS-I: A tool for assessing risk of bias in non-randomised studies of interventions. BMJ 355:i4919.

    WCRF/AICR (World Cancer Research Fund/American Institute for Cancer Research). 2007. Food, nutrition, physical activity, and the prevention of cancer: A global perspective. Washington, DC: AICR.

    Whiting, P., P. Davies, J. Savovic, D. M. Caldwell, and R. Churchill. 2013. Evidence to inform the development of ROBIS, a new tool to assess the risk of bias in systematic reviews. http://www.bristol.ac.uk/media-library/sites/social-community-medicine/robis/ROBIS%20Report%204_9.pdf (accessed July 16, 2017).

    Whiting, P., J. Savovic, J. P. Higgins, D. M. Caldwell, B. C. Reeves, B. Shea, P. Davies, J. Kleijnen, R. Churchill, and ROBIS group. 2016. ROBIS: A new tool to assess risk of bias in systematic reviews was developed. J Clin Epidemiol 69:225-234

    Suggested Citation:"6 Evidence Review: Judging the Evidence for Causal Relationships." National Academies of Sciences, Engineering, and Medicine. 2017. Guiding Principles for Developing Dietary Reference Intakes Based on Chronic Disease. Washington, DC: The National Academies Press. doi: 10.17226/24828.
    ×

    This page intentionally left blank.

    Suggested Citation:"6 Evidence Review: Judging the Evidence for Causal Relationships." National Academies of Sciences, Engineering, and Medicine. 2017. Guiding Principles for Developing Dietary Reference Intakes Based on Chronic Disease. Washington, DC: The National Academies Press. doi: 10.17226/24828.
    ×
    Page 129
    Suggested Citation:"6 Evidence Review: Judging the Evidence for Causal Relationships." National Academies of Sciences, Engineering, and Medicine. 2017. Guiding Principles for Developing Dietary Reference Intakes Based on Chronic Disease. Washington, DC: The National Academies Press. doi: 10.17226/24828.
    ×
    Page 130
    Suggested Citation:"6 Evidence Review: Judging the Evidence for Causal Relationships." National Academies of Sciences, Engineering, and Medicine. 2017. Guiding Principles for Developing Dietary Reference Intakes Based on Chronic Disease. Washington, DC: The National Academies Press. doi: 10.17226/24828.
    ×
    Page 131
    Suggested Citation:"6 Evidence Review: Judging the Evidence for Causal Relationships." National Academies of Sciences, Engineering, and Medicine. 2017. Guiding Principles for Developing Dietary Reference Intakes Based on Chronic Disease. Washington, DC: The National Academies Press. doi: 10.17226/24828.
    ×
    Page 132
    Suggested Citation:"6 Evidence Review: Judging the Evidence for Causal Relationships." National Academies of Sciences, Engineering, and Medicine. 2017. Guiding Principles for Developing Dietary Reference Intakes Based on Chronic Disease. Washington, DC: The National Academies Press. doi: 10.17226/24828.
    ×
    Page 133
    Suggested Citation:"6 Evidence Review: Judging the Evidence for Causal Relationships." National Academies of Sciences, Engineering, and Medicine. 2017. Guiding Principles for Developing Dietary Reference Intakes Based on Chronic Disease. Washington, DC: The National Academies Press. doi: 10.17226/24828.
    ×
    Page 134
    Suggested Citation:"6 Evidence Review: Judging the Evidence for Causal Relationships." National Academies of Sciences, Engineering, and Medicine. 2017. Guiding Principles for Developing Dietary Reference Intakes Based on Chronic Disease. Washington, DC: The National Academies Press. doi: 10.17226/24828.
    ×
    Page 135
    Suggested Citation:"6 Evidence Review: Judging the Evidence for Causal Relationships." National Academies of Sciences, Engineering, and Medicine. 2017. Guiding Principles for Developing Dietary Reference Intakes Based on Chronic Disease. Washington, DC: The National Academies Press. doi: 10.17226/24828.
    ×
    Page 136
    Suggested Citation:"6 Evidence Review: Judging the Evidence for Causal Relationships." National Academies of Sciences, Engineering, and Medicine. 2017. Guiding Principles for Developing Dietary Reference Intakes Based on Chronic Disease. Washington, DC: The National Academies Press. doi: 10.17226/24828.
    ×
    Page 137
    Suggested Citation:"6 Evidence Review: Judging the Evidence for Causal Relationships." National Academies of Sciences, Engineering, and Medicine. 2017. Guiding Principles for Developing Dietary Reference Intakes Based on Chronic Disease. Washington, DC: The National Academies Press. doi: 10.17226/24828.
    ×
    Page 138
    Suggested Citation:"6 Evidence Review: Judging the Evidence for Causal Relationships." National Academies of Sciences, Engineering, and Medicine. 2017. Guiding Principles for Developing Dietary Reference Intakes Based on Chronic Disease. Washington, DC: The National Academies Press. doi: 10.17226/24828.
    ×
    Page 139
    Suggested Citation:"6 Evidence Review: Judging the Evidence for Causal Relationships." National Academies of Sciences, Engineering, and Medicine. 2017. Guiding Principles for Developing Dietary Reference Intakes Based on Chronic Disease. Washington, DC: The National Academies Press. doi: 10.17226/24828.
    ×
    Page 140
    Suggested Citation:"6 Evidence Review: Judging the Evidence for Causal Relationships." National Academies of Sciences, Engineering, and Medicine. 2017. Guiding Principles for Developing Dietary Reference Intakes Based on Chronic Disease. Washington, DC: The National Academies Press. doi: 10.17226/24828.
    ×
    Page 141
    Suggested Citation:"6 Evidence Review: Judging the Evidence for Causal Relationships." National Academies of Sciences, Engineering, and Medicine. 2017. Guiding Principles for Developing Dietary Reference Intakes Based on Chronic Disease. Washington, DC: The National Academies Press. doi: 10.17226/24828.
    ×
    Page 142
    Suggested Citation:"6 Evidence Review: Judging the Evidence for Causal Relationships." National Academies of Sciences, Engineering, and Medicine. 2017. Guiding Principles for Developing Dietary Reference Intakes Based on Chronic Disease. Washington, DC: The National Academies Press. doi: 10.17226/24828.
    ×
    Page 143
    Suggested Citation:"6 Evidence Review: Judging the Evidence for Causal Relationships." National Academies of Sciences, Engineering, and Medicine. 2017. Guiding Principles for Developing Dietary Reference Intakes Based on Chronic Disease. Washington, DC: The National Academies Press. doi: 10.17226/24828.
    ×
    Page 144
    Suggested Citation:"6 Evidence Review: Judging the Evidence for Causal Relationships." National Academies of Sciences, Engineering, and Medicine. 2017. Guiding Principles for Developing Dietary Reference Intakes Based on Chronic Disease. Washington, DC: The National Academies Press. doi: 10.17226/24828.
    ×
    Page 145
    Suggested Citation:"6 Evidence Review: Judging the Evidence for Causal Relationships." National Academies of Sciences, Engineering, and Medicine. 2017. Guiding Principles for Developing Dietary Reference Intakes Based on Chronic Disease. Washington, DC: The National Academies Press. doi: 10.17226/24828.
    ×
    Page 146
    Suggested Citation:"6 Evidence Review: Judging the Evidence for Causal Relationships." National Academies of Sciences, Engineering, and Medicine. 2017. Guiding Principles for Developing Dietary Reference Intakes Based on Chronic Disease. Washington, DC: The National Academies Press. doi: 10.17226/24828.
    ×
    Page 147
    Suggested Citation:"6 Evidence Review: Judging the Evidence for Causal Relationships." National Academies of Sciences, Engineering, and Medicine. 2017. Guiding Principles for Developing Dietary Reference Intakes Based on Chronic Disease. Washington, DC: The National Academies Press. doi: 10.17226/24828.
    ×
    Page 148
    Suggested Citation:"6 Evidence Review: Judging the Evidence for Causal Relationships." National Academies of Sciences, Engineering, and Medicine. 2017. Guiding Principles for Developing Dietary Reference Intakes Based on Chronic Disease. Washington, DC: The National Academies Press. doi: 10.17226/24828.
    ×
    Page 149
    Suggested Citation:"6 Evidence Review: Judging the Evidence for Causal Relationships." National Academies of Sciences, Engineering, and Medicine. 2017. Guiding Principles for Developing Dietary Reference Intakes Based on Chronic Disease. Washington, DC: The National Academies Press. doi: 10.17226/24828.
    ×
    Page 150
    Suggested Citation:"6 Evidence Review: Judging the Evidence for Causal Relationships." National Academies of Sciences, Engineering, and Medicine. 2017. Guiding Principles for Developing Dietary Reference Intakes Based on Chronic Disease. Washington, DC: The National Academies Press. doi: 10.17226/24828.
    ×
    Page 151
    Suggested Citation:"6 Evidence Review: Judging the Evidence for Causal Relationships." National Academies of Sciences, Engineering, and Medicine. 2017. Guiding Principles for Developing Dietary Reference Intakes Based on Chronic Disease. Washington, DC: The National Academies Press. doi: 10.17226/24828.
    ×
    Page 152
    Suggested Citation:"6 Evidence Review: Judging the Evidence for Causal Relationships." National Academies of Sciences, Engineering, and Medicine. 2017. Guiding Principles for Developing Dietary Reference Intakes Based on Chronic Disease. Washington, DC: The National Academies Press. doi: 10.17226/24828.
    ×
    Page 153
    Suggested Citation:"6 Evidence Review: Judging the Evidence for Causal Relationships." National Academies of Sciences, Engineering, and Medicine. 2017. Guiding Principles for Developing Dietary Reference Intakes Based on Chronic Disease. Washington, DC: The National Academies Press. doi: 10.17226/24828.
    ×
    Page 154
    Suggested Citation:"6 Evidence Review: Judging the Evidence for Causal Relationships." National Academies of Sciences, Engineering, and Medicine. 2017. Guiding Principles for Developing Dietary Reference Intakes Based on Chronic Disease. Washington, DC: The National Academies Press. doi: 10.17226/24828.
    ×
    Page 155
    Suggested Citation:"6 Evidence Review: Judging the Evidence for Causal Relationships." National Academies of Sciences, Engineering, and Medicine. 2017. Guiding Principles for Developing Dietary Reference Intakes Based on Chronic Disease. Washington, DC: The National Academies Press. doi: 10.17226/24828.
    ×
    Page 156
    Suggested Citation:"6 Evidence Review: Judging the Evidence for Causal Relationships." National Academies of Sciences, Engineering, and Medicine. 2017. Guiding Principles for Developing Dietary Reference Intakes Based on Chronic Disease. Washington, DC: The National Academies Press. doi: 10.17226/24828.
    ×
    Page 157
    Suggested Citation:"6 Evidence Review: Judging the Evidence for Causal Relationships." National Academies of Sciences, Engineering, and Medicine. 2017. Guiding Principles for Developing Dietary Reference Intakes Based on Chronic Disease. Washington, DC: The National Academies Press. doi: 10.17226/24828.
    ×
    Page 158
    Suggested Citation:"6 Evidence Review: Judging the Evidence for Causal Relationships." National Academies of Sciences, Engineering, and Medicine. 2017. Guiding Principles for Developing Dietary Reference Intakes Based on Chronic Disease. Washington, DC: The National Academies Press. doi: 10.17226/24828.
    ×
    Page 159
    Suggested Citation:"6 Evidence Review: Judging the Evidence for Causal Relationships." National Academies of Sciences, Engineering, and Medicine. 2017. Guiding Principles for Developing Dietary Reference Intakes Based on Chronic Disease. Washington, DC: The National Academies Press. doi: 10.17226/24828.
    ×
    Page 160
    Suggested Citation:"6 Evidence Review: Judging the Evidence for Causal Relationships." National Academies of Sciences, Engineering, and Medicine. 2017. Guiding Principles for Developing Dietary Reference Intakes Based on Chronic Disease. Washington, DC: The National Academies Press. doi: 10.17226/24828.
    ×
    Page 161
    Suggested Citation:"6 Evidence Review: Judging the Evidence for Causal Relationships." National Academies of Sciences, Engineering, and Medicine. 2017. Guiding Principles for Developing Dietary Reference Intakes Based on Chronic Disease. Washington, DC: The National Academies Press. doi: 10.17226/24828.
    ×
    Page 162
    Suggested Citation:"6 Evidence Review: Judging the Evidence for Causal Relationships." National Academies of Sciences, Engineering, and Medicine. 2017. Guiding Principles for Developing Dietary Reference Intakes Based on Chronic Disease. Washington, DC: The National Academies Press. doi: 10.17226/24828.
    ×
    Page 163
    Suggested Citation:"6 Evidence Review: Judging the Evidence for Causal Relationships." National Academies of Sciences, Engineering, and Medicine. 2017. Guiding Principles for Developing Dietary Reference Intakes Based on Chronic Disease. Washington, DC: The National Academies Press. doi: 10.17226/24828.
    ×
    Page 164
    Suggested Citation:"6 Evidence Review: Judging the Evidence for Causal Relationships." National Academies of Sciences, Engineering, and Medicine. 2017. Guiding Principles for Developing Dietary Reference Intakes Based on Chronic Disease. Washington, DC: The National Academies Press. doi: 10.17226/24828.
    ×
    Page 165
    Suggested Citation:"6 Evidence Review: Judging the Evidence for Causal Relationships." National Academies of Sciences, Engineering, and Medicine. 2017. Guiding Principles for Developing Dietary Reference Intakes Based on Chronic Disease. Washington, DC: The National Academies Press. doi: 10.17226/24828.
    ×
    Page 166
    Suggested Citation:"6 Evidence Review: Judging the Evidence for Causal Relationships." National Academies of Sciences, Engineering, and Medicine. 2017. Guiding Principles for Developing Dietary Reference Intakes Based on Chronic Disease. Washington, DC: The National Academies Press. doi: 10.17226/24828.
    ×
    Page 167
    Suggested Citation:"6 Evidence Review: Judging the Evidence for Causal Relationships." National Academies of Sciences, Engineering, and Medicine. 2017. Guiding Principles for Developing Dietary Reference Intakes Based on Chronic Disease. Washington, DC: The National Academies Press. doi: 10.17226/24828.
    ×
    Page 168
    Suggested Citation:"6 Evidence Review: Judging the Evidence for Causal Relationships." National Academies of Sciences, Engineering, and Medicine. 2017. Guiding Principles for Developing Dietary Reference Intakes Based on Chronic Disease. Washington, DC: The National Academies Press. doi: 10.17226/24828.
    ×
    Page 169
    Suggested Citation:"6 Evidence Review: Judging the Evidence for Causal Relationships." National Academies of Sciences, Engineering, and Medicine. 2017. Guiding Principles for Developing Dietary Reference Intakes Based on Chronic Disease. Washington, DC: The National Academies Press. doi: 10.17226/24828.
    ×
    Page 170
    Suggested Citation:"6 Evidence Review: Judging the Evidence for Causal Relationships." National Academies of Sciences, Engineering, and Medicine. 2017. Guiding Principles for Developing Dietary Reference Intakes Based on Chronic Disease. Washington, DC: The National Academies Press. doi: 10.17226/24828.
    ×
    Page 171
    Suggested Citation:"6 Evidence Review: Judging the Evidence for Causal Relationships." National Academies of Sciences, Engineering, and Medicine. 2017. Guiding Principles for Developing Dietary Reference Intakes Based on Chronic Disease. Washington, DC: The National Academies Press. doi: 10.17226/24828.
    ×
    Page 172
    Suggested Citation:"6 Evidence Review: Judging the Evidence for Causal Relationships." National Academies of Sciences, Engineering, and Medicine. 2017. Guiding Principles for Developing Dietary Reference Intakes Based on Chronic Disease. Washington, DC: The National Academies Press. doi: 10.17226/24828.
    ×
    Page 173
    Suggested Citation:"6 Evidence Review: Judging the Evidence for Causal Relationships." National Academies of Sciences, Engineering, and Medicine. 2017. Guiding Principles for Developing Dietary Reference Intakes Based on Chronic Disease. Washington, DC: The National Academies Press. doi: 10.17226/24828.
    ×
    Page 174
    Suggested Citation:"6 Evidence Review: Judging the Evidence for Causal Relationships." National Academies of Sciences, Engineering, and Medicine. 2017. Guiding Principles for Developing Dietary Reference Intakes Based on Chronic Disease. Washington, DC: The National Academies Press. doi: 10.17226/24828.
    ×
    Page 175
    Suggested Citation:"6 Evidence Review: Judging the Evidence for Causal Relationships." National Academies of Sciences, Engineering, and Medicine. 2017. Guiding Principles for Developing Dietary Reference Intakes Based on Chronic Disease. Washington, DC: The National Academies Press. doi: 10.17226/24828.
    ×
    Page 176
    Suggested Citation:"6 Evidence Review: Judging the Evidence for Causal Relationships." National Academies of Sciences, Engineering, and Medicine. 2017. Guiding Principles for Developing Dietary Reference Intakes Based on Chronic Disease. Washington, DC: The National Academies Press. doi: 10.17226/24828.
    ×
    Page 177
    Suggested Citation:"6 Evidence Review: Judging the Evidence for Causal Relationships." National Academies of Sciences, Engineering, and Medicine. 2017. Guiding Principles for Developing Dietary Reference Intakes Based on Chronic Disease. Washington, DC: The National Academies Press. doi: 10.17226/24828.
    ×
    Page 178
    Suggested Citation:"6 Evidence Review: Judging the Evidence for Causal Relationships." National Academies of Sciences, Engineering, and Medicine. 2017. Guiding Principles for Developing Dietary Reference Intakes Based on Chronic Disease. Washington, DC: The National Academies Press. doi: 10.17226/24828.
    ×
    Page 179
    Suggested Citation:"6 Evidence Review: Judging the Evidence for Causal Relationships." National Academies of Sciences, Engineering, and Medicine. 2017. Guiding Principles for Developing Dietary Reference Intakes Based on Chronic Disease. Washington, DC: The National Academies Press. doi: 10.17226/24828.
    ×
    Page 180
    Suggested Citation:"6 Evidence Review: Judging the Evidence for Causal Relationships." National Academies of Sciences, Engineering, and Medicine. 2017. Guiding Principles for Developing Dietary Reference Intakes Based on Chronic Disease. Washington, DC: The National Academies Press. doi: 10.17226/24828.
    ×
    Page 181
    Suggested Citation:"6 Evidence Review: Judging the Evidence for Causal Relationships." National Academies of Sciences, Engineering, and Medicine. 2017. Guiding Principles for Developing Dietary Reference Intakes Based on Chronic Disease. Washington, DC: The National Academies Press. doi: 10.17226/24828.
    ×
    Page 182
    Suggested Citation:"6 Evidence Review: Judging the Evidence for Causal Relationships." National Academies of Sciences, Engineering, and Medicine. 2017. Guiding Principles for Developing Dietary Reference Intakes Based on Chronic Disease. Washington, DC: The National Academies Press. doi: 10.17226/24828.
    ×
    Page 183
    Suggested Citation:"6 Evidence Review: Judging the Evidence for Causal Relationships." National Academies of Sciences, Engineering, and Medicine. 2017. Guiding Principles for Developing Dietary Reference Intakes Based on Chronic Disease. Washington, DC: The National Academies Press. doi: 10.17226/24828.
    ×
    Page 184
    Suggested Citation:"6 Evidence Review: Judging the Evidence for Causal Relationships." National Academies of Sciences, Engineering, and Medicine. 2017. Guiding Principles for Developing Dietary Reference Intakes Based on Chronic Disease. Washington, DC: The National Academies Press. doi: 10.17226/24828.
    ×
    Page 185
    Suggested Citation:"6 Evidence Review: Judging the Evidence for Causal Relationships." National Academies of Sciences, Engineering, and Medicine. 2017. Guiding Principles for Developing Dietary Reference Intakes Based on Chronic Disease. Washington, DC: The National Academies Press. doi: 10.17226/24828.
    ×
    Page 186
    Suggested Citation:"6 Evidence Review: Judging the Evidence for Causal Relationships." National Academies of Sciences, Engineering, and Medicine. 2017. Guiding Principles for Developing Dietary Reference Intakes Based on Chronic Disease. Washington, DC: The National Academies Press. doi: 10.17226/24828.
    ×
    Page 187
    Suggested Citation:"6 Evidence Review: Judging the Evidence for Causal Relationships." National Academies of Sciences, Engineering, and Medicine. 2017. Guiding Principles for Developing Dietary Reference Intakes Based on Chronic Disease. Washington, DC: The National Academies Press. doi: 10.17226/24828.
    ×
    Page 188
    Suggested Citation:"6 Evidence Review: Judging the Evidence for Causal Relationships." National Academies of Sciences, Engineering, and Medicine. 2017. Guiding Principles for Developing Dietary Reference Intakes Based on Chronic Disease. Washington, DC: The National Academies Press. doi: 10.17226/24828.
    ×
    Page 189
    Suggested Citation:"6 Evidence Review: Judging the Evidence for Causal Relationships." National Academies of Sciences, Engineering, and Medicine. 2017. Guiding Principles for Developing Dietary Reference Intakes Based on Chronic Disease. Washington, DC: The National Academies Press. doi: 10.17226/24828.
    ×
    Page 190
    Suggested Citation:"6 Evidence Review: Judging the Evidence for Causal Relationships." National Academies of Sciences, Engineering, and Medicine. 2017. Guiding Principles for Developing Dietary Reference Intakes Based on Chronic Disease. Washington, DC: The National Academies Press. doi: 10.17226/24828.
    ×
    Page 191
    Suggested Citation:"6 Evidence Review: Judging the Evidence for Causal Relationships." National Academies of Sciences, Engineering, and Medicine. 2017. Guiding Principles for Developing Dietary Reference Intakes Based on Chronic Disease. Washington, DC: The National Academies Press. doi: 10.17226/24828.
    ×
    Page 192
    Suggested Citation:"6 Evidence Review: Judging the Evidence for Causal Relationships." National Academies of Sciences, Engineering, and Medicine. 2017. Guiding Principles for Developing Dietary Reference Intakes Based on Chronic Disease. Washington, DC: The National Academies Press. doi: 10.17226/24828.
    ×
    Page 193
    Suggested Citation:"6 Evidence Review: Judging the Evidence for Causal Relationships." National Academies of Sciences, Engineering, and Medicine. 2017. Guiding Principles for Developing Dietary Reference Intakes Based on Chronic Disease. Washington, DC: The National Academies Press. doi: 10.17226/24828.
    ×
    Page 194
    Next: 7 Intake-Response Relationships and Dietary Reference Intakes for Chronic Disease »
    Guiding Principles for Developing Dietary Reference Intakes Based on Chronic Disease Get This Book
    ×
    Buy Paperback | $65.00 Buy Ebook | $54.99
    MyNAP members save 10% online.
    Login or Register to save!
    Download Free PDF

    Since 1938 and 1941, nutrient intake recommendations have been issued to the public in Canada and the United States, respectively. Currently defined as the Dietary Reference Intakes (DRIs), these values are a set of standards established by consensus committees under the National Academies of Sciences, Engineering, and Medicine and used for planning and assessing diets of apparently healthy individuals and groups.

    In 2015, a multidisciplinary working group sponsored by the Canadian and U.S. government DRI steering committees convened to identify key scientific challenges encountered in the use of chronic disease endpoints to establish DRI values. Their report, Options for Basing Dietary Reference Intakes (DRIs) on Chronic Disease: Report from a Joint US-/Canadian-Sponsored Working Group, outlined and proposed ways to address conceptual and methodological challenges related to the work of future DRI Committees. This report assesses the options presented in the previous report and determines guiding principles for including chronic disease endpoints for food substances that will be used by future National Academies committees in establishing DRIs.

    1. ×

      Welcome to OpenBook!

      You're looking at OpenBook, NAP.edu's online reading room since 1999. Based on feedback from you, our users, we've made some improvements that make it easier than ever to read thousands of publications on our website.

      Do you want to take a quick tour of the OpenBook's features?

      No Thanks Take a Tour »
    2. ×

      Show this book's table of contents, where you can jump to any chapter by name.

      « Back Next »
    3. ×

      ...or use these buttons to go back to the previous chapter or skip to the next one.

      « Back Next »
    4. ×

      Jump up to the previous page or down to the next one. Also, you can type in a page number and press Enter to go directly to that page in the book.

      « Back Next »
    5. ×

      Switch between the Original Pages, where you can read the report as it appeared in print, and Text Pages for the web version, where you can highlight and search the text.

      « Back Next »
    6. ×

      To search the entire text of this book, type in your search term here and press Enter.

      « Back Next »
    7. ×

      Share a link to this book page on your preferred social network or via email.

      « Back Next »
    8. ×

      View our suggested citation for this chapter.

      « Back Next »
    9. ×

      Ready to take your reading offline? Click here to buy this book in print or download it as a free PDF, if available.

      « Back Next »
    Stay Connected!