6

Evidence Review: Judging the Evidence for Causal Relationships

Previous chapters illustrate the current process and framework (see Chapter 1, Figure 1-2) as well as the relevant activities to date (see Chapter 2) for developing Dietary Reference Intakes (DRIs) based on indicators of adequacy, toxicity, or chronic disease. Although the current DRI process considers all three indicators, this committee has been asked to provide recommendations and guiding principles for developing chronic disease DRIs, specifically. As mentioned in Chapter 1, a key activity relevant to chronic disease endpoints within the purview of DRI committees is evaluating the certainty of the evidence with regard to two main questions: (1) What are acceptable levels of confidence that the relationship between an NOFS and a chronic disease is causal? and (2) If a causal relationship exists, what are acceptable levels of confidence in the intake-response relationship data, what are approaches for identifying and characterizing such relationship, and if appropriate, to recommend chronic disease DRIs? This chapter discusses the methodological tools that can be used to answer the first question (does a causal relationship exist?). The chapter starts with an overview of the conduct of a systematic review, as a modern tool to answer scientific questions and to provide context for the reminder of the chapter. It also provides an overview of the systems and tools that are often used to evaluate scientific evidence. Finally, it discusses in more depth the strengths and cautions in applying one of those systems, Grading of Recommendations, Assessment, Development and Evaluation (GRADE), in determining causal associations between nutrients or other food substances (NOFSs) and chronic diseases for DRI development. Resources for more detailed application of the procedures are provided. At the end of the chapter, the

guiding principles as foundations for a scientifically credible chronic disease DRI process are listed. A list of key terms and their definitions is in Box 6-1.

Throughout the report and particularly within this chapter, the committee makes a clear distinction between the task of the systematic review team, which conducts the systematic review, and the task of the DRI committee, which reviews the totality of evidence and recommends chronic

disease DRIs, if appropriate. A systematic review is a scientific investigation that focuses on a specific question and that uses explicit, planned scientific methods to identify, select, assess, and summarize the findings of similar but separate studies. It may or may not include a quantitative synthesis (meta-analysis) of the results from separate studies (IOM, 2011b). A systematic review is typically conducted by a group of experts in the process itself

that includes subject matter experts (e.g., a systematic review team) and in consultation with external subject matter experts (e.g., a technical expert panel). In contrast, the DRI committee’s review consists of drawing conclusions about the totality of the findings. The guidelines and recommendations in this chapter are envisioned to be followed in the context of the current process, where DRI committees do not conduct systematic reviews themselves but are the recipients of such reviews. Based on the systematic reviews, the DRI committee is tasked with the review of the totality of the evidence, including evidence about quantitative intake-response relationships described in Chapter 7, and, if appropriate, with recommending a DRI. It is further assumed that the initial and any subsequent systematic reviews and protocols are conducted independently of the DRI committee. The first use of this type of systematic review and evidence review process for DRI purposes was reflected in the development of the 2011 update of the DRIs for calcium and vitamin D, which included consideration of chronic disease endpoints (Brannon et al., 2014; IOM, 2011a) (see the vitamin D case example in Chapter 3).

This chapter particularly considers the nutrition-specific issues outlined in Chapter 3. It draws on other sections of the report that discuss methods to measure dietary intake (see Chapter 4), and disease outcomes (see Chapter 5) and leads up to the discussion, in Chapter 7, of approaches to specifying intake-response relationships when causation has been established with sufficient certainty (i.e., the second main question). In addition, although it is not the task of this report to provide direction in moving to guidelines and policies, in various places the committee offers comments on how the conduct of systematic reviews can and should anticipate the guideline decisions that follow them as best as possible, in order to facilitate that process.

OVERVIEW OF THE SYSTEMATIC REVIEW PROCESS

In the context of setting DRIs, the systematic review process is an essential activity before the totality of the evidence is reviewed by the DRI committee. Figure 6-1 is an overview of an ideal systematic review as conceptualized in the IOM report Finding What Works in Health Care: Standards for Systematic Reviews (IOM, 2011b). Depending on the context for a systematic review and the group that conducts it, the specific nature of the activities may vary. This section includes a description of some of the activities highlighted as especially important in the context of developing DRIs. These include formulating the questions, registering the systematic reviews and adopting an appropriate format for reporting methods and results, and anticipating additional information needs beyond the systematic review results.

Problem Formulation, Scoping and Developing a Systematic Review Protocol

The initial selection of particular NOFSs and health outcomes is within the purview of the federal agencies (see Figure 1-2), which prioritize NOFSs and health outcomes for study based on public health priorities and other factors. Whatever the choices, each one implies some tentative perspective that a scientific basis and a literature exist to proceed. Therefore, once the general scientific questions that form the basis for the systematic evidence review have been formulated, an important next preliminary step is to sample the available literature, to determine that it is sufficiently robust to make the review useful for each of the central scientific questions identified. Often, a preliminary literature review may be available, including previously conducted systematic reviews and meta-analyses. If not, a preliminary “scoping” review should be conducted (see, e.g., Brannon et al., 2016). If the studies are clearly insufficient, then setting a DRI is not scientifically justifiable. In addition to assessing the literature’s breadth, depth, and relevance, the scoping exercise could have other potential dividends, such as determining the need for translation services and identifying additional keywords to aid in searching. This step is different from conducting the comprehensive literature search. Scoping may also identify systematic reviews, including meta-analyses already published on nutrition-disease topics. These previously published systematic reviews can help target relevant literature and also can be incorporated as part of the evidence review. Instruments are available to assess systematic review quality (e.g., AMSTAR [Pieper et al., 2014; Shea et al., 2007, 2009] and risk of bias [ROBIS] [Whiting et al., 2013, 2016]). It may be possible to add recent research reports to existing systematic reviews for reasons of efficiency if the search is comprehensive up to a particular date and all other quality criteria are met (Chung et al., 2012; Garner et al., 2016; Shekelle et al., 2009).

Any comprehensive systematic review can have multiple scientific questions (e.g., multiple dimensions of efficacy and safety of an NOFS-focused intervention, including nutrient-nutrient interactions, varied effects in subpopulations defined by age or other characteristics, and effects on multiple disease outcomes or surrogate markers). Each question should be specific and clearly defined, and narrow enough to be aligned with the evidence base under review. Framing questions as specific scientific hypotheses is essential.

The structure for questions to guide systematic and evidence reviews involves detailed and explicit characterization of the study level and design details, including target population, candidate interventions, and outcomes important to individuals. The PICO (population, intervention, comparator,

and outcome) structure¹ is considered helpful in bringing focus and order to the questions at hand (Guyatt et al., 2011b). In the context of chronic disease DRIs, key questions would be formulated, and the parameters of interest to the sponsoring agency or agencies in consultation with a technical expert panel would be specified using the PICO framework. The vitamin D case example in Chapter 3 demonstrates how a map of the various biological pathways of interest can be created as an analytic framework to guide the process of framing the key questions (see Figure 3-1). The intervention and comparator in these questions are implicit, i.e., effects of interest are not specified with respect to a given level of vitamin D intake. Data abstraction methods that identify the P, I, C, and O elements for each study are included in the systematic review protocol. As noted in Chapter 3, because everyone in the population consumes NOFSs at some level, for the purpose of establishing a causal relationship, some questions may be framed as an increase or decrease relative to existing consumption patterns rather than above or below a certain threshold.

As another example, formulated questions for systematic reviews to support the World Health Organization (WHO) guideline on free sugars² were as follows: “What is the effect of a decrease or increase in free sugars in adults and children?” and “What is the effect of restricting intake of free sugars to below 10 percent of total energy intake?”—accompanied by a definition of the term “free sugars” (WHO, 2015). Annex 6-1 shows the full detail of the free sugar questions in a PICO table format that specifies population considerations, exposure definitions, comparisons, confounders, effect modifiers and intermediates, outcomes, and time frames (WHO, 2015). Evidence on these questions about sugar was gathered through several systematic reviews (peer reviewed and published) in order to inform the eventual guideline. The questions in Box 6-2, also drawn from systematic reviews to support WHO guideline development, illustrate more complex NOFS-chronic disease questions using the general PICO approach (Brouwer, 2016; Mensink, 2016). As with the free sugar questions, these questions frame “interventions” about intake level in terms of increases or decreases, sometimes related to specific thresholds in association with particular outcomes—in this case blood lipids as surrogates for noncommunicable disease risk. Other questions address effects of different types

___________________

¹ Some groups use PICOTS: population, intervention, comparators, outcomes, timing, and setting.

² Free sugars is defined by the United Nations World Health Organization and the Food and Agriculture Organization as “all monosaccharides and disaccharides added to foods by the manufacturer, cook, or consumer, plus sugars naturally present in honey, syrups, and fruit juices” (WHO, 2015, p. 1). In the United States, the term “free sugars” is generally equivalent to “added sugars,” the term used by the 2015-2020 Dietary Guidelines for Americans.

of saturated or trans fat and substitution or replacement effects that are relevant to food manufacturing.

Systematic Review Registration and Report Format

Once the systematic review team is assembled and as the systematic review process begins, the committee strongly suggests registering the review in advance with the international prospective register of systematic reviews (PROSPERO).³ This registration applies to the initial systematic review and any subsequent reviews needed. Registering the review in advance with the PROSPERO system is mandatory for the Agency for Healthcare Research and Quality (AHRQ) Evidence-based Practice Center (EPC) program, which currently leads the systematic reviews for setting DRIs and has value in several ways, such as avoiding duplication of effort and promoting transparency. In planning the report, the team should follow the Preferred Reporting Items for Systematic Reviews and Meta-Analyses for Protocols (PRISMA-P) approach,⁴ which specifies structure and completeness for reporting systematic reviews and meta-analyses.

Anticipating Additional Literature Needs When Planning Systematic Reviews

When planning a systematic review, anticipating downstream needs for literature searches and guideline creation can be important, particularly if the systematic review cannot be repeated or re-opened after completion (e.g., due to financial or other constraints). Examples of such activities

___________________

³ See https://www.crd.york.ac.uk/PROSPERO.

⁴ See http://www.prisma-statement.org.

include collecting literature on validation of potential surrogate markers or biomarkers of intake, and the general health characteristics or diets of the target populations of interest, identifying disease risks in those groups of special interest, searching for published computational or other public health decision models that could aid in recommending the DRI, identifying studies of adverse effects of selected NOFS interventions, and cataloging relevant literature on population interventions. These tasks will be accomplished by the systematic review team as it develops its search strategy.

The committee also notes that mechanistic studies, whether in humans, animals, or various in vitro systems, may be important to generating hypotheses regarding the ability of an NOFS to affect the risk of developing a chronic disease. The World Cancer Research Fund Expert Panel on Diet, Nutrition, and Cancer Prevention, for example, used a hierarchy of robustness based on categorization of human and animal experimental findings into three classes of which only one class was thought to be useful for making judgments about diet and cancer risk (WCRF/AICR, 2007). Although it is not generally practical to include mechanistic studies in systematic reviews of the effect of NOFSs on health, mechanistic studies can contribute to addressing the study question by providing additional information. Thus, it would be important to anticipate mechanistic research questions at the start of the systematic review process. Box 6-3 lists some of the important information that can be gathered from mechanistic studies.

Forming the Systematic Review Team, Technical Expert Panel, and DRI Committee

Developing chronic disease DRIs depends on the contributions of many stakeholders, including the federal agencies, groups of experts, and the public. Within the current process, three groups of experts—the systematic review team, the technical expert panel, and the DRI committee—are involved in different capacities and time frames. Briefly, the systematic review team conducts the systematic review in consultation with the technical expert panel, and the DRI committee reviews the totality of the evidence and recommends chronic disease DRIs, if appropriate. Formation of these three groups necessitates a formal process. Although no dictates on group structure and the disciplines represented in any of the three groups have been established, experience suggests that several types of expertise are of great value. While the systematic review team includes mainly methodologists, the technical expert panel would include relevant content experts in nutrition science, toxicology, scientific study design and analysis, public health, biostatistics, nutrition epidemiology and chronic disease epidemiology, and disease pathogenesis. In terms of the DRI committee, disciplines include content experts for the main themes of the review, methodologists

experienced with the evidence review approach, biostatisticians, and persons with experience in guideline construction and policy applications. The size of the group should be consonant with the breadth of review topics, and available resources.

To maintain objectivity, scientific rigor, and acceptability of a systematic review, it is important to recognize and manage potential bias and conflict of interests of the individuals conducting or providing input into the systematic review (IOM, 2011b). AHRQ, which is the agency that in the current DRI process has been responsible for administering the systematic review for DRI development, has policies to reduce the risk of bias from financial and non-financial interests. This requirement was also highlighted in a recent report from the National Academies of Sciences, Engineering, and Medicine (the National Academies) in the context of advisory panels (NASEM, 2017). Specifically, the report made recommendations on the selection process, including disclosure of how conflict of interest and biases are identified and managed, of members of the Dietary Guidelines Advisory

Committee.⁵ The National Academies, which has convened all DRI committees in the past, has policies for assessing potential conflicts of interest among committee members.⁶ To promote transparency and enhance public and stakeholder assurances of committees, the National Academies publishes the biographies of individual members for public review and comment.

Regardless of the subject matter and the disciplines required, it is important to establish a mechanism for communication among the three groups that will ensure a moderate level of communication while also protecting against inappropriate influence on the systematic review methods.

SYSTEMS OF EVIDENCE SYNTHESIS AND THEIR APPLICATIONS

The committee concluded that addressing diet-chronic disease relationships requires a credible, established, structured system for systematic review and review of the totality of the evidence, for all important topics that may be translated into population policy, and that this applies where “diet” refers to specific NOFSs, as is the focus of this DRI-oriented report, as well as when applied to foods or entire dietary patterns. The guiding principles discussed throughout the chapter reflect the fact that systematic reviews and evidence reviews in general are well-developed scientific activities for informing policy and practice. The several different systematic review and evidence review systems that have been applied to nutrition have many features in common. Requirements for such systems include:

• A record of robust experience and precedent to approach important scientific issues, including those involving questions about health effects of NOFS consumption,
• Well-structured protocols that can accommodate multiple major and ancillary scientific issues,
• Sufficient clarity and transparency to be useful for evidence review and policymaking groups,
• Sufficient detail to be replicable and to withstand expert peer review,
• The ability to cogently address causal pathways, which can potentially be used to develop policy recommendations, and
• A core set of methods stable enough to be applied over time but sufficiently flexible to be sensitive to the important details for different content areas. The methods should improve and evolve with thoughtful input from content experts and the user communities

___________________

⁵ The Dietary Guidelines Advisory Committee independently evaluates the scientific evidence and makes recommendations about how the Dietary Guidelines for Americans could be revised.

⁶ See www.nationalacademies.org/coi.

The committee acknowledges that no system will meet all needs and contingencies equally, and that all systems have strengths and weaknesses. The committee identified and reviewed key features of a variety of available evidence review and systematic review systems and instruments, contained in Annex 6-2, in sufficient detail to confirm that they have much in common conceptually although they vary in intent and comprehensiveness. The Annex includes systems or instruments intended only for assessing specific elements of primary empirical research reports, such as “risk-of-bias” tools, or instruments to evaluate the quality of reporting and completed systematic reviews. As others have concluded (Dwyer et al., 2016), all of these systems and instruments were deemed to have important value and apply similar principles to the review of the evidence. Some instruments for reviewing evidence are intended only for RCTs while others apply to both trials and observational studies. Some systems are used for a variety of issues related to primary and secondary prevention issues, including nutrition-related issues (AHRQ, 2014; USPSTF, 2015). Another system is used for assessing toxic environmental exposures (NTP, 2015) and others have been adapted for specific nutrition-related uses (NEL, 2015; WCRF/AICR, 2007). Still other systems are designed to evaluate the quality of reporting of completed systematic reviews and meta-analyses (e.g., AMSTAR [Pieper et al., 2014; Shea et al., 2007, 2009]).

THE “GRADE” (GRADING OF RECOMMENDATIONS, ASSESSMENT, DEVELOPMENT AND EVALUATION) SYSTEM: DEFINITION AND JUSTIFICATION

After reviewing the systems in Annex 6-2 and based on the knowledge and experience of committee members about what would meet the needs of DRI committees and users of DRI reports, the committee adopted the GRADE system as the reference point for the evidence reviews relating to NOFS-chronic disease considerations.

The GRADE system is an approach for rating certainty in the evidence in systematic reviews and guidelines. The system is designed for examining alternative management strategies or interventions, which may include no intervention or current best management. Although originally developed in the context of health care (diagnosis, screening, prevention, and therapy), it can be (and has been) applied to public health questions. In the particular case of evaluating the associations of NOFSs with chronic disease outcomes, GRADE will be used to assess the certainty in the association between an NOFS and benefit and harm, which implies causality. In addi-

tion to rating the certainty of a body of evidence, the GRADE approach includes tools for grading the strength of recommendations (see also Chapter 7). The rationale for the selection of GRADE is presented below.

Meets Criteria for an Appropriate Evidence Review Tool

The GRADE system meets the requirements for an established system of guidance for evidence synthesis. It has been used extensively in many scientific domains, and comprehensive documentation and precedent exist to address many of the issues relevant to NOFS-chronic disease questions. A particular advantage of the GRADE system is that it provides a comprehensive and clearly structured taxonomy for rating the certainty of evidence and extensive, specific guidance for making the ratings. In addition, GRADE working subgroups continue to address specific methodological or subject matter issues as needed. Publications for using GRADE include a six-part series of papers in the British Medical Journal (Guyatt et al., 2008a,b,c,d; Jaeschke et al., 2008; Schunemann et al., 2008), a large series of papers in the Journal of Clinical Epidemiology (Andrews et al., 2013a,b; Balshem et al., 2011; Guyatt et al., 2011a,b,c,d,e,f,g,h), and two additional papers focusing on the framework for translating evidence to guideline decisions (Alsonso-Coello et al., 2016a,b). The online GRADE Handbook (Schunemann et al., 2013) summarizes the guidance in these articles.

As long as the fundamental tenets are followed and a clear rationale for specific decisions stated, GRADE can accommodate particular perspectives. Indeed, the merit of this system is not that all members of a group using it will ultimately agree, but that it provides a structured, explicit, and transparent system for making decisions. Therefore, the GRADE working group strongly discourages modifications to the approach itself and recommends adherence to the currently available methods and applications. However, the GRADE system is neither rigid nor static. GRADE continues to evolve, and the GRADE working group is open to participation and collaboration from those facing challenges in applying the system, such as for informing nutritional guidelines in the context of the many unique conceptual and methodological issues outlined in Chapter 3.

Is in Wide Use and Applicable to Nutrition Policy Questions

The GRADE system provides an approach to rating the certainty of evidence for the outcomes relevant to optimizing health, either in terms of personal habits and lifestyle, or in decisions about health care interventions. It has been adopted by more than 100 organizations including WHO (2014), the Cochrane Collaboration, the American College of Physicians, the National Institute for Health and Care Excellence (NICE), and

UpToDate^®), increasing its use and familiarity. GRADE has been applied in more than 100 countries. This facilitates the ability to compare reviews developed by diverse users. GRADE has been applied to nutrition-related systematic reviews (Garcia-Larsen et al., 2016; Mayhew et al., 2016; Naude et al., 2014; Pennant et al., 2015; Santesso et al., 2012) and is the method generally used by WHO to support guideline development, including for nutrition topics.

Has Criteria for Assessing Strength of Recommendations

The GRADE working group has seriously considered the issues of policy guidelines creation after evidence review, though the latter is not the task of this report. GRADE separates questions about confidence in estimates of effect in studies from questions about the strength of recommendations. This is critical for preserving the integrity of the judgments made about causal associations. Strong recommendations are far more likely when evidence is of high or moderate certainty and far less likely when evidence is of low or very low certainty. Nevertheless, when desirable and undesirable consequences and outcomes are closely balanced, it is possible to have a weak recommendation in the context of high certainty of the evidence. Similarly, GRADE has identified five paradigmatic situations in which strong recommendations are warranted in the face of low certainty evidence (Andrews et al., 2013b).

APPLYING THE GRADE APPROACH FOR SYNTHESIZING EVIDENCE ABOUT THE RELATIONSHIP BETWEEN AN NOFS AND CHRONIC DISEASE

This section is not intended to provide specific methods or protocols, which are published in peer-reviewed journals and the online handbook (Schunemann et al., 2013). The section describes how the GRADE system (see Figure 6-2 for an overview) is applied to evidence review in general, with particular suggestions and examples for application to the development of NOFS-chronic disease DRIs. In particular, this section addresses four critical and challenging issues in conducting an evidence review and determining the certainty of the evidence using the GRADE system: (1) selecting and ranking the outcomes, (2) addressing multiple comparators, (3) developing evidence summaries, and (4) determining the certainty of the evidence for each outcome. Published information that provides additional detail is highlighted.

It is envisioned that future DRI committees will review the evidence synthesis provided to them by applying the GRADE approach. Through that process, the committees will make judgments about causality and,

where indicated, intake-response. Thus, although some of the activities shown in Figure 6-2 and described above under the “overview of systematic review” are not conducted by DRI committees, they are essential to ensuring that DRIs are established with the best possible evidence. For example, as Figure 6-2 illustrates and as described earlier in this chapter, in the GRADE approach, a systematic review of the relevant scientific literature by the systematic review team precedes the work of the DRI committees to evaluate the totality of the evidence. The following discussion assumes that the systematic review and subsequent evidence reviews by the DRI committee are largely independent and that these processes precede and are separate from the development of guidelines, as described in Chapter 1 and above.⁷

Selecting and Ranking Health and Disease Outcomes

GRADE emphasizes the importance of identifying all outcomes that are meaningful to the population to whom a recommendation or guideline will be applied. For example, a particular NOFS may have causal relationships with several disease outcomes, beneficial or harmful. For risk assessment purposes, effects on physical and mental health are of primary importance. Effects on individual well-being (quality of life) may be considered during the risk management process, but these general effects are more relevant to consuming foods or following dietary patterns than to the intake of specific NOFSs.

Selecting the chronic disease outcomes to be considered as the basis for a DRI is a two-step process. First, all important chronic disease outcomes are identified during the formulation of the PICO questions. For the WHO free sugars guideline (WHO, 2015), several indicators were identified for body weight or body adiposity outcomes, and the outcome of dental caries was specifically defined as not including dental erosion (see Annex 6-1 for free sugar guidelines PICO specifications). In this step, designating and recording other secondary outcomes from relevant scientific reports is likely to be valuable, even if not originally part of the review’s goal. Defining longer-term health outcomes from a nutritional intervention will be central for developing chronic disease DRIs. If available, general health outcomes, such as all-cause mortality or disability rates, may inform summative population effects of the NOFS under consideration, both positive and adverse. The inclusion of such broad outcomes is of particular importance for NOFS

___________________

⁷ In other contexts, these activities may be performed as a continuum in which the same committee acts as or liaises to the technical advisory group before the conduct of systematic reviews, receives the systematic review, proceeds with evidence review, and then develops policy recommendations.

questions because so many different outcomes may be affected, directly or indirectly. However, because of the many other NOFSs and other factors (e.g., other behaviors, medication use) that may be in the same pathway, attributing total mortality to a specific NOFS is challenging. The systematic review protocols also may include relevant surrogate markers (see Chapter 5 and the discussion of indirectness in this chapter). As mentioned in Chapter 5, conceptually, surrogate markers have no importance in themselves unless they are evaluated as “qualified” to serve as surrogates for the important policy relevant outcome. The protocol also should specify the time frame to which the outcomes apply (e.g., the interval between intervention and expected effects, such as 1 year, 5 years, or a lifetime).

Dealing with Multiple Comparators

The structure of the PICO questions, i.e., specifying one comparator for a given intervention, requires modification when more than two alternative courses of action exist. Under these circumstances, all potential courses of action can be characterized as alternative interventions in a series of questions in which the intervention is compared with each relevant alternative, or as a single intervention with multiple comparators by group (possible alternatives within a single question). For example, one could ask in one question “what is the effect of replacing saturated fat with monounsaturated fatty acids or various types of polyunsaturated fatty acids on cardiovascular disease outcomes” but extract data to support examination of each type of replacement separately. However, when multiple alternatives exist, it is unlikely that the primary literature will have compared each alternative to every other alternative. In this case, inferences regarding the relative merits of the alternatives require indirect comparisons. A novel statistical methodology for simultaneously considering all direct and indirect comparisons, network meta-analysis, is now available for dealing with such situations. GRADE has developed guidance for rating certainty of evidence in the context of network meta-analysis (Puhan et al., 2014).

Developing Evidence Summaries

Identifying, collecting, and evaluating published scientific studies (i.e., the evidence) is described in detail in the GRADE handbook (Schunemann et al., 2013). Once the studies have been identified, and the evidence evaluated, GRADE suggests aggregating the findings into two types of evidence summary tables: evidence profiles and summary-of-findings tables (Guyatt et al., 2011a). Although this task would be conducted by the systematic review team, not the DRI committee, it is an essential task that allows the committee to assess the totality of the evidence. The DRI committee would

review the evidence tables in order to make their own decisions regarding the certainty of the evidence for each outcome. Any discrepancies should be clearly documented and justified. As mentioned above, sufficient interaction between the DRI committee and the systematic review team and technical expert panel will ensure a more efficient process because any differences in judgments would have been discussed before the DRI committee receiving the final systematic review.

For binary or other categorical outcomes, evidence profiles include, at a minimum, a list of all outcomes, the numbers of participants and studies addressing these outcomes, the fundamental study designs (RCTs or observational studies), and measures of both the relative and absolute effect of the intervention compared to control. For continuous outcomes, evidence profiles differ only in the presentation of the effects. Possible presentation of continuous outcomes includes natural units and a number of transformations—standardized mean differences, minimal important difference units, and ratio of means, and binary outcomes—which will then be presented as relative and absolute effects.

RCTs start with a high certainty of evidence. Following the GRADE approach (see below under Factors Determining the Certainty of Evidence), the systematic review team will specify judgments (i.e., not serious, serious, or very serious) about specific factors (i.e., risk of bias, imprecision, inconsistency, indirectness, and publication bias) that may warrant rating down the certainty of evidence about specific outcomes from high to moderate (or low) and influence the judgment about the overall certainty of evidence. These factors, which also apply to observational studies, are discussed in the following section.

Ideally, for observational studies, the evidence profiles from the systematic review team will document each reason for rating down, and may also document factors for rating up, such as the presence of large effect sizes, dose-response gradients, and coverage of all plausible biases (see the following section for more detail). These can reinforce rather than undermine inferences regarding the effects of NOFS interventions. For each outcome and factor, the specific judgment would vary. Furthermore, the overall certainty rating would vary for each outcome.

Summary-of-findings tables are identical to evidence profiles, with one exception: they do not document judgments for each factor (rating up or down) but only the final overall certainty rating and specification of the domains that were rated down or up.

In practice, full documentation of the evidence reviewed can require multiple tables in order to characterize all important details, usually published as supplemental material in conjunction with the systematic review itself. For example, the systematic review of prospective studies of associations of saturated fat and trans fat with noncommunicable diseases as part

of a WHO guideline development process (de Souza et al., 2015) required six appendixes in order to provide details of the search strategy, supplemental tables to describe characteristics of studies reviewed and included in the GRADE summary (20 tables), characteristics of studies reviewed but not included in the GRADE summary, supplemental figures (68 figures), and two final GRADE summary tables, one each for saturated fat and trans fat and including all outcomes.

Figure 6-3 lists the 20 summary tables from the de Souza et al. review (de Souza et al., 2015). It shows that tables may address methodological issues such as the accuracy of measurements (eTable 1) and reported dietary intakes (eTable 20) in addition to findings grouped by study design or analytic approach, exposure, and outcomes. These tables are typically documented with footnotes to clarify definitions, analytic issues, and reasons for rating up or down, as appropriate (Santesso et al., 2016). The GRADE summary tables for saturated fat and trans fat were annotated with 40 and 86 footnotes, respectively. The format for GRADE table entries is shown in Figure 6-4, using the example for trans fat.

Once the systematic review team submits to the DRI committee its systematic review with evidence profiles and summary-of-findings tables for all outcomes of interest, positive and adverse, the committee evaluates the evidence presented, notes and justifies agreements and disagreements, and evaluates the importance of the health outcomes. The committee first makes a decision on the certainty of the evidence with regard to a causal relationship for each outcome they view as important. If such a relationship exists, then the DRI committee characterizes the certainty of the intake-response relationships and considers benefits and harms and their relative importance, as explained in Chapter 7.

Factors Determining the Certainty of Evidence

Initial Rating Based on Study Design: RCTs Versus Observational Studies

GRADE initially rates the certainty of the evidence as high, moderate, low, or very low with the possibility of changing the rating—up or down—based on the factors described below (i.e., risk of bias, imprecision, inconsistency, indirectness, and publication bias). Because randomization—when concealed, practiced with an intention-to-treat approach to the analysis, and with a sufficiently large sample size—deals with the problem of prognostic balance between intervention and control groups, RCTs begin with a high certainty of evidence. However, as discussed in Chapter 3, many NOFS-chronic disease questions will not have been addressed through RCTs for reasons of feasibility or cost. Also, the trials that do exist might have limited value for several reasons related to the form or dose of the

NOFS, study duration, study population, or other factors. In the GRADE approach, RCTs may be rated down for these and other reasons as the following sections describe.

Chapter 3 also discusses the fact that the relevant evidence on NOFSs and chronic disease risk is mostly observational, with large cohort studies playing a central role. In the GRADE system, conclusions based solely on observational studies are judged as having low certainty that a causal relationship is present because of concerns about residual confounding (i.e., confounding that remains after statistical adjustment), bias, and the accuracy of dietary exposure assessment. For observational studies, the certainty that the relationship is causal may be rated up to moderate under specific circumstances. Observational studies also provide information that is used in assessing intake-response relationship and in risk management. However, as noted in Chapter 3, observational studies that have a retrospective case-control or cross-sectional design may be excluded from eligibility for inclusion in systematic reviews because of their lower suitability for supporting judgments about causal relationships, in comparison to prospective studies. If included, this type of evidence would almost always be rated as low or very low quality, for reasons discussed earlier.

Addressing potential confounding typically requires appropriate use of individual-level data capable of characterizing both chronic disease risk and exposure propensity in the study population. Also, when dietary intake measurement depends on self-report, which is affected by individual characteristics, the study exposure assessment would need to have been shown to correlate strongly with a suitable objective measure of exposure (e.g., exposure biomarker) (see also Chapter 4). In the above-cited systematic review of trans fatty acids and noncommunicable disease risk, the authors used a three-tiered rating system to classify the quality of exposure assessment and applied this during their rating of study quality (de Souza et al., 2015, Appendix 2, eTable1).

Observational studies that address both sources of bias (confounding and exposure assessment) in a convincing way, and that report clear intake and chronic disease relationships, can potentially contribute usefully to establish chronic disease DRIs. However, these studies remain as low certainty evidence unless (1) the effect on the outcome is large (relative effect of two or more⁸), (2) a dose-response gradient is present, or (3) all important biases would either diminish an effect that is present or create an effect where none is observed. In Figure 6-4, the observation of a dose response led to rating up, by one level (from low to moderate) for the body of evidence on associations of both coronary heart disease (CHD) mortality and total

___________________

⁸ For continuous variables, a statistical method of conversion to binary variables should be used.

CHD events with intake of total trans fatty acids (based on table footnotes 15 and 22 in the source; not shown in the figure). In contrast, all evidence relating total trans fatty acids to other outcomes and industrial trans fatty acids to CHD outcomes was rated down from low to very low due to one or more of the following certainty factors: serious risk of bias, serious inconsistency, serious imprecision (footnotes 29, 36, 43, 50, and 57 in the source; not shown in the figure). The nature of these factors, which are especially critical with respect to observational studies, is discussed in the next sections. Bodies of evidence comprising observational studies that are subject to being rated down on the basis of serious limitations would raise substantial uncertainty with respect to supporting causal relationships.

Risk of Bias of Individual Studies (Internal Validity and Study Limitations)

One of the factors considered in assessing the certainty of a body of evidence is the risk of bias of the individual studies. Systematic reviews are expected to critically appraise individual studies and evaluate the totality of evidence for each outcome in terms of the risk of bias (i.e., not serious, serious, or very serious). Key features to consider in risk of bias assessment for RCTs are concealment of randomization, blinding, co-intervention,⁹ loss to follow-up, sufficient experimental contrast, and conduct of an intention-to-treat analysis. Although GRADE does not mandate the use of particular instruments to assess risk of bias in RCTs, guidance is offered. One suggestion is to use a modified version of the Cochrane Risk of Bias instrument (Guyatt and Busse, 2017).

For observational studies, important features to consider in risk of bias assessment are selection of exposed and unexposed individuals from different cohorts, flawed measurement of exposure and outcome (see Chapters 4

___________________

⁹ Interventions other than the treatment under study that are applied differently to the treatment and control groups.

and 5), failure to accurately measure all important prognostic factors and to adjust for these factors in the analysis, and loss to follow-up. Because of the diversity in study designs, the availability of a universal tool to assess risk of bias for all study designs is unlikely. It is also not within the study scope or feasible for this committee to delineate an exhaustive list of questions related to risk of bias assessment. The CLARITY Group at McMaster University offers candidate instruments for measuring risk of bias within observational studies (Busse and Guyatt, 2017a,b; Tikkinen et al., 2017a,b). Also, it is expected that DRI committees will include sufficient expertise in epidemiological study designs and their potential limitations, to be able to appropriately assess the risk of bias of the individual studies. As discussed below, the field of nutrition has no well-accepted risk-of-bias tools. There are no tools that prompt for evaluation of certain methodological features that are encountered with respect to assessing causal associations of NOFSs or other aspects of diet with chronic diseases (Chung, 2017). Current efforts are being directed at formally modifying risk-of-bias tools with the addition of questions that are relevant for nutrition (see also Chapter 3 for unique characteristics of nutrition research). Some nutrition questions are topic dependent, such as those related to 25(OH)D assay methods, while others are more generic, such as compliance issue for RCTs and measurement errors/biases in dietary assessment methods (see also Chapter 4). Because intake-response relationships are very important for ultimately developing DRIs, any biases in dietary assessment methods need to be considered in a risk-of-bias assessment by the addition of questions relevant to nutrition (see Chapter 4).

For RCTs involving NOFS exposures from foods, blinding of the intervention exposure is often not possible. The exception is in some controlled feeding studies, but these are usually of short duration and dependent on intermediates as outcome variables. Also, unlike trials of pharmaceuticals, the control group is always exposed to some level or form of the NOFS through intake of foods even in most trials of NOFSs given as a pill supplements. In such cases, unless the baseline exposure of subjects is considered in the trial, the real question being examined in nutrition RCTs involves the health benefits or risks of higher versus lower levels of intake, rather than specific exposure levels of the NOFS. For RCTs, therefore, a question related to whether baseline exposure has been measured and considered would be necessary in order to assess the certainty of the evidence.

In the current absence of a validated, nutrition-specific tool, the Cochrane risk-of-bias tool for RCTs¹⁰ and Newcastle-Ottawa Scale¹¹ for cohort studies, supplemented with items specific to nutrition have been used

___________________

¹⁰ See http://methods.cochrane.org/bias/assessing-risk-bias-included-studies.

¹¹ See http://www.ohri.ca/programs/clinical_epidemiology/oxford.asp.

in the field of nutrition. Another risk-of-bias tool used in nutrition is the Nutrition Evidence Library Bias Assessment Tool (NEL BAT).¹² This tool is based on Cochrane risk-of-bias domains (Selection Bias, Performance Bias, Detection Bias, Attrition Bias) and it is tailored by study design, with different sets of questions applying to RCTs (14 questions), non-RCTs (14 questions), and observational studies (12 questions). However, the tool has no nutrition-specific items.

Inconsistency of Results

Inconsistency is defined as heterogeneity or variability of study results. Issues to consider regarding consistency of results are the extent to which point estimates are similar, the extent to which confidence intervals overlap, and statistical analyses, including I², a measure of variability, and tests for heterogeneity (Guyatt et al., 2011f). Consistency of outcomes may depend on matters related to the design and conduct of the systematic review itself, such as the sharpness of study question focus, and the inclusion/exclusion criteria for studies included in the analysis, such as the precision of the outcome assessment.

When results have large inconsistencies, effect modifiers can be explored. Subgroup analyses may be misleading, and methodologists have developed criteria to assess the credibility of such analyses. These criteria include whether evidence comes from comparison within or between studies; whether chance can explain the apparent subgroup differences; whether the study questions were developed a priori, deriving from among a small number of hypotheses; and whether the expected direction of the apparent effect was accurately specified in advance (Sun et al., 2010).

Indirectness of Evidence

The available evidence may not provide an ideal match with the question of interest. This occurs, for example, when studies are not available for certain age groups or particular ethnic groups with respect to exposures or outcomes of interest. For example, the WHO systematic review on free sugars and dental caries identified only studies of children, suggesting that conclusions about adults would be indirect and therefore subject to being rated down on this basis. However, in this particular case, a judgment of serious indirectness did not occur, with the following justification:

___________________

¹² See http://www.nel.gov/topic.cfm?cat=3384.

The evidence reviewed by Te Morenga et al. relating intake of free sugars to body weight outcomes raised the potential for another type of indirectness based on short study duration (Te Morenga et al., 2012). The RCTs that addressed the impact of free sugar consumption on weight were all relatively short term. This could be considered only indirect evidence for what would happen over a more extended period. However, the evidence was not rated down for serious indirectness on the basis that, “These short-term studies were of sufficient duration to detect a change on the outcome of interest and, thus, the data provide relevant evidence for the association between increased free sugars intakes and weight gain” (Te Morenga et al., 2012, supplement 2, footnote 4).

Other types of indirectness are situations in which available studies of individual NOFSs do not reflect dietary patterns of a population of interest or when only surrogate markers rather than the disease outcomes of interest have been studied. The comment in the above quotation from Moynihan and Kelly about the range of populations studied addresses the breadth of population coverage. In the case of fatty acids, evidence from blood lipids, a possible surrogate marker, was used to facilitate judgments about certain questions, but the overall evidence base included data on mortality and other events.

Studies also may provide only indirect comparisons in other ways. The effect of Diet A versus Diet B may be of interest, but no direct comparisons are available. Yet, if both Diet A and Diet B have been compared to Diet C, this allows the possibility of an indirect comparison between A and B. Such indirect comparisons are at greater risk of producing misleading results than are direct comparisons. However, this linking approach may be useful in complementing information not available in direct comparisons.

Imprecision

In the context of nutritional studies, examination of the boundaries of the confidence interval for a given overall effect informs the decision

regarding precision. Imprecision is the random error in studies that results in wide 95 percent confidence intervals around the estimate of the effect. This differs from inconsistency, which refers to the consistency across individual studies. For each outcome, the authors of each study, or ideally the DRI committee, evaluate the decision regarding whether a proposed intervention or comparator is preferable by selecting estimates at both ends of the confidence interval, to see whether these alternatives would change the interpretation. If it would, then one rates down the evidence for imprecision. For instance, in Figure 6-4, evidence from cohort studies of total trans fatty acid on ischemic stroke was rated down for imprecision because of 95 percent confidence intervals that suggested benefit at the lower bound and exceeded the threshold for harm at the upper bound (de Souza et al., 2015, appendix 6, footnote 27).

This guidance has one exception. If a study or studies report a large or very large effect with a relatively small number of events, experience has shown that the results do not stand the test of time. Therefore, under these circumstances, even if the confidence interval standard suggests satisfactory precision, the study certainty may be rated down on the grounds of imprecision.

If a DRI committee wished to be quantitative about the decisions to rate down a body of evidence under these circumstances, it could specify a magnitude of effect (typically a modest relative risk reduction (RRR), for example ≤30 percent) and calculate the sample size required, in a single study, to have adequate power (80 or 90 percent) with the usual Type 1 error threshold of 0.05 and assume the baseline risk in the studies available. The sample size that results from this exercise is labeled the optimal information size (OIS). If the total number of individuals in a systematic review is less than the number of individuals generated by a conventional sample size calculation for a single adequately powered trial, or the OIS, the committee should consider rating down for imprecision (Schunemann et al., 2013). Many online calculators are available for sample size calculation.¹³ Alternatively, Table 6-1 provides a guide for what happens, given the number of events observed in the studies, when this exercise is conducted with RRRs of 20 to 30 percent. For example, if an RRR of ≤30 is chosen when the total number of events is 100 or less, the thresholds for precision will most likely not be met (Guyatt et al., 2011e; Pogue and Yusef, 1997).

Publication Bias

Publication bias is defined as the systematic under-estimation or an over-estimation of the underlying beneficial or harmful effect due to

___________________

¹³ See http://www.stat.ubc.ca/~rollin/stats/ssize/b2.html.

TABLE 6-1 Optimal Information Size Implications When an Adequate Power and Magnitude of Effect Is Desired

SOURCE: Adapted from Guyatt et al., 2011e.

the selective publication of studies. In the GRADE approach, options include rating down when the evidence comes from a number of small industry-funded trials, creating and evaluating a funnel plot, and using one of a number of statistical tests that evaluate the asymmetry of the data (small studies showing larger effects than the larger studies) (Guyatt et al., 2011d). The methods guide on grading the strength of a body of evidence by AHRQ describes a conceptual framework and recommends both qualitative and quantitative methods, such as the funnel plot, to assess publication bias. Although both of these groups offer important guidance, they also recognize that measuring publications bias is challenging (AHRQ, 2014; Guyatt et al., 2011d) and that there are no completely satisfactory strategies. This is particularly the case when the body of evidence is from observational studies because assessors might not know if the published observational studies are representative of the studies conducted due to the lack of published or registered protocols. With this in mind, the terms GRADE suggests using are “undetected” and “strongly suspected” publication bias. In the examples in Figure 6-4, publication bias was not assessed formally due to too few studies.

Factors That Can Increase the Certainty of the Evidence

Large Magnitude of an Effect

Rating up for large effect size is typically applied in the context of observational studies. GRADE’s rule of thumb is that evidence may be rated up one level if the odds ratio or relative risk is >2.0 or <0.5 or two levels if the relative risk is >10 or <0.1. The WHO free sugars review (Te Morenga et al., 2012) rated up certainty in evidence based on a large effect of sugar consumption on dental caries. Although large effects are rarely seen with any nutrition intervention unless individuals are deficient in a particular nutrient, the committee concluded that uprating the confidence in the body of evidence of observational studies requires a large effect in the health outcome. This requirement provides higher assurance that the association could not be due to residual confounding.

Intake-Response

Another reason for rating up evidence certainty is observing an intake-response gradient, in this case between the NOFS and the health endpoint of interest. An intake-response gradient implies that the magnitude of the outcome increases in a linear or non-linear fashion with increasing magnitude of exposure and the question being answered would be: “What is the level of confidence that an intake-response relationship gradient exists?” In contrast, Chapter 7 addresses not only whether an intake-response relationship exists but also how to characterize it quantitatively.

The type of evidence needed for rating up the certainty of the evidence based on an intake-response relationship is less stringent than that needed to characterize the relationship quantitatively. For instance, intake levels may be categorical (low, medium, high), so that while evidence of an increase in response with intake exists, a quantitative relationship cannot be determined. Alternatively, one might have high confidence that the “slope” of a linear relationship is greater than zero, but because of potential bias toward the null, confidence is lower as to what the value of the slope might be. In attempting to make this distinction, this report reserves the term quantitative intake-response for a relationship that can be characterized quantitatively.

Effect of Plausible Residual Confounding

Very rarely, evidence certainty may be rated up when all plausible confounding factors would minimize an observed effect, or create an effect when an effect has not been observed. A systematic review that reported

higher death rates in private for-profit versus private not-for-profit hospitals provides an example of the phenomenon of plausible confounding (Guyatt et al., 2011h). Plausible biases included patients in the not-for-profit hospitals being sicker and higher numbers of patients with excellent private insurance coverage in the for-profit hospitals, both of which would be expected to result in better outcomes for private for-profit hospital patients. Because these biases would diminish the observed effect, even if they were not accounted for in the analysis, the circumstances surrounding this confounder might lead the evidence panel to consider the evidence from observational studies as moderate rather than low certainty (Guyatt et al., 2011h).

Rating the Overall Certainty of Evidence

Within the GRADE approach, the evidence is rated by outcome and study design. That is, observational studies are rated separately from RCTs for each outcome. Within evidence from observational studies, those with different designs may also be rated separately. This is illustrated in the list of supplemental tables from a systematic review of observational studies of fatty acids and noncommunicable diseases in Figure 6-3. Typically, certainty of evidence differs across outcomes. For each outcome, when evidence from both RCTs and observational studies is available, the focus should be on the higher certainty evidence, and the overall rating of certainty should be that of the higher certainty. With regard to the overall certainty of evidence, GRADE specifies that it should be designated as the lowest detected among the critical outcomes.

ACCEPTABLE LEVEL OF EVIDENCE THAT THE RELATIONSHIP IS CAUSAL

As already mentioned, once it receives the systematic review with evidence profiles and summary-of-findings tables for all outcomes of interest, positive and adverse, the DRI committee will evaluate the evidence presented, note and justify agreements and disagreements, and evaluate the importance of the health outcomes. The DRI committee first makes a decision on the certainty of the evidence with regard to a causal relationship for each outcome it views as important. If such a relationship exists, then the DRI committee will characterize the certainty of the intake-response relationships, consider benefits and harms and their relative importance, and recommend chronic disease DRIs levels or ranges when appropriate, as explained in Chapter 7.

Determining causation between exposures and outcomes has been a longstanding philosophical as well as practical challenge in population

research as well as for science in general. As discussed above, RCTs, as true experiments, provide the best overall evidence of causation, but the actual certainty in the evidence depends on the certainty of those trials. Other criteria and systems for approaching causation exist, such as the Bradford Hill criteria (Hill, 1965). Parascandola and Weed reviewed the epidemiological literature on causation and found multiple definitions (Parascandola and Weed, 2001).

The Options Report (Yetley et al., 2017) had one set of options that directly addressed causation. Four options were presented, shown in Box 6-4, relating to the levels of confidence that an association between an NOFS exposure and an outcome is causal. All the options portrayed are feasible approaches to defining acceptable levels. The GRADE approach addresses the issue of whether the association between an exposure and an outcome is causal (Schunemann et al., 2013). After considering these approaches in detail and based on the committee’s evaluation of the GRADE approach, the recommendation is that to accept the likelihood of a causal relationship

between an NOFS and a chronic disease outcome, a moderate rating of the relevant overall evidence base derived using GRADE criteria is needed. Lesser levels of confidence in causation should not lead to actionable recommendations or, in the case here, the development of chronic disease DRIs. Presumably, to move forward with developing chronic disease DRIs, determination of an intake-response in data from observational studies as part of the evidence rating also would be needed.

The GRADE system does not specify either a minimum number of studies or participants (although it does provide guidance for minimum number of participants to avoid rating down certainty for imprecision) or the characteristics of the study design (e.g., observational studies have established the causal relationship of smoking and lung cancer) necessary for determining that a relationship has at least moderate certainty, and

therefore is likely to be causal. In the same manner, the DRI committees will apply their judgment relative to this matter.

GUIDING PRINCIPLES AS FOUNDATION FOR A CHRONIC DISEASE DRI PROCESS

Guiding principles suitable for the evidence review process are summarized in Box 6-5. These principles reflect the fact that evidence review in general is well developed as a scientific activity for informing policy and practice. The guiding principles are the foundation of a rigorous scientific process for setting chronic disease DRIs.

REFERENCES

de Souza, R. J., A. Mente, A. Maroleanu, A. I. Cozma, V. Ha, T. Kishibe, E. Uleryk, P. Budylowski, H. Schunemann, J. Beyene, and S. S. Anand. 2015. Intake of saturated and trans unsaturated fatty acids and risk of all cause mortality, cardiovascular disease, and type 2 diabetes: Systematic review and meta-analysis of observational studies. BMJ 351:h3978.

Dwyer, J. T., K. H. Rubin, K. L. Fritsche, T. L. Psota, D. J. Liska, W. S. Harris, S. J. Montain, and B. J. Lyle. 2016. Creating the future of evidence-based nutrition recommendations: Case studies from lipid research. Adv Nutr 7(4):747-755.

Garcia-Larsen, V., S. R. Del Giacco, A. Moreira, M. Bonini, D. Charles, T. Reeves, K. H. Carlsen, T. Haahtela, S. Bonini, J. Fonseca, I. Agache, N. G. Papadopoulos, and L. Delgado. 2016. Asthma and dietary intake: An overview of systematic reviews. Allergy 71(4):433-442.

Garner, P., S. Hopewell, J. Chandler, H. MacLehose, H. J. Schunemann, E. A. Akl, J. Beyene, S. Chang, R. Churchill, K. Dearness, G. Guyatt, C. Lefebvre, B. Liles, R. Marshall, L. Martinez Garcia, C. Mavergames, M. Nasser, A. Qaseem, M. Sampson, K. Soares-Weiser, Y. Takwoingi, L. Thabane, M. Trivella, P. Tugwell, E. Welsh, E. C. Wilson, H. J. Schunemann, and Panel for Updating Guidance for Systematic Reviews. 2016. When and how to update systematic reviews: Consensus and checklist. BMJ 354:i3507.

Guyatt, G., and J. W. Busse. 2017. Modification of Cochrane tool to assess risk of bias in randomized trials. https://www.evidencepartners.com/resources/methodological-resources (accessed May 12, 2017).

Guyatt, G. H., A. D. Oxman, R. Kunz, Y. Falck-Ytter, G. E. Vist, A. Liberati, H. J. Schunemann, and Grade Working Group. 2008a. Going from evidence to recommendations. BMJ 336(7652):1049-1051.

Guyatt, G. H., A. D. Oxman, G. E. Vist, R. Kunz, Y. Falck-Ytter, P. Alonso-Coello, H. J. Schunemann, and Grade Working Group. 2008b. GRADE: An emerging consensus on rating quality of evidence and strength of recommendations. BMJ 336(7650):924-926.

Guyatt, G. H., A. D. Oxman, R. Kunz, R. Jaeschke, M. Helfand, A. Liberati, G. E. Vist, H. J. Schunemann, and Grade Working Group. 2008c. Incorporating considerations of resources use into grading recommendations. BMJ 336(7654):1170-1173.

Guyatt, G. H., A. D. Oxman, R. Kunz, G. E. Vist, Y. Falck-Ytter, H. J. Schunemann, and Grade Working Group. 2008d. What is “quality of evidence” and why is it important to clinicians? BMJ 336(7651):995-998.

Guyatt, G., A. D. Oxman, E. A. Akl, R. Kunz, G. Vist, J. Brozek, S. Norris, Y. Falck-Ytter, P. Glasziou, H. DeBeer, R. Jaeschke, D. Rind, J. Meerpohl, P. Dahm, and H. J. Schunemann. 2011a. GRADE guidelines: 1. Introduction-GRADE evidence profiles and summary of findings tables. J Clin Epidemiol 64(4):383-394.

Guyatt, G. H., A. D. Oxman, R. Kunz, D. Atkins, J. Brozek, G. Vist, P. Alderson, P. Glasziou, Y. Falck-Ytter, and H. J. Schunemann. 2011b. GRADE guidelines: 2. Framing the question and deciding on important outcomes. J Clin Epidemiol 64(4):395-400.

Guyatt, G. H., A. D. Oxman, G. Vist, R. Kunz, J. Brozek, P. Alonso-Coello, V. Montori, E. A. Akl, B. Djulbegovic, Y. Falck-Ytter, S. L. Norris, J. W. Williams, Jr., D. Atkins, J. Meerpohl, and H. J. Schunemann. 2011c. GRADE guidelines: 4. Rating the quality of evidence—Study limitations (risk of bias). J Clin Epidemiol 64(4):407-415.

Guyatt, G. H., A. D. Oxman, V. Montori, G. Vist, R. Kunz, J. Brozek, P. Alonso-Coello, B. Djulbegovic, D. Atkins, Y. Falck-Ytter, J. W. Williams, Jr., J. Meerpohl, S. L. Norris, E. A. Akl, and H. J. Schunemann. 2011d. GRADE guidelines: 5. Rating the quality of evidence—Publication bias. J Clin Epidemiol 64(12):1277-1282.

Guyatt, G. H., A. D. Oxman, R. Kunz, J. Brozek, P. Alonso-Coello, D. Rind, P. J. Devereaux, V. M. Montori, B. Freyschuss, G. Vist, R. Jaeschke, J. W. Williams, Jr., M. H. Murad, D. Sinclair, Y. Falck-Ytter, J. Meerpohl, C. Whittington, K. Thorlund, J. Andrews, and H. J. Schunemann. 2011e. GRADE guidelines 6. Rating the quality of evidence—Imprecision. J Clin Epidemiol 64(12):1283-1293.

Guyatt, G. H., A. D. Oxman, R. Kunz, J. Woodcock, J. Brozek, M. Helfand, P. Alonso-Coello, P. Glasziou, R. Jaeschke, E. A. Akl, S. Norris, G. Vist, P. Dahm, V. K. Shukla, J. Higgins, Y. Falck-Ytter, H. J. Schunemann, and Grade Working Group. 2011f. GRADE guidelines: 7. Rating the quality of evidence—Inconsistency. J Clin Epidemiol 64(12):1294-1302.

Guyatt, G. H., A. D. Oxman, R. Kunz, J. Woodcock, J. Brozek, M. Helfand, P. Alonso-Coello, Y. Falck-Ytter, R. Jaeschke, G. Vist, E. A. Akl, P. N. Post, S. Norris, J. Meerpohl, V. K. Shukla, M. Nasser, H. J. Schunemann, and Grade Working Group. 2011g. GRADE guidelines: 8. Rating the quality of evidence—Indirectness. J Clin Epidemiol 64(12):1303-1310.

Guyatt, G. H., A. D. Oxman, S. Sultan, P. Glasziou, E. A. Akl, P. Alonso-Coello, D. Atkins, R. Kunz, J. Brozek, V. Montori, R. Jaeschke, D. Rind, P. Dahm, J. Meerpohl, G. Vist, E. Berliner, S. Norris, Y. Falck-Ytter, M. H. Murad, H. J. Schunemann, and Grade Working Group. 2011h. GRADE guidelines: 9. Rating up the quality of evidence. J Clin Epidemiol 64(12):1311-1316.

Hill, A. B. 1965. The environment and disease: Association or causation? Proc R Soc Med 58:295-300.

IOM (Institute of Medicine). 2011a. Dietary Reference Intakes for calcium and vitamin D. Washington, DC: The National Academies Press.

IOM. 2011b. Finding what works in health care: Standards for systematic reviews. Washington, DC: The National Academies Press.

Jaeschke, R., G. H. Guyatt, P. Dellinger, H. Schunemann, M. M. Levy, R. Kunz, S. Norris, J. Bion, and Grade Working Group. 2008. Use of GRADE grid to reach decisions on clinical practice guidelines when consensus is elusive. BMJ 337:a744.

Mayhew, A. J., R. J. de Souza, D. Meyre, S. S. Anand, and A. Mente. 2016. A systematic review and meta-analysis of nut consumption and incident risk of CVD and all-cause mortality. Br J Nutr 115(2):212-225.

Mensink, R. P. 2016. Effects of saturated fatty acids on serum lipids and lipoproteins: A systematic review and regression analysis. Geneva, Switzerland: World Health Organization.

Moynihan, P. J., and S. A. Kelly. 2014. Effect on caries of restricting sugars intake: Systematic review to inform WHO guidelines. J Dent Res 93(1):8-18.

NASEM (National Academies of Sciences, Engineering, and Medicine). 2017. Optimizing the process for establishing the Dietary Guidelines for Americans: The selection process. Washington, DC: The National Academies Press.

Naude, C. E., A. Schoonees, M. Senekal, T. Young, P. Garner, and J. Volmink. 2014. Low carbohydrate versus isoenergetic balanced diets for reducing weight and cardiovascular risk: A systematic review and meta-analysis. PLOS ONE 9(7):e100652.

NEL (Nutrition Evidence Library). 2015. Nutrition Evidence Library—2015 Dietary Guidelines Advisory Committee systematic reviews. https://www.cnpp.usda.gov/nutrition-evidence-library-2015-dietary-guidelines-advisory-committee-systematic-reviews (accessed April 28, 2017).

NTP (National Toxicology Program). 2015. Handbook for conducting a literature-based health assessment using OHAT approach for systematic review and evidence integration. https://ntp.niehs.nih.gov/pubhealth/hat/noms/index-2.html (accessed July 16, 2017).

Parascandola, M., and D. L. Weed. 2001. Causation in epidemiology. J Epidemiol Community Health 55(12):905-912.

Pennant, M., M. Steur, C. Moore, A. Butterworth, and L. Johnson. 2015. Comparative validity of vitamin C and carotenoids as indicators of fruit and vegetable intake: A systematic review and meta-analysis of randomised controlled trials. Br J Nutr 114(9):1331-1340.

Pieper, D., T. Mathes, and M. Eikermann. 2014. Can AMSTAR also be applied to systematic reviews of non-randomized studies? BMC Res Notes 7:609.

Pogue, J. M., and S. Yusuf. 1997. Cumulating evidence from randomized trials: Utilizing sequential monitoring boundaries for cumulative meta-analysis. Control Clin Trials 18(6):580-593; discussion 661-666.

Puhan, M. A., H. J. Schunemann, M. H. Murad, T. Li, R. Brignardello-Petersen, J. A. Singh, A. G. Kessels, G. H. Guyatt, and Grade Working Group. 2014. A GRADE Working Group approach for rating the quality of treatment effect estimates from network meta-analysis. BMJ 349:g5630.

Santesso, N., E. A. Akl, M. Bianchi, A. Mente, R. Mustafa, D. Heels-Ansdell, and H. J. Schunemann. 2012. Effects of higher-versus lower-protein diets on health outcomes: A systematic review and meta-analysis. Eur J Clin Nutr 66(7):780-788.

Santesso, N., A. Carrasco-Labra, M. Langendam, R. Brignardello-Petersen, R. A. Mustafa, P. Heus, T. Lasserson, N. Opiyo, I. Kunnamo, D. Sinclair, P. Garner, S. Treweek, D. Tovey, E. A. Akl, P. Tugwell, J. L. Brozek, G. Guyatt, and H. J. Schunemann. 2016. Improving GRADE evidence tables part 3: Detailed guidance for explanatory footnotes supports creating and understanding GRADE certainty in the evidence judgments. J Clin Epidemiol 74:28-39.

Schunemann, H. J., A. D. Oxman, J. Brozek, P. Glasziou, R. Jaeschke, G. E. Vist, J. W. Williams, Jr., R. Kunz, J. Craig, V. M. Montori, P. Bossuyt, G. H. Guyatt, and Grade Working Group. 2008. Grading quality of evidence and strength of recommendations for diagnostic tests and strategies. BMJ 336(7653):1106-1110.

Schunemann, H., J. Brozek, G. Guyatt, and A. Oxman. 2013. Introduction to GRADE Handbook. http://gdt.guidelinedevelopment.org/app/handbook/handbook.html (accessed May 16, 2017).

Shea, B. J., J. M. Grimshaw, G. A. Wells, M. Boers, N. Andersson, C. Hamel, A. C. Porter, P. Tugwell, D. Moher, and L. M. Bouter. 2007. Development of AMSTAR: A measurement tool to assess the methodological quality of systematic reviews. BMC Med Res Methodol 7:10.

Shea, B. J., C. Hamel, G. A. Wells, L. M. Bouter, E. Kristjansson, J. Grimshaw, D. A. Henry, and M. Boers. 2009. AMSTAR is a reliable and valid measurement tool to assess the methodological quality of systematic reviews. J Clin Epidemiol 62(10):1013-1020.

Shekelle, P., S. Newberry, M. Maglione, R. Shanman, B. Johnsen, J. Carter, A. Motala, B. Hulley, Z. Wang, D. Bravata, M. Chen, and J. Grossman. 2009. Assessment of the need to update comparative effectiveness reviews: Report of an initial rapid program assessment (2005-2009), Rockville, MD: Agency for Healthcare Research and Quality. https://www.ncbi.nlm.nih.gov/pubmed/21204320 (accessed July 16, 2017).

Sun, X., M. Briel, S. D. Walter, and G. H. Guyatt. 2010. Is a subgroup effect believable? Updating criteria to evaluate the credibility of subgroup analyses. BMJ 340:c117.

Te Morenga, L., S. Mallard, and J. Mann. 2012. Dietary sugars and body weight: Systematic review and meta-analyses of randomised controlled trials and cohort studies. BMJ 346:e7492.

Tikkinen, K. A., J. W. Busse, and G. Guyatt. 2017a. Tool to assess risk of bias in observational studies of natural history of medical symptoms/conditions in general populations. https://www.evidencepartners.com/resources/methodological-resources (accessed May 16, 2017).

Tikkinen, K. A., R. Cartwright, and G. Guyatt. 2017b. Tool to assess risk of bias in studies of prognosis. https://www.evidencepartners.com/resources/methodological-resources (accessed May 16, 2017).

USPSTF (U.S. Preventive Services Task Force). 2015. Procedure manual. https://www.uspreventiveservicestaskforce.org/Page/Name/methods-and-processes (accessed July 16, 2017).

WCRF/AICR (World Cancer Research Fund/American Institute for Cancer Research). 2007. Food, nutrition, physical activity, and the prevention of cancer: A global perspective. Washington, DC: AICR.

Whiting, P., P. Davies, J. Savovic, D. M. Caldwell, and R. Churchill. 2013. Evidence to inform the development of ROBIS, a new tool to assess the risk of bias in systematic reviews. http://www.bristol.ac.uk/media-library/sites/social-community-medicine/robis/ROBIS%20Report%204_9.pdf (accessed July 16, 2017).

Whiting, P., J. Savovic, J. P. Higgins, D. M. Caldwell, B. C. Reeves, B. Shea, P. Davies, J. Kleijnen, R. Churchill, and ROBIS group. 2016. ROBIS: A new tool to assess risk of bias in systematic reviews was developed. J Clin Epidemiol 69:225-234.

WHO (World Health Organization). 2014. WHO handbook for guideline development. Geneva, Switzerland: World Health Organization.

WHO. 2015. Sugars intake for adults and children. Guideline. Geneva, Switzerland: World Health Organization.

Yetley, E. A., A. J. MacFarlane, L. S. Greene-Finestone, C. Garza, J. D. Ard, S. A. Atkinson, D. M. Bier, A. L. Carriquiry, W. R. Harlan, D. Hattis, J. C. King, D. Krewski, D. L. O’Connor, R. L. Prentice, J. V. Rodricks, and G. A. Wells. 2017. Options for basing Dietary Reference Intakes (DRIs) on chronic disease endpoints: Report from a joint US-/ Canadian-sponsored working group. Am J Clin Nutr 105(1):249S-285S.

ANNEX 6-1
PRIORITY QUESTIONS IN THE FORMAT OF POPULATION, INTERVENTION, CONTROL, AND OUTCOMES (PICO)¹⁴ FOR GUIDING THE SYSTEMATIC REVIEWS IN SUPPORT OF THE WORLD HEALTH ORGANIZATION GUIDELINE ON FREE SUGARS INTAKE FOR ADULTS AND CHILDREN

What is the effect of a decrease or increase in free sugars intake in adults and children?

What is the effect of restricting intake of free sugars to below 10 percent of total energy intake?¹⁵

___________________

¹⁴ Reprinted with permission from WHO (World Health Organization). 2015. Sugars intake for adults and children. Guideline. Geneva, Switzerland: World Health Organization.

¹⁵ Less than 10 percent of total energy intake is the existing population nutrient intake goal for free sugars (WHO/FAO, 2003).

___________________

¹⁶ The percentage of fat (i.e., adipose tissue) that a person’s body contains.

ANNEX 6-1 REFERENCES

ANNEX 6-2
ASSESSMENT TOOLS USED IN SYSTEMATIC REVIEWS

TABLE ANNEX 6-2-1 Examples of Tools for Risk of Bias (or Quality) Assessment for Individual Studies Included in the Systematic Reviews

^a See http://www.nel.gov/topic.cfm?cat=3384 (accessed July 24, 2017).

^b See http://methods.cochrane.org/bias/assessing-risk-bias-included-studies (accessed July 24, 2017); http://handbook.cochrane.org (accessed July 24, 2017); Higgins et al., 2011.

^c See https://sites.google.com/site/riskofbiastool/welcome/home (accessed July 24, 2017); Sterne et al., 2016.

^d See http://www.ohri.ca/programs/clinical_epidemiology/oxford.asp (accessed July 24, 2017).

^e See https://ntp.niehs.nih.gov/pubhealth/hat/noms/index-2.html (accessed July 24, 2017).

TABLE ANNEX 6-2-2 Tools for Evaluating the Strength (or Quality) of the Body of Evidence*

* Any grading tool for strength of the body of evidence can be applied to grade the body of evidence of any study design that is defined by the inclusion criteria of the systematic reviews. If a systematic review included both observational studies and randomized controlled trials (RCTs), the strength of the body of evidence is rated as such. If a systematic review only included RCTs, the strength of the body of evidence can only rate the body of RCT evidence.

^a See www.gradeworkinggroup.org (accessed July 24, 2017).

^b See http://www.nutrigrade.net (accessed July 24, 2017); Schwingshackl et al., 2016.

^c See http://www.nel.gov/topic.cfm?cat=3368 (accessed July 24, 2017); https://health.gov/dietaryguidelines/2015-scientific-report/05-methodology.asp (accessed July 24, 2017).

^d See http://effectivehealthcare.ahrq.gov/index.cfm/search-for-guides-reviews-and-reports/?pageaction=displayproduct&productid=318 (accessed July 24, 2017); Berkman et al., 2015.

^e See https://www.uspreventiveservicestaskforce.org/Page/Name/methods-and-processes (accessed July 24, 2017).

^f See https://ntp.niehs.nih.gov/pubhealth/hat/noms/index-2.html (accessed July 24, 2017).

^g WCRF/AICR, 2007.

TABLE ANNEX 6-2-3 Tools for Evaluating the “Quality” (or Reporting Quality*) of Systematic Reviews

* Only ROBIS was developed to assess the risk of bias of a systematic review. AMSTAR is primarily a rating of reporting quality and cannot assess the other potential biases in a systematic review. AMSTAR is generally used as a quality assessment tool for complex systematic reviews that want to include existing systematic reviews to determine whether a systematic review has sufficient quality to be included.

^a See https://amstar.ca/index.php (accessed July 24, 2017); Pieper et al., 2014; Shea et al., 2007, 2009.

^b Kung et al., 2010; Pieper et al., 2015.

^c See http://www.bristol.ac.uk/social-community-medicine/projects/robis (accessed July 24, 2017); Whiting et al., 2013, 2016.

ANNEX 6-2 REFERENCES

This page intentionally left blank.