6

Current Approaches to Examining the Evidence: Key Findings

The National Nutrition Monitoring and Related Research Act of 1990 states that the Dietary Guidelines for Americans (DGA)¹ should provide “nutritional and dietary information and guidelines . . . based on the preponderance of the scientific and medical knowledge which is current at the time the report is prepared.”² As written in the Statement of Task (see Box 1-3), this committee of the National Academies of Sciences, Engineering, and Medicine (the National Academies) was requested by Congress to review “(2) how the Nutrition Evidence Library (NEL) is compiled and utilized, including whether NEL reviews and other systematic reviews and data analysis are conducted according to rigorous and objective scientific standards; and (3) how systematic reviews are conducted on longstanding DGA recommendations, including whether scientific studies are included from scientists with a range of viewpoints.”³ To respond to these requests, this chapter summarizes the approach taken by this National Academies committee to review and evaluate the processes used in examining the evidence that underlies the DGA recommendations.

This chapter is divided into sections to reflect the types of analyses traditionally used by the Dietary Guidelines Advisory Committee (DGAC):

___________________

¹ Refer to Chapter 1, Box 1-1, for an explanation of how the term DGA is used throughout this National Academies report.

² National Nutrition Monitoring and Related Research Act of 1990, Public Law 101-445, 101st Cong. (October 22, 1990), 7 U.S.C. 5341, 104 Stat. 1042–1044.

³ Consolidated Appropriations Act, 2016, Public Law 114-113, 114th Cong. (December 18, 2015), 129 Stat. 2280–2281.

(1) original NEL systematic reviews, (2) existing systematic reviews, meta-analyses, and reports, (3) food pattern modeling, and (4) descriptive data analyses. For each type of analysis, this National Academies committee first considered the types of research questions that are relevant for the DGAC; the types of analysis that are appropriate to address these questions; and how the 2015 DGAC’s review of evidence was conducted. This chapter then discusses ways in which the process can be strengthened or enhanced to best support the DGA in the future. Finally, this National Academies committee considered the necessity for and availability of high-quality data for use in each of the four types of analyses used by the 2015 DGAC.

OVERVIEW OF THE TYPES OF RESEARCH QUESTIONS AND ANALYSES USED BY THE DGAC

This National Academies committee analyzed the questions answered by the 2015 DGAC and categorized the questions in the DGAC’s evidence review—inclusive of all evidence types—into three broad groupings: eating patterns, prevalence of disease, and relationships between diet and health (see Table 6-1). In some instances, previous DGACs have used multiple analyses for reviewing the evidence to address a specific question.

To understand the prevalence of disease in the overall population, the DGAC asked a series of descriptive questions. Analyses of U.S. population data were used to estimate the number of Americans living with certain chronic diseases.

Questions related to eating patterns included examination of (1) current patterns of food and nutrient consumption in the United States and (2) how changes in food choices would alter dietary intakes. While these areas are related, three main types of questions were addressed (i.e., descriptive, relational, and predictive questions); the 2015 DGAC used

TABLE 6-1 Types of Research Questions Asked and Examples from the 2015 DGAC Scientific Report

NOTES: This table summarizes approaches the 2005, 2010, and 2015 DGACs have taken in their evidence reviews, including the types of research questions asked. It does not offer all possible types of analyses that could be used to answer these questions. DGAC = Dietary Guidelines Advisory Committee; NHANES = National Health and Nutrition Examination Survey; USDA = U.S. Department of Agriculture.

four general types of analyses to answer the questions. To evaluate current eating patterns, the 2015 DGAC asked descriptive questions about the food and nutrient intakes of Americans. These were answered using analyses of U.S. population data such as the National Health and Nutrition Examination Survey (NHANES). To project the effect of changes in eating patterns, predictive questions were asked to anticipate potential outcomes. These types of questions have been addressed through food pattern modeling, a kind of analysis that predicts the effect of the recommended changes.

Given the DGAC’s mandate to “promote health and prevent disease,” of particular interest and concern for the 2015 DGAC were relationships between diet and health and disease outcomes (including the nature of the relationships, as well as intermediate outcomes) (HHS/USDA, 2015b). Therefore, many of the questions answered by the 2015 DGAC were relational questions. These questions were answered using original systematic reviews, and/or existing systematic reviews, meta-analyses, and reports in the literature.

Methodological Approaches to Different Types of Questions

Over time, the types of analyses used by DGACs to develop the scientific basis for the DGA have evolved. Descriptive data analyses were available for use by the DGAC since the origin of the guidelines, but data on dietary intakes were only formally considered by the DGAC as recently as 2010. Food pattern modeling was available and used to produce recommended intakes of food groups since the 1990 edition. Relationships between diet and health were historically based on ad hoc expert examination of the existing literature. However, as the science of evidence review evolved, more standardized methods of systematic review have been considered and employed in the DGAC’s review of the evidence. The NEL was introduced in the 2010 cycle. The 2015 DGAC used the following types of analyses:

• NEL systematic reviews: Comprehensive reviews of the literature that adhere to established principles, as well as updates of existing systematic reviews
• Existing sources of evidence: Evaluations of sources of evidence such as published systematic reviews, meta-analyses, and reports
• Food pattern modeling: A type of sensitivity analysis that incorporates various data inputs, constraints, goals, and assumptions to inform food patterns and resulting nutrient profiles, as well as to answer various questions regarding the effects of modifications to food patterns

• Descriptive data analyses: A type of analysis used to answer descriptive questions about overall population trends and population subgroups

DGACs have also considered invited expert testimonies and public comments.

ASSESSMENT OF SYSTEMATIC REVIEW METHODS

Systematic reviews provide a synthesis of relevant existing research on a particular topic.⁴ Systematic reviews are a significant source of evidence for the DGAC. Prior to 2010, DGACs relied on existing reports available in the published literature, or drew conclusions based on their own review of the evidence. The NEL is a program housed in the U.S. Department of Agriculture (USDA) Center for Nutrition Policy and Promotion, and it conducts systematic reviews to inform federal nutrition policy and programs (USDA, 2017). It was developed in part to provide support, as well as a structure and protocol, for the DGAC to conduct original systematic reviews (USDA/HHS, 2016).

The use of systematic reviews has varied across cycles with respect to the use of original and existing systematic reviews, meta-analyses, and reports. The 2005 DGAC answered around 44 percent of questions with an evidence-based literature review (17 out of 32 total questions), and roughly the same percentage with existing publications. With the introduction of the NEL in 2010, the number of systematic reviews used by the DGAC increased. During the 2010 cycle, 76 percent of the questions (44 out of 59 total questions) were answered by an original systematic review, while existing publications were used to answer about 12 questions (20 percent). In the 2015–2020 DGA, 25 percent of the questions (23 of 91 total questions) were answered by an original systematic review, while existing publications were used to answer 44 percent of the questions (40 of 91 total questions).⁵ Notably, systematic review methodology has become increasingly common, and thus more reviews were likely available for use by the DGAC in 2015 than in 2010. Despite some variation across DGA cycles, systematic reviews, both previously existing and conducted de novo, have served as a key source of evidence. The devel-

___________________

⁴ Systematic reviews are designed to answer a specific question(s). The DGAC process for selecting and refining topics precedes the development of systematic review questions and is described in detail in Chapter 5. Systematic review questions are developed according to the criteria outlined below in Step 1 of the NEL process.

⁵ These numbers were calculated based on the 2005, 2010, and 2015 DGAC reports. See Appendix C for a complete list of questions answered by the 2015 DGAC.

opment of the NEL has led to centralization and standardization of the systematic review process across the most recent DGA cycles.

Questions That Systematic Reviews Are Intended to Address

Systematic reviews provide important insights into the relationships between diet and health. For example, such questions as “What is the relationship between dietary patterns and risk of cancer?” and “What is the relationship between sodium intake and cardiovascular disease outcomes?” were the subject of two systematic reviews in the 2015 DGAC Scientific Report. This type of question intends to assess the stated relationship between a particular aspect of diet within a defined population, with respect to a particular intervention and defined health outcome, and with consideration for known potential confounders. Using systematic reviews to understand the nature and types of these relationships also means distinguishing between causality and associations, depending on the study types and data available. The quality of the studies available may also limit the ability to make certain inferences, and careful consideration of quality and risk of bias of included studies in systematic reviews is important.

The 2015 DGAC also used systematic reviews to consider the relationships of other factors influencing diet and/or affecting health outcomes of interest. For example, the questions “What is the relationship between neighborhood and community access to food retail settings and weight status?” and “What is the impact of obesity prevention approaches in early care and education programs on the weight status of children ages 2 to 5 years?” consider weight status as a health outcome of interest.

Questions of relationship can also be developed in such a way to assess the effect of a particular dietary factor on a health or disease outcome, including intermediate outcomes. Examples of this type of question include “What effect does the interrelationship of sodium and potassium have on blood pressure and cardiovascular disease outcomes?” from the 2015 DGAC Scientific Report, and, “What are the effects of dietary stearic acid on LDL cholesterol?” from the 2010 DGAC Scientific Report.

According to NEL protocol, systematic review questions are developed and prioritized in advance of the decision to use an existing systematic review, meta-analysis, or report, or to conduct a de novo systematic review. The process of identifying, evaluating, and deciding whether or not an existing systematic review should be included or excluded requires a different set of considerations than conducting an original systematic review (see “Assessment of the NEL Process for Using Existing Systematic Reviews, Meta-Analyses, and Reports” beginning on page 167).

The USDA Nutrition Evidence Library and Its Approach to Conducting Original Systematic Reviews for the DGAC

The NEL is staffed by nutrition scientists and research librarians with systematic review expertise; for both the 2010 and 2015 DGACs, NEL staff provided support for all original (de novo) systematic reviews.

The design of the NEL protocol for original systematic reviews is based on published methodologies from the Agency for Healthcare Research and Quality (AHRQ), Cochrane, the Academy of Nutrition and Dietetics, and the Institute of Medicine (IOM) (USDA/HHS, 2016). Standardized methodology and protocols are used for the purpose of promoting transparency, minimizing bias, and ensuring the development of high-quality systematic reviews. For original NEL systematic reviews, the review team is composed of the following:

• A DGAC subcommittee (four to seven members), for the purpose of providing expertise specific to the review topic and knowledge of the field;
• One or more NEL analysts, who assist the DGAC in planning, facilitating, conducting, and documenting the systematic review to ensure alignment with NEL methodology;
• One or more NEL librarians, who manage the development, implementation, refinement, and documentation of the search strategy; and
• NEL abstractors, individuals with advanced degrees in nutrition or a related field, who assist in data extraction and risk of bias assessment.

The size of the systematic review team varies based on the project’s needs, but at a minimum, one librarian and one analyst are assigned a lead role.

NEL librarians and analysts are required to be trained in systematic review methodology. New staff members are required to undergo approximately 150 hours of training over the course of several months before independently performing any of the steps in the systematic review process (USDA/HHS, 2016). NEL abstractors receive approximately 10 initial hours of training from the NEL staff and an email orientation to the specific systematic review project, as well as ongoing training as needed throughout the project. Prior to approval for participation, evidence abstractors are required to disclose potential financial, professional, and intellectual conflicts of interest (USDA/HHS, 2016).

The six steps in the NEL original review process are as follows:

1. Topic identification and systematic review question development;
2. Literature search, screening, and selection;

3. Data extraction and risk of bias assessment;
4. Evidence description and synthesis;
5. Conclusion statement development and evidence grading; and
6. Identification of research recommendations.

The DGAC makes all substantive decisions during each step, while NEL staff assists in executing and documenting those decisions and ensuring that the process adheres to established NEL methodology. Table 6-2 provides an overview of each step in the process, specifying the roles of the NEL staff, DGAC, and tools used. Even for the steps in Table 6-2 that specify the NEL as the primary actor, the DGAC still provides oversight and direction, and reviews and approves products.

Step 1: Topic Identification and Systematic Review Question Development

Topics are identified by the DGAC. During the topic identification process, the DGAC determines additional information about the topic, including the target population, public health outcomes of interest, and relevant references as appropriate.

The U.S. Department of Health and Human Services (HHS) and USDA suggest topic selection criteria, based on the scope and purpose of the DGAC. These criteria focus on the role of the DGA to inform public health action and policy in promoting population health and reducing the risk of disease (Millen, 2017). Also taken into consideration is the likelihood that the results of including the topic in an evidence review will: “(1) inform decisions about federal public health food and nutrition policies and programs, or (2) represent an area of major public health concern, uncertainty, and/or a knowledge gap that is critical to public health policy” (USDA/HHS, 2016). Both scope and importance are considered in selecting topics.

For each suggested topic, the rationale for review, target population, and public health outcomes of interest are outlined, and the approach for examining the evidence for the topic is recommended by the DGAC. These steps apply to all topics, regardless of the type of analysis used to examine the evidence (i.e., original NEL systematic review; existing high-quality systematic reviews, meta-analyses, and report(s); food pattern modeling; descriptive data analyses).

If requested by the DGAC, the NEL provides support at this initial stage of the evidence assessment by conducting exploratory searches. The goals of exploratory searches are to determine whether sufficient evidence is available to warrant a systematic review, to refine search terms and health outcomes of interest, and to provide information to estimate the

TABLE 6-2 Overview of NEL Systematic Review Steps

NOTE: AMSTAR = Assessing the Methodological Quality of Systematic Reviews; BAT = Bias Assessment Tool; DGAC = 2015 Dietary Guidelines Advisory Committee; N/A = not applicable; NEL = Nutrition Evidence Library; PICO = population, interventions/exposures, comparators, and intermediate and/or health or dietary outcomes of interest.

^a For the purposes of this table, this column notes only specialized tools. NEL protocol specifies the methods used at each of these steps, which are outlined in detail in the text.

^b If no existing systematic reviews, meta-analyses, or reports are identified in the literature search, this step is omitted.

time and resources needed for a systematic review on a specific topic. It is not to provide details on results or conclusions.

For topics that are selected to be addressed with systematic reviews—either original NEL systematic reviews, or existing high-quality reports when available—questions are developed by the DGAC with assistance from federal staff according to the PICO (population, intervention/exposure, comparator, and outcome of interest) framework. The PICO framework outlines the following elements of interest to be included in the question: target population and subpopulations, the intervention and/or exposure, the main comparator, and selected outcomes. Systematic review questions are reviewed and further refined in an iterative process, integrating input from the DGAC to ensure appropriate focus and specificity. The subsequent development of an analytic framework is intended to ensure that the final systematic review question(s) considered critical elements that may have affected the outcome. In addition to the PICO elements, the analytic framework for each systematic review includes key definitions, potential confounders, and a list of all systematic review questions for a particular topic, if more than one was asked. The analytic framework serves as a visual representation of the overall scope of the project and is available publicly during DGAC meetings, online after DGAC meetings, and once the systematic review is completed and the DGAC Scientific Report is released.

Step 2: Literature Search, Screening, and Selection

Search For each systematic review project, the lead NEL librarian, working in collaboration with the DGAC and NEL analyst(s), is responsible for developing a search strategy to identify relevant literature and for documenting the search terms, electronic databases searched, and appropriate search refinements.

First, DGAC subcommittees establish a priori inclusion and exclusion criteria for each systematic review based on a set of standard criteria developed by the NEL for the purposes of promoting consistency across systematic reviews and ensuring relevance to the U.S. population. These criteria can be revised by the DGAC subcommittee members if needed based on the topic of the systematic review to address any unique considerations. For example, questions examining the relationship between community food environments or food access and weight status are limited to only include U.S. populations, but for questions on relationships between dietary patterns and cancer, the population is expanded to include individuals from countries with a high or very high human development index (as defined according to the 2012 Human Development Index) (HHS/USDA, 2015b). To promote objectivity and minimize opportunity

for bias, any post hoc revisions to the criteria are discouraged by the NEL protocol, and if changes have to be made, the date and justification for the revision are documented in the search strategy.

The NEL inclusion and exclusion criteria cover study design, risk of bias, language, publication status, and health status of study subjects, along with the rationale for selections (USDA/HHS, 2016). Specifically, the standard criteria for DGAC systematic reviews include studies published in English in peer-reviewed journals in generally healthy populations, including populations with elevated chronic disease risk, or a mix of individuals with and without the disease or health outcome of interest. Studies are excluded if they were conducted in exclusively diseased populations or nongeneralizable subsets of the population, were unpublished or in the grey literature, or were published in languages other than English. Studies are not excluded based on a risk of bias assessment, although this is considered in later grading of the overall quality of the evidence (USDA/HHS, 2016). Study designs that are included and excluded may be dependent on the most appropriate design feasible for addressing a particular topic or question. However, standard NEL protocol states that randomized and nonrandomized controlled trials, prospective and retrospective cohort studies, case-control studies, and pre/post studies with a control are included, while cross-sectional studies, uncontrolled studies, pre/post studies without a control, and narrative reviews are excluded (USDA/HHS, 2016). Existing systematic reviews and meta-analyses are identified in a duplication assessment and may be used at the discretion of DGAC subcommittees to replace or augment an original NEL systematic review.

To test the search strategy and identify any potential errors, the NEL librarian performs a preliminary search in PubMed, using PubMed operators and search terms, and previews the results. The search strategy is then peer reviewed by another NEL librarian for the following elements:

1. “The accuracy of translating the research questions into search concepts and terminology;
2. Proper use of search operators, fields, limiters or filters, and spelling of syntax of search terms/strings;
3. The accuracy of adapting the search strategy for each database;
4. Inclusion of relevant subject headings such as Medical Subject Headings (MeSH) with free-text terms; and
5. Provision of additional relevant search terms and/or original databases” (USDA/HHS, 2016).

The NEL librarian makes any necessary revisions during the peer-review process, shares the search strategy with the DGAC subcommittee for

review, and subsequently finalizes the search strategy after all additional revisions noted by the DGAC subcommittee are made. PubMed, Embase, and Cochrane are the standard databases searched, but other topic-specific databases may be searched depending on the research question (USDA/HHS, 2016). All databases searched are listed in the search plan and results. The final search is conducted in the selected electronic databases.

Screening The results of the search are independently screened by two NEL analysts via title, abstract, and full-text review. The goal of screening is to review the search results and determine whether each article meets the inclusion and exclusion criteria. A third analyst or member of the DGAC is available to resolve conflicts between the two analysts.

Selection The resulting list of included and excluded articles is reviewed and approved by the DGAC subcommittee. A manual search of the reference sections of included articles is also performed to ensure all relevant articles are included, and to identify any potential gaps in the search.

Duplication assessment Depending on the topic and literature identified, NEL staff can conduct a duplication assessment to identify any existing high-quality systematic reviews, meta-analyses, or reports that answer the systematic review topic or question of interest. Existing systematic reviews, meta-analyses, and reports may be used to either replace an original systematic review or to supplement a systematic review as an additional source of evidence.

If an existing systematic review, meta-analysis, or report is identified during the search and screening process, the DGAC subcommittee is responsible for determining if and how it should be used, based on the report’s relevance to the systematic review question of interest, the quality of the report, the timeliness of the report, and with consideration for reference overlap. The assessment is based on PICO elements, AMSTAR rating, and the date range of the existing systematic review (see “Assessment of the NEL Process for Using Existing Systematic Reviews, Meta-Analyses, and Reports” beginning on page 167 for more information). If multiple existing reports are identified on the same topic and the conclusions are similar, the NEL can combine them in consideration of overall evidence; if conclusions differ, they can be used for background information, but are not deemed an appropriate source of evidence. If no existing high-quality reports are identified, the NEL proceeds with an original systematic review on the identified topic and questions.

Step 3: Data Abstraction and Risk of Bias Assessment

In preparation for the review and summary of the evidence, data relevant to the systematic review question are extracted and risk of bias is assessed for each article included in a systematic review. A standardized evidence extraction form is developed by the NEL analyst and approved by the DGAC subcommittee to ensure all relevant data are collected. These forms are organized generally by study characteristics, participant characteristics, exposure(s)/independent variable(s), outcome(s)/dependent variable(s), limitations/risks of bias, funding, and related articles. Specific instructions are also provided to ensure all relevant information is collected (e.g., for Dietary Assessment Method, example instructions may specify: “enter name and/or type of instrument used and a brief description of tool, if it was validated for the study sample, number of data collection points, and which data points were used for diet assessment”) (USDA/HHS, 2016). Data extraction can be done with assistance from NEL abstractors.

After completing the data extraction, risk of bias (internal validity) in individual studies is assessed using the NEL Bias Assessment Tool (BAT). The NEL BAT was developed to assess the risk of bias of individual studies included in NEL systematic reviews, and is based on existing risk of bias tools, including those developed by the Agency for Healthcare Research and Quality and Cochrane, and follows a question/answer format (Higgins and Green, 2011; USDA/HHS, 2016; Viswanathan et al., 2012, 2013; West et al., 2002). The tool is designed to assess four types of bias, including

1. Selection bias, through assessment of inclusion/exclusion criteria, recruitment, allocation of participants, and baseline distribution of confounders;
2. Performance bias, through assessment of adherence to study protocol by the participants and investigators, unplanned concurrent exposures, and blinding of the participants and investigators;
3. Detection bias, through assessment of blinding of the outcome assessors, outcome measures, and statistical methods; and
4. Attrition bias, through assessment of follow-up length and attrition (high/differential).

Each of these assessments is facilitated by a set of targeted questions specific to randomized controlled trials, nonrandomized controlled trials, or observational studies. For example, to assess blinding of participants on randomized or nonrandomized controlled trials, the NEL BAT question is “Were participants blinded to their intervention or exposure status?” Each question can be answered with one of four responses: yes,

no, cannot determine, and not applicable. NEL protocol specifies that “yes” or “no” responses should be selected if sufficient information is provided in the study to clearly indicate the answer to the question; if no or insufficient information is available in the study, “cannot determine” should be selected; and “not applicable” should be selected if the question is not applicable to the study. For quality control purposes, the NEL BAT is completed in a dual process where both the evidence abstractor and an NEL analyst independently complete the bias assessment. Any conflicts identified are to be resolved by the abstractor and analyst, with assistance from another NEL staff member, if needed (USDA/HHS, 2016).

The analyst combines the extracted data and limitations identified via the NEL BAT into a spreadsheet referred to as the evidence grid to assist the DGAC subcommittee’s review of the evidence (USDA/HHS, 2016).

Step 4: Evidence Description and Synthesis

The DGAC subcommittee reviews the evidence grid and the full-text manuscripts of the articles identified in the search. This review is facilitated by a series of probing questions provided by the NEL analyst, referred to as the evidence portfolio worksheet, which are independently completed by each DGAC subcommittee member. These questions vary in their focus and aim and are intended to aid the DGAC members in comparing and contrasting the studies reviewed, and to assist with subsequent development of a conclusion statement along with a grade of the overall quality of the evidence. Some of the questions are intended to evaluate the characteristics affecting the quality of the study and potential considerations, and include the following:

• “Whether the reported effects of a study are likely to be the true effects of the intervention/exposure,
• Whether the sample size of a study is sufficient to avoid type I and II errors,
• Whether a study is designed to directly examine the link between the intervention/exposure and the outcome(s) of interest in the systematic review question, and
• Whether a study is generalizable to the U.S. population of interest” (USDA/HHS, 2016).

Other questions focus on the specific intervention/exposure and outcome(s) of interest for the systematic review. Study limitations, consistency of results, methodological differences resulting in disagreement in outcomes, significance of results, and reliability across multiple independent research groups are also noted.

The NEL analyst compiles information from the DGAC subcommittee’s review of the evidence portfolio to facilitate drafting of the evidence description and synthesis by a DGAC member(s) or the analyst, which includes descriptive information about the review and a summary of findings. The draft synthesis of evidence is reviewed by the DGAC subcommittee; a minimum of three subcommittee members are required to provide feedback before the synthesis can move forward (USDA/HHS, 2016). The synthesis of evidence generally compares and contrasts the interventions/exposures and outcome(s) of interest, methodology, and results. Also included in the final evidence synthesis is a discussion of the themes of the systematic review question, an overview table providing a summary of the results and key study characteristics, an assessment of the body of evidence according to the aspects outlined in the NEL grading rubric, and the resulting research recommendations and rationale.

Step 5: Conclusion Statement Development and Evidence Grading

The collection, description, and synthesis of evidence are subsequently used by the DGAC subcommittee to develop a conclusion statement in response to the systematic review question. Conclusion statements are written in a clear and concise manner and include relevant information important for consideration, including a statement acknowledging general agreement among the studies on which the conclusion was based, and/or an explanation of any areas of disagreement. Per NEL protocol, conclusion statements are not to address areas outside of the body of evidence reviewed and are not intended to express implications. Drafting conclusion statements, as with the evidence description and synthesis, is an iterative process involving both DGAC subcommittee and NEL staff. The DGAC subcommittee’s role is to ensure appropriate interpretation and communication of evidence, while the NEL staff’s role is to review draft conclusions to ensure they met the protocol. If discussions throughout the evidence synthesis and drafting of conclusion statements necessitates clarifications or changes to the evidence portfolio, these are made by the NEL staff as appropriate.

Each conclusion statement is accompanied by a grade of the strength of the evidence supporting the conclusion; the grade is not applicable to individual studies. The grade is determined by the DGAC subcommittee according to specific criteria laid out by the NEL and based on five elements: internal validity, adequacy, consistency, impact, and generalizability.

1. Internal validity refers to the “likelihood that the reported effects are the true effects of the intervention/exposure and not over- or

2. underestimates resulting from bias due to study design or conduct” and is based on information gathered in completing the NEL BAT (USDA/HHS, 2016).

3. Adequacy of the evidence is determined based on the number of “studies overall, studies by independent research groups, studies with sample sizes that are sufficient to avoid type I and II errors, and participants overall” (USDA/HHS, 2016).
4. Consistency of the evidence is judged on three elements of the findings: “(1) direction, (2) size of effect/degree of association, and (3) statistical significance” (USDA/HHS, 2016).
5. Impact of the evidence is determined by: “(1) the directness with which the study designs examine the link between the intervention/exposure and outcome of interest in the systematic review question, (2) the statistical significance, and (3) the practical/clinical significance” (USDA/HHS, 2016).
6. Generalizability of the evidence to the U.S. population is considered with regard to the study samples and the intervention/exposure and outcomes studied.

For each of these five elements, the DGAC determines whether the overall body of evidence in each area is strong, moderate, or limited, or whether a grade is not assignable. Each DGAC subcommittee member evaluates the body of evidence and assigns a grade for each of those elements independently, and then differences are noted and discussed among the DGAC subcommittee members. Through discussion, the DGAC subcommittee arrives at a grade for the conclusion statement, reflective of its evaluation of the overall body of evidence as outlined in the NEL grading rubric. The grades used for conclusion statements also fall into one of these four categories: strong, moderate, limited, and grade not assignable (see Table 6-3). Draft and final conclusion statements and grades are presented at public meetings.

Step 6: Identification of Research Recommendations

Research recommendations are initially developed and drafted during the evidence description and synthesis step to reflect gaps and/or limitations in the body of evidence, but can be revised and updated to reflect the continued discussions concerning conclusions and grading of evidence before being finalized. Emerging topics in particular can be included in the DGAC Scientific Report with a rationale describing the need for additional research.

TABLE 6-3 Description of Grades for Conclusion Statements Used by the USDA Nutrition Evidence Library

SOURCE: USDA/HHS, 2016.

Availability and Accessibility of Systematic Reviews

Public availability of original systematic reviews is part of the NEL protocol. For example, the NEL systematic reviews are documented in their entirety and, following the completion of the review and the publication of the DGAC Scientific Report, are posted on the NEL website (NEL. gov) to promote transparency, accessibility, and reproducibility. Systematic reviews conducted for the DGAC are discussed at DGAC public meetings. Throughout the 2015 DGAC’s deliberations, public comments were accepted, and comments about the systematic reviews under way were welcomed and reviewed by the 2015 DGAC. The completed evidence portfolio that is posted online following the publication of the DGAC Scientific Report includes

1. A conclusion statement;
2. A grade of the overall quality of evidence;
3. Key findings;
4. Research recommendations;

5. Evidence summaries giving the description and synthesis of the evidence along with the risk of bias assessment, references, and research recommendations;
6. An analytic framework, including the systematic review question(s); and
7. Search plan and results, including search parameters, selection criteria, and the final list of included and excluded articles, with brief explanations of reasons for exclusion (USDA/HHS, 2016).

The NEL staff drafts a technical abstract for each systematic review, which is posted on the NEL website along with the details of the full systematic review. The technical abstract is designed to provide key details of the systematic review in an easily accessible and standard format, similar in nature to a technical abstract prepared for peer-reviewed publications or scientific meetings, but longer and including more detail. Technical abstracts are reviewed by the DGAC subcommittee members before posting. Each technical abstract is titled with the systematic review question it describes, and includes five sections: background, conclusion statement, methods, findings, and discussion. Within these sections, key details of the systematic review are described, including the rationale and objective, data sources, study eligibility criteria, participants, interventions, study appraisal, synthesis methods, results of the systematic review and appraisal of the body of evidence along with the grade of the conclusion statement, and limitations and implications of key findings (USDA/HHS, 2016).

Approach to Non-DGAC Systematic Reviews

The NEL was created to support the DGAC, as well as conduct nutrition-related systematic reviews for federal partner agencies, such as those within USDA and HHS. As a result, the NEL has two separate protocols for conducting systematic reviews: one for DGAC-requested systematic reviews, and one for non-DGAC systematic reviews.⁶ The two protocols have many similarities and use the same steps, but there are key differences.

The fundamental difference between the two protocols is that for DGAC-requested systematic reviews, the DGAC is the approver and “authors” the systematic review (USDA/HHS, 2017). In the protocol for non-DGAC systematic reviews, the NEL authors the work and relies on a technical expert collaborative to provide domain expertise. The technical

___________________

⁶ In this discussion, details of the steps for the non-DGAC systematic review protocol were derived from two reports (USDA, 2012, 2014).

expert collaborative reviews key decisions and provides technical advice as needed (USDA, 2012, 2014). This difference drives much of the variation at the procedural level (e.g., key decisions made by the DGAC are instead made by the NEL).

Other key differences between the two protocols include the tools used. In Step 3, data extraction and risk of bias assessment, the protocol for DGAC systematic reviews employs the NEL BAT to evaluate bias (USDA/HHS, 2016). The non-DGAC systematic review protocol uses the Academy of Nutrition and Dietetics Research Design and Implementation Checklist to assess methodological rigor⁷ (USDA, 2012, 2014). To conduct Step 5, developing conclusion statements and grading the evidence, different sets of criteria are used to evaluate strength of the body of evidence. The DGAC systematic review protocol employs the criteria of internal validity, adequacy, consistency, impact, and generalizability (USDA/HHS, 2016). The non-DGAC systematic review protocol, however, uses criteria adapted and validated by the Academy of Nutrition and Dietetics: quality, quantity, consistency, generalizability, and public health impact (USDA, 2012, 2014).

Evaluation of the NEL Original Systematic Review Process

Several organizations have developed guidance on conducting systematic reviews, including AHRQ, Cochrane, and the Institute of Medicine, which were all cited in the development of the NEL protocol (USDA/HHS, 2016). To assess the NEL process, the systematic review process from the 2015 DGAC Scientific Report was outlined by this National Academies committee according to systematic review steps adapted from the report Finding What Works in Health Care: Standards for Systematic Reviews (IOM, 2011). These steps, as well as the roles of the NEL and DGAC in the process, are described in detail in Table 6-4. This National Academies committee reviewed each step in the systematic review process. Although the standards presented in Table 6-4 are aspirational and will likely not all be met in every systematic review, they do highlight several opportunities for improvement in the NEL de novo systematic review process, as discussed in the next section.

Findings

Original systematic reviews can help the DGAC answer questions regarding the relationship between diet and health if a synthesis of the

___________________

⁷ The Academy of Nutrition and Dietetics’ Research Design and Implementation Checklist uses the quality ratings of positive, neutral, or negative.

TABLE 6-4 Description of the Roles of the NEL and DGAC in the 2015 NEL Process Related to Conducting Systematic Reviews

NOTES: This table describes the NEL systematic review protocol as based on the 2015 DGAC process. The numbered systematic review steps are adapted from the Institute of Medicine report Finding What Works in Health Care: Standards for Systematic Reviews. The systematic review team was considered to include NEL staff, abstractors, and the DGAC members. DGAC members were not considered to be users/stakeholders. DGAC = 2015 Dietary Guidelines Advisory Committee; N/A = not applicable; NEL = Nutrition Evidence Library; OGE = Office of Government Ethics; PICO = population, intervention/exposure, comparator, and outcome of interest; USDA = U.S. Department of Agriculture.

^a The NEL has established standard processes (e.g., inclusion and exclusion criteria) to help promote consistency across NEL reviews and to ensure that the evidence being considered in each systematic review is applicable to the U.S. population and relevant to public health- and nutrition-oriented policies and programs. These standard processes have been reviewed and informed by federal stakeholders to ensure policy relevance.

^b This standard was adapted to apply to the NEL process.

SOURCES: DGAC, 2014; HHS/USDA, 2013; IOM, 2011; USDA/HHS, 2016.

evidence does not already exist. The utility of original systematic reviews depends on the availability of high-quality studies that are implemented appropriately to ensure impartiality of the reviews. The DGAC protocol could be better structured to support independence of NEL de novo systematic reviews.

The roles of the NEL staff and the DGAC are not clearly delineated and overlap at many steps of the systematic review protocol, potentially limiting the objectivity of results. For example, in evaluating the body of evidence, the DGAC’s process for synthesizing evidence and drafting conclusion statements appears to be facilitated by the NEL staff (i.e., NEL staff develops probing questions for the DGAC’s review of the evidence as well as compiles information received from the DGAC), but the statements are both “drafted” by the DGAC and then “reviewed” by the DGAC. The position of the DGAC as the driver of each step in the systematic review process, from the designing of the search strategy to the grading of the body of evidence, does not consistently promote an independent process. This National Academies committee’s evaluation

also identified the challenge of combining a systematic review process with the process for developing DGA recommendations, as best practices have shown that development of guidelines generally requires a more thorough separation of steps. While still allowing for the necessary iterations and communication between DGAC and NEL staff, this National Academies committee believes there is an opportunity to limit the overlap in roles and ensure the necessary expertise is included appropriately in each step by redesigning the process to clearly delineate the roles of both the DGAC and the NEL staff (see Chapter 4).

Because systematic reviews synthesize the evidence presented in individual studies, it is critical that the primary studies are of high quality. Nutrition studies present several methodological challenges, one in particular being self-reported dietary intake data. Because methods for acquiring dietary intake data vary and have the potential to introduce bias into the systematic review outcomes, they should be taken into account in the development of inclusion/exclusion criteria and appropriately managed in analyses whenever possible (see Box 4-4 for a discussion on using self-reported dietary intake data).

Throughout the entire process of conducting systematic reviews, it is unclear how and with what frequency NEL methods are updated. For example, since the NEL BAT was developed, other organizations have made several improvements in assessing the risk of bias in systematic reviews (Higgins and Green, 2011; Viswanathan et al., 2012). However, these updates have not yet been reflected in the NEL BAT. Additionally, while NEL de novo systematic reviews are publicly available, they are not peer reviewed. Maintaining up-to-date methods for conducting systematic reviews in a rapidly evolving field depends on collaboration with outside organizations and implementing ongoing training in best practices.

In addition, appropriately interpreting the results of systematic reviews, and subsequently integrating these results with other analyses, is an important element in developing conclusions. The interpretation of results and integration of analyses are subjective and require careful consideration. Whereas many steps are in place earlier in the NEL systematic review process to help identify potential limitations in the data available (e.g., inclusion/exclusion criteria, risk of bias assessment), it is unclear how these limitations are taken into account in the interpretation of results.

Conclusion

Overall, the NEL process for conducting original systematic reviews is thorough and adheres to several of the existing systematic review standards in the field. However, the overall protocol needs to be strengthened

to improve the efficiency of the NEL process and minimize the introduction of bias. Clear delineation of the roles of individuals and groups involved at various steps in the NEL process are key to developing appropriate conclusions. The NEL ought to use the most appropriate, validated, and standardized methods whenever possible. Ensuring up-to-date methods are adopted and implemented in the NEL process depends on engaging in ongoing training and collaboration efforts with other organizations conducting systematic reviews, and could increase the usefulness of the NEL. Systematic reviews including observational studies in particular will need to be carefully evaluated in the interpretation of results and development of conclusions.

ASSESSMENT OF THE NEL PROCESS FOR UPDATING SYSTEMATIC REVIEWS

Because the NEL has only been used to support two editions of the DGAC Scientific Report, the only opportunity to update systematic reviews conducted previously by the NEL was in the 2015 DGAC. In one instance, the update combined two questions into one and expanded the terminology around the exposure of interest in the search to broaden the scope of the systematic review, while keeping the target population and outcomes the same. In the other instances, the same systematic review was repeated with updated search dates for the purpose of capturing articles published after the original systematic review was conducted. Updates of NEL systematic reviews were conducted and documented according to NEL de novo systematic review methods (HHS/USDA, 2015b). Methods for conducting an update to an existing systematic review not original to the NEL will be discussed in the assessment of existing systematic reviews, meta-analyses, and reports.

Findings

Systematic reviews can be updated to take into account new evidence since the last review was conducted. The NEL process for updating systematic reviews reflects the process for original systematic reviews, and the findings identified for conducting original systematic reviews apply also to the process for updating systematic reviews.

In the 2015 DGAC Scientific Report, a clear explanation of why a decision was made to update or not update a systematic review was not publicly available. Updating systematic reviews can be done for a number of reasons and takes several forms (AHRQ, 2014; Garner et al., 2016; Higgins and Green, 2011). In some cases, such as abstracting new data or significantly adjusting the methods used, updating a systematic review

can require more resources than conducting a de novo review. Reasons to update a systematic review could include any one or multiple of the following scenarios:

1. The systematic review question to be answered and the methods to be used remain unchanged, but the review does not include recently published studies on the topic. This gap requires adding recently published studies. This approach also assumes that either the results or the abstracted data from previous systematic reviews are available so that quantitative analyses could be performed, if needed.
2. If the systematic review question is changed (different from an existing systematic review), it may require abstracting new data from publications used in previous systematic reviews. An outcome of interest or metric may also change (e.g., measurement of dietary intake). In this case, a new search may or may not be needed.
3. Methods used in systematic reviews also evolve (e.g., grading of evidence, the use of different types of self-reported dietary data); adhering to the latest standards in performing an update sometimes requires using data abstracted from primary publications in a previous systematic review.

Conclusion

It was not clear why the 2015 DGAC chose to update some systematic reviews and not others. Updates of NEL systematic reviews generally ought to be conducted on a needs-based approach and in accordance with the NEL systematic review protocol for de novo systematic reviews. Regardless of the reason to update a systematic review, an update needs to consider all relevant research. Updating a systematic review may require collecting additional data or performing new analyses. As a result, newly published studies may be added or previously included studies may be excluded based on refined methods. This includes previously appraised research, because advances in knowledge about mechanisms of action, interactions of nutrients, or other factors that affect the outcome may reflect new understandings and could be integrated into a new conclusion statement.

Because new information and publications often drive updates, ongoing surveillance of literature is necessary to keep systematic reviews up to date and minimize duplication of efforts. Ongoing surveillance can also identify existing systematic reviews that may replace the need to conduct an update of a systematic review.

ASSESSMENT OF THE NEL PROCESS FOR USING EXISTING SYSTEMATIC REVIEWS, META-ANALYSES, AND REPORTS

Many groups around the world are now conducting systematic reviews, often on the same topic. Because systematic reviews require significant amounts of time, expertise, and costs to conduct, a search should be made to identify existing and ongoing systematic reviews before a new systematic review is undertaken. With limited resources, it would be advantageous to leverage existing systematic reviews, meta-analyses, and reports to minimize unnecessary replication of efforts and to share results with others.

However, existing systematic reviews may not fully address the question or population of interest, they may be outdated, or they may not meet current methodological standards. Furthermore, the quality of the systematic reviews—and hence their reliability—may vary. These concerns can make use of existing systematic reviews challenging but do not necessarily invalidate these systematic reviews. The concerns must be carefully analyzed and the challenges in using them understood. If a decision is made to proceed with using existing systematic reviews, documenting the rationale and explaining how any challenges are mitigated provides transparency. Documenting the reasons that existing systematic reviews have been assessed but not included will assist in reconciling potential differences in the results across different systematic reviews.

DGAC Approach to Using Existing Systematic Reviews, Meta-Analyses, and Reports

In addition to conducting original systematic reviews, the DGAC has used existing systematic reviews, meta-analyses, and reports to answer its questions. The 2015 DGAC was the first committee to develop and document a standardized process and criteria for including existing systematic reviews. The process paralleled several of the steps in the de novo NEL systematic review process.

Identifying Existing Systematic Reviews, Meta-Analyses, and Reports

As systematic review questions were developed and prioritized, the DGAC collaborated with the NEL to develop an analytic framework. At this point, before the literature search and screening begins, existing systematic reviews, meta-analyses, and reports may have been identified a priori by DGAC subcommittee members aware of current literature. The 2015 DGAC also requested literature and references on specific topics through public comments. If a report from an authoritative source was identified and met the criteria for inclusion, a literature search was still

conducted according to NEL protocol to identify any additional reports on the topic. Existing reports may also have been identified during a duplication assessment in the early stages of the literature screening and selection process in preparation for a de novo systematic review (USDA/HHS, 2016). In all cases, the existing reports were required to meet the criteria for inclusion.

Criteria for Inclusion

Existing systematic reviews, meta-analyses, and reports were assessed by the NEL and the DGAC to determine if they met the predetermined inclusion/exclusion criteria. In some cases, federal DGAC support staff other than the NEL supported the DGAC in its review of existing systematic reviews, meta-analyses, and reports. The DGAC based its determination on four criteria: (1) the relevance to the systematic review question of interest, (2) the quality of the report, (3) the timeliness, and (4) reference overlap, if multiple systematic reviews were identified.

The relevance to the systematic review question was determined through comparison of the existing report to the established scope of the question outlined in the analytic framework, including PICO stipulations. Inclusion and exclusion criteria used in the existing report were also compared to those outlined for the question to judge relevance. For the 2015 DGAC, the methodological quality of the existing report was evaluated based on the AMSTAR tool, which considers 11 areas of methodological quality elements to assess.⁸ In the AMSTAR tool, a systematic review receives 1 “point” for each item appropriately fulfilled, with a total score of 11 possible. To meet the inclusion criteria set by the DGAC in 2015, systematic reviews and/or meta-analyses must have scored 8 or higher (USDA/HHS, 2016). Timeliness of the systematic review was based on whether the date range set in the inclusion/exclusion criteria for the existing systematic review matched the date range set in the search strategy for the systematic review question of interest. In cases where multiple existing reports were identified, the references lists were examined for overlap. If individual studies overlap between system-

___________________

⁸ These 11 criteria are (1) a priori research design established, (2) study selection and data extraction completed by two independent reviewers, (3) comprehensive review of literature conducted, (4) status of publication defined in inclusion criteria, (5) list of included and excluded studies provided, (6) characteristics of included studies provided, (7) scientific quality of included studies assessed and documented, (8) scientific quality of included studies considered in analysis and conclusions drawn, (9) appropriate methods applied for combining findings of studies, (10) assessment of the likelihood of publication bias included, and (11) conflict of interest in included studies and systematic review noted (Shea et al., 2007).

atic reviews, care was taken to ensure that the individual studies were not included multiple times, to reduce the potential for overestimating results (USDA/HHS, 2016).

Evidence Summary and Synthesis

If eligible for inclusion, a summary and synthesis of the evidence from existing reports was developed by federal DGAC support staff. In some cases, targeted questions may have been prepared by the federal staff to facilitate the DGAC members’ identification of themes and key findings from the systematic reviews. The review of evidence specific to each systematic review question was outlined in the 2015 DGAC Scientific Report (at the same level of detail as with original systematic reviews, including a conclusion statement and grade, implication statement, and summary of the review of evidence), and more detailed evidence descriptions were provided in appendixes to the 2015 DGAC Scientific Report (HHS/USDA, 2015b). Although the information provided in the 2015 DGAC Scientific Report varied slightly in the presentation and type of information for a particular systematic review question, at minimum, the report included an evidence portfolio with a summary table of included studies. For some systematic review questions, additional information such as the search strategy and analytic framework were included. Excluded studies, with briefly stated reasons for exclusion, were also provided as either a complete reference list of excluded studies or the number of excluded studies.

Historically, the DGAC has also considered existing authoritative reports published by federal agencies or leading scientific organizations in its evidence review. For these questions, an evidence portfolio was not provided, because the conclusions were drawn directly from published reports. For example, several questions in the 2015 DGAC Scientific Report intended to address evidence on physical activity and health outcomes were based on conclusions from the Physical Activity Guidelines Advisory Report, 2008, and associated publications (HHS/USDA, 2015b).

Findings

Use of existing systematic reviews, meta-analyses, and reports may be beneficial, considering the significant time and resources needed to conduct original systematic reviews. Using existing systematic reviews also serves to limit the duplication of efforts across groups conducting systematic reviews. However, inclusion of existing systematic reviews, meta-analyses, and reports depends on their quality and relevance in relation to the specific topic and question that is being considered. As is the case

for de novo systematic reviews, it is critical that the individual studies included in existing systematic reviews, meta-analyses, and reports are of high quality and adhere to standard methods.

Currently, the DGAC’s criteria note that systematic reviews must achieve an AMSTAR score of 8 or higher to be included; however, limitations to the AMSTAR tool have been identified (Burda et al., 2016; Faggion, 2015; Wegewitz et al., 2016), and this measure alone is not sufficient to determine the quality of a systematic review. As methods continue to advance, the DGAC criteria will need to adjust accordingly.

Within the DGAC’s stated criteria of inclusion, there are additional considerations, some of which are inherent to comparing reports, and others are specific challenges that may result in inabilities to use the existing systematic reviews as is. The assessment of published systematic reviews is based on reported information, which can vary widely across systematic reviews. A common challenge in using existing systematic reviews and relying on reported information is not having the necessary information to allow independent verification of the validity of the analyses and conclusions. Without the abilities to verify, the user of an existing systematic review has to trust the veracity of the reported information. Alternatively, the user may decide to use the existing systematic review as a framework and abstract only sufficient information from the original studies to carry out the necessary independent assessment.

Registration of the systematic review protocol on the PROSPERO website may provide additional information to assess whether the systematic review adhered to its original intent and methods. Archiving of data in websites such as the Systematic Review Data Repository provide opportunities for readers to assess the comprehensiveness and accuracy of the abstracted data used in systematic reviews. Although resources such as the Systematic Review Data Repository could increase transparency within the NEL process, their use is not required across organizations conducting systematic reviews, and many published systematic reviews may not have done so.

A more challenging problem occurs if disagreements in results and conclusions occur among multiple systematic reviews. The reason for discrepancy may sometimes be apparent, such as the obvious differences in the eligibility criteria or large differences in publication date and hence studies included. Discrepancies caused by subtle differences in the eligibility criteria or how such criteria were operationalized may be difficult to ascertain.

Missing data in the original systematic review may lead to an inability to include the systematic review, or the systematic review team may need to abstract additional information not reported by the original team that conducted the systematic review. It is difficult to know how the other

team operationalized eligibility criteria, even if the written criteria appear the same. Different methods used to assess the limitations of primary studies and grade the strength of evidence can also present challenges.

In some cases, the NEL staff, in determining inclusion of an existing systematic review, may be able to simply perform a new literature search to bring the existing review up to date, if they determined that the data abstraction and analyses performed by the original authors were accurate, and their interpretations were correct. In these cases, the NEL and DGAC would need to ensure data abstraction and interpretation were harmonized with the NEL protocol. Even if an existing systematic review is found not to be completely suitable because of the nature of the question(s) asked, the list of studies identified may still be helpful in conducting a new systematic review.

Conclusion

In summary, use of existing systematic reviews, meta-analyses, and authoritative reports from leading organizations is generally appropriate and encouraged by this National Academies committee, with the understanding that they ought to be relevant, timely, and of high quality. Efficiency and use of time and resources must be weighed carefully in using an existing systematic review compared to conducting a de novo review (Smith et al., 2011; Whitlock et al., 2008). However, opportunities exist to strengthen the current method of identifying existing systematic reviews, meta-analyses, and reports. Ongoing surveillance of the literature can serve to identify existing systematic reviews while maximizing resources. Surveillance can also help identify authoritative reports for use by the DGAC. There are also opportunities to leverage the Systematic Review Data Repository at AHRQ to further enhance the value and usefulness of the NEL to the nutrition research community. All systematic reviews and reports ought to meet the criteria for inclusion specified in the NEL protocol, with consideration for appropriate methods in cases of missing and unreported data.

EVALUATION OF METHODOLOGY AND USE OF FOOD PATTERN MODELING

Since the 1990 DGA Policy Report, specific food-intake guidance has been provided to help the public meet nutrient needs while moderating intake of other dietary constituents. Such guidance has been presented as a single food guide or multiple eating patterns (the USDA food patterns were substantially revised and formally described beginning in the 2005 DGAC Scientific Report), but the intention has remained the same: translate

nutritional recommendations into food intake recommendations that take account of the totality of the diet. In each instance, USDA has conducted food pattern modeling to derive this guidance for the DGAC.

Questions Food Pattern Modeling Is Intended to Address

Food pattern modeling, which assesses the nutrient content of various possible eating patterns based on typical choices within food groups, can be used to address a range of specific questions (see Appendix C). But the overarching question it seeks to answer is, “How well do varying combinations and amounts of food groups meet the Dietary Reference Intakes and potential recommendations in the DGA?” (Britten et al., 2006a,b). This is an important issue, given the myriad nutritional profiles of basic foods and the complex array of constraints involved in achieving nutritional adequacy while moderating consumption of energy and other dietary constituents. In effect, food pattern modeling shows how diets could be developed to meet those constraints. Three different patterns developed by USDA were featured in the 2015–2020 DGA Policy Report—“Healthy US-Style,” “Healthy Mediterranean-Style,” and “Healthy Vegetarian”—as “examples of healthy eating patterns that can be adapted based on cultural and personal preferences” (HHS/USDA, 2015a). The Dietary Approaches to Stop Hypertension (DASH) dietary pattern is also mentioned in the 2015–2020 DGA Policy Report as an “example of a healthy eating pattern . . . [with] . . . many of the same characteristics as the Healthy US-Style Eating Pattern” (HHS/USDA, 2015a). DASH was not derived via food pattern modeling; it was developed for a randomized controlled clinical trial to study the effect of that diet on cardiovascular risk factors.

Current Methods Used to Derive Evidence: Steps in Process

Food pattern modeling that both informs and reflects the DGA recommendations has been an iterative process, at times developed concurrently with the DGA, with input from both the DGAC and federal staff (Britten et al., 2006a). For the 2015 DGAC, USDA staff from the Center for Nutrition Policy and Promotion were designated by their leadership to work on food pattern modeling. They worked extensively with the DGAC in addressing possible modifications to the patterns through their support of DGAC committees. However, the DGAC and federal staff had unique roles in the process (see Figure 6-1).

The methods USDA employed to conduct the food pattern modeling required inherent assumptions in addressing the questions in Appendix C (Britten et al., 2006b). The first was that each specific question referred to the “total” rather than a “foundation” diet. Unlike other approaches to food

guidance that focus only on major nutrient-bearing food groups (i.e., the foundation), the food pattern modeling accounted for the totality of the diet (i.e., the entire energy allotment). The second assumption was that nutrient values associated with each pattern were based on typical food choices, but in their most nutrient-dense form. That is, by design, the nutrient profile associated with each food group corresponded to average nutrient values associated with a population-weighted mix of pure foods in that group.

The exact procedure involved in food pattern modeling depends on the specific question being addressed, but the general process is outlined in Figure 6-1 and described below.

Step 1: Establish Energy Levels

The first step in formulating the food patterns is establishing the set of energy levels for which discrete patterns will be developed. Energy requirements vary widely by gender, age, height, weight, and activity level (IOM, 2000), and the patterns that result from the modeling must cover the needs of nearly everyone in the population. Consequently, establishing the appropriate range of energy levels is key to the exercise. Using the IOM’s formulas for calculating Estimated Energy Requirements (EERs) (IOM, 2005) for sedentary individuals, USDA determined that 1,000 to 3,200 kcal is an appropriate range. EERs represent the average energy intake predicted to maintain energy balance. USDA divided that range into 200-kcal increments, identified age/gender groups associated with each energy level, and then developed distinct patterns for each (Britten et al., 2006b).

Step 2: Establish Nutrient Goals

The second step in creating the patterns is to set the goals for a range of micro- and macronutrients at each energy level, based on the age/gender group(s) corresponding to that level. Goals for nutritional adequacy in the food patterns are based on the IOM’s Recommended Dietary Allowances (RDAs) or Adequate Intakes (AIs), rather than the Estimated Average Requirements (EARs), because the patterns are intended to serve as guides for individuals in planning their intakes (see Appendix E). Patterns can exceed the RDA or the AI for some nutrients to meet the goals for others if they do not exceed the Tolerable Upper Intake Level (UL). For example, the goal for total fiber is set at 14 gm per 1,000 kcal, the formula used to set the AI. Moderation goals for nutrients that tend to be overconsumed are set at the Acceptable Macronutrient Distribution Range (AMDR) for the macronutrients and at less than the UL for sodium.

Step 3: Establish Food Groups

The next step is establishing the food groups, which form the “building blocks” of the pattern. Although the names have varied, five major food groups have formed the basis of USDA’s food guidance for the past several decades: fruits, vegetables, grains, protein foods, and dairy. In addition, subgroups within several of these groups are constructed to target specific choices. For example, the grains group is subdivided into whole and refined grains, to encourage more frequent consumption of whole grains. In effect, dividing the group allows the modeling to change one subgroup more, less, or in a different direction than another, as needed to reach nutritional goals. The current list of food groups and subgroups in USDA’s Healthy US-Style Eating Pattern and Healthy Mediterranean-Style Pattern is shown in Box 6-1; more discrete subgroups in the protein foods group (eggs, legumes, soy products, nuts, and seeds) replace meat, poultry, and eggs in the Healthy Vegetarian Eating Pattern.

Step 4: Develop Food Groups Composites and Nutrient Profiles

Once the food groups have been established, modelers identified the nutrient contributions from standard amounts of each to estimate the nutrient totals associated with varying quantities of the food groups and subgroups. A nutrient profile for each group and subgroup was determined by developing a weighted composite of representative foods from each group. These representative foods, and the weight each one receives in the composite, were determined based on national surveys. For example, potatoes made up most of the starchy vegetables composite because they represented the largest share of all starchy vegetables consumed; corn and peas were also included, but in lesser amounts. However, there was one important qualification: all candidate foods were in their leanest form, prepared without the addition of fat, oil, or sugar (Marcoe et al., 2006). For example, the dairy group was represented by fat-free fluid milk. This means, for example, if someone chooses low-fat vanilla yogurt instead of fat-free, any energy ascribed to the fat and added sugars must come from the relatively small “calories for other uses” allowance.

Step 5: Model Inputs and Constraints

The last step in deriving the proposed patterns—modeling the inputs and constraints to establish the amounts from each food group to achieve nutrient targets—is done iteratively to determine the best fit. The resulting patterns can also be compared to estimates of usual intakes of various foods from a recent National Health and Nutrition Examination Survey or other population surveys (see “Descriptive Data Analyses: Evaluation of Methodology and Use” beginning on page 180).

Review of Food Pattern Modeling

The methods and results of the food pattern modeling have been described in detail and peer reviewed via a number of manuscripts in the nutrition literature (Britten et al., 2006a,b; Marcoe et al., 2006). However, there is a difference between the kind of review a manuscript receives describing something that has already been completed and an invited review by outside experts to critique and advance a methodological process. Food pattern modeling could benefit from such a review.

Role of the DGAC in Food Pattern Modeling

The other steps shown in Figure 6-1 are initiated by the DGAC (i.e., review findings from systematic reviews along with proposed eating patterns and suggesting new goals for some nutrients and other options

for modifying patterns), but USDA works in close collaboration with the DGAC to complete them. These steps may also be conducted iteratively, as they relate to evaluating the patterns and their possible modifications, based on the DGAC’s assessment of findings from the systematic reviews and other descriptive data analyses. Many of the questions shown in Appendix C involve these steps. For example, identifying the amount of any nutrient in the overall pattern that is contributed by each of the food groups is done to determine each group’s significance to the overall pattern and the extent to which it helps attain appropriate levels of nutrients of concern. If a nutrient tends to be limiting, options can be explored for modifying the patterns, based on which food groups are good sources and accompanying data on usual intakes of those food groups. The preset energy levels infer a “zero-sum game,” so increases of one food group necessarily call for decreases in something else.

Findings

Modeling in general can be helpful in examining complex systems, and the food pattern modeling conducted by USDA has tremendous potential for showing the simultaneous effects of altering food intake patterns, given certain assumptions. Food pattern modeling is dependent on the accuracy of the assumptions, which need to be presented transparently to facilitate broad understanding of the methodology. These assumptions could be better structured to use the most current evidence available and be flexible enough to adjust to reflect new research, if necessary.

The preset energy levels and nutritional goals that serve as constraints in the modeling exercise are drawn from the relevant IOM standards (IOM, 2000). Energy levels designated to correspond to each sex and age group assume a sedentary activity level—rather than erring on the side of a larger energy allowance (which would make meeting nutrient needs easier)—because Americans tend to be both sedentary and overweight. The use of the RDA when available, and the AI otherwise, is fitting for guidance aimed at individuals rather than groups (IOM, 2000; Murphy and Barr, 2006). However, when comparing population usual intakes to proposed food group recommendations, it is important to note that the standards on which food group recommendations are based are intended to cover the requirements of nearly everyone in the population and so are likely higher than most people’s needs. This is an important caveat because when recommendations are set to ensure the adequacy of almost everyone, as in this case, subsequent analyses examining, for example, whether Americans meet the recommendations, does not necessarily mean that most Americans are eating inadequate diets relative to their nutrient needs. In the long term, systems modeling may help

identify if different recommendations are necessary for different groups or individuals and the potential effect of broader versus more tailored recommendations.

To date, a relatively limited set of food groupings has been used in the modeling. These groupings are defined based on nutrient content and their traditional place in the American diet. For example, the dairy group has always had a place in USDA guidance, because foods in that group are both rich sources of calcium (supplying a high amount per 100 gm of the food) and important sources (serving as a major contributor in U.S. diets) (Hoy and Goldman, 2014). However, some subcultures do not include dairy foods in their cuisine, and other segments of the population are lactose intolerant or allergic to dairy proteins. Recent iterations of USDA eating patterns have allowed soy beverages to substitute for animal milk, but that does not entirely address the mismatch between the guidance and the food and beverage preferences of many individuals.

The output of the food pattern modeling is in terms of the total daily quantity of foods from each group. The alternative would be to recommend a specific number of daily servings of a particular size, but the term servings can be confusing because portion sizes vary widely. Quantities for disparate foods within the groups are standardized with the use of “equivalents.” For example, in the grain group, a half cup of cooked rice and one slice of bread are both considered to equal one ounce equivalent of grains.

Another issue is the lack of clarity regarding whether certain food groups and their amounts are necessary, or just sufficient, to achieve the modeling goals. Considering the example previously mentioned with dairy, it is not clear whether there is any other way to meet energy and calcium goals than by the inclusion of the recommended amounts of dairy foods. Another example is the vegetable subgroups: dark-green, red and orange, and legumes are undoubtedly targeted because they provide nutrients typically lacking in diets, but the question remains: are the starchy and other vegetables necessary (e.g., could the targeted subgroups alone supply all the vegetables)? Likewise, why are meats, poultry, and eggs all combined in one protein foods subgroup, given their differing nutrient profiles? It seems the idea of a food group’s traditional role in the American diet may serve as an underlying constraint on the modeling, and a greater array of food group combinations and amounts may be possible that would meet energy and nutrient goals if that constraint were lifted. Again, by representing the pathways between different types of food and the resulting diet and then nutritional intake, systems models can show what may occur with various changes and replacements (e.g., what would happen if different foods were replaced by others?). Conducting a range of sensitivity analyses could provide insight into these relationships.

Employing and modeling different standards of “typical consumption”—operationalized by composite nutrient profiles weighted to reflect population averages—are critical, as they help evaluate what the population’s average nutrient intake would be if they followed the recommendations under varying circumstances. The approach taken by DGAC is in contrast with others that rely on especially nutrient-dense foods (such as salmon, apricots, or almonds), which might result in insufficient nutrient intakes when the patterns are put into practice with more typically consumed foods. However, the range of expected nutrient intakes, as well as the average, could be obtained if the variability in intakes were accounted for. As an example, the composite nutrient profile for fruits is weighted toward orange juice, because it is a major fruit source in the population, and yet not everyone consumes orange juice. It would be helpful to perform some sensitivity analyses to see how dependent the estimated nutrient totals are to the inclusion of orange juice. It also might be useful to have separate composites for different age and sex groups if choices within the food groups vary by those demographics. Finally, typical consumption is inferred for choices within, not among, food groups because some food groups, although rarely consumed, are necessary to achieve nutrient adequacy.

As mentioned previously, although foods selected to represent the nutrient profile of each group are based on typical consumption, they are also in their leanest form, prepared without the addition of fat, oil, or added sugar. This approach seems intuitively contradictory because Americans typically eat foods with an overabundance of such additions; nonetheless, there is justification to examine potential patterns this way. Specifically, this prescription is followed precisely so the modelers can determine how much fat, oil, and sugars can be added to the diet. In other words, the purest forms of foods from nutrient-bearing food groups are assumed at the outset, to see how much of the energy allotment they alone require. A key result of the food pattern modeling is demonstrating there is a very small allowance for discretionary calories when trying to meet nutrient goals within energy levels appropriate for most Americans. In effect food modeling was used to set limits on substances such as fat, oil, and sugars rather than using a physiologic or metabolic end point. Systems models can account for the different additions and modifications (e.g., additives, cooking styles, preparation methods) that may occur to food prior to their consumption.

As defensible as it is to construct the patterns with the leanest, no-sugar-added form of all foods, this raises a key point regarding translation of the patterns. The most recent version of the DGA included some instruction regarding this point, in a figure titled “Hidden Components in Eating Patterns,” but it is not at all clear whether consumers understand

these distinctions. Obviously, if the recommended food group amounts were followed, but with foods in their usual forms, and additional discretionary calories were added, the total energy in the patterns would exceed the designated energy level. Alternatively, perhaps other assumptions could be made in the modeling to incorporate foods as consumed (such as including the saturated fat and sodium from cheese in the dairy composite, in proportion to its consumption).

A similar issue arises with sodium. Most representative foods in the composites are in a no-salt-added form; exceptions include bread (not low-sodium) and a small amount of ham in the meat composite. The relatively limited inclusion of foods containing salt and other sources of sodium results in patterns that approximate the UL without the addition of salt at the table, which is instructive regarding the austerity of the sodium UL.

Conclusion

In summary, the current process for food pattern modeling and its use in informing the DGAC review of the evidence is generally well designed for the questions it is intended to answer. The inputs, constraints, assumptions, and goals are all documented, and standard state-of-the-art data inputs are used. However, enhancements to the process could allow food pattern modeling to respond to a broader range of research questions, increasing its usefulness to the general population. Proposed enhancements include moving toward systems modeling, incorporating other factors and mechanisms that may affect the food composition and choice, further breaking down and representing the heterogeneity of the population and their behaviors, establishing more and different tailored scenarios, and conducting sensitivity analyses to determine how critical various food groups are as well as other key drivers. Sensitivity analyses can also help determine how robust the findings are to the inherent variability in food intakes and composition, and the resulting effect under a wider range of circumstances.

DESCRIPTIVE DATA ANALYSES: EVALUATION OF METHODOLOGY AND USE

Other types of questions that the DGAC is interested in answering are questions of prevalence, trends, and population-level food intake (see Table 6-1 and Appendix C), which are best answered with descriptive data. Previous DGACs have used national data to guide information regarding dietary, nutritional, and health status of the U.S. population. USDA and HHS staff with access to the data and expertise to conduct

analyses have assisted in preparing these descriptive data analyses for the DGAC. Following the 2015 DGA cycle, these federal staff were collectively referred to as the Data Analysis Team (DAT). The post-hoc development of the DAT intends to improve efficiency regarding procurement and analyses of data requests of the DGAC. Aside from accessibility to appropriate data and willingness to assist the DGAC, no specific criteria for membership or inclusion on the DAT have been noted. The data were made publicly available through online access and inclusion in the 2015 DGAC Scientific Report’s references and appendixes. Different from the questions completed by the NEL systematic reviews, the questions addressed using descriptive data analyses did not go through a grading rubric and were not graded. The DGAC acknowledged this issue by taking the “strengths and limitations of data analyses into account in formulating conclusion statements” (HHS/USDA, 2015b).

Questions That Descriptive Data Analyses Are Intended to Address

Descriptive data analyses are important with regard to providing information about the current status and trends in food and nutrient intakes among the population of the United States. Such information helped the DGAC evaluate adequacy of nutrient intake to determine whether there were “nutrients of concern” that may have been under- or overconsumed, as well as to indicate whether consumption should increase or decrease according to population consumption patterns (see Chapter 7 for additional information on determining nutrients of concern). The nutrient intake data provided information to determine whether nutrient intake of the U.S. population met the Dietary Reference Intakes (DRIs) recommendations. Data reporting the prevalence of diet-related diseases and conditions in the U.S. population aided the DGAC in its determination of the scope of current dietary problems. The specific questions addressed by these descriptive data analyses are outlined in Appendix C.

Sources of Descriptive Data and Methods Used by the 2015 DGAC

Whereas most of the analyses used data collected as part of NHANES, other important data sources were used by the 2015 DGAC, including government and nongovernment sources. These included the National Health Interview Survey and the 2014 report of the American Heart Association (HHS/USDA, 2015b) among others, as shown in Table 6-5.

A number of methods were used in the 2015 DGAC’s assessment of available surveillance data to describe dietary and health status. Although stepwise descriptions of each analysis conducted to inform the

TABLE 6-5 Summary of Data Sources Used by the 2015 DGAC for Descriptive Data Analyses

NOTE: AHA = American Heart Association; NHANES = National Health and Nutrition Examination Survey; NHIS = National Health Interview Survey; SEARCH Study = SEARCH for Diabetes in Youth study; SEER = National Cancer Institute’s Surveillance, Epidemiology, and End Results Program; USDA-ARS NND = U.S. Department of Agriculture’s Agricultural Research Service National Nutrient Database for Standard Reference, Release 27.

^a Includes data on energy and selected nutrients.

SOURCE: HHS/USDA, 2015b.

2015 DGAC Scientific Report were not available, the sources of data were included in the DGAC’s description of the evidence (HHS/USDA, 2015b). The 2015 DGAC drew all dietary intake data from NHANES What We Eat in America (WWEIA), which offers detailed food intake and behavior information, as well as national- and subgroup-level estimates of usual dietary intakes.

NHANES WWEIA uses the interviewer-administered 24-hour dietary recall, which captures detailed dietary information for the past 24 hours through a verbal interview with participants. The 24-hour dietary recall method can provide rich detail about foods consumed. Although it measures intake only on a given day, and can be used to assess the mean intake for a population group, multiple 24-hour recalls for at least a representative subsample of individuals are required to estimate the distribution of usual intake for a population. Two recalls were collected for most individuals in WWEIA. The statistical method used by the DAT to esti-

mate the distributions of usual nutrient and food intakes in a population was developed by scientists within NCI; thus, the method is colloquially and broadly referred to as the “NCI [National Cancer Institute] method” (NCI, 2015; Tooze et al., 2006).

Findings

This National Academies committee reviewed the approach of the DGAC, in collaboration with relevant federal partners (the DAT). The analytic approach used in determining the proportion of the population that is inadequate or at risk of adverse health effects owing to excess consumption adheres to the recommendations by the DRI committee on diet assessment, including the collection of multiple 24-hour dietary recalls, estimation of usual intake distributions, and comparisons of EARs to those distributions.

Another key element in the development of the guidelines is the availability of current data for use by the DGAC. This depends on both the timely release of data as well as timely analysis by the DAT. The descriptive data analyses used by the DGAC provide important inputs at the outset of the DGAC’s review of evidence, including current prevalence of disease, trends, and population nutrient intakes, to inform further analyses and inform the identification of nutrients of concern. However, in the current process, these analyses are not initiated until the convening of the DGAC. Preparing these descriptive analyses in advance could maximize the DGAC’s time. However, one caveat is that the most current data should be used by the DGAC, so any significant updates to datasets released after the initial analyses would need to be integrated throughout the DGAC’s process. Additionally, recognizing that available population subgroup data may not fully represent the diversity of the U.S. population, exploring options to expand these data could be beneficial.

In reviewing descriptive data analyses conducted to inform past DGACs by various federal agencies and offices based on availability and relevance, this National Academies committee sees the introduction of the DAT as an opportunity for improving reporting of data analyses and centralizing efforts. For example, a central Web-based location that includes links to all the data analyses would be useful in increasing transparency and comparability of descriptive data analyses across DGA cycles.

This National Academies committee recognizes that data analyses depend on the availability of current and high-quality data collection efforts (e.g., surveillance data), and that advancements in the methods of data collection at the population and subgroup levels can also lead to improvements in the understanding of population health and disease prevalence and trends.

Conclusion

Overall, this National Academies committee found that the descriptive data analyses being conducted are appropriate to answer the questions proposed by the DGAC, and the availability of current and high-quality data continues to contribute to the most accurate analyses possible. Timely access to results of descriptive data analyses could maximize the DGAC’s time and have important and lasting benefits for the DGA to affect changes in population health.

QUALITY OF DIETARY DATA ACROSS ALL TYPES OF EVIDENCE

Dietary intake data are central to the development of dietary guidelines, and form the basis of most studies examined in the systematic evidence-based reviews and analyses of food pattern modeling and consumption patterns of the population. The nutrition field has relied on respondent self-report as a means of assessing dietary intake to capture the totality of the diet. However, substantial limitations exist.

Regarding self-report, one of the limitations is measurement error, or the difference between the true value of a parameter of interest and the value obtained through assessment. There are two main types of measurement error: random and systematic. Random error occurs when measures scatter randomly around the true value. In dietary research, day-to-day variation in intakes is an example of this type of error. In descriptive research, random error does not affect the mean but leads to wider distributions and an overestimation of tail probabilities; in examining relationships, random error leads to attenuation and loss of power. Systematic error, or bias, occurs when measures tend to deviate from truth in the same direction; the widespread tendency to underreport intakes is an example of bias. Differential bias, such as the tendency toward greater underreporting among persons with obesity relative to other individuals, is particularly problematic because it confounds relationships between diet and health. In descriptive research, systematic error leads to inaccurate estimates of both the mean and the distribution. In examining associations, relationships can be either exaggerated or attenuated, depending on the source of the bias, and loss of power can result.

Different methods of collecting self-report dietary intake data exhibit varying degrees of each type of measurement error (NCI, 2017). Dietary recalls, which form the foundation of population dietary monitoring studies such as the NHANES WWEIA, generally exhibit more within-person random error because they are conducted on a 24-hour basis. Food frequency questionnaires, which have been used in most epidemiologic cohort studies, display relatively more bias.

Many researchers have described the effects of measurement error and what steps can be taken to address it, such as statistical adjustment and data cleaning methods that exclude implausible data (Freedman et al., 2011, 2017; Goldberg et al., 1991; Huang et al., 2005; Prentice and Huang, 2011; Tooze et al., 2010). Random within-person error can be corrected with repeat administrations and statistical modeling, and it is therefore generally less problematic than bias. Systematic error cannot be eliminated by repeated measures, but its effects can be addressed with the use of reference measures, such as biomarkers or a less biased instrument, on at least a subsample of individuals (NCI, 2017). Methods are continually being developed to address both types of error more effectively.

CONCLUSION

The types of analyses used in the DGAC Scientific Report include original systematic reviews; existing systematic reviews, meta-analyses, and reports; food pattern modeling; and descriptive data analyses. All types of analyses will continue to provide important information in the DGA going forward, and they need to be based on validated, standardized, and up-to-date methods and processes.

The NEL systematic review process overall is thorough and based on rigorous scientific standards. However, this National Academies committee identified opportunities for improvement in several areas where the NEL can be aligned with best practices. These areas include (1) a clear delineation of roles of the Dietary Guidelines Scientific Advisory Committee (DGSAC) and the NEL, (2) training for NEL staff, (3) collaboration with other systematic review groups, and (4) supportive infrastructure to keep NEL processes up to date. Additionally, ongoing surveillance of existing systematic reviews, meta-analyses, and reports can allow for less duplication of efforts and ensure outdated systematic reviews are updated and existing systematic reviews are used whenever possible. In using existing systematic reviews, the relevance, timeliness, and quality of the systematic review should be carefully considered according to NEL protocol and methodological standards.

Food pattern modeling and descriptive data analyses both provide unique and important inputs into the DGA process and could be improved with advanced and standardized methods. The current process for food pattern modeling operates within several assumptions, some of which are necessary and some of which can be updated to improve outcomes. Advancing methodology in the food pattern modeling process to account for variability in eating patterns could improve outcomes and increase the usefulness of the USDA food patterns. Conducting some descriptive

data analyses and initial food pattern modeling in advance of the DGAC’s convening, depending on the availability of current and high-quality data, could maximize time and increase efficiency of the DGA process.

REFERENCES

Huang, T. T., S. B. Roberts, N. C. Howarth, and M. A. McCrory. 2005. Effect of screening out implausible energy intake reports on relationships between diet and BMI. Obesity Research 13(7):1205-1217.

IOM (Institute of Medicine). 2000. Dietary Reference Intakes: Applications in dietary assessment. Washington, DC: National Academy Press.

IOM. 2005. Dietary Reference Intakes for water, potassium, sodium, chloride, and sulfate. Washington, DC: The National Academies Press.

IOM. 2011. Finding what works in health care: Standards for systematic reviews. Washington, DC: The National Academies Press.

Marcoe, K., W. Juan, S. Yamini, A. Carlson, and P. Britten. 2006. Development of food group composites and nutrient profiles for the MyPyramid food guidance system. Journal of Nutrition, Education, and Behavior 38(Suppl 6):S93-S107.

Millen, B. January 10, 2017 (unpublished). Understanding the committee’s methodology for topic formation and reviewing the evidence base. Presentation to the Committee to Review the Process to Update the Dietary Guidelines for Americans, Washington, DC, January 10.

Murphy, S. P., and S. I. Barr. 2006. Recommended dietary allowances should be used to set daily values for nutrition labeling. American Journal of Clinical Nutrition 83(5):1223s-1227s.

NCI (National Cancer Institute). 2015. Usual dietary intakes: The NCI method. Rockville, MD: NCI.

NCI. 2017. Dietary assessment primer. Rockville, MD: NCI.

Prentice, R. L., and Y. Huang. 2011. Measurement error modeling and nutritional epidemiology association analyses. Canadian Journal of Statistics 39(3):498-509.

Shea, B., J. Grimshaw, G. Wells, M. Boers, N. Andersson, C. Hamel, A. Porter, P. Tugwell, D. Moher, and L. Bouter. 2007. Development of AMSTAR: A measurement tool to assess the methodological quality of systematic reviews. BMC Medical Research Methodology 7(1):10.

Smith, V., D. Devane, C. M. Begley, and M. Clarke. 2011. Methodology in conducting a systematic review of systematic reviews of healthcare interventions. BMC Medical Research Methodology 11(1):15.

Tooze, J. A., D. Midthune, K. W. Dodd, L. S. Freedman, S. M. Krebs-Smith, A. F. Subar, P. M. Guenther, R. J. Carroll, and V. Kipnis. 2006. A new statistical method for estimating the usual intake of episodically consumed foods with application to their distribution. Journal of the American Dietetic Association 106(10):1575-1587.

Tooze, J. A., V. Kipnis, D. W. Buckman, R. J. Carroll, L. S. Freedman, P. M. Guenther, S. M. Krebs-Smith, A. F. Subar, and K. W. Dodd. 2010. A mixed-effects model approach for estimating the distribution of usual intake of nutrients: The NCI method. Statistics in Medicine 29(27):2857-2868.

USDA. 2012. A series of systematic reviews on the effects of nutrition education on children’s and adolescents’ dietary intake. Alexandria, VA: Food and Nutrition Service, USDA.

USDA. 2014. A series of systematic reviews on the relationship between dietary patterns and health outcomes. Alexandria, VA: Food and Nutrition Service, USDA.

USDA. 2015. USDA food patterns. Washington, DC: USDA.

USDA. 2017. Nutrition Evidence Library. Alexandria, VA: Center for Nutrition Policy and Promotion, USDA.

USDA/HHS. 2016 (unpublished). Dietary Guidelines for Americans: Process brief, sections 4–5. Prepared for the Committee to Review the Process to Update the Dietary Guidelines for Americans.

USDA/HHS. 2017 (unpublished). Dietary Guidelines for Americans: Process brief, sections 6–7. Prepared for the Committee to Review the Process to Update the Dietary Guidelines for Americans.

Viswanathan, M., N. D. Berkman, D. M. Dryden, and L. Hartling. 2012. Development of the RTI item bank on risk of bias and precision of observational studies. Journal of Clinical Epidemiology 65(2):163-178.

Viswanathan, M., N. D. Berkman, D. M. Dryden, and L. Hartling. 2013. AHRQ methods for effective health care. In Assessing risk of bias and confounding in observational studies of interventions or exposures: Further development of the RTI item bank. Rockville, MD: Agency for Healthcare Research and Quality.

Wegewitz, U., B. Weikert, A. Fishta, A. Jacobs, and D. Pieper. 2016. Resuming the discussion of AMSTAR: What can (should) be made better? BMC Medical Research Methodology 16(1):111.

West, S., V. King, T. S. Carey, K. N. Lohr, N. McKoy, S. F. Sutton, and L. Lux. 2002. Systems to rate the strength of scientific evidence. Vol. 47. Prepared by the Research Triangle Institute, University of North Carolina Evidence-Based Practice Center. Rockville, MD: Agency for Healthcare Research and Quality.

Whitlock, E. P., J. S. Lin, R. Chou, P. Shekelle, and K. A. Robinson. 2008. Using existing systematic reviews in complex systematic reviews. Annals of Internal Medicine 148(10):776-782.