To evaluate the results of the literature search, the committee first used the organizational scheme described above to consider (1) the validity and (2) the generalizability of the studies and publications identified in its research taxonomy. Although the committee appreciates the range of evidence types and the strengths and weaknesses of various methodological approaches, it determined that the most useful approach was to examine research questions relevant to its statement of task with observational studies conducted in a real-world context. From this perspective, the quantitative approach of the randomized controlled trial was deemed limited because of the use of an artificial setting and the need to control variables that may have application to understanding and interpreting consumer behavior. Relevant experimental studies were not identified in the search and thus were not included in the evidence review.
The types of research studies that provided the most valid evidence for examining the behavior of participants in food assistance programs were field experiments and survey-based studies. The committee identified a range of observational evidence, including population-based surveys and impact studies based on secondary data analysis, that reflected the complexity of variables relevant to the outcomes of interest, as well as the generalizability of the evidence to pertinent research questions. The committee considered the merits of these studies on the basis of the methodological approach used and the overall quality of the research. Additional support for the findings from observational studies was obtained from evidence-based reviews, meta-analyses, and qualitative studies.
Finally, in addition to the body of peer-reviewed evidence from journal publications, the committee considered peer-reviewed evidence from government reports, in particular, relevant reports from the U.S. Department of Agriculture’s (USDA’s) Economic Research Service, as well as nonreviewed publications from stakeholder and nongovernmental organizations. Although government reports undergo a rigorous peer review, the process differs from that for peer-reviewed journal articles in that the review is not blinded. Specifically, the Office of Management and Budget Guidelines for Ensuring and Maximizing the Quality, Objectivity, Utility, and Integrity of Information Disseminated by Federal Agencies state: “In general, an agency conducting a peer review of a highly influential scientific assessment must ensure that the peer review process is transparent by making available to the public the written charge to the peer reviewers, the peer reviewers’ names, the peer reviewers’ report(s), and the agency’s response to the peer reviewers’ report(s).”1 While
1Guidelines for Ensuring and Maximizing the Quality, Objectivity, Utility, and Integrity of Information Disseminated by Federal Agencies, Federal Register, Volume 67, Issue 24, p. 5465 (February 5, 2002).