questions might address how interventions may need to be adapted, what training is needed for staff, or how to sustain effects over time.
Two terms relevant to evidence quality and standards in evidence-based medicine or public health are key to the discussion here and throughout this report: internal validity (the level of certainty of the causal relationship between an intervention and the observed outcomes), hereafter referred to as “level of certainty,” and external validity (the extent to which research results can be generalized to other populations, including individuals, settings, contexts, and time frames, which speaks to issues of the transferability and application of findings to real-world settings), hereafter referred to as “generalizability.” (See Appendix A for formal definitions.)
In relation to Type 2 and perhaps Type 3 questions, studies to date have tended to emphasize the level of certainty of the causal relationship between an intervention and the observed outcome (e.g., well-controlled efficacy trials) while giving limited attention to the intervention’s generalizability (e.g., the translation of science to the various circumstances of practice) (Glasgow, 2008; Glasgow et al., 2006; Green and Glasgow, 2006; Klesges et al., 2008). For example, Klesges and colleagues (2008) reviewed 19 childhood obesity studies to assess the extent to which dimensions of external validity were reported. Their work reveals that some key contextual variables (e.g., cost, program sustainability) are missing entirely in the peer-reviewed literature on obesity prevention. Because generalizability is deemphasized or compromised in study selection and evidence screening procedures based on the hierarchical framework of evidence-based medicine, concern has been raised that applying that hierarchy’s narrow inclusion criteria results in a limited pool of sources for subsequent evidence synthesis (Dixon-Woods et al., 2001). Further, it is increasingly acknowledged that the traditional hierarchy works less well in public health and health care arenas involving large-scale, complex social interventions than in the arena of medical therapies. Evidence standards for community-based public health problems call for taking a broader perspective to locate useful forms of evidence. The types of evidence that are available for assessing both level of certainty and generalizability were a key consideration when the committee developed the framework proposed in this report.
While the highest-quality source of evidence is generally deemed to be the RCT, much of the evidence available for environmental and policy strategies to reduce obesity risk is not derived from such trials. For example, a randomized design is seldom useful in policy research because the scientist cannot randomly assign exposure (the policy), and understanding of the policy process is often gained from qualitative methods (e.g., case studies). In addition, reductionist approaches for inferring causality (common in medicine) tend to reduce a problem to a single cause (Davidovitch and Filc, 2006), whereas obesity is likely to be influenced by a complex web of causation (Hill and Peters, 1998; Kumanyika and Brownson, 2007; Kumanyika et al., 2002).