Page 11
Theme 3.
Evidence and Inference: Consistency and Variation Revisited
At the core of science is a commitment to rigorous reasoning, method, and the use of evidence. The final session of the workshop was designed to take a step back from the specific issues of how federal agencies support science and how science can inform education practice, and to focus on the “first principles” of evidentiary and inferential reasoning. To help it deliberate about the scientific principles of education research, the committee assembled a panel of scholars from a range of scientific disciplines and professions who provided their perspectives on the ways evidence and inference are used in their fields. Panelists brought expertise from education assessment, linguistic anthropology, labor economics, law, and the emerging interdisciplinary field of systematic synthesis.
The panel began with a talk by an expert in education assessment, whose scholarly work has focused on the identification of “first principles” of evidentiary inference and reasoning. His presentation served as a frame for subsequent presentations and discussions. 6 In his introductory remarks, he stressed the difference between data and evidence: “Datum becomes evidence in some analytic problem when its relevance to conjectures being considered is established.” Any piece of evidence, he argued, is almost always “incomplete, inconclusive, and amenable to multiple explanations.” He also said that using evidence to make inferences—explanations, conclusions, or predictions based on what we know and observe—is always done in the presence of uncertainty.
Evidence is almost always incomplete, inconclusive, and amenable to multiple explanations...we always reason in the presence of uncertainty. —Robert Mislevy |
The panel discussion following illustrated the same theme of consistency and variation that surfaced in previous sessions. Despite dramatic variability with respect to the goals, methods, and products of the exercise, a consensus among the panelists on the basic tenets of reasoning about evidence began to develop.
6 His presentation and the some of the quotes provided here first appeared in: Schum, D.A. (1994). The Evidential Foundations of Probabilistic Reasoning. New York: Wiley.
Page 12
BACK TO BASICS: EXPLICIT REASONING, RIGOROUS METHOD, PUBLIC PROCESS
Each panelist described at least three common characteristics of effective inferential reasoning: (1) a visible and explicit form of argument with a clear delineation of constructs, frameworks and theories (explicit reasoning); (2) the identification and explanation of patterns, variations, and rival hypotheses (rigorous method); and (3) a commitment to clear and accessible documentation of how inferences are made (public process).
A crucial aspect of systematic syntheses is that they need to make public, and therefore open to scientific scrutiny, the methods of the synthesis process...a commitment to revising reviews periodically is a good model for how the accumulation of evidence can work... —Larry Hedges |
Panelists described the rigors of reasoning about evidence. Although accomplished by different means (e.g., abduction, induction, and deduction), “inferential force”—as it was described by one panelist—is created by moving among the data, explaining the warrants for each step in the inferential chain, and adding appropriate qualifiers and conditions. Inferences are strengthened by subsequently and iteratively making predictions and testing them, seeking the best explanation by considering and eliminating alternative or rival hypotheses, describing possible unseen mechanisms, and revising frameworks to account for unexpected data or results.
Several panelists emphasized the importance of making the inferential process publicly available to encourage scrutiny by the professional community. Subjecting claims to criticism and engaging in a debate about the warrants for knowledge was explicitly identified by several panelists as an indication of the health of the knowledge-generating enterprise.
Controlled experiments are challenged by the replicability criterion: Does the experiment match the practice...that would be in operation if the programs were to be adopted? They also are associated with high costs and there may be ethical constraints. Uncontrolled experiments (observational studies) face selection bias. Avoiding this bias...is always difficult. —Glen Cain |
THE SPECIALIZATION OF INFERENTIAL REASONING: VARIATION IN GOALS, METHODS, AND PRODUCTS
Although each of the panelists made clear the importance of rigorous thinking in making inferences and claims, the variability in the ways in which different fields and disciplines treat evidence and inference was also apparent. The legal scholar on the panel made this point explicitly, arguing “the coherence and elegance of a particular perspective on inference should not be mistaken for the omnipotence of any such perspective” and the inferential reasoning process will always involve “a wide array of conceptual and perceptual processes.”
The panel presentations and subsequent discussions illustrated this inherent specialization in evidentiary and inferential reasoning. Specifically, the goal of the inference and its intended product gave rise to much of the variability in the method of the reasoning
Page 13
process across fields. Contrasting ethnographic techniques with traditional social science methods, for example, one panelist, a linguistic anthropologist, identified the difference in the objective of each type of research explicitly: “The goal [of ethnography] is to understand how things connect rather than how to isolate a measure.” The end product, therefore, is also different: “[Ethnography] is theory-generating rather than theory-testing.”
[In the traditional sciences], if you end up with a concept you didn't have before you started, your career is over... In my field [ethnography], if you don't end up with a new concept that you didn't have before you started, your career is over. —Michael Agar (emphasis added) |
The labor economist on the panel traced the evolution of econometric methods that model the relationship between inputs and outputs. He commented on the now well-known tension between controlled experiments and observational studies, identifying the relative strengths and weaknesses of each strategy.
Legal practice and scholarship blend a variety of inferential techniques. Since law is not concerned with identifying fundamental principles, the legal scholar on the panel suggested that there are limits to the parallels that can be drawn between inference in law and inference in science. He did, however, suggest one way in which scientists might learn from lawyers, judges, and legal scholars from the adversarial system of American justice, reminding the scientists these actors are “good at identifying multiple sources of uncertainty.”
Often you find that study findings may contradict one another in terms of statistical significance, the sign of the effect, or the magnitude of the effect... these conditions are not unique to education, and in fact cover a wide swath of the sciences... experimental ecology, some fields of chemistry, medicine, psychology, and public health... —Larry Hedges |
Another panelist described the problem of inference in systematic syntheses. This interdisciplinary field, he explained, enables comparisons across studies to produce summary descriptions of bodies of research evidence. He noted that the problems of drawing inferences from multiple studies are the same across disciplines and similar to those of drawing inferences in individual studies.