National Academies Press: OpenBook

Science, Evidence, and Inference in Education: Report of a Workshop (2001)

Chapter: Theme 3: Evidence and Inference: Consistency and Variation Revisited

« Previous: Theme 2: The Interface of Research and Practice in Education: Linking Quality with Utility
Suggested Citation:"Theme 3: Evidence and Inference: Consistency and Variation Revisited." National Research Council. 2001. Science, Evidence, and Inference in Education: Report of a Workshop. Washington, DC: The National Academies Press. doi: 10.17226/10121.
×

Page 11

Theme 3.

Evidence and Inference: Consistency and Variation Revisited

At the core of science is a commitment to rigorous reasoning, method, and the use of evidence. The final session of the workshop was designed to take a step back from the specific issues of how federal agencies support science and how science can inform education practice, and to focus on the “first principles” of evidentiary and inferential reasoning. To help it deliberate about the scientific principles of education research, the committee assembled a panel of scholars from a range of scientific disciplines and professions who provided their perspectives on the ways evidence and inference are used in their fields. Panelists brought expertise from education assessment, linguistic anthropology, labor economics, law, and the emerging interdisciplinary field of systematic synthesis.

The panel began with a talk by an expert in education assessment, whose scholarly work has focused on the identification of “first principles” of evidentiary inference and reasoning. His presentation served as a frame for subsequent presentations and discussions. 6 In his introductory remarks, he stressed the difference between data and evidence: “Datum becomes evidence in some analytic problem when its relevance to conjectures being considered is established.” Any piece of evidence, he argued, is almost always “incomplete, inconclusive, and amenable to multiple explanations.” He also said that using evidence to make inferences—explanations, conclusions, or predictions based on what we know and observe—is always done in the presence of uncertainty.

Evidence is almost always incomplete, inconclusive, and amenable to multiple explanations...we always reason in the presence of uncertainty.

—Robert Mislevy

The panel discussion following illustrated the same theme of consistency and variation that surfaced in previous sessions. Despite dramatic variability with respect to the goals, methods, and products of the exercise, a consensus among the panelists on the basic tenets of reasoning about evidence began to develop.

6 His presentation and the some of the quotes provided here first appeared in: Schum, D.A. (1994). The Evidential Foundations of Probabilistic Reasoning. New York: Wiley.

Suggested Citation:"Theme 3: Evidence and Inference: Consistency and Variation Revisited." National Research Council. 2001. Science, Evidence, and Inference in Education: Report of a Workshop. Washington, DC: The National Academies Press. doi: 10.17226/10121.
×

Page 12

BACK TO BASICS: EXPLICIT REASONING, RIGOROUS METHOD, PUBLIC PROCESS

Each panelist described at least three common characteristics of effective inferential reasoning: (1) a visible and explicit form of argument with a clear delineation of constructs, frameworks and theories (explicit reasoning); (2) the identification and explanation of patterns, variations, and rival hypotheses (rigorous method); and (3) a commitment to clear and accessible documentation of how inferences are made (public process).

A crucial aspect of systematic syntheses is that they need to make public, and therefore open to scientific scrutiny, the methods of the synthesis process...a commitment to revising reviews periodically is a good model for how the accumulation of evidence can work...

—Larry Hedges

Panelists described the rigors of reasoning about evidence. Although accomplished by different means (e.g., abduction, induction, and deduction), “inferential force”—as it was described by one panelist—is created by moving among the data, explaining the warrants for each step in the inferential chain, and adding appropriate qualifiers and conditions. Inferences are strengthened by subsequently and iteratively making predictions and testing them, seeking the best explanation by considering and eliminating alternative or rival hypotheses, describing possible unseen mechanisms, and revising frameworks to account for unexpected data or results.

Several panelists emphasized the importance of making the inferential process publicly available to encourage scrutiny by the professional community. Subjecting claims to criticism and engaging in a debate about the warrants for knowledge was explicitly identified by several panelists as an indication of the health of the knowledge-generating enterprise.

Controlled experiments are challenged by the replicability criterion: Does the experiment match the practice...that would be in operation if the programs were to be adopted? They also are associated with high costs and there may be ethical constraints. Uncontrolled experiments (observational studies) face selection bias. Avoiding this bias...is always difficult.

—Glen Cain

THE SPECIALIZATION OF INFERENTIAL REASONING: VARIATION IN GOALS, METHODS, AND PRODUCTS

Although each of the panelists made clear the importance of rigorous thinking in making inferences and claims, the variability in the ways in which different fields and disciplines treat evidence and inference was also apparent. The legal scholar on the panel made this point explicitly, arguing “the coherence and elegance of a particular perspective on inference should not be mistaken for the omnipotence of any such perspective” and the inferential reasoning process will always involve “a wide array of conceptual and perceptual processes.”

The panel presentations and subsequent discussions illustrated this inherent specialization in evidentiary and inferential reasoning. Specifically, the goal of the inference and its intended product gave rise to much of the variability in the method of the reasoning

Suggested Citation:"Theme 3: Evidence and Inference: Consistency and Variation Revisited." National Research Council. 2001. Science, Evidence, and Inference in Education: Report of a Workshop. Washington, DC: The National Academies Press. doi: 10.17226/10121.
×

Page 13

process across fields. Contrasting ethnographic techniques with traditional social science methods, for example, one panelist, a linguistic anthropologist, identified the difference in the objective of each type of research explicitly: “The goal [of ethnography] is to understand how things connect rather than how to isolate a measure.” The end product, therefore, is also different: “[Ethnography] is theory-generating rather than theory-testing.”

[In the traditional sciences], if you end up with a concept you didn't have before you started, your career is over... In my field [ethnography], if you don't end up with a new concept that you didn't have before you started, your career is over.

—Michael Agar (emphasis added)

The labor economist on the panel traced the evolution of econometric methods that model the relationship between inputs and outputs. He commented on the now well-known tension between controlled experiments and observational studies, identifying the relative strengths and weaknesses of each strategy.

Legal practice and scholarship blend a variety of inferential techniques. Since law is not concerned with identifying fundamental principles, the legal scholar on the panel suggested that there are limits to the parallels that can be drawn between inference in law and inference in science. He did, however, suggest one way in which scientists might learn from lawyers, judges, and legal scholars from the adversarial system of American justice, reminding the scientists these actors are “good at identifying multiple sources of uncertainty.”

Often you find that study findings may contradict one another in terms of statistical significance, the sign of the effect, or the magnitude of the effect... these conditions are not unique to education, and in fact cover a wide swath of the sciences... experimental ecology, some fields of chemistry, medicine, psychology, and public health...

—Larry Hedges

Another panelist described the problem of inference in systematic syntheses. This interdisciplinary field, he explained, enables comparisons across studies to produce summary descriptions of bodies of research evidence. He noted that the problems of drawing inferences from multiple studies are the same across disciplines and similar to those of drawing inferences in individual studies.

Suggested Citation:"Theme 3: Evidence and Inference: Consistency and Variation Revisited." National Research Council. 2001. Science, Evidence, and Inference in Education: Report of a Workshop. Washington, DC: The National Academies Press. doi: 10.17226/10121.
×
Page 11
Suggested Citation:"Theme 3: Evidence and Inference: Consistency and Variation Revisited." National Research Council. 2001. Science, Evidence, and Inference in Education: Report of a Workshop. Washington, DC: The National Academies Press. doi: 10.17226/10121.
×
Page 12
Suggested Citation:"Theme 3: Evidence and Inference: Consistency and Variation Revisited." National Research Council. 2001. Science, Evidence, and Inference in Education: Report of a Workshop. Washington, DC: The National Academies Press. doi: 10.17226/10121.
×
Page 13
Next: Synthesis and Next Steps for the Committee »
Science, Evidence, and Inference in Education: Report of a Workshop Get This Book
×
Buy Paperback | $21.00 Buy Ebook | $16.99
MyNAP members save 10% online.
Login or Register to save!
Download Free PDF

Research on education has come into the political spotlight as the demand grows for reliable and credible information for the guidance of policy and practice in the education reform environment. Many debates among the education research community feature questions concerning the nature of evidence and these questions have also appeared in broader policy and practice arenas. Inquiry has generally, over the past years, created bodies of scientific knowledge that have profound implications for education. Dramatic advances in understanding how people learn, how young children acquire early reading skills, and how to design and evaluate educational and psychological measurements is a good example of this. However, the highly contextualized nature of education and the wide range of disciplinary perspectives that rely on it have made the identification of reducible, generalizable principles difficult and slow to achieve.

Due to this, the U.S. Department of Education's National Educational Research Policy and Priorities Board (NERPPB) has asked the NRC to establish a study committee to consider the scientific underpinnings of research in education. The committee consists of members with expertise in statistics, psychology, sociology, anthropology, philosophy of science, history of education, economics, chemistry, biology, and education practice. The committee worked with the three questions in mind: What are the principles of scientific quality in education research?, How can research-based knowledge in education cumulate?, and How can a federal research agency promote and protect scientific quality in the education research it supports?.

A workshop was held on March 7-8, 2001 that was organized into three main sessions: Supporting Scientific Quality at the Federal level, The Interface of Research and Practice in Education, and Evidence and Inference. Science, Evidence, and Inference in Education: Report of a Workshop summarizes this workshop through these three ideas. The report also includes what the committee plans to do next, the workshop agenda, and information on the workshop's participants and speakers.

  1. ×

    Welcome to OpenBook!

    You're looking at OpenBook, NAP.edu's online reading room since 1999. Based on feedback from you, our users, we've made some improvements that make it easier than ever to read thousands of publications on our website.

    Do you want to take a quick tour of the OpenBook's features?

    No Thanks Take a Tour »
  2. ×

    Show this book's table of contents, where you can jump to any chapter by name.

    « Back Next »
  3. ×

    ...or use these buttons to go back to the previous chapter or skip to the next one.

    « Back Next »
  4. ×

    Jump up to the previous page or down to the next one. Also, you can type in a page number and press Enter to go directly to that page in the book.

    « Back Next »
  5. ×

    Switch between the Original Pages, where you can read the report as it appeared in print, and Text Pages for the web version, where you can highlight and search the text.

    « Back Next »
  6. ×

    To search the entire text of this book, type in your search term here and press Enter.

    « Back Next »
  7. ×

    Share a link to this book page on your preferred social network or via email.

    « Back Next »
  8. ×

    View our suggested citation for this chapter.

    « Back Next »
  9. ×

    Ready to take your reading offline? Click here to buy this book in print or download it as a free PDF, if available.

    « Back Next »
Stay Connected!