refine theories, they design research studies that entail the collection of data in some form. Those data are then analyzed, results or findings are arrived at, and interpretations of those results are made. These interpretations then can be used to frame future research and guide policy making and program design and implementation.

Replicability and Falsifiability

All theories must be falsifiable. In other words, any theory derived from a study must be sufficiently elaborated so that other scientists can replicate the study and collect additional empirical data to either corroborate or contradict the original theory. It is this willingness to abandon or modify a theory in the face of new evidence that is one of the most central defining features of the scientific method.

The extent to which one particular theory can be viewed as uniquely supported by a particular study depends on the extent to which alternative explanations have been ruled out. A particular research result is never equally relevant to all competing theoretical explanations. A given experiment may be a very strong test of one or two alternative theories but a weak test of others.

Validity and Generalizability

Validity is defined as the extent to which the instrument is actually measuring what the researcher intends it to measure. External validity concerns the generalizability of the conclusions to the larger population and setting of interest. Internal and external validity are often traded off across different methodologies. The alleged trade-off between internal and external validity presents some interesting questions. In what sense can a biased estimate (one that is inaccurate for the whole population) be said to be generalizable? What we mean is that we are willing to risk a small amount of bias for a large increase in confidence that the estimate generalizes to a much larger set of children and programs. Willingness to take that risk requires some confidence that the size of the bias introduced by lack of experimental control is small relative to the bias introduced by applying an unbi-



The National Academies | 500 Fifth St. N.W. | Washington, D.C. 20001
Copyright © National Academy of Sciences. All rights reserved.
Terms of Use and Privacy Statement