In the simplest study of an intervention, one group of subjects who receive the intervention (the treatment group) is compared with another group of subjects (the control group) who do not. When the control group receives no other intervention, it serves to depict the counterfactual: what would happen in the absence of the intervention. Many studies, however, are more elaborate and may involve multiple interventions and controls.
An experiment is a study in which the investigator controls the selection of the subjects who may receive the intervention and assigns them to treatment and control groups at random. Experiments can be conducted in highly controlled settings, such as in a laboratory, or in the field, such as at a school, so as to better reflect the context in which an intervention would be implemented in practice. The former assess efficacy, or whether the intervention produces the intended effect. The latter, called randomized controlled field trials (RCFTs), assess effectiveness, or whether the intervention produces the intended effect in practice.
One important advantage of RCFTs is that secondary variables do not confound the effects of an intervention. That is, in an ideal study, an investigator wants to compare the effects of an intervention on a treatment group that is as similar as possible to the control group in all important respects except for having received the intervention. But this ideal can be affected by secondary or intervening variables—other factors by which the treatment group differs from the control group but are not of primary interest—which confound the effects of the intervention. These factors can influence the outcome of an experiment. In an RCFT, however, these secondary variables do not necessarily need to be controlled for in the design or the analysis: randomization obviates even the need to identify the secondary variables.
For many policy purposes, however, the effects of secondary variables are often critical, especially when the intervention is implemented as the result of a policy action. For this reason, the designs of RCFTs are often complex and incorporate individual differences among subjects and contextual variables so that their effects can be analyzed.
Even for the most rigorously conducted RCFTs, however, the results from one setting may not generalize to all other settings. Consequently, it may be difficult to identify “what works” in different settings from just one RCFT. Moreover, the use of RCFTs may be limited because they often require much time and expense in comparison with other approaches, or they may be precluded by ethical considerations.