parent to other services in the community. This means that families enrolled in the same program were, in practice, experiencing different interventions.
ROLE OF RESEARCH
Most individuals who are involved with home visiting programs agree that it is important to assess the effectiveness of home visiting as a strategy for service delivery. Research is integral to home visitation. Many programs have been evaluated individually and summaries of research have been compiled (for example, Olds and Kitzman, 1993). A recent addition to this work is a meta-analysis by Mark Appelbaum and Monica Sweet, of the University of California San Diego (Appelbaum and Sweet, 1999). Appelbaum and Sweet combined and analyzed the results of evaluations of 60 home visiting programs that (1) existed in the United States between 1965 and the present, (2) worked with normally developing children, (3) used home visits as its primary means of service delivery, and (4) could isolate the effects of home visiting.
The meta-analysis was intended to summarize the state of evaluation results of home visiting, and to indicate where research focus might be placed in the future. Because home visiting programs work to help children through their parents, both child and parent outcomes were studied (see Box 1 for a complete list of the outcomes). Child and parent outcomes were, on average, slightly larger for home-visited groups than for control groups; effect sizes for 7 of the 10 outcome groups significantly differed from zero (range: −042 to .481). There was some evidence that the estimated effect sizes were larger for programs evaluated using quasi-experimental methods than for programs evaluated using random assignment study design. The reasons for this are unclear. It may be the result of unmeasured preexisting differences between the program and control groups in these studies, or because of some other factor that distinguishes these two groups of programs. Appelbaum and Sweet cautioned, however, that actual effect sizes were small to moderate and may not translate to practically significant improvements for home-visited children.
Because programs were so varied and numerous, Appelbaum and Sweet were also able to investigate whether or not program characteristics influence program effectiveness. Results of these analyses were mixed. Often, programs did not enhance those outcomes corresponding to their primary goals. In addition, there was no clear and consistent evidence that type of staff, child age during intervention, program length, or the intensity/frequency of the intervention were related to effect sizes in a systematic way. The findings regarding program intensity contradict other reviews of the literature (see, for example, Heinecke et al., 1988; Powell and Grantham-McGregory, 1989) and evaluations of individual programs (for example, ongoing analyses conducted by Burchinal, Brooks-Gunn, and Lopez discussed below) and may reflect some of the problems associated with comparing results across very disparate programs.
One of the messages from the meta-analysis: those involved with home visiting programs need to decide what standards of evidence should be used when assessing the effectiveness of programs. At this point in time, there is little consistency between individual home visiting programs regarding the match between program goals and outcome measures, and no consensus on the magnitude of outcomes that warrant labeling a specific program as successful. This makes evaluating programs difficult. In addition, despite extensive programmatic experience, it is difficult to make definitive statements about such issues as how much training is needed or which type of staff (professionals or paraprofessionals) is best, due to a lack of systematic efforts to evaluate components of home visiting. Workshop participants agreed that home visiting programs need to set standards for themselves. Having such standards would help researchers conduct and interpret evaluations, as well as help staff, program participants, and policymakers know what to expect from a program.
The choice of research methods is an essential component of the debate concerning standards of evidence. The speakers and participants at the workshop voiced a range of opinions about the most effective approaches for assessing home visiting interventions. Some argued that randomized trials are the best way to test the causal connection between a service program and outcomes. They provide the greatest (though never complete) assurance that program effects are due to the program rather than to some other influence on or characteristic of participating families. As such, randomized trials give evaluations of home visiting programs scientific credibility. However, speakers also cautioned that randomized trials are difficult to implement and are sometimes not well designed; that differential attrition from control and program groups poses challenges to the initial randomization and must be carefully considered in all analyses; and that they do not address every important question about interventions. Some at the workshop even suggested that the process of conducting random assignment of families to control and program groups perverts the goals and therefore the effectiveness of home visitation.
The workshop participants generally agreed on the importance of putting decisions about evaluation methods in the context of two questions: (1) Given the question that is being asked,
what is the best method for answering it? (2) What is the nature and scope of the policy decisions that the research is guiding? Not all questions require randomized trials, such as questions about the factors that affect parental engagement in home visiting. As Mary Wagner, of the Center for Education and Human Services at SRI International, pointed out, those involved with home visiting need to decide on a good combination of research designs to get a full picture of how these programs are working. Robert Granger, of the Manpower Demonstration Research Corporation, highlighted the practical necessity of sometimes reserving intensive (and therefore expensive) research designs, such as randomized trials, for “big deal” policy decisions, such as those regarding whether and how much to invest in a particular strategy or program. In other words, the choice of research methods should be linked to the magnitude of the resources they will require relative to the magnitude of the public resources that are at stake in the decisions the research will inform. Other designs that demand less of programs, such as case studies or longitudinal follow-up of individual programs, may be both adequate and appropriate for addressing more fine-grained decisions about such things as, for example, implementation issues or the role of community context.
Discussion of other aspects of research methods included the staging of research, the importance of longitudinal studies, and the need for greater collaboration between researchers and practitioners. David Olds, of the University of Colorado at Denver and the Kempe Prevention Research Center for Family and Child Health, and designer and evaluator of the Nurse Home Visitation Program, pointed out the importance of staging research, which involves doing focus groups, pretesting, and pilot work before executing the main evaluation. For example, the evaluation of the Nurse Home Visitation Program in Elmira, New York, included a year of work prior to initiating the main study, including a small-scale, randomized trial to “make sure that they were on the right track.” A study this comprehensive is rare, however, not only for home visiting but for early interventions in general.
Concerning longitudinal studies, Appelbaum commented that those targeted for home visiting interventions are not an easy group to help and one should not be surprised at an absence of significant evidence of program effects after a short period of time. He suggested that looking at trajectories of change over time may reveal more dramatic results because small initial differences can set parents and children on a different course that only becomes more evident later in development.
Workshop participants, in a repeated refrain, also encouraged researchers and practitioners to work together to build evaluations into programs from the beginning. For example, Tammy Mann, of the Early Head Start National Resource Center based at ZERO TO THREE: National Center for Infants, Toddlers and Families, emphasized the importance of developing methodologies so that program staff are constantly thinking about what they are doing and how they are doing it. She noted, “It seems that far too little attention has been focused on developing research designs, methods, and measures that are valued, respected, and understood by program managers so that they can become increasingly able to naturally incorporate both quantitative and qualitative evaluation strategies into their programs and, in turn, make necessary refinements in program design.”