Learning from Experience: Evaluating Early Childhood Demonstration Programs (1982)

Below is the uncorrected machine-read text of this chapter, intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text of each book. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.

The Health Impact of Early Childhood Programs: Perspectives from the Brookline - Early Education Project Melvin D. Levine and Juclith S. Palfrey In the evaluation of early childhood programs the health status of the children enrolled is often a prominent issue. Planners and policy makers are likely to ask whether participation in such programs enhances children's health and, if so, whether the gains are substantial enough to justify the costs. In early childhood programs for which improved health is not a primary objective, program planners may want to know if the addition of a health-monitoring component would be cost-effective. As evaluators survey early childhood programs and their impact on health, they may consider the possibility of modifying the content of traditional preventive health care. They may examine the feasibility of collaborative service models that include the consoli- dation of early education and preventive pediatrics, so that communities can shape and upgrade simultaneously the health care and developmental monitoring of children. At first glance it might seem that the health of children is easily amenable to evaluation and measurement. Accurate numbers, however, are difficult to obtain and are often misleading. In this paper we outline some salient clinical and methodological issues that have become appar- ent to us in working at the Brookline Early Education Project analyzing the health impact of a comprehensive early childhood project. The first section of the paper delineates a number of critical issues facing evaluators. The second section discusses the scope of "health" by detailing the various background and process factors that need to be considered during evaluation. The third section outlines specific questions evaluators can ask as they measure the impact of health. Finally, drawing on our experiences with the Brookline Early Education Project, the last section 57

58 reviews seven evaluation prototypes and discusses matching alternative evaluation strategies to specific questions. RELEVANT ISSUES FACING EVALUATORS Defining Health and Its Borders Health is more than the absence of disease. It is the absence of handicap, social and emotional discord, and environmental stress as well as the presence of resil- iency, stamina, and homeostasis. There is growing aware- ness that traditional pediatric health cannot be viewed apart from psychosocial, behavioral, developmental, and educational status (Richmond, 197S; Rutter et al., 1970; Haggerty et al., 1975). Functional health and its promotion have increasingly become the purview of the pediatrician working in conjunction with professionals from other disciplines (Levine, Brooks, and Shonkoff, 1980). Health issues are likely to involve other areas. For example, it is essential for those managing children with musculoskeletal defects to address the functional (i.e., gross and fine motor and psychosocial) impacts of such handicaps. Professionals helping neurologically impaired children must involve themselves in the assessment of higher-order cognitive function, self-esteem, behavioral adjustment, and related family issues. Health maintenance must include anticipatory guidance and counseling, for patterns of behavior are as much within the domain of child health as are infectious diseases and specific organ disorders. Table 1 samples the broad spectrum of child health disorders. So many factors are involved that it is easier to describe what should be included under the rubric of "health" than to isolate issues irrelevant to health maintenance. Describing Health Status Characterizing the health status of groups of children is even more difficult than characterizing individual health. Since universally acceptable scoring and weighting systems do not exist, the health evaluation of a cohort enrolled in an early childhood project can be costly to obtain and difficult to interpret.

59 TABLE 1 Some Negative Health Outcomes Whose Effects Early Education Projects Are Intended to Minimize 1. Poor growth and/or nutrition 2. Sensory deficits 3. Chronic illness and symptoms 4. Poor utilization of health services 5. Recurrent trauma 6. Neurological disorders 7. Neurodevelopmental dysfunctions 8. Psychosocial mobility 9. Mental retardation/multiple handicapping conditions 10. Life-threatening diseases A group's health status cannot be presented as a simple inventory of existing symptoms and conditions. Evaluators who wish to characterize the health status of a group of children need to take into account past medical events, family history, and current health. In addition, there must be estimates of vulnerability and resiliency, descriptions of health practices and knowledge (nutrition, exercise, and total environment), and accounts of medical service utilization. Describing Health Change After defining the limits of health as a subject matter and developing the descriptors to characterize group health status, evaluators must find measures of health change. This can be particularly challenging in the preschool child, as the morbidity itself evolves with age and many of the dysfunctions and disorders are self-limited or transient.

60 Sometimes changes in health are more apparent than real because different measures are used at different ages. Certain reflexes, for example, can be elicited in children who are three to six months old but not thereafter. Hearing and vision are easier to measure after three years of age than before. ~~~ Some orthopedic dit-ticult~es may not be evident until a child attempts to walk. Children may have immunity to certain diseases at specific ages but not beyond. The descriptors of the health of children thus depend to some extent on which window one looks through at what time. Just as problematic for evaluators is the high prevalence of self-limited acute or subacute illnesses and the spontaneous health resiliency of young children (single, 1964; Miller et al., 1960). This makes it difficult to study both the occurrence of and recovery from acute disease. It may be particularly hard to attribute symptom abatement to treatment effects. Behavioral and developmental disorders of early child- hood reveal considerable instability over time as well. Although some researchers (Thomas and Chess, 1975; Taft, 1978) have suggested that behavioral characteristics may be maintained from infancy through childhood, others (Bell et al., 1971; Carey et al., 1977) have demonstrated that children who have "behavior problems" at age two or three may not be the same children who have difficulty in school. It may be impossible to identify precise endpoints of health change. For example, one may not be able to determine whether a child has had one prolonged episode of otitis media that never really healed or multiple ear infections (Giebink and Quie, 1978). This difficulty impedes any precise accounting of numbers of acute illness episodes during a given period. The measurement of health change is complicated for three reasons: (1) the actual content of health and morbidity evolves with age, (2) many conditions undergo spontaneous remission, and (3) some disorders are closely associated with others and are therefore indistinguishable from one another. Therefore, before looking at the impact of a program on health, evaluators should develop appro- priate methods of characterizing change: The health (of a group or individual) may vary depending on the period of time under scrutiny. The measures of health should therefore be dynamic, depending on the age and development of the children in a program. The measurement of health "progression" must somehow be differentiated from normal

61 chronological change. To document enhancement, evaluators must show that a particular child or group of children at the end of two years improved in overall health character- istics. As difficult as this may be, such documentation stands as a critical requisite for the evaluation of health as a progressive phenomenon. Dealing With Low Prevalence Rates in Pediatric Morbidity Pediatric illness differs fundamentally from its adult counterpart. In the latter, a relatively small number of major illnesses (e.g., hypertension, obesity, coronary heart disease, cancer, and diabetes) are likely to be highly prevalent within a population. Evaluators of adult health programs may be able to measure the impact of a program on these distinct entities and thereby generalize about health status and program-induced change. In contrast, there is no single chronic organic condition of childhood common enough to scrutinize in such a fashion without a very large sample. Therefore, in studying chronic medical conditions in a service program for chil- dren, it is often necessary to employ aggregate ratings that "lump" children with such disparate conditions as congenital heart disease, juvenile rheumatoid arthritis, nephrotic syndrome, recurrent urinary tract infection, and asthma for purposes of analysis. The alternative is to use samples that are too large for most early childhood projects. Resiliency in childhood also differentiates child health from adult health. Most child health setbacks are likely to be acute and self-limited, leaving no scars or aftereffects. Cataloging of such events shows that in a single year most children have been ~sick" as many as four or five times (single, 1966). For children such morbidity is par for the course, developmentally appro- priate, perhaps immunologically necessary, and ultimately inconsequential (Mortimer, 1968). Evaluators thus need to weight acute self-limited disease very differently in children. Absence of Data on Normalcy In assaying pediatric health status, we are hampered by a paucity of data on normalcy and normal variation.

62 While information does exist regarding the prevalence of specific chronic diseases or congenital anomalies and the incidence of some acute illnesses, this information tends to reflect major social class differences (Morris, 1979), serious problems with reporting (Brewer and Kakalik, 1979; Bureau of Education for the Handicapped, 1979), and inadequacies in many of the measuring techniques (Balinsky and Berger, 1975). ~ - This distortion makes it especially difficult to determine if the health status of a partic- ular group of children is below or beyond what ordinarily might be expected. Normative data are even more deficient in assessing developmental status, behavior, family functioning, and health care utilization patterns. Much of traditional medical research has had the benefit of normative data. For example, it is possible to study the effects of a medication on a patient's glucose level, since norms for blood sugar are available. For many of the aggregate measures of community child health, however, no such norms exist (Starfield, 1974). What is "normal" or "to be expected" for a particular population must almost always be reestablished in undertaking evaluation research. In many instances this requires the use of comparison groups or control populations. Selecting Outcome Measures In documenting program effects on health, one critical issue is the precise outcomes to be measured at designated outcome points. One might be tempted to consider only the prevalence of morbidity as an outcome measure. This, of course, leaves out such issues as parental health knowledge, patterns of use, and children's health stamina. Furthermore, it is unlikely that the straightforward rendering of morbidity statistics constitutes an adequate reflection of project effectiveness. An early support project may be beneficial without diminishing the prevalence of a disorder. For example, one would not expect a program to lessen the occurrence of myopia, yet an effective effort might result in a reduction of previously undetected or untreated nearsightedness. In fact, an early childhood project may exert its greatest effects not on prevalence figures but on awareness, management, coping, and the prevention of complications. Another issue in selecting outcome measures is the need for a method of weighting. Health outcomes can be measured in terms of their severity, their impact (on

63 function, on families, on society), and their relevance, so that composite morbidity may be subdivided into significance for treatment (prescriptive implications) and potential for impairment of future health or function (predictive weight). Even a low prevalence of disorders that are likely to thwart academic function or behavioral adjustment may be more important than a high occurrence of such disorders as flat feet. Evaluators might also want to select outcome measures that have significant implications for treatment. A project should be judged more harshly if it missed problems that were treatable than if it overlooked those for which no therapy was available. Outcome measures should not be too global, particularly with regard to developmental and behavioral assessments. A project that uses IQ as an outcome measure will not be pinpointing the prevalence of problems with attention, language, or other isolated information-processing deficits that can seriously impair function (Levine, Brooks, and Shonkoff, 1980). Similarly, the results of a developmental screening test by themselves are unlikely to be sufficient to describe a project's impact on children's development (Meter, 1973; Meissels, 1978). A more comprehensive picture would include parental reports of function, direct observations of behavior, or specific teacher accounts of skills, abilities, and interest. One challenge for those evaluating early childhood projects is the identification of measures that can be used to provide an in-depth assessment of function to determine whether the program has diminished or minimized the effects of so-called low-severity, high-prevalence dysfunctions of childhood, which include specific learning disabilities, primary attention deficits, and various forms of psychosocial maladaptation during the school years. In delineating outcome measures, evaluators should consider the objectives of the project under scrutiny. For instance, if health is a high priority of a given project and if assurance of primary care is an explicit goal, then it is appropriate to determine if the project has met that goal by assessing patterns of health care utilization of the enrolled children. This approach may be less relevant in projects that have had only an incidental commitment to health. Finally, within a given population uniformity of outcome measurement may be unrealistic and inappropriate, especially in programs emphasizing the individualization

64 of services. To measure gains, evaluators may need to specify "target subgroups. n For a subgroup with hearing deficits, incremental growth in vocabulary may be a better measure of program effectiveness than mean developmental scores at age five. Evaluators can identify areas in which they would expect or hope to see progress for particular children. _ _ _ They might also weigh expectations against accomplishments. For example, a project may not be able to diminish the prevalence of problems with short-term memory in children, but it might be able to achieve a generalized improvement in the reading proficiency of children with short-term memory problems, a gain that would surpass what would be expected for nonparticipating children with this developmental dysfunction. Thus, although a project may not diminish the severity or prevalence of short-term memory problems, it may manage to have "better Hopers," more competent readers, and perhaps happier children within this target subgroup. In some cases a project may want to evaluate only specific target groups to demonstrate program effects. When an evaluation becomes this focused, however, either large numbers or elegant small sample designs are needed to demonstrate that intervention has been successful. - Assessing the Cost-Effectiveness of a Health Program Evaluation A major challenge exists in the calculation of a cost-benefit ratio for health program evaluations. There is constant pressure to balance the expense and difficulty of acquiring a particular set of data against its ultimate value for children and its relevance to the objectives of a project. It may be simple to determine immunization rates for a particular population, but if the project is located in a town where most children are well immunized anyway, despite its economy this will not be a useful way of measuring program impact. On the other hand, if a service model is likely to improve a family's ability to . . . . . . cope with behavior problems, a series of expensive measures of behavior and parenting may be most relevant (Haggerty et al., 1975; Roghmann et al., 1973; Haggerty, 1965). There can be no one set of criteria for evaluating all projects. Those aspects of health chosen for evaluation will depend largely on the nature of the community, the objectives of the program, the

65 availability of evaluation funds, and current public policy questions. Identifying The Evaluation Consumer In designing an evaluation of health outcome, it is essential to understand the needs and priorities of those for whom the evaluation is intended. The content of an analysis depends largely on its intended audience. It can be argued, however, that all evaluations should be able to undergo some degree of rigorous scrutiny, even if intended primarily for nonacademics. Purely anecdotal reports and testimonials are inadequate measures of health care provision under any circumstances; highly esoteric statistical analyses, on the other hand, may have little impact on school committee decision making or on the deliberations of legislators. Often evaluations must apply several formats, each designed for a unique constitutuency. Timing and Staging A critical issue for evaluation is timing. The health effectiveness of a program can be documented while it is in progress. Alternatively, one could consider assessment of its impact at the end of a project or at a short or long interval following termination. Decisions about timing must consider what is being measured and demon- strated. If a major goal is to minimize morbidity and suffering and to cushion the traumatic impact of daily events and environments, then it is crucial to offer evaluations while the project is in progress. If the goal is to look at the long-range effects of intervention or general service, postintervention analyses are needed. Findings inevitably reflect the timing of an evaluation, and the implications can be great, especially for issues of cost-effectiveness. Because of the instability of health conditions in childhood and the high degree of resiliency, the timing of evaluations significantly influences the attribution of program effects, which can be misleading from a public policy viewpoint. For example, if children in a particular program have less difficulty adjusting to the first weeks of kindergarten than nonparticipants, evaluators may feel that they have documented a measurable effect. It may turn out, however,

66 that the two groups are virtually indistinguishable if they are evaluated in mid-February of the kindergarten year. Allocators of resources would wonder if great expenditures to promote "educational readiness" were really worthwhile if their effects were demonstrable only during the earliest months of kindergarten and thereafter washed-out. Traditional health issues follow a similar pattern. If a child's flat feet are detected in an early childhood project, but the child has no pain or functional limita- tion and it is documented that the finding would otherwise not have emerged until the first or second grade, what has been gained? In any event, by the time the child is eight or nine, the parents are likely to be aware of the condition. In that case, what is the value of early detection? Assuming that the condition is discovered early and the child given corrective shoes, does it really make a difference (Bleck, 1971; Cowell, 1977)? In some cases it may be better not to diagnose a problem that is going to resolve itself or that may not cause symptoms or require treatment for several years. It may be that years after a program ends there is little difference in the prevalence of previously undiagnosed findings. One might argue that ultimately the important problems will be detected. There is a danger that early detection may obligate programmers to unnecessary expenditures for interventions, especially for conditions that are likely to remit spontaneously. Once again, it is important to review the objectives of a project. The timing of an evaluation of program effects should relate to the objectives. In stating objectives there should be some consideration of the anticipated or desired duration of effects. Outcome measures can then be timed to assess these accordingly. Having delineated these issues for the evaluation of program effects on health, we now turn to a more detailed examination of the measurement of health status. The next section describes this as a necessary step in demon- strating the influence of a specific program on health. THE SCOPE OF CHILD HEALTH As early childhood programming expands, the literature from education (e.g., U.S. General Accounting Office, 1979; Lazar, 1977; Bronfenbrenner, 1975; White, 1975; Zigler and Valentine, 1979) and pediatrics (e.g.,

67 Richmond, 1975; Thomas and Chess, 1975; Taft, 1978; Carey et al., 1977; Roghmann et al., 1973; Klaus and Kennell, 1976; Morris et al., 1976; Badger et al., 1976) makes it clear that the scope of child health that can be addressed is very wide. Clearly, evaluators should focus on those aspects of a health program that are most likely to reveal efficacy. In our work at the Brookline Early Education Project (Pierson, 1974) and the school clinics at Children's Hospital Medical Center in Boston (Levine, 1979), we have found it most helpful to define (1) the background health characteristics of the children, (2) the ongoing health and developmental processes at home and in the program, and (3) the outcomes that the program intends to achieve. Only with these areas clearly defined does it become possible to address specific questions regarding program impact. Background Variables From the intrauterine period onward, the experiences of children vary significantly. Some endure prenatal and postnatal trauma, some are born into impoverished and disorganized families, some inherit genetic disorders, and some fail to receive adequate nurturance. Others, because of their constitutional makeup, never adjust optimally to their milieu and continually hunger for greater satisfaction from it. Still others arrive with ease, cope readily, and manifest little or no disability in dealing with the external world. Programs dealing with young children generally take these variations into account for staffing and pro- grammatic reasons. To do so for outcome measurement is equally critical. Children who are more "at risk" will require greater levels of service; their outcomes may turn out to be excellent, but they differ from children not at risk. For instance, the most pertinent outcome ma F^' ~ mid ~-~1 ~= serf child might be ease with 611 ~ ~ ~ ~ ~ =- ~ ~ ~ ale ~ ~ - . a hearing aid, skill in using a total communication system, and ability to attend a normal school for at least some of the day. On the other hand, outcome measures for a normally hearing child from a socio- economically depressed and disorganized home might be assurance of primary health care, money for food, and an adequate after-school, supervised program. Because the health needs of children are so varied, programmers, monitors, and policy makers should keep

68 sight of "targeted" outcomes. Figure 1 illustrates diagrammatically the idea that individual children require customized health programming. Some entering an early childhood project need little more than routine preventive health care. Others, because of early negative factors, need problem-focused surveillance, while still others demand active intervention. Obviously, those in the preventive or surveillance group may require more active intervention at ~ Chant point during the project. Among the most common risk factors are perinatal stresses, genetic predispositions, low socioeconomic status, and negative critical life events (setbacks). Each of these needs to be assessed separately, since they may have a differential effect on outcome. Perinatal Influences Major perinatal complications have been shown to place children at risk for developmental sequelae. However, there is currently considerable debate in the perinatal literature about the ability to predict dysfunction from perinatal catastrophes, especially in the wake of advanced intensive-care technology (Alberman and Goldstein, 1970; Davie et al., 1972). At most, one can say that a child who sustained prenatal or postnatal trauma or illness may be at higher risk of developmental dysfunction in the future. Those most likely to fulfill such predictions are newborns who weigh less than 1,200 grams (Stewart et al., 1977; Kopelman, 1978), those who suffered intra- uterine growth retardation, the so-called small-for-dates babies (Neligan et al., 1976), those born in outlying hospitals who were transported to regional centers for intensive care (Cassady, 1975; Chance et al., 1973), and those identified in the first few days as neurologically impaired (Nelson and Ellenberg, 1979). While there is still much to be learned about the connections between perinatal problems and later outcomes, early biological events should be recorded so that out- comes can be measured against them. This is most true for children who are in the double jeopardy of early physical stress and socioeconomic hardship or deprivation. A number of studies have shown that these children are at considerable risk (Werner et al., 1971; Institute of Medicine, 1973; Knobloch and Pasamanick, 1966; Sameroff and Chandler, 1975), and their health outcomes definitely

69 Background Variables a) Perinatal Status b) Genetic Handicaps c) Social and Environmental Factors d) Critical Life Events Entry \ Point ~ . ~ / / Normal Preventive Needs Early "Outcomes" a) Temperament/Behavior b) Devel opment c) Health / 1 Type of Service Need Special Surveillance Needs The Outcomes Active Intervention and Special Preventive Needs FIGURE 1 Customized health programming for children in an early childhood project. can be used as a targeted index of a given program's total performance. A major problem with the literature on the relationship between perinatal trauma and later life has been the lack of uniform outcome measures. Studies have used different chronological endpoints, including 1 year (Fitzhardinge, 1975; Goldstein et al., 1976), 4 years (Broman et al., 1975), 7 years (Davie et al., 1972), and 10 years (Nelson and Ellenberg, 1979). Most investigations have applied  developmental (Tilford, 1976) or intelligence quotients (Fitzhardinge, 1975; Broman et al., 1975); some have inspected functional outcomes, such as school performance (Rubin et al., 1973). Many have accepted neurological disability as the norm for children with perinatal stress

70 and thus have reported surprise or delight when a relatively small percentage manifest such problems (Stewart and Reynolds, 1974). Others have assumed that any neurological or learning disability, even of the most subtle degree, is a negative outcome and have therefore reported large percentages of abnormal-consequences (Rubin et al., 1973; Fitzhardinge and Steven, 1972). It will be important for program monitors to be aware of these vagaries of criteria and to be as explicit as possible when determining which outcomes to follow and where to draw the lines. Genetic Disorders Genetic diseases, such as Down's syndrome, other chromosomal anomalies, phenylketonuria (PKU), and inborn errors of carbohydrate and lipid metabolism are known to predispose children to poor health and developmental outcome (Milunsky, 1975). Early education projects, particularly those offering "infant stimulation" are often designed to help such children (Hayden and McGuiness, 1977; Bricker and Iacino, 1977). It is crucial that entry characteristics on genetically handicapped children be registered and the natural history of their disorders well understood. For this group, targeted outcome measures are appropriate (Tjossem, 1976). As an example, children with treated PKU have been shown to have significant weaknesses of perceptual motor function that are disproportionate with their overall intellectual levels (Koff et al., 1977). A global cognitive index (such as a standard IQ test) used as an outcome measure would fail to assess the impact of a program designed for early intervention for such children, whereas a specific look at perceptual motor functioning would do so. Likewise, any assessment of a Down's syndrome program should gear the outcome standards for growth and development along a developmentally appropriate scale (Smith and Wilson, 1973). Those analyses that have shown the effectiveness of early intervention have used a targeted approach, and the evaluators have been familiar with the natural history of the particular disorders (Horton, 1976). Finally, in the case of genetic disabilities, outcome measures should be designed specifically to address the question: Did this program succeed in preventing secondary disability in these children? In the words of

71 Meter (1975:386), who has written extensively on this subject, ". . . the prevention of DO (developmental disability) is also a relative phenomenon, in the sense that the prevention of further disability through early intervention is still prevention of otherwise inevitable further deterioration, although the total disability may not be lessened or fully compensated. The specific genetic inheritance of a DD person may establish certain ceiling limitations for growth and development, but even those lowered ceilings will probably not be reached without appropriate intervention." Socioeconomic and Environmental Influences One of the strongest predictors of school performance is socioeconomic status. Furthermore, there is a disproportionate amount of illness among poor children (E. Newberger et al., 1976; National Research Council, 1976). Head Start and other early childhood programs have incorporated health components specifically for this reason (Richmond, 1966). Documentation of the socioeconomic status of children within a given project is important for service allocation and allows those who are monitoring the program's efficacy to determine how well health goals have been met for the disadvantaged and to identify gaps that remain to be filled by health and welfare agencies. Knowing that a proportion of children within a program are at a socioeconomic disadvantage, program monitors need to assess sources of primary medical care (Haggerty, 1976; Harvard Child Health Project Task Force, 1977), dental care (Gortmaker, 1979), nutritional adequacy (C. Newberger et al., 1976; Folman, 1977), and home safety (Taylor and Newberger, 1979). While these are outcomes important for all youngsters, they are particularly salient for poor children. Experience with Head Start and Follow Through has indicated that programmatic gains tend not to be sustained if children cease to receive stimulation. Caldwell's group has pointed to the fact that the extent of home participation can be measured systematically (Elardo et al., 1975). For those programs enrolling children at high socioeconomic risk, outcome measures designed to estimate the extent to which there has been family participation and family growth through the program are increasingly recognized as of major benefit (Zigler and Valentine, 1979; Richmond, 1966).

72 A goal of the Brookline Early Education Project has been to teach parents an advocacy role vis-a-vis schools and other institutions. Mothers and fathers within the program have indicated to interviewers they feel that their increased competence in working with and advocating for their children will promote a healthier environment (Weiss, 1979). For programs working with high-risk socioeconomic groups, parental competence and advocacy could serve as a powerful outcome measure; such measurement, however, remains elusive. Other environmental influences, such as family intact- ness, the quality of housing, patterns of nurturance, and cultural milieu, are likely to be important to document and consider in individualizing services. In evaluations, such data can help document project impacts on specific subgroups. Health and Development Over Time While the entry characteristics mentioned above are known to be related to some of the negative outcomes listed in Table 1, community accountability is not limited to those children whose vulnerability is readily identified in the first few months or years. Close community surveillance is justified by the findings of Smith and Phillips (1978) that 45 percent of a group of severely developmentally delayed children (excluding Down's syndrome) were not identified until after they were 18 months old, and 16 percent of these 131 children were not diagnosed as handicapped until they were 49 to 60 months old. Child health and development are characterized by trends and flux. The fact that a child has pneumonia once may have almost no significance, but if he or she has pneumonia multiple times, a serious immunodeficiency lung abnormality, or cystic fibrosis may be involved. Similarly in development, a child may not walk until 18 months of age but then progress normally or may walk late, talk late, and be cognitively delayed. In the Brookline Early Education Project, among a "community" of nearly 300 children, the importance of trends was underscored by an analysis of risk status during the first six months of life. Risk groups were defined on the basis of physical, developmental, neuro- logical, and perinatal findings at three separate time points: two weeks, three months, and six months.

73 Considerable instability of membership in the risk groups was demonstrated, indicating that early childhood projects that admit only children at risk should have flexible admission criteria and that project directors must be aware of the constantly emerging service needs of children outside the project as well as those enrolled at a specific time point (Levine, Palfrey, Lamb, et al., 1971). A parallel analysis of temperamental characteristics from birth to six months indicates the same instability of findings, suggesting that enrollment based on tempera- mental findings should perhaps be flexible as well (Kronstadt et al., 1979). One way of using trends is to look at the changes in children at both ends of the spectrum. This has been the analysis design of the Educational Testing Service studies on Head Start. Using this design the group has been able to isolate factors responsible for the maintenance of the good effects of Head Start and to begin to make recommendations for specific program components (Shipman, no date). Without an analysis of trends, this could not have been done. As programs are analyzed, trends must be kept in mind. The health and developmental factors that should be followed carefully over time include health status, sensory abilities, temperament and behavior, and developmental performance. In addition to trends, certain individual health events can have detrimental effects on child outcomes. It is important, therefore, that these be registered if and when they occur. Of specific importance because of their known effects on the central nervous system are meningitis and encephalitis, major head trauma, and life-threatening diseases. Furthermore, it is important to document "critical life events that can affect a child's develop- ment. Such events as parental separation, divorce, or death may set a child's development off course and help explain an intermediate or ultimate outcome. Other critical events, such as a move or birth of a sibling, may have temporary but significant impact (either positive or negative). Finally, the documentation of abuse and neglect is important as one assesses health outcomes. Those projects that have been family-centered should be able to document fewer episodes of domestic turmoil and abuse than would be otherwise predicted. Acute health and critical life events such as those listed above pose a problem for those who are documenting outcomes, since they do not necessarily occur at the

74 beginning or the end of the project, and they have little regularity. Despite these issues, major acute events often carry with them effects on the lives of children, and careful documentation of them may help sort out important health outcomes. "Graduates Health Profile Although evaluators should be judicious in their choice of timing for analysis, an overall perspective can be obtained at the end of a program, as assessment is made of the extent to which initial health objectives for children were met. For example, the goals of the Brookline Early Education Project (as conceived by the superintendent of schools) were that no child would arrive at kindergarten with any undetected health or developmental problem and that remediation would be in place prior to school entry for children with problems (Pierson, 1974). Table 1 is a checklist of such detect- able problems (a negative outcomes list), which evaluators of early childhood projects might want to use as a conceptual "graduate health profile. In dealing with early diagnosis and intervention, it is essential that project planners and evaluators be fully aware of the negative outcomes likely to occur during the school years and whose effects the early projects are intended to minimize. In planning service and evaluation in an early childhood project, it is therefore helpful to identify those children who appear to be at risk for such disorders (Oberklaid and Levine, 1980). That is not to say that all such children are in fact identifiable during the preschool years--some may be, others may not. Moreover, children who are likely to have learning disabilities or other kinds of subtle dysfunctions during the school years may not manifest them until they are challenged with specific kinds of academic or cognitive tasks. In addition, although some children may appear to reveal dysfunction during the preschool years, they may function normally later on. At the beginning, during, and after an early education program, it is helpful to have a good descriptive account of the developmental health of those in the program. It is clear that with increasing knowledge of subtle, "low- severity, high-prevalence" handicaps of childhood, developmental descriptions of children should go beyond the simple, traditional milestones.

75 Listed below are examples of negative health outcomes, some of whose antecedents may be detected, described, and treated in an early childhood project: 1. Poor Growth and/or Nutrition Children suffering from poor growth or nutrition may stand out in a group setting more than they would in the family. Danger signals of growth failure, poor eating habits, or emotional deprivation may be apparent to those involved with the child on a day-to-day basis. 2. Sensory Deficits Hearing and vision defects, which impair function in school, can be readily detected in early childhood (Palfrey et al., 1980; Strangler et al., 1980). Such deficits, however, are often acquired after school entry. 3. Chronic Illness and Symptoms Chronic illness affects childhood performance in multiple ways--not the least of them being the loss of school days. The burden of chronic disease in children is often accompanied by major psychological and social problems (Sultz et al., 1972). Furthermore, children who need to undergo major or recurring hospitalization may suffer functional setbacks as a result. Although there is little a project can offer to prevent most chronic disease, there are a variety of strategies that can be used to help children cope with or control symptoms. 4. Poor Utilization of Health Services ~- Although it is now rare for children not to have access to primary care, there are still demographic pockets where health care provision is inadequate (E. Newberger et al., 1976; Lowe and Alexander, 1974), and it is exactly these children who are at greatest risk on all health and social factors. 5. Recurrent Trauma The child who suffers from multiple accidents may be permanently impaired. Further- more, the injuries may have been inflicted intentionally and therefore indicative of serious family and social pathology (Smith and Simpson, 1975; Kempe and Helfer, 1972). 6. Neurological Disorder Neurological disorders can be detected and managed, but it would be unreasonable to expect an early education program to have a major impact on basic disease processes. 7. Neurodevelopmental Dysfunctions Children with minor necrologic markers (Wolfe and Hurwitz, 1966; Vukovich, 1968), serious attention deficits (Levine and Oberklaid, 1980), sequencing problems (Ruder and Denckla,

76 1976), motor delays (Levine, Oberklaid, Ferb, et al., 1980), language disabilities (Rutter and Martin, 1972; Wiig and Semel, 1976; Oberklaid et al., 1979; Denckla, 1978), visual spatial dysfunction (Kephart, 1971; Robinson and Schwartz, 1973), memory problems on "developmental output failure" (Levine and Meltzer, 1981), or with combinations of these symptoms are all at serious disadvantage in school. To the extent possible, early childhood programs should address these issues. 8. Psychosocial Morbidity Major negative outcomes for children in the behavioral or psychosocial sphere are disorders of personality development, affect, or self-esteem. These may interfere seriously with learning and growth and thwart optimal developmental health (Connolly, 1971; Rutter, 1974; Simmons and Tymchok, 1973). 9. Mental Retardation and Multiple Handicapping Conditions Early childhood programs have the potential of reducing the serious burdens faced by retarded and handicapped children by enhancing the normal aspects of their lives, encouraging those with handicaps to interact with other children. 10. Life-Threatening Diseases Specialized programs for children with life-threatening diseases and their families may help to ease their suffering and pain. ~, ~ With regard to health outcomes then, evaluations of early childhood programs should involve background characteristics of the population (e.g., premature versus term babies, child abuse versus normal environment); the evaluators should analyze the program variables as well as health and developmental trends; and finally, they should consider which of the 10 health outcomes their program was best suited to address; the evaluators can then move on to pose specific evaluation questions. MEASUREMENT OF HEALTH OUTCOMES an early chldhood program on children's health, the questions that can be asked depend to a great extent on the program's stated goals. For instance, one program may entail only the assurance that children are obtaining health care "somewhere"; another may strive to achieve a particular level of health care for its enrolled children; and a third When assessing the impact of program may be directly involved in the provision of some health services. Clearly, the depth and sophistication of analysis will vary with program characteristics.

77 To measure a program's health impact, evaluators can choose from a number of questions, including the following: 1. Did the project assume the completion of standard health maintenance (i.e., regular physical examinations, immunizations, and screening)? 2. Did the project assist in the detection of health problems? 3. Did the project prevent health problems? 4. Did the project's intervention help reduce the incidence or the effect of specific health problems? 5. At what cost were these health activities carried out? Some of the questions are likely to apply universally to early childhood programs, while others would be relevant only in a program that had targeted a specific health outcome. (See Table 2.) Standard Preventive Measures Assurance of Adequate Health Care A major demonstrable health contribution is access to good health care services. For example, there was dramatic, demonstrable change in the provision~of health services to children in Berkshire County, Massachusetts, with the initiation of the Berkshire Health Program (Whitfield and Walker, no date). One can document the extent to which programs have facilitated the use of available, comprehensive, and affordable health care for the children enrolled. Many authors have pointed to the fact that the fragmented system of health care in the United States has significant inequities and gaps (Institute of Medicine, 1973; E. Newberger et al., 1976; National Research Council, 1976). Specifically, many poor and rural areas are underserved medically, and, furthermore, the quality of health care is inconsistent, even in areas where sufficient personnel are available. Poor children are still more likely to receive care in public clinics characterized by unattractive physical surroundings, long waiting times, overburdened and sometimes impersonal staff, and dependence on hospital emergency service for medical treatment, whereas middle-class children are likely to benefit from more personalized private care.

78 t) a .,_, o P4 o O · - s C~ o . - . - ~: s - ~: a) s a, ~s o o ~5 ~4 o P. u) ~n tU £ ~3 m ~: E~ ~ E ^.r, ~ O O tC5 ~ O s O O  ~: ~n a O s C) s ~n o u, C aJ ~ ~ ~n ~, ~ O os C} JJ . - G) C: ~ S O 3 ~: o - ~: 4, ~4 C 4~ ~ C C) _ a' 4~ F Q ~ L. 4~ ~a o F o ~: eq e 4~ ~:: ~ a.' 4 eC G) 3 .o 4, eq 411 F C) a, a, F U) O a  C Ll ~ C ~'- ~:- ~:> :^ :~ D D , ~,Q D D D D n _ ~_ ~ 0 0 tn 0 eq 0 ~n 0 0 0 U) 0 U) ce 0 U) 0 0 0 0 0 o 0 0 0 0 0 0 0 0 0 Z Z P4 ~Z Z z ~p4 P4 ~ s s s s s s s s s q: q ~cr ~. ~, 0 0 r' 3 ~ ~ ~ ~ ~o ~0 + + + + + ' ' + + 1 1 1 + ' + + + + + + + + + + + + + + + c ~n G ~4 ~F U] F ~G, ~ ~4) 0 0 ~D ~v ~O  tn ~0 ~n ~a U) ~U] 4 ~F dJ _ qS 05 a ~4 ~0 F S ~_t C) E ~. - 4 ~C ~C ~Ll ~O ~U) C x O ~Q4 ~a, U] 0 a' ~C ~c ~c eq ~a' 0 s ~a) 0 ~0 -4 ~C E JJ Cn F ~E D J~ N . - Q' () 0 ~4 0 ~c O U) ~_I 0 ~O ~a, a_1 C 41 ' ~U) C O - ~O ~ u, ~0 ~a, ~_t 0 ._' c s E c) ~(., C a) ~So C s ~t) o. ~4 u ~a ~: ~z ~m ~u ~: o P~ D CO - o E o ,4 JJ tn a) s 4~ S 4J 0 c QJ F · - C F Ul E o a, v 0 - s v o q~ a, v .,4 ~4 o L 0 ~ · - C  C F O .,' V ~ 0 eq S V V S V CD C E + o :~ v 0 .,, s v o 4) .. V V ~ o `4 Z (24

79 Monitors of child health programs may not be able to assess the quality of care the children are receiving but they can document two important outcomes relating to comprehensiveness. First, do the children attend a clinic designed to provide continuing (i.e., not episodic) care (Institute of Medicine, 1977)? Do they have a single primary care source as opposed to multiple facilities or the dependence on emergency rooms and outpatient clinics (Levy et al., 1979)? One possible outcome measure entails "documentation of engagement in a primary care source." Our experience at the Brookline Early Education Project (where 97 percent of families have maintained such a source) has been that six-month updates of this measure have been necessary both for service and documentation purposes. Second, it is also helpful to determine if appropriate preventive services are available for children; these issues are addressed below. Besides documenting the availability and comprehensive- ness of health care, one can determine whether there exist barriers to access of care. In some cases these are physical (Harvard Child Health Project Task Force, 1977; Reynolds et al., 1976); in others, financial (Morris, 1979; E. Newberger et al., 1976; Harvard Child Health Project Task Force, 1977). The experience with Medicaid over the past 15 years has dramatically indicated that cost does stand as a major barrier to health care. Studies by the U.S. Public Health Service (1976) have shown that "in 1964, prior to Medicare and Medicaid, the poor of all ages made fewer physician visits per year than the non-poor did, but by 1974, the poor were using physician services at a somewhat higher rate than the rest of the population. n Similarly, Gortmaker has recently shown (1979:18) that the rate of dental service use is directly proportional to the availability of Medicaid and other insurance payments for such care e In addition, in California it has been shown that identification of handicapped children is directly proportional to funds for such endeavors (Office of the Auditor General of California, 1979). With the advent of Medicaid, in those states in which the program is generously funded, cost may stand as less of a barrier to very poor children than it may to lower-middle-class children. Estimates of child health expenditures per year are in the neighborhood of $300 per child. Monitors of early childhood programs may find that this places a particular burden on those families

80 not covered by insurance and on those families covered by proprietary insurance companies, such as Blue Cross/Blue Shield, that do not always pay for preventive services. In addition, children with serious handicaps may not be adequately covered by insurance companies because they are seen as too risky (U.S. Public Health Service, 1976). Financial barriers to care should be recognized and adjustments made through advocacy for young children. Enhanced access is a major health outcome for early childhood programs, particularly in the presence of geographic, financial, or other barriers. This "spin-off is of major interest to a number of audiences concerned with the provision of services to young children. Screening In some cases early childhood programs themselves may sponsor health screenings. Others may stop at ensuring that screening has been performed elsewhere for the children within their programs. In either case the percentage of participants who have undergone standard screening procedures may serve as a useful outcome marker. Frankenburg (1974) points to a number of criteria for relevant screening measures. These include prevalence, importance, cost effectiveness, and interventions avail- able. Monitors of health outcomes for children can assess the extent to which early education programs have accomplished screening in the following areas: 1. Vision The American Academy of Pediatrics (1972) has published guidelines for screening the vision of young children. Prior to age three it is not really feasible to obtain accurate measures of visual acuity, but children can be checked for structural anomalies, squint, and tumors by means of observation and history. After age three there are a number of procedures that can be applied to measure acuity (i.e., Allen, 1957; Lippmann, 1974, 1975; Sheridan, 1970). As part of the evaluation it is certainly reasonable to determine whether this service has been provided or arranged for in a preschool program. 2. Hearing A series of international conferences have recommended periodic screening of children for hearing loss (Joint Committee on Infant Hearing Screening, 1971, 1972). Although there is a highly specialized technology available to detect hearing

81 impairment in newborn infants, it is not economically feasible for mass screening nor does it address the issue of acquired hearing loss. For these reasons there is consensus that periodic screening supplemented by a thorough history is the most justifiable approach to the screening of young children for hearing loss (Palfrey et al., 1980). Documentation can Include the number or children screened for hearing and the ages at which it was accomplished. 3. Lead Intoxication Screening Early childhood demonstration projects are frequently aimed at services to poor children. A major epidemic among poor children has been poisoning from the ingestion of peeling lead paint and plaster (Chisholm, 1971; Center for Disease Control, 1975). Needleman et al. (1979) have shown that chronic or repeated ingestion of small amounts of lead can cause behavioral symptomatology. With current techniques, screening for lead poisoning is simple and accurate. One measure of the adequacy of monitoring of child health, then, might be the provision of an annual or semiannual screening for lead poisoning, especially in geographical areas of high prevalence. . . . . . _ 4. Anemia Between 2 and 9 percent of preschool children suffer from anemia almost entirely on the basis of iron deficiency. Oski and his coworkers (Oski and Hinig, 1978; Webb and Oski, 1973) have documented the behavioral consequences of such anemia. In addition, anemia is a marker of other nutritional needs. In early childhood programs, screening for anemia can serve as a possible indicator of poor health status or of family needs for nutrition education (Folman, 1977). Such screening could help policy makers determine the extent to which supplemental food programs, such as those incorporated within Head Start, have been valuable and to what extent they should be continued or augmented. 5. Sickle Cell Screening Sickle cell anemia is a serious disease affecting approximately 1 percent of the black population. For those programs serving blacks, the proportion of children who have had sickle cell testing can be used as one point for evaluation. An additional measure with regard to sickle cell screening might consist of an educational survey of parents to determine their understanding of sickle cell anemia and sickle cell trait. 6. Dental Screening A recent preschool nutrition survey indicated that throughout the United States the prevalence of caries is 2.6 to 3.8 per child (Folman, 1977). Other studies have indicated a higher prevalence

82 of carious teeth among poor children and those attending Head Start centers than among middle-class children. Dental screening is a first step in the prevention of caries, and its provision can be used to assess the adequacy of overall health services in a project. As dental screening is undertaken and recorded, a variety of policy implications emerge. As with anemia, dental status may be an indirect measure of nutritional status. Policy makers can determine the extent to which educational and nutritional services are being provided to families. In addition, a variety of staffing needs may be demonstrated, including the necessity for dental services within a particular program. Immunizations In addition to screening, preschool projects can monitor immunization status, which is one major measure of a child's health. Recent studies indicate that as many as 40 percent of the nation's children are not adequately immunized (Center for Disease Control, 1977). Reasons for this include poor health care distribution, family mobility, and noncompliance as well as recent public apathy about the importance of childhood immunization. Clearly, the documentation of immunization levels as a program outcome measure is of direct benefit to the individual program. In the larger sense, it is helpful on the local, state, and federal levels for the document- ation of important epidemiological information. In addition, a recent study by Minear and Guyer (1979) as well as a study by the Medical Foundation of Massachusetts (Gottlieb and Wechsler, 1976) have indicated that the level of immunization within a community can be greatly enhanced by close and tenacious monitoring at either the clinic or the school level. Physical Examination Completion of periodic physical examinations can serve as an outcome measure related to child health practices for a given program. The actual frequency of such assessments is a matter of continuing controversy. Local standards should be reviewed, and recommendations should be made to participants in a project. Compliance could be a useful health-related outcome.

83 Developmental Screening To date, guidelines for developmental screening are at a more primitive stage than those for vision, hearing, lead, and anemia screening (Meter, 1973; Meissels, 1978), in part because developmental screening is particularly complex and time-consuming. Those involved in the process have become increasingly convinced that the best approach is a comprehensive assessment that includes a substantial, if brief, look at the following areas: (1) gross motor skill, (2) fine motor function, (3) visual motor integra- tion, (4) receptive language, (5) expressive language, (6) memory, (7) experiential learning, and (8) behavior (Levine, Oberklaid, Ferb, et al., 1980). As an outcome measure, monitors of child preschool programs can document whether screening procedures have been carried out over the time period of the program. In addition, it may be worthwhile to describe what type of developmental assessment was used and what staff members carried out the test. The importance of such documenta- tion is that developmental screening on a large scale has not been performed in this country. However, with the Early and Periodic Screening, Diagnosis, and Treatment Program (Frankenburg and North, 1974), the Education for All Handicapped Children Act (Palfrey et al., 1978), and state laws recently generated to comply with the federal regulations, states are being asked to perform develop- mental assessments of young children. To the extent that large numbers of data about a variety of assessments can be gathered, this will help to determine what sorts of assessment are of most value. The developmental status of participants can be an important measure for a project. The demonstration of higher mean performance levels or a smaller proportion of suboptimal "scores" can be convinc- ing evidence of program efficacy. - . Beyond Screening: Indicating numbers of cases of any disorder detected in a program can help policy planners in given cities or rural areas become familiar with the major preschool health problems in their areas. One would not expect to find the same prevalence rates for all disorders in all locations. A project may need to demonstrate a reduction in the prevalence of one or more conditions in order to argue convincingly for a significant health impact. This process may be difficult for several reasons:

84 (1) existing or expected prevalence rates may not be available or obtainable, (2) a nonparticipant comparison group may be needed, and (3) reporting may be incomplete even within the program. Prevention Early childhood programs may be able to demonstrate prevention in a number of important areas, including health care neglect, child abuse and neglect, accidents, and malnutrition. These are primarily areas in which family involvement is needed and in which a clear goal must be set in order to identify effects. To document prevention, evaluators would need data substantiating specific problems in these areas prior to entry into the program (i.e., preprogram prevalence data) or a well- matched contemporaneous control group or large-scale norms for the conditions under study. Intervention Many early childhood programs are in effect early intervention programs. The children enter because of handicaps or at-risk status, and attempts are made to alleviate their handicaps or to decrease the special risk. Demonstrating the effects of intervention requires meticulous attention to background variables, program design, and outcomes. The question always in the evaluator's mind is: Would this child have been the same in the absence of the program? This question may not be thoroughly answered, but the compilation of data on similar children inside and outside a program as well as the comparison of youngsters in dissimilar programs will help evaluators judge the likely effect of a program. Equipped with a number of possible questions, the health evaluator must make decisions regarding the method of evaluation. The next section discusses seven prototypes of health evaluation. PROTOTYPES OF EVALUATION The selection of one or more prototypes for evaluation of health services is dependent on multiple factors. First, as noted above, a project is most likely to show

85 gains in areas that are consistent with its objectives. Thus, evaluation plans should include systematic scrutiny of those areas of health that were specifically targeted for prevention or intervention. Second, the choice of an evaluation prototype depends on the nature and number of resources available for the evaluation process. Certainly some of the evaluation plans discussed below are far more costly (financially and in terms of human resources) than others. Third, there may be ethical constraints on a particular type of evaluation, especially with regard to the use of comparison groups who receive little or no intervention. Fourth, the choice is influenced by the availability of adequate measuring instruments for the type of evaluation desired. If, for example, it is felt that the major health impact will be on parenting, then one may have to decide whether there are good outcome- measuring instruments for this. If not, another type of evaluation may be needed that does not require the documentation of specific outcomes. Fifth, the best type of evaluation depends to some extent on the numbers of children involved and more particularly on the kind of morbidity one wishes to assess. If the latter has a low prevalence, comparison group studies that will be able to demonstrate statistically significant differences will be hard to achieve. In general, the smaller the quantitative differences in outcome, the larger the numbers of children that will need to be involved. Six, as noted above, it is critically important to determine the nature of the constituency for whom the evaluation is undertaken. That which will persuade one audience may be ineffective or irrelevant to the deliberations of another. The prototypes we describe in this section are by no means exhaustive. Other forms of evaluation might be suggested. The seven systems described below are based on our own experience in planning and implementing an evaluation for the Brookline Early Education Project. Comparison Studies In health-related studies of outcome that use a comparison group, statistically significant differences should emerge when one compares a treatment with a nontreatment group. This methodology carries with it numerous intellectual and ethical hazards. When it is effective, a comparative methodology is likely to be the most convincing, especially to scientific or academic

86 groups. It is the most amenable to quantitative statis- tical analysis. In applying this form of evaluative strategy, the following questions need to be considered: · Should the comparison group be selected and randomized at the same time as the program group? · Should the comparison group be followed concur- rently and evaluated periodically at the same time as the treatment group? · Is it possible for the comparison group to be evaluated in a truly "blind" fashion? Or is it likely that independent evaluators will still know which children were part of a program? · Are the outcomes to be measured likely to yield relevant differences between the groups that are great enough to have statistical significance? · To what extent will the comparison group receive intervention? More specifically, if pathological findings occur during evaluations, will some form of feedback, surveillance, or active intervention be recommended, despite the fact that the children are not in the program? · Would the design be strengthened by supplementing or replacing ongoing comparison groups with cross- sectional, i.e., "nonlongitudinal," groups? To overcome possible intervention effects for the comparison group, one might want to recruit new subjects for comparison at certain points in the project. · What is one prepared to do if the comparison group and the treatment group turn out to be ill matched on various extraneous factors (e.g. parent educational level, and birth history)? How can this be avoided? , socioeconomic status, These issues are critical to the design of such evaluations. Once they are dealt with, one can proceed with the selection of the precise outcome measures desired. After selecting the outcome measures, it is helpful to develop mock tables to determine the likelihood of various kinds of outcomes and thus ensure that the numbers in the comparison and experimental groups will be adequate to demonstrate significant differences that may occur. In many instances, comparison studies need to focus on differential impacts on targeted subgroups. For example, if one wanted to study program effects on children with

87 chronic diseases or on those with sociodemographic risk factors, one would need to be certain that there were comparable and sufficient numbers of such target children in the experimental and comparison groups. The relatively low prevalence of most chronic medical problems in child- hood can certainly have an impact on the nature of the outcome study. In the Brookline Early Education Project, it was necessary to develop clusters of morbidity so that they could be measurable in sufficient quantity. For example, among the 300 children studied, there were not enough premature infants or infants born with jaundice. However, the use of a composite rating system to charac- terize a subgroup with high or moderately high levels of perinatal risk enabled us to amass a large enough group to evaluate possible program effects. Such a process does run the risk of "mixing apples and oranges" or scrutinizing artifical categories. When comparison groups are not recruited at the same time as the experimental groups, it is difficult to match them. If they are noncontemporaneous, it is likely that the children have undergone changes over time. Even with contemporaneous comparison groups, there are likely to be volunteer effects. Parents who agree to have their children evaluated without benefit of services may be a very different kind of group from those who agree to participate in a project. For this reason, whenever possible it is best to have a random assignment of a comparison group that is selected at the same time and from the same pool as the subjects in the regular program. . . Comparison group studies are most credible if the evaluators of the children are not part of the project itself and are unable to distinguish between participants and nonparticipants. This can be difficult, because often times much outcome data need to be obtained through history taking, during which it is possible, if not likely, that an evaluator will discover whether a child has been in the program Outcome Studies Without Comparison Groups In certain instances it may be possible to perform outcome studies without comparison groups. Such descrip- tive analyses can be convincing, especially if the outcomes measured are comparable with those of other studies or else face valid in general. The following example might be useful: If in a particular project one

88 of the major goals is to minimize or prevent the effects of child abuse and neglect, it might be possible to study this without a comparison group. If good data are avail- able about local community prevalence rates for such problems, then the project's documentation of a diminished occurrence may demonstrate effectiveness. One problem for such studies is the likelihood of better reporting within the project than in the community at large, which can tend to diminish the differences. Conditions such as child abuse are likely to be underreported in the nonprogram group. If the results are dramatic enough, prevalence data may not even be necessary. For example, if a project has not had a single instance of child abuse, that fact has a high level of face validity and does not require the invocation of comparison groups or other studies. In summary, if good data are available from a comparable population, or if a project is likely to have dramatic face-valid findings, an outcome study without a comparison group can be an effective and economical evaluation system. Longitudinal Study of Findings Sometimes it is difficult to draw a clear line between program documentation and evaluation. A careful account of what has occurred in a project can in itself serve as one dimension of evaluation. Early in the history of the Brookline Early Education Project we were asked a key question ot interest to Public DolicY makers: ~ "What are you finding, and what are you doing about it?" The need to be responsive to this inquiry led to the development of a method that we have called the Lonqitudinal Study of Findings (Pierson et al., 1980). It is a project-auditing system with the primary stress on a basic unit called a "finding," defined as a diagnostic observation suggesting service need. The latter might consist of direct intervention or ongoing monitoring and surveillance. An interdisciplinary team met to discuss each child as he or she passed the age of 42 months. There was an account of all findings for each child, derived from direct classroom experience, physical examinations, neurodevelopmental assessments, psychological tests, and parent reports. Each finding for each child was rated according to a series of relevant measures including severity, certainty (versus the equivocal nature of a

89 particular finding), predictive significance, prescriptive significance, prior knowledge of the finding (i.e., whether this was the first documentation of the finding), category of finding (e.g., developmental, health, educa- tional, psychosocial) and modalities of management, treatment, or observation. Each finding was subject to longitudinal tracking. After one year there was accountability for each finding, e.g., whether it had vanished, diminished, changed in character, remained the same, or gotten worse. Compliance with the treatment program also was rated. Through the Longitudinal Study of Findings a number of basic questions could be addressed: . What did you find when you looked at this popula- tion of children at a particular age? What did you decide to do based on what you found? What proportion of your findings were predictive - of later problems with health or function? . What proportion of your findings were in fact remediable or amenable to some form of treatment? . What proportion of your findings were both predictive of later problems and remediable? · What kinds of treatments did you recommend? · What proportion of your children required which kinds of intervention? · Which forms of intervention were most likely to engender compliance? · What 12-month treatment-related outcomes were seen with regard to the findings? The Longitudinal Study of Findings as used in the Brookline Early Education Project was a useful method of auditing program documentation; it has, however, like other methods, had some shortcomings. First, the study was undertaken without a comparison group, making it ~ i ff i ~.,1 t t~ Attr ibute the resolution of findings to program effects. Second, such an audit system (at least as it was carried out) is expensive, involving teams of professionals in prolonged discussions of findings. One can argue that this activity is also an important dimen- sion of service, since it requires systematic thinking on the part of the program staff about the needs of partici- pating children. Third, certain aspects of the Longitudinal Study of Findings are necessarily arbitrary. For example, the system used to classify findings is subjective. If one uncovers "hyperactivity" in a child,

so for example, is it a medical finding? Is it develop- mental? Educational? Psychosocial? In our experience many such dilemmas occurred; the formulation of precise ground rules was essential, and a detailed scoring manual was compiled. Fourth, the Longitudinal Study of Findings is essentially a pathological model. Ideally, findings should include not only problems but also identified strengths and their outcomes, although this would, of course, increase the cost of the process. Finally, to be useful the Longitudinal Study of Findings must make distinctions in the effects of findings from child to child. A particular abnormality in one child may suggest greater significance and service need than the same finding in another child. For example, a child with a language disability who comes from a deprived environment is likely to have a worse prognosis than a child with similar language delays in a more enriched milieu. The Longitudinal Study of Findings prototype can be of value to public policy makers. For one thing, it enables one to estimate personnel requirements in an early child- hood project. If it turns out that language disabilities have a high prevalence, then it may follow that the special educators or early educators in the project should be trained specifically to deal with these dysfunctions. The necessity and/or time requirements for a nurse or physician may depend on the yield of medical findings in . · . a particular project, although this will vary from site to site. The Longitudinal Study of Findings can help answer one question that is particularly germane: What would have happened to this group of children if the program did not exist? By looking at an inventory of findings within the project and by estimating their prescriptive and predictive effects, one can begin to assess the toll of neglect. One can develop an argument about those findings that would go undetected and untreated were the project not in place. One can then examine the cost to children and the community of this degree of neglect. Such data can argue for or against the value of an early childhood program affecting health. Case Argument Studies An economical and often compelling form of evaluation can be undertaken as "case argument studies." We have used the word "argument" to differentiate such evaluation processes from mere testimonials or anecdotal accounts.

91 In a case argument study one presents a series of clinical examples that include convincing evidence of the impact of the project on a child's or a family's health. The evaluators need to choose successful examples and assemble documented case studies demonstrating program effects. It may be necessary to refer to the literature (i.e., a normative base) on various subjects in order to make a convincing argument that without this project a particular child's health would have been compromised. It is not sufficient to describe programs and changes: Convincing arguments should be made that the changes related causally to program effects and that the particular findings related in the case were unlikely to be self-limited or transient. A case argument study should try to prove the fact that alternative services in the community could not deal as well with this particular child's health issues. One can liken a case argument study to an attorney's brief for use in court. As an argument for the success of a program, it cannot rest its case on circumstantial evidence but must have multiple interlocking substantia- tions for the efficacy of the program in a particular case. To summarize, a good case argument study of a child should include the following: · Full description of the child's conditions and/or vulnerabilities and their severity. · Argument about the potential negative consequences of relative neglect of these issues. · Analysis of the cost (financial and human) of neglect or delay. · Consideration of the cost of detection in the program. · Presentation of the likelihood of alternative detection or management in the community were the program not in existence. · Description of the outcome and the likelihood of durable effects of the program on the child. Such case argument studies can be convincing to public policy makers. Even if quantitative program effects cannot be demonstrated for a project, a selection of well-chosen cases can justify its existence. To cite an extreme example, one might argue that if a project with 300 children in it can prevent 2 of them from becoming juvenile delinquents by age 13, it has more than justified its existence and can be deemed to have a favorable

92 cost-benefit ratio. In all likelihood, it is best for a project to present a number of diverse case argument studies. It is most helpful if these studies can relate directly to some of the primary objectives of the program. Process Studies of Health Process studies constitute another important prototype for health evaluation. Documentation of parental satisfaction, attitudinal changes, and sensitivity to health needs are among the relevant dimensions of process evaluation in this area. Process studies can be undertaken using standardized parent interviews or questionnaires. In several investigations, health diaries have been used to document feelings and behaviors related to health. Such diaries can be useful in both process and outcome studies. As part of standardized interviews, parents can be asked about their overall levels of satisfaction with the health aspects of an early education project. Listed below are examples of useful questions: · Did you feel that the doctors (or nurses) in the project were sympathetic and understood your child's needs? · Were you comfortable or somewhat afraid about asking them questions? · Did you often have to wait a long time to be seen for a health examination? · Did the health personnel use words you didn't understand? . . . . Do you think that the health personnel were good with your child or baby? · Did you think that the feedback you received from them was adequate? · Did they often make you afraid? · Did the personnel in the project communicate well with your own doctor? · Do you feel that the health part of this project was helpful even though you have your own doctor outside the project? · Did the health part of this project help you in any way to use your own doctor better? · Do you feel more knowledgeable about health issues as a result of participating in this project?

93 . Do you have more confidence in your own ability to make health-related decisions now that you have been in the project? · Have you switched physicians or sources of medical care while you have been in the project? · Do you think the project had anything to do with these changes? · Can you describe anything in the health area that you are doing differently now as a result of having been in the project? The answers to questions like these can be assembled in such a way as to give a good composite picture of the effect of a program on behavior and attitude. One can also relate, at least qualitatively, a sense of the degree of satisfaction with the health aspects of a project. In interpreting such data, it is of course critical to bear in mind that satisfaction and efficacy may be very differ- ent dimensions. There can be a vast discrepancy between what people think they want from a project and what they actually need. A process study may be more effective in getting at "wants" than at needs. Process studies need not be limited to parents. In the Brookline Early Education Project we undertook a process study of local pediatricians to determine the impact of the project's early-school health services on the practicing community (Hanson and Levine, 1980). The local physicians' satisfaction, awareness, and sensitivity to the project were assayed through a standardized ques- tionnaire. An important advantage of process studies is their ability to evaluate the impact of a project on a broader array of constituents, including those providing existing services, personnel in the schools, professionals within the project itself, trainees, and those responsible for the future care of the children. Another advantage is that process studies can be an ongoing activity, providing relatively immediate feedback and evaluation throughout the life of the program. Tracer Studies The use of tracer studies can be economical and effective in evaluating the health impact of a project (Kessner et al., 1974). In these investigations a few key measures, consistent with the objectives of the program, are isolated and sought within the program. The

94 tracers used should be well documented in the literature, so that expected prevalence estimates can be obtained. For example, a good tracer for the efficacy of a health- related project might be the immunization rate of its participants. Several other tracers might also be selected. For example, in an evaluation of pediatric practices undertaken several years ago, the frequency of throat cultures was used. This was thought to be a good index of the thoroughness of a pediatrician. Was he or she in the habit of prescribing antibiotics without cultures? Or was a culture usually taken first? In an early childhood project, three or four tracers might indicate efficacy, such as the prevalence of accidents or accidental poisoning, hospitalization rates, alterations in the use of emergency rooms, the existence of a primary care source, the ability to name a dentist, or the existence of certain kinds of safety devices in the home or automobile. The assumption underlying the selection of tracers is that they somehow typify the overall health status of a child. Sometimes there can be an inherent circularity in this, particularly when the objectives are too close to the tracer. For example, if parents in a project were given safety caps to insert in electric outlets to prevent shock, and the existence of such devices in the parents' homes was used as a tracer, the outcome might not be representative of health status in general. Tracer studies may or may not entail a comparison group. They can be descriptive insofar as there exist data from other studies or face validity for each specific tracer. Cost-Benefit Studies Cost factors can be a part of the prototypes of evaluation mentioned in this chapter. Often it is possible to integrate measurements of costs and benefits into assessments of outcome or process. An analysis of series of outcomes might entail a careful examination of the expenditures that produced these outcomes. In a case argument study it can be important to document the cost per child of various evaluations. As projects increas- ingly merge health and early education activities, it is essential to document additional costs. Fundamental questions need to be answered: a

95 · Should a project have its own health education screening or service component? Or should it use existing pediatric services in the community? How extensive should the health component of an early childhood project be? · If children are to receive health examinations, what should they include? Which components of health evaluation are least likely to be covered by other programs in the community? · Are there clear savings to be had by consolidating health and educational services? For example, if assessments are to be made of educational readiness in young children, is there some economy to be derived from combining these with a preschool physical, neurological, and sensory examination? Does the combination of such services yield diagnostic benefits that might not be present were they fragmented? The answers to these questions can be derived as part of project evaluations. However, they will never be uniformly applicable throughout the United States. The nature of existing resources, the goals of a particular project, the nature of a population served, the values of existing service providers, and public policy makers are all likely to have strong impacts on the analysis of costs and benefits. Choosing a Prototype While the prototypes listed above are certainly not the only ones available and combinations are possible, evaluators must choose among alternative designs. Clearly the best way to make the choice is to start with the question for which an answer is desired, since certain questions dictate certain approaches. Table 3 matches the types of questions that have been raised at the Brookline Early Education Project with the seven suggested evaluation prototypes. As indicated in the table, a given prototype may be appropriate for one question or one project but not for another. For instance, when we wanted to know the prevalence of hearing defects, an outcome study with or without a comparison group and a longitudinal study of findings were both appropriate, while case arguments, process studies, and tracer studies were not. When our interest was the prevention of early school dysfunction, clearly

96 . a' c a~ u m ~ 1 · - V 0 ~ 0 V ~q C) rl o o . - C} ~5 ~4 a' ~: . - y o o m s . - a, S~ a ~rl ~Q o U] a o o ~4 o .,, m ~: E~ 0 ~ · - t0  E~ u~ 0 UO 0 ~ ·~4 O ~ V P U) 0 JJ c a, a, ~ :' 0 ~ C~ ,0 V ~ C) V] ~S 0 a , ~ c ~5 ~ · - ~ V ~ V U) C . - · - ~ ~ C~ C O C ~ ·- O 0 1 ~ Ll ., 0 V O U] ~ :' o (} V Ll ~ 5 O O C) S C V V o ~ · - U] 0 3 ·^ 0 C a, O · - 0 - V V) ~ a, O O ~ s ~ V V o ~ · - L4 0 3 c' 0 0 a, c ~q:' V · -~  0~C o v ~ 0 0 O o ~: ~ Z ~n 0 c va, 0 . -~ a, C o ~ v CL 0 0 O o Z 0 0 ~o 4d ~5 . - a, a, ~C o ~ v 0~ 0 0 O O ~: ~ Z 0 0 C v a, ~5 ._, a~ h 0' C o .~) . 00 0 O O ~: ~ Z 0 0 C v a' ~a ~ . - a' C ~C o 4., - 0 0 O O () Z 0  0 V `0 ~ ·-1 aJ ~a' 04 C o a~ 0 0 ·4 0 0 0 0 a a' V ~ ·,4 C' Ll ~C o ~ V CL 0 0 0~ 0 0 ~: ~ Z 3 3 :^ O o C I E E I `- - 3 q3 O a, a O I I Z I I 3 3 :>, O O C ~:  O I I Z I I O I I O I I Z I I Z I I E E 3 · - s · - ~a = ~ · - £ E~ ·,4 ~5 E :C · - ~ 3 3 ~ o O £ ,~ ~ E E E ·,1 ·_4 ·~4 ~a ~ (D a~ £ £ £ 1 , ,~ ·.- a, V C c ~ a, ~ c ~ a) a' oc ~ s ~ .,~, a, V 0 V 0 ~ ~ ·~4 ~ V O :5 ~ s · - :e · - s s a, ·rl · - £ ~ !r  E E 1 ~ . - · - 3 o a, ~ :£ £ 0 1 1 Z I I . - Q) £ 0 3 0 ~ ,~ s s ._' cr V' ~ ·~ - · - GJ z: 2 £ ._, E a£, · - ~ 3 3 a, 0 0 £ E ._1 s s o, a' ·^ · - £ 3: 3: n ~.  0 a, ·a v s ~ V V s 3 I E , ~ s 3 r~ O · - X ~ s .. - E E .,. · - 3 == - 1 a, £ o' ~ .-, .^ :~: ~ :~: C ~- 0 .~' ~ v v C ~ Ll V 0 C E C s . - ~: E E · - s · - o~ t)-~4 a ~0 1 1 £ =: : ~Z I I I E E 1 · - · - 3 ~ ~ O a) aJ £ :~  0 1 1 Z 1 1 E ~a GJ £ 3 3 3 O o O O I I Z I I O I I Z I I E a, . - · - £ ~ 5: 0 O ._, ·^ ~4 V ~ a .,' C ~ ~ C) ~ :> · - o V ~ ~ ~ C) 0 ~ a, C C C C ~ · - ~ Q} 0 1 1 Z 1 1 . - a, ~: 3 3 0 1 1 Z 1 ~  O I I Z 1 1 0 1 1 Z I I O I I Z I I c ._, V V ~. C ~ C 3 O 0 s V C G5 0

97 only the outcome studies with comparison groups or cost-benefit studies would suffice. For a question such as parent satisfaction, elaborate comparison group studies were neither necessary nor appropriate, and information obtained from a process study would have limited applicability. The scope of child health is very wide, and evaluators of early childhood programs should plan carefully before they launch a health evaluation, defining the variables they want to use (particularly background and outcome), the questions they want to answer, and then select the one or two evaluation prototypes that are most likely to yield answers. CONCLUSION In surveying the various prototypes for the evaluation of the health impact of a program, it is clear that they are not mutually exclusive. In many instances projects may want to apply more than one prototype to assess a program's efficacy. None of these is foolproof; all need careful application and meticulous interpretation. A large project may need separate evaluations of specific aspects of health care influence. For example, if one can demonstrate that a particular project benefited the health of children in some way, one may then proceed to ask: What aspect(s) of the program had the greatest influence in this regard? It may be that health education made the difference. Or it may be that specific diagnostic examinations or fastidious feedback to the local physician was the major positive influence. Isolating one or more elements of service that were particularly useful obviously has public policy implications. A future project may try to allocate its resources to only those aspects of health services that are likely to have the greatest payoff. Thus, all programs should analyze subcomponents of their health services in order to discern the most beneficial elements. Measuring the impact of an early childhood project on health has significant implications for medical professionals. It is likely that many or One same methods can be applied to the examination of medical program efficacy. The study of evaluation research can therefore reap benefits for health care research as well as education and public policy determination. If the technology of evaluation is to continue to grow and meet . . . . . . ~

98 the needs of public policy makers and investigators, cross-fertilization between disciplines is likely to accelerate the process. We will have achieved a great deal if this paper can help foster such collaboration. REFERENCES Alberman, E. D., and Goldstein, H. (1970) The "at risks register, a statistical evaluation. British Journal of Preventive and Social Medicine 24:129-135. Allen, H. F. (1957) Testing visual acuity in preschool children: norms, variables and a new picture test. Pediatrics 19:1093-1100. American Academy of Pediatrics, Committee on Children with Handicaps (1972) Vision screening in preschool children. Pediatrics 50:966-967. Baddeley, A. (1976) The Psychology of Memory. New York: Basic Books. Badger, E., Burns, D., and Rhoads, B. (1976) Education for adolescent mothers in a hospital setting. American Journal of Public Health 66:469-472. Balinsky, W., and Berger, R. (1975) A review of the research on general health status indices. Medical Care 13:283-295. Bell, R. Q., Weller, G. M., and Walding, M. F. (1971) Newborn and preschoolers: organization of behavior and relations between periods. Monograph of the Society for Research in Chi_ Development 36(1-2). Bleak, E. E. (1971) The shoeing of children--sham or science? Developmental Medicine and Child Neurology 13:188-195. Brewer, G. D., and Kakalik, J. S. (1979) Handicapped Children. New York: McGraw Hill Bricker, D. D., and Iacino, R. (1977) Early intervention with severely/profoundly handicapped children. In E. Sontag, ea., Educational Programming for the Severely and Profoundly Handicapped. Reston, Va.: Council for Exceptional Children. .

99 Broman, S. H., Nichols, P. L., and Kennedy, W. A. (1975) Preschool IO: Prenatal and Developmental Correlates. New York: John Wiley & Sons, Bronfenbrenner, U. (197S) Is early education effective? In H. J. Leichter, ea., The Family as Educator. New York: Teachers College Press, Columbia University. Education for the Handicapped Progress Toward a Free, Appropriate Public Education. A Report to Congress on the Implementation of Public Law 94-142: The Education for All Handicapped Children Act. HEW Publication No. (OK) 79-05003. Washington, D.C.: U.S. Department of Health, Education, and Welfare. Carey, W. B., Fox, M., and McDevitt, S. C. (1977) Bureau of (1979) Temperament as a factor in early school adjustment. Pediatrics 60:621-624. Cassady, G. (1975) Perinatal outcome and referral age. Pediatrics 56:160. , Center for Disease Control (1975) Increased Lead Absorption and Lead Poisoning in Young Children. Atlanta, Gal: U.S. Department of Health, Education, and Welfare. Center for Disease Control (1977) Si~mmarv of Immunization Status: ~ Preliminary Report: U.S. Immunization Survey, 1976. Atlanta, Gal: U.S. Department of Health, Education, and Welfare. Chance, G. W., O'Brien, M. J., and Swyer, P. R. (1973) Transportation of sick neonates, 1972: an unsatisfactory aspect of medical care. Canadian Medical Association Journal 109:847. Chisholm, J. J. (1971) Lead poisoning. Connolly, C. (1971) Social and emotional factors in learning disabilities. In H. R. Myklebust, ea., Progress in Learning Disabilities. Vol. II. New York: Grune ~ Stratton, Inc. Cowell, H. R. (1977) Shoes and shoe corrections. Of North America 24:791-797. Scientific American 224:15-23. Pediatric Clinic l

100 Davie, R., Butler, N., and Goldstein, H. (1972) From Birth to Seven: The Second Report of the National Child Development Study. London: Longman. Denckla, M. (1978) Naming of object-drawings by dyslexic and other learning disabled Children. Brain and Language 3:231. Dingle, J. H. (1964) Illness in the Home. Cleveland, Ohio: The Press of Case Western Reserve University. Dingle, J. H. (1966) The common cold and common cold like illnesses. Medical Times 94:186-190. Elardo, R., Bradley, R., and Caldwell, B. M. (1975) The relation of infant's home environments to mental test performance from six to thirty-six months: a longitudinal analysis. Child Develooment 46 71-76. Fitzhardinge, P. Me (1975) Early growth and development in low birthweight infants following treatment in an intensive care nursery. Pediatrics 56:162-172. Fitzhardinge, P. M., and Steven, E. M. (1972) The small for date infant. II. Neurological and intellectual sequelae. Pediatrics 50:50-57. Folman, S. J. (1977) Nutritional Disorders of Children: - Prevention, Screening and Follow-up. HEW Publication No. (HSA) 77-5104 Washington, DeCe UeSe Department of Health, Education, and Welfare. Frankenburg, W. K. (1974) Selection of diseases and tests in pediatric screening. Pediatrics 54:612-616. Frankenburg, W. K., and North, A. F. (1974) A Guide to Screening for the Early and Periodic Screening, Diagnosis and Treatment Program (EPSDT) Under Medicaid. HEW Publication No. (SRS) 74-24516. Washington, D.C.: U.S. Department of Health, Education, and Welfare. Giebink, G. S., and Quie, P. G. (1978) Otitis media: the spectrum of middle ear inflammation. Annual Review of Medicine 29:285-306.

101 Goldstein, K. M., Caputo, D. V., and Taub, H. B. (1976) The effects of prenatal and perinatal complications on development at one year of age. Child Development 47:613-621. Gortmaker, S. L. (1979) Access to and Utilization of Ambulatory Medical and Dental Services Among Children in Genesee County, Michigan. Community Child Health Studies, Harvard School of Public Health, Cambridge. Gottlieb, N. H., and Wecheler, H. (1976) Immunization levels in Boston schools--a second look. New England Journal of Medicine 294:1459. Haggerty, R. J. (1965) Family diagnosis: research methods and their reliability for studies of the medical-social unit, the family. American Journal of Public Health 55:1521-1533. Haggerty, R. J. (1976) ~ Who will monitor access? Pediatrics - 57:169-170. Haggerty, R. J., Roghmann, K. J., and Pless, I. B. (1975) Child Health and the Community. New York: John Wiley & Sons, Inc. Hanson, M. A., and Levine, M. D. (1980) Early school health: an analysis of its impact on primary care. Journal of School Health 50:577-580. Harvard Child Health Project Task Force (1977) Toward a Primary Medical Care System Responsive to Children's Needs. Cambridge, Ballinger Publishing Co. Hayden, A. H., and McGuiness, G. D. (1977) Bases for early intervention. In E. Sontag, ea., Educational Programming for the Severely and Profoundly Handicapped. Reston, Va.: Council for Exceptional Children. Horton, K. B. (1976) Early intervention for hearing-impaired infants and young children. In T. Tjossem, ea., Intervention Strategies for High Risk Infants and Young Children. Baltimore, Md.: University Park Press. Institute of Medicine (1973) Infant Death: An Analysis by Maternal Risk and Health Care. Washington, D.C.: National - Academy of Sciences.

102 Institute of Medicine (1977) Primary Care in Medicine. A n~finitio Washington, D.C.: National Academy of Sciences. Joint Committee on Infant Hearing Screening (1971) Joint committee statement on infant hearing screening. Journal of the American Speech and Hearing Association 13:79. Joint Committee on Infant Hearing Screening (1972) Joint committee statement on infant hearing screening. Journal of the American Speech and Hearing Association 16:160. Kempe, C. H., and Helfer, R. E., eds. (1972) Helping the Battered Child and His Family. Philadelphia, Pa.: J. B. Lippincott Co. Kephart, N. C. (1971) The Slow Learner in the Classroom. Columbus, Ohio: Charles E. Merrill. Kessner, D. M., Snow, C. K., and Singer, J. (1974) The Assessment of Medical Care for Children. Volume 3. Washington, D.C.: Institute of Medicine. Klaus, M. H., and Kennell, J. H. (1976) Maternal-Infant Bonding. St. Louis, Mo.: C. V. Mosby Co. Knobloch, H., and Pasamanick, B. (1966) Prospective studies on the epidemiology of reproductive causally: methods, findings and some implications. Merrill-Palmer Quarterly of Behavior and Development 12:27-43. Koff, E., Boyle, P., and Pueschel, S. M. (1977) Perceptual-motor functioning in children with phenylketonuria. American Journal of Diseases of Children 131:1084-1087. Kopelman, A. E. (1978) The smallest preterm infant. American Journal of Diseases of Children 132:461-462. Kronstadt, D., Oberklaid, F., Ferb, T., and Swartz, J. (1979) Infant behavior and maternal adaptation in the first six months of life. American Journal of Orthopsychiatry 49(3):454-464. Lazar, I. (1977) The Persistence of Preschool Effects: A Long-Term Follow-up of 14 Infant and Preschool Experiments. Report prepared for the Adminis- tration for Children, Youth, and Families, U.S. Department of Health, Education, and

103 Welfare by the Education Commission of the States. Levine, M. D. (1979) The School Function Program: Profile of a General Pediatrics Consultative Service Model. Report prepared for the Robert Wood Johnson Foundation under Grant No. 4293. Levine, M. D., and Oberklaid, F. (1980) Hyperactivity--symptom complex or complex symptom. American Journal of Diseases of Children 134:409-414. Levine, M. D., Brooks, R., and Shonkoff, J. P. (1980) A Pediatric Approach to Learning Disorders. New York: Wiley Medical. Levine, M. D., and Meltzer, L. (1981) Developmental output failure: impaired productivity in the school aged child. Pediatrics 67:18-25. Levine, M. D., Oberklaid, F., Ferb, T. E., Hanson, M. A., Palfrey, J. S., and Aufseeser, C. L. (1980) The pediatric examination of educational readiness: validation of an extended observation procedure. Pediatrics 66:341-349. Levine, M. D., Palfrey, J. S., Lamb, G. A., et al. (1977) Infants in a public school system: the indicators of early health and educational need. Pediatrics 60:579-587. Levy, J. D., Bonanno, R. A., Schwartz, C. G., and Sanofsky, P. A. (1979) Primary care: patterns of use of pediatric medical facilities. Medical Care 17:881-893. Lippmann, O. (1974) Directions for Use of the H.O.T.V. Test. The Good-Lite Company, Forest Park, Ill. Lippmann, O., Illiterate, E., Frankenburg, W. K., and Camp, B. W., eds. (1975) Pediatric Screening Tests. Ill.: Charles C Thomas. Springfield, Lowe, C. U., and Alexander, D. F. (1974) Health care of poor children. In A. Schorr, ea., Children and Decent People. New York: Basic Books. Meter, J. (1973) Screening and Assessment of Young Children at Developmental Risk. President's Committee on Mental Retardation. HEW Publication No. OS-73-90. Washington, D.C.: U.S. Department of Health, Education, and Welfare.

104 Meter, J. (1975) Early intervention in the prevention of mental retardation. Pp. 385-409 in A. Milunsky, ea., Prevention of Genetic Disease and Mental Retardation. Philadelphia, Pa.: W. B. Saunders Company. Meissels, S. J. (1978) A Guide to Early Childhood Developmental Screening. Massachusetts State Department of Education. Miller, F. J. W., Court, S. D. M., Walton, W. S., and Knox, E. G. (1960) Growing Up in Newcastle Upon Tyne. London: Oxford University Press. Milunsky, A. (1975) Prevention of Genetic Disease and Mental Retardation. Philadelphia, Pa. e We B · - Saunders Company. Minear, R. E., and Guyer, B. (1979) Assessing immunization services at a neighborhood health center. Pediatrics 63:416-419. Morris, A. G., London, R., and Glick, J. (1976) Educational intervention for preschool children in a pediatric clinic. Pediatrics 57:765-768. Morris, J. N. (1979) Social inequalities undiminished. Lancet 1(8107):87-90. Mortimer, E. A. (1968) Frequent colds. Pp. 211-215 in M. Green and R. J. Haggerty, eds., Ambulatory Pediatrics. ~ W. B. Saunders Company. Philadelphia, Pa.: National Research Council (1976) Toward a National Policy for Children and Families. Advisory Committee on Child Development. Washington, D.C.: National Academy of Sciences. Needleman, H. L., Gunnoe, C., Levitan, A., et al. (1979) Deficits in psychologic and classroom performance of children with elevated destine lead levels. New England Journal of Medicine 300:659-665. Neligan, G. A., Kolvin, I., Scott, N. M., et al., eds. (1976) Born too soon or born too small. Clin. Dev. Med. 61. Philadelphia, Pa.: Spastics International Medical Publications, J. Lippincott Co.

105 Nelson, K. (1979) B., and Ellenberg, J. H. Neonatal signs as predictors of cerebral palsy. Pediatrics 64:225-232. Newberger, C. M., Newberger, E. H., and Harper, G. P. (1976) The social ecology of malnutrition in childhood. In J. Lloyd-Still, ea., Malnutrition and Intelligence. Lancaster, Pa.: Medical and Technical Publishing Co. Newberger, E. H., Newberger, C. M., and Richmond, J. B. (1976) Child health in America: toward a rational public policy. Milbank Memorial Fund Quarterly (Summer):249-298. Oberklaid, F., and Levine, M. D. (1980) Precursors of school failure. Review 2:1 (July). Pediatrics in Oberklaid, F., Dworkin, P., and Levine, M. D. (1979) Developmental behavioral dysfunction in preschool children. American Journal of Diseases of Children 133:1126-1131. Office of the Auditor General of California (1979) Special Education Financing Warrants Review Report, No. 843. Sacramento, Calif. Oski, F. A., and Hinig, A. S. (1978) The effects of therapy on the developmental scores of iron-deficient infants. Journal of Pediatrics 92:21-25. Palfrey, J. S., Mervis, R. C., and Butler, J. (1978) New directions in the evaluation and education of handicapped children. New England Journal of Medicine 298:819-824. Palfrey, J. S., Hanson, M. A., Norton, S., et al. (1980) Selective hearing screening for very young children. Clinical Pediatrics 19:473-477. Pierson, D. (1974) The Brooklyn Early Education Project: model for a new education priority. Childhood Education 50:132-136. Pierson, D. E., Levine, M. D., Ferb, T. E., and Wolman, R. (1980) Auditing Multidisciplinary Assessment Procedures: A System Developed for the Brooklyn Early Education Project. Paper presented at the Third International Conference on Early Identification of Children Who Are Developmentally "At Risk," Teton Village, Wyo., September 22-26.

106 Public Health Service (1976) Forward Plan for Health FY 1978-1982. Washington, D.C.: U.S. Department of Health, Education, and Welfare. Reynolds, R. C., Banks, S. A., and Murphee, A. H. (1976) The Health of a Rural Community. Gainesville, Fla.: University of Florida Press. Richmond, J. B. (1966) Communities in action: a report on Project Reading Teacher 19:323-331. Head Start. Richmond, J. B. (1975) An idea whose time has arrived. Pediatric Clinics of North America 22:517-523. Robinson, M. E., and Schwartz, L. B. (1973) Visuo-motor skills and reading ability: a longitudinal study. Developmental Medicine and Child Neurology 15:281. Roghmann, K. J., Hecht, P. K., and Haggerty, R. J. (1973) Family coping with everyday stress: self reports from a household survey. Journal of Comparative Family Studies 4(1):49-62. Rubin, R. A., Rosenblatt, C., and Balow, B. (1973) Psychological and educational sequelae of prematurity. Pediatrics 52:352-363. Rudel, R., and Denckla, M. (1976) Relationship of IQ score and reading score to visual, spatial and temporal matching tasks. Journal of Learning Disabilities 9:169. Rutter, M. (1974) Emotional disorder and educational underachievement. Childhood 49:249. Archives of Disease in - Rutter, M., and Martin, J. A. M., eds. (1972) The child with delayed speech. Clinics in Developmental Medicine, No. 43. London: Spastics International Medical Publications Rutter, M., Tizard, J., and Whitmore, K. (1970) Education, Health and Behaviour. London: John Wiley & Sons, Inc. Sameroff, A., and Chandler, M. (1975) Reproductive risk and the continuum of caretaking casualty. In F. Horowitz, ea., Review of Child Development Research, Vol. 4. Chicago, Ill.: University of Chicago Press. Sheridan, M. (1970) Stycar Vision Test Manual. 2nd rev. ed. Windsor, Berks, England: NFER Publishing Co.

107 Shipman, V.C. (no Maintaining and Enhancing Early Intervention date) Gains. Abridged version of Project Report 76-21 prepared for the Office of Child Develop- ment, U.S. Department of Health, Education, and Welfare, under Grant No. H-8256. Simmons, J. Q., and Tymchak, A. (1973) The learning deficits in childhood psychosis. Pediatric Clinics of North America 20:665-680. Smith, B., and Phillips, C. J. (1978) Identification of severe mental handicap. Child: Care, Health and Development 4:195-203. Smith, D. W., and Wilson, A. A. (1973) The Child with Down's Syndrome. Philadelphia, Pa.: W. B. Saunders Company. Smith, S., and Simpson, K. (1975) The Battered Child Syndrome. London: Butterworth. Starfield, B. (1974) Measurement of outcome: a proposed scheme. Milbank Memorial Fund Quarterly (Winter):39-50. Stewart, A., and Reynolds, E. O. R. (1974) Improved prognosis for infants of very low birtb-weight. Pediatrics 54:724-735. Stewart, A. L., Turcan, D. M., Rawlings, G., et al. (1977) Prognosis for infants weighing 100 grams or less at birth. Archives of Disease in Childhood 52:97-104. Strangler, S. R., Huber, C. J., and Routh, D. K. (1980) Screening Growth and Development of Preschool Children. New York: McGraw-Hill Book Company. Sultz, H. A., Schlesinger, E. R., Mosher, W. E., and Feldman, J. G. (1972) Long Term Childhood Illness. Pittsburgh, Pa.: University of Pittsburgh Press. Taft, L. T. (1978) Child development: prenatal to early childhood. Journal of School Health (May):281-287. Taylor, L., and Newberger, E. H. (1979) Child abuse in the international year of the child. New England Journal of Medicine 301:1205-1212. Thomas, A., and Chess, S. (1975) Temperament and Development. New York: Brunner/Mazel.

108 Tilford, J. A. (1976) The relationship between gestational age and adaptive behavior. Merrill-Palmer Quarterly 22:319-326. Tjossem, T. (1976) Early intervention: issues and approaches. In T. Tiossem, ea., I for High Risk Infants and Young Children. Baltimore, Md.: University Park Press. U.S. General Accounting Office (1979) Early Childhood and Family Development Programs Improve the Quality of Life for Low Income Families. HRD-79-40. February 6. Vukovich, D. M. (1968) Pediatric neurology and learning disabilities. In H. R. Myklebust, ea., Progress in Learning Disabilities. Vol. I. New York: Grune & Stratton, Inc. Webb, T. E., and Oski, F. A. (1973) Iron deficiency anemia and scholastic achievement in young adolescents. Journal of Pediatrics 82:827. Weiss, H. B. (1979) Parent Support and Education: An Analysis of the Brookline Early Education Project. Thesis presented to the faculty of the Graduate School of Education, Harvard University. Werner, E. E., Beirman, J. M., and French, F. E. (1971) The Children of Kauai: A Longitudinal Study from the Perinatal Period to Age Ten. Honolulu: University of Hawaii Press. White, B. L. (1975) Critical influences in the origins of competence. ~t =~2 ~ 21:243-266. Whitfield, T., and Walker, D. (no Personal communications regarding ongoing date) studies of the Community Child Health Studies Group at Harvard School of Public Health. Wiig, E., and Semel, E. (1976) Language Disabilities in Children and Adolescents. Columbus, Ohio: Charles E. Merrill. Wolff, P. H., and Hurwitz, I. (1966) The choreiform syndrome. Developmental Medicine and Child Neurology 8:160-165. Zigler, E., and Valentine, J., eds. (1979) Project Head Start: A Legacy of the War on Poverty. New York: The Free Press.