National Academies Press: OpenBook

Learning From Experience: Evaluating Early Childhood Demonstration Programs (1982)

Chapter: Comprehensive Family Service Programs: Special Features and Associated Measurement Problems

« Previous: Preschool Education for Disadvantaged Children
Suggested Citation:"Comprehensive Family Service Programs: Special Features and Associated Measurement Problems." National Research Council. 1982. Learning From Experience: Evaluating Early Childhood Demonstration Programs. Washington, DC: The National Academies Press.
×
Page 203
Suggested Citation:"Comprehensive Family Service Programs: Special Features and Associated Measurement Problems." National Research Council. 1982. Learning From Experience: Evaluating Early Childhood Demonstration Programs. Washington, DC: The National Academies Press.
×
Page 204
Suggested Citation:"Comprehensive Family Service Programs: Special Features and Associated Measurement Problems." National Research Council. 1982. Learning From Experience: Evaluating Early Childhood Demonstration Programs. Washington, DC: The National Academies Press.
×
Page 205
Suggested Citation:"Comprehensive Family Service Programs: Special Features and Associated Measurement Problems." National Research Council. 1982. Learning From Experience: Evaluating Early Childhood Demonstration Programs. Washington, DC: The National Academies Press.
×
Page 206
Suggested Citation:"Comprehensive Family Service Programs: Special Features and Associated Measurement Problems." National Research Council. 1982. Learning From Experience: Evaluating Early Childhood Demonstration Programs. Washington, DC: The National Academies Press.
×
Page 207
Suggested Citation:"Comprehensive Family Service Programs: Special Features and Associated Measurement Problems." National Research Council. 1982. Learning From Experience: Evaluating Early Childhood Demonstration Programs. Washington, DC: The National Academies Press.
×
Page 208
Suggested Citation:"Comprehensive Family Service Programs: Special Features and Associated Measurement Problems." National Research Council. 1982. Learning From Experience: Evaluating Early Childhood Demonstration Programs. Washington, DC: The National Academies Press.
×
Page 209
Suggested Citation:"Comprehensive Family Service Programs: Special Features and Associated Measurement Problems." National Research Council. 1982. Learning From Experience: Evaluating Early Childhood Demonstration Programs. Washington, DC: The National Academies Press.
×
Page 210
Suggested Citation:"Comprehensive Family Service Programs: Special Features and Associated Measurement Problems." National Research Council. 1982. Learning From Experience: Evaluating Early Childhood Demonstration Programs. Washington, DC: The National Academies Press.
×
Page 211
Suggested Citation:"Comprehensive Family Service Programs: Special Features and Associated Measurement Problems." National Research Council. 1982. Learning From Experience: Evaluating Early Childhood Demonstration Programs. Washington, DC: The National Academies Press.
×
Page 212
Suggested Citation:"Comprehensive Family Service Programs: Special Features and Associated Measurement Problems." National Research Council. 1982. Learning From Experience: Evaluating Early Childhood Demonstration Programs. Washington, DC: The National Academies Press.
×
Page 213
Suggested Citation:"Comprehensive Family Service Programs: Special Features and Associated Measurement Problems." National Research Council. 1982. Learning From Experience: Evaluating Early Childhood Demonstration Programs. Washington, DC: The National Academies Press.
×
Page 214
Suggested Citation:"Comprehensive Family Service Programs: Special Features and Associated Measurement Problems." National Research Council. 1982. Learning From Experience: Evaluating Early Childhood Demonstration Programs. Washington, DC: The National Academies Press.
×
Page 215
Suggested Citation:"Comprehensive Family Service Programs: Special Features and Associated Measurement Problems." National Research Council. 1982. Learning From Experience: Evaluating Early Childhood Demonstration Programs. Washington, DC: The National Academies Press.
×
Page 216
Suggested Citation:"Comprehensive Family Service Programs: Special Features and Associated Measurement Problems." National Research Council. 1982. Learning From Experience: Evaluating Early Childhood Demonstration Programs. Washington, DC: The National Academies Press.
×
Page 217
Suggested Citation:"Comprehensive Family Service Programs: Special Features and Associated Measurement Problems." National Research Council. 1982. Learning From Experience: Evaluating Early Childhood Demonstration Programs. Washington, DC: The National Academies Press.
×
Page 218
Suggested Citation:"Comprehensive Family Service Programs: Special Features and Associated Measurement Problems." National Research Council. 1982. Learning From Experience: Evaluating Early Childhood Demonstration Programs. Washington, DC: The National Academies Press.
×
Page 219
Suggested Citation:"Comprehensive Family Service Programs: Special Features and Associated Measurement Problems." National Research Council. 1982. Learning From Experience: Evaluating Early Childhood Demonstration Programs. Washington, DC: The National Academies Press.
×
Page 220
Suggested Citation:"Comprehensive Family Service Programs: Special Features and Associated Measurement Problems." National Research Council. 1982. Learning From Experience: Evaluating Early Childhood Demonstration Programs. Washington, DC: The National Academies Press.
×
Page 221
Suggested Citation:"Comprehensive Family Service Programs: Special Features and Associated Measurement Problems." National Research Council. 1982. Learning From Experience: Evaluating Early Childhood Demonstration Programs. Washington, DC: The National Academies Press.
×
Page 222
Suggested Citation:"Comprehensive Family Service Programs: Special Features and Associated Measurement Problems." National Research Council. 1982. Learning From Experience: Evaluating Early Childhood Demonstration Programs. Washington, DC: The National Academies Press.
×
Page 223
Suggested Citation:"Comprehensive Family Service Programs: Special Features and Associated Measurement Problems." National Research Council. 1982. Learning From Experience: Evaluating Early Childhood Demonstration Programs. Washington, DC: The National Academies Press.
×
Page 224
Suggested Citation:"Comprehensive Family Service Programs: Special Features and Associated Measurement Problems." National Research Council. 1982. Learning From Experience: Evaluating Early Childhood Demonstration Programs. Washington, DC: The National Academies Press.
×
Page 225
Suggested Citation:"Comprehensive Family Service Programs: Special Features and Associated Measurement Problems." National Research Council. 1982. Learning From Experience: Evaluating Early Childhood Demonstration Programs. Washington, DC: The National Academies Press.
×
Page 226
Suggested Citation:"Comprehensive Family Service Programs: Special Features and Associated Measurement Problems." National Research Council. 1982. Learning From Experience: Evaluating Early Childhood Demonstration Programs. Washington, DC: The National Academies Press.
×
Page 227
Suggested Citation:"Comprehensive Family Service Programs: Special Features and Associated Measurement Problems." National Research Council. 1982. Learning From Experience: Evaluating Early Childhood Demonstration Programs. Washington, DC: The National Academies Press.
×
Page 228
Suggested Citation:"Comprehensive Family Service Programs: Special Features and Associated Measurement Problems." National Research Council. 1982. Learning From Experience: Evaluating Early Childhood Demonstration Programs. Washington, DC: The National Academies Press.
×
Page 229
Suggested Citation:"Comprehensive Family Service Programs: Special Features and Associated Measurement Problems." National Research Council. 1982. Learning From Experience: Evaluating Early Childhood Demonstration Programs. Washington, DC: The National Academies Press.
×
Page 230
Suggested Citation:"Comprehensive Family Service Programs: Special Features and Associated Measurement Problems." National Research Council. 1982. Learning From Experience: Evaluating Early Childhood Demonstration Programs. Washington, DC: The National Academies Press.
×
Page 231
Suggested Citation:"Comprehensive Family Service Programs: Special Features and Associated Measurement Problems." National Research Council. 1982. Learning From Experience: Evaluating Early Childhood Demonstration Programs. Washington, DC: The National Academies Press.
×
Page 232
Suggested Citation:"Comprehensive Family Service Programs: Special Features and Associated Measurement Problems." National Research Council. 1982. Learning From Experience: Evaluating Early Childhood Demonstration Programs. Washington, DC: The National Academies Press.
×
Page 233
Suggested Citation:"Comprehensive Family Service Programs: Special Features and Associated Measurement Problems." National Research Council. 1982. Learning From Experience: Evaluating Early Childhood Demonstration Programs. Washington, DC: The National Academies Press.
×
Page 234
Suggested Citation:"Comprehensive Family Service Programs: Special Features and Associated Measurement Problems." National Research Council. 1982. Learning From Experience: Evaluating Early Childhood Demonstration Programs. Washington, DC: The National Academies Press.
×
Page 235
Suggested Citation:"Comprehensive Family Service Programs: Special Features and Associated Measurement Problems." National Research Council. 1982. Learning From Experience: Evaluating Early Childhood Demonstration Programs. Washington, DC: The National Academies Press.
×
Page 236
Suggested Citation:"Comprehensive Family Service Programs: Special Features and Associated Measurement Problems." National Research Council. 1982. Learning From Experience: Evaluating Early Childhood Demonstration Programs. Washington, DC: The National Academies Press.
×
Page 237
Suggested Citation:"Comprehensive Family Service Programs: Special Features and Associated Measurement Problems." National Research Council. 1982. Learning From Experience: Evaluating Early Childhood Demonstration Programs. Washington, DC: The National Academies Press.
×
Page 238
Suggested Citation:"Comprehensive Family Service Programs: Special Features and Associated Measurement Problems." National Research Council. 1982. Learning From Experience: Evaluating Early Childhood Demonstration Programs. Washington, DC: The National Academies Press.
×
Page 239
Suggested Citation:"Comprehensive Family Service Programs: Special Features and Associated Measurement Problems." National Research Council. 1982. Learning From Experience: Evaluating Early Childhood Demonstration Programs. Washington, DC: The National Academies Press.
×
Page 240
Suggested Citation:"Comprehensive Family Service Programs: Special Features and Associated Measurement Problems." National Research Council. 1982. Learning From Experience: Evaluating Early Childhood Demonstration Programs. Washington, DC: The National Academies Press.
×
Page 241
Suggested Citation:"Comprehensive Family Service Programs: Special Features and Associated Measurement Problems." National Research Council. 1982. Learning From Experience: Evaluating Early Childhood Demonstration Programs. Washington, DC: The National Academies Press.
×
Page 242
Suggested Citation:"Comprehensive Family Service Programs: Special Features and Associated Measurement Problems." National Research Council. 1982. Learning From Experience: Evaluating Early Childhood Demonstration Programs. Washington, DC: The National Academies Press.
×
Page 243
Suggested Citation:"Comprehensive Family Service Programs: Special Features and Associated Measurement Problems." National Research Council. 1982. Learning From Experience: Evaluating Early Childhood Demonstration Programs. Washington, DC: The National Academies Press.
×
Page 244
Suggested Citation:"Comprehensive Family Service Programs: Special Features and Associated Measurement Problems." National Research Council. 1982. Learning From Experience: Evaluating Early Childhood Demonstration Programs. Washington, DC: The National Academies Press.
×
Page 245
Suggested Citation:"Comprehensive Family Service Programs: Special Features and Associated Measurement Problems." National Research Council. 1982. Learning From Experience: Evaluating Early Childhood Demonstration Programs. Washington, DC: The National Academies Press.
×
Page 246
Suggested Citation:"Comprehensive Family Service Programs: Special Features and Associated Measurement Problems." National Research Council. 1982. Learning From Experience: Evaluating Early Childhood Demonstration Programs. Washington, DC: The National Academies Press.
×
Page 247
Suggested Citation:"Comprehensive Family Service Programs: Special Features and Associated Measurement Problems." National Research Council. 1982. Learning From Experience: Evaluating Early Childhood Demonstration Programs. Washington, DC: The National Academies Press.
×
Page 248
Suggested Citation:"Comprehensive Family Service Programs: Special Features and Associated Measurement Problems." National Research Council. 1982. Learning From Experience: Evaluating Early Childhood Demonstration Programs. Washington, DC: The National Academies Press.
×
Page 249
Suggested Citation:"Comprehensive Family Service Programs: Special Features and Associated Measurement Problems." National Research Council. 1982. Learning From Experience: Evaluating Early Childhood Demonstration Programs. Washington, DC: The National Academies Press.
×
Page 250
Suggested Citation:"Comprehensive Family Service Programs: Special Features and Associated Measurement Problems." National Research Council. 1982. Learning From Experience: Evaluating Early Childhood Demonstration Programs. Washington, DC: The National Academies Press.
×
Page 251
Suggested Citation:"Comprehensive Family Service Programs: Special Features and Associated Measurement Problems." National Research Council. 1982. Learning From Experience: Evaluating Early Childhood Demonstration Programs. Washington, DC: The National Academies Press.
×
Page 252
Suggested Citation:"Comprehensive Family Service Programs: Special Features and Associated Measurement Problems." National Research Council. 1982. Learning From Experience: Evaluating Early Childhood Demonstration Programs. Washington, DC: The National Academies Press.
×
Page 253

Below is the uncorrected machine-read text of this chapter, intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text of each book. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.

Comprehensive Family Service Programs: Special Features and Associated Measurement Problems Kathryn Hewett, with the assistance of Dennis Deloria INTRODUCTION A few years ago a team from the Denver, Colorado, U.S. General Accounting Office (GAO) visited a child develop- ment demonstration program in Gering, Nebraska, as part of a routine review of such federally funded programs in the region. At this Child and Family Resource Program, the GAO team was surprised and impressed with the many types of services provided and with the program's approach to planning and delivering services to each family. Intrigued, they looked at 3 more of the 11 demonstration programs in other areas and subsequently produced a glowing report on the comprehensive family service model as implemented by the Child and Family Resource Program. Their report called attention to several aspects of the model that distinguished it from most traditional child development programs: the emphasis on families rather than children, the approach of developing with parents a distinct plan of service and activities for each family, and the program's role as a coordinator of services in the community for each family. So enthusiastic was the GAO team about the model that they recommended it to Congress as a prototype for future child development program legislation (GAO, 1979). The Child and Family Resource Program is one of several research and demonstration programs developed by federal and private researchers that attempt to influence the development of children working with families. A number of demonstration programs funded by the Head Start Bureau of the Administration for Children, Youth, and Families (now part of the U.S. Department of Health and Human Services) have been delivering comprehensive family services for nearly a decade (15th Anniversary Head Start 203

204 Committee, 1980). Among these programs are the Child and Family Resource Program, Home Start, the Parent Child Centers, and the Parent Child Development Centers. Most of these programs had roots in child development inter- vention programs developed by researchers such as Levenstein, Weikart, Gordon, Klaus and Gray, Lally and Honig, White and Watts, and many others. Initiated as experiments in providing basic Head Start services, these programs showed the influence not only of the child development philosophies of Head Start but also of philosophies about parent involvement and community political action that characterized Head Start in the late 1960s and early 1970s. At the same time other researchers were developing family oriented programs with the aim of enhancing child development. Such programs as the Brookline Early Education Project, the Syracuse Families Project, and the Family Networks Project at the Merrill-Palmer Institute resulted from these efforts. In general these programs combine early childhood intervention and family support in various degrees, providing them directly by program staff and indirectly through coordination of other service agencies in the community. It is the combined emphasis on child and family and the broad array of available services that makes them "comprehensive." The federal demonstration programs, and some of the private ones as well, provide a full range of health, nutritional, and educational services similar to those available to children in Head Start, along with broad social and educational activities for parents, such as job counseling, child care assist- ance, health screening, housing improvement, and legal and other services. They go beyond Head Start and other early intervention programs in enabling greater parent participation in determining the direction of the program, in emphasizing both child development and family services, , in assuming a greater role in coordinating services in the community for participating families, and in improving family services in general for the community. This paper emphasizes the federal demonstration program models, particularly the Child and Family Resource Program and Home Start. Of course, this paper represents only one approach to issues affecting families. It does not survey the range of policies and programs that cur- rently affect family life. For example, this paper does not focus on entitlement programs, such as Title XX, which legislates broad services for low-income families

205 (e.g., day care, homemaker service, family planning, etc.) Nor does this paper consider policies and programs affecting family life in countries other than the United States. Some of the provocative implications of policies in Eurone and Asia can be found in the writings of Kahn and Kamerman (1975). There are also important single service programs (such as those for day care or health) that influence the quality of family life (see Travers et al. and Levine and Palfrey, in this volume). The set of models discussed in this chapter were selected to illustrate important issues in measuring the effects of family programs. First, these models have been implemented in a variety of urban and rural settings nationwide and have served many different subcultural groups of families: Appalachian, urban white, black, Hispanics (both Puerto Rican and Chicano), native Americans, and many other regional groups. Second, they have attempted to implement a personal- ized, direct approach to delivering services that involves the establishment of relationships with families. Compared with an entitlement program that theoretically serves all eligible families, these programs have a more intensive approach, serving fewer families in greater depth. Their approach is similar in some important ways to a clinical community mental health model of family service and in many other important respects is different, too. Finally, all of the federal and private family service programs mentioned were conducted simultaneously with a research program; thus, these programs and the evaluations associated with them represent the most recent attempts to grapple with the research problems inherent in family service programs. Thus far the research suggests that however promising the comprehensive family service models may be, they pose conceptual and practical problems for research that are not yet solved. In this paper the problems posed by family service programs and the efforts to grapple with these problems are organized around three features common to most of the programs mentioned. These features, in turn, have major implications for measurement: · the mix of support and intervention; · the emphasis on individualized treatment; and · the role of integrating community services for families.

206 Throughout this paper the discussion of programs and measurement issues rests on a broad general assumption that it is necessary to consider not only measurement but also the social context in which programs operate and the values of different groups who may have a stake in the program. Several researchers have observed that research questions and measures concerning child development have historically reflected the values of individuals and of society at any given time (Kagan et al., 1978; Kessen, 1979). While this observation may apply to all research about children and families, it is particularly true of research concerning social programs that have diverse constituencies. This is not merely a philosophical stance. Because of the comprehensive and individualized nature of the programs, a great many domains of family life might be affected by program participation. The challenge to the researcher is to adopt a framework for research that helps to make explicit the different values, adopt or impose some priority among the effects of interest, and select measures capable of both detecting effects of primary interest and describing other possible effects of interest to other constituencies. OVERVIEW OF F=ILY POLICY RESEARCH Trends in Family Research Family research has roots in many disciplines: rural and urban sociology, economics, and many areas of psychology--developmental, educational, and clinical. Until recently, research in these areas proceeded in parallel, with little exchange across disciplines. Generally, policy research concentrated on outcomes for children and on defining program treatments that children received, but it largely ignored outcomes for family members other than the mother, for families as a whole or for social groups, institutions, or communities. A review of family and family-related policy research published in 1978 concluded that although more than 26,000 studies of the family had been conducted since 1974, the questions posed by family-oriented research represented "new questions for policy, and for behavioral science as well" (Newbrough et al., 1978:85). The "new" questions were concerned with the reciprocal influences among family members, the relationships between families and other social organizations or institutions (extended families,

207 schools, and social programs) over childrearing, and the intended and unintended effects of intervention programs. Overall, the questions signal a broadening of concern, from the child as primary focus of research and policy to the family and the social groups and institutions that are intertwined with family life. Because this broadening of concern is reflected in federal and privately sponsored research, it is useful to summarize the more general views that are often associated with it. · Families remain important units for the socializa- tion of children, and outcomes for families as well as children must be the concern of social programs and policies. · Families are characterized by a dynamic process of development as a group that is similar to but not the same as the development of its individual members. In addition, certain states in the normal course of family development are likely to be sources of stress, as are the extraordinary events of family life (Hill and Mattessich, 1977). · Families are both social and psychological units with various sizes, memberships, and values, which reflect subcultures, regions, and ethnic and religious backgrounds (Hill, 1971; Nobles, 1976). · There is an "ecology of human development" (Bronfenbrenner, 1977) in which the family is an important one of many interrelated social groups and institutions which overlap in the care and socialization of children (Hertz, 1976). The Newbrough report summarizes the work and findings of many researchers when it suggests moving federal policies away from programs of categorical services (i.e., programs that provide only health or only child development services) toward providing support in many forms to families based on varying needs from family to family. Furthermore, the report suggests that such programs of support should include mediating structures in social environments (churches, kinship networks, and others). Optimal development of children depends on an ever- broadening circle of influences. These influences begin with the child's individual ability and temperament and move to the immediate family and household, to the extended family, its history and social place, and beyond, to the informal groups and formal institutions of society.

208 It is clear that such views about the interdependence of families and social institutions imply a complex concep- tualization of family policy that must be reflected in research. Assumptions About Family Policy The foregoing summary of family research implies that measures of program effects are influenced by assumptions concerning values and social and personal responsibility. Such issues will provide discussion and disagreement among policy makers and researchers. Neither policy nor research, however, can be made without recognizing the importance of such values. Consequently, we present some of the assumptions we make about current family policy. First, we assume that a climate of belief persists concerning the value of providing some types of service or support to American families, although models and systems of service delivery may vary from agency to agency. Second, equity among different income and class groups will continue to be important in determining policy, although the political and economic mood of the country in mid-1981 raises questions about the differ- ential effects of budget cuts on poor, working-class, and middle-class families. While the limitations of program eligibility for low-income families have both desirable and undesirable consequences, those families will continue to receive some types of federal support. Third, we assume that the federal government will maintain some distance from direct intervention in family affairs but will continue to exert indirect influence on families through the types of financial and service structures adopted (services, programs, and stipends) and the types of coordination required between agencies, schools, courts, etc. Fourth, we assume that there is some awareness and an acceptance of the diversity of family types and life- styles in this country. If we accept the integrity of various family types and ways of functioning, we can also assume that it is difficult to agree on a single treat- ment, attitude, or behavior that is the "best" way to enhance child and family development. What is ideal for children and families may vary by region, community, subculture, or developmental stage. Finally, we assume that most policy decisions are political and may not directly reflect the use of

209 research. Traditional outcome research is only one of several types of information that policy makers use, depending on the timing and political issues surrounding the decision. However, it is the responsibility of researchers to be aware of the assumptions and implications of the research questions asked and the methods used (Weiss, 1976). COMPREHENSIVE FAMILY PROGRAMS Program Descriptions In 1973 the Administration for Children, Youth, and Families (formerly the Office of Child Development) initiated the Child and Family Resource Program as part of the Head Start Bureau's research and demonstration program. There are 11 of these programs across the country, 1 in each of the 10 regions and 1 representing the Indian and Migrant Division. Each program receives approximately $130,000 per year to serve a minimum of 80 families; some programs serve as many as 350 families. The Child and Family Resource Program is a family oriented child development program, providing support services to families with children from the prenatal period through age eight. It meets children's needs by working through the family as a unit and provides continuity in serving children during the major stages of their early development. This is accomplished through three program components: (1) an infant-toddler component serving parents and their children in the prenatal period through age three; (2) Head Start, for families with children ages three to five; and (3) a preschool-school linkage component, to ensure a smooth transition from preschool into the early elementary school grades. Another distinctive feature of the program is its emphasis on a comprehensive assessment of each family's strengths and needs and the development with the family of an individualized plan for services. There is much variation in the structure and staff rules from program to program. Some programs have separate staff providing child development services, social services, and health services to families; in others a single person has primary contact with the family, although this person is backed up by specialists in the three areas. All programs provide a mix of home visits and center activities for mothers and children. All have a variety of educational

210 and recreational activities for parents in addition to child development education. Recent information and evaluations of the program can be found in Nauta (1981), Nauta and Johnson (1981), and Johnson et al. (1980). Home Start operated actively as a demonstration program from 1972 through 1975; it was designed to demonstrate methods of delivering comprehensive Head Start-type services to children and parents for whom a center-based program was not feasible. Much of the program--parent education, social services, and child activities--took place in the home. All of the 16 programs also had center meetings, where children and parents got together two or three times a month. Many of the programs were in rural areas; guidelines were explicit in the intention to provide services as families identified them. In 1975 six Home Start programs became regional training centers for providing assistance to Head Start programs that wanted to adopt a home-based component. Like the Child and Family Resource Program, Home Start emphasized the whole family and not just children; unlike the Child and Family Resource Program, Home Start served only families with preschool-age children and enrolled them for two years at most. Information about Home Start can be found in Hewett (1978) and Love et al. (1976). The Brookline Early Education Project is a privately developed and funded experimental program in a single service site, Brookline and Boston, Massachusetts. Initially based on the child development philosophy and research of Burton White and the work of White and Watts at the Harvard Preschool Project, the program sought to provide support and parent education to mothers of chil- dren from the prenatal period to age three. The program provided a physical center and educational materials on a lending basis; families were encouraged to use the resource center and to take material home with them. Each family also had a visitor who helped the family acquire educational information and practical experience in playing with and teaching their own children. The visitors also provided assistance to mothers in need for coping with everyday family problems. The program also included comprehensive health screening and treatment referral services. Both low-income ana mlaale-class children were enrolled in the program. Three levels of service were offered, and families were randomly assigned to the levels for purposes of research. (Information about the program and its evaluation can be found in the paper by Levine and Palfrey in this volume.)

211 Program Features Three characteristics common to family service program models) were selected for discussion in this paper: the mix of support and intervention in the structure and content of family service programs, the use of individual- ized treatment for families, and the program's role as coordinator of services for families in the local community. These features were selected because they illustrate important aspects of family service programs that distinguish them from the more traditional child development intervention programs. These same features also constitute difficult problems for measurement of program outcomes. We noted earlier that the broad goals of comprehensive service programs and their individualized treatment of families create a long list of possible outcomes of potential interest in evaluating the programs. Effects may be anticipated in health and in educational, psycho- logical, and social functioning for children, parents, and other family members. There may also be effects on the ways in which families and individuals in other social groups or institutions interact as well as effects for the services generally available to families in the site of program operation. Possible effects can be organized into three broad groups. The first set of effects are those for individuals within the family household itself. These may include effects on child development, adult development, parent education, parent-child interaction, and intrafamily relationships (e.g., marital or sibling). Child development intervention programs have traditionally concentrated on studying effects on the cognitive and Throughout the paper "model" is used to refer to a specific program that was created by federal or private research with a particular set of guidelines and goals (e.g., the Child and Family Resource Program, Home Start, the Brookline Early Education Project). The term "program" refers to the local project(s) implemented under those guidelines according to a specific model. There are 11 programs following the Child and Family Resource Program model; 16 programs following the Home Start model. For the privately developed models, one project represented the program model.

212 physical development of children and on parent-child (especially mother) interaction. Another set of effects are those concerning the relationships between families and the informal organizations (social groups, extended families) and formal institutions (schools, agencies) of society. Effects of this sort might be observed in the role of parent involvement in the child's public schooling or the ability of parents to obtain regular income for the family. A third set of effects are those for the service delivery community at the site of program implementa- tion. Since some program models have as their goal improvement of services for families in the community, examples of these effects are improved prenatal care or new links between agencies serving families in the community. The researcher must decide which set of effects is most important and, within each set, which particular outcomes are of greatest interest. The different groups interested in family programs have different views of the priority among these effects, which is suggested by many aspects of a program's philosophy and practice. Each of the three features and the measurement problems associated with them exert influence on the type of effects selected for study and the methods selected for measuring them. In the discussion that follows the potential effects and values of different constituencies, which are mentioned but not emphasized, are provided primarily as a context for the discussion of measurement problems. The Mix of Support and Intervention The family programs described in this paper have been influenced by many social programs and lines of social and psychological research. Inherent in many of the programs and research is a dynamic tension between intervention in the lives of children and families and support of their strengths and capabilities. The two views have been characterized by some researchers as the "deficit" and the "strength" models of family functioning. Both support and intervention are implied by the very broad guidelines that defined the Child and Family Resource Program, Home Start, and other programs funded by the Administration for Children, Youth, and Families. Support was implied by the wording of the Child and

213 Family Resource Program guidelines to build on the existing strengths of families and to enhance the total development of children, by working through the family and by offering diverse social and psychological services as needed by each family. Intervention was also implied by the emphasis on educational activities for children and parents and by the guidelines that limited eligibility to low-income families. Theoretically it is possible to see support and intervention as two different, essentially philosophical approaches to the operation of programs--philosophies that influence program structure, content, and the nature of the relationships between staff and families. It can also be argued, however, that the two are inseparable-- that providing support to families can serve as an intervention and that change, the primary goal of intervention, is more likely to occur in an atmosphere of support for parents. Because support and intervention are intertwined in most programs, it is difficult to distinguish them from each other. The list below comprises somewhat stereotypical attributes of the two approaches, as a way of contrasting the hypothetical extreme for each approach: INTERVENTION Change desired in children, mother Change expected during or immediately after treatment Treatment provided for one period of enrollment, though this may be intense for one to two years Standard treatment defined by professionals outside the family SUPPORT - Change in children, parents, and other family members, social institutions Change may be short term but more likely to take place episodically over years Treatment may be intensive at first but intermittent over years, as needed or desired by family Treatment individ- ualized and deter- mined by family and staff; emphasis on parent imitative

214 Implied deficit model of family; family problems most important Professional staff, often with educational and social work roles filled by different staff Emphasis on cognitive develop- ment for children and parent education, the latter focused on child development and educational topics Nondeficit model implied; strengths as well as needs important Staff with multiple roles; mix of professional and paraprofessional staff Child development and parent educa- tion important, but but other needs of family may take precedence; broader range of social and psychological services In fact, none of the programs mentioned in this paper are solely interventionist or supportive in their approach. Programs with different mixes of support and intervention differ on many dimensions, thus influencing research decisions about: · Who is expected to change . . . (children, parents, or social agencies). · What is expected to change (childrearing attitudes, coping behavior, the use of services, or quality of housing). · When change is expected to occur (immediately, within months, or over a period of years). · Who is responsible for initiating and accomplishing change (parents, professionals, or members of the extended family or social network). The mix of support and intervention is the source of much ambiguity in the family service concept. The ambi- guity in philosophy and program implementation encourages ambiguity in expectations by various concerned groups about what such programs do or should do and the criteria by which they should be studied. Understanding the mix in family programs should help researchers know how to look at the treatment provided by the programs as well as what effects and criteria for measurement may be most appropriate.

215 Three characteristics of programs are often implemented in different ways according to whether the program's philosophy is toward support or intervention in its approach: · The relative emphasis on education and cognitive development versus a range of social, educational, and psychological services. · The role of parents in determining treatment. · Policies about family enrollment and participation. The tendency for traditional intervention programs to emphasize cognitive and physical development for children rather than a range of developmental services has been mentioned. Support-oriented programs may have educational components and may place high values on certain types of education, especially for parents, but the child's school readiness or performance is not the primary aim or cnz' a development activities. Likewise, traditional inter- vention approaches emphasized certain aspects of maternal teaching behavior (such as verbal behavior or the ability to structure learning activities); a support approach may be more attentive to the affective quality of the mother and child relationship or the socialization aspects of childrearing as indicated by the mother's interest and need to know. The role of parents in determining treatment can be observed directly by looking at their participation in planning or organizing activities. It can also be observed indirectly in the structure of the program and in staff attitudes toward family participation in program operations. - ~ ~ ~ ~ ~ ~ staff as professionals and families, and the more parents have to say about the types of services and activities they get and the more a program leans toward support rather than intervention. The less formal the distinction is between Policies concerning family enrollment and participation also illustrate philosophies of intervention and support. Although Home Start and the Child and Family Resource Program were intended for low-income families, there was variation in the types of families actually recruited as well as variation in program expectations for participa- tion by families. Some programs recruited families already interested in Head Start; others sought out low-income mothers with new babies through clinics and hospitals. Some programs limited the enrollment of families in crisis, believing

216 that other agencies could better serve them. Others chose to emphasize enrollment of families of particular types (e.g., single parents, teenage mothers, and rural families). Expectations for participation varied primarily in the length of time enrolled or the intensity or frequency of participation while enrolled. Home Start families were enrolled for two to three years and were expected to participate regularly: weekly home visits, group activities for children every other week, and parent group activities a few times a month were common. In the Child and Family Resource Program, however, there was more variation among programs in expectations for participation. Some required contracts with families specifying a schedule for participation similar to Home Start; others allowed great differences in intensity or frequency of participation. A family could be quite active (one or more contacts with the center and weekly home visits) or relatively inactive (monthly telephone calls initiated by the family). Variation depended on whether the family was new to the program (usually more active), was undergoing crisis, or was temporarily too busy with other familiy business to participate in program activities. The variations in type of family enrolled and in levels of participation have far-reaching implications for the treatments delivered by programs and the outcomes expected as a result of the program. For example, gains in chil- dren's mental development in Home Start and the Child and Family Resource Program seem linked to regular participa- tion over a fairly short period of time (less than two years); this finding is consistent with those from intervention-type programs in short-term effects for mental development. For longer-term gains, anecdotal evidence from the Child and Family Resource Program suggests that changes in family circumstances or the coping skills of mothers take much longer; program staff report changes in some families in these areas after two years and in others after four or five. Thus, different levels of participation may be appropriate for producing outcomes in different areas of family functioning. Likewise, different levels of participation may also be appropriate for different types of families. Some individuals seem able to make a commitment to the program and participate regularly right away; others develop similar relationships of trust with program staff only after long periods of weeks or months. Many parents

217 expressed the view that the appeal of the program was that "it was there when we needed it"--in other words, it functioned truly as a resource and support program. Not only do programs often have different expectations for participation depending on the type of family or the area of greatest family need, but they also may vary in the intensity of service or in expectations for participa- tion over time for specific families. Thus, one family may participate regularly for two years, then leave the program; another may participate irregularly at first for a year and then regularly for two; still another may participate regularly for two years, then decrease steadily in participation over the next three years, stepping up contact with the program in times of special crisis. All three families might be served within a single program. In fact, length of time enrolled and intensity of participation are major ways in which treatment has been individualized in these demonstration programs. The Individualization of Treatment Individualization of treatment is entirely consistent with, even inseparable from, the mix of support and inter- vention in family service programs. Because individual- ization is closely related to the mix of support and intervention and because it is central to the philosophy of these programs, we provide an overview of individual- ization, followed by a discussion of some attempts to study programs that provide such treatments. Though something of a misnomer when applied to a family unit, "individualization" refers to the process of planning for and the resulting pattern of activities and services a family actually gets as a result of partici- pating in a family service program. Most programs mentioned in this paper have some individualized treatment; the Child and Family Resource Program has the most, as treatment is determined to a large part by the parents in conjunction with the staff. To understand individualization it is necessary to understand the process by which plans for families are made and what aspects of the program are actually varied. In the Child and Family Resource Program the planning begins with a thorough assessment to identify what parents want and children need. A specific plan is developed, recorded, and approved; the plan, which varies

218 in detail from program to program, is reviewed and revised periodically. Areas considered in assessment include the health of all household members; needs for necessities such as housing, clothing, food, and utilities; arrange- ments for adequate income, which may include education, training, or job interviews; and needs for social services such as legal assistance, day care, or recreation. In many programs, less tangible personal goals may be con- sidered, such as social activities, new skills training, or personal counseling. A developmental assessment of children and full health screening are also included; sometimes a nurse visits the home to conduct health inventories. The plan that results from this assessment is approved verbally or in writing. In some programs, parents meet with program staff and representatives of other agencies who provide services identified in the family's plan. What is individualized? Structurally, one important source of variation is the mix of direct services (by program staff; e.g., informal counseling of health education by a program nurse) and indirect services, through referrals to other agencies. Another source of VarlatlOI1 IS the mix or center- ano nome-oasea aches that parents elect. In some programs the mother may choose to have monthly home visits from a teaching staff member and weekly sessions for her child at the center. In other programs all families are expected to participate in the same schedule of center and home activities; variation may occur with the grouping of certain types of parents (single mothers, teenage mothers) and activities tailored to their interests. Another source of variation is the content of home visits and center group meetings. Which of the many aspects of child and family development (health, social and emotional relationships, etc.) are covered and how these topics are addressed (group discussion, role playing, etc.) are important determinants of program treatment. Finally, variation occurs in the nature of the relationship between parents and staff. This variation is difficult to characterize but concerns the degree to which the parents are self-sufficient in identifying goals, interacting with children, and participating in Such variation in the relationship program activities. between staff and parents may reflect the style of the staff member, the program philosophy, or the circum- stances under which the parents came to the program

219 (e.g., court referral, interest in child development, desire for social contact). In short, virtually all aspects of the program may be varied. A summary of common variation includes the following: · The mix of direct and referral services. · The mix of center- and home-based activities. · The types of actions and goals identified for families. · The time of participation (regular weekday, weekends for working parents). · The ~~~-~~-~ parents, ~_' Involvement ot other family members (grand- siblings, other care givers). one type of child development or parent education activities offered or the relative emphasis on child versus family services. While these aspects can be used to identify the type and degree of individualization in a program, they are collected as either input or treatment variables in program evaluations. In most family service programs the number of dimensions on which treatment may vary is large, a fact that poses a fundamental problem for researchers. Measurement Problems The foregoing discussion pointed out the complex nature of family service program models that deliver individual- ized treatment--treatment that combines support and inter- vention in different degrees for specific families and in different proportions across programs. Several problems for measurement of such programs were mentioned. First, comprehensive family programs are difficult to measure because they have broad goals for diverse effects on families and communities. These programs have many constituencies (parents and children, program staff, policy makers, advocates, taxpayers, researchers) whose values dictate different priorities for program goals and effects. Second, the mix of support and intervention in an individualized program makes definition of treatment particularly problematic. In effect, there are as many treatments as there are families. Clustering families by the patterns of service they receive is difficult because

220 the combinations of potentially important treatment variables are numerous. Third, it is difficult to determine which of the many variations in treatment might be the important ones. Assuming that program goals might be agreed upon among some concerned groups and treatment reasonably well defined, there is still Ohm Problem of being unable to determine which aspects of treatment were responsible nor the effects observed. - , ~ _ · . . ~ Thus, replication of "successful" program features would be impossible. Finally, there is the problem of assigning value to different patterns of treatment and outcome and of determining whether the treatment and outcome make sense for the individual child or family case for which it was designed. The inability to evaluate the appropriateness of treatment or of treatment-outcome relationships is a fundamental problem in evaluating these programs; it raises other questions about the nature of public and private services to families: · Whose values are more "right," parents or staff? When do parents know what's best for their children? What constitutes a staff member's "providing alternatives" in childrearing, and what constitutes imposing one's values on another? · What are the costs to families and society of substituting professional relationships with those that were formerly provided by families, churches, or other community organizations? · What is the effect of providing role models in the form of staff members who may have social or political views different from the family members who enroll? What is the implication of providing support and intervention only to low-income families or to mothers who are not working full time? Many of the problems described above are higher-order problems of conceptualization, definition, and values. Without clarification of these problems, however, no methodological solutions are likely to produce the answers to the most important research questions to be asked about these programs. Many of the approaches taken by researchers to minimize these problems are aimed at clarifying such higher-order problems.

221 ASSESSING THE PROCESS OF INDIVIDUALI ZED TREATMENT The discussion of support, intervention, and individualization above was intended to illustrate the most important characteristics of the family service model: When implemented as mandated, these programs may determine a unique set of goals and treatment for each enrolled family. Theoretically r then, it might be impossible to evaluate outcomes for participating families except on a family-by-family basis. In recent years, description and measurement of program treatment has been justified on the grounds that it is useful in interpreting patterns of outcome observed between treatment and control groups (Hewett et al., 1979). There seem to be many reasons why description and measurement of treatments are desirable in themselves: The general goals and philosophical mix of support and intervention may be different from model to model and community to community, the needs of families and communities that help shape local programs differ, the identities and views of different groups may be unclear, treatment is likely to be individualized, the expectations and relationships among staff and families also vary. In terms of evaluation research, the description or defini- tion of program goals and treatment are often referred to as the study of "process," as distinct from "outcome" (Ross) et al., 1979; Goodwin and Driscoll, 1980). At least three types of "process" variables have been distinguished that are appropriate in studying family service programs: · Indicators that the program delivered the services mandated by guidelines or dictated by program goals (Zigler and Trickett, 1978). · Indicators of how much (hours of contact, number of visits) or what type of treatment (education, one-to-one relationship) was delivered. · Indicators of the dynamic relationships through which treatment was delivered (staff-family, parent- child, staff-community agencies). All three types of process variables may be captured through quantitative and qualitative measures, although the nature of relationships lend themselves more readily to qualitative measures in general. We believe that the nature of family service programs requires the flexible and selective use of many quali

222 tative and quantitive measures to capture all three types of program processes and treatment. Not all studies need to collect data about goals, treatment, and processes in equal proportions; the selection of types of process variables and appropriate measures is dictated, of course, by the types of questions to be answered by the research. Three general questions, which should be asked for any family service program, are important for understanding and assessing such programs: · What goals are expressed for this program, and what constituencies (local and other) do the goals represent? · What actually goes on in the program, and what measures of process (treatment) can be used to capture it? Can any be used to relate treatment to outcome? · What issues for policy or measurement are raised by the study treatment or process? What anticipated outcomes are suggested? What goals are expressed for this program, and what constituencies do the goals represent? The notion of identifying program goals first is a basic tenet of evaluation research. With regard to family service programs, it is easy to see that broad guidelines may be interpreted locally in different ways in response to different local conditions. More difficult still, there are likely to be different sets of goals that represent the groups who have a stake in the program, both at the community and the state or federal levels. Policy makers may want to provide service and to learn the forms a program may take if left to local implementa- tion. Program directors may see their programs as social advocacy groups, mental health facilities, vehicles for individual self-help, or preventive and compensatory child development programs. Parents may see the program as a source of accessible advice about childrearing, an entry into a community system of child care, or a chance to make new friends and participate in social or educational activities. But these are not the only concerned groups. Evalua- tors and social policy advocates in academic and political positions may see programs as laboratories for human development studies, as threats to the natural order among family members and groups in society, as platforms for grass-roots political organization, or as models for preventive community mental health.

223 Identifying goals may be a complex task, even for a single set of constituents. The Brookline Early Education Project emphasized the goals and expectations of the parents, collecting tremendous quantities of process data from parents about initial expectations, experiences with various program practices, and satisfaction. The Home Start evaluation compiled case studies of the historical and organizational roots of each local demonstration program and included a case study of the national program office and its development of the federal program model. Eventually the evaluator or researcher must choose some goals as the basis for structuring evaluations; the choice of a set of goals (whether those of policy makers, other researchers, or program staff) represents an implicit stand about their relative importance. Disproportionate attention to goals can be a trap for evaluators (Weiss and Rein, 1979); it is important to recognize the limitations of the use of goals sets or other contextual data in defining treatment. What actually goes on in the program, and what measures of process (treatment) can be used to capture it? Can any be used to relate treatment to outcome? What goes on in a program can be characterized both qualitatively and quantitatively--measured or described, according to the purposes of the research. Selection of a particular group or process within the program for in-depth study is dictated by the orientation of the evaluator and the goals selected for study. Qualitative methods are often appropriate to compile a thorough description of the purposes and realization of the program. Especially for programs based on a new model or guidelines, the very process of compiling a description of the program is likely to be useful to evaluators and to new audiences for the program. For evaluators the descriptive qualitative study will help focus attention on the most important program features as well as those most difficult to measure. Preparation for such a study may identify descrepancies between goals and practice and raise questions about unanticipated outcome. The issues and questions raised in a good descriptive study may be useful in themselves for policy makers and may serve as a basis for subsequent decisions about study design made by evaluators. In the Brookline Early Education Project the largely descriptive process study raised many important questions about the "appropriate" role of staff who visit homes in different subcultures served by the program. In the Child

224 and Family Resource Program both descriptive and process studies were used. The descriptive study attempted to determine through interviews with staff and informal observation of program activities whether there were common models actually in operation among the 11 sites. On the basis of this descriptive study it was observed that although certain methods of delivering particular services such as health or infant and toddler services were common to several of the local programs, no overall models of program structure could be identified. Such observations might have bearing on guidelines for future programs of this type. Another aspect of the evaluation of the Child and Family Resource Program used a different qualitative method--ethnography--to capture the interpersonal processes between staff and families. Evaluations of educational programs have used this method for studying certain types of questions (Stake, 1978); at present, an ethnographic study is under way in five sites. At each site, trained ethnographers who are familiar with the community are spending extensive time with families during a six-month period to understand the role of the program and other social networks or programs. Ethnographers are also observing among program staff the process that goes along with maintaining relationships. With the exception of the ethnographic study, the descriptive studies described above relied on some forms of quantitative information (e.g., number of group meetings) in assembling the qualitative picture. In general, the qualitative and descriptive methods require continuous cross validation between the evaluation staff visiting the program and the program staff; these methods are both subjective and time-consuming. As an initial step in assessing such programs, however, the investment of time and the mix of subjective observation and simple quantitative data are necessary for a broad understanding of what actually takes place in the program. Because it attempts to assemble a whole picture of the program and its various meanings to staff and participants, a descrip- tive program study is one of the major sources of judgment about the quality and appropriateness of service. Qualitative description is sometimes disregarded because it cannot be related to outcome measures. How- ever, from studies of parent education programs up to 1978, Goodson and Hess (1978) observed that only the most global treatment variables were common across programs

225 and useful in relating treatment to outcome. Among the variables were the presence of a structured curriculum, the role of parents in decision making, the combination of home and center activities, and the evidence of a one-to-one relationship between staff and families. Thus, even the most broad qualitative aspects of the program can be useful in characterizing treatment. Quantitative measures of treatment are also important. Most of the demonstration programs and private family service programs of the Administration for Children, Youth, and Families used quantitative treatment measures. Some were used simply to describe program operations; others were related to outcomes for families. We consider two types of quantitative treatment measures: systems of records kept by the program staff and systems of observations of program activities by evaluators outside the program. Systems of Records Typically, systems of records include detailed records of services and activities delivered to clients and are maintained by program staff. They often record numbers for referrals, transportation, home visits, center group sessions, and other types of contact between staff and In programs characterized by individualized records are typically kept for specific An evaluator may use records kept by the program internally2 (which may differ in detail, regularity, and content) or require that a separate system of evaluation records be kept. Typically these systems of records attempt to capture measures of participation, although some evaluations also attempt to record family stress, motivation to partici- pate, staff perceptions of families, and other character- istics of the family-staff relationship. In Home Start and the Child and Family Resource Program, participation families. treatment, families. . . . . . . . 2This was the approach used by the General Accounting Office team in the review of the Child and Family Resource Program described at the beginning of this paper. They were not concerned with comparability of treatment across programs.

226 measures were found to be useful as covariables to explain different patterns of outcome for children in the program. In Home Start, children's gains on cognitive and language outcome measures dropped or disappeared when families made fewer than two visits per month. In the Child and Family Resource Program, children's outcomes on a develop- mental measure (Bayley, 1969) were related to a complex measure of participation (rate over time of participation in both home visits and center activities). These findings suggest that there is an effect of participation in program activities beyond mere enrollment in the program, although there were no overall differences in development between children in the Program and a . - . comparison group not receiving program services. While this is a successful use of treatment variables to explain outcomes, it captures only the broadest aspects of treatment. Attempts to capture the more individualized nature of treatment in the Child and Family Resource Program have horn made with the use of familY-sPecific goal records kept by staff as part of the program s study or treatment. There are many technical problems with this approach, some of which have been documented in evaluation literature (Kiresuk and Sherman, 1968; Kiresuk et al., 1978) e However, as the approach has not been widely used in family service programs and does provide useful descrip- tive information, we will describe some of the problems encountered with this approach. Goal setting and goal attainment have been used as measures of outcome primarily in mental health settings making use of certain types of treatment: short-term, behaviorally oriented approaches in which there is explicit agreement between client and therapist about what they will try to accomplish, how long it will take, and what each will be expected to do. These character- istics of the treatment process are similar to the explicit goal-setting process that is common in the individualized treatment of the Child and Family Resource Program. The forms currently used record a description of the goal, who it concerns (the mother, child, or whole family), the type of goal (health, employment), and when it is expected to be completed. the staff report on the status of the goals: changed, much progress, no progress, dropped. In this spare format, goal records provide a profile for each family of the focus of treatment, the types and sequence At regular intervals,

227 of services used, and whether many goals are dropped or changed--a possible signal of disinterest or mismatch between staff and family. Aggregated over all families, goal records can provide a rough empirical picture of the program's overall treatment emphasis, whether on immediate needs, such as housing, or long-range goals for mothers, such as employment. Problems with goal attainment as a treatment measure exist at several levels. One is validity: Goal records may not represent the family's true goals and private reasons for coming to the program or the implicit goals the staff may be pursuing for the parent (such as more regular attendance at the program). Others include the reliability of the records, the level of detail at which a goal should be stated, the difficulty in distinguishing between services that are part of treatment (like health screening) and goals, and the incentives for staff to maintain such records regularly. As outcome measures, records kept by staff give an automatic advantage to staff or programs that are already record oriented or sufficiently practiced or educated to be able to translate their work well into record form. In programs in which former program parents often become staff members, they may lack education, practice, or a "professionalized" attitude toward recordkeeping, which serves to penalize their families on that measure. Aside from the many practical problems with goal records as measures of treatment, there is a more important conceptual problem with using them as outcome measures. It is virtually impossible to assign values to different clusters or sequences of goals. Are goals for maternal employment and housing repair more important than goals for participation in program activities and group activities for a preschool child? Is limited progress toward a goal such as "to settle custody rights with the children's father" more or less important than completion of a goal such as Into obtain hearing and visual screening for the child"? Clearly, attainment can only w~ MA i" t~=rmc! of :` n~rt;~lil~r family. not as a In ~ of_ ~ ~ J ~ ~ C ~ ~ ~ _ ~ J _ ~ ~ ~ ~ ~ ~ , basis for aggregating or comparing outcomes among families. There are no models of what is normative or desirable for each family (except what will generally enhance the development of the child); thus, the study of goals and goal attainment describes the individualized treatment but does not help determine whether it is an appropriate treatment for the family.

228 Systems of Observation Systems of observation, both in-person and videotape, have been used to capture treatment variables. In Home Start the observations were modest, carried out by field observers who accompanied home visitors to record the content of the visit, the focus of activity (parent or child), and a few other variables. While this information was not used to study outcomes, it did have immediate consequences for policy from the national program office. From these observations it became apparent that although the focus of Home Start was on the parent, most home visitors spent a majority of time interacting with the children. Subsequently, the national program office provided additional direction and opportunities for training to help home visitors work effectively with adults. Videotaped observation systems have been used to capture process variables (as in the evaluation of the Parent Child Development Centers) or outcome (as in the Child and Family Resource Program). These systems are usually prohibitively expensive for large evaluations. In the Child and Family Resource Program they were used only at selected sites. What additional questions are raised by the process measures? A final important function of process studies is to identify additional questions about the program under study. Researchers routinely suggest further research on the basis of their studies; less frequently, however, researchers try to identify how their methods failed to capture what they wanted and what that failure suggests for different formulations of the problems being investigated. Weiss and Rein (1969) undertook such an analysis when they observed that they could not detect the effects of a program's efforts to change communities because they looked primarily at changes in individuals. Furthermore, because much policy research is conducted to answer specific questions posed by policy makers, evaluators are encouraged to frame their research as answers to specific questions in order to be useful and effective (see Deloria and Brookins, in this volume). In addition to answering the questions posed, researchers must formulate and raise questions that emerge from research and the problems encountered in research. This view of research as hypothesis generating rather than hypothesis testing is one that has been identified by a number of psychologists and other researchers (e.g.,

229 Weiss, 1972). They maintain that research methods and strategies must be continously modified in order to capture and explain the phenomena they investigate. Studies of treatment and process may take many different forms, using a variety of methods to answer or raise questions about programs. Process studies can furnish a basis for understanding what goes on in a program as well as a basis for selecting appropriate outcomes for study and the methods likely to capture them. And, on a more limited basis, process measures can help relate treatment to outcome. ASSESSING OUTCOMES FOR INDIVIDUALIZED PROGRAMS As elaborated above, family programs with broad goals and individualized treatments require good process evaluation in order to describe what the programs intended to do and what treatments they provided to children and families. Likewise, the goals and treatments of these programs pose problems for the evaluation of outcomes as well as process. This section discusses four such problems in evaluating outcomes: Assessing outcomes across multiple domains. Assessing multiple outcomes within a single Selecting an appropriate unit of analysis. Selecting appropriate comparisons. Although each problem has distinct implications for evaluating impact, all four are interrelated. These problems arise directly from characteristics of the comprehensive family service programs that we have used as illustrations throughout this paper--Home Start and the Child and Family Resource Program. The problems are best exemplified by reference to specific features of past and current evaluations of these two programs. Features of the Program Evaluations Evaluations of Home Start and the Child and Family Resource Program were conducted at different times during the past decade. The evaluation of the Child and Family Resource Program was able to build and expand on the design of the Home Start evaluation (e.g., by including an ethnographic study of process and outcome). In many

230 important respects, however, the basic design and analyses of the two evaluations were similar and certain common evaluation features help illustrate the problems selected for discussion. Both Home Start and the Child and Family Resource Program evaluations were charged with investigating multiple possible outcomes for families and children as a result of the program. In choosing to emphasize the role of the family in child development, federal program designers implicitly accepted the assumption that a wide range of outcomes could be addressed directly or indirectly through program intervention or support. In addition to the child's own temperament and abilities, outcomes were expected in the behaviors of parents and other members of the immediate household and in the functioning of the family in relation to neighbor- hood support and social service resources. Programs were also expected to mediate the influences of forces outside the family; for example, opportunities for employment or education in the local community or state regulations concerning Aid to Families with Dependent Children (AFDC). Furthermore, there was an explicit assumption that programs could bring about gradual changes in community institutions or child care policies. There were three potential units or levels of analysis In both evaluations: the individual (child or parent), the family or household unit, and the institution level, which could include outcomes for local service providers or for broad (state or federal) program policies. The institutional level was qualitatively different from the other two and is discussed in the "Integration of Service" section of this paper, which focuses on outcomes for children and families only. Evaluations of both Home Start and the Child and Family Resource Program organized diverse outcomes into study domains that correspond more or less to areas of psychological or sociological research. Prior work in each research area provided a basis for the selection of variables and measures that would capture change in each domain. Both evaluations eventually included outcome domains representing child development, parent-child interaction, maternal and child health, family circumstances (income, parental education, housing quality, and employment), and family-social interaction (the use of informal social networks, social service resources, and individual coping skills). Each domain was represented by several variables. For example, in

231 the parent-child interaction domain for the Home Start evaluation, there were variables representing such constructs as maternal teaching style and frequency and type of parent-child interaction. Another feature shared by the evaluations was the use of a treatment comparison group design with random assign- ment to groups at each program site included in the evaluation. In general, outcome analyses compared mean differences between treatment and comparison groups using all variables in all domains for all families, and further analyses related process and outcomes for treatment families only. All three features shared by the evaluations reflected accepted practice in psychological and educational research. They were implemented because they would answer certain types of policy questions concerning outcomes for families receiving individualized treatment incorporating multiple services compared with families participating in no such program (or receiving different, unsystematic services). Assessing Outcomes Across Multiple Domains Essentially this problem is the inevitable consequence of attempting to assess outcomes from individualized treatment based on the family's expressed needs and goals. From among the many services available in all domains, a family might choose and receive services or benefits in only two or three. There might be as many treatments and as many patterns of outcome as there are families. One family (or individual within it) may need health services in addition to child development services; another may need job training and child care in addition to child development services. Since program emphasis and treatments vary by domain over time for a family, expectations for outcomes in the domains of treatment or need can logically be expected to vary as well. In both programs there was consensus among program staff that outcomes in two domains--child development and parent-child interaction--could and should be assessed for all families. Nonetheless, there was no agreement about which of the two child-related domains was more important, nor which of the other domains, expected to vary with family need, was most important. This is a conceptual as well as methodological problem. It reflects an important lack of integrative

232 theoretical models for many of the changes or interactions expected to take place within families or between families and social groups, including the Child and Family Resource Program. The problem is not an absence of models alto- gether. There are many models of cognitive and socio- emotional development in children and of psychological and social functioning in families. There are also theories (and implied models) about the reciprocal influences in family-child development (Bell and Harper, 1977) and about how families and social institutions interact (see Newbrought et al., 1978, for a discussion and summary of recent research). There are recent promising attempts to characterize the "ecology of human development" (Bronfenbrenner, 1977), linking the development of the individual child to the functioning of the family and to the wider social systems that support or inhibit that functioning. However, these new theoretical developments have thus far not been widely applied in program evaluation. The competing models and the absence of widely accepted conceptualizations for unifying such models force continued reliance on analyses within outcome domains. It has another result as well. Without well-defined models of the relationship among outcomes in several domains, it is difficult for evaluators to justify emphasis on a particular domain or to fully explore relationships among domains on theoretical grounds. Inevitably, resources must be expended on collecting and analyzing data in each domain; the necessity to be broad in analyzing several domains often precludes in-depth analyses. In the Home Start and the Child and Family Resource Program evaluations, the domains of child development and parent-child interaction were emphasized because of their prominence in the program guidelines and because of the expressed interest of policy makers. While such decisions were legitimate given the goals of the programs and the federal agency, they did limit resources that might have been applied to the problem of understanding outcomes for families as a whole, in different domains, over time. Assessing Outcomes Within a Single Domain Within-domain analyses, combining a number of separate variables, is a microcosm of the previous problem; within a particular domain it is also often impossible to combine

233 variables in a meaningful way. The difficulty in combin- ing them varies by domain, to the degree that previous research in that domain has provided theoretical models and empirical groundwork. There are numerous theories and studies linking specific features of child development to one another and linking features of parent-child interaction to one another. In contrast, in the domain of family-social interaction, theoretical and empirical work has been fragmented. For example, while there is descriptive work relating individual coping behavior to social and institutional supports (Belle et al., 1980), little has been done to develop a theoretical framework for explaining the dimensions of such relationships. An example of a simple within-domain analysis illustrates the problems inherent in analyzing outcomes both within domains and across multiple domains. In the Child and Family Resource Program evaluation, program families at six sites were compared with non- program families on some variables in the domain of family circumstances. One was employment status; another was family income. The findings showed no significant differences on employment status, although somewhat fewer Child and Family Resource Program mothers were employed. The findings also showed that non-Child and Family Resource Program families reported significantly more income than Child and Family Resource Program families. Taken separately these findings seem to show that the Child and Family Resource Program had no effect on jobs and an adverse effect on income. Yet if the two vari- ables are considered together, a more complex picture emerges showing informative differences between sites. For instance, higher family income is reported in sites and groups with more working mothers. Since most Child and Family Resource Programs encourage mothers to stay at home so they can concentrate on parenting, the lower employment and reported income figures reflect this; but in some sites, the Child and Family Resource Program encouraged mothers to work, and the employment rates and reported income are higher in these sites. To further assess the ultimate value of encouraging mothers to work or not to work, we would have to cross domains and examine the respective child development gains. Long-range child development gains resulting from mothers staying at home could conceivably offset the short-term losses in employment and income. Only with more complete within-domain and across-domain analyses can reasons for underlying differences be discovered.

234 The difficulty of combining variables within some domains and of analyzing them into a comprehensive picture of outcomes across domains creates several problems. It makes impossible a view of status and change in the family overall, either at a single point or over time. It makes difficult the compilation of a comprehensive picture for each family or individual in relation to the individualized treatment received. And it may cause evaluators to draw false conclusions about the variables or domains they have studied because they cannot interpret their results. Finally, in the absence of models or methods for combining outcomes in diverse domains, evaluators And n~li~v makers maw anions ~rml~lahl ~ ~ ~ ~ ~ _ = ~ ~ · _ ~ ~ ~ ~ ~ ~ resources co''ece~ng data ot marginal utility or analyzing them in ways that do not prove to be useful or interpret- able. Careful consideration of research questions and the issues raised by the program must guide priorities for analysis among variables in a single domain and for combining variables across domains. When such priorities have been set in the past, child development and parent- child interaction have received the most emphasis; perhaps it is not the time to turn additional resources to the exploration of the complex issues of combining outcomes within and across domains. Selecting a Unit of Analysis In Home Start and the Child and Family Resource Program, two units of analysis were recognized as most important: the individual and the family. Although the program emphasis was on the family, analyses nonetheless used the individual child or parent almost exclusively. In part this problem is similar to the Previous two. That is, if it is difficult to aggregate outcomes for an individual within and across domains, then it will also be difficult to compile and analyze outcome data at the family level, since this involves outcomes for multiple family members. For example, health outcomes are considered separately for children and mothers, rather than for a family as a whole, which would be more consistent with the program's approach. This is a continuing problem because of the stated goals and actual emphasis of the program on the family--not only the family as a group of members at any one time, but also the family as it develops over time. Here the lack of clear developmental models seems most crucial, although a

235 number of family psychologists and sociologists have attempted to grapple with this lack (Duvall, 1977; Burr et al., 1979). For the most part, however, methods that combine information on several family members or on the family as a unit are those developed for clinical work or research with families (Jacob, 1975; Gurman and Kniskern, 1978). It seems appropriate that some of these methods be explored for their utility in evaluating outcomes for family programs. Selecting Appropriate Comparisons In line with the analysis of single variables or measures within domains, the use of treatment and comparison group designs has been the primary means of attributing impact or change to the program. While there are some problems with using only such designs in evaluating some types of compensatory education programs (Campbell and Boruch, 1975) and practical problems (e.g., attrition) associated with using such designs in longi- tudinal research, there are benefits in the approach (Cook and Campbell, 1979). It seems appropriate to retain such designs in future evaluations when questions are to be answered about families receiving one treatment versus another. However, a simple treatment-comparison group design does not preclude the use of other standards for assessing change in individuals and families. Treatment-comparison group designs might well be supple- mented by other approaches to help answer such questions as the following: . . . . . . · How is a particular child or family developing relative to its own prior status or functioning (the family as its own comparison)? · How are program children or families functioning compared with selected national norms for similar populations? · How is a particular pattern of outcomes related to specific aspects of treatment? · How are patterns of outcome different by important family variables, such as age of mother or household type (e.g., single mother)? Answers to these types of questions are important in evaluating outcomes because they help clarify how the programs work or do not work with different individuals

236 and families. Particularly when heterogeniety of both treatment and comparison groups may make it difficult to detect effects based on group means, it is important to understand the relationships between treatment and outcome. It is desirable to have such information in the context of each family, as contextual information may be crucial in interpreting results. Recommendations Taken together, these four interrelated problems suggest that thinking about evaluation designs and analytic methods must be broadened in assessing outcomes for family problems. Additional approaches should supplement, not replace, the experimental and quasi- experimental designs using randomized treatment and control groups now in use. recommendations. First, continuing attention should be paid to the use and further development of theoretical models of family development, family-child interaction, and family-social interaction. Attempts to operationalize existing models should be continued. Second, current experimental and quasi-experimental designs should be supplemented by studies or alternate designs that use the family as a unit of analysis and compare the family's growth with selected others and with their own status at earlier time(s). Third, additional exploration should be undertaken of methods that might be used in supplementary designs or analyses to capture outcome patterns at the family level as well as the multidimensional quality of those patterns. Thus, increased attention might be given to certain forms of multivariate and profile analyses. These methods exist but have not been fully explored for their utility in capturing the multidimensional nature of family status or change over time. In addition, methods developed for assessing clinical work with families should also be investigated for their appropriateness in assessing both process and multi- dimensional outcome. Among these methods might be single-case experimental analyses that have used primarily clinical data (Hersen and Barlow, 1976; Kazdin, 1977) or the case argument method described for children's health measurement by Levine and Palfrey (in this volume). Social anthropological methods using participant-observers We make three general

237 also might be useful; this approach has been used successfully by the Child and Family Resource Program evaluation to obtain qualitative data about treatment- outcome relationships. The methods of data collection and analysis are documented in Johnson (1981) and in Travers et al. (1981). INTEGRATION OF FAMILY SERVICES The third important feature of family service programs is their approach to coordinating services for families within a local community. This approach has potential effects both on the families themselves and their use of services and, more broadly, on the services available at the local, state, and federal levels. This section describes how such integration works and what problems it poses for measurement of outcomes. Family services have proliferated during the past two decades, posing difficulties for both families and policy makers. The confusing array of services available to many low-income families has created a need for programs that can integrate disparate services, to make them more readily accessible to families. We can call this integration function a "linkage" or "brokerage" service to families. The linkage or brokerage service to families contrasts sharply with the traditional single services provided by service agencies, such as health clinics, day care centers, job training programs, food stamp programs, legal assistance clinics, and housing programs. Each agency characteristically provides a narrow cluster of related family services, and for the most part the services available from each agency do not overlap with those of the others. The proliferation of single-service providers has created a strong need for linkage services. It is not uncommon for an urban, low-income family to have access to several hundred services from a like number of separate providers--with each provider having its own office hours, eligibility rules, application forms, contact persons, and separate office locations. This bewildering array of options presents formidable obstacles to families that are already sorely stressed by the routine daily demands of running their households. This creates a dilemma for families: They cannot solve some of the problems facing them without outside

238 help, yet they cannot easily surmount their immediate problems to thread through the bureaucratic barriers to get help. Comprehensive service programs such as the Child and Family Resource Program and Home Start fill a major need in this regard. They are designed to serve as brokers of services for familes, matching available community services to specific needs of particular families. When a family enrolls it receives help in identifying and contacting service agencies from among the spectrum of agencies available in their community. Not only does this limit the family's primary institutional contact to a single program but it also limits it to a single person. The Child and Family Resource Program uses a person called a "family advocate" for nearly all contacts between the program and the family, and this person typically goes to the family's home rather than requiring a family member to visit the staff office. In Home Start the home visitor serves a similar role. The family advocates and the home visitors are specially trained to identify pressing family needs and to link individual families to the proper pro- gram or agencies that can best assist them with these needs. In this arrangement the family members do not have to become experts themselves in the identification and use of community services. We might point out that these Programs are not merely brokers; they also provide many individual services directly. They do carry the brokerage concept consider- ably beyond that usually found in other service programs. Once families are enrolled in a linkage program, such as the Child and Family Resource Program, they receive many kinds of assistance in obtaining services from other sources. The first assistance that families get is information. Nearly all linkage programs provide lists of locally available services, with telephone numbers of people to contact. The lists can serve as permanent reference sources for the families; they often include several hundred contacts in urban areas. Monthly newsletters are often used to explain individual single-service agencies to families in more detail. Program staff often conduct a needs assessment to help families determine which of the many available services would be most helpful for them. The needs assessments are usually updated periodically as family circumstances change, and they serve the program staff as a plan of action for working with each family. The procedure involves either formal or informal discussions with

239 parents to learn their views and wishes and to discuss suggestions of the program staff. The needs assessment results in a joint agreement between the parent and staff on priorities for services. Once the assessment is completed, program staff may either refer the parent to particular service agencies, call the agencies to make appointments for the parent, or actually accompany parents to appointments, arranging for transportation and baby-sitting if necessary. If some payment is necessary, such as for medical care, the program may pay all or part of the costs. In many cases the program follows up the appointment to ensure that the necessary services were provided. When services are withheld from families without good cause or are of poor quality, program staff often serve as advocates for promoting changes in the delinquent service provider--either by organizing parents to represent themselves or by directly petitioning the agency, its board of directors, or the state or federal funding source. As a last resort, linkage programs may directly provide services to families if no other community sources exist. Assumptions About Integration of Services Most linkage programs are based on several unstated but clear assumptions. They first assume that a mismatch exists between families in need and the available services--since, if family needs and the available services were perfectly matched, there would be no need for linkage programs. This assumption of a mismatch further assumes an excess of services over families--that some services are avail- able but unused because families are unable to gain access to them. Where there are fewer services than families in need, the assumption becomes somewhat modified. In its modified form it assumes that more services could be made available if proper advocacy actions were taken. As implemented, most linkage programs carry out advocacy functions for the families rather than for the service programs: They assume that some familes are not getting the services they need, to which they have a right. They seldom assume the opposite--that some families are getting services for which they have minimal need or little right (because they are not fully eligible,

240 for example) . This latter function may be likened to that of a monitor, preventing the misuse of services. This "monitoring" function is more than hypothetical, since many programs do it to a minimal extent under a different name. The realities of limited service avail ability and unequal distribution may actually permit monitoring activities to produce an overall increase in effective family benefits if some are shifted from less needy to more needy families. The mechanism in linkage programs for carrying out this monitoring function is the needs assessment, whereby the services a family needs most are distinguished from those needed least. Few if any families can receive all available benefits for which they are eligible, so priorities must be decided; fortunately, few if any families need all benefits. In the best of all possible worlds, every family would always thrive on its own. When it is not so, social institutions can intervene--preferably by temporarily helping the family to become self-sufficient, but also by providing continuous, ongoing services if necessary. In our less than perfect world, most linkage programs assume that their primary role is to get services to families, and the more services the better. This has both desirable and undesirable consequences. It is desirable because the families served are often those furthest from the main stream of society, with the fewest advocates of any kind, and in great need of assistance. It is undesirable, because easy access to services may prevent the growth of family self-sufficiencY and independence. Some program critics maintain that many program staff deliberately promote family dependence to justify their own reasons for being, even if only subconsciously. The ultimate cost of this dependence may well prove high both to the family and to society. We present a point of view close to that of the programs: that there are many families who are not getting the services they need and that in general the more they get the better. But we should recognize that the underlying problems are much more complex and that future evaluations may have to reflect the more complex viewpoints and conflicting values. Integration of Services: Expected Program Effects Many beneficial outcomes may result from the hierarchy of linkage activities; some affect the families, some

241 affect the single-service agencies, and still others affect state and federal service programs. Effects on Families Comprehensive service programs filling a linkage or brokerage role can affect families by: eligible. . Increasing family awareness of available services. Increasing family enrollment in services when Increasing the use of available services. · Improving family circumstances because of services and reducing family stress. · Improving "consumer" awareness--fostering better knowledge of the family's rights and of the service agency's responsibilities for providing responsive, quality services. · Improving "consumer" advocacy involvement for upgrading the quality of services. The first four (awareness, enrollment, use, and improved circumstances) must logically be considered together. They are sequentially dependent such that the first (greater awareness of services) must logically occur before the second (greater enrollment) can occur; like- wise, the second must precede the third, and the third must precede the fourth. Moreover, none of the four effects can occur unless "linkable" services are avail- able in the community in the first place. Rural areas, especially, may have so few services that comprehensive service programs must use most of their resources to provide services directly themselves. In addition to the four sequential effects of aware- ness, enrollment, use, and improved family circumstances, there are also two "consumer" effects that may result from comprehensive service programs. One effect is an increase in the family's knowledge of its own rights and of the service agencies' responsibilities for providing responsive, quality services. The other is increased advocacy involvement for upgrading the quality of community services. We can consider these, too, to be sequential, since families are unlikely to become active advocates for service improvement unless they first know what the services should be, then note the shortcomings.

242 Effects on Service Agencies In addition to effects on families, linkage programs such as the Child and Family Resource Program will often produce changes in the agencies that provide services. Possible effects include the following: Simplified service utilization (through standardized application forms, adoption of consistent eligibility criteria, or additional neighborhood centers, for example). · Increased resources, such as additional funds or staff to serve more families or additional "found" resources to meet special family needs. · Better coordination of services among local providers, leading simultaneously to less overlap of services and fewer gaps. There are many ways that comprehensive service programs can encourage agencies to make single services easier to use. One way is simply by keeping them informed about the kinds of problems families encounter while attempting to use the services. For many conscien- tious single-service agencies, merely having a problem brought to their attention is enough to encourage action. Failing that, the comprehensive service program can actively petition the agency's head, the board of directors, or the funding source until improvement begins Staff from an established program can often get results when parents alone cannot, because the staff can draw on wider community or political support and because the staff usually have better skills than parents for coping with bureaucratic resistance. Comprehensive service programs can sometimes encourage single-service agencies to obtain more resources. For instance, by recruiting new families the comprehensive service program often overloads the single-service agency so seriously that the agency can then document the greater need to its board or funding agency. Or new wells of "found" resources might be developed--such as finding community volunteers who can provide needed professional services or establishing informal family support networks, such as other enrolled families, neighbors, community groups, and churches, to carry out functions formerly performed by paid staff. One of the most natural functions of comprehensive service programs is the coordination of like services .

243 across agencies, neighborhoods, and service domains. Single-service agencies usually have some leeway to shift the emphasis of their services, and the comprehensive service program staff are usually well located to spot such needs and bring them to an agency's attention. Effects on State or National Service Programs Sometimes a comprehensive service program can influence the larger state or federal systems that support the networks of local single-service agencies. Such effects might include the following: · Better coordination between services at the state and federal levels (by standardizing eligibility require- ments, application forms, referral networks, and funding procedures). · Increased resources to serve more families, perhaps in more states and counties, or to provide a higher level of services to families now served. Pooling staff or facilities to provide services jointly that existing state or federal programs could not provide alone. These and other related effects tend to happen only occasionally, but they do happen and may have substantial impact. For example, family enrollment procedures became simpler when Head Start managers ruled that AFDC or other public assistance eligibility was sufficient evidence for Head Start eligibility. Also, many states have adopted the Head Start Performance Standards for Title XX Day Care Programs, which opens the way for shared facilities and shared child-recruiting procedures. In yet another example, Head Start and the U.S. Department of Agriculture have agreed that every Head Start center is eligible for food program funds. This decision removes the need to screen every Head Start program for eligibility (thereby saving much time and effort) and frees Head Start funds for other family needs (such as child dental care, for which there are few funds). Comprehensive service programs can sometimes help increase federal and state resources. For example, when a state either lacks a Title XX plan or has a weak plan, local Head Start staff may use their knowledge of family needs and federal law, and state officials may encourage the appropriate state agency to prepare an expanded Title

244 XX plan (thereby qualifying the state for increased federal funds). Comprehensive service programs can also help match complementary services. An example of pooling staff resources occurred when. national Head Start managers and managers of the Early and Periodic Screening, Diagnosis, and Treatment (EPSDT) program agreed to operate a national experimental program that used local Head Start staff to find and recruit children eligible for these health services. This sharing enabled both programs to serve low-income families better: Head Start by freeing health funds for other family needs, and EPSDT by reaching families that it could not otherwise find without Head Start's established community recruiting network. Evaluation Problems and Approaches Evaluating Effects on Families By measuring the first four family effects in sequence we can determine the success of a comprehensive service program for improving families' circumstances, perhaps its most important goal. But if a program has little effect on these circumstances, an examination of this sequence of four effects will also indicate where the program is falling short. Such information is useful to program managers in upgrading deficient local projects. In measuring the four family effects, the sequence must be examined for each family individually. For example, it must be known that EPSDT child health services are not available in a family's community so that a comprehensive services program is not erroneously deemed ineffective because it did not inform the family about EPSDT or enroll it. Likewise, if EPSDT is available but a family does not meet eligibility requirements, the program cannot be held responsible. Or if the family does not need EPSDT services, as determined by their needs assessment, the program would not be expected to affect the family's use of EPSDT. These and other conditions must be examined individually, for each family, to preserve interpretability in an evaluation. Multiple data sources are needed to determine a comprehensive service program's success at each stage. First, program staff should be asked about the avail- ability of community services; for thoroughness, their responses should be independently cross-checked against

245 other community information sources to minimize inaccuracies and omissions. Many questions requiring answers are so detailed that only the single-service providers can accurately answer them: How many services do they provide? What kinds? What areas do they serve? What are their eligibility requirements? Are they capable of serving additional families? Then, following the sequence, we need to determine on a family-by-family basis if each family is aware of the single-service agency, if the family needs the service (according to the individual family's needs assessment), and if the family meets eligibility requirements. Families can be asked directly if they are aware of the services, but other people will usually have to be consulted to determine whether a family is eligible. One of the first activities the staff of the Child and Family Resource Program and Head Start undertake each year is determining which of their families are eligible for food stamps, EPSDT benefits, and welfare assistance, so the families can be enrolled immediately. A comprehensive service program's apparent effectiveness for enrolling families in services is limited, of course, by the number of eligible but unenrolled families it receives in the first place. Since the families themselves seldom know whether they are eligible, the evaluators have to approach other people for this information. Moreover, eligibility rules are usually complex and vary by community or state, even for many federally available services. The evalu- ators themselves may have to perform calculations of income eligibility for each family and match them individually to the requirements for each service, to determine the potential new enrollments possible. Increases in program enrollment can be measured without matching the effects for individual families, but the maximum percentage enrollable cannot; it is a vital policy statistic because it reveals the overall need for a service and the current degree of success reached in meeting the need. The services that families actually use can often be determined by asking families to name them. However it is usually necessary to determine the precise name of a service program, its funding source, or its legislative authority--details families seldom know. This information must typically be laboriously gathered by directly contacting the sources of service cited by parents, although at times linkage program staff know. Rough estimates of the extent to which families use a service

246 can often be obtained directly from them, but precise levels of use are difficult to determine. Improved family circumstances can sometimes be measured by equating them to service use (as, for example, when food stamps can be given a dollar value and counted as an increase in family income), but usually indirect means are needed to determine the improvement. When a family change is drastic, such as finding a substantial job for a previously unemployed father, little risk is faced in inferring improved family circumstances. In searching for subtler effects, the problems grow more difficult. The effects of preschool on children are usually less clear, although many people seem to agree that favorable results from selected child tests often imply an improve- ment in family circumstances. Families can ultimately be asked directly if they feel their circumstances have improved or if they experience less family stress. We can always assume (after the fashion of pollsters) that if families say things are better (or worse) then they are better (or worse), even though appearances may suggest otherwise. In areas such as preventive health services, improved status can be determined only by using compli- cated medical procedures that are simply not feasible in most evaluations. Thus, apart from a few notable exceP- tions, the ultimate goal of improved family circumstances is the most difficult aspect of program success to assess. The effect of increased family consumer awareness and advocacy usually results from rather obvious involvement of the families and can frequently be determined by direct questions to the families or program staff. Typical questions ask about parent participation on policy councils, school boards, task forces, political organizations, or ad hoc groups. Evaluating Effects of Local, State, and National Programs Few individual families are aware of changes in the service providers over time (because, for example, a family usually applies only once for a service and has no way of knowing previous or subsequent application procedures). Long-term staff employed by the linkage program are often aware of changes and trends in the providers they work with. Since they typically go through service application procedures with family after family, linkage program staff can frequently identify changes

247 that are subtle--all the more so if they actively advocated the changes. Most often, however, changes in the single-service agencies can only be identified with any certainty by direct longitudinal review of the agencies themselves. This may involve a study at the community level or it may involve a study of an entire national service delivery network. Because the designs required for this type of evaluation are different from ones for assessing family impacts, it often becomes difficult to carry them out concurrently with evaluations of family effects. Approximate descriptions of changes in service providers can be obtained by gathering anecdotes from appropriate community residents about the kinds of changes they remember seeing. This kind of study need not be longitudinal, thereby considerably reducing the cost. Lazar (1970) conducted such a study on the community impact of Head Start. Changes in service programs at the state and national levels can be identified through changes in regulations, funding levels, legislation, reorganization, staff time allocation, and so on; these can be ~measured" by examin ing official program documents; observing operating practices; and interviewing federal, regional, or state policy makers and managers. Once measurements are taken at the state or federal level there are so many prevailing influences that it becomes almost impossible to attribute any changes found back to a particular comprehensive service program. This does not diminish the policy benefits of straightforwardly describing the changes, however, even without scientific support for the causes of the changes. Evaluators should also be alert for unintended conse- quences of comprehensive service program activities. For example, outreach and referral activities may be so effec- tive that the single-service providers are inundated beyond their capacities, eroding service quality and exceeding available funds or staff capabilities. This may create new problems at the state and federal levels as well as at the local level. - CONCLUS ION Our discussion has focused on two of the comprehensive family service programs developed as demonstrations under the auspices of Head Start and the Administration for

248 Children, Youth, and Families. Three essential features of these programs were described: the mix of support and intervention, the emphasis on individualized program treatment, and the role of the program in coordinating community services. Each of the features was shown to have particular implications for the kinds of evaluation questions posed and the methods used to answer them. The mix of support and intervention was seen as a natural result of the programs' broad goals and of their acceptance of the strengths and different values of families and communities. Individualized treatment also was an expression of the broad goals. Treatment was varied by family along many dimensions, such as the types of activities and services emphasized, the role of parents in the program, and the length and intensity of participation expected. Third, the programs' role as "brokers" of services and advocates for families implies both support or change for families as consumers of . . service ana change in the services or policies concerning families at the local, state, and even federal levels. Thus, the programs are designed to support and bolster the development of the children, the functioning within the family or household unit, and the ways in which families and social institutions interact. This broad concept of program action necessitates a broadening of methods and perspectives in program evaluation. One area of evaluation to be expanded is the use of process studies to describe and characterize what actually happens in the program between families and staff, among staff, and between staff and community institutions. Different types of process studies can and should be used to identify and clarify goals and constituencies for family programs, to describe what the programs do, and to help relate what goes on in the programs (the process and/or treatment) to what results from them (the out- comes). Process studies also help raise questions and generate hypotheses about unanticipated outcomes and also look at questions about the relationships among different family and child programs at a federal administrative and policy level. We make several recommendations concerning the study . . of outcomes in order to address the problems posed by Individualized treatments for families in several domains of family life over time. One is to retain the quasi- experimental designs currently in use but to supplement them with additional, smaller studies or analyses to enhance their interpretability. Exploration of methods

249 used in clinical work or educational psychology but less commonly used in program evaluation is urged in order to give a whole picture of outcomes in several domains for families or groups of families over time. Increased use of atheoretical indicators, ethnography, and descriptive analyses for synthesizing different types of data also is discussed. Overall, while the programs and the methodological problems associated with evaluating them are complex, the endeavor is not overwhelming. It requires careful speci- fication and rating of the questions that are to be answered at any one time for any particular program. also requires the recognition Anal many quest ~ ~= concerning the evaluation of family support programs are ultimately questions of values and social policy. Perhaps most important, the problems, as we have described them, derive from the richness and complexity of the programs, their goals, and practices and of the families themselves. It would be a mistake for evaluators merely to complain about such complexity or to adopt new methods or perspectives that would eliminate important variety for the sake of precision and manageability. At this point in the development of family programs and of research about families, it is important for evaluators to try to capture this richness and variety in different ways, using multiple methods and perspective. Under- standing outcomes at the individual, family, and social levels demands that we consider outcome to be (like families and programs) multidimensional, multiply determined, occurring within a particular context, and changing over time. It REFERENCES B~yley, N. (1969) Bayley Scales of Infant Development. New York: Psychological Corporation. Bell, R. Q., and Harper, L. V. (1977) Child Effects on Adults. University of Nebraska Press. Belle, D., et al, (1980) Lives in Stress: Lincoln, Neb.: A Context for Depression. Cambridge, Mass.: Harvard University Press. Bronfenbrenner, U. (1977) Toward an experimental ecology of human development. American Flat 32:513-531.

250 Burr, W. R., Hill, R., Nye, F. I., and Reiss, I. L., eds. 1979) Contemporary Theories About the Family. Vol. I and II. Campbell, D. T., and Boruch, Re (1975) New York: The Free Press. Making the case for randomized assignment to treatment by considering the alternatives: six ways in which quasi-experimental evaluations in compensatory education tend to underestimate effects. In C. A. Bennett and A. A. Lumsdaine, eds., Evaluation and Experiment. New York: Cook, T. D., and Campbell, D. T. (1979) Quasi-Experimentation. Rand-McNally. Duvall, E. (1977) Lippincott Co. General Accounting Office (1979) Early Childhood and Family Development Programs Improve Quality of Life for Low Income Families. Washington, D.C.: General Accounting Office. February. Goodson, B. D., and Hess, R. (1978) The effects of parent training programs on child performance and behavior. In B. Brown, ea., Found: Long-Term Gains From Early Intervention. Boulder, Colo.: Westview Press. Goodwin, W. L., and Driscoll, L. A. (1980) Handbook for Measurement and Evaluation in Early Childhood Education. Family Development. Academic Press. Chicago, Ill. Philadelphia, Pa.: J. B. San Francisco, Calif.: Jossey-Bass, Inc., Publishers. Gurman, A. S., and Kniskern, D. P. (1978) Research on marital and family therapy: progress, perspective, and prospect. In S. L. Garfield and A. E. Bergin, eds., Handbook of Psychotherapy and Behavior Change. 2nd ed. New York: John Wiley & Sons, Inc. 15th Anniversary Head Start Committee (1980) Head Start in the 1980's: Review and Recommendations. Report by the 15th Anniversary Head Start Committee. Edward Zigler, Chairman. September. Herson, D., and Barlow, R. (1976) Single Case Experimental Design. New York Pergamon Press.

251 Hertz, T. W. (1976) The Ecology of Child Development: Research Issues. The Child/The Family/Environment/ Services. A Report for Planners of Research and Development. Prepared for the Interagency Panel on Early Childhood Research and Develop- ment. Washington, D.C.: The George Washington University. December. Hewett, K. H. (1978) Partners With Parents: The Home Start Experience With Children and Their Families. Office of Human Development Services. DHEW Publication No. 78-31106. Washington, D.C.: U.S. Department of Health, Education, and Welfare. Hewett, K., Connell, D., Affholter, D., and Weiss, S. (1979) Evaluation of the Child and Family Resource Program: Volume I: Design Report Draft. AAI No. 78-111. Cambridge, Mass.: Abt Associates, Inc. Hill, R. (1971) Strengths of Black Families. Washington, D.C.: National Urban League. Hill, R., and Mattessich, P. (1977) Reconstruction of Family Development Theories: A Progress Report. Paper presented at the meeting of the National Council on Family Relations. San Diego, October. Jacob, T. (1975) Family interaction in disturbed and normal families: a methodological and substantive review. Psychological Bulletin 82:33-65. Johnson, L., Nauta, M. J., and Hewett, K. D. (1980) Evaluation of the Child and Family Resource Program: Phase III Program Study Report. AAI No. 80-91. Cambridge, Mass.: Abt Associates, Inc. Johnson, L. (1981) Methodology for the Ethnographic Study: The Child and Family Resource Program Evaluation. Cambridge, Mass.: Abt Associates, Inc. Kagan, J., Kearsley, R. B., and Zelazo, P. R. (with the assistance of C. Minton) (1978) Infancy: Its Place in Human Development. Cambridge, Mass.: Harvard University Press. Kamerman, S. B., and Kahn, A. J. (1978) Family Policy: Government and Families in

252 Fourteen Countries. University Press. New York: Columbia Kazdin, A. (1977) Methodological issues in single case analysis. ~n~rnA1 Of clinical and Consulting Psychology 34. Kessen, W. (1979) The American child and other cultural interven- tions. American Psychologist 34:915-920. Kiresuk, T. J., and Sherman, K. E. (1968) Goal attainment scaling: a general method for evaluating community mental health programs. Community Mental Health Journal 4(6):443-453. Kiresuk, T. J., Calsyn, R. J., and Davidson, W. S. (1978) A critique of goal attainment scaling. In T. Cook, ea., Evaluation Studies Review Annual 3:700-715. Lazar, I., Hubbell, V. R., Murray, H., Roche, M., and Royce, J. (1977) Persistence of Preschool Effects: Final Report. Grant No. 18-76-07843, Administration for Children, Youth, and Families. Washington, D.C.: U.S. Department of Health, Education, and Welfare. Office of Human Development Services. Love, J. M., Nauta, M. J., Coelen, C. G., Hewett, K. D., and Ruopp, R. R. (1976) Home Start Evaluation Study. High/Scope Educational Research Foundation and Abt Associates, Inc. Cambridge, Mass.: Abt Associates, Inc. Nauta, M., and Johnson, L. (1981) Evaluation of the Child and Family Resource Pro ram Phase III Research Report. g Cambridge, Mass.: ~ Nauta, M. (1981) Abt Associates, Inc. Evaluation of the Child and Family Resource Program: Phase III Executive Summary. Cambridge, Mass.: Abt Associates, Inc. Newbrough, J. R., Dokecki, P. R., Dunlop, K. H., Hogge, J. H., and Simpkins, C. G. (1978) Families and Family-Institution Transactions in Child Development. Final report. Contract No. 105-77-1045, Administration for Children, Youth, and Families, Office of Human Development Services. Washington, D.C.: U.S. Department of Health, Education, and Welfare.

253 Nobles, W. W. (1976) A Formulative and Empirical Study of Black Families: Final Report. San Francisco, Calif.: Westside Community Mental Health Center, Inc. Rossi, P. H., Freeman, H. E., and Wright, S. R. (1979) Evaluation: A Systematic Approach. Beverly Hills, Calif.: Sage Publications. Stake, R. tl978) The case study method in social inquiry. Educational Researcher 7(2):~-8. Travers, J., Johnson, L., Lynnell, J., and Irwin, N. (1981) An Ethnographic Perspective on the Child and Family Resource Program. Cambridge, Mass.: Abt Associates, Inc. Weiss, C. Program Effectiveness. N.J.: Prentice-Hall. (1972) Evaluation Research: Methods for Assessing Englewood Cliffs, Weiss, R. S., and Rein, M. (1969) The evaluation of broad-aim programs: a cautionary case and a moral. Annals of the American Academy of Political and Social Science 385.133-142 Zigler, E., and Trickett, P. (1978) IQ, social competence and evaluation of early childhood intervention programs. American Psychologist 1:789-798.

Next: The Evaluation Report: A Weak Link to Policy »
Learning From Experience: Evaluating Early Childhood Demonstration Programs Get This Book
×
MyNAP members save 10% online.
Login or Register to save!
  1. ×

    Welcome to OpenBook!

    You're looking at OpenBook, NAP.edu's online reading room since 1999. Based on feedback from you, our users, we've made some improvements that make it easier than ever to read thousands of publications on our website.

    Do you want to take a quick tour of the OpenBook's features?

    No Thanks Take a Tour »
  2. ×

    Show this book's table of contents, where you can jump to any chapter by name.

    « Back Next »
  3. ×

    ...or use these buttons to go back to the previous chapter or skip to the next one.

    « Back Next »
  4. ×

    Jump up to the previous page or down to the next one. Also, you can type in a page number and press Enter to go directly to that page in the book.

    « Back Next »
  5. ×

    To search the entire text of this book, type in your search term here and press Enter.

    « Back Next »
  6. ×

    Share a link to this book page on your preferred social network or via email.

    « Back Next »
  7. ×

    View our suggested citation for this chapter.

    « Back Next »
  8. ×

    Ready to take your reading offline? Click here to buy this book in print or download it as a free PDF, if available.

    « Back Next »
Stay Connected!