Beyond the Blueprint: Directions for Research on Head Start's Families (1996)

This appendix consists of excerpts from two reports by advisory committees of the Administration for Children, Youth, and Families. The first (pp. 58-66) is "Overall Strategy and General Principles" from Head Start Research and Evaluation: A Blueprint for the Future. Recommendations of the Advisory Panel for the Head Start Evaluation Design Project. (DHHS publication no. ACY 91-31195, September 1990, Head Start Bureau, Administration for Children, Youth, and Families, Office of Human Development Services, U.S. Department of Health and Human Services).

The second (pp. 60-71) is "Executive Summary" from Creating a 21st Century Head Start: Final Report of the Advisory Committee on Head Start Quality and Expansion. (U.S. Government Printing Office 1994—515032/86864, December 1993, U.S. Department of Health and Human Services, Washington, D.C.).

Future Head Start research and evaluation efforts should be guided by a clearly defined overall strategy and general principles. This section presents and discusses the eight principles formulated by the Panel.

1. Head Start research and evaluation planning should be organized around two principal questions:

• Which Head Start practices maximize benefits for children and families with different characteristics under what types of circumstances?

• How are gains sustained for children and families after the Head Start experience?

Research initiatives designed to answer these questions offer high promise of producing information that will lead to continuing improvements in the quality of Head Start and other early childhood programs. Moreover, careful consideration of these pivotal questions will help ACYF identify Head Start factors and post-Head Start experiences that extend or attenuate the positive effects of the Head Start program.

The nature of these two principal questions defines a new generation of research and evaluation which differs significantly from past studies. First, they require much finer grained analyses, particularly with respect to independent variables. Second, they require information specific to various subgroups of the Head Start population. Third, they recognize the contribution of the knowledge of context to the interpretation of data. Fourth, they acknowledge family functioning, both as a goal in itself and its importance in the mediation of the child's development. Each of these research and evaluation implications will be discussed further in this report.

2. An overall research strategy rather than a single large scale study is the appropriate framework for addressing critical Head Start research and evaluation questions.

The Panel recommends strongly against a single large scale study of Head Start as the principal mechanism for seeking answers to the pivotal research questions highlighted above. The methodological requirements for the new generation of research and evaluation issues do not lend themselves to large scale evaluations that treat Head Start as a single program. Head Start is not, in any simple sense, a uniform "treatment." The common denominator of Head Start programs nationwide is conformity to a set of regulatory performance standards that reflect comprehensive service requirements in education, parent involvement, social services, and health services. But programs are allowed and encouraged broad flexibility in how they deliver the required component services. Moreover, Head Start programs serve children in different regions and subeconomies of American society. They serve a number of minority groups and address issues of bilingualism and multiculturalism. They embody a variety of programmatic formulas and inventions created by local Head Start staff to respond to the unique needs of families in their communities.

An overall strategy is needed to extend existing theory and state-of-the-art research methods and to provide a more comprehensive and in-depth knowledge base for improving the quality of services provided to children and families. The Panel proposes three key approaches to guide this new strategy. These are implementation of an integrated set of studies, use of diverse methodologies and identification of marker variables.

An advantage of multiple studies, as opposed to a single large project, is the capacity for a cluster of studies to complement one another at a single point in time and to build upon one another in incremental stages over time. A further advantage is the ability to cross-validate findings using different methodologies and to test the hypotheses with different subgroups of programs and participants.

The research and evaluation studies should be designed to yield results that are interpretable for specific subgroups of children and/or families and for specific localities. These findings, taken as a whole, could address the major questions of interest. In addition, the studies should be designed and conducted by a consortium of investigators who would contribute a number of different perspectives and areas of expertise to the effort. Such an arrangement permits much more control over the quality of the data than is possible in large scale studies that are under the direction of an individual investigator or contracting firm.

The necessity for an integrated and coordinated set of studies cannot be overemphasized. Although a large number of studies of Head Start were conducted over the past 25 years (particularly in the first dozen years of the program's existence), what exists is a fractionated accumulation of studies that do not build upon one another. These efforts have yielded relatively little in the way of an organized body of knowledge. Head Start research and evaluation, in general, has not been based upon well formulated program and policy questions.

The need for different research designs depends on the state of knowledge and the particular issues being explored. The proposed strategy draws upon diverse methodologies including case, ethnographic, correlational, quasi-experimental, and experimental studies. The common denominator of these various designs should be an overall conceptual framework guided by the principal questions.

Correlational and quasi-experimental studies could be used to test hypothetical causal models, models of ''What works best for whom?'' and of factors conducive to the maintenance of gains. Such studies could also identify variables that suggest causal influences or "active ingredients." Experiments could be designed to test these hypotheses through treatment manipulations or modifications, or additions to existing Head Start programs in randomized trials. This is one example of how a particular issue could be pursued through several stages of inquiry. There are many variations on this theme depending on the particular questions to be explored and the available research and evaluation resources.

It is important to note that, in the view of the Panel, randomized studies designed to compare the effects of Head Start against the effects of nonparticipation ("treatment vs. no treatment") are generally no longer viable options. First, as ACYF progresses toward the

Administration's goal of universal services for all eligible children, the potential for withholding services to form a control group, already difficult for ethical and practical reasons, will cease to be an option. Second, in view of the expansion of state and public school preschool programs and developmental child care, it is unrealistic to expect to find in most communities a representative group of "untreated" eligible children, even if Head Start services are not provided.

There are a variety of ways to respond to these and other constraints on experimental options posed by changing societal realities. One approach would be to use random assignment at the level of individual Head Start programs, centers, classrooms, or groups of home visitors to test various experimental "add ons." The control comparison would be an individual Head Start program (or unit or units) from the same subpopulation that did not receive the "add ons."

Case and ethnographic studies have played an increasingly important role in social science methodology. First, they are particularly helpful for hypothesis formulation. Second, knowledge of how programs actually operate, why something works or does not work, why some strategies work for children and families with a particular set of characteristics and not for others and the conditions under which they work are best informed by qualitative methodologies. Quantitative research does not replace qualitative knowledge, but instead builds upon it and requires it for valid interpretation. Both quantitative and qualitative research must conform to appropriate scientific criteria for assembling and analyzing evidence in order to insure the validity, reliability, and replicability of the findings.

In the past, one of the detriments to the use of qualitative methods has been the high cost of the methodology. However, if high quality research is to be conducted to respond to the critical issues in Head Start, then future Head Start studies must use both qualitative and quantitative methods in a complementary fashion.

Another necessary condition for an integrated and coordinated set of studies is the identification of a set of marker variables for child functioning, family functioning, program characteristics, and community characteristics. These core variables will make it possible to tie separate studies together. Identification of marker variables should take place early in the implementation of the next series of Head Start studies, either as an integral part of the initial studies or as a separate activity. Additional variables would be added to particular studies, depending on the specific questions under investigation. Marker variables should be sensitive to racial, ethnic, linguistic, and cultural diversity among Head Start subpopulations.

In summary, the Panel recommends a strategy that involves an integrated set of small scale studies involving different methodologies, different subgroups of the Head Start population and multiple investigators. However, use of small scale studies does not mean that findings and policy decisions will be based on small samples. Nor does this mean that these studies would be simple in design. Rather, the recommended strategy will facilitate the convergence of results from multiple sources that will collectively incorporate robust findings from sophisticated designs. Thus the study outcomes will say something meaningful about the impact on subgroups of program participants.

Child outcomes should remain a central thrust of Head Start studies. However, the focus of future Head Start research and evaluation efforts should be widened to include a broader array of child outcomes and to encompass outcomes for families, communities, and institutions.

Head Start is a two generation program that, in addition to the social competence goals for children, addresses goals for parents and other family members as priority outcomes in their own right. For example, Head Start's initiatives in the areas of family literacy, job training, and family support incorporate objectives of parental educational attainment, employability, and family self-sufficiency. ACYF's research agenda should give high priority to issues of family functioning, parent involvement in their children's development, family support, and family variables as mediating influences on child functioning.

Examples of salient family outcomes of a Head Start experience include:

• improved parenting skills;

• increased parent-child interaction to promote child development and learning and to strengthen the family system;

• improved expectations for children's future success in school;

• increased parent involvement in schools and other community institutions;

• reduced risk factors associated with family stress, including family violence, child abuse and neglect and substance abuse;

• reduced dependence on welfare and heightened parental skills related to economic self-sufficiency (for example, improved literacy, adult education, and employability); and

• improved access to and utilization of community family support services.

In pursuit of its multiple program goals, Head Start has forged new relationships with agencies at the Federal, state and local levels and served as a catalyst in developing partnerships among human services agencies. The nature of such collaboration is an area of inquiry in terms of the impact of Head Start's efforts on policy and on the delivery of social programs and must be taken into account in research design and in the selection of independent variables.

5. Multiple indicators and methods should be employed in the measurement of important outcomes.

Evaluations should utilize, to the fullest extent possible, outcome indicators that are reflective of the generally understood goals of Head Start and that convey the program's important effects. These outcome measures should be readily understandable to parents, program staff, policymakers, and the general public. In addition to carefully selected standardized tests, outcome measures should include such straightforward indicators as

whether or not a child is at grade level, is placed in a special class, or does or does not have an undetected (or identified but untreated) medical condition.

Researchers should be mindful, however, that there are often complex dimensions of "simple" measures of the type recommended. For example, whether a child is retained in grade in the public schools is a relatively simple indicator. Why the child is retained in grade, the criteria by which this decision is made, and the implications for the child's future success in school may vary greatly within and across different school systems. Similarly, whether or not a child is in a special education class or experiencing "pull out" instruction is relatively easy to establish. What this means in terms of the child's placement in an appropriate educational environment that will best meet the child's learning and developmental needs is more complex.

In general, previous Head Start studies used norm-referenced tests either of I.Q. or achievement as the single method for measuring program outcomes. There has been well justified criticism of the use of I.Q. tests, particularly for minority children. However, there are other norm-referenced tests that are useful in determining the child's acquisition of certain skills. In fact, familiarity with "taking tests" and the particular skills involved in the test process can serve the child well in terms of future academic demands.

Norm-referenced tests should be only one of several methods for measuring child status. Indicators of a child's status, such as reading ability, number competence, or ability to function as a confident social participant in classroom and school processes are best arrived at through multiple methods, including observations, and ratings by teachers, parents, and peers.

Care should be taken in the selection of child outcome instruments to keep in mind that the ultimate purpose is to assess program effectiveness. With respect to school readiness and achievement, for example, the focus should be on the extent to which the influences of Head Start, school and the family combine to provide the child with opportunities to learn and function in an educational setting at an optimal level consistent with his or her ability.

The overall Head Start evaluation agenda should strive to balance the following elements: validity of outcome measures, ease of data collection for both researchers and program staff, multimethod assessment strategies, and an understanding of the underlying dynamics that the indicators purport to represent.

6. Data collection procedures and techniques must be valid and appropriate for the particular research question and the specific population.

Any norm-referenced instruments used with particular subpopulations must be valid for the subpopulation under consideration. Interviews and questionnaires administered to parents must take into account literacy level and linguistic usage of the respondents. In order to insure reliability and validity of the responses, these instruments may require special adaptations for different subpopulations. Of particular importance is attention to predictive validity. Reliable instruments may differ substantially in their capacity to predict the same outcome for different subpopulations. Therefore, most instruments will, of necessity, have to be pilot-tested prior to their use in research and evaluation studies.

All instruments should be administered individually to Head Start children. To optimize measurement comparability across years, individually administered tests also should be employed in later grade school follow-ups.

7. Program variation must be explored while searching for explanations of differential outcomes.

There is no single active ingredient in Head Start that is the key to program outcomes. Local Head Start programs are complex organizational entities that interact with equally complex family and community ecologies. This calls for a combination of holistic research strategies (including organizational, case study, and ethnographic methodologies) together with multivariate statistical explorations involving natural variation, experimental, and quasi-experimental designs. Researchers should be alert to the possibilities that interaction of program attributes may be more important in accounting for outcomes than individual program attributes considered in isolation.

The following program variables are among those that are particularly important:

• staff characteristics and behaviors (including education, years of experience, credentials and certification, length and type of training, and knowledge of early childhood education and child development);

• classroom composition (group size and child-staff ratio) and, for home-based programs, family-home visitor case loads;

• nature and intensity of interactions among children and between children and staff;

• curricular strategy, including the extent to which the curriculum is well-planned and appropriately delivered;

• nature and intensity of parent involvement;

• staff-parent interaction;

• nature and frequency of home visiting and other family contacts;

• delivery of comprehensive services, including nature of linkages with health, nutrition, social and educational agencies;

• length and intensity of child's participation per day, week, and year;

• age at onset of intervention and number of years of child's and family's participation in the program;

• degree of program responsiveness to identified needs of participating children and families;

• extent and nature of flexibility exercised by the local program in tailoring its delivery system to specific circumstances and resources in the community;

• organizational climate;

• administrative and personnel issues (including staff compensation and turnover); and

• program auspices.

Head Start research should capitalize on the insights of prior and ongoing research in early childhood education, child development, and other related fields in exploring these variables. For example, organizational research is currently bringing new understandings to the functioning of public schools. Similar perspectives on examining Head Start programs as organizational entities can provide powerful research tools. Areas of inquiry include leadership, how decisions are made, the role of parent participation on policy councils, grantee-delegate agency linkages, the relationships of local programs and ACYF Regional Offices, and approaches to program implementation. Related administrative and personnel issues are worthy of priority attention, as suggested by recent findings in child care research showing the strong relationship between staff salaries and turnover and child outcomes.

In addition to variation across programs, variation within programs must be explored. Children and families vary considerably in the attributes with which they enter Head Start programs. These attributes will differ in their interactions with program characteristics. Therefore, even families with children in the same classrooms will vary in their Head Start experiences. Traditional methodology assumes a uniform treatment across families, at least among those with children in the same classroom. In studies which explore the reasons for differential effects, it is crucial to incorporate consideration of such variation in the initial design of the study.

8. Head Start research and evaluation studies can be greatly enhanced by building on the existing strengths of programs and program staffs.

The Panel felt strongly that researchers need to approach all studies with a focus on identifying the strengths of current programs and program staffs, rather than emphasizing deficits (for similar reasons that the approach to children and families should emphasize strengths rather than deficits). Research projects are most likely to succeed if researchers include program staff in all stages of the project, including the initial development of hypotheses. Collaboration between the research and program communities calls for joint planning with clearly defined roles and benefits for all parties.

Studies of existing programs are most likely to bear fruit if they involve a search for excellence both in overall programs and in particular strengths within programs. In addition, experimental designs to test program features will be most successful if those program "add ons" are considered by Head Start staff as new and exciting additions to their current programs. It also is important to provide increased funding to offset the costs of implementing the added features.

Although cooperative efforts between program and evaluation staff are essential, data collection procedures must include safeguards to insure that findings are not subject to the criticism of respondent bias. Such safeguards might include the use of ratings by independent observers not connected with a program. In addition, program staff should receive guarantees that negative findings will not be used by administrators to penalize the program or eliminate its staff. These guarantees are likely to reduce respondent bias and enhance the quality of the data.

The Panel's chief concern is that high quality research is conducted and used to make future decisions about programs that can profoundly affect the lives of poor children and their families. In addition to the recommended principles that undergird the research strategy discussed above, the following corollary principles involve budgetary considerations.

• High quality research is expensive. A smaller number of adequately funded studies is preferred to a larger number of inadequately funded efforts.

• The first year of all large scale and complex studies should be devoted to design and pilot testing. For all studies, there should be sufficient lead time prior to the formal data collection stage to permit careful pilot testing and the refining and fine-tuning of the design, measurement instruments and procedures.

The Advisory Committee on Head Start Quality and Expansion was created in June 1993 to review the Head Start program and make recommendations for improvement and expansion. After six months of deliberations, the 47 members of the Advisory Committee seek to open a new chapter in the history of the program.

Launched in 1965 as a comprehensive child development program, Head Start has provided a beacon of hope and support to more than 13 million low-income children and their families across the United States through the provision of education, health, social services, parent involvement, and disability services.

Yet the world of Head Start today is dramatically different than in 1965. Today the needs of families and children who live in poverty are more complicated and urgent than ever before—from children who have lived with violence and substance abuse to families with interrelated problems of homelessness, lack of education, and unemployment. Since Head Start began, the face of poverty has changed to include more single parents, and increasing numbers of working parents. In addition, the recognition and importance of promoting family literacy has increased.

Over the past 28 years, the landscape of community services has changed dramatically. There are new roles and enhanced capacities for serving young children and their families. Today we also have new knowledge about the attributes of services and supports that are effective in changing long-term outcomes for young children, new knowledge about the importance of the first three years of life, and new knowledge and appreciation for the continuum of developmental and comprehensive services needed before school and into the early years to help children succeed in school.

In order to develop a set of recommendations for the future of the Head Start program, the Advisory Committee reviewed existing data and reports on Head Start and consulted with a wide variety of individuals and groups across the country. The Committee found that, after a period of rapid expansion, Head Start can be proud of many successes yet still needs to address existing quality problems and to be refocused to meet the challenges of a new age. The Advisory Committee found that:

• Head Start has been successful in improving the lives of many low-income children and their families and in serving as a national laboratory for early childhood and family support;

• Most Head Start programs provide quality services, however, the quality of programs is uneven across the country;

• Head Start needs to be better equipped to serve the diverse needs of families;

• There continues to be a large unmet need for Head Start services; and

• In many communities and states, Head Start, public schools, and other early childhood programs and providers responsible for addressing the needs of young children and families operate in isolation from one another, without adequate resources, planning, and coordination.

As the Advisory Committee looks forward to the next century, we envision an expanded and renewed Head Start which serves as a central community institution for low-income children and their families. The Head Start of the 21st century:

• Ensures quality and strives to attain excellence in every local program;

• Responds flexibly to the needs of today's children and families, including those currently unserved; and

• Forges new partnerships at the community, state, and federal levels, renewing and recrafting these partnerships to fit the changes in families, communities, and state and national policy.

In order to respond to these issues, and to create a 21st Century Head Start, the Advisory Committee sets forth a set of recommendations to the federal government, Head Start providers, and the nation at large. These recommendations implement three broad principles.

The Advisory Committee believes that the quality of services must be a first priority. We should strive for excellence in all Head Start programs by focusing on staffing and career development, improving the management of local programs, reengineering federal oversight to assure accountability, providing better facilities, and strengthening the role of research.

The Advisory Committee recommends that the Department of Health and Human Services (HHS) develop new initiatives to utilize qualified ''mentor teachers'' to provide supervision and support to classroom staff, establish competency-based training for staff who work directly with families, ensure sufficient staffing levels to serve children and families effectively, and continue to increase compensation.

In the area of management, the Advisory Committee recommends an expanded emphasis on management training; strengthening financial management policies and practices; supporting strategic planning through a multi-year "phased-in" expansion strategy; updating the Performance Standards in health, parent involvement, social services, and education; and developing performance measures.

In addition, the Advisory Committee recommends an assessment of the training and technical assistance system, a review and strengthening of Head Start monitoring, training of regional and central office staff, and the assurance of prompt action to deal with low performing grantees.

2. We must expand the number of children served and the scope of services provided in a way that is more responsive to the needs of children and families.

The Advisory Committee reaffirms the concept that all eligible children in need of Head Start should be served. At the same time the Committee remains committed to investments in quality as a top priority. Head Start should focus on the needs of children in the context of their families and communities by enhancing family services and increasing parent involvement, assessing needs and planning strategically, reaching out to children and families currently unserved, promoting full day and full year programs where needed, and improving services to families with younger children. The overwhelming majority of Advisory Committee members recommend a new initiative to expand Head Start services to families with younger children. Some Advisory Committee members believe that further study is needed to explore ways of serving additional families with children under age three prior to launching an initiative.

The Advisory Committee recommends that HHS review and expand current resources used for family services, parent education, and family literacy, and that increased efforts be made to involve parents in all aspects of the program. The Committee recommends that as Head Start programs move forward toward the goal of serving all eligible children, they should be encouraged to assess their total program in order to balance the needs for quality, scope of services, and number of children served. Particular

attention and support are recommended to address the special needs of Indian and migrant programs.

3. We must encourage Head Start to forge partnerships with key community and state institutions and programs in early childhood, family support, health, education, and mental health, and we must ensure that these partnerships are constantly renewed and recrafted to fit changes in families, communities, and state and national policies.

Because no program, no matter how excellent, can go it alone, we must ensure that Head Start join forces with other providers in the community and state. As a partner, Head Start can not only maximize its own resources, but can use its leadership to influence other service providers to adopt the core concepts that have made Head Start such a success.

Head Start and public schools should renew commitments to ensure continuity of services by providing developmentally appropriate programs, parent involvement, and supportive services from Head Start through the primary grades.

Head Start should form new partnerships at the state and local level, and with the private sector, to provide more coordinated services to families. Head Start should also play a central role on behalf of low-income children and families in emerging initiatives, particularly in national service, health reform, education reform, family preservation and support and welfare reform.

In summary, the Advisory Committee supports the goal of ensuring that all eligible children and their families receive high quality Head Start services, that programs are tailored to meet the needs of todays families and communities, and that sufficient resources are made available to meet these goals.

The Advisory Committee believes that the recommendations and principles set forth in this report must inform Head Start program decisions at all levels. The recommendations must guide priorities and the use of existing as well as new resources to ensure quality services that children need to enter school ready to learn and that families need to achieve self-sufficiency.

The Advisory Committee recommends that HHS act promptly to develop an implementation plan based on the ideas set forth in this report. First priority should be given to ensuring quality and striving for excellence.

The process of setting priorities should also draw on the best available information and input from Head Start and the larger community.

In concluding, the Advisory Committee on Head Start Quality and Expansion urges the Department to see this report as a step in an overall effort to improve early childhood and family support services for all children in the United States, and particularly for those most vulnerable. HHS should continue to show leadership in looking across programs to ensure that policies consistently promote quality services for young children and their families.