Longitudinal Surveys of Children (1998)

Some collect data in personal interviews; others rely on observation, administrative records, or time diaries to supplement the data. In addition to surveys conducted in the United States, longitudinal surveys in England, Canada, and China were represented at the workshop.

To facilitate the workshop discussion, a summary overview for each survey was prepared, providing a brief introduction, a statement about the central substantive issues that guided development of the survey, the study design, the sampling strategies, and the constructs being assessed. Data collection instruments and methods also were listed, as was information on the current status of the survey. This information appears in Appendixes A and B.

The workshop agenda was structured to allow for discussion of cross-cutting methodological and conceptual issues, identification of common policy questions, and consideration of mechanisms for ongoing information sharing. In advance of the workshop, participants prepared brief statements noting special challenges and issues, and time was allotted at the workshop to address these concerns in open discussion. Ample time was allowed for brainstorming and additional open exchange among the participants. The agenda appears in Appendix C.

Although the agenda was designed to be flexible, allowing ample time for discussions and time for participants to raise issues of particular significance to their survey, the workshop did not cover many methodological issues of relevance and importance to longitudinal studies in general and longitudinal studies on children in specific.

This report is intended to be used by representatives from federal and state agencies engaged in research, data collection, program evaluation, and policy analysis pertaining to children and youth; administration officials engaged in policy formulation, implementation, and evaluation; congressional staff; public and private sponsors of research on children, youth, and families; and data users in academic communities and research organizations that focus on children and their well-being.

Four major issue areas emerged from the workshop: (1) theoretical issues, (2) methodological issues, (3) adequacy of data, and (4) dissemination of data. The topics represent the participants' cross-cutting observations and reflect the issues and challenges researchers face. Participants' ideas regarding the next steps to be taken in conducting longitudinal research on children and youth appear at the end of the report.

Since longitudinal studies span many years, often more than a decade, providing data that are representative of the current population becomes a challenge; such data are usually better collected by cross-sectional surveys. Although many longitudinal surveys are designed to provide continuous representation of the

population or of sampled cohorts, it can be difficult to continue to meet the changing demands for policy relevance, which is often facilitated by including new question sequences into the study design. In addition to the theoretical and methodological issues is the need for critical assessments of existing data on children, child development, and child well-being and careful consideration of topics to be added to future data collections on children, youth, and their families. At the core of the discussions was the issue of ensuring that data will be available to answer current and future policy questions, many of which concern issues of cause and effect and are increasingly focused on children and youth.

Participants agreed that children and youth constitute a special population for research, involving both constraints and challenges. A range of developmental issues across infancy, childhood, and youth must be addressed and studied. Thus, traditional survey instruments or interviewing techniques often require reassessment when the focus shifts from adults to children as respondents and subjects or to adults as informants about children or adolescents. Perceptions of respondent burden and cooperation may also differ. Special consents often must be granted when children and adolescents are involved as survey subjects.

Workshop participants discussed the extent to which specific theoretical perspectives guided the design of the surveys. They also discussed the theoretical underpinnings of measurements used in longitudinal studies on children, including such specific measures as those used in capturing issues related to child development.

Workshop participants debated the extent to which their surveys were guided by a specific theoretical framework or based on a specific theoretical model—for example, a model derived from developmental psychological principles or perspectives founded on behavioral genetics, sociological, or biological theories. The National Institute of Child Health and Human Development's (NICHD) Study of Early Child Care was guided by developmental psychology, and the hypotheses to be tested by the study were spelled out prior to data collection, providing a unique opportunity to study both predicted and unpredicted connections between processes and outcomes. This appears to be an exception rather than the rule. Most study designs are not based on a specific theoretical model. Some participants thought that surveys, especially those conducted by the federal government, could not commit to one specific theoretical model. However, participants agreed that, in order to study children and youth longitudinally and to capture transitions in their development, it is important to know what is expected to happen at different ages. Experts agree that a good survey design offers data to

a broad audience and allows for a range of hypotheses to be tested. The measures included in the study usually are not driven by one specific model of development. Once the data have been collected, they can inform questions that arise from a range of theoretical and policy perspectives.

The National Longitudinal Study of Adolescent Health (Add Health) serves as an example of a survey that allows comparisons of sociological and biological models. Based on a behavioral genetics design, it links children with different kinds of genetic relatedness. A large sampling frame allows the study of combinations, such as twins, half-siblings, cousins, and biologically unrelated children living in the same household, as well as the more common nontwin siblings. At the same time, its sample provides aggregations of respondents into peer, family, school, and neighborhood context that are central to prevailing ecological theories of development.

Workshop participants agreed that, in order to understand outcomes for children and youth, surveys must include environmental influences that go beyond the home to the neighborhood, child care provider, school settings, and perhaps even the work site and the judicial system. The workshop participants also agreed that surveys need to consider multiple dimensions of child well-being. Many of the more recent surveys represented at the workshop incorporated this insight into their design. For example, the Early Childhood Longitudinal Study assesses children along a number of physical, social, emotional, and cognitive development dimensions—an important distinction from past educational studies that focused almost exclusively on cognitive outcomes. Thus, this study is guided by a framework of children's development and schooling, emphasizing the child and the interaction among the child's family, school, and community. Analyses of the Early Head Start Evaluation will be based in part on the participating programs and theories of change, which specify their expected outcomes and program strategies for achieving them.

Participants called for research and development of short scales for implementation in surveys. Such research should include considerations of alternative methodological approaches, such as balanced incomplete block spiraling (BIB). In this approach, a many-item scale is split into overlapping blocks, and each person answers only a few items. The participants also considered it important to provide realistic guidelines about what constructs are amenable to being assessed in the context of surveys.

Workshop participants discussed the goal of assessing children's development in several critical domains, combined with the practical constraints imposed by a survey's methodology. Critical domains of child development include cognitive and noncognitive areas (e.g., physical, social, and emotional development). Processes as well as outcomes should be captured (e.g., approaches to learning

and approaches to problem solving). The child development community and the survey research community offer different perspectives on the best data collection strategies and often use different instrument validation tools. The participants, very aware of these issues, concurred that it is an enormous challenge to get valid data on children and their development in the context of a survey.

Participants recognized that the scales administered in assessments must be appropriate for the ages of the children and youth in the study and that they must be recognized by experts in the field—such as developmental psychologists—as valid and accurate indicators. However, financial and human project resources impose practical constraints on how intensively any given aspect of child development can be assessed. The participants thought, for example, that current clinical assessment scales are not always appropriate for surveys. These scales or batteries are developed for clinical use with individuals—where accuracy is critically important—or for research within relatively small and homogeneous samples. When the objective is not clinical treatment but the estimation and description of relationships among variables and the study population is heterogeneous, it may be both valid and necessary in terms of time and cost of assessment to work with a well-designed subset of assessment items that can be readily administered in the survey setting. Furthermore, the personnel available to administer the assessments often are not professionals in the field but hired data collectors who must be given special training. Thus, the assessment scales must be brief and amenable to standardization and computerized instrumentation.

Workshop participants addressed how information is gained from direct and indirect assessments. Indirect assessments are included in many surveys and may involve videotaping behavior. For example, the Early Head Start evaluations videotape interactions between mothers and children at three ages. Most surveys depend on a mixture of assessment methodologies.

Measurement of change is a central goal in longitudinal studies, and the participants discussed adaptive testing methodologies to achieve good measurement throughout the range of skills being assessed. It is important that floor and ceiling effects be avoided, e.g., that test items allow for detection of growth for children performing at both the lower and upper end of the distribution of difficulty. The Early Childhood Longitudinal Study's kindergarten cohort was offered as an illustration of a survey that uses a two-stage adaptive approach. Here, the degree of success in answering questions early in a test sequence determines the level of difficulty of succeeding questions. This type of approach is also valuable when the time available for testing is limited; children spend most of their testing time responding to items that are at or near their ability level rather than items across the entire ability distribution.

Workshop participants also recognized that language use and capacity influence many assessments. Sometimes, because of a lack of multilingual assessment instruments and multilingual data collectors, assessments are conducted in English even if this is not the subject's first language or the language spoken by

the child. Thus, for a study such as the New Immigrant Survey, the design of more appropriate assessment instruments is a critical challenge.

Finally, participants noted the importance of capturing and measuring the impact of children's environments. It is often difficult to separate school effects from neighborhood effects, and the concept of neighborhood is not always clearly conceptualized in the study design. Social science concepts such as cohesion and informal social control may be captured, but measures linking service availability and service utilization at the community level to outcomes for children and youth are lacking. Participants called for better measures of the influence of environment on child development, especially the impact of the neighborhood.

Some methodological challenges are inherent to all longitudinal surveys. Providing representative and current data are among such challenges. From its inception, a longitudinal survey collects data into the future from the same observational units. Thus, a survey's design entails using the proper sampling frame to select a sample representative of that frame. Different strategies are used to ensure that the sample remains representative of the universe it purports to represent over time. As the subjects in the sample age, the sample must be refreshed or augmented to compensate for attrition and other changes. The survey must also be designed to ensure that the data provide timely information. Staying current and addressing emerging policy issues continue to be challenges for those who design longitudinal surveys. Some surveys address this need by augmenting the survey scope, adding supplemental modules for occasional coverage of specific topics. These challenges face all longitudinal surveys, not just surveys of children. For longitudinal surveys of children, however, workshop participants believed it to be especially important to learn the best ways to track respondents and minimize nonresponse.

Keeping track of respondents who participate in longitudinal surveys—especially those who move or for whom there is no fixed address—is critical to the success of any such survey. It is not uncommon for children to change households, to move as a household, or to change schools during the year or between years with or without changing home addresses, creating both operational and analytical difficulties. This is a particular challenge for those who want to include special populations, such as children involved in foster care, children whose parents have separated or divorced, and children of high-risk families (e.g., low-income families). Although time-tested procedures for tracking respondents are in place, workshop participants emphasized the importance of keeping up with technological advances, such as World Wide Web or CD-ROM-based address

directories, and using innovative approaches to track respondents, including establishing toll-free phone numbers for respondents to contact researchers, offering monetary incentives, and developing special tracking networks.

Nonresponse is an issue for all those involved in carrying out surveys, but it is of particular concern in longitudinal data collections, especially those that gather data from different sources and multiple respondents. Obtaining respondents' cooperation is not a trivial issue, but workshop participants agreed that this is an area where survey designers can exercise some control. Perceptions of the burdens on respondents and their impact on response rates may vary by the socioeconomic status of the families involved as well as the children's ages. Workshop participants concurred that too little is known about what works and for whom, particularly with reference to subsamples of children, youth, and families. They called for more tests of the types of incentives, especially monetary, that successfully reduce nonresponse.

Workshop participants agreed that abundant data on children and youth already have been collected but that there is little or no recent synthesis across subject-matter areas. Suggestions about ways to inventory and synthesize what is known included mining existing datasets to create topic-specific data files, creating collections of topic-specific research papers, and engaging in joint analysis efforts to enhance the knowledge of best current practices in the field. It is evident that data are lacking on some variables of keen current interest. Little is known about how children and adolescents spend their time, for example, or about the role of fathers in children's development. Participants expressed concern about the paucity of relevant policy data to study the effects on children and youth of program and policy changes in such areas as welfare reform. The ensuing discussion focused on three areas in which data appear to be lacking or inadequate: time use, fatherhood, and program evaluation.

Time use of children and adolescents has received increased attention in recent years, but there is little current empirical data on the topic. A few of the surveys represented at the workshop collect such data through diaries. For example, in the Panel Study of Income Dynamics (PSID), parents or caregivers record children's activities in diaries for each child for two 24-hour periods, weekends and weekdays, from birth through age 12. When completed, the diaries are collected and the entries are edited by interviewers.

Workshop participants discussed the value of such data to get a picture of how children and youth use their time and to gain insights about input from parents, child care providers, and school personnel. A diary is a unique data collection method and perhaps one of the best ways to get a picture of how children use their time and to gain insight about input from their parents, child care providers, and school personnel. However, the workshop participants expressed a general concern about the validity of time-use data and concurred that more research is needed on measurement errors.

Data on fathering and father-child interactions are inadequate. Workshop participants called specifically for more survey research on the impact of fathers on child development, as relatively little is known about the roles that both biological and social fathers play in children's development. Traditionally, mothers are believed to be the best survey respondents for questions regarding their children. But interviewing fathers in addition to mothers may enhance understanding of family dynamics and the context in which children develop. This approach poses additional methodological challenges and costs, especially when interviewing fathers involves tracking and locating absent ones (Peters et al., 1996).

The impact of absent fathers on children's development is a topic of current national concern, and several of the studies represented at the workshop collect data on this topic. For example, the PSID gathers information from mothers about absent fathers and makes an attempt to locate them, often successfully. Beyond direct assessments of fathering, recent data from the Add Health study concur with previous findings that youth for whom father presence or absence is identified show different behavioral profiles than youth for whom this information is lacking altogether. There appear to be many ways in which fatherhood data can enhance understanding of child development. The NICHD is funding a major interview and videotaping survey of the fathers of Early Head Start children.

At the workshop, considerable discussion focused on the extent to which existing surveys can contribute to efforts to assess the effects of recent governmental policy changes on children, youth, and families. In particular, participants debated the ways in which survey data can provide information about the impact of recent welfare reform legislation¹ on the outcomes of children and youth. In

general, few surveys include both the child outcome data and input data, such as program participation, that are needed to link findings to welfare reform. Moreover, given the extensive state- and county-level variations in implementation of the law, data that can be disaggregated to this level are needed. Few if any of the existing surveys can provide this level of data. The Early Head Start Evaluation includes information from local site visits that might be useful in monitoring the effects of welfare reform. The Census Bureau's Survey of Program Dynamics (SPD) also would provide much-needed data (see Appendix A for additional details on this survey). Novel child and family content is due to be piloted in spring 1999; however, at the time of the workshop the amount of information that will be collected was still being determined. Overall, the experts believe that available surveys can provide only inferential insights into the effects of welfare reform on children and youth and that most are not able to track the effects.²

Overall, workshop participants cited as a challenge the need to obtain pre- and post-reform data to determine what outcomes can be attributed to policy change. They also noted the special challenges inherent in program evaluations that differ from those facing national longitudinal studies. These include the collection of program and policy data and the critical importance of considering when it is appropriate to assess program or policy outcomes. Often, the desire to study groups that are representative of a national population is at odds with the desire to study comparable groups receiving competing treatments.

Workshop participants believed public release and dissemination of data to be a controversial and challenging issue. Tensions exist surrounding the wide-

spread interest in timely release of datasets to the public, concerns about proper documentation to guard against inappropriate uses of data, and fears about breaches of respondent confidentiality. Data release and disclosure were cited as among the most common challenges faced by those conducting longitudinal surveys. Although the time between data collection and public release varies, it has become increasingly simple to obtain data with the advent of computer disks, the Internet, and electronic data file transfers.

Universal availability of data, however, raises a concern for proper analytical applications of data (e.g., when and how to use weighting, how to portray outcome measures). Furthermore, the trend is to seek ways to link data across datasets and to increase the knowledge base for smaller geographic units. It is central in this context to ensure that, when data are made available to the public, the process does not inadvertently disclose information about individual respondents. In addition to confidentiality, consent is a crucial issue for those involved in releasing data, such as videotaped data from longitudinal surveys. Amid growing concerns about confidentiality and privacy in this computer era, workshop participants agreed that efforts are needed both to develop protocols for the research community and to educate the public on these matters.

The public's need for up-to-date information, especially for program evaluations, puts pressure on researchers to release data as quickly as possible after collection. In the past, research findings made their way into print, and the data themselves were eventually released to the public and then archived. Today, data are often available simultaneously with or before extensive analysis takes place. Early releases, however, frequently do not include all data adjustments, such as adjustments for nonresponse or imputations for missing data.

For some data collections, initial data analyses occur inhouse before the data files are made available to the public. For example, there is a two-year inhouse research period before data from the Denver Youth Survey of the Program of Research on the Causes and Correlates of Delinquency are available for public use. In contrast, the National Center for Education Statistics (NCES) believes that its first priority is to produce statistics for the public and to make the data available to the public as quickly as possible, although some restrictions are imposed. In the NICHD Study of Early Child Care, early site-specific data files were released only to the principal investigators at the site that provided the data, not to all sites involved in the study. These data have not yet been released to the public. At times, data files are released under contractual arrangements only. The Add Health study has released data to the public using only half of the sample; the full set of data is available only to contractual users of the data.

Computer technology makes it possible for a broad audience with various levels of statistical sophistication to access data from surveys. Workshop participants discussed whether users who have not been involved in a survey's design and data collection can fully understand the complexities of the data collected and questioned the ability of less experienced users to apply proper weights to data extracts or to calculate standard errors. Participants agreed that a survey's protocol must be clear, concise, and accessible to the user community and that efforts should be made to facilitate proper use of the data—for example, by providing computer programs for computing standard errors, rather than requiring users to do their own calculations. Beyond such efforts to avoid misuses of data, workshop participants did not think they should take responsibility for the public's use or interpretation of data. On the contrary, they thought that further analyses of data may lead to innovative ways of thinking about patterns in the data, new insights, and discoveries. And although ample opportunity exists for individuals to misinterpret data that are disseminated, peer reviews serve as gatekeepers for the scientific community.

In addition to proper analytical applications of data, universal availability also raises concern about confidentiality and data disclosure. The trend in the social sciences field is to seek ways to link data (e.g., linking survey data to state administrative data) and thus to increase the range of information that is available for smaller geographical units. Administrative data can also be used as a validation tool to confirm individual reports of, for example, public benefits received. Data can be linked in many different ways. Some longitudinal studies are designed to pursue personal links to respondents, such as ties to absent fathers, parents, children, and siblings. Some studies also link children and adolescents to their child care providers and their educational environment. Often, a respondent's Social Security number is required for such links to be made successfully, opening the door to perceptions of inappropriate intrusion.

The practice of data linkages raises many ethical issues. It remains a challenge to ensure that data confidentiality is not breached. Workshop participants discussed techniques for disguising the identity of respondents, such as different strategies of adding noise to the data (e.g., random components that increase variability but, on average, do not contribute to the magnitude of estimated effects). Depending on the techniques used, it may be possible to identify an individual respondent using just a few variables. Participants agreed that, although linkages across datasets can enhance the utility of survey data in many ways, caution is warranted, particularly when samples of children and youth are involved for whom issues of custody, juvenile crime, and consent for medical