4
Study Design, Data Collection, and Analysis

This chapter begins with an overview of the National Children’s Study (NCS) design. It then describes, critiques, and makes recommendations on sampling design and data collection plans and their impact on quality control and response burden. Finally, data analysis and dissemination plans developed for the NCS are described and recommendations provided for improvement.

DESIGN SUMMARY

The NCS is designed as a longitudinal investigation of a nationally representative sample of 100,000 births to residents in the United States during the years 2008-2012. The births selected for the NCS will be identified from a probability sample of households chosen with standard survey sampling techniques. A sample cohort of births will be identified from a sample of all noninstitutionalized and cognitively/mentally competent women ages 18-44 who currently live in the national sample of households. The children born to these women during the recruitment period will in turn be followed until they are 21 years old. A variety of physical examinations and interview surveys, conducted in person and by telephone, will be completed during the follow-up period.

The NCS birth cohort will be identified from a stratified cluster sample of households chosen in two or more stages (or selection steps), with random selection used in each stage and the number of stages depending on the



The National Academies | 500 Fifth St. N.W. | Washington, D.C. 20001
Copyright © National Academy of Sciences. All rights reserved.
Terms of Use and Privacy Statement



Below are the first 10 and last 10 pages of uncorrected machine-read text (when available) of this chapter, followed by the top 30 algorithmically extracted key phrases from the chapter as a whole.
Intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text on the opening pages of each chapter. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.

Do not use for reproduction, copying, pasting, or reading; exclusively for search engines.

OCR for page 101
4 Study Design, Data Collection, and Analysis T his chapter begins with an overview of the National Children’s Study (NCS) design. It then describes, critiques, and makes recommenda- tions on sampling design and data collection plans and their impact on quality control and response burden. Finally, data analysis and dissemi- nation plans developed for the NCS are described and recommendations provided for improvement. DESIgN SuMMARy The NCS is designed as a longitudinal investigation of a nationally rep- resentative sample of 100,000 births to residents in the United States during the years 2008-2012. The births selected for the NCS will be identified from a probability sample of households chosen with standard survey sampling techniques. A sample cohort of births will be identified from a sample of all noninstitutionalized and cognitively/mentally competent women ages 18-44 who currently live in the national sample of households. The children born to these women during the recruitment period will in turn be followed until they are 21 years old. A variety of physical examinations and interview surveys, conducted in person and by telephone, will be completed during the follow-up period. The NCS birth cohort will be identified from a stratified cluster sample of households chosen in two or more stages (or selection steps), with ran- dom selection used in each stage and the number of stages depending on the 0

OCR for page 101
0 THE NATIONAL CHILDREN’S STUDY RESEARCH PLAN pattern of housing in the particular parts of the country that are selected.1 Some key features of the NCS sample design are presented in Table 4-1. The use of established random selection methods in each sampling stage will ensure that the NCS samples of households, eligible women of childbearing age, and births are national probability samples. In the first stage, 110 individual counties or small groups of contiguous counties were randomly chosen (by staff at a federal agency partner, Centers for Disease Control and Prevention/National Center for Health Statistics) to serve as the set of primary sampling units (PSUs). Each of these PSUs was selected with probability proportional to size (PPS), with the actual 1999-2002 count of live births serving as the measure of size for each PSU. Since multiple PSUs were chosen in some of the largest metropolitan areas, the 110 PSUs are found in 105 different “study locations or sites.” Fieldwork will begin in two waves (in 2009 and 2011) on random subsets of the remaining 98 study locations (the Vanguard Centers will undertake sampling, participant enrollment, and data collection in selected PSUs beginning in mid-2008). Within each sample PSU, 10-15 small groups of neighboring census blocks or block groups, called “segments,” are to be randomly chosen. All children born to all eligible female residents of all households located in the 1,100-1,650 sample segments will be included in the study, implying that no subsampling is to be done within sample seg- ments. To improve the efficiency of screening for pregnancies among the women ages 18-44 who are identified in selected households, women who are more likely to become pregnant will be monitored more intensively than the other women. These women may also be recruited through the health care providers they visit during pregnancy and delivery. Data collection and other survey process activities are to be completed using a common National Institutes of Health organizational structure, in which direct governance is provided by program staff of the National Institute of Child Health and Human Development (NICHD) to a network of centers tied to multiple study locations and a coordinating center, all of which are separately answerable to NICHD. A contract to serve as NCS coordinating center has already been awarded to Westat, Inc., of Rock- ville, Maryland, as have contracts to 24 centers, including the seven Van- guard Centers. NICHD anticipates the need for about 13 to 15 additional centers. Developed within this organizational framework, the field operations plan prepared by Westat includes some elements of standardization and 1 Whileonly two stages of selection are described here, additional stages may be added as needed to accommodate the largest primary sampling units (consisting of more than 500 seg- ments) and high-density urban segments with high-rise residential buildings and other features that make it difficult to achieve targeted household sizes for that stage of selection.

OCR for page 101
0 STUDY DESIGN, DATA COLLECTION, AND ANALYSIS central process control but also the potential for independent local process accountability, since separate contracts would be negotiated by the centers that would conduct the fieldwork in the 105 study locations. Westat’s over- all responsibility is to integrate its specific process tasks with those done by the other centers. This means that in addition to assuming responsibil- ity for survey instrument development, field staff training, and subsequent sampling activity, Westat must plan and orchestrate all activities surround- ing sample recruitment and retention, as well as the collection, processing, and analysis of all data that are obtained during the life of the study. Its coordinating role will require the methods and materials it develops to be incorporated into a written manual of operations. Finally, Westat must also see that all of these activities are completed efficiently, with high quality, and in a timely manner. The general responsibility of individual centers will be to appropriately and effectively apply the methods and use the materials that Westat develops. SAMPLINg DESIgN NCS sample selection follows well-established principles of popula- tion sampling. The most important feature is the strong commitment to random sampling methods in all stages of selection, combined with an effective use of sampling stratification selection with PPS selection in the first sampling stage. This approach is designed to ensure that the initial sample of births will be nationally representative. The extent to which the final data set is representative, of course, will depend on the study team’s success in maintaining the statistical integrity of the sample through a prolonged period of follow-up, as noted below. Another critical sampling feature is the use of the same approach (i.e., complete enumeration of a random sample of segments) in each sample PSU. Allowing individual cen- ters to develop their own sampling methods within PSUs could have made analysis of NCS data unnecessarily complex and created the possibility that within-PSU sampling methods were incompatible or inappropriate. Conclusion 4-1: We strongly endorse the use of probability sampling to select the NCS national sample of births. Other specific features of the NCS sample design were found to be strategically sound. The expected size of the proposed NCS sample is suffi- ciently large to justify the equal-probability sample of births, and PPS selec- tion combined with equal numbers of approximately equalized second-stage sampling units (i.e., segments) is the preferred way to yield equal selection probabilities of households, eligible women, and births. While oversam- pling important population subgroups is often used in national surveys to

OCR for page 101
0 THE NATIONAL CHILDREN’S STUDY RESEARCH PLAN TAbLE 4-1 Key Features of the Sample Design for the National Children’s Study Sampling unit and Frame Source Stratification (What is being sampled and from (Stratify by what? Which sample Stage what list?) allocation approach?) • 92 roughly equal-sized strata County/small group of contiguous 1 • Stratification by county size and counties Primary sampling unit (PSU) % of births to Hispanic women, % of births to Black women, % Defined so that each final PSU is of births to Asian women, and % expected to have at least 2,000 births of low-birth-weight newborns during 1999-2002; subdivided into “geographic units” for subsampling, as needed Frame: Grouped listing of 3,141 counties • 18 self-representing (or certainty) geographic unit 1a (Optional) PSUs, in 13 of the largest PSUs • Also by geographic proximity Formed and subsampled in large PSUs consisting of >500 segments somehow (e.g., by sorting on location?) Frame: Constructed list of subgroups of segments specially formed with PSU strata • 18 self-representing (or certainty) Segment 2 PSUs, in 13 of the largest PSUs • Also “geographically” somehow Groups of census blocks or block groups (BGs); approximately equal and (perhaps) by other segment in population size; intermediate characteristics selection of “chunks” as needed Frame: Constructed list of segments • None Chunk 2a (Optional) Formed and subsampled in high- density urban segments with large HH counts Frame: Constructed list of chunks in larger segments NOTES: The eligible-for-study population consists of births to noninstitutionalized and cognitively/mentally competent female residents (ages 18-44) during a designated 4-year en- rollment period, based on the woman’s location of residence at the time of birth.

OCR for page 101
0 STUDY DESIGN, DATA COLLECTION, AND ANALYSIS Sample Selection Overall Sample Size (How will random selection be used?) • Some type of without-replacement • 110 PSUs in 105 locations: 7 Vanguard probability proportional to size (PPS) sites + 98 other “study locations” random selection method — 18 self-representing PSUs in 13 study — Measure of size = actual 1999-2002 locations count of the number of births to — One nonself-representing PSU selected residents in the county(ies) by PPS in each of 92 strata • Rollout in waves: Vanguard first; then 3 random subsets of the remaining study locations • Select one with PPS in each second-stage • One geographic unit per “stratum” stratum — Measure of size = 1999-2002 estimated number of births • Using a field-verified list of USPS carrier • 10-15 segments selected in most sample addresses, visiting all households, PSUs enrolling all eligible females and resulting births among those who become pregnant during a 4-year sample recruitment and enrollment period (i.e., no subsampling within segments) • One at random within segment • 1 chunk per segment • Using a field-verified list of USPS carrier • 10-15 segments/chunks selected per addresses, visiting all households, sample PSU enrolling all births occurring in them during the 4-year enrollment period, as in selected segments (i.e., no subsampling within chunks) “Noninstitutionalized” is taken to mean not currently living in an institutional setting (e.g., hospital, nursing home, prison, assisted living facility, etc.), college dormitory, or military barracks. USPS = U.S. Postal Service.

OCR for page 101
0 THE NATIONAL CHILDREN’S STUDY RESEARCH PLAN increase the precision of comparisons between subgroups, the likelihood of there being many interesting subgroups to oversample in the NCS (defined by race/ethnicity, exposure to toxic substances or air, etc.), combined with limitations of oversampling methods, made an equal-probability design an appropriate approach for a sample of 100,000 births (Kalsbeek, 2003). The overall size of the proposed sample will support estimates for most impor- tant national population subgroups defined by race and ethnicity. Another positive feature of the proposed design is the use of established geopolitical area units (blocks and block groups) to define segments. Since statistical information from each decennial census is routinely prepared at the block and block group levels, potentially useful ancillary information from the U.S. Census Bureau and other sources will be readily available to describe contextual conditions and differential nonresponse. A third positive feature is the decision to select the NCS birth cohort sample by screening households for women prior to pregnancy to yield higher sample coverage and avoid the complications of frame multiplicity that result from provider-based approaches (Lesler and Kalsbeek, 1992). While sampling prepregnant women by screening a sample of households may avoid frame problems, it is the more expensive approach of the two because of the relatively low percentage of women screened in a general population sample of households who will actually become pregnant dur- ing the sample accrual period. The decision to delineate discovered women by their probability of becoming pregnant by asking them questions whose answers correlate with pregnancy (age, prior birth history, sexual activity, etc.) is conceptually sound, although it was not clear how accurately these data will predict the proportion of monitored women ages 18-44 who will become pregnant. This predicted pregnancy ratio among participating women will be a key component in the estimates of the ratio of the number of households needed to enroll about 1,000 births in each PSU. Conclusion 4-2: While we endorse the decision to select an equal prob- ability national sample of births as a reasonable strategy given the many key scientific objectives of the NCS, we recognize that a proportionate representation of the study’s target population will result in estimates for some subgroups that are not as precise as they would be had those groups been oversampled. Other features of the NCS sample design raise important issues, how- ever. A first and difficult one is the lower bound of the age criterion for female eligibility. NCS design specifications currently call for monitoring women ages 18-44 in selected households (instead of the more common 15-44 year range to define reproductive age in women’s health studies). About 3.4 percent of all births will be omitted. The decision to exclude

OCR for page 101
0 STUDY DESIGN, DATA COLLECTION, AND ANALYSIS births to teenage mothers ages 15-17, whose birth outcomes, such as low birth weight (Martin et al., 2007) and infant mortality (Mathews and MacDorman, 2007), are often more unfavorable than for older teens and women in their 20s and 30s may therefore skew findings of health and development outcomes of the children of the youngest women bearing children. Given the importance of teen pregnancy for both the mothers and their children, it would be desirable to include adolescent mothers in this study, however, the ethical issues in drawing them into the study would present a significant barrier. Although they might be liberated minors for purposes of medical care for themselves and their children, they might not be considered so for purposes of research. The regulations for the protec- tion of human research participants (45 CFR Part 46) present barriers to recruiting this population. Permission of one or both parents of the teen may often be required, whether or not the teen is pregnant. In addition, at least some institutional review boards (IRBs) might conclude that this study poses more than minimal risk given its onerousness, thereby requiring an enhanced level of scrutiny. Also, these young women may not want to divulge that they are preg- nant, and many pregnancies may be unwanted. Interviewing younger women, many of whom live with parents, would also introduce additional ethical issues. Obtaining parental consent to participate in the research might also involve overcoming hurdles such as potentially strained relation- ships with parents due to the pregnancy. These conditions may adversely affect response rates and retention rates. The panel discussed this issue and, recognizing these difficulties, endorses the decision of the NCS staff to omit births to women ages 15-17 from the NCS. Another set of concerns with the NCS sample design is related to sam- pling within PSUs: The current proposal calls for forming approximately equal-sized segments in each PSU and completely enumerating a random sample of 10-15 of them. One of these issues stems from the decision to include all households in selected segments, given the stated goal to enroll approximately 1,000 births per PSU. The difficulty here concerns what is needed to successfully meet the PSU sample size target: either (a) an accurate estimate of the household-to-enrollee ratio for births or (b) the flexibility to vary the number of assigned households in each PSU if accu- rate estimates of this ratio are unavailable. The problem with completely enumerating selected segments consisting of dozens or hundreds of house- holds is that there is limited flexibility to manipulate the number of assigned sample households to hit the target of 1,000 enrolled births. Of course, one could choose to prematurely stop fieldwork when the sample size target is reached, but this partial implementation of the recruitment protocol for each sample household could seriously compromise the statistical integrity

OCR for page 101
0 THE NATIONAL CHILDREN’S STUDY RESEARCH PLAN of the NCS birth cohort and thus seriously bias population estimates from a respondent sample that favors the easier-to-recruit cases (Kalsbeek et al., 1994). Conclusion 4-3: The process of identifying births from a national sample of households is complex and subject to numerous sources of attrition of uncertain magnitude. Because of this, it will be difficult to predict how many households must be initially selected to produce a probability sample of 1,000 births in each of the NCS sites. Recommendation 4-1: The NCS should consider modifying the sampling design to allow for flexibility in increasing the num- ber of study participants in the event that the estimated number of screened households needed to reach 1,000 births per PSu is incorrect. One approach would be to choose more segments than currently planned and to subsample from the planned complete enumeration listing of households located in each segment. By subsampling within segments and creating initially assigned and reserve sample components, there would be more flexibility to vary the number of households assigned for fieldwork. Subsampling thus would put less pressure on the accuracy of the household- to-enrolled-birth ratio and the decision as to how large the segments must be. Another complementary strategy to more precisely reach within-PSU sample size targets would be to invest resources during the pilot phase of field operations in the Vanguard Centers on estimating components of the ratio and to observe the determinants of change in this ratio among these sites. Components would correspond to attrition occurring between the following sequential steps of recruitment: assignment, location, screening, and rostering households, followed by female screening, participation in pregnancy monitoring, pregnancy, and giving birth to a live infant. A related concern is the approach to enumerating households within segments during the “list and screen” phase of sample recruitment. Because of the high cost and less than complete coverage of traditional methods of household listing, the NCS study design calls for an approach that uses vendor-supplied USPS carrier route addresses found in sample segments as a confirmatory source to assist field staff who will be doing within-segment household listing in the traditional way. While this type of adjudicated enu- meration of households makes good sense, it should be noted that in our understanding the statistical and practical utility of USPS postal addresses for household listing in area probability samples like the NCS is still in the early stages of development.

OCR for page 101
0 STUDY DESIGN, DATA COLLECTION, AND ANALYSIS Recommendation 4-2: The NCS should consider the proposed household enumeration approach to be experimental and should conduct carefully designed field studies to clearly establish the statistical and practical implications of the proposed adjudicated listing approach. The investigators should also consider the impact of the dynamic state of the set of residential households during the 4-year time period in which sample enrollment will occur.2 Specifically, methods should be developed to deal with the differential impact of the relatively more socially dynamic and mobile population subgroups (e.g., young singles, new immigrants) on final sample composition. Finally, the role of the sample design to ensure a broad diversity of exposure to environmental agents like nonpersistent pesticides, outdoor/ indoor air pollution, and aeroallergens was unclear and underdeveloped. While the current design specifications do address which particulate expo- sures are being considered (NCS Research Plan, Vol. 1, p. 6-3 and Tables 6-2 and 6-3), and the discussion of segment stratification within PSUs includes a passing reference to stratifying by “environmental measure” (among others), there is no clear indication of the specific steps that will be taken to ensure that the NCS sample will contain a wide range of envi- ronmental chemical exposures of relevance to the study hypotheses. Some examples of sources of ambient environmental source, or exposure, data that can be used to assign segment exposure are the Environmental Protec- tion Agency’s (EPA’s) Air Quality System (AIS), the Interagency Monitoring of Protected Visual Environments (IMPROVE) network, and point- and area-source primary particulate matter emissions information from the EPA National Emissions Inventory (NET). Recommendation 4-3: To ensure a diverse exposure profile in the sample, the NCS should consider a careful assessment of variation in ambient exposure to chemical agents within each PSu. If the set of segments in a PSu can be classified by combined exposure to a group of important chemical agents, this information could then be used to form varying exposure-level strata for segment sampling in each PSu and thus ensure a range of ambient exposure to relevant environmental agents. 2 The target population of households thus becomes any household existing for any length of time during the 4-year enrollment period.

OCR for page 101
0 THE NATIONAL CHILDREN’S STUDY RESEARCH PLAN DATA COLLECTION The NCS will rely on approximately 39 study centers, each of which is responsible for data collection in one or more of the 105 study locations. The study centers are selected through a competitive process based on evaluation criteria, such as demonstrated data collection capabilities, the ability to build extensive community networks for recruiting and retaining the sample of women and newborns, and a demonstrated commitment to the protection of individual respondents’ information. The study centers will hire and train data collection staff, work to ensure community engage- ment, and provide scientific support and consultation for the study. During the course of the study, the study centers will collect data at participants’ homes and in clinical settings, including the infant’s place of delivery. Other data will be collected by mail, telephone, or in person. Biological samples from mother, father, and child as well as air, water, soil, and dust from the child’s environment will also be collected. This is a monumental, multifaceted data collection effort. The multi- organization data collection model relies on government staff directing a central coordinating center that not only provides scientific support for the study’s program office but also is the nexus for the management and flow of the myriad data collection activities. The coordinating center must provide the centralized release, control, and maintenance of the sample, as well as the uniform training and quality-control activities to ensure a standardized national data collection. The coordinating center must have strong capabili- ties in locating sample members as well as in data collection itself, in the event that some study centers fail to meet response rate expectations. The complex decentralized data collection system requires a significant training program for interviewers and their supervisors, frequent communication, monitoring, and follow-up to achieve the goal of creating a standardized and uniform national data set. While decentralization maximizes the participation and involvement of scientists and researchers in the data collection efforts, it is an unusual model for collecting data from a large national probability population sam- ple. The overarching operational goal of the NCS is to collect completely standardized survey, biological, and environmental data from as many of its 100,000 sampled births as possible. Study sites and centers should be invisible to this process, in the sense that all are contributing completely comparable data to the national collective. A more conventional data col- lection model, used in such large health-related surveys as the National Longitudinal Study of Adolescent Health, is for a single, well-established data collection entity to control data collection efforts and maximize re- sponse rates. Apart from a regional network of supervisors responsible for interviewing, hiring, and quality control, there are no “sites” or “centers”

OCR for page 101
 STUDY DESIGN, DATA COLLECTION, AND ANALYSIS in this model. Of course, elements of the NCS data collection, such as the collection of biological and environmental samples, are extraordinary. But the bulk of the collected data will come from conventional interviews with the parents of the 100,000 children and, eventually, the children themselves. The proposed site-based design threatens the quality of the NCS national data collection efforts if the sites are not managed and monitored very closely by the NCS staff and the coordinating center. Managing and coordinating the data collection activities of 39 study centers and their quality-control processes will be a formidable challenge. Government staff and the coordinating center will probably develop the detailed specifications for hiring interviewers, for hiring staff to supervise data collection activities, for the uniform training modules that are admin- istered to study center trainers, and for certifying trainers and interview- ers to conduct the various data collections and guidelines for monitoring interviewers’ workload to lead to uniform and consistent data collections. The coordinating center must ensure the 39 study centers implement quality assurance procedures, including maximizing response rates, for all aspects of the study. This model requires substantial staff involvement to ensure its success, particularly high-level staff with substantial survey research and data collection skills and experience. Also it is likely that there will be site turnover during a 20-year study period from both loss of key personnel and the 5-year competitive renewal process. This turnover will also place a quality/training burden on the coordinating center. Despite these data quality coordination and control measures, there is likely to be substantial variability across sites in the implementation of the data collection protocols, some of which may threaten NCS study objec- tives. Despite persistent monitoring of data quality and procedures across all the sites, some sites are likely to fail to meet the high response rate and quality data collection standards required of this study. A data collection model that uses many fewer organizations responsible for data collection is logistically more realistic, practical, and likely to ensure uniform data collection procedures. In particular, using fewer organizations, particularly those that have proven capabilities in collecting high-quality national data in a uniform and consistent manner, would provide a greater likelihood of producing high-quality national data. Conclusion 4-4: The data collection model adopted by the NCS is complex, will challenge the abilities of the staff and coordinating center to achieve a uniform and consistent national data collection, and may compromise key study objectives. Data quality will be maximized and missing data minimized if data collection staff receive uniform and comprehensive training along with

OCR for page 101
 THE NATIONAL CHILDREN’S STUDY RESEARCH PLAN close supervision and periodic evaluation of their work. The interview situations faced by the data collection staff will be challenging because of the interview length and survey content. Quality-control procedures, such as random repeat interviews or repeat data collection, while burdensome for study participants, are an important component of quality control to identify poor-quality interview staff and incorrectly coded interviews. Study location production and quality-control results, particularly with regard to response rates, are critical to ensuring a uniform data collection system across the nation. The monitoring of productive data-quality goals is criti- cal to a successful data collection, since some study centers may not meet the quality-control guidelines and data collection production goals. Quality control of the instrument design is equally important, and the use of current best practices is critical, including the careful evaluation of potentially invasive questions, cognitive testing, literacy and cultural sensi- tivity evaluation, and implementing best practices for translations. The NCS uses several data collection modes for questionnaire data, in addition to col- lecting biological specimens, environmental samples, and medical examina- tion data. The research plan provides general descriptive information about many aspects of the study, but it does not provide specific information to determine whether best practices are being used in the design and testing of survey instrumentation. In particular, the content of the baseline inter- view is substantial, but there is little information describing the processes, procedures, and criteria for the items and approaches adopted. While some details may not be appropriate for inclusion in a research plan, an indica- tion that best practices are used in designing and testing questionnaires for the study, including any planned experiments or tests prior to the start of collection, is important. Similarly, little information is available in the research plan about the treatment of language-minority respondents and the procedures for testing questionnaires in languages other than English. Thorough cognitive testing of survey instruments needs to be undertaken at the Vanguard sites for language and cultural validity. Conclusion 4-5: The NCS research plan does not provide sufficient information on the use of data collection guidelines and quality-control procedures to enable evaluation of the planned implementation of a uniform national data collection system. To accomplish the goals of the study, a substantial amount of informa-

OCR for page 101
 STUDY DESIGN, DATA COLLECTION, AND ANALYSIS tion is needed on a wide variety of topics in household and clinical settings. The estimated response burden through the first 2 years of data collection is approximately 30 hours for women seen prior to pregnancy and 26 hours for women seen in the first trimester.3 The research plan notes that any single face-to-face data collection will be limited to no more than 4 hours. Other aspects of the collection, such as the clinical setting visits, will require additional time from the study participants. The study requires a substantial commitment of time by each study participant. From a practical point of view, eligible respondents, even after agreeing to participate, will prefer to minimize their participation time. While study staff are aware of the respon- dent burden associated with the collection and have made efforts to reduce content and consequently interview time, the unique nature of the study and the many stakeholders in the study together will lead to substantial pressure to add additional items to an already ambitious data collection program. The effect of the real or perceived respondent burden can result in lower quality data or no data at all. Staff sensitivity to the public’s interest and tolerance of the amount of time the study requires is important to the successful implementation of the study, but there is little information in the research plan to suggest that this issue is a concern. Conclusion 4-6: The NCS research plan does not address directly the issue of respondent burden, except to say that “some” effort is being made to reduce it, nor does the plan make clear the total number of hours the respondent must commit to the study. In particular, in light of the estimate of the interview length (4 hours) for the baseline interview, a critical collection for the study, the research plan pays little attention to respondent burden and its impact on the quality of the data. Initial response rates and sample retention are key quality indicators for longitudinal surveys. Initial response rates are particularly important, since so little information is available about the women who fail to participate in the initial wave of data collection. The NCS’s target initial response rates for study sites are in the 65-75 percent range. Given the burdensome nature of the data collection, the upper end of this range is ambitious but highly desirable to ensure the overall quality of NSC data. Some of the factors that depress the response rate are the sensitive nature of pregnancy, sexual rela- tions, and fecundity status for some women and the likely disproportion- ate representation of undocumented migrants and other high-risk groups 3 This response burden includes the actual home visits, follow-up visits to collect the envi- ronmental sampling equipment, time spent completing the self-administered questionnaires, phone calls, and the prenatal ultrasounds. It covers all the contacts from pregnancy (or before) through 24-month phone contact. It also includes the birth visit.

OCR for page 101
 THE NATIONAL CHILDREN’S STUDY RESEARCH PLAN among pregnant women. The response rate goal can be met only if response rates are as high as possible across all sites. Without a more centralized data collection structure or extremely close supervision, study sites are likely to have varied success in implementing the quality-control procedures, obtain- ing cooperation with the participants, and conducting the interviews. Recommendation 4-4: The NCS should consider ways in which the survey data collection could be consolidated into a smaller number of highly qualified survey organizations. While assumptions are based to some extent on the experience of other data collections, incentives to achieve response in other data collections, and the expectation that community involvement will help sample recruit- ment, the NCS research plan does not explicitly address the best methods and procedures for achieving target baseline response rates. Nor, in any significant way, does it address methods to influence sample units to partici- pate in this long-term study. There is little information on the community engagement model and how the model will be effective for the long-term recruitment of sample participants, nor is there much information on the use of incentives to improve cooperation. Furthermore, there does not ap- pear to be any description of planned experiments on methods to improve recruitment. These issues are critical to ensure a successful baseline inter- view and need to be addressed. While attaining a high baseline survey response is critical for the NCS, retaining sample cases over the life of the study is also of central impor- tance. Maintaining the representativeness of the sample over time is key to the acceptance of nationally representative results from the study. Nonpar- ticipation in a longitudinal survey can occur in several ways, and two, in particular, require different approaches to sample retention. First, sample members who initially refuse require interviewer follow-up procedures aimed at overcoming obstacles to their participation. Interviewer training and good supervisory monitoring and controls help in this regard. Second, sample cases who move to locations out of their sample areas must be located and followed to their new residence. Little is said in the research plan about how the study expects to maximize retention of sample cases, particularly regarding follow-up for these two groups of nonrespondents, nor is there any information concerning how long and how often refusal nonresponse and nonlocatable cases are to be included in the sample eligible to be interviewed. Typically, the largest loss in sample occurs in the early stages of a longitudinal study. The NCS assumption of a 2 percent sample loss each year is a reasonable target after the survey is under way for a few years and provided good locating procedures have been established. The NCS

OCR for page 101
 STUDY DESIGN, DATA COLLECTION, AND ANALYSIS study, however, is likely to experience the most sample loss during the most intensive part of the data collection—the first year—and should devote considerable resources to minimizing these sample losses. While the NCS study staff recognize the sample retention issues and are aware of the need to track sample cases and minimize attrition, there is little discussion of this in the research plan. Conclusion 4-7: The NCS research plan provides little information concerning best methods for sample recruitment to achieve initial and follow-up target response rates, sample maintenance and sample re- tention procedures for implementation at the study sites, community involvement plans consistent with the uniform implementation of data collection procedures, or contingency plans to support study sites that do not achieve target response rates. A successful national data collection program requires the establish- ment of standards across data collection sites, such as: • A clear set of measurable production and quality goals for each NCS field site. These measures should include, for example, the fol- lowing: interviewer training and fieldwork evaluation scores, stan- dard American Association of Public Opinion Research (AAPOR) unit response rates at each sampling level, components of standard AAPOR unit nonresponse rates, profiles of interview length for individual survey instruments, profile of total respondent burden, cutoff rate for completed interviews, item nonresponse rates for key study outcome variables, the annual cohort retention rate, the number of adverse events (e.g., cases of child neglect), as well as any other appropriate measures of the effectiveness of fieldwork. • A plan for closely and continuously monitoring data collection quality measures, such as those given above. The plan should in- clude guidelines for remedial interventions when sites do not meet expectations, including retraining of data collection specialists, evaluation of supervisory staff or staffing levels at each site, and, if necessary, replacement of site field staff with field staff from the coordinating center or some other qualified organization with relevant survey or data collection experience. Speed in recognizing and resolving these problems is essential. • A plan for the systematic collection and annotation of regional environmental measures in each of the sites. Recommendation 4-5: because of the complexity of the proposed organizational model for data collection and the difficulty of main-

OCR for page 101
 THE NATIONAL CHILDREN’S STUDY RESEARCH PLAN taining the quality and uniformity of data collection procedures across a large number of study sites, the NCS program office should establish and monitor strict standards for enrollment, reten- tion, and data collection at each of the study sites and be prepared to take immediate corrective action if sites do not meet high-quality standards in data collection. The research plan does not describe how the Vanguard Centers will be used to serve as test sites to identify operational, procedural, and question- naire design issues prior to their implementation at other study sites. These centers have the unique opportunity to serve as data collection laboratories for the study. While this has occurred on a limited basis, much more could be accomplished. Formal experimentation in Vanguard Centers of proce- dures or alternative instruments should be encouraged, although the task will be challenging, since the amount of time available prior to full-scale implementation of the study is limited. Operational procedures need to be defined, tested, and refined prior to going into the field. The Vanguard Centers should be the agents for such testing and refining. For example, a plan for sample size reestimation based on the first year’s experience of the Vanguard Centers could be developed. That is, us- ing the data from the first year of enrollment, the study team should reesti- mate the number of households that must be sampled to attain the overall study goal of 100,000 births. The sample size reestimation will involve quantifying components of attrition in the household sample associated with household screening, recruitment of eligible female residents, their be- coming pregnant and giving birth, and enrollment of the birth in the study. The Vanguard Centers can also help develop an initial assessment of the measures that are potentially useful predictors of nonresponse or attrition and whether environmental measures have sufficient variability to support analyses with adequate power. Recommendation 4-6: The NCS should prepare a plan for moni- toring progress of the study in reaching its sample size goals. As part of the plan, the NCS should take advantage of the experience of the Vanguard Centers to evaluate initial enrollment rates, the effectiveness and potential respondent burden of the interview instrument, and the ability of the Vanguard Centers to obtain the required household environmental measures reliably. A program as multifaceted and complicated as the NCS cannot resolve all the methodological issues related to the various types of data collection that will be used over the life of the study. The study is likely to encounter unsolved questions in how best to measure participant lifestyles or attitudes

OCR for page 101
 STUDY DESIGN, DATA COLLECTION, AND ANALYSIS about environmental exposures, or in how to analyze some of the complex genomic and biological data. The research plan did not explicitly describe any substudies, such as randomizing participants to different incentive treatments, randomizing the order of the formulations of some domains in the interview, or randomizing different strategies for asking sensitive questions. Determining best practices during the study requires an ongoing program of research and experimentation. Research must be directed to short- and long-term goals, resulting in information on which to base data collection decisions. The conduct of the research must be a priority for the NCS staff and have direct application to the study. While the research plan provides substantial discussion of many topics, there appears to be no formal funded program of methodological research, including pilot studies, that has staff resources assigned to it. Conclusion 4-8: The NCS research plan does not address the ongoing methodological needs of the study—to study data collection procedures and instruments, conduct experiments, and evaluate the quality of the survey operations and the quality of the data—nor does the plan ad- dress the best use of the Vanguard Centers. Recommendation 4-7: To resolve issues that arise during data col- lection, the NCS should set aside sufficient resources to maintain an ongoing program of methods research and field experimentation. Among the issues that might be addressed in this research are the reliability and validity of previously untested survey questions and measurement strategies, the effectiveness of sample retention pro- cedures, predictors of response outcomes associated with sample initial recruitment and subsequent annual retention, error implica- tions of unit nonresponse, adjustment strategies for unit nonre- sponse, and methods for dealing with item nonresponse. DATA ANALySIS AND DISSEMINATION Since all data will be stored on the coordinating center’s servers, all analyses will use centrally prepared analytic data sets. In addition to the usual codebooks, standardized formatting, and technical documentation, the data sets prepared by the coordinating center should include data ed- ited, imputed, and weighted using current best practices. Detailed recom- mendations about analysis methods are not appropriate at this stage, but the following general principles should be followed: • Data sets should include guidance on the best methods for analy-

OCR for page 101
 THE NATIONAL CHILDREN’S STUDY RESEARCH PLAN sis. The data sets from this study will be analyzed by many teams, often with different goals for an analysis. The guidance should discuss how methods may vary according to whether the goal of an analysis is the estimation of a relationship between exposure and outcome, an estimate of the exposure of a particular set of environmental factors in subsets of the U.S. population, or the prediction of the number of developmental disorders in subsets of the population. • Data sets should include documentation on the models used to compensate for unit, wave, and item nonresponse. While it will be impossible to predict all uses of the data, the statistical team cre- ating the data sets should work closely with project investigators when constructing the data sets. • All data sets should include complete documentation on the meth- ods used to impute, the assumptions about analyses used when developing the imputation procedures, and the potential strengths and weaknesses of these methods. • To help analysts separate the model-building and model-testing phases of their analyses, it would be helpful if the coordinating center were to use the sample design to develop four independent divisions of the sample and include a variable defining these inde- pendent quarter samples in the master NCS data set. Data analysis plans for the main analyses are not part of the research plan. In particular, while many research hypotheses are described, the research plan does not discuss how, when, or by whom the research will be conducted. The analysis plan for studying birth defects from impaired glucose metabolism, distributed at the September 2007 meeting of the panel, added substantial detail to the general analysis plans outlined in the research plan. Even that plan, however, does not have the specificity needed to prepare data collection instruments that both support analysis plans and minimize respondent burden. In smaller, less comprehensive studies, a detailed analysis plan is the primary tool for clarifying or sharpening study questions and for designing data collection instruments. Analysis plans should contain the following features: • Detailed information about possible predictors, including the scale or units of measurement, and, when possible, references to questions in interviews or assays being conducted by central laboratories. • Specification of the methods that will be candidates for miti- gating the effect of unit nonresponse, study attrition, and item nonresponse.

OCR for page 101
 STUDY DESIGN, DATA COLLECTION, AND ANALYSIS • Before final approval, each analysis plan should be reviewed by a group consisting of senior scientists and statisticians from the study team who are not involved with the study being reviewed. • Analysis plans should include strategies to mitigate the model overfitting that can arise when stepwise model selection is rou- tinely used to identify predictors. These strategies could include a prespecification of what variables should always be included in a data set, the use of shrinkage methods such as Lasso (Tibshirani, 1996), and routine use of either training and validation samples or cross-validated model evaluation. Data from the NCS will provide unprecedented opportunities for the study team and other investigators to learn more about the prevalence of de- velopmental disabilities, the environmental chemicals to which children are exposed, and the relationships between those disabilities and environmental exposures. Past experience with virtually all national data sets is that the research value of the data is maximized when as many skilled analysts as possible are able to access the data for original and replication analyses, and when the peer-review process judges the quality of the analyses performed. In light of this, the national data set that will be gathered as the result of an enormous expenditure of public funds should be made available as soon as is practical to the general scientific community for detailed analyses. The panel acknowledges the formidable challenges in ensuring that the confi- dentiality of study participants is protected but notes that a number of data dissemination models, such as the Census Bureau’s network of data centers, have successfully balanced the potentially competing goals of data access and respondent protection. The current research plan does not discuss any plans for the dissemination of data to the scientific community. Recommendation 4-8: The NCS should begin planning for the rapid dissemination of the core study data, subject to respondent protection, to the general research community and for supporting the use of the data after dissemination. The costs of implementing this plan should be estimated and set aside in future NCS budgets. Dissemination includes not only the publication of findings through reports and scientific papers and the production of documented data files for researchers, but also active support in the use of NCS data by the broadest possible range of qualified investigators. The dissemination plan should include the following elements:

OCR for page 101
0 THE NATIONAL CHILDREN’S STUDY RESEARCH PLAN • A timeline for making common elements of the data available to the general research community as soon as they have been cleaned and documented. • A plan for analytic support for investigators wishing to use the data, including analysis consultation, documentation of models used for developing unit nonresponse adjustments to the sampling weights and imputation of data elements subject to missingness, and recommended models for adjusting for measurement error or nondetectable levels in environmental variables.