The National Children’s Study (NCS) was authorized by the Children’s Health Act of 2000 (Public Law 106-310) and is being implemented by a dedicated Program Office in the Eunice Kennedy Shriver National Institute of Child Health and Human Development (NICHD). In the words of the Program Office, the NCS is planned to be a “longitudinal observational birth cohort study to evaluate the effects of chronic and intermittent exposures on child health and human development in U.S. children.” The NCS would be the first study to collect a broad range of environmental exposure measures for a national probability sample of about 100,000 children followed from birth or before birth to age 21 and has great potential value, as summarized in Box 1-2 in Chapter 1.
Detailed plans for the NCS were developed by 2007 and reviewed by an outside panel (see National Research Council and Institute of Medicine, 2008). At that time, sample recruitment for the NCS Main Study was scheduled to begin in 2009 and to be completed within about 5 years. However, results from the initial seven “Vanguard Study” (pilot) locations, which recruited sample cases in 2009-2010, indicated that the proposed household-based recruitment approach would be more costly and time consuming than planned. In response, the Program Office implemented a number of pilot tests in 2011 to evaluate alternative recruitment methods. Based on these results, the study design was revised in early 2013, and a tentative 2015 start date was set for the Main Study.
In March 2013, Congress requested a review of the revised study design by a panel of the National Research Council and Institute of Medicine, stipulating that contracts were not to be let for the NCS Main Study until 60 days after
completion of the review report. Congress specifically stated the panel study would be “to conduct a comprehensive review and issue a report regarding proposed methodologies for the NCS Main Study, including whether such methodologies are likely to produce scientifically sound results that are generalizable to the United States population and appropriate subpopulations.”
NICHD specified that the panel’s review should cover such aspects of the Main Study design as the national probability sample’s overall sample size and design; the use of hospitals and birthing centers as the primary sampling unit; the relative size of the prenatal and birth strata in the probability sample; the size of the supplemental convenience sample; optimal use of sibling births; use of health care providers to refer prospective participants; the proposed study visit schedule, with emphasis on more frequent data collection in pregnancy and early childhood; the proposed approach to assess health and developmental phenotypes; and the proposed approach to define and characterize health disparities.
The National Children’s Study has the potential to add immeasurably to scientific knowledge about the impact of environmental exposures, broadly defined, on children’s health and development in the United States. The panel supports a number of elements of the proposed design for the NCS Main Study, including;
- the use of a national equal probability sample for a large cohort of births,
- the concept of the study as a data collection platform with a focus on health and development guided by exemplar scientific hypotheses,
- the inclusion of siblings born within the 4-year recruitment window, and
- the collection and storage of biological and environmental samples to permit subsequent analysis.
The panel does not endorse two other aspects of the proposed design: (1) the plan to recruit only one-half of the 90,000 births in the probability sample prenatally and the other half at the time of birth and (2) the proposed convenience samples. The scientific consensus on the importance of prenatal exposures on child health and development strongly supports the need to recruit almost all of the births prenatally (excepting only when the mother does not seek prenatal care). The panel’s cost analysis shows that it is feasible to have close to 100 percent prenatal recruitment by dropping the planned convenience samples, which add little scientific value.
For other aspects of the proposed design—including the choice of hospi-
tals as primary sampling units instead of geographic areas as in the early pilot testing, the quality of available hospital sampling frames, whether stratification is adequate for understanding health disparities, the details of the sampling and recruitment strategies, the scientific merit of the proposed exemplar hypotheses that are to guide data collection, the schedule and content of data collection in early waves, and the extent of data collection burden on respondents—the panel did not receive sufficiently detailed information from the Program Office to assess them.
Because of this lack of information and related reasons (detailed in Chapter 6), the panel concludes that achieving a scientifically grounded and cost-effective design and implementation for the Main Study will require expansion of the scientific expertise in the Program Office, establishment of an authoritative multidisciplinary oversight structure to review the Program Office’s decisions, and regular independent outside review.
The panel evaluated whether the current NCS plan embraces a scientific approach that identifies current and anticipated future subject domains and measures that are of high priority for understanding children’s health and development. Such an approach is needed to guide key study design elements, such as the target population, the sampling strategy, and the schedule and content of data collection.
The panel endorses a number of the general elements of the NCS study design framework that are consistent with best design principles for longitudinal birth cohort studies. These elements include a conception of the study as a data collection platform (noted above) and a dynamic conception of health and disease, which calls for measuring health status, disease conditions, symptoms, and behaviors rather than just existing disease categories. For the latter element, however, insufficient details about the additional measures of conditions and symptoms to be collected precluded the panel’s ability to determine whether the burden on respondents could be excessive.
The panel also accepts the NCS assertion that scientific discovery during the more than two-decade life of the study will be facilitated by using “exemplar hypotheses” to guide the study design development rather than enumerating an extensive list of specific hypotheses. At the same time, the panel finds that the exemplar hypotheses proposed for the NCS are not sufficiently well developed to guide sample design and data collection in the early waves, nor is there a clear plan to identify lines of inquiry that could lead to important exemplar hypotheses to guide data collection in later waves.
As noted above, the panel does not endorse one important aspect of the proposed NCS design: the plan to recruit half the cases prenatally and half at birth. This plan does not adequately reflect the growing scientific consensus on
the importance of prenatal influences on child health and development. The panel recommends that the NCS use prenatal recruitment and data collection for almost all sample members, making an exception only for the small percentage of births to mothers who did not receive prenatal care. As part of its evaluation, the panel commissioned a cost analysis to enable it to identify alternative sample designs that could achieve nearly 100 percent prenatal recruitment and be largely cost neutral with the design as currently proposed.
The current NCS design calls for a national, equal probability of selection sample of 90,000 births using a stratified list sample design: hospitals are the primary sampling units, and prenatal care providers whose patients deliver at the selected hospitals are the secondary sampling units. In the proposed design, recruitment would occur during the prenatal period in providers’ offices for half of the sample and shortly after birth in the hospitals for the other half of the sample. The proposed sample design also includes 10,000 “convenience sample” births, 5,000 of which would be born to first-time mothers recruited prior to their child’s conception. The other 5,000 would be available to address a variety of issues that the NCS would define later: examples include populations exposed to natural disasters, such as hurricanes or industrial accidents, and children likely to experience disparities in health outcomes and not adequately represented in the 90,000-birth main sample.
The panel endorses a number of key aspects of this sample design, including its large size, use of probability-based selection methods, and intent to sample births with approximately equal selection probabilities. Plans for sample stratification (for example, by race and ethnicity) were insufficiently detailed for the panel to evaluate whether they are adequate to respond to the mandate for health disparities research priorities in the Children’s Health Act of 2000.
For a variety of reasons (detailed in Chapter 2), the panel judges the added value of the 10,000-birth convenience sample cases to be substantially less than the value of additional cases in the probability sample, recruited prenatally. The panel’s cost analysis shows that, holding total field costs constant, the elimination of the planned convenience samples would allow the probability sample to increase in size from 90,000 births, half of which are recruited prenatally, to nearly 95,000 births, nearly all of which are recruited prenatally.
In addition to moving from household-based recruitment to provider-based recruitment, a key change in the NCS sample design since the 2008 review is from a county-based to a hospital-based primary sampling frame. Very few details regarding the hospital-based frame were made available to the panel, in part because of a decision by NCS to change the source of the hospital list in early fall 2013. A list of hospitals and birthing centers might provide a useful frame for the NCS first-stage sample: however, the panel lacked information
concerning the quality of the proposed frame to determine whether it is justified for the NCS to shift from a county- to a hospital-based approach.
Another change from the initial 2008 design is the proposed inclusion of all siblings born subsequently to the originally sampled “target” children and within the planned 4-year enrollment period. The panel endorses this change, as siblings provide many analytic advantages, most prominently collection of preconception exposure information for second and higher-parity births.
Overall, however, the documents that NCS made available to the panel did not provide sufficient details for an evaluation of whether the proposed sample would meet the minimal standards of a scientifically based sample design required for large national data collections. The panel considers it vital for the NCS to develop a detailed sampling plan and recruitment strategy for the Main Study, using survey experts who have extensive experience in conducting large national surveys. The resulting detailed plan should also be appropriately reviewed.
The panel was asked to consider the proposed frequency of data collection, comparing the early and later years of the study. Although there is reasonable scientific justification to conduct more frequent data collection during the prenatal period and early years, the panel did not receive an adequate explanation of the scientific basis for the specific proposed schedule of visits.
With regard to study measures and methods, scientific quality is enhanced by using the most valid and standardized data collection measures and methods that are feasible, while maintaining sufficient flexibility to assess emerging domains of scientific inquiry. Since the panel did not receive information on specific study protocols, data collection methods, or study instruments, its review could not address the scientific merit or quality of these aspects of the NCS data collection.
The NCS Program Office has an elaborate process for the identification of study domains, data collection protocols, and instruments. As described in the documents, the process would be guided by a conceptual framework for health and development, including a review of existing standardized assessment methods, and would involve consultation with multiple advisory bodies. Although the process is elaborate, the NCS did not provide to the panel any documentation or draft protocols or instruments that would demonstrate its effectiveness as a means to develop scientifically and operationally optimal data collection protocols.
In addressing its goals of understanding the effects of chronic and intermittent environmental exposures on child health and human development, the NCS proposes to devote considerable resources to obtaining biological and environmental samples and storing them in ways that make them available for
future investigations. The panel endorses this method of ensuring that future innovations in measurement are incorporated into the study.
Throughout the panel’s study, the Program Office staff impressed us with their willingness to provide responses to the panel’s myriad questions on a tight time schedule. However, many of those responses did not provide adequate, well-justified scientific answers. The panel concludes that the office lacks sufficient in-house expertise in relevant scientific and survey research disciplines to enable it to use effectively the input it receives from contractors and advisory groups for design and operational decisions for the NCS Main Study. Moreover, the panel concludes that the current management structure—with the Program Office making decisions for the NCS without authoritative oversight that reflects all areas of relevant expertise—is not likely to produce the optimal design for a study that needs to be implemented in a scientifically grounded and cost-effective manner.
Cost-effective and scientifically grounded operation of the NCS Main Study requires a broader and deeper base of scientific expertise for the Program Office; an authoritative multidisciplinary oversight structure to ensure that the decisions of the Program Office are appropriately vetted by all relevant experts; and provision for periodic comprehensive reviews of the study by an independent outside group. The panel’s recommendations to address these shortcomings are below.
CONCLUSIONS AND RECOMMENDATIONS
The NCS Main Study offers enormous potential, but it also presents a large number of conceptual, methodological, and administrative challenges. In addition, funding uncertainties make it difficult to plan a study of this magnitude and duration. Like the scientists associated with the study itself, we are eager for it to succeed. We present our recommendations in the hope that, as it goes forward, the NCS will achieve its intended objective to examine the effects of environmental influences on the health and development of American children.
The panel’s detailed conclusions can be found in Chapter 7.
RECOMMENDATION 2-1: The scientific framework for the National Children’s Study should be based on current understanding of the determinants of children’s health and development and an informed consideration of the likely future trajectory of scientific discovery. The paradigms of developmental biology and life-course epidemiology, coupled with findings from other social and behavioral science research on the prenatal and
early life periods, should guide development of the design for the Main Study.
RECOMMENDATION 2-2: In order to facilitate scientific discovery during and after National Children’s Study data are gathered, the Main Study should use a national probability sample with the largest feasible sample size and an approximately equal probability of selection design, and it should recruit nearly all of the cohort as early in pregnancy as possible.
RECOMMENDATION 2-3: In order to facilitate scientific discovery during and after National Children’s Study (NCS) data are gathered, the Main Study should use valid and standardized data collection measures and methods, while maintaining flexibility to revise or develop new instruments. The NCS should also use state-of-the-art procedures to collect, archive, and provide access to biological and environmental specimens for future analyses.
RECOMMENDATION 2-4: The proposed strategy for the National Children’s Study Main Study to collect detailed data on children’s health status, conditions, symptoms, and behaviors should be followed to the extent possible, taking into account constraints of costs, operational feasibility, and the need to not overburden respondents.
RECOMMENDATION 2-5: While the panel appreciates the possible scientific value of gathering preconception exposure information on 5,000 first-birth children as part of the National Children’s Study Main Study, this supplemental sample should be dropped because of high costs, the lack of any evidence of the value of such a sample, the lack of detailed plans for both selection and analysis, and potential limitations in the proposed data collection schedule.
RECOMMENDATION 2-6: The other supplemental convenience samples proposed for the National Children’s Study Main Study should be dropped from the design, including samples of children exposed to natural disasters or geographically defined environmental exposures, samples of additional members of disadvantaged groups, and samples of siblings born outside the 4-year birth window. The potential added value of the supplemental sample cases is less than the value of the additional cases in the probability sample they would replace, specifically, the value of the additional prenatal cases in the probability sample.
RECOMMENDATION 3-1: The National Children’s Study Main Study sample should be stratified by characteristics that will achieve variability in socioeconomic status within important population groups to support analysis of health disparities, as well as achieving variability in environmental exposures and geography to support analysis of relationships between exposures and health outcomes.
RECOMMENDATION 3-2: A detailed plan for sampling, recruitment, and minimizing attrition bias for the National Children’s Study (NCS) Main Study should be fully developed and evaluated by sampling and survey experts independent from the NCS and approved by the proposed independent oversight committee before the study moves forward.
RECOMMENDATION 4-1: Prior to proceeding with the Main Study, the National Children’s Study (NCS) should develop scientifically well-grounded exemplar hypotheses that should be used to guide and evaluate decisions regarding the NCS design and data collection schedule and domains.
RECOMMENDATION 4-2: Because hypotheses will change over time, the National Children’s Study should implement a strong and public process to revise and develop new exemplar hypotheses to guide future study implementation, engaging with the extramural and intramural research communities.
RECOMMENDATION 4-3: The National Children’s Study Main Study should collect data during the prenatal period at multiple times for as many of the study participants as the budget will allow.
RECOMMENDATION 4-4: Although the panel does not endorse the current proposal for a substantial birth enrollment stratum, if the National Children’s Study (NCS) Main Study retains such a stratum, the NCS should conduct a full pilot test of recruitment and data collection during the birth visit before the Main Study is implemented.
RECOMMENDATION 4-5: The National Children’s Study Program Office should document and provide justification for development of the data collection schedule, content, and methods now and going forward. The documentation should be sufficient to guide use of the study data by future researchers.
RECOMMENDATION 4-6: The National Children’s Study Program Office should finalize the study visit data collection protocols that it
intends to use for the Main Study (including questionnaires and other measurements), at least through age 1, and then pilot test the protocols before implementing the Main Study. The protocols and findings of the pilot tests should be peer reviewed and approved by the proposed independent oversight committee prior to initiating the Main Study.
RECOMMENDATION 4-7: The relevance to health disparities should be an explicit criterion for selecting the constructs that will be assessed as part of the National Children’s Study (NCS) Main Study, the measures that will be used to assess them, and the timing of the assessments. The NCS should obtain input from experts on health disparities in childhood as part of the documented process through which the measures for inclusion are selected, and the measures should be approved by the proposed oversight committee.
RECOMMENDATION 4-8: The National Children’s Study should consider producing an “early release” version of the data from the Main Study that includes data collected in the early years of each wave’s data collection cycle and makes those data available to analysts under the terms of restricted access data centers.
RECOMMENDATION 5-1: Given the goal for the National Children’s Study (NCS) to understand the links of environmental exposures to child health and development and its cost structure, if major reductions in the cost of the study need to be made, they should be reductions in sample size rather than exposure domains. Along with such a decision to reduce the sample size, the NCS should reconsider whether to oversample minorities in order to maintain the ability to evaluate health disparities with a reduced sample.
RECOMMENDATION 6-1: The National Institute of Child Health and Human Development (NICHD) should consider and implement one or more means to enhance the scientific expertise of the National Children’s Study (NCS) Program Office by recruiting experts in relevant fields from within the National Institutes of Health, other federal agencies, and outside government. In addition, NICHD should consider contracting with experts outside of government to work part time on the NCS as a means to bolster the scientific expertise that is focused on the NCS.
RECOMMENDATION 6-2: The National Institutes of Health should strengthen the oversight and leadership of the National Children’s Study (NCS) by establishing an oversight scientific management structure to
include a full range of relevant expertise, with review and approval authority for NCS design decisions.
RECOMMENDATION 6-3: The National Children’s Study (NCS) Program Office should establish a mechanism, such as a study section like those in the National Institutes of Health, or use a qualified independent organization to conduct periodic comprehensive outside scientific reviews of the design and operations of the NCS Main Study. To facilitate the work of such a committee and transparency for the study more generally, the NCS Program Office should promptly post on its website all scientific studies conducted for the NCS.