The National Children's Study 2014: An Assessment (2014)

Summary

The National Children’s Study (NCS) was authorized by the Children’s Health Act of 2000 (Public Law 106-310) and is being implemented by a dedicated Program Office in the Eunice Kennedy Shriver National Institute of Child Health and Human Development (NICHD). In the words of the Program Office, the NCS is planned to be a “longitudinal observational birth cohort study to evaluate the effects of chronic and intermittent exposures on child health and human development in U.S. children.” The NCS would be the first study to collect a broad range of environmental exposure measures for a national probability sample of about 100,000 children followed from birth or before birth to age 21 and has great potential value, as summarized in Box 1-2 in Chapter 1.

Detailed plans for the NCS were developed by 2007 and reviewed by an outside panel (see National Research Council and Institute of Medicine, 2008). At that time, sample recruitment for the NCS Main Study was scheduled to begin in 2009 and to be completed within about 5 years. However, results from the initial seven “Vanguard Study” (pilot) locations, which recruited sample cases in 2009-2010, indicated that the proposed household-based recruitment approach would be more costly and time consuming than planned. In response, the Program Office implemented a number of pilot tests in 2011 to evaluate alternative recruitment methods. Based on these results, the study design was revised in early 2013, and a tentative 2015 start date was set for the Main Study.

In March 2013, Congress requested a review of the revised study design by a panel of the National Research Council and Institute of Medicine, stipulating that contracts were not to be let for the NCS Main Study until 60 days after

completion of the review report. Congress specifically stated the panel study would be “to conduct a comprehensive review and issue a report regarding proposed methodologies for the NCS Main Study, including whether such methodologies are likely to produce scientifically sound results that are generalizable to the United States population and appropriate subpopulations.”

NICHD specified that the panel’s review should cover such aspects of the Main Study design as the national probability sample’s overall sample size and design; the use of hospitals and birthing centers as the primary sampling unit; the relative size of the prenatal and birth strata in the probability sample; the size of the supplemental convenience sample; optimal use of sibling births; use of health care providers to refer prospective participants; the proposed study visit schedule, with emphasis on more frequent data collection in pregnancy and early childhood; the proposed approach to assess health and developmental phenotypes; and the proposed approach to define and characterize health disparities.

OVERALL ASSESSMENT

The National Children’s Study has the potential to add immeasurably to scientific knowledge about the impact of environmental exposures, broadly defined, on children’s health and development in the United States. The panel supports a number of elements of the proposed design for the NCS Main Study, including;

• the use of a national equal probability sample for a large cohort of births,
• the concept of the study as a data collection platform with a focus on health and development guided by exemplar scientific hypotheses,
• the inclusion of siblings born within the 4-year recruitment window, and
• the collection and storage of biological and environmental samples to permit subsequent analysis.

The panel does not endorse two other aspects of the proposed design: (1) the plan to recruit only one-half of the 90,000 births in the probability sample prenatally and the other half at the time of birth and (2) the proposed convenience samples. The scientific consensus on the importance of prenatal exposures on child health and development strongly supports the need to recruit almost all of the births prenatally (excepting only when the mother does not seek prenatal care). The panel’s cost analysis shows that it is feasible to have close to 100 percent prenatal recruitment by dropping the planned convenience samples, which add little scientific value.

For other aspects of the proposed design—including the choice of hospi-

tals as primary sampling units instead of geographic areas as in the early pilot testing, the quality of available hospital sampling frames, whether stratification is adequate for understanding health disparities, the details of the sampling and recruitment strategies, the scientific merit of the proposed exemplar hypotheses that are to guide data collection, the schedule and content of data collection in early waves, and the extent of data collection burden on respondents—the panel did not receive sufficiently detailed information from the Program Office to assess them.

Because of this lack of information and related reasons (detailed in Chapter 6), the panel concludes that achieving a scientifically grounded and cost-effective design and implementation for the Main Study will require expansion of the scientific expertise in the Program Office, establishment of an authoritative multidisciplinary oversight structure to review the Program Office’s decisions, and regular independent outside review.

STUDY DESIGN

The panel evaluated whether the current NCS plan embraces a scientific approach that identifies current and anticipated future subject domains and measures that are of high priority for understanding children’s health and development. Such an approach is needed to guide key study design elements, such as the target population, the sampling strategy, and the schedule and content of data collection.

The panel endorses a number of the general elements of the NCS study design framework that are consistent with best design principles for longitudinal birth cohort studies. These elements include a conception of the study as a data collection platform (noted above) and a dynamic conception of health and disease, which calls for measuring health status, disease conditions, symptoms, and behaviors rather than just existing disease categories. For the latter element, however, insufficient details about the additional measures of conditions and symptoms to be collected precluded the panel’s ability to determine whether the burden on respondents could be excessive.

The panel also accepts the NCS assertion that scientific discovery during the more than two-decade life of the study will be facilitated by using “exemplar hypotheses” to guide the study design development rather than enumerating an extensive list of specific hypotheses. At the same time, the panel finds that the exemplar hypotheses proposed for the NCS are not sufficiently well developed to guide sample design and data collection in the early waves, nor is there a clear plan to identify lines of inquiry that could lead to important exemplar hypotheses to guide data collection in later waves.

As noted above, the panel does not endorse one important aspect of the proposed NCS design: the plan to recruit half the cases prenatally and half at birth. This plan does not adequately reflect the growing scientific consensus on

the importance of prenatal influences on child health and development. The panel recommends that the NCS use prenatal recruitment and data collection for almost all sample members, making an exception only for the small percentage of births to mothers who did not receive prenatal care. As part of its evaluation, the panel commissioned a cost analysis to enable it to identify alternative sample designs that could achieve nearly 100 percent prenatal recruitment and be largely cost neutral with the design as currently proposed.

SAMPLE DESIGN

The current NCS design calls for a national, equal probability of selection sample of 90,000 births using a stratified list sample design: hospitals are the primary sampling units, and prenatal care providers whose patients deliver at the selected hospitals are the secondary sampling units. In the proposed design, recruitment would occur during the prenatal period in providers’ offices for half of the sample and shortly after birth in the hospitals for the other half of the sample. The proposed sample design also includes 10,000 “convenience sample” births, 5,000 of which would be born to first-time mothers recruited prior to their child’s conception. The other 5,000 would be available to address a variety of issues that the NCS would define later: examples include populations exposed to natural disasters, such as hurricanes or industrial accidents, and children likely to experience disparities in health outcomes and not adequately represented in the 90,000-birth main sample.

The panel endorses a number of key aspects of this sample design, including its large size, use of probability-based selection methods, and intent to sample births with approximately equal selection probabilities. Plans for sample stratification (for example, by race and ethnicity) were insufficiently detailed for the panel to evaluate whether they are adequate to respond to the mandate for health disparities research priorities in the Children’s Health Act of 2000.

For a variety of reasons (detailed in Chapter 2), the panel judges the added value of the 10,000-birth convenience sample cases to be substantially less than the value of additional cases in the probability sample, recruited prenatally. The panel’s cost analysis shows that, holding total field costs constant, the elimination of the planned convenience samples would allow the probability sample to increase in size from 90,000 births, half of which are recruited prenatally, to nearly 95,000 births, nearly all of which are recruited prenatally.

In addition to moving from household-based recruitment to provider-based recruitment, a key change in the NCS sample design since the 2008 review is from a county-based to a hospital-based primary sampling frame. Very few details regarding the hospital-based frame were made available to the panel, in part because of a decision by NCS to change the source of the hospital list in early fall 2013. A list of hospitals and birthing centers might provide a useful frame for the NCS first-stage sample: however, the panel lacked information

concerning the quality of the proposed frame to determine whether it is justified for the NCS to shift from a county- to a hospital-based approach.

Another change from the initial 2008 design is the proposed inclusion of all siblings born subsequently to the originally sampled “target” children and within the planned 4-year enrollment period. The panel endorses this change, as siblings provide many analytic advantages, most prominently collection of preconception exposure information for second and higher-parity births.

Overall, however, the documents that NCS made available to the panel did not provide sufficient details for an evaluation of whether the proposed sample would meet the minimal standards of a scientifically based sample design required for large national data collections. The panel considers it vital for the NCS to develop a detailed sampling plan and recruitment strategy for the Main Study, using survey experts who have extensive experience in conducting large national surveys. The resulting detailed plan should also be appropriately reviewed.

STUDY CONTENT

The panel was asked to consider the proposed frequency of data collection, comparing the early and later years of the study. Although there is reasonable scientific justification to conduct more frequent data collection during the prenatal period and early years, the panel did not receive an adequate explanation of the scientific basis for the specific proposed schedule of visits.

With regard to study measures and methods, scientific quality is enhanced by using the most valid and standardized data collection measures and methods that are feasible, while maintaining sufficient flexibility to assess emerging domains of scientific inquiry. Since the panel did not receive information on specific study protocols, data collection methods, or study instruments, its review could not address the scientific merit or quality of these aspects of the NCS data collection.

The NCS Program Office has an elaborate process for the identification of study domains, data collection protocols, and instruments. As described in the documents, the process would be guided by a conceptual framework for health and development, including a review of existing standardized assessment methods, and would involve consultation with multiple advisory bodies. Although the process is elaborate, the NCS did not provide to the panel any documentation or draft protocols or instruments that would demonstrate its effectiveness as a means to develop scientifically and operationally optimal data collection protocols.

In addressing its goals of understanding the effects of chronic and intermittent environmental exposures on child health and human development, the NCS proposes to devote considerable resources to obtaining biological and environmental samples and storing them in ways that make them available for

future investigations. The panel endorses this method of ensuring that future innovations in measurement are incorporated into the study.

PROJECT MANAGEMENT

Throughout the panel’s study, the Program Office staff impressed us with their willingness to provide responses to the panel’s myriad questions on a tight time schedule. However, many of those responses did not provide adequate, well-justified scientific answers. The panel concludes that the office lacks sufficient in-house expertise in relevant scientific and survey research disciplines to enable it to use effectively the input it receives from contractors and advisory groups for design and operational decisions for the NCS Main Study. Moreover, the panel concludes that the current management structure—with the Program Office making decisions for the NCS without authoritative oversight that reflects all areas of relevant expertise—is not likely to produce the optimal design for a study that needs to be implemented in a scientifically grounded and cost-effective manner.

Cost-effective and scientifically grounded operation of the NCS Main Study requires a broader and deeper base of scientific expertise for the Program Office; an authoritative multidisciplinary oversight structure to ensure that the decisions of the Program Office are appropriately vetted by all relevant experts; and provision for periodic comprehensive reviews of the study by an independent outside group. The panel’s recommendations to address these shortcomings are below.

CONCLUSIONS AND RECOMMENDATIONS

The NCS Main Study offers enormous potential, but it also presents a large number of conceptual, methodological, and administrative challenges. In addition, funding uncertainties make it difficult to plan a study of this magnitude and duration. Like the scientists associated with the study itself, we are eager for it to succeed. We present our recommendations in the hope that, as it goes forward, the NCS will achieve its intended objective to examine the effects of environmental influences on the health and development of American children.

The panel’s detailed conclusions can be found in Chapter 7.