7
Summary, Conclusions, and Recommendations
This chapter brings together the panel’s conclusions and recommendations from the preceding chapters for the design of the National Children’s Study (NCS) Main Study. We present them in the context of the overall conceptual framework proposed for the NCS: to serve as a “data platform,” with a focus on child health and development and the study design principles that flow from that framework. In the first section below, we present that framework and our findings on how the current design of the NCS meets the principles of the framework.
The chapter then summarizes the panel’s assessment of the nine issues it was specifically asked to address regarding the design for the NCS:
- the national probability sample’s overall sample size and design;
- the size of the supplemental convenience sample(s);
- the optimal use of sibling births;
- the relative size of the prenatal and birth strata in the probability sample;
- the use of hospitals and birthing centers as the primary sampling units;
- the use of health care providers to refer prospective participants;
- the proposed approach to define and characterize health disparities;
- the proposed approach to assess health and developmental phenotypes; and
- the proposed study visit schedule, with emphasis on more frequent data collection in pregnancy and early childhood.
These nine issues are discussed under two headings—the design, size, and composition of the Main Study probability sample (issues 1-6) and the content and visit schedule (issues 7-9). The final two sections of this chapter cover the costs of data collection and the overall leadership of the NCS.
It is difficult to judge the impact of the panel’s recommendations on the cost or timing of the Main Study. It may be that the recommendations from Chapters 2 through 5 will not cost more than the opaque process currently underway, especially given the commitment to pilot testing of the NCS Program Office. However, the panel received no hard evidence that there is a specific plan driving the work to launch the Main Study and hence no way to determine the impact of the panel’s recommendations on cost and timing. The recommendations to enhance the scientific expertise, oversight, and periodic outside review of the NCS may result in delays in implementation of the Main Study. However, the panel believes that the quality, utility and cost-effectiveness of the Main Study will ultimately be enhanced thereby.
CONCEPTUAL FRAMEWORK AND IMPLICATIONS FOR DESIGN
The panel endorses the overall conceptual framework proposed for the NCS, in which it is to function as a data collection platform with a focus on health and development (see Chapter 2). From this conception of the NCS as a data platform, the panel identified the following overarching principles that are important to reflect in the study design. It then considered the proposed NCS design in light of the principles.
• Principle: A scientific framework that encompasses current and anticipates future domains of high-priority scientific inquiry is needed to guide key study design elements, such as the target population, the sampling strategy, and the schedule and content of data collection.
o Finding: The framework for the NCS is not currently as well developed as required to meet the principle. In addition, the specific design feature to recruit equal numbers of cases into the sample prenatally and at birth does not fully reflect the growing consensus in the scientific literature on the importance of prenatal influences on child health and development.
• Principle: Scientifically robust exemplar hypotheses are needed to guide sample design and early-wave data collection, while decisions about data to be collected in later waves should leave room to take into account new hypotheses that emerge over the course of the study.
o Finding: The proposed exemplar hypotheses for the NCS are not currently as well developed as required to guide sample design and data collection in the early waves, nor is there a long-range plan spelling out lines of inquiry that must be pursued early on to
support development of new instruments for collection of information later in the project.
• Principle: A probability sample ensures that results generalize to the population from which the sample is drawn.
o Finding: The proposed design largely incorporates this principle, with the exception of its convenience samples.
• Principle: A stratified national sample in which children have an approximately equal chance of selection is required to support multiple goals. For the NCS, these goals include estimating relationships between exposures and health outcomes, analyzing health disparities, and attaining representation of children in key demographic and geographic subgroups roughly in proportion to their representation in the population.
o Finding: Although the proposed NCS design largely incorporates the study’s goals, its stratification plan needs to be carefully evaluated to ensure that it responds to research on the effects of exposures on health outcomes, as well as health disparities research priorities, as mandated in the Children’s Health Act of 2000.
o Finding: The panel was not provided with sufficient information to determine if further reductions in sample size (beyond those in the current design) that may become necessary due to costs would jeopardize the study’s analytical requirements.
• Principle: As large a sample size as possible within budget constraints is needed to provide statistical power for current and future scientific discoveries.
o Finding: The proposed NCS design largely reflects this principle, with a proposed national probability sample of 90,000.
o Finding: The proposed supplemental samples of 10,000 do not add sufficient value to the study to warrant their inclusion, and they detract from the national probability sample’s potential size and prenatal coverage.
• Principle: Scientific quality is enhanced by using the most valid and standardized data collection measures and methods that are feasible while maintaining sufficient flexibility to assess emerging domains of scientific inquiry.
o Finding: The panel was not provided sufficient information with which to evaluate this aspect of the NCS design.
• Principle: Scientific discovery is enhanced when the potential for future innovations in measurement is incorporated into the study.
o Finding: The plan for collection and storage of biological and environmental samples meets this principle and is appropriate to make them available for future investigations.
o Finding: Details about how potential innovations will be adopted were not provided to the panel.
• Principle: Discoveries related to health conditions are facilitated by a dynamic conception of health and disease, which calls for measuring health status, disease conditions, symptoms, and behaviors rather than just existing disease categories.
o Finding: The proposed design embraces this principle, but it does not provide sufficient details for the panel to assess whether the burden imposed on respondents by the additional questions on the conditions and symptoms is excessive.
o Finding: The process for specifying the measures to be collected appears to be large, unwieldy, and unsuitable for field implementation, and the operationalization of this process would benefit from timely and transparent scientific consultation.
• Principle: Discovery is facilitated if data are released as early and as completely as possible, with due regard for the protection of confidentiality.
o Finding: The proposed design endorses this principle but would profit from lessons provided by data release schedules and methods followed in other national studies to achieve timely release.
• Principle: Transdisciplinary discovery and statistical sophistication are enhanced when all relevant scientific expertise is integrated into the project management structure.
o Finding: The NCS design, as described, indicates a lack of sufficient scientific expertise, which is a major weakness of the study.
• Principle: The study design should be as cost effective and efficient as possible.
o Finding: The panel was not provided sufficient information with which to evaluate this aspect of the NCS design.
The above conceptual framework, resulting design principles, and panel findings, as well as the scientific literature on children’s health and development, lead to the panel’s recommendations regarding the overall design of the NCS.
RECOMMENDATION 2-1: The scientific framework for the National Children’s Study should be based on current understanding of the determinants of children’s health and development and an informed consideration of the likely future trajectory of scientific discovery. The paradigms of developmental biology and life-course epidemiology, coupled with findings from other social and behavioral science research on the prenatal and early life periods, should guide development of the design for the Main Study.
RECOMMENDATION 2-2: In order to facilitate scientific discovery during and after National Children’s Study data are gathered, the Main Study should use a national probability sample with the largest feasible sample size and an approximately equal probability of selection design, and it should recruit nearly all of the cohort as early in pregnancy as possible.
RECOMMENDATION 2-3: In order to facilitate scientific discovery during and after National Children’s Study (NCS) data are gathered, the Main Study should use valid and standardized data collection measures and methods, while maintaining flexibility to revise or develop new instruments. The NCS should also use state-of-the-art procedures to collect, archive, and provide access to biological and environmental specimens for future analyses.
RECOMMENDATION 2-4: The proposed strategy for the National Children’s Study Main Study to collect detailed data on children’s health status, conditions, symptoms, and behaviors should be followed to the extent possible, taking into account constraints of costs, operational feasibility, and the need to not overburden respondents.
The panel further offers two recommendations about the proposed supplemental samples for the NCS.
RECOMMENDATION 2-5: While the panel appreciates the possible scientific value of gathering preconception exposure information on 5,000 first-birth children as part of the National Children’s Study Main Study, this supplemental sample should be dropped because of high costs, the lack of any evidence of the value of such a sample, the lack of detailed plans for both selection and analysis, and potential limitations in the proposed data collection schedule.
In making this recommendation, the panel also took into consideration the loss of the opportunity to recruit more prenatal cases if the preconception group is retained.
RECOMMENDATION 2-6: The other supplemental convenience samples proposed for the National Children’s Study Main Study should be dropped from the design, including samples of children exposed to natural disasters or geographically defined environmental exposures, samples of additional members of disadvantaged groups, and samples of siblings born outside the 4-year birth window. The potential added value of the supplemental sample cases is less than the value of the additional cases
in the probability sample they would replace, specifically, the value of the additional prenatal cases in the probability sample.
Regarding the scientific consensus on the importance of beginning data collection during the prenatal period cited in Recommendation 2-1 (above), the panel notes that the Program Office did not provide a scientific rationale to support the proposed change to enroll one-half of the probability sample at birth instead of enrolling as many cases as possible prenatally, as in the original design. The Program Office suggested that resource constraints led to this design, but it did not provide cost estimates for its proposed design or for any alternative design models. The panel conducted its own cost analysis for recruitment and data collection under alternative designs (see Chapter 5 and Appendix B): it demonstrates that a 100 percent national probability sample, with as much prenatal recruitment as possible,1 has approximately the same field cost and yields nearly as many total cases as the proposed sample of 45 percent recruitment prenatally, 45 percent recruitment at birth, and 10 percent convenience or supplemental cases.
PROBABILITY SAMPLE DESIGN, SIZE, AND COMPOSITION
In the proposed design for the probability sample as described in materials provided to the panel, the first design decision is to define the study’s target population: it is to consist of all births in the United States (50 states and the District of Columbia) but excluding births to women who do not deliver in hospitals or birthing centers and births to women at hospitals with too few births to be included on the sampling frame. A second design decision involves the overall size of the probability sample.
CONCLUSION 3-1: The panel endorses the proposed target population of all births in the United States during a specified time period consisting of 4 full calendar years, as well as the proposed sample exclusions from this target population.
CONCLUSION 3-2: Because of the lack of explicit hypotheses in the study design, it is not possible for the panel to judge whether the proposed sample size is justified on the basis of the study’s objectives.
A large appropriately stratified national probability sample in which children have an approximately equal chance of selection would be one that
____________________
1As noted in Chapter 4, birth recruitment would be needed for the relatively small number of women who do not receive prenatal care or who receive prenatal care from a provider not on the provider list frame.
largely ensures that children in key demographic and geographic subgroups are represented roughly in proportion to their representation in the population. Such a sample could achieve the needed geographic dispersion within key demographic groups to facilitate analysis of health disparities. However, while stratification is a key mechanism for assuring this chance of selection, little information was available concerning what kind of stratification will be possible. In addition, other studies have found significant differential attrition among subgroups of particular interest to the assessment of health disparities: this is a key concern given the 21-year life of the NCS. Differential attrition will affect the ultimate composition of the sample and may reduce its value for assessing health disparities if not anticipated and addressed using oversampling.
CONCLUSION 3-3: By adopting an equal probability of selection design for the National Children’s Study, it is likely that the sample sizes for a number of subgroups of interest may be inadequate for some important types of analysis. These subgroups are likely to include minorities in the U.S. population who are known to be on the negative side of health disparities and to have higher attrition in longitudinal studies. However, the absence of explicit study hypotheses and objectives makes it difficult to identify these important population subgroups and their associated sample size requirements.
RECOMMENDATION 3-1: The National Children’s Study Main Study sample should be stratified by characteristics that will achieve variability in socioeconomic status within important population groups to support analysis of health disparities, as well as achieving variability in environmental exposures and geography to support analysis of relationships between exposures and health outcomes.
Insufficient information was provided to the panel to assess the coverage, feasibility, and other aspects of the first stage of the proposed design—using a list of hospitals as primary sampling units (PSUs)—because development of the sampling plan had not been completed as of February 2014. With the proposed design, the secondary sampling stage would be prenatal care providers from the sampled hospitals, split into two strata: the prenatal stratum and the birth stratum. The ultimate sampling stage would be the sampling of pregnant women from selected providers or the sampling of women just after birth at selected hospitals.
CONCLUSION 3-4: The panel has not been provided with sufficient detail on the planned hospital-based sample design and recruitment strategy to judge their merits and scientific validity or determine potential coverage bias and the availability of appropriate stratification variables.
CONCLUSION 3-5: The panel has not been provided with sufficient justification for moving to hospital-based primary sampling units from the sampling approach previously proposed by the National Children’s Study (NCS) for discussion at the 2013 NCS Workshop (see National Research Council and Institute of Medicine, 2013) and based on Vanguard Study pilot testing—namely, county-based primary sampling units with hospitals as secondary sampling units and providers as third-stage sampling units.
Assessment of the proposed sample design, when completed, should include comparisons with the previous design. Any comparison needs to include a cost-effectiveness analysis of the options and an assessment of the ability to ensure coverage and to control for such characteristics as race and ethnicity, socioeconomic status, age, and marital status to ensure the sample will support evaluation of health disparities.
Because a geographic-based first stage sample design has already been developed and would need only to be updated for population changes and because it appears feasible to sample prenatal care providers within geographic areas based on the Vanguard Study experience, the panel questions the decision to move to a hospital-based approach.
In its Vanguard Study, the NCS tested both provider-based recruitment and provider-based sampling approaches. While the panel was provided with limited information about what was done and how well it worked, the available information does indicate that provider-based sampling followed by recruitment of women is feasible. However, the panel is concerned about the possibility of high rates of nonparticipation, particularly by women in groups important for understanding health disparities: it will be important for the NCS to monitor participation rates by important categories, such as race and ethnicity, socioeconomic status, age, and marital status.
CONCLUSION 3-6: Assuming that participation in the National Children’s Study Main Study follows the patterns in the Vanguard Study, the cumulative response rate to birth for the prenatal stratum would be between 28 and 32 percent, and the rate to age 12 would be 22 to 25 percent—very small fractions of the eligible sample. The cumulative response rate to age 21 would be 18 to 21 percent. A thorough analysis of nonresponse bias is clearly indicated, and in any case will be required by the U.S. Office of Management and Budget.
CONCLUSION 3-7: The high rates of cumulative nonresponse expected in the National Children’s Study pose a severe risk for nonresponse bias that may not be mitigated by weighting adjustments, potentially making some study results invalid.
A range of other aspects of the current sampling plan were considered by the panel.
CONCLUSION 3-8: Enrolling siblings as members of the National Children’s Study sample provides many analytic advantages, most prominently the gathering of preconception exposure information for second- and higher-order births. The panel endorses current plans to recruit siblings born after the initially recruited child—but only within the 4-year recruitment interval associated with the original primary sampling unit for the target birth—and to continue to follow these children until age 21.
CONCLUSION 3-9: Weight adjustment and screening are viable options for accounting for the fact that subsequent siblings have more than one way to enter the sample. The panel was not provided sufficient information to recommend one over the other. In either case, detailed information on prior births to the mother will need to be collected.
CONCLUSION 3-10: As of February 2014, the currently proposed hospital-based sample design for the National Children’s Study had not been sufficiently developed or documented to support an evaluation.
CONCLUSION 3-11: The identification, sampling, and recruitment of women at the time of birth has not been sufficiently pilot tested, using a representative set of hospitals, to support any conclusion about this feature of the design.
RECOMMENDATION 3-2: A detailed plan for sampling, recruitment, and minimizing attrition bias for the National Children’s Study (NCS) Main Study should be fully developed and evaluated by sampling and survey experts independent from the NCS and approved by the proposed independent oversight committee before the study moves forward.
The proposed approach to assess health and developmental phenotypes, the proposed study visit schedule, and the development of content for the NCS reflect a conceptualization of health and development that is a substantial advance from the one that was reviewed in the previous evaluation by the National Research Council and Institute of Medicine (2008). The breadth of the conceptualization encompasses most of the issues affecting child health and development and provides many dimensions that could be linked to environmental exposures. In addition, the flexibility to use data to generate a variety of phenotypes, rather than focus on specific diagnoses, seems promising. How-
ever, needed details on the operationalization and effectiveness of these new approaches were not provided to the panel.
Exemplar hypotheses are a valuable way to guide sample design and early-wave data collection, while the data collected in later waves need to be able to adapt to hypotheses that emerge over the course of the study. Such hypotheses need to be carefully formulated in the context of the overall goals of a study.
CONCLUSION 4-1: A strategy of using a few exemplar hypotheses rather than stating a large list of hypotheses requires that the planners of the National Children’s Study ensure that the exemplar hypotheses are important and scientifically robust to guide the study design and data collection.
RECOMMENDATION 4-1: Prior to proceeding with the Main Study, the National Children’s Study (NCS) should develop scientifically well-grounded exemplar hypotheses that should be used to guide and evaluate decisions regarding the NCS design and data collection schedule and domains.
RECOMMENDATION 4-2: Because hypotheses will change over time, the National Children’s Study should implement a strong and public process to revise and develop new exemplar hypotheses to guide future study implementation, engaging with the extramural and intramural research communities.
In addition to exemplar hypothesis, other aspects of the NCS content have not been sufficiently detailed to be used for design decisions.
CONCLUSION 4-2: While using a dynamic health phenotype concept to plan the content of the National Children’s Study appears to be a promising strategy, the panel lacked sufficient information to judge whether the implementation of such an approach would be feasible given constraints imposed by respondent burden and overall study costs.
CONCLUSION 4-3: Many of the principles and concepts guiding development of the study design and the concept of having processes for developing future hypotheses and study content are consistent with the study platform framework for the National Children’s Study. However, it is not clear whether and how those principles and concepts can be effectively used to design the study content.
The schedule of data collection is a key design element for any study, and it is especially important for a large-scale longitudinal study such as the NCS. Other critical elements in any study design are enrollment and the protocols for
data collection. In addition, two key elements for the NCS are exposure to a wide range of hormonal, chemical, and other environmental factors and a focus on health disparities. In all these areas, the information provided to the panel lacked sufficient information or scientific justification for the current design for the NCS Main Study. More broadly, the overall processes for developing the details needed for careful design is unclear.
CONCLUSION 4-4: The panel agrees that more intensive data collection in the early years of the National Children’s Study is important, but the panel did not receive sufficient scientific justification to assess the merits of the specific data collection schedule.
RECOMMENDATION 4-3: The National Children’s Study Main Study should collect data during the prenatal period at multiple times for as many of the study participants as the budget will allow.
CONCLUSION 4-5: The strategy of the National Children’s Study (NCS) to enroll a substantial proportion of participants at the time of the child’s birth poses substantial logistical and operational challenges that have not been adequately tested in the NCS Vanguard Study.
RECOMMENDATION 4-4: Although the panel does not endorse the current proposal for a substantial birth enrollment stratum, if the National Children’s Study (NCS) Main Study retains such a stratum, the NCS should conduct a full pilot test of recruitment and data collection during the birth visit before implementation.
CONCLUSION 4-6: Exposure assessment, including collection of biological and environmental specimens during multiple study visits beginning during the prenatal period, is a critical component of the National Children’s Study in addressing the mandate of the Children’s Health Act of 2000 and fulfilling the study’s goal to serve as a platform for future scientific inquiry.
CONCLUSION 4-7: The processes for developing content for the National Children’s Study are complicated, and insufficient documentation has been provided to demonstrate that the processes will be effective.
RECOMMENDATION 4-5: The National Children’s Study Program Office should document and provide justification for development of the data collection schedule, content, and methods now and going forward. The documentation should be sufficient to guide use of the study data by future researchers.
RECOMMENDATION 4-6: The National Children’s Study Program Office should finalize the study visit data collection protocols that it intends to use for the Main Study (including questionnaires and other measurements), at least through age 1, and then pilot test the protocols before implementing the Main Study. The protocols and findings of the pilot tests should be peer reviewed and approved by the proposed independent oversight committee prior to initiating the Main Study.
CONCLUSION 4-8: Based on the information provided, the panel concludes that the National Children’s Study plan has paid insufficient attention to how health disparities should be taken into account in the development of the schedule visit and content of the Main Study.
RECOMMENDATION 4-7: The relevance to health disparities should be an explicit criterion for selecting the constructs that will be assessed as part of the National Children’s Study (NCS) Main Study, the measures that will be used to assess them, and the timing of the assessments. The NCS should obtain input from experts on health disparities in childhood as part of the documented process through which the measures for inclusion are selected and the measures should be approved by the proposed oversight committee.
A final data issue for any longitudinal study concerns the release of its data.
CONCLUSION 4-9: The panel endorses the general structure of the data distribution plans for the National Children’s Study (NCS), although it fails to understand the need for a 2-year lag between the availability of analytic data and their release to the research community. Subject to confidentiality concerns, timely and complete data access are vital to maximize the scientific value of the NCS and have been achieved by other federal government surveys, which ought to serve as models for the NCS.
RECOMMENDATION 4-8: The National Children’s Study should consider producing an “early release” version of the data from the Main Study that includes data collected in the early years of each wave’s data collection cycle and makes those data available to analysts under the terms of restricted access data centers.
For virtually all large-scale studies there are tradeoffs between everything one would like to do and what the budget can support. Those tradeoffs need to be made on the basis of realistic assumptions and careful analysis of the costs
of various aspects of the study. Because the NCS has not undertaken such an analysis and in order to fulfill the panel’s charge for a comprehensive review of the plans for the Main Study, the panel undertook an analysis of the NCS’s likely field costs. Although field costs are only part of overall study costs, they are the ones most likely to be affected by the sample design features considered by the panel.
CONCLUSION 5-1: The panel estimates that the field costs of the current plan for the National Children’s Study over its first 7 years will total $1.495 billion. This estimate does not include a number of other sizable contributors to overall study costs.
CONCLUSION 5-2: The major drivers of field costs for the National Children’s Study Main Study are sample size and number of interviews, with an in-person interview costing more than twice as much as a telephone interview. In contrast with contacting and gaining the cooperation of respondents, modest changes in interview length contribute minimally to cost.
CONCLUSION 5-3: For the same field costs and with the elimination of the 10,000 supplemental samples—5,000 nulliparous women and a 5,000 convenience sample—the National Children’s Study could afford to enroll a predominantly prenatal probability sample of 96,000 cases with no other changes to the proposed data collection protocol.
RECOMMENDATION 5-1: Given the goal for the National Children’s Study (NCS) to understand the links of environmental exposures to child health and development and its cost structure, if major reductions in the cost of the study need to be made, they should be reductions in sample size rather than exposure domains. Along with such a decision to reduce the sample size, the NCS should reconsider whether to oversample minorities in order to maintain the ability to evaluate health disparities with a reduced sample.
LEADERSHIP OF THE NATIONAL CHILDREN’S STUDY
Throughout the panel’s deliberations, the NCS Program Office staff were cooperative and responsive, providing timely responses to the panel’s many questions. Yet the panel repeatedly found that the answers to its questions were less than what would be needed to carry out its charge for a comprehensive review. The lack of detailed information to many basic questions about the design of the NCS and the lack of scientific justification for many of the design
decisions for the Main Study led the panel to deep concern about the overall leadership and management of the NCS.
CONCLUSION 6-1: Cost-effective and scientifically grounded operation of the National Children’s Study (NCS) Main Study requires a broader and deeper base of scientific expertise than currently exists in the NCS Program Office; an authoritative multidisciplinary oversight structure to ensure that the decisions of the Program Office are appropriately vetted in all relevant areas of expertise; and a provision for periodic comprehensive reviews of the study by an independent outside group.
RECOMMENDATION 6-1: The National Institute of Child Health and Human Development (NICHD) should consider and implement one or more means to enhance the scientific expertise of the National Children’s Study (NCS) Program Office by recruiting experts in relevant fields from within the National Institutes of Health, other federal agencies, and outside government. In addition, NICHD should consider contracting with experts outside of government to work part time on the NCS as a means to bolster the scientific expertise that is focused on the NCS.
RECOMMENDATION 6-2: The National Institutes of Health should strengthen the oversight and leadership of the National Children’s Study (NCS) by establishing an oversight scientific management structure to include a full range of relevant expertise, with review and approval authority for NCS design decisions.
RECOMMENDATION 6-3: The National Children’s Study (NCS) Program Office should establish a mechanism, such as a study section like those in the National Institutes of Health, or use a qualified independent organization to conduct periodic comprehensive outside scientific reviews of the design and operations of the NCS Main Study. To facilitate the work of such a committee and transparency for the study more generally, the NCS Program Office should promptly post on its Website all scientific studies conducted for the NCS.
