As authorized by the Children’s Health Act of 2000, the National Children’s Study (NCS) is planned to be a “longitudinal observational birth cohort study to evaluate the effects of chronic and intermittent exposures on child health and human development in U.S. children…” with “a cohort size of approximately 100,000 children and a national probability sample” (NICHD, 2013b, p. 3; see Box 1-1). The NCS will follow children from before birth or at birth to 21 years of age with great promise to contribute to the understanding of the impact of the environment, broadly defined, on the health and well-being of children (see Box 1-2).
The Eunice Kennedy Shriver National Institute of Child Health and Human Development (NICHD) Office of the Director has primary responsibility for planning and coordinating the NCS. This chapter provides the detailed charge to the panel, a brief history of the NCS, a summary of the proposed design of the NCS Main Study, the approach of the panel, and an overview of this report.
Section 1508 of the Consolidated and Further Continuing Appropriations Act, 2013, states:1
That the Director (of NIH) shall contract with the National Academy of Sciences within 60 days of enactment of this Act to appoint an expert Institute of Medicine/National Research Council (IOM/NRC) panel to
1Available: http://thomas.loc.gov/cgi-bin/query/z?c113:H.R.933.enr [March 2014].
conduct a comprehensive review and issue a report regarding proposed methodologies for the NCS Main Study, including whether such methodologies are likely to produce scientifically sound results that are generalizable to the United States population and appropriate subpopulations: Provided further, That no contracts shall be awarded for conducting the Main Study until at least 60 days after the IOM/NRC report has been available to the public.
The NICHD requested that the Committee on National Statistics of the National Research Council, in collaboration with the Board on Children, Youth,
Children’s Health Act of 2000 Public Law 106-310
SEC. 1004. Long-Term Child Development Study
a. PURPOSE. It is the purpose of this section to authorize the National Institute of Child Health and Human Development (NICHD) to conduct a national longitudinal study of environmental influences (including physical, chemical, biological, and psychosocial) on children’s health and development.
b. IN GENERAL.—The Director of the National Institute of Child Health and Human Development shall establish a consortium of representatives from appropriate Federal agencies (including the Centers for Disease Control and Prevention, the Environmental Protection Agency) to—
(1) plan, develop, and implement a prospective cohort study, from birth to adulthood, to evaluate the effects of both chronic and intermittent exposures on child health and human development; and
(2) investigate basic mechanisms of developmental disorders and environmental factors, both risk and protective, that influence health and developmental processes.
c. REQUIREMENT.—The study under subsection (b) shall—
(1) incorporate behavioral, emotional, educational, and contextual consequences to enable a complete assessment of the physical, chemical, biological, and psychosocial environmental influences on children’s well-being;
(2) gather data on environmental influences and outcomes on diverse populations of children, which may include the consideration of prenatal exposures; and
(3) consider health disparities among children, which may include the consideration of prenatal exposures.
NOTE: Congress subsequently passed Public Law 110-154 in 2007 to rename the institute as the Eunice Kennedy Shriver National Institute of Child Health and Human Development.
SOURCE: The full text of the law is available at http://www.gpo.gov/fdsys/pkg/PLAW-106publ310/html/PLAW-106publ310.htm [December 2013].
and Families of the National Research Council and the Institute of Medicine, assemble an ad hoc panel to conduct this congressionally mandated review of the design of the NCS Main Study.2 The NICHD charge to the panel is narrower than the “comprehensive review” language in the congressional act:
An ad hoc panel will conduct a congressionally mandated review of the design of the National Children’s Study (NCS) Main Study. The NCS is intended to follow a cohort of 100,000 children identified at or before birth through age 21 years. The study consists of a pilot or Vanguard Study, currently under way, and a Main Study, which is not to begin until after the expert panel has delivered its report. Based on Vanguard Study experience and other input, as of the meeting of the NCS Federal Advisory Committee on February 26, 2013, the Eunice Kennedy Shriver National Institute of Child Health and Human Development (NICHD) proposed a multi-stage probability design for 90 percent of the expected sample size of children for the Main Study. The first stage would select a national probability sample of hospitals and birthing centers from a national listing. A second stage would list prenatal care providers that “feed” patients for delivery at the selected hospital or birthing center, and a third stage would attempt to recruit pregnant women seeing these providers during their prenatal period. In addition, some women may need to be enrolled at the hospital at delivery. The remaining 10 percent of the Main Study sample is set aside for targeted populations to address additional questions of scientific interest, including subsequent births to women selected into the probability sample. The children enrolled in the Main Study will have a wide range of data collected about them, their parents, and their environment at specified intervals over the life of the study.
The charge to the expert panel is to review this proposed design with regard to the sampling frame, the sample design, the recruitment and retention process, and broad aspects of the interview schedule and data collection procedures to determine their scientific merit and, in particular, to determine the expected generalizability of results to a national population and population groups. The panel’s review will cover such aspects of the Main Study design as the national probability sample’s overall sample size and design; the use of hospitals and birthing centers as the primary sampling unit; relative size of the prenatal and birth strata in the probability sample; the size of the supplemental convenience sample; optimal use of sibling births; use of health care providers to refer prospective participants; proposed study visit schedule, with emphasis on more frequent data collection in pregnancy and early childhood; proposed approach to assess health and developmental phenotypes; and proposed approach to define and characterize health disparities.
2The NCS Main Study is distinguished from the pilot “Vanguard Study,” which is currently under way.
The panel will deliver a report with conclusions and recommendations at the end of the study that take cognizance of logistical and resource constraints as provided by NICHD.
The NCS has had a long gestation period, which in part reflects the challenges in addressing some of the issues identified in the panel’s charge. Fluctuations in its budget reflect some of this history: see Table 1-1.3 The table
3For a brief history of the NCS up to about 2008, see National Research Council and Institute of Medicine (2008).
The Promise of the National Children’s Study
As envisioned by Congress, the National Children’s Study (NCS) has great potential to address the major effects on and high costs of child morbidity due to potentially preventable conditions in the United States. This potential reflects the growing body of investigation showing that social, economic, and environmental exposures that are encountered prenatally and during early life and that accumulate throughout childhood have long-term consequences for social, emotional, physical, and cognitive well-being in adulthood. “Nearly all domains of later health experience, including cardiovascular disease, various cancers, respiratory disease, cognitive decline, and psychological impairment, have been associated with early-life exposures of one kind or another” (Lawlor et al., 2009, p. 897).
Research on early life factors has often stemmed from the exploration of data from historical birth cohorts (e.g., Barker et al., 1989a, 1989b) and from new kinds of analyses of existing birth cohorts (e.g., Hardy and Kuh 2009; Nybo Andersen et al., 2009). The results from those studies, in turn, have stimulated efforts to revitalize older pregnancy and birth cohorts by collecting new information and to establish new cohorts. Both the number and scope of published studies drawing data from birth cohort studies have increased over the past decade (Lawlor et al., 2009).
As Lawlor et al. (2009, p. 897) note, a number of recent national birth cohorts, including the Danish National Birth Cohort, the Norwegian Mother and Child Cohort, and the U.S. National Children’s Study: “have recruited, or plan to recruit, 100,000 parents and children, in order to determine the genetic and life-course influences on childhood health, development and/or common complex diseases in adulthood.” A more recent birth cohort study of a similar magnitude and scope is the Japan Environment and Children’s Study that started recruiting in 2011. The United Kingdom has a strong tradition of national birth cohort studies going back to the 1940s (see Pearson, 2011).* The new Life Study in the United Kingdom is the latest example: it is planned to include 80,000 births to be recruited in pregnancy
shows that no money was appropriated for implementation of the NCS until 2007. The timeline associated with the evolution of the NCS is summarized in Box 1-3 and is described in more detail below. During the early 2000s, the NICHD convened multiple workshops and commissioned several reports on NCS design options. By 2002, a federally chartered advisory group, the NCS Federal Advisory Committee, was established, and in 2003 the NICHD set up a NCS Program Office to plan and manage the NCS.
The decision that a nationally representative probability sample should be a core element of the study was reached in June 2004 by agreement of the NCS Program Office, the Advisory Committee, and an NCS Sampling Design Workshop Panel. Later that year the NCS, in collaboration with the National Center for Health Statistics, developed the NCS first-stage sample, comprising 110
and born in 2015. As noted above, these studies are similar in purpose and scope to the National Children’s Study. In many of these countries, unlike the situation in the United States, there are population registers, school records, administrative records for health care providers, and disease registers that facilitate sample design, data linkage, and analysis and thereby potentially decrease data collection costs. However, the scope and quality of the information available in administrative records is often less than what can be gathered in the survey-based approach of the NCS.
The NCS is planned to be the most comprehensive study of child health and development in the world. The study is intended to enroll a nationally representative sample of nearly 100,000 U.S. babies at or before birth and follow them through age 21, gathering detailed information about their environmental exposures, health conditions, and social, emotional, and cognitive development. As Guttmacher et al. (2013, p. 1873) note:
Other U.S.-based longitudinal studies provide information about children’s health, growth, or development. None of them, however, are as large as the NCS, entail collecting such detailed biologic and environmental data and samples, or include longitudinal phenotyping from before birth through the age of 21. To maximize data interoperability, the NCS is being coordinated with similar studies in other countries (including France, Japan, Britain, and Canada), but none of those studies will examine the same populations or environmental factors as the NCS.
The proposed NCS data collection program has the potential to take advantage of emerging innovations in the biological, social-psychological, and environmental sciences, maximizing the chances for scientific discoveries regarding child health and development. Successfully carried out, the NCS has the potential to enable U.S. scientific leadership in child health-related science for the next several decades.
TABLE 1-1 Interagency and Congressional Funding for the National Children’s Study (dollars in millions)
|Fiscal Year||Nominal Valuea||Present Value (2012)b|
|Funding Provided by NICHD During the Planning Phase|
|Funding Appropriated by Congress During the Implementation Phase|
|2013||Up to 165.0||Up to 162.6|
|2014||Up to 165.0||Not available|
aNominal value budget information for 2000 through 2013 taken from http://www.nationalchildrensstudy.gov/about/funding/Pages/interagencycongressionalfunding.aspx [April 2014]; 2014 budget information from 2015 appropriations language from http://officeofbudget.od.nih.gov/pdfs/FY15/Appropriation%20Language.pdf [April 2014].
bThe present value is computed using the price index of the Bureau of Economic Analysis for government nondefense consumption expenditures dated January 2, 2014; see http://www.bea.gov/national/nipaweb/SS_Data/Section3All_xls.xls [April 2014].
primary sampling units in 105 locations around the country. Beginning in 2005, the NCS Program Office awarded a number of contracts to design and implement the study. The initial NSC research plan was completed in June 2007, and the NICHD requested an external review of the plan, which is the subject of the National Research Council and Institute of Medicine (2008) publication.
The 2007 Study Design
The early architects of the NCS designed a sampling strategy to deliver a national probability sample of 100,000 children using a stratified area sample design that was expected to provide preconception environmental exposure information for about 25 percent of these children and early prenatal information for about 90 percent (see details in Chapter 3). The area sample of 110 primary sampling units (counties, groups of counties, or parts of counties) was designed so that each unit would experience a minimum of 2,000 births during a 4-year birth enrollment period, which would yield a target of 1,000 births for
Timeline of the National Children’s Study
|2000||National Children’s Study (NCS) authorized by Congress|
|2002||Federal Advisory Committee for NCS established|
|2003||NCS Program Office established within National Institute of Child Health and Human Development|
|2004||Nationally representative sample determined to be core element of study design|
|2005||First-stage sample (counties or groups of counties) drawn by National Center for Health Statistics|
1. Appropriation for NCS authorized by Congress
2. NCS research plan completed
|2008||National Research Council/Institute of Medicine review of NCS research plan published|
1. Field operations begun in 7 initial Vanguard locations
2. Change in NCS leadership
3. NCS decision made to expand pilot study to 30 additional locations using field contractors that had already been selected (but not activated) to implement the Main Study in those locations. All 37 locations become part of Vanguard Study
|2010||Alternative recruitment strategies tests begin fieldwork (10 locations each for 3 alternatives)|
1. Provider-based sampling pilot initiated in 3 additional locations, bringing the Vanguard study to a total of 40 locations to test 5 different recruitment strategies
2. Workshop on sampling strategies sponsored by NCS
3. Workshop on environmental exposures and measures sponsored by the Environmental Protection Agency, National Institute of Environmental Health Sciences, and NCS
1. National Research Council/Institute of Medicine Workshop on sample design conducted and summary issued
2. National Research Council/Institute of Medicine Panel on the Design of the National Children’s Study and Implications for the Generalizability of Results established
|2014||Publication of National Research Council/Institute of Medicine Panel’s report|
each unit over the birth period with each birth having an equal probability of selection. Secondary sampling units (called segments) were defined as aggregations of contiguous census blocks that, when summed together within each primary sampling unit, would yield the desired target for the primary sampling unit. The secondary sampling only occurred in the primary sampling units that were designated to participate in one of the recruitment pilot (or Vanguard) studies (see below). The “measure of size” for the segments was estimated from the number of births in the census blocks (geocoded to the mother’s place of residence) during a prior 5-year period. Segments were randomly sampled, although in most of the sampling units, contiguous segments were clustered into geographical strata with one segment sampled per stratum in order to enhance the likelihood that the random sample of segments would be diverse and geographically distributed across the primary sampling units.
The original design of the study was organized around hypotheses in 28 topic areas, referred to as core or meta-hypotheses. These hypotheses focused on 7 priority outcome areas:
- neurodevelopment and behavioral,
- child health and development,
- obesity and growth,
- injury, and
- reproductive development.
Each topic had several specific hypotheses with power calculations to demonstrate that the hypotheses could be addressed with the anticipated size of the NCS study population, and specific items of data collection were linked to these topics.
Implementation and Testing
As indicated above, the NCS Program Office was established in 2003 with a few staff and no direct funding. The initial direct appropriation for the NCS was to NICHD, and after 2009 the appropriation for the NCS was to the Office of the Director of the National Institutes of Health. According to the NICHD website, “the NICHD Office of the Director has primary responsibility for planning and coordinating the National Children’s Study.”4 The NCS Program Office directs the implementation of the day-to-day operation of the study and houses scientists, staff on detail from other organizations, program analysts, and
4Available: https://www.nichd.nih.gov/about/org/od/ncs/Pages/index.aspx [March 2014].
contractors.5 In 2014, the NCS Program Office had 18 staff members listed on the NCS website. Of them, 4 held medical degrees, 9 held advanced degrees equivalent to a Ph.D.,6 and the director had both an M.D. and a Ph.D. Six of the 18 were contracting officer representatives.
The NCS Program Office awarded contracts in 2005 to academic programs in seven of the 110 sampled primary sampling units to pilot test the household recruitment and data collection protocols. The NCS uses the term “Vanguard Study” to refer to all pilot-testing activities used to plan the NCS Main Study.7 The seven units used for the initial pilot testing were called the initial Vanguard locations, and the academic programs that were contracted to implement the field work were called the initial Vanguard centers. Field operations began in 2009 in the seven initial Vanguard locations8 using a household enumeration and screening strategy to identify eligible women for recruitment. The goal was to enroll approximately 1,750 pregnant women through these seven study locations (250 per location) during 12 months of data collection. The enrolled families were considered to be a “Vanguard cohort” that would be followed for 21 years and used for pilot testing future data collection protocols. The plan was for the NCS Main Study to begin after the 12 months of enrollment in the initial Vanguard locations, with the field work and participant enrollment beginning in three annual waves of approximately one-third of the 110 sampled primary sampling units. The initial Vanguard centers would participate in the Main Study as part of wave 1, while continuing to follow the Vanguard cohort participants.
In early summer 2009, the NCS Program Office indicated concern that the initial enrollment in the Vanguard Study was lower than anticipated. Although the response rates at various stages of recruitment were comparable to or higher than those for other large studies,9 the pilot study yielded fewer pregnant
5Available: http://www.nationalchildrensstudy.gov/about/organization/programoffice/Pages/default.aspx [March 2014]. Includes the names of 18 key staff members, many with a linkage to a brief bio.
6Fields of study of staff with a Ph.D. included cell biology, physiological molecular biology, environmental health science, sociology, epidemiology, and health policy and management with a concentration in bioethics. For the other three individuals with a Ph.D. or equivalent, no additional information was available.
7The purpose of the Vanguard Study is to pilot test data collection protocols and methods, participant follow-up, personnel training, etc. The panel understands and data users should expect that there will be some changes as aspects of the study design are tested and revised. For this reason and others, data collected in the Vanguard Study will often not be comparable with data collected in the Main Study.
8Recruitment began in two locations in February 2009 and in the remaining five locations in April 2009. The implementation was phased across segments (the secondary sampling units), so recruitment did not begin in some segments until November 2009.
9For details, see http://www.nationalchildrensstudy.gov/about/organization/advisorycommittee/Pages/ParkC-NCSAC-Jan2010-Recruitment-final.pdf [March 2014, p. 12].
women and births than had been anticipated.10 Based on this information, “the NCS Program Office projected that enrollment using a household recruitment approach could take much longer to complete and cost more than anticipated” (NICHD, 2013d, p.5). In the summer of 2009, the NCS leadership was changed (for details see Wadman, 2009, pp. 20-21). Among other issues related to this change, it had been reported that the revised projected cost of the study was significantly more than previously planned, even without accounting for new recruitment results. In the fall of 2009, the new NCS leadership decided to conduct additional recruitment pilot studies rather than to proceed immediately to the Main Study.
The field study centers contracted to implement the Main Study11 became the platform for testing the additional recruitment alternatives, and the funding that was appropriated for the Main Study instead supported the Vanguard Study. This phase of pilot testing was called the alternative recruitment pilot of the Vanguard Study. To test recruitment strategies, the NCS Program Office developed three alternative recruitment strategies and implemented each of them in 10 of the primary sampling units that had been sampled for the Main Study. The locations were selected to be geographically distributed throughout the country and have different population sizes and characteristics, although they were not formally considered to be a representative or probability subsample of the NCS Main Study primary sampling units. The three alternative recruitment strategies were an enhanced household enumeration strategy, a direct outreach recruitment strategy, and a provider-based recruitment strategy (NICHD, 2013d, p. 5). Although the study populations were recruited using different methods, the segment-based secondary sampling units were still used to define residential eligibility.
Evaluation of the three strategies indicated the provider-based recruitment to be the most efficient method of recruiting pregnant women. However, even that approach was inefficient in more populous locations because many different groups were providing prenatal care to women living in the same segment. Since sample eligibility was determined from the mother’s location of residence at the time of delivery, many different provider groups had to be engaged. Therefore, the NCS began planning for a new provider-based sampling pilot study that eliminated the geographic secondary sampling units and instead defined the prenatal care providers as secondary sampling units (NICHD, 2013d, p.6). At the end of 2013, with the provider-based sampling study still
10By the conclusion of the initial Vanguard pilot in April 2010, only about 800 pregnant women had been enrolled or about 45 percent of the goal. For details, see http://www.nationalchildrensstudy.gov/about/organization/advisorycommittee/Pages/A1_NCS_Alt_Recruitment_Schema.pdf [March 2014, p. 4].
11In 2012, the NCS Program Office contracted with four centers, called Regional Operating Centers, to continue the Vanguard Study and to follow families that were previously enrolled. The contracts with the original 40 Vanguard Study centers were subsequently discontinued.
under way, preliminary results indicated that it was an efficient secondary sampling strategy. A challenge that was identified with all of the recruitment methods based on a geographic primary sampling unit, including the provider-based sampling method, was that a large number of hospitals had to be engaged in order to collect data and biological specimens at the time of delivery.
The NCS Program Office sought input from the NCS advisory committee and stakeholders on an appropriate sampling design for the Main Study during 2012. The NCS Program Office prepared materials (NICHD, 2012) to provide background and discussion of alternative approaches including probability samples, convenience samples, and hybrid approaches. Sample design was also the topic of a workshop held in May 2012.12 In its budget request for 2013, prepared in May 2012,13 NCS stated
In assessing alternative sampling strategies, National Children’s Study and NIH leadership considered the overall scientific goals and which of these could be achieved with different strategies. Additional considerations were costs, based on Vanguard data, and the reality of flat or shrinking budgets for biomedical research. As a consequence, NIH now proposes that the Main Study sampling frame be based on provider location. One approach for developing such a sampling frame would be to use providers associated with specific health plans. Such an approach would have several advantages in terms of cost and feasibility, but would abandon the geographic based probability sample. Consequently, the enrolled population would no longer be a national probability sample but, instead, a well described cohort followed longitudinally.
First New Sample Design Proposed in 2013
In 2013, NICHD requested that the National Research Council and Institute of Medicine convene a workshop for a public discussion of a proposed new sampling plan for the NCS Main Study, which differed from the approach suggested in its budget request for 2013. In this plan, hospitals identified as secondary sampling units would be selected from a geographically selected primary sampling unit. Prenatal care providers whose patients delivered at selected hospitals (or birthing centers) would serve as third-stage sampling units. A sample of women who visit one of the sampled prenatal care providers and expect to deliver at one of the selected birthing centers or hospitals would be recruited.
One new feature of this plan was that the sample of 100,000 births would be split into a probability sample of 90,000 with two enrollment strata and an
12See summary of workshop discussion at http://www.nationalchildrensstudy.gov/research/workshops/Pages/potential-sampling-strategies-minutes-may-2012.pdf [May 2014].
unspecified convenience sample of 10,000. For the probability sample, 45,000 children would be sampled from prenatal care providers using the selected hospitals, and 45,000 children would be sampled at birth from the same selected hospitals. The NCS explained this allocation in practical terms: “NCS would need more time and resources to enroll pregnant women from prenatal care providers than from birth facilities” (NICHD, 2013b, p. 25). The allocation of the sample between the prenatal care providers and hospitals was discussed at the workshop; the workshop discussions are summarized in National Research Council and Institute of Medicine (2013).
The sampling plan provided to the panel by the NCS Program Office in August 2013 for this study (NICHD, 2013b) was yet further changed from previous plans. This plan relied on a list frame of hospitals from which to select primary sampling units, replacing the geographic-based area sample that was previously proposed. Prenatal providers whose patients deliver at the selected birthing hospitals and birthing centers would serve as secondary sampling units. The current proposal retains the 45-45-10 distribution of the sample among prenatal care providers, birthing hospitals, and the convenience sample, respectively.
At the time the panel was drafting this report, the specific methodologies of the sampling plan were still under development, and many important details were not available to the panel. For example, between August and October of 2013, the frame under consideration switched from a list of hospitals available from the American Hospital Association augmented with a list of birthing centers to a list of hospitals assembled from the 2010 State Inpatient Database maintained by the Agency for Healthcare Research and Quality of the U.S. Department of Health and Human Services.
Several NCS features other than the sampling frame changed since the 2008 review. The current proposal conceptualizes the study less as a vehicle for testing current hypotheses and more as a platform for future researchers to develop and investigate new hypotheses using the previously completed study instruments and archived biological and environmental specimens. Consequently, the current proposal no longer relies on specific hypotheses to define the content of the study. Instead, “the proposed plan was developed using several exemplar hypotheses so it is hypothesis informed but not hypothesis limited” (NICHD, 2013b, p. 34; Appendix 2). The NCS Program Office initially illustrated the impact of sample size by presenting tables using the same hypotheses and same computer code for calculating odds ratios that were used for the 2007 design (p. 36). Later, NICHD listed five exemplar hypotheses (see NICHD, 2013d). The concept of using exemplar hypotheses is discussed in Chapter 2, and the implementation of this strategy is discussed in Chapter 4.
The NCS is also now described as a platform for researchers covering a broad conception of health domains (NICHD, 2013b, p. 36). The NCS will not focus on classifying participants into predetermined disease categories, but instead will collect a set of primary observations and events to enable researchers to apply their own health criteria and form “cases” (p. 31).
Although the use of domains and primary observations, rather than predetermined disease categories, does not necessarily require a shift from the original disease-oriented outcomes of the NCS, the current proposed plan tends to deemphasize a focus on disease outcomes and gives greater emphasis to positive health and development domains. It provides a detailed conceptualization of health and development in seven domains:
- physical health,
- psychosocial health,
- neurodevelopmental health,
- health behaviors,
- social environment, and
- physical environment.
Each of these domains has subdomains. The NCS will use this conceptualization of health as multidimensional and dynamic to guide the selection of assessments.
With the current plan, NCS proposes to use phenotypes and profiles to describe participants. The NCS explains (NICHD, 2013b, p. 30):
The term phenotype is used for the observable characteristics including morphology, physiology, developmental stage, behavior and products of behavior…. The term profile is used for the larger concept of phenotype plus environmental context. A profile includes observable characteristics about the participant plus information about the environment such as air particle measures, noise level, family structure and dynamics, access to health care, etc.
Thus, at each assessment a participant will be assessed using a health phenotype framework, as well as through collection of environmental data and biospecimens. The NCS will also directly collect some health condition information on all participants.
The panel held three meetings to review the design of the NCS, two of which included 2-hour periods that were open to the public. During the open meetings, NCS staff were present, as were representatives of a group of NCS
Vanguard Study principal investigators, the American Academy of Pediatrics, the Children’s Environmental Health Network, and the Environmental Protection Agency’s Office of Children’s Health. In addition, the panel engaged two consultants, Randall Olsen of Ohio State University and Lisa Schwartz of Mathematica Policy Research, to provide the expert cost analysis included in this report.
In preparation for the panel’s work, the NCS Program Office provided a background document to the panel (NICHD, 2013b) and made available background documents it prepared for meetings of its NCS Federal Advisory Committee (NICHD, 2013a, 2013c, 2014b). The volume of background material provided to the panel was considerably less than that provided in the previous external review (see National Research Council and Institute of Medicine, 2008). The earlier study was asked to review a comprehensive research plan that was 745 pages long, including multiple appendixes. For this study, the NCS Program Office supplied a 56-page document that provided an overview of the conceptual framework and processes that would result in the final design of the NCS Main Study.
The panel prepared several sets of questions that were sent to NCS for response. Panel questions and NCS responses were in writing, sent and received by email between the NRC study director and the panel’s official point of contact at NCS or the director of NCS. The documents (NICHD, 2013d, 2013e, 2013f, 2013g, 2013h, and 2013j) contain the NCS responses to panel questions. The document (NICHD, 2013i), also provided at the panel’s request, is a summary of an interim report concerning the frame and sample of hospitals prepared by NCS consultants. The document (NICHD, 2014a) provides NCS comments on the panel’s cost analysis assumptions. Appendix A summarizes the information provided by the NCS Program Office and the questions asked by the panel for which information was not provided. The panel members also reviewed publicly available documents on the NCS Website and searched the peer-review literature for publications related to the NCS and other large national birth cohort studies.14
The panel concluded that to meet its charge of evaluating the “scientific merit” of the Main Study, the NCS Program Office would need to provide specific documentation on the sampling design, the sample frame, the data collection protocols, and the study instruments, rather than just describe the conceptual framework, strategies, and anticipated processes to design the Main Study. For example, one key to assessing expected generalizability of results would be detailed information about the proposed sampling plan based on a list of hospitals: the completeness of the list, the number of primary sampling
14The separate reference list for NICHD documents includes both those that are publicly available and those that were provided directly to the panel and are available in the Public Access File of the National Academy of Sciences.
units to be selected, stratification, and so on. The panel did not receive such information about the primary sampling units in the current design; the NCS Program Office indicated that the sampling plan was still being developed.
Similarly, the panel was asked to comment on the proposed study visit schedule; the proposed approach to assess health and developmental phenotypes; and the proposed approach to define and characterize health disparities. With these tasks specified, and given that the Vanguard Study had been in place for more than 4 years and that the Main Study was scheduled to begin soon,15 the panel expected to receive draft final protocols and data collection instruments to assess. However, the panel did not receive these protocols and instruments, apparently because they are still being developed (NICHD, 2013c, 2014b).
Although there was substantial interaction between the panel and the NCS Program Office leaders and staff, the panel could not determine whether all requested documentation was not provided primarily because the study design, protocols, and instruments are incomplete and could not be provided or because there was a difference in perspective between the panel and the NCS Program Office regarding the scope of the panel’s charge. Specifically, it was never clear whether the NCS Program Office thought that the panel could evaluate the scientific merit and generalizability of the Main Study by reviewing only the concepts, strategies, and proposed processes rather than reviewing actual documentation of the final proposed study design, protocols, and initial instruments. Regardless of the underlying reason, the documentation the panel requested was not provided: thus, the panel concluded that its review could not address the scientific merit or quality of these critical aspects of the NCS.
The remainder of the panel’s report includes six chapters, two reference lists, and three appendixes. Chapter 2 describes and comments on overarching features of the current NCS study design, including the conceptual framework and its evolution from a hypothesis-driven and disease-specific approach to a data collection platform with a focus on health. It also describes key issues that drive the sample design, including the utility of the proposed convenience samples. The chapter addresses the following key items the panel was asked to consider: the national probability sample’s overall sample size and design; the relative size of the prenatal and birth strata in the probability sample; the
15The NCS Program Office stated that they “anticipate that the Main Study will start a few months after the Institute of Medicine/National Research Council report is published.” See http://www.nationalchildrensstudy.gov/newsandevents/announcements/Pages/ncsreceivesFY2013appropriationapril2013.aspx [March 2014].
proposed approach to assess health and developmental phenotypes; and the size of the supplemental convenience sample.
Chapter 3 describes and evaluates the proposed sample design in light of the study’s history and resource constraints, with more detail on topics introduced in Chapter 2. It comments on statistical issues related to the survey design, as well as on the treatment of special populations. The chapter addresses the following key items the panel was asked to consider: the national probability sample’s overall sample size and design; the use of hospitals and birthing centers as the primary sampling unit; the relative size of the prenatal and birth strata in the probability sample; the optimal use of sibling births; and the use of health care providers to refer prospective participants.
Chapter 4 addresses the proposed study visit schedule, the content of collections of data and samples, and the approach to address health disparities. It also reviews the proposed approach to the dissemination of information once the study begins. The chapter comments on the following key items from the panel’s charge: the proposed study visit schedule, with emphasis on more frequent data collection in pregnancy and early childhood; and the proposed approach to define and characterize health disparities.
Chapter 5 summarizes the panel’s analysis of field costs associated with various alternative sample composition assumptions.
Chapter 6 comments on the study’s scientific leadership and need for independent study oversight and periodic outside review.
Chapter 7 summarizes the panel’s conclusions and recommendations.
The general reference list includes all documents cited in the report except those from NICHD. The NICHD reference list includes the documents from the agency cited in the report, both those that are publicly available and those that are available in the National Academy of Science’s Public Access File.
Appendix A documents the information requested by the panel and information provided by NCS. Appendix B describes the scope and assumptions underlying the panel’s field cost analysis in Chapter 5. Appendix C provides brief biographies of the panel members and staff.