This chapter addresses issues related to the leadership and oversight of the National Children’s Study (NCS). Although the charge to the panel from the National Institute of Child Health and Human Development (NICHD) does not list this topic explicitly, the panel determined that it is a critical issue underlying the centerpiece of the panel’s charge, which is the design of the Main Study and whether it “will produce scientifically sound results.”
The legislation authorizing the NCS assigns responsibility for the conduct of the study to the NICHD (Section 1004a). It also requires (Section 1004b) the director of NICHD to establish a consortium of representatives from appropriate federal agencies (set up as the Interagency Coordinating Committee), including the Centers for Disease Control and Prevention (CDC) and the U.S. Environmental Protection Agency (EPA), to “plan, develop, and implement a prospective cohort study….” While the authorizing language is ambiguous regarding whether NICHD or the federal agency consortium is “in charge” of the NCS, the NICHD Website states that the Office of the Director (which includes the NCS Program Office) has primary responsibility for planning and coordinating the NCS, and the NCS Website states that the NCS Program Office is responsible for the day-to-day operation of the NCS. The director of the NCS Program Office reports to the NICHD director and the director of the National Institutes of Health (NIH) on high-level design decisions. Beginning in 2010, with the decision that the previously approved sample design should be
changed, the NCS Program Office has been engaged in activities that are best described as study design and planning in addition to the ongoing operation of the Vanguard Study.
To provide input to the NCS, there is not only the Interagency Coordinating Committee, but also an NCS Federal Advisory Committee, which is chartered to advise the directors of NIH, NICHD, and the NCS Program Office. These two committees meet quarterly. For the Vanguard Study, the Program Office established a separate steering committee that included Program Office staff and field contractors, including university researchers. As the Vanguard Study was expanded, the steering committee was enlarged, and an executive steering committee was formed. The Program Office has also contracted for scientific expertise in particular areas, such as sample design and questionnaire content, through regular contract mechanisms, as well as through the establishment of various working group and committees. In recent years, it has provided publicly only summaries from this contracted work.
Problems with Decision Processes and Documentation
Despite these oversight and advisory structures, the panel finds that the processes by which study decisions are made and vetted are opaque. Moreover, decisions in a number of important instances are not well documented and do not appear to reflect the breadth or depth of relevant scientific expertise even though the NCS has engaged many well-qualified scientists since it began.
More broadly, it is the panel’s view that the current management structure, with the Program Office in overall control of decision making for the NCS, is not likely to produce the optimal design for a study that needs to be implemented in a scientifically grounded and cost-effective manner. The panel’s assessment on this issue is based on observations and experiences of panel members during its work and the preceding workshop (see National Research Council and Institute of Medicine, 2013); the panel’s evaluation of the quality and completeness of the documentation provided to it (see Appendix A); and the experiences of panel members with the work of the Program Office over the course of the 6 years since the publication of the prior review of the National Children’s Study (National Research Council and Institute of Medicine, 2008).
The panel found several troubling problems, including:
- proposed changes in sample designs with inadequate evidence on the quality of the proposed sample frames and lack of comparative analysis of the costs and benefits of the proposed changes;
- the absence of concrete plans for the supplemental sample components, the most costly of which would be the preconception sample of 5,000 nulliparous women;
- the apparent absence of a well-specified cost model for evaluating tradeoffs across alternative designs, an example of which is an inability to estimate the cost of crucial design components, such as prenatal as opposed to birth recruitment;
- insufficient detail concerning findings from the Vanguard Study1 and incomplete study design plans, even though the Vanguard Study has been pilot-testing study design options and data collection protocols since 2009;
- lack of scientific documentation and specifications to support the proposed sample design, content, and other key decisions for the Main Study; and
- absence of a realistic and well-specified schedule of stages and decision points that demonstrate how and when a final, well-documented design for the Main Study would be complete and field implementation could begin. At the time of implementation, instrumentation should be available particularly for the early phases up to 1 year of age.
Need for Added Expertise
The panel sincerely appreciates the positive tone of interactions with the Program Office staff and with their expressions of willingness to provide responses to the panel’s questions. However, as noted above and detailed in Appendix A, many of the responses were not fully responsive to the panel’s requests. Thus, the panel is concerned that the Program Office may not have sufficient in-house expertise in relevant scientific and survey research disciplines2 to enable it to function effectively in using the input it receives from contractors and advisory groups for design and operational decisions for the NCS Main Study.
Based on the material sent to the panel, the panel concludes that ongoing expertise is needed in sampling and weighting in longitudinal surveys, not only to draw the original sample, but also to deal with issues of attrition and missing data. In addition, expertise is needed in the design and management of
1The Program Office stated that” it plans to disseminate Vanguard data and is currently expecting to make its first release in late 2014” (NICHD 2013d, p. 86) and also stated that “many analyses are still pending, but early results have informed the sampling and recruitment design” (NICHD 2013g, p. 7). It also stated that the “Vanguard survey data have been released to 28 writing teams and 4 supplemental methodological survey projects for analysis for primary publication” (NICHD 2013f, p. 6).
2As noted in Chapter 1, the NCS Program Office has a staff of 18 who have advanced degrees in a variety of relevant disciplines. They are responsible for management and coordination of many contracts and advisory bodies. Although the current staff may have some of the areas of expertise recommended by the panel, the extent of their expertise was not evident in the documentation received by the panel. Moreover, the current staff are not likely to have time to conduct or stay abreast of research in relevant areas.
epidemiologic research in pregnant women and children in large, longitudinal studies. Expertise and experience are also needed in the planning and conduct of pilot studies, the management of multiple contracts, the development and maintenance of a sample collection system, and the timely production of analyzable datasets. Such expertise would require familiarity with the multiple disciplines required for this effort and how to deploy them efficiently. In particular, considering the NCS goal to assess the role of environmental factors on child health and development, the Program Office needs to have significant expertise in exposure assessment, expertise that encompasses a broad conceptualization of “environment.”
Expertise in measurement theory and development is likely also needed. The NCS will face the need to have succinct, valid, and reliable measures throughout its course. For some issues, the measures can be developed with existing instruments; for others, as suggested by the Program Office, it may require adapting or shortening such instruments and establishing their validity and reliability. The panel also believes that the Program Office needs greater familiarity with the constructs and measures to help in guiding the discussion of the various working groups and committees to obtain concrete, pragmatic advice for implementation of the NCS. The Program Office needs to focus those discussions to obtain timely input and measurement suggestions. In contrast, the panel found that the working groups, while very thoughtful, have produced reports and documents that are relatively remote from near-term issues of implementation of the NCS.
Needed Oversight and Outside Review
In addition to the need for added depth and breadth of expertise in the NCS Program Office, the panel believes that changes are needed to strengthen the oversight and review structures for the NCS. Specifically, because of the complexity, cost, and scientific importance of the NCS and the wide range of expertise it requires in such areas as sampling, survey methods, environmental exposure measurement, health disparities and health phenotypes, the panel believes that an authoritative, multidisciplinary oversight structure for the NCS is required to ensure that the decisions of the Program Office are appropriately vetted in all relevant areas. In addition, the panel deems it critical that regular comprehensive reviews by an independent outside group, with appropriate multidisciplinary expertise, be conducted roughly every 3 years in order to ensure that the Main Study operates cost effectively to maximize the scientific utility of the information. At present, there is no known provision for such reviews.
CONCLUSION 6-1: Cost-effective and scientifically grounded operation of the National Children’s Study (NCS) Main Study requires a broader
and deeper base of scientific expertise than currently exists in the NCS Program Office; an authoritative multidisciplinary oversight structure to ensure that the decisions of the Program Office are appropriately vetted in all relevant areas of expertise; and a provision for periodic comprehensive reviews of the study by an independent outside group.
The panel is not the appropriate body to make specific recommendations to address the three areas of concern it has identified regarding the scientific leadership and decision making structure for the NCS. Instead, we offer below some approaches for consideration in each area—scientific expertise, oversight, and periodic outside review.
Methodological and Substantive Expertise
To develop a broader and deeper base of scientific expertise, the panel urges the NCS Program Office to consider ways and means to bring in outside scientists with relevant backgrounds and experience to supplement its in-house staff. The NCS has established the National Children’s Study Scholars Program,3 but it is limited to federal employees, and the positions are not salaried and so have to be supported by the scientist’s sponsoring federal agency. To engage the expertise it needs, the NCS could provide support for appointments in the Program Office of up to several years through such mechanisms as interagency personnel agreements (IPAs) for scientists with needed expertise, not only from other federal agencies, but also from academic institutions and private research organizations.
Requiring a commitment to spend full time at the NCS for an extended period of time could be a deterrent to attracting the very best research scientists from academia. To overcome this barrier, the Program Office could also establish a competitive contract mechanism to select and fund outstanding academic research scientists to collaborate part time (e.g., 20-40 percent) on the NCS to provide expertise and possibly function as co-principal investigators for project domains. Such contracts, to be most effective, are best handled with individuals separately from other contracts for such activities as data collection.
Other approaches could also be considered to strengthen the base of scientific expertise that is available to the NCS. For example, it could be useful to increase and formalize collaborations, at the levels of both programs and individual scientists, between the Program Office and the intramural research programs of the NICHD, the National Institute of Environmental and Health
3For details, see http://www.nationalchildrensstudy.gov/opportunities/Pages/scholars.aspx [April 2014].
Sciences (NIEHS) and other institutes of the NIH; with CDC, including the National Center for Health Statistics (NCHS) and the National Center for Environmental Health; and with EPA.
RECOMMENDATION 6-1: The National Institute of Child Health and Human Development (NICHD) should consider and implement one or more approaches to enhance the scientific expertise of the National Children’s Study (NCS) Program Office by recruiting experts in relevant fields from within the National Institute of Health, other federal agencies, and outside government. In addition, NICHD should consider contracting with experts outside of government to work part time on the NCS as a means to bolster the scientific expertise that is focused on the NCS.
To improve the oversight structure for the NCS and to ensure that the study design and implementation protocols are as scientifically based and cost-effective as possible, the panel suggests a model for study oversight, which builds on the language in the Children’s Health Act of 2000.
Under this model, the NIH would establish a scientific management group,4 the members of which would have the authority to review and approve the study hypotheses, design, methods, measures, cost, and instruments on an ongoing basis. Such an interagency scientific management group would include senior-level scientists from the NICHD, NIEHS, EPA, and CDC, selected by their agencies, who would be assigned to contribute substantial effort in a review capacity for the NCS, at least until the Main Study is fully launched. The management group could also consider selecting some nongovernmental experts to serve on the committee. As the Main Study progresses, the composition of the committee and level of effort could change in light of different needs for expertise to address different developmental milestones in children’s health and development.
This group would make decisions about the overall design and operation of the NCS. For greater effectiveness, responsibility for specific aspects could be assigned to the participants from particular agencies within an overall agreed to framework: for example, sample design and overseeing field operations to NCHS, environmental exposure measures to NIEHS and EPA, and child development to NICHD. This structure would be responsive to Section 1004b of the legislation and would enhance scientific legitimacy of the NCS.
4A “scientific management group” is equivalent to the concept of an independent oversight committee that is included in some of the recommendations earlier in this report. An interagency scientific management group is one example of how an independent oversight committee could be established.
This administrative structure could be implemented by changing the current Interagency Coordinating Committee from being an advisory, coordinating body to a study oversight committee with approval authority subject to final review by the NIH director. Alternatively, a scientific management group could be established in addition to the Interagency Coordinating Committee with higher level agency leaders who would focus on the oversight function. Such a committee should have unrestricted access to NCS processes and data that are the basis for study design decisions by the NCS Program Office.
Either of the approaches to establishing an appropriate oversight structure just outlined would be responsive to the Children’s Health Act of 2000. In addition, the NCS Federal Advisory Committee would continue in operation, advising the scientific management group and expecting substantive responses to its recommendations.
RECOMMENDATION 6-2: The National Institutes of Health should strengthen the oversight and leadership of the National Children’s Study (NCS) by establishing an oversight scientific management structure to include a full range of relevant expertise, with review and approval authority for NCS design and major management decisions.
Periodic Outside Review
For such a complex, long-running, and costly enterprise as the NCS Main Study, it is important to have periodic formal outside scientific review in addition to strengthening the scientific expertise devoted to the study, providing for authoritative ongoing oversight, and continuing the standing federal advisory committee. There are several possible mechanisms for accomplishing such reviews, which should generally be conducted every few years and involve not only review of documents, but also meetings with program and contractor staff, field visits to observe data collection and processing, and other activities as necessary to ensure a comprehensive review.
Models for such outside review include the use of a specially appointed visiting committee, such as the National Science Foundation convenes every few years for reviews of each of its major programs, or the appointment of a study section as used by NIH, or the commissioning of a qualified independent organization to conduct the review. The review group, however constituted, would report to the oversight scientific management group and the NIH director and its reports should be publicly released.
RECOMMENDATION 6-3: The National Children’s Study (NCS) Program Office should establish a mechanism, such as a study section like those in the National Institutes of Health or qualified independent organization, to conduct periodic comprehensive outside scientific reviews
of the design and operations of the Main Study. To facilitate the work of such a committee and transparency for the study more generally, the NCS Program Office should promptly post on its Website all scientific studies conducted for the NCS.
The panel notes that large-scale, long-running longitudinal studies in the United States are run either by statistical agencies with extensive survey and statistical experience (e.g., the National Longitudinal Surveys of the Bureau of Labor Statistics, which contracts for the data collection) or by university-based centers that have one or more principal investigators who bring deep scientific and survey design knowledge to the project with funding from a federal grant or cooperative agreement (e.g., the Health and Retirement Study and the Panel Study of Income Dynamics at the University of Michigan and the National Longitudinal Study of Adolescent Health at the University of North Carolina). We also note that three large British birth cohort studies are housed in the same university-based research institute. We know of no example in which a funding agency, such as NICHD, not only provides the funding for a survey but also controls the design decisions and overall operations with contracts for data collection and other functions. The NCS Program Office obtains advice from researchers and relevant federal agencies, but that advice is not binding for design or operational decisions.
In a review of six longitudinal birth cohort studies, Golding (2009) commented on management structures.5 Four of those reviewed were housed in a university, and two were managed by government agencies. Golding (2009, p. 27) suggests that “basing the director and the study in a university is likely to be the best way to ensure the study’s scientific and ethical integrity.” However, other national birth cohort studies not mentioned by Golding also are managed by government agencies. The panel did not reach a consensus on whether to recommend one model or the other, but the majority of panel members generally believe that both management models could be effective. The panel judges that the current management structure is not functioning effectively and encourages the NIH director to carefully reconsider all options before settling on a structure that is most likely to result in a more scientifically based and cost-effective NCS main study.
5One of the studies reviewed was the NCS; two other studies, the Danish National Birth Cohort and the Norwegian Mother and Child Cohort Study, enrolled 100,000 births by 2002 and 2008, respectively.