Appendix A
Communications Between the Panel and the NCS Program Office
During the course of its work, the National Children’s Study (NCS) Program Office sent several background documents to the panel, and the panel sent to the Program Office several sets of questions about the current proposed design of the NCS. All questions and responses were sent and received by email between the panel’s study director and the staff person designated as the point of contact or the director of the Program Office. This appendix details the timing of those panel communications with the Program Office. Table A-1, which provides a summary of the information flows, also lists information requested by the panel that was not provided by the Program Office.
August 2013 The first set of questions was sent to the Program Office on August 1, before the first panel meeting. These questions focused on the sample design. The primary purpose of this set of questions was to facilitate discussion during the panel’s first meeting and to inform the Program Office that the panel was interested in seeing detailed descriptions of proposed methodologies and data used in justifying decisions. On August 8, the Program Office provided the panel with its briefing document (NICHD, 2013b). On August 16, during the part of the panel’s meeting that was open to the public, the panel questioned the cost analysis used by NCS to justify its decision about having one-half of the probability sample recruited at birth and also asked when we would receive details about the hospital frame and sample.
September 2013 After the initial public meeting with Program Office staff, the panel prepared 11 sets of questions, 9 of which focused on the 9 topics
in the charge to the panel. The other two sets of questions concerned issues the panel deemed important: NCS plans to disseminate data and the NCS’s planned use of interviewing modules that subsets of respondents would receive. All 11 sets of questions were sent to the Program Office by September 9. On September 30, the panel recognized gaps in the questions that had been sent on September 9 and sent another set of questions—11 in all—to NCS by email. On September 30, just prior to the government shutdown and at the urging of the panel, the Program Office provided draft responses to the 11 question sets sent to the Program Office on September 9. Given the volume and detailed nature of the questions, the Program Office stated that its responses should be considered preliminary, were provided so that the panel could continue its work during the shutdown, and might be revised in the future.
October 2013 The NCS Program Office delivered a preliminary version of responses to the questions sent on September 30, and a final version of its responses to the panel’s original 11 sets of questions on October 23 (NICHD, 2013d). This document also included summary information about NCS plans. On October 30 the panel sent its fourth set of questions to NCS. These questions focused on information not previously provided by NCS, in addition to asking for clarification and detail on a number of points made in the responses that the panel had already received.
November 2013 On November 9, the panel received final responses from NCS (NICHD, 2013e) to the second set of 11 questions, which had been sent to NCS on September 30. Responses to most of the October 30 questions were received in three parts on November 8 (NICHD, 2013f), November 26 (NICHD, 2013g), and November 27 (a draft of responses to most of the remaining questions.) In a number of instances responses were either still not complete, or indicated a lack of understanding of the question. For example, in response to “the panel would like to see details about how differing four year recruitment periods will be reconciled during the estimation process,” NCS described the operational need to roll out the sample over time.
On November 19 the panel’s cost consultants met with NCS leadership to discuss assumptions for the cost analysis. On November 23 the panel’s study director reminded NCS of the panel’s need for cost information, stating that cost data are critical to the assessment of the NCS proposed design of the Main Study because cost is cited by NCS as the rationale for key decisions.
In response to cost questions, NCS stated that Program Office staff had met with the panel’s cost consultants and had given them all the required information.
December 2013 The panel’s cost consultants did not agree with the Program Office’s assessment and formulated a set of cost questions that were communicated to the Program Office on December 6. On December 17, the panel received responses to some but not all of those cost questions. After pointing out that the responses were incomplete, a more complete set of responses was received on December 20 (NICHD, 2013j). However, even in this document, for example, the question “What are the cost metrics for these pilot efforts?” (p. 2) was not answered by the Program Office.
On December 17, NCS provided the panel with a summary of the preliminary report concerning the design for hospitals as primary sampling units (NICHD, 2013i) but not the report itself. NCS also provided the final version of responses to the last set of questions, which had been sent on October 30 (NICHD, 2013h).
January 2014 At the invitation of the panel, NCS provided comments on the panel’s cost analysis (NICHD, 2014a).
Other Information The Program Office provided the panel with background documents prepared for the three meetings of the NCS Advisory Committee that were held in the time frame of the panel’s activities (NICHD 2013a, 2013c, and 2014b). In searching for additional information about the NCS on the study’s website, the panel found detailed research results only in white papers dated 2004 and 2005, which the website states are not a current, accurate representation of the plans for the NCS and are provided for historical purposes only.
| Topic | Information Provided and Source | Information Not Provided | ||
| Requested Information | Source | Explanation of Response | ||
| Sample Size and Design | General description: NICHD (2013b, pp. 16-17, 20-28, 40-41) and NICHD (2013d, pp. 11-16, 42-53) | (1) Number of hospitals, providers, women to be sampled at each phase; (2) costs and variances | NICHD (2013d, p. 51) and NICHD (2013h, p. 1) | Response said requested information not available because sample design still under development. |
| Useful detail about recruitment process and success at various stages in Vanguard Study, including assessment of sample representativeness: NICHD (2013b, pp. 26-28), NICHD (2013d, pp. 5-10), NICHD (2013e, pp. 1-3), and NICHD (2013f, pp. 3-4) | Rationale for moving from area-based sample to hospital-based sample | NICHD (2013d, pp. 52, 57) and NICHD (2013g, p. 4) | First response said hospital approach will minimize bias and save resources. Second response cited greater operational flexibility, easier to find a replacement should a sampled hospital decline. No additional details provided. | |
| Details of planned reconciliation of data when each hospital has a different 4-year birth period | NICHD (2013d, p. 50) and NICHD (2013h, p. 3) | First response said NCS will do reconciliation. Second response had no reference to reconciliation. No additional details provided. | ||
| (1) Power calculations for specific exposures and prevalence to justify sample size; (2) precision of key estimates needed to determine sample size | NICHD (2013d, pp. 43-46, 50); for (1) reformulated in NICHD (2013h, p. 2); for (2), reformulated in NICHD (2013g, p. 2) | (1) Response said any proposed exposure or outcome with defined prevalence fits within the framework used by NCS, but calculations not available because PSU sample design not complete. Second response stated that this sample size (50,000 births) “is adequate for hypotheses that we and others have proposed to make exposure outcome associations.” (2) Response said NCS should be general purpose. | ||
| Hospitals as PSUs | General description: NICHD (2013b, pp. 22-25) and NICHD (2013d, pp. 11-13, 16, 54-58) | Details of proposed sample of hospitals including quality of the frame. | NICHD (2013d, pp. 54-58), NICHD (2013f, p. 1), NICHD (2013h, p.1) and NICHD (2013j, p. 3) | Response said details not available because sample design under development. |
| Brief summary of NCS consultants’ initial report: NICHD (2013i, pp. 1-2) | ||||
| Summary of Vanguard Study experience recruiting women at hospitals: NICHD (2013f, pp. 2-3) and NICHD (2013g, p. 2) | ||||
| Topic | Information Provided and Source | Information Not Provided | ||
| Requested Information | Source | Explanation of Response | ||
| Prenatal Providers as SSUs | General description: NICHD (2013b, pp. 20-24) and NICHD (2013d, pp. 13-15, 70-74) | Detail about quality of list frame of providers | NICHD (2013d, pp. 70-71) and NICHD (2013h, p. 1) | First response described construction of list. Quality not mentioned. Second response provided more detail on construction of list. |
| Summary of Vanguard Study experience sampling and recruiting both providers and women at providers: NICHD (2013f, pp. 3-4) and NICHD (2013g, p. 1) | ||||
| Size and Cost of Birth and Prenatal Strata | General description: NICHD (2013b, pp. 24-25), NICHD (2013d, pp. 12-16, 59-63) | Cost analysis that was used to justify large birth stratum | NICHD (2013d, pp. 59-60) | Response said a subsequent review of submitted cost data is suggesting a need to reevaluate this cost estimate; no additional details provided. |
| Some cost information provided verbally to panel’s cost consultants and in NICHD (2013j) | Detailed cost information, including cost metrics for Vanguard Study | NICHD (2013d, pp. 59-63), NICHD (2013h, pp. 1-3), and NICHD (2013j, p. 2) | Many details not provided. | |
| Reviewed and commented on panel’s cost appendix: NICHD (2014a) | ||||
| Convenience Samples | General description: NICHD (2013b, pp. 25-26) and NICHD (2013d, pp. 16-18, 64-66) | Detail about composition of 5,000-child convenience samples (other than nulliparous women) | NICHD (2013d, pp. 64-66) and NICHD (2013h, p. 3) | First response said convenience sample composition not defined until sample design complete and PSUs selected. Second response said unique exposures sample only one of multiple options. |
| Sample of nulliparous women: NICHD (2013f, p. 5) and NICHD (2013g, pp. 10-11) | ||||
| Sibling Births | General description: NICHD (2013b, p. 17) and NICHD (2013d, pp. 19-24, 67-69) | Methodology for computing selection probabilities of sibling births | NICHD (2013d, p. 68) and NICHD (2013f, p. 4) | Response said siblings included with certainty. No additional details provided. |
| Study Visit Schedule | General description: NICHD (2013b, pp. 32-33) and NICHD (2013d, pp. 35-41, 75-77) | Rationale behind selection of specific observation points and data to be collected | NICHD 2013d, p. 75) and NICHD (2013h, p. 2) | Response cited need for frequent measurements during periods of rapid change; no additional details provided. |
| Topic | Information Provided and Source | Information Not Provided | ||
| Requested Information | Source | Explanation of Response | ||
| Approach to Measurement | Health phenotype concept: NICHD (2013b, pp. 29-32) and NICHD (2013d, pp. 26-34) | (1) Information that spans early waves of interviewing, specifically about content, timing, and sample allocation details; (2) process used to select domains and subdomains and related measures (Vanguard study has data collection experience up to 12 months of age); (3) operationalization of this process; (4) measures used and how selected | NICHD (2013d, pp. 46-48 and 87-99); reformulated in NICHD (2013g, pp. 7-10) | (1) Response referred to general background section of report. Appendix states “specified visits include what is currently being tested/planned in the Vanguard Study and are possible measures to be collected in the Main Study.” Table shows only categories of measures (environmental sample, biospecimen, questionnaire.) (2) Response stated Vanguard Study not designed to examine prevalence of exposures or outcomes or examine exposure-outcome relationships. (3) Response stated that within each domain and subdomain measures were selected to address specific topics. (4) Measures not provided. |
| General description of content and content development process: NICHD (2013b, pp. 34-35, 42-47) and NICHD (2013d, pp. 25-32, 34-35, 89-99) | ||||
| Exposure assessment: (1) summary of earlier exposure assessment workshops:1 NICHD (2013b, pp. 48-51); (2) summary of Vanguard Study research | (1) Clarification of information collected for birth stratum, and, if retrospective assessment is an objective, provide specific documentation | NICHD (2013d, pp. 43-44, 48, 76), and for (1), NICHD (2013g, p. 10) | (1) Response states NCS is exploring the option of self-collected environmental samples for the birth stratum, and that the purpose of a postbirth visit is to collect information about persistent environmental. | |
| in environmental collections: NICHD (2013g, pp. 8-10) | of validity of methods for the full spectrum of environmental factors. (2) Besides environmental exposures (dust, etc.) what other measures of the family environment are considered important. (3) Strategy for nonpersistent exposures | exposures. No additional information about retrospective assessment provided. (2) Response states that NCS will capture a full spectrum of environmental and social factors. Details not provided. (3) Nonpersistent exposures would rely on probability and timing to detect. | ||
| Health Disparities | General description: NICHD (2013b, pp. 11-12, 19-20, 26-28) and NICHD (2013d, pp. 78-81) | Panel requests NCS reaction to their judgment that oversampling within the context of the main study is the way to best address health disparities | NICHD (2013d, pp. 78-81), and NICHD (2013g, p. 11) | Response states that choosing groups to oversample is challenging and oversampling reduces the precision of estimates of the overall population. |
| Useful detail about recruitment by demographic category from Vanguard Study: NICHD (2013d, pp. 5-10) and NICHD (2013e, pp. 1-3). |
| Topic | Information Provided and Source | Information Not Provided | ||
| Requested Information | Source | Explanation of Response | ||
| Issues Related to Scientific Merit and Generalizability | NCS staff qualifications | NICHD (2013e, p. 5) | No response provided. | |
| Schedule for launching main study | NICHD (2014e, p. 6) | Main study field work anticipated to begin in fiscal year 2015; no additional details provided. | ||
NOTES: PSUs = primary sampling units; SSUs = secondary sampling units.
1 Workshop sponsored by National Children’s Study, U.S. Environmental Protection Agency, and National Institute of Environmental Health Sciences.
