The panel sought to ground its recommendations for the design of the National Children’s Study (NCS) in an understanding of the nature of likely field costs and by calculating the field costs of alternative sample designs and field strategies over the period of respondent recruitment. Although this topic was not explicitly included in the charge to the panel from the National Institute of Child Health and Human Development (NICHD), the panel judged that an evaluation of the scientific merits of the proposed methodologies of the NCS Main Study required an understanding of the relative benefits and costs of design options. It is important to note that the cost figures we have developed are based only on major costs of data and specimen collection and thus understate, perhaps substantially, total project costs over the period we considered. (Examples of the omitted cost categories are provided in Appendix B.)
In the course of developing our estimates, we provided the NCS Program Office with a list of the major assumptions used in our cost model (see Appendix B for a more complete description of our assumptions). The Program Office agreed with substantially all of our cost assumptions, and we accepted most of the changes to our assumptions and estimates (either up or down) suggested by Program Office staff. One noteworthy exception concerned what we believe is
the Program Office’s likely underestimation of the costs of persuading hospitals and providers to cooperate with the study.1
The previous chapters have identified three design issues with important cost implications: (1) the fraction of the main probability sample that would be recruited at the offices of prenatal providers affiliated with targeted hospitals (rather than being recruited when they presented for delivery); (2) the inclusion of a supplemental sample of nulliparous women to be recruited prior to the conception of their first births; and (3) the number of in-home and telephone interviews associated with each study participant. We modeled a total of eight different designs involving these dimensions.
In estimating field costs associated with alternative designs, our model only includes field operation tasks for which these issues would have significant cost effects. As detailed in Appendix B, it does not attempt to account for the differential costs associated with overall management at the program or contractor level, which include purchasing or building the sample frame, managing the sample and preparing sample weights, programming questionnaires, tracking and storing environmental or biological specimens, data entry, verification, transmission and management, archiving, documentation, and dissemination. In addition, because the NCS plan proposes that the sample of births be drawn over 4 years for each of four groups of primary sampling units, with the start-up of each birth window for each of these groups spread over 4 consecutive years, the enrollment phase will extend for 7 years. Thus, we carried out our cost model for 7 years, since the cost consequences of most of the design issues we investigate will have occurred during that period.
Given the magnitude of the estimated 7-year total field costs of all of the scenarios we investigated—$1.3 to $1.6 billion—we begin by noting the most sizable components of field costs. We present these cost drivers in decreasing order of significance.
1. Sample Size: Pursuing a cohort of 100,000 is the primary driver of field costs. Even modest changes in this figure will generate substantial
____________________
1As detailed in Appendix B, the current NCS experience is based on the Vanguard Study. We judged that the involvement of academic medical centers in the Vanguard Study brought considerable name recognition to the recruitment effort. The kinds of national field organizations that are proposed for the Main Study are less likely to bring the same sort of name recognition and are likely to need more time, effort, and expense to secure the cooperation of hospitals and providers, and, possibly, even respondents. Although it is not possible to estimate these additional costs with much precision, additional outreach efforts, incentive payments, and more time on site to recruit respondents could translate into an additional $100 million in recruiting costs over the 7-year period (see Appendix B for the details on our assumptions).
dollar changes. A corollary to this fact is that if large cost adjustments are necessary, sample size is a necessary dimension in which to make large changes in projected costs.
2. Recruitment Costs: The cost of recruiting the baseline sample of 100,000 births is roughly $231 million or about 15 percent of the 7-year estimated field costs of the project. The total includes the costs of recruiting hospitals and providers, as well as the costs of recruiting the women whose births will become members of the probability sample. This cost varies somewhat with the fraction of the sample enrolled at delivery rather than prenatally.
3. Interviewing Costs: At roughly $1,000 each in current dollars, adding or dropping one face-to-face interview for each respondent changes 7-year total field costs by about $100 million. Adding or dropping one telephone interview per respondent changes 7-year total field costs by roughly $40 million.
4. A Supplemental Sample of Nulliparous Women: Because several women (about 3.6 based on current NCS estimates) need to be enrolled and given an in-home interview in order to yield 1 woman with a birth that will enter the sample, the cost of obtaining 5,000 preconception first births enrolled in the study is quite large, about $76 million.
5. Fraction Recruited Prenatally: Recruiting respondents prenatally and administering interviews prior to birth, rather than recruiting them at birth, increases project costs by $30 million for every 10 percentage point increase in the fraction of the total sample that is recruited prenatally.
6. Interview Length: Increasing by 5 minutes the length of all of the 1.05 million interviews administered in the first 7 years of the study, including both face-to-face and telephone interviews, increases total field costs by only $10 million. Attempting to save money by reducing interview length is thus relatively ineffective when compared with the other cost drivers.
FIELD COST ESTIMATES: ALTERNATIVE MODELS
Chapters 2 and 4 provide the scientific basis for recruiting into the probability sample as many women as possible prior to birth. Chapter 2 questioned the high cost and the lack of evidence of value of the 5,000-birth supplemental sample of preconception nulliparous women and the scientific rationale of supplemental samples of 5,000 that might be used for special purposes. Thus, the panel asked its cost consultants to explore a variety of approaches to maximizing prenatal recruitment while staying within the field cost constraints associated with the current design proposed for the NCS. This section presents the data on eight models that incorporate different assumptions about the extent of
prenatal recruiting, the inclusion of the supplemental samples, and reductions in the total number of interviews.
- Model 1–baseline (current design): 100,000 total births, consisting of 90,000 recruited as part of the probability sample (including subsequent siblings) and 10,000 recruited into the various supplemental samples. Of the 90,000 in the probability sample, the mothers of roughly 45,000 would be enrolled at their prenatal providers and receive one or two prenatal interviews. The remaining 45,000 would be recruited at the time of birth. Of the 10,000 supplemental-sample births, the mothers of 5,000 first births would be recruited prior to conception and the mothers of another 5,000 would be recruited at birth for the supplemental convenience sample.
- Model 2–maximum prenatal recruitment: This model isolates the additional expenses associated with maximizing prenatal recruitment in the current design. It is the same as the baseline except that 97 percent of the births recruited in the probability sample of 90,000 (including siblings) and the supplemental convenience sample receive prenatal interviews, along with the preconception sample of first births. We assume that 3 percent of births cannot be recruited prenatally and would be enrolled at delivery (see Chapter 2).
- Model 3–maximum prenatal recruitment into the probability sample but no preconception first-birth supplemental sample: This model isolates the incremental cost of the preconception supplemental sample. It assumes there is no supplemental sample of 5,000 preconception first births, and the probability sample is increased by 5,000 to 95,000; it retains the 5,000 in the convenience sample and assumes they are recruited prenatally. As in Model 2, 97 percent of births in the probability sample (and their subsequently enrolled siblings) are enrolled prenatally and the remaining 3 percent are enrolled at delivery.
- Model 4–drop one in-home and one telephone interview: This model and the next two make small changes in the number or types of interviews. This model adopts the assumptions in Model 3 and drops one face-to-face and one telephone interview in the child’s first year of life.
- Model 5–drop one telephone interview: This model also adopts the assumptions in Model 3 and drops one telephone interview in the child’s first year of life.
- Model 6–drop one face-to-face interview: This model also adopts the assumptions in Model 3 and drops one face-to-face interview in the child’s first year of life.
- Model 7–have only one prenatal interview: This model also adopts the assumptions in Model 3 and includes only one interview prior to
-
birth for the 97 percent of the 95,000 in the probability sample that are recruited prenatally and the 5,000 in the convenience sample.
- Model 8–maximum prenatal recruitment, no supplemental samples, and cost neutrality: This model is similar to Model 3, but it incorporates maximum prenatal recruitment and cuts the number of sampled children to maintain cost neutrality with the current design. Its cost assumptions are the same as the baseline except that 97 percent of births in the probability sample (including siblings) are enrolled prenatally, and there is no preconception first-birth cohort or convenience sample. Under this model, 96,000 children could be enrolled in the study.
Table 5-1 shows the costs and number of births that could be included under these eight models. The second column shows the 7-year field costs, in millions, associated with these eight models. So, for example, the first row shows that the estimated field costs for the current study design total $1.495 billion over 7 years. The third column shows the incremental field savings (or costs) associated with Models 2-8 relative to the baseline cost projection for the field work. The fourth column shows, for each model, the size of the sample that would be needed to make that field strategy cost neutral with the baseline model. The central message of Table 5-1 is that, although the costs of administering the prenatal protocol to women recruited at their providers are higher than the costs of recruiting them at delivery, by eliminating the preconception cohorts for nulliparous women (but retaining preconception cohorts in the sibling samples), the NCS can attain cost neutrality either with modest changes to the data collection protocol or by modifying the sample size. If, as recommended in Chapter 2, the other 5,000 supplemental sample component of the proposed design is dropped, then, at cost neutrality, the size of the probability sample is either close to or larger than what is achievable with the baseline specification.
CONCLUSION 5-1: The panel estimates that the field costs of the current plan for the National Children’s Study over its first 7 years will total $1.495 billion. This estimate does not include a number of other sizable contributors to overall study costs.
Turning from 7-year total field costs to the associated pattern of annual field costs, Figure 5-1 shows a linear increase in annual field costs from about $60 million in the first year to more than $300 million in the fourth year of the study. For reference, the fiscal 2010-2012 congressional appropriations for the NCS were each around $190 million; the fiscal 2013 and 2014 appropriations
TABLE 5-1 Field Costs of Alternative Sample Designs
| Model and Sample Size* | Total Field Cost (in millions of $) | Base Field Cost–Modeled Field Cost (in millions of $) | Size of Main Sample (including siblings) for Field Cost Neutrality |
| N = 100,000 | |||
| Baseline: current design—one-half prenatal and one-half birth recruitment for 90,000 births, plus 5,000 preconception supplemental sample and 5,000 convenience sample (Model 1) | $1,495 | NA | NA |
| Maximum prenatal recruitment plus preconception supplemental sample, and convenience sample (Model 2) | $1,631 | ($135) | 89,179 |
| Maximum prenatal recruitment and convenience sample but no preconception supplemental sample (Model 3) | $1,542 | ($47) | 96,256 |
| Drop one in-home and one telephone interview from Model 3 (Model 4) | $1,348 | $147 | 113,940 |
| Drop one telephone interview from Model 3 (Model 5) | $1,492 | $3 | 100,256 |
| Drop one in-home interview from Model 3 (Model 6) | $1,398 | $98 | 108,836 |
| Have only one prenatal interview in Model 3 (Model 7) | $1,488 | $7 | 100,600 |
| N = 96,000 | |||
| Maximum prenatal recruitment, no preconception or other supplemental sample and cost neutrality (Model 8) | $1,495 | ($0) | 96,256 |
NOTES: All cost figures are based only on costs of fielding the study and thus do not represent the full cost of the NCS; see text for discussion. Cost of convenience sample assumed to be equal to cost of equal-sized prenatal sample.
*See text for details of the panel’s models; see Appendix B for details of cost assumptions and other information about the panel’s cost modeling exercise.
FIGURE 5-1 Annual field costs for three models of the NCS sample design.
NOTES: The costs include only field costs. See text for discussion.
Model 1(baseline): NCS program office proposed plan (N = 100,000)
Model 3: Maximum prenatal recruitment but no preconception supplement sample (N = 100,000)
Model 8: Maximum prenatal recruitment, no preconception supplemental sample, and cost neutral (N = 96,000)
were up to $165 million,2 respectively. It is important to remember that these are appropriations for total project costs and not just the field costs that we modeled.
It is rarely, if ever, possible to accommodate all desired elements in the design of a single study, no matter how large, so that tradeoffs between cost and coverage always have to be considered before finalizing a design. An evaluation of those tradeoffs has to rest on an understanding of the likely benefits and costs
____________________
2Section 1508 of the Consolidated and Further Continuing Appropriations Act, 2013 states: “That $165,000,000 shall be for the National Children’s Study (NCS), except that not later than July 15, 2013 the Director [of the NIH] shall estimate the amount needed for the NCS during fiscal year 2013, taking into account the succeeding proviso, and any funds in excess of the estimated need shall be transferred to and merged with the accounts for the various Institutes and Centers of NIH in proportion to their shares of total NIH appropriations made by this Act.” The panel uses the term “up to” to describe this type of appropriation.
associated with the options under consideration. The panel was surprised to learn that the NCS does not appear to have developed a cost model for considering the likely tradeoffs. Consequently, the panel constructed the models above of some of the most important recruitment and interviewing costs likely to be incurred during the first 7 years of the Main Study. We note again that these components represent only a fraction of the total costs associated with the NCS.
As discussed above and in Table 5-1 and Figure 5-1 (above), the panel estimates total field costs of the current NCS design at about $1.5 billion, with annual costs ranging from $60 million in the first year to a little over $300 million in the fourth year. The average annual field cost is $214 million, significantly more than the most recent congressional appropriation for fiscal 2014 of up to $165 million. And that appropriation is not only for field costs, but for all costs of the NCS.
CONCLUSION 5-2: The major drivers of field costs for the National Children’s Study Main Study are sample size and number of interviews. In-person interviews cost more than twice as much as telephone interviews. In contrast to contacting and gaining the cooperation of respondents, modest changes in interview length contribute minimally to cost.
CONCLUSION 5-3: For the same field costs and with the elimination of the 10,000 supplemental samples—5,000 nulliparous women and a 5,000 convenience sample—the National Children’s Study could afford to enroll a predominantly prenatal probability sample of 96,000 cases with no other changes to the proposed data collection protocol.
It is to be hoped that adequate funding will be provided to carry out the proposed NCS Main Study design, after it has been refined and reviewed. But if adequate funding is not provided, tradeoffs will be required.
RECOMMENDATION 5-1: Given the goal for the National Children’s Study (NCS) to understand the links of environmental exposures to child health and development and its cost structure, if major reductions in the cost of the study need to be made, they should be reductions in sample size rather than exposure domains. Along with such a decision to reduce the sample size, the NCS should reconsider whether to oversample minorities in order to maintain the ability to evaluate health disparities with a reduced sample.
