The National Children's Study 2014: An Assessment (2014)

Appendix B

Field Costs for the National Children’s Study: First 7 Years

The panel was charged with evaluating the proposed methodologies for the National Children’s Study (NCS) Main Study, including whether such methodologies are likely to produce scientifically sound results. In trying to carry out this part of our charge, the panel found that the relevant cost information was not available (see Chapter 5). Thus, in order to assess alternative approaches with key design elements, such as sampling frame and design, the recruitment and retention process, and broad aspects of the interview schedule and data collection, the panel commissioned two consultants—Lisa Schwartz of Mathematica Policy Research and Randall Olsen of Ohio State University—to construct a cost model of several design options. They collected cost information relevant to the modeling at meetings and through telephone conversations with staff from the NCS Program Office. As explained below, most of the NCS Program Office cost suggestions were adopted and used in this analysis, and the Program Office staff were invited to “fact-check” the model assumptions and results. Key results of the cost analysis are summarized in Chapter 5. This appendix details the assumptions behind the cost model.

SCOPE OF THE COST ANALYSIS

The assessment was limited to the field costs that would vary to a material degree for the design options the panel considered. Specifically, the budget models focused on recruitment, retention, and data collection costs through the first 7 years of the Main Study. Our general objective was to assess the cost

implications of key design tradeoffs that were roughly consistent with the total estimated field costs of the current proposed Main Study design.

Despite the large 7-year total field costs we estimate—some $1.3 to $1.6 billion—it is important to realize that our estimates omit many potentially sizable study costs, although they are not materially affected by total sample size or recruitment strategy. Those omitted costs include

• project management and oversight of contractors by the National Institute of Child Health and Development (NICHD);
• any costs associated with the Vanguard Study and pretesting at any sites;
• any costs associated with purchasing or building the sample frame;
• design and supervision of the sampling activities and the production of sample weights;
• design and review of questionnaire and specimen collection protocols and the preparation, printing and distribution of training materials or other supplies and consumables to the field;
• costs to assay environmental samples or analyze blood samples other than what the Program Office determines is necessary for quality control;
• security tasks to keep the NCS compliant with the Federal Information Security Management Act of 2002;
• management of the data collection contractors;
• management of the data flow to and from the field to whichever contractor is handling the data;
• programming of the questionnaires, configuration and distribution of laptops, including troubleshooting repair, replacement, and inventory control;
• tracking and storing all specimens; and
• data management, including storage, data cleaning, variable creation, geocoding, appending of other data to the master database, documentation, generation of user data files whether for public access or restricted access, user support, user outreach of any kind, and publicity and promotion of the data.

MODELING ASSUMPTIONS

To the extent possible, the panel constructed a cost model based on information provided by the NCS Program Office. When cost information was not available, we relied on the experience of our consultants (who have a combined 40 years of experience with survey research and several contract survey research firms) to develop reasonable estimates. The cost models reflect the breadth of

their experience in budgeting various components of large, complex data collection efforts.

For the most part, the Program Office agreed with our cost assumptions. When their review suggested changes to our assumptions (either up or down), we usually revised our models accordingly. These changes to our initial assumptions did not produce major changes to the overall estimates. As described in the penultimate section of this appendix, we questioned only two NCS Program Office assumptions as being inconsistent with the experience of the consultants who conducted the cost analysis. In one case, the need for some in-person interviews at birth, the consultant’s assumption was used in the cost analysis.

We took as our baseline model the design as described by the NCS Program Office in documents submitted to the panel.¹ We begin with a high-level summary of the designs included in our cost models. Across all models, we assume the yield from provider-based sample recruiting is measured by the number of prenatal enrollments that lead to a sampled birth. That is, if a woman agrees to the study and makes an appointment for the first prenatal interview, her agreement is not counted as a successful recruitment until and unless the woman completes the first prenatal interview and then stays with the study through childbirth.

Based on the information provided to the panel, we assume that 250 hospitals are the primary sampling units (PSUs).² Also based on Program Office information, we assume that 50 percent of women who are approached will refuse to participate and that of the women who agree to participate, 20 percent will drop out of the study after their first prenatal interview and before birth. The Program Office estimates the average enrolled prenatal birth mother will receive 1.5 prenatal interviews. Accounting for attrition, this means about 1.7 prenatal visits are conducted for every birth brought into the study by prenatal provider-based recruitment.

The secondary sampling units are non-hospital-based providers (i.e., practices), and we assume, on average, five such providers are recruited per sampled hospital (N = 1,250 providers). Recruitment of hospitals and providers and sample enrollment occurs in four sequential groups of PSUs. Each group of PSUs is tracked for 4 years, but the initial year for each group is staggered: that is, recruitment in group A takes place in years 1 through 4,

____________________

¹Our work is based on materials received from the Program Office through January 2014, but we note that there may have been subsequent changes to this design, which would not be reflected in our cost assessments. However, the major cost drivers are parameterized in the spreadsheet, which is available at http://sites.nationalacademies.org/DBASSE/CNSTAT/National_Childrens_Study_2014/index.htm [July 2014]. .

²At the time of this report, NCS anticipated that between 200 and 300 hospitals would comprise the study’s primary sampling units and estimated that they would recruit five providers per hospital.

group B in years 2 through 5 and so forth.³ This approach spreads out the higher costs of enrollment and the heavy first-year data collection over more years so that peak costs are not as high as they otherwise would be. Using this approach, peak costs occur in year 4 when all PSUs are involved in active recruitment or data collection. The last birth would be enrolled in year 7 of the data collection phase: hence, our analysis is budgeted over 7 years. In years 8 and later, the data collection costs we estimate fall until they reach a steady state of about $50 million (plus an adjustment for inflation). This approach reflects a case load averaging 50,000 active cases per year⁴ minus attrition.

We note that for modeling purposes, recruitment costs are influenced by which of two data collection protocols is used for a case. As noted above, the prenatal protocol, the more costly of the two, includes, on average, 1.5 prenatal in-home interviews, which include environmental and biospecimen collection. All women recruited prenatally and all women in the preconception sample once pregnancy is confirmed receive this protocol. The protocol for women recruited at delivery includes a take-home kit for collecting and shipping environmental samples: in all other ways the women recruited after delivery receive the same schedule of interviews. All women recruited at the hospital and women in the non-probability (non-preconception) sample receive the protocol for women recruited at delivery.

The scientific value of the prenatal sample led the panel to consider options for increasing the prenatal fraction of the sample without increasing the total field costs, which could be done in a variety of ways. As we detail below, dropping the preconception sample and then making 97 percent of the sample prenatal through provider-based recruitment comes very close to achieving the goal. To match the 7-year cost of the baseline model, the sample only has to be reduced to about 96,000. This result reflects the high cost of the preconception sample, primarily due to the large number of women who must be tracked to achieve 5,000 enrolled births.

The Cost Models

The panel’s cost analysis is based on eight models, of which the first is the baseline:

____________________

³At the time of this report, NCS anticipated that provider-based recruitment will occur over a 3-year period in order to ensure that children born to enrolled women are born in year 4 of recruitment; delivery-based recruitment will occur over 4 years.

⁴After age 5, children will be monitored once every 2 years. Thus, to monitor 100,000 children, there will be 50,000 cases per year.

• Model 1–baseline (current design): 100,000 births (including siblings), with 90,000 probability sample (half recruited prenatally, half at birth), 5,000 preconception and 5,000 other convenience supplemental sample. Of the 100,000 births, 5,000 are recruited preconception and, if the mothers become pregnant within the 4-year window for recruitment, they also receive the prenatal sample protocol. The baseline protocol administers an average of 1.5 prenatal interviews for the women sampled prenatally. NCS estimates a 20 percent attrition rate between the initial prenatal, face-to-face interview and delivery, so the effect on costs is equivalent to an additional 0.2 prenatal interviews (for a total of 1.7) administered per woman enrolled in the prenatal protocol. The baseline model budget includes performing the full complement of assessments in the first year: a hospital interview and, for 15 percent of the sample, an in-home visit immediately after birth; a telephone interview at 3 months and 9 months; and in-home assessments at 6 months and 1 year. The last two assessments are performed for all birth events. The content of the various interviews may differ for the prenatal and birth sample mothers, but we assume these differences are not material in terms of overall costs.
• Model 2–maximum prenatal recruitment: Same as the baseline except 97 percent of the births, including those in the other supplemental convenience sample, are enrolled prenatally. We assume that in order to sample a pro rata share of women who are not eligible for prenatal, provider-based recruitment, 3 percent must be enrolled at delivery.
• Model 3–maximum prenatal recruitment but no preconception first-birth supplement sample: Same as Model 2 except no preconception cohort is recruited, and the probability sample is increased by 5,000.
• Model 4–drop one in-home and telephone interview: Same as Model 3 except eliminate one face-to-face and one telephone interview in the child’s first year of life.
• Model 5–drop one telephone interview: Same as Model 3 except eliminate one telephone interview in the child’s first year of life.
• Model 6–drop one face-to-face interview: Same as Model 3 except eliminate one face-to-face interview in the child’s first year of life.
• Model 7–just one prenatal interview: Same as Model 3 except women have only one prenatal interview prior to birth.
• Model 8–maximum prenatal recruitment, no preconception supplemental sample, no other convenience sample, and cost neutrality: Same as Model 3 except there are no convenience samples and only 96,000 children are enrolled in the study.

Cost Model Assumptions

In this section, we provide information on some of the key cost parameters for the NCS field work, in five areas: general, sample recruitment and retention, interviewer recruitment and training, data collection, and specimen processing.

General Assumptions

• Inflation is estimated at 2 percent per year with field work beginning in 2015. Insofar as field work begins after 2015, all numbers should be increased by 2 percent per year. Should inflation accelerate, figures would need adjustment.
• Attrition is based on how long the mother has been in the study, not the age of the child. In families that include siblings, attrition by the mother results in attrition for all her children. We assume in the second year, 90 percent of mothers remain in the study; in year 3, 87 percent; in year 4, 85 percent and then a 1 percent decline in participation rate every year thereafter through year 7.
• All births to sampled women inside the 4-year recruitment period are included in the study. For a 100,000-child sample, about 8.8 percent of the children are subsequent siblings of the sampled births, based on the panel’s analysis of fertility data. For sampled pregnancies occurring later in the cluster’s 4-year recruitment window, there is less time for eligible siblings to be born within the 4-year window. As a result, the 8.8 percent subsequent sibling projection may be lower than one might intuitively expect.
• Indirect costs and fees are 57 percent. The fringe rate for interviewers is 20 percent and 40 percent for professional staff. Insofar as telephone interviewing is used, we assume it is done from a call center with a total, all-inclusive staff cost of $30 per hour. This assumption may be low if calling is done from a national survey firm’s premises.

Sample Recruitment and Retention

• 250 hospital PSUs and 1,250 practice providers are recruited at 100 professional hours of senior level staff time per entity (150,000 professional staff hours).
• Retention of hospitals and practices in the study requires one on-site meeting per year. Hence, during the 4-year recruitment period, there would be four on-site meetings with each participating hospital and three on-site meetings with each participating practice.
• The costs of institutional review boards (IRBs) are borne by the NCS Program Office: we assume that one central IRB will approve the study

• and that no additional review boards will require approval. Thus, no IRB-related costs are included in our analysis.

• Hospital staff require a stipend for collecting cord blood, cord samples, and placenta samples at birth. Estimated hourly rate for hospital staff to collect biospecimens is $100, unloaded, and it is estimated that biospecimen collection will take, on average, 0.5 hours per birth.
• 80 percent of hospitals and 70 percent of providers agree to participate, factoring in refusals to the provider recruitment model.
• To secure cooperation, a one-time monetary incentive to hospitals of $500 per hospital is included.
• A combination of monetary and nonmonetary incentives to providers is estimated at $500 per year per practice.
• To recruit the prenatal sample, field staff are on site at providers’ offices for 4 hours per day, 2 days per week for a 3-year period.
• To recruit the birth sample, field staff are on site at hospitals for 20 hours per week over 4 years. On-site time does not include the hours needed to complete the in-hospital interview.
• 25 percent of the recruitment costs can be assigned to each of the four clusters of hospitals. The costs for each cluster are not distributed evenly over the 4 years of recruitment within each cluster. Rather, we assign 40 percent of the cluster costs to the first year of recruiting within a cluster, 25 percent to year 2, 20 percent to year 3 and 15 percent to year 4. This assumption reflects the up-front costs for recruiting hospitals and providers.
• The strategy for selecting the convenience samples has not been defined with enough exactitude to generate reliable budget estimates. We have assumed NCS can obtain this supplementary sample at roughly the same unit cost as either the prenatal or delivery sample.
• The participation rate among recruited women is 50 percent, requiring a sample of about 11,650 per year for each of the four major cluster groups to meet the overall target sample size.

Interviewer Recruitment and Training

• Interviewer recruitment is 30 hours of a field supervisor’s time per recruit, which includes time to recruit interviewers who will do the phlebotomies.
• Training is 80 hours, including time to train for phlebotomy, and requires 15 days of hotel and per diem at $300 per day. Travel to training sites is projected at $600 per trainee.
• Telephone interviewer training is projected at 40 hours and does not require travel by trainees. We assume this training will be done in person at the call center.

Data Collection

• All births include an in-hospital interview of the mother and the collection of the placenta and cord blood and maternal urine and saliva, blood, and dried blood spots from the infant.
• 15 percent of births will need an age 0 in-home interview to collect environmental samples, biospecimens (blood and breast milk, at a minimum) that respondents do not complete and return themselves.
• One field interviewer is sent to households to complete an in-person interview and specimen collection.
• 65 percent of all in-person cases are “easy” and require 5.5 hours per respondent interviewed for contact activities (including travel, appointment setting, rescheduling, and packing specimens).
• 25 percent of all in-person cases are “hard” and require 9 hours per respondent interviewed for contact activities.⁵
• In-person interviewing in the home is 2 hours per interview including consent, answering respondent questions and maintaining rapport, anthropometrics, and specimen collection. This time is in addition to the time required to locate, make contact, travel, and secure cooperation with a respondent.
• All in-person interviews include a venous blood draw from the mother or child.
• Interviewer travel expenses $40 to complete an easy case and $110 for a hard case.
• Respondent fees average $75.
• Post-birth telephone interviews are 1 hour per interview. Telephone interviewers require 2 hours to secure cooperation for easy cases and 4.5 hours for hard cases. Hard telephone cases will also require, on average, $50 each in other expenses and some will require a personal visit to secure cooperation.⁶
• Laptops and smartphones, including connection and Internet charges are $750 per year per interviewer.
• Anthropometric equipment (tapes, scales, calibration weights, stadiometers blood pressure meters, etc.) is $4,000 per interviewer in year 1, $250 annually thereafter, based on experience in the Vanguard Study. We assume a 3-year lifetime for this equipment, making the

____________________

⁵Difficult cases often require repeated visits due to broken appointments, as well as visits to determine where the respondent is located and to secure additional locating information from former neighbors. Younger households are often more mobile, which makes some cases difficult because of locating problems and not just because of reluctance or resistance to cooperation. The cases that are not “easy” or “hard” are noninterviews.

⁶Other expenses include additional mailings and an in-house locating effort, such as electronic database searches, to obtain new contact information.

• annual cost $1,550 per interviewer. The cost of Oragene kits and specimen supplies is $50 per case. The cost for collecting, packing, mailing, labeling and assaying 5 percent of the environmental and blood samples⁷ is $190 per in-person interview.

• The expected interview completion rate is 90 percent.
• Field management is estimated at 1 hour of field supervisor time for every 5 hours of field interviewer time, and 1 hour of higher level supervision for every 6 hours of field supervisor time.
• Field interviewers’ rate of pay is $23 per hour to compensate for the specialized skills that are needed to do phlebotomy; field supervisor pay is $35 per hour; and the next higher level of supervision is set at $40 per hour. We do not budget overall field management and central direction as the scope for such work is unspecified as yet.

Specimen Processing

• No lab work will be done on the environmental samples; lab work will be done on 5 percent of the venous blood samples for quality control only.
• Shipping costs for specimens from the field to the central repository are approximately $135,000 per year per 1,000 participants.

Our assumptions do not include the costs of repository processing and storage of biological and environmental samples, which we estimate at $126 million over the first 7 years ($18 million in annualized costs), based on an inventory of 42,000,000 primary and derivative samples. We note these for informational purposes only as they are significant.

COSTS AND COST DRIVERS

Based on these assumptions and the resources implied for the NCS design, we estimated the costs of our models. We begin this section with the major factors that influence the costs.

• Sample size—Pursuing a cohort of 100,000 is the primary driver of costs. Even modest cost perturbations at the interview level, given the number of planned interviews and the sample size, will generate large impacts in dollar amounts, if not in percentage terms. A corollary to this fact is that if large cost adjustments are necessary, sample size is a necessary dimension to include.

____________________

⁷NCS plans to assay or analyze 5 percent of environmental and biological samples for quality control purposes only.

• Recruitment at hospitals and providers—The cost of recruiting the probability sample is roughly $231 million or roughly 15 percent of the 7-year costs of the project. This cost varies with the fraction of the sample enrolled at delivery rather than prenatally.
• Interviewing costs—At roughly $1,000 each (in current dollars), adding or dropping one face-to-face interview for each respondent changes costs by about $100 million. Changing the number of telephone interviews per respondent by one changes total costs by about $40 million. As noted above, we assume that to obtain a satisfactory rate of completed remote interviews, some respondents will need to be interviewed or at least contacted in person to secure an interview.
• Preconception sampling—Because several women (about 3.6 based on NCS program office estimates) need to be tracked in order to yield one woman with a birth entering the sample, the cost of obtaining 5,000 preconception births enrolled in the study is about $76 million.
• Fraction in the prenatal protocol group—Changing respondents from being recruited at providers and administered prenatal interviews rather than being recruited at birth and not receiving prenatal interviews increases project costs by $30 million for every 10 percentage point increase in the fraction receiving the prenatal protocol. This cost reflects both the cost of recruitment and the cost of prenatal interviews, both to women who enroll in the study and the women who attrite before delivery.
• Interview length—Reducing by 5 minutes the length of every interview administered in the first 7 years of the study, both face to face and telephone, reduces costs over the first 7 years by $10 million. Attempting to save money by reducing interview length is relatively ineffective and will come at far higher price, scientifically, than eliminating an entire interview. We believe this is the least attractive approach to cost containment of all the alternatives we explored.

For the baseline and models 2 through 8, Table B-1 shows our estimated field costs for the first 7 years of the NCS, the cost differential relative to the baseline, and the sample size that would make each model cost neutral with respect to the baseline. (This table also appears as Table 5-1 in Chapter 5). A notable feature of the table is that while moving to an overwhelmingly prenatal protocol for data collection increases costs, those additional costs are substantially offset by eliminating the preconception cohort. Modest reductions in the schedule of interviews or modest reductions in total sample size (or both) result in costs that are roughly equivalent to baseline costs. These alterations make it possible to collect a probability sample of births that is larger than the baseline proposal. Other adjustments are possible, but this provides a framework for looking at costs and reconciling them with budgets.

TABLE B-1 Field Costs of Alternative Sample Designs

NOTES: All cost figures are based only on costs of fielding the study and thus do not represent the full cost of the NCS; see text for discussion. Cost of convenience sample assumed to be equal to cost of equal-sized prenatal sample.

^*See text for details of the panel’s models.

TABLE B-2 Year-by-Year Cost Comparisons Across Models (in millions of $)

NOTE: See text for discussion.

Table B-2 shows the estimated yearly costs for the eight models as well as the 7-year total. The data show that, despite spreading sample recruitment costs over 7 years instead of four, field costs peak at a level above the fiscal 2014 appropriation level. Attempting to spread recruitment costs over even more years may reduce the peak but increase total program costs.

Questionable Assumptions

The panel questioned only two Program Office assumptions as being inconsistent with the experience of the consultants who conducted the cost analysis for us, recruiting costs and need for follow-up. First, the Program Office estimates that the level of effort needed to recruit for the Main Study will be the same as that required for the Vanguard Study. However, the original Vanguard Study contractors were primarily universities, many of which also had medical schools and university-affiliated hospitals. Staff affiliated with those universities or contractors acting on their behalf recruited hospitals, providers, and women for the study. In our judgment, these affiliations and the universities’ community-level “brand equity” likely had two effects: (1) they made it more likely that the entities they approached agreed to participate; and (2) the level of effort needed to gain that cooperation was less than it would otherwise be. We judged that it is unlikely that a contract research organization will be as effective in garnering support from a random sample of hospitals, providers, and women as an effort undertaken by a university or medical center with strong national or regional name recognition. While all of our models incorporate the Program Office’s recruiting assumptions, we note in Chapter 5 that the study may incur substantially higher costs if, in practice, contract research firms must expend more effort than estimated to achieve the study’s recruitment goals. A number of changes to recruiting assumptions could result from this change in staffing model:

• In addition to 100 hours of professional staff time needed to recruit hospitals, up to 300 additional hours may be needed to gain endorsement from professional organizations (such as the American Hospital Association); a similar number of hours of professional staff time may be needed to gain support from professional organizations that can encourage providers to participate in the study.
• Two rather than one on-site meeting may be needed per year per hospital and per provider to recruit and retain them in the study; in addition, monthly phone meetings may be needed to address issues and maintain relationships.
• Contract research firms may host webinars for participating practices to encourage the sharing of best practices (no webinars were included in the baseline model).

• Practices may require a substantially higher incentive than the budgeted $500 per year for hospitals to participate: our baseline model assumes a one-time payment of $5,000 for practices to secure participation.
• Field staff may need to be on site for more hours each week to successfully recruit women for the study. For the delivery sample, we assumed field interviewers would need to be on site 33 hours per week (the baseline model assumed 20 hours per week); for the provider-based sample, we assumed field staff would be on site 16 hours per week (the baseline model assumed 8 hours per week).

Second, in all of our models including the baseline we assumed 15 percent of cases would need a follow-up visit after birth to retrieve specimens or conduct an interview. The Program Office assumes this would not be necessary, but we judge that the many distractions surrounding a birth, the possibility of rapid discharge from the hospital before the in-hospital interview is conducted, and the possibility some mothers simply would not get around to collecting environmental samples all suggested we allow for the need for an in-home visit shortly after birth.

Limitations of Our Cost Model

Apart from limiting its estimates to only the field costs of the NCS, our cost analysis has other noteworthy limitations. We believe our estimates are reasonable, but there are reasons that contractors may bid costs that differ from those presented here. For example, if the contracts are fixed-price rather than cost—reimbursement contracts, the uncertainties inherent in performing such a large and complex project may lead bidders to build in a “cushion” to cover uncertainties, such as subtle changes in the interpretation of the scope of work, delays created by procurement problems, or other administrative or governmentwide disruptions. If the work scope is not clearly defined due to uncertainties over specimen collection or interview content, contract modifications become more likely, and such changes are inherently less competitive, increasing costs. There is also the risk of a contractor winning a bid that is based on a low tender price and then being unable to perform the work within the dollar limits or finds the work technically more demanding than expected. Such a problem with one part of the project, say, technological infrastructure, can easily affect other parts. Subtle changes to questionnaires can, depending on the change, have serious cost implications that only become evident when they are fielded.

There are also risks inherent in survey projects for which respondent reaction and attitudes have unpredictable cost implications. For example, the plan to draw blood from the mother at every personal visit is a very aggressive plan in terms of the demands it places on respondents. This may lead to a less

cooperative respondent pool that may require more time, effort, and incentive payments to overcome. Alternatively, these demands may reduce costs because respondents leave the study in greater numbers. There are also regulatory risks. If regulations to which the NCS is subject become more stringent, this could easily generate unpredictable cost increases. These kinds of problems are not specific to the NCS, of course.

This page intentionally left blank.