The NCVS is largely designed and conducted for BJS by the Census Bureau. Complex survey contracts cannot be wisely administered without highly sophisticated statistical and methodological expertise. Federal statistical agencies that successfully contract out their data collection (either to the Census Bureau or a private contractor) generally have mathematical statisticians and survey methodologists who direct, coordinate, and oversee the activities of the contractor. While many of the BJS staff are labeled “statisticians,” the panel observed the lack of statistical expertise that is crucial in dealing with the trade-offs of costs, sample size, numbers of primary sampling units, interviewer training, questionnaire length, use of bounding interviews, etc. The expressions of displeasure about the Census Bureau’s management of the NCVS were not matched with BJS statistical analyses and simulations of design alternatives that might offer better outcomes for the agency. Furthermore, the panel thinks that the number of of BJS full-time staff dedicated to the analysis of NCVS data and the generation of reports is insufficient to exploit the full value of the survey and to navigate its challenging future. Some of the issues that require analysis (e.g., the effects of declining response rates on estimates, trade-offs of waves and questionnaire length) need statistical and methodological expertise that goes beyond current in-house capabilities.
Following the lead of other federal statistical agencies, BJS could usefully enhance statistical expertise on its staff with a program of outside research funds. When federal agencies form useful partnerships with academic researchers, they can reduce their overall costs of innovation. BJS has a track record of small research grants connected to the NCVS. The panel applauds these and urges an expansion to tackle the real methodological issues facing the NCVS.
Recommendation 5.2: BJS should perform additional and advanced analysis of NCVS data. To do so, BJS should expand its capacity in the number and training of personnel and the ability to let contracts.
One reason that the panel thinks that technical staffing and external research are important is that many of the questions posed about the NCVS have not been evaluated sufficiently for us to provide recommendations to BJS on the final design of the survey. The panel thinks that this is the long-term result of “eating its seed corn,” of using the operating budget too much to release the traditional reports and too little to scope out the problems of the future. It was well known 15 years ago that household survey response rates were falling; the impact on survey costs of these falling rates was clear (de Leeuw and de Heer, 2002). Federal statistical agencies (see CNSTAT’s Principles and Practices of a Federal Statistical Agency) must consistently