• formally test and evaluate the selected new procedure(s); and
• clearly communicate with data users at each step.
RECOMMENDATION 10-15 The Bureau of Justice Statistics should follow five steps when contemplating a major methodological change in one of its major surveys: (1) develop the conceptual background for the need for change and alternative sampling, estimation, or survey methodologies; (2) formally review these concepts with a broad set of data users to decide on a strategic direction; (3) formally review the statistical and survey methodological issues and proposed changes with technical experts in the broader statistical community; (4) formally test and evaluate the new procedures, their feasibility, and their impact; and (5) clearly communicate with data users at each step.
The panel offers two concrete examples regarding the need for the above recommendation: the recent change to account for series victimization and an earlier decision to use data from the bounding interview in estimation. In both cases, BJS concluded that a methodology change was needed and had informal, if not formal, discussions with a number of data users who weighed in on the issues. However, the third step is also important, and it appears to have received less attention. In the example of series victimization, BJS appears not to have engaged fully with specialists on outliers in the statistical community to help evaluate a wider range of alternative procedures to adjust (or not adjust) for these outliers. In the example of the bounding interview, the current adjustments that involve weighting the first wave data down to the average level of waves 2-7 are suspect. As more recent analysis has shown, the “recency” effect, along with notable attrition rates over the life of the survey, raises the question of whether these adjustments to the data are the best approach. Again, engaging more statistical expertise to examine alternatives would have been appropriate when this adjustment procedure was considered.
Following the decision to change, communication with data users needs to be frequent and clear. For example the publication Criminal Victimizations, 2011 (Bureau of Justice Statistics, 2012a) stated that the series victimization change was being implemented in that, and future, publications. The panel did not find a notice that the change was being made retroactively on the online database. More and frequent communication is always a good policy.
BJS has a very important mission to provide estimates of criminal victimizations within the United States, both annual rates of those victimiza-