9
Synopsis of Potential Errors in the
National Crime Victimization Survey
The mission of the Bureau of Justice Statistics (BJS) is “to collect, analyze, publish, and disseminate information on crime, criminal offenders, victims of crime, and the operation of justice systems at all levels of government” (Bureau of Justice Statistics, n.d.-c). A major tool in accomplishing this mission is the National Crime Victimization Survey (NCVS), which provides a rich source of information on criminal victimizations, the victims, and the specifics of their criminal victimizations. In a 2008 report, the National Research Council found that the NCVS provided the only annual source of national data on criminal victimization with detail at the incident level. The report’s first recommendation (3-1) was that “BJS must ensure that the nation has quality annual estimates of levels and changes in criminal victimization” (National Research Council, 2008, p. 3). This panel strongly concurs with that recommendation. To further that mission, this panel has focused specifically on the quality of the measurements of rape and sexual assault with the purpose of helping improve quality in BJS’s estimates of those types of criminal victimizations.
The NCVS has both sampling and nonsampling error, as does every survey. The panel examined the error structure of the NCVS (see Chapters 7 and 8) in considerable detail with the intent of identifying areas that might be particular problems. We reiterate that the NCVS is an omnibus survey with the primary goal of estimating many types of criminal victimization. Its survey design is geared toward this broad goal, and it does not and cannot incorporate separate features for measuring each different type of victimization. Yet the survey’s basic goal and design create problems for measuring rape and sexual assault. This chapter summarizes the discussions
in Chapters 7 and 8 that the panel believes are most problematic for measuring rape and sexual assault. The chapter concludes with several specific recommendations. The bulk of the panel’s recommendations are included in Chapter 10.
POTENTIAL SOURCES OF ERROR: SUMMARY
The NCVS uses a classical area sampling design based on selection in multiple stages to create a sample of housing units. This basic design is used for a number of general household surveys conducted by the Census Bureau, and it is a good design for an omnibus type survey. For estimates of rape and sexual assault, however, a limitation of the NCVS sample design is its current inability to control the sampling error for estimates of these low-incidence events. Although the sample size of the NCVS is large, rape and sexual assault are low-incidence events and the size of the estimated coefficients of variation (CVs)1 for those estimates at the national level allows considerable year-to-year fluctuation (see Chapter 7). In addition, also because of the low incidence of rape and sexual assault, BJS has to pool multiple years of data to get estimates of rape and sexual assault for important subpopulations (e.g., by race, gender, marital status, or age group).
In 2011, BJS changed its process for counting series victimizations. The effect of the change was substantial for estimates of rape and sexual assault: these victimizations were undercounted in the past, but series victimizations (based on only a few reports) now account for almost 40 percent of the national estimates of rape and sexual assault. The handling of these outlier reports has created additional concern about year-to-year fluctuation in the estimates of rape and sexual assault.
The overall response rates on the NCVS have remained fairly high, and there is little item nonresponse coded for specific rape and sexual assault screener questions. However, the panel is concerned about panel attrition, particularly because those individuals most likely to drop out—younger people and individuals not living as a couple—may be at greater risk for rape and sexual assault. The panel’s judgment is that individuals who respond on the NCVS but do not want to report a specific victimization will refuse the question indirectly by responding that no victimization occurred. This item nonresponse would then be coded as a legitimate zero.
Although BJS clearly defines what it intends to measure regarding rape and sexual assault, because of the omnibus design of the screener, these concepts are poorly translated through the data collection instruments. The in-
________________
1The CV is the standard error of a survey estimate divided by the estimate itself (expressed as a percent). It provides a relative measure of the sampling error associated with survey estimates.
struments use oversimplified terms that can be interpreted in different ways. This situation creates the potential for specification error in the NCVS.
Several issues with the NCVS could lead to measurement error associated with the questionnaire, the data collection mode, and the interaction between interviewers and respondents. These issues include comprehension of key questions and respondents’ willingness to answer certain questions. The latter could be affected by the mode of data collection and interviewer gender. There continues to be inadequate training of interviewers and monitoring of the interviews on the NCVS.
All surveys have error associated with their processing. BJS publishes little information about its processing methods or errors found. The lack of such transparency makes it difficult for NCVS data users to fully understand NCVS estimates and their limitations.
OBSTACLES TO HIGH-QUALITY ESTIMATES
Based on the potential errors in the NCVS discussed in Chapters 7 and 8 and summarized above, the panel identified four major obstacles for accurately estimating incidences of rape and sexual assault.
1. a sample design that is inefficient for measuring these low-incidence events,
2. the context of “crime” that defines the survey,
3. a lack of privacy for respondents in completing the survey, and
4. the use of words with ambiguous meaning for key measures in the questionnaire.
These obstacles form the basis for the panel recommendations in Chapter 10 to create a separate survey to measure rape and sexual assault.
Inefficiency of the Sample Design
The NCVS is a general criminal victimization survey that targets the noninstitutionalized population of the United States with questions about different types of criminal victimizations, including crimes against property and people. Of the 19 million criminal victimizations identified through the NCVS in 2011, 64 percent were property crimes, which are dominated by theft; 29 percent were violent crimes (against people) but not classified as serious; and the remaining 7 percent of criminal victimizations were classified as serious violent crimes,2 of which only 1 of the percentage points
________________
2Serious violent crimes include robbery and aggravated assault, but not murder and kidnapping, which are not estimated through the NCVS.
FIGURE 9-1 Criminal victimizations by type, NCVS 2011.
SOURCE: Data from Criminal Victimization 2011 (Bureau of Justice Statistics, 2012a).
was rape and sexual assault (see Figure 9-1). Thus, from a statistical perspective, rape and sexual assaults are statistically rare incidents.
The rarity of an attribute in a population presents unique challenges to efficient and effective sampling to estimate the proportion of the population with that attribute. In this situation, the difficulty can be seen by observing the CVs for various types of criminal victimizations, with a lower number meaning a more precise estimate (see footnote 1). Nationally, in 2011, the CV for property crime was 2 percent, for serious violent crime it was 7 percent, and for rape and sexual assault it was 14 percent (Bureau of Justice Statistics, 2012a). The NCVS sample design is not well suited to measure the uncommon attribute of having experienced a rape or sexual assault in the past 6 months (see Conclusion 7-1 in Chapter 7).
This concern is intrinsic to the basic design of the NCVS. BJS and the Census Bureau could address this issue by implementing a very large increase to the existing sample size of the NCVS or by modifying the NCVS sample design to focus more efficiently on subpopulations at greater risk for sexual victimization. (See Chapter 10 for a discussion of such a focused design.) However, the NCVS is an omnibus victimization survey whose goal is to measure a broad range of criminal victimizations. If it were to focus on the subpopulation at risk for a specific type of victimization (rape and sexual assault), then it would likely make the survey design much less efficient for measuring other types of criminal victimizations across the whole survey population.
Criminal Context of the Survey
The NCVS is a criminal victimization survey. It is introduced that way to household members, and the questionnaire goes through a listing of different types of crimes, asking each respondent whether any of these crimes has happened to him or her. As discussed in Chapter 8, victims may not always think of a rape or sexual assault as a crime, particularly if the respondent knows the offender. In addition, some victims may fear disclosure to police and may associate a government crime survey too closely with law enforcement. The panel concludes that the context of a criminal victimization survey is a major barrier to accurate reporting of incidents of sexual victimizations (see Conclusion 8-7 in Chapter 8).
BJS could address this issue by moving the measurement of rape and sexual assault to a separate survey (as recommended in Chapter 10) or by changing the entire context of the NCVS to something more neutral. The modification of the NCVS into a survey that has a “neutral context” would entail a major redesign of that survey (including its name), and such a change would undermine other goals of the survey. The context of criminal victimization appears to work well for reporting other, less sensitive, victimizations. A redesign of the NCVS to remove the “criminal context” on this large omnibus survey would make it difficult to effectively communicate the purpose of the survey to respondents, and it would likely negatively affect the reporting of many other types of victimizations.
Lack of Privacy in Responding
Providing a respondent with privacy is an important prerequisite for any survey that deals with sensitive questions. Privacy, specifically from other household members, is critical for accurately responding to inquiries about rape and sexual assault, in part because the victim often knows the offender. In fact, the offender may be a household member. The current NCVS data collection protocols do not provide sufficient privacy (see Conclusion 8-8 in Chapter 8). Some privacy gains would be possible on the NCVS by switching to a self-administered mode of data collection. This switch would be a major change for the survey, and it would likely have both beneficial and detrimental effects on collecting information about other types of criminal victimizations.
The panel concludes that such a change of data collection mode is important to make, but that this change alone would still not provide the level of privacy that is needed for reporting sexual victimizations. As long as multiple members of the household are interviewed with an identical questionnaire, an adequate level of privacy is unlikely to be obtained. Each household member knows the questions that are asked on the survey and
the duration of the interview based upon their own responses. An individual household respondent may continue to have concerns about reporting an incident of rape or sexual assault under those circumstances. To achieve the privacy level needed for questions regarding rape and sexual assault, the survey protocol would also need to change to targeting only a single person in each household or to deploy a “matrix” design in which each household member would be screened for different types of criminal victimizations. If implemented for the NCVS as a whole, then either approach would severely reduce the number of responses obtained for each specific type of victimization. Without an accompanying increase in sample size, this change would negatively affect the precision for estimates of all types of victimizations.
Ambiguous Terms in the Questionnaire
The current NCVS screening questionnaire uses a series of cues to aid respondents in remembering past victimizations. As part of those cues, it uses the terms rape and sexual assault without any further elaboration as to what those terms mean. Although BJS has precise meanings for these terms, they are not provided to respondents. It is not reasonable to assume that all individual respondents will interpret these words identically or as BJS anticipates (see Conclusion 8-6 in Chapter 8). More behaviorally specific words that describe a specific set of actions would lead to more accurate responses on these questions.
Such a behavioral approach is not completely alien to the NCVS. Some of the screening questions for assault do use some behaviorally specific language: a respondent is not asked whether he or she has been assaulted, but is asked questions that describe specific behaviors—“Has anyone attacked or threatened you … with anything like a baseball bat, frying pan, scissors, or stick?” It would be possible to make behaviorally specific changes on the current NCVS regarding rape and sexual assault with only a modest effect on other parts of the questionnaire. However, there would need to be specific screening for rape and sexual assault that is separate from and in addition to the more generalized cuing sequences currently used in the screener. Another difficulty is that the NCVS interviews children as young as 12 years of age, and there may be parental objections to the behaviorally specific questions. With a separate survey as recommended by the panel, there would be more options available to BJS on handling respondents between the ages of 12 and 17.
The majority of the panel’s recommendations specifically address new directions for measuring rape and sexual assault (see Chapter 10). In this
section, the panel offers several general recommendations to BJS regarding improved documentation and future research that grew out of the panel’s review of the NCVS. With regard to research, we consider the unaddressed topic of child victimization.
The error review contained in Chapters 7 and 8 and summarized above identifies potential errors on the NCVS. The panel did not have the time or resources to conduct a complete error profile of the NCVS, which would include measuring the levels of error and their effects on the estimation of rape and sexual assault. The panel recommends this action to BJS.3
RECOMMENDATION 9-1 The Bureau of Justice Statistics should conduct an in-depth total error profile of the National Crime Victimization Survey, specifically focusing on estimation of rape and sexual assault. This profile should be made available to public data users.
Throughout its work, the panel found that the publicly available documentation was sometimes difficult to locate on the BJS website, and the major methodology document was several years out of date. The age and lack of clarity of existing documentation of the NCVS inhibits a complete and accurate understanding of the NCVS methodology.
RECOMMENDATION 9-2 The Bureau of Justice Statistics should provide complete documentation of the methodology in its surveys, details of survey response, costs, and the individual components of total survey error. The documentation should be made publicly available and easy to access.
Although BJS has commissioned a number of research studies over the past 10 years and many of the final reports of those studies are on the BJS website, they are inadequately organized to allow data users easy access. An interested data user has to know that a particular research study was conducted and then do a “hit or miss” search to see whether a research report on that topic was posted. In addition, the document Survey Methodology for Criminal Victimization in the United States (Bureau of Justice Statistics, 2008b) has been periodically updated, but the most recent update appears to be 2008.
________________
3To provide guidance in implementing this recommendation, here are some examples of error profiles in the literature: Brooks and Bailar (1978); Chakrabarty and Torres (1996); Doyle and Clark (2001); Jabine, King, and Petroni (1990); Kalton et al. (2000); and U.S. Energy Information Administration, 1996. Kasprzyk and Kalton (2001) review the use of quality profiles in U.S. statistical agencies and discuss their strengths and weaknesses for survey improvement purposes.
RECOMMENDATION 9-3 The Bureau of Justice Statistics should clearly lay out its research agenda on the agency’s website and link the agenda topics to completed reports. For research in progress, the website should include information about what is being done and when a report is expected.
The NCVS and this report do not address the victimization of children under the age of 12 years, a serious and neglected problem in the overall measurement of sexual victimization. There is a partial accounting of the victimization of children because several of the public health—oriented surveys that measure lifetime prevalence of rape and sexual assault have asked respondents their age when first victimized. For example, the National Intimate Partner and Sexual Violence Survey estimated that of individuals who reported having been raped in their lifetime, 12.3 percent of females and 27.8 percent of males were first raped before the age of 12 (Black et al., 2011).
The panel believes that the victimization of young children is a serious problem that requires further attention. No ongoing survey captures this information, with child protective services providing the major source of information on these victimizations. Unfortunately, this situation is similar to when the Uniform Crime Reports was the single source reporting on the sexual victimization of adults, with resulting concerns about underreporting to authorities. However, the issues associated with interviewing children mean that the measurement of this type of victimization needs a specialized approach—it cannot be measured through the same vehicle and the same methods that are used for adults and older children. The panel did not have the time and resources to pursue this extensive area of research. Instead, it recommends that BJS explore options to measure these sexual victimizations in the future. This might include the inclusion of retrospective questions to adults about childhood victimizations.
RECOMMENDATION 9-4: The Bureau of Justice Statistics should begin research to explore options for measuring the incidence of rape and sexual assault of children younger than 12.