9


Synopsis of Potential Errors in the
National Crime Victimization Survey

The mission of the Bureau of Justice Statistics (BJS) is “to collect, analyze, publish, and disseminate information on crime, criminal offenders, victims of crime, and the operation of justice systems at all levels of government” (Bureau of Justice Statistics, n.d.-c). A major tool in accomplishing this mission is the National Crime Victimization Survey (NCVS), which provides a rich source of information on criminal victimizations, the victims, and the specifics of their criminal victimizations. In a 2008 report, the National Research Council found that the NCVS provided the only annual source of national data on criminal victimization with detail at the incident level. The report’s first recommendation (3-1) was that “BJS must ensure that the nation has quality annual estimates of levels and changes in criminal victimization” (National Research Council, 2008, p. 3). This panel strongly concurs with that recommendation. To further that mission, this panel has focused specifically on the quality of the measurements of rape and sexual assault with the purpose of helping improve quality in BJS’s estimates of those types of criminal victimizations.

The NCVS has both sampling and nonsampling error, as does every survey. The panel examined the error structure of the NCVS (see Chapters 7 and 8) in considerable detail with the intent of identifying areas that might be particular problems. We reiterate that the NCVS is an omnibus survey with the primary goal of estimating many types of criminal victimization. Its survey design is geared toward this broad goal, and it does not and cannot incorporate separate features for measuring each different type of victimization. Yet the survey’s basic goal and design create problems for measuring rape and sexual assault. This chapter summarizes the discussions



The National Academies | 500 Fifth St. N.W. | Washington, D.C. 20001
Copyright © National Academy of Sciences. All rights reserved.
Terms of Use and Privacy Statement



Below are the first 10 and last 10 pages of uncorrected machine-read text (when available) of this chapter, followed by the top 30 algorithmically extracted key phrases from the chapter as a whole.
Intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text on the opening pages of each chapter. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.

Do not use for reproduction, copying, pasting, or reading; exclusively for search engines.

OCR for page 153
9 Synopsis of Potential Errors in the National Crime Victimization Survey T he mission of the Bureau of Justice Statistics (BJS) is “to collect, analyze, publish, and disseminate information on crime, criminal offenders, victims of crime, and the operation of justice systems at all levels of government” (Bureau of Justice Statistics, n.d.-c). A major tool in accomplishing this mission is the National Crime Victimization Survey (NCVS), which provides a rich source of information on criminal victim- izations, the victims, and the specifics of their criminal victimizations. In a 2008 report, the National Research Council found that the NCVS provided the only annual source of national data on criminal victimization with de- tail at the incident level. The report’s first recommendation (3-1) was that “BJS must ensure that the nation has quality annual estimates of levels and changes in criminal victimization” (National Research Council, 2008, p. 3). This panel strongly concurs with that recommendation. To further that mis- sion, this panel has focused specifically on the quality of the measurements of rape and sexual assault with the purpose of helping improve quality in BJS’s estimates of those types of criminal victimizations. The NCVS has both sampling and nonsampling error, as does every survey. The panel examined the error structure of the NCVS (see Chapters 7 and 8) in considerable detail with the intent of identifying areas that might be particular problems. We reiterate that the NCVS is an omnibus survey with the primary goal of estimating many types of criminal victimization. Its survey design is geared toward this broad goal, and it does not and cannot incorporate separate features for measuring each different type of victimization. Yet the survey’s basic goal and design create problems for measuring rape and sexual assault. This chapter summarizes the discussions 153

OCR for page 153
154 ESTIMATING THE INCIDENCE OF RAPE AND SEXUAL ASSAULT in Chapters 7 and 8 that the panel believes are most problematic for mea- suring rape and sexual assault. The chapter concludes with several specific recommendations. The bulk of the panel’s recommendations are included in Chapter 10. POTENTIAL SOURCES OF ERROR: SUMMARY The NCVS uses a classical area sampling design based on selection in multiple stages to create a sample of housing units. This basic design is used for a number of general household surveys conducted by the Census Bureau, and it is a good design for an omnibus type survey. For estimates of rape and sexual assault, however, a limitation of the NCVS sample design is its current inability to control the sampling error for estimates of these low-incidence events. Although the sample size of the NCVS is large, rape and sexual assault are low-incidence events and the size of the estimated coefficients of variation (CVs)1 for those estimates at the national level al- lows considerable year-to-year fluctuation (see Chapter 7). In addition, also because of the low incidence of rape and sexual assault, BJS has to pool multiple years of data to get estimates of rape and sexual assault for im- portant subpopulations (e.g., by race, gender, marital status, or age group). In 2011, BJS changed its process for counting series victimizations. The effect of the change was substantial for estimates of rape and sexual assault: these victimizations were undercounted in the past, but series victimizations (based on only a few reports) now account for almost 40 percent of the national estimates of rape and sexual assault. The handling of these outlier reports has created additional concern about year-to-year fluctuation in the estimates of rape and sexual assault. The overall response rates on the NCVS have remained fairly high, and there is little item nonresponse coded for specific rape and sexual assault screener questions. However, the panel is concerned about panel attrition, particularly because those individuals most likely to drop out—younger people and individuals not living as a couple—may be at greater risk for rape and sexual assault. The panel’s judgment is that individuals who re- spond on the NCVS but do not want to report a specific victimization will refuse the question indirectly by responding that no victimization occurred. This item nonresponse would then be coded as a legitimate zero. Although BJS clearly defines what it intends to measure regarding rape and sexual assault, because of the omnibus design of the screener, these con- cepts are poorly translated through the data collection instruments. The in- 1 The CV is the standard error of a survey estimate divided by the estimate itself (expressed as a percent). It provides a relative measure of the sampling error associated with survey estimates.

OCR for page 153
SYNOPSIS OF POTENTIAL ERRORS 155 struments use oversimplified terms that can be interpreted in different ways. This situation creates the potential for specification error in the NCVS. Several issues with the NCVS could lead to measurement error associ- ated with the questionnaire, the data collection mode, and the interaction between interviewers and respondents. These issues include comprehension of key questions and respondents’ willingness to answer certain questions. The latter could be affected by the mode of data collection and interviewer gender. There continues to be inadequate training of interviewers and moni- toring of the interviews on the NCVS. All surveys have error associated with their processing. BJS publishes little information about its processing methods or errors found. The lack of such transparency makes it difficult for NCVS data users to fully under- stand NCVS estimates and their limitations. OBSTACLES TO HIGH-QUALITY ESTIMATES Based on the potential errors in the NCVS discussed in Chapters 7 and 8 and summarized above, the panel identified four major obstacles for ac- curately estimating incidences of rape and sexual assault. 1. a sample design that is inefficient for measuring these low-incidence events, 2. the context of “crime” that defines the survey, 3. a lack of privacy for respondents in completing the survey, and 4. the use of words with ambiguous meaning for key measures in the questionnaire. These obstacles form the basis for the panel recommendations in Chapter 10 to create a separate survey to measure rape and sexual assault. Inefficiency of the Sample Design The NCVS is a general criminal victimization survey that targets the noninstitutionalized population of the United States with questions about different types of criminal victimizations, including crimes against property and people. Of the 19 million criminal victimizations identified through the NCVS in 2011, 64 percent were property crimes, which are dominated by theft; 29 percent were violent crimes (against people) but not classified as serious; and the remaining 7 percent of criminal victimizations were clas- sified as serious violent crimes,2 of which only 1 of the percentage points 2  Seriousviolent crimes include robbery and aggravated assault, but not murder and kidnap- ping, which are not estimated through the NCVS.

OCR for page 153
156 ESTIMATING THE INCIDENCE OF RAPE AND SEXUAL ASSAULT Serious Violent Rape & Sexual Crimes, Except Assault, 1% Rape & Sexual Assault, 6% Property Violent Crimes, Crimes, 64% Not Serious, 29% FIGURE 9-1  Criminal victimizations by type, NCVS 2011. SOURCE: Data from Criminal Victimization 2011 (Bureau of Justice Statistics, 2012a). was rape and sexual assault (see Figure 9-1). Thus, from a statistical per- spective, rape and sexual assaults are statistically rare incidents. The rarity of an attribute in a population presents unique challenges to efficient and effective sampling to estimate the proportion of the population with that attribute. In this situation, the difficulty can be seen by observing the CVs for various types of criminal victimizations, with a lower number meaning a more precise estimate (see footnote 1). Nationally, in 2011, the CV for property crime was 2 percent, for serious violent crime it was 7 percent, and for rape and sexual assault it was 14 percent (Bureau of Justice Statistics, 2012a). The NCVS sample design is not well suited to measure the uncommon attribute of having experienced a rape or sexual assault in the past 6 months (see Conclusion 7-1 in Chapter 7). This concern is intrinsic to the basic design of the NCVS. BJS and the Census Bureau could address this issue by implementing a very large in- crease to the existing sample size of the NCVS or by modifying the NCVS sample design to focus more efficiently on subpopulations at greater risk for sexual victimization. (See Chapter 10 for a discussion of such a focused design.) However, the NCVS is an omnibus victimization survey whose goal is to measure a broad range of criminal victimizations. If it were to focus on the subpopulation at risk for a specific type of victimization (rape and sexual assault), then it would likely make the survey design much less ef- ficient for measuring other types of criminal victimizations across the whole survey population.

OCR for page 153
SYNOPSIS OF POTENTIAL ERRORS 157 Criminal Context of the Survey The NCVS is a criminal victimization survey. It is introduced that way to household members, and the questionnaire goes through a listing of dif- ferent types of crimes, asking each respondent whether any of these crimes has happened to him or her. As discussed in Chapter 8, victims may not always think of a rape or sexual assault as a crime, particularly if the re- spondent knows the offender. In addition, some victims may fear disclosure to police and may associate a government crime survey too closely with law enforcement. The panel concludes that the context of a criminal victimiza- tion survey is a major barrier to accurate reporting of incidents of sexual victimizations (see Conclusion 8-7 in Chapter 8). BJS could address this issue by moving the measurement of rape and sexual assault to a separate survey (as recommended in Chapter 10) or by changing the entire context of the NCVS to something more neutral. The modification of the NCVS into a survey that has a “neutral context” would entail a major redesign of that survey (including its name), and such a change would undermine other goals of the survey. The context of criminal victimization appears to work well for reporting other, less sensitive, victim- izations. A redesign of the NCVS to remove the “criminal context” on this large omnibus survey would make it difficult to effectively communicate the purpose of the survey to respondents, and it would likely negatively affect the reporting of many other types of victimizations. Lack of Privacy in Responding Providing a respondent with privacy is an important prerequisite for any survey that deals with sensitive questions. Privacy, specifically from other household members, is critical for accurately responding to inquiries about rape and sexual assault, in part because the victim often knows the offender. In fact, the offender may be a household member. The current NCVS data collection protocols do not provide sufficient privacy (see Con- clusion 8-8 in Chapter 8). Some privacy gains would be possible on the NCVS by switching to a self-administered mode of data collection. This switch would be a major change for the survey, and it would likely have both beneficial and detrimental effects on collecting information about other types of criminal victimizations. The panel concludes that such a change of data collection mode is important to make, but that this change alone would still not provide the level of privacy that is needed for reporting sexual victimizations. As long as multiple members of the household are interviewed with an identical questionnaire, an adequate level of privacy is unlikely to be obtained. Each household member knows the questions that are asked on the survey and

OCR for page 153
158 ESTIMATING THE INCIDENCE OF RAPE AND SEXUAL ASSAULT the duration of the interview based upon their own responses. An individ- ual household respondent may continue to have concerns about reporting an incident of rape or sexual assault under those circumstances. To achieve the privacy level needed for questions regarding rape and sexual assault, the survey protocol would also need to change to targeting only a single person in each household or to deploy a “matrix” design in which each household member would be screened for different types of criminal victimizations. If implemented for the NCVS as a whole, then either approach would severely reduce the number of responses obtained for each specific type of victimiza- tion. Without an accompanying increase in sample size, this change would negatively affect the precision for estimates of all types of victimizations. Ambiguous Terms in the Questionnaire The current NCVS screening questionnaire uses a series of cues to aid respondents in remembering past victimizations. As part of those cues, it uses the terms rape and sexual assault without any further elaboration as to what those terms mean. Although BJS has precise meanings for these terms, they are not provided to respondents. It is not reasonable to assume that all individual respondents will interpret these words identically or as BJS anticipates (see Conclusion 8-6 in Chapter 8). More behaviorally specific words that describe a specific set of actions would lead to more accurate responses on these questions. Such a behavioral approach is not completely alien to the NCVS. Some of the screening questions for assault do use some behaviorally specific lan- guage: a respondent is not asked whether he or she has been assaulted, but is asked questions that describe specific behaviors—“Has anyone attacked or threatened you . . . with anything like a baseball bat, frying pan, scis- sors, or stick?” It would be possible to make behaviorally specific changes on the current NCVS regarding rape and sexual assault with only a modest effect on other parts of the questionnaire. However, there would need to be specific screening for rape and sexual assault that is separate from and in addition to the more generalized cuing sequences currently used in the screener. Another difficulty is that the NCVS interviews children as young as 12 years of age, and there may be parental objections to the behaviorally specific questions. With a separate survey as recommended by the panel, there would be more options available to BJS on handling respondents between the ages of 12 and 17. GENERAL RECOMMENDATIONS The majority of the panel’s recommendations specifically address new directions for measuring rape and sexual assault (see Chapter 10). In this

OCR for page 153
SYNOPSIS OF POTENTIAL ERRORS 159 section, the panel offers several general recommendations to BJS regarding improved documentation and future research that grew out of the panel’s review of the NCVS. With regard to research, we consider the unaddressed topic of child victimization. The error review contained in Chapters 7 and 8 and summarized above identifies potential errors on the NCVS. The panel did not have the time or resources to conduct a complete error profile of the NCVS, which would include measuring the levels of error and their effects on the estimation of rape and sexual assault. The panel recommends this action to BJS.3 RECOMMENDATION 9-1 The Bureau of Justice Statistics should conduct an in-depth total error profile of the National Crime Victim- ization Survey, specifically focusing on estimation of rape and sexual assault. This profile should be made available to public data users. Throughout its work, the panel found that the publicly available docu- mentation was sometimes difficult to locate on the BJS website, and the major methodology document was several years out of date. The age and lack of clarity of existing documentation of the NCVS inhibits a complete and accurate understanding of the NCVS methodology. RECOMMENDATION 9-2 The Bureau of Justice Statistics should provide complete documentation of the methodology in its surveys, details of survey response, costs, and the individual components of total survey error. The documentation should be made publicly available and easy to access. Although BJS has commissioned a number of research studies over the past 10 years and many of the final reports of those studies are on the BJS website, they are inadequately organized to allow data users easy access. An interested data user has to know that a particular research study was conducted and then do a “hit or miss” search to see whether a research report on that topic was posted. In addition, the document Survey Meth- odology for Criminal Victimization in the United States (Bureau of Justice Statistics, 2008b) has been periodically updated, but the most recent update appears to be 2008. 3  To provide guidance in implementing this recommendation, here are some examples of error profiles in the literature: Brooks and Bailar (1978); Chakrabarty and Torres (1996); Doyle and Clark (2001); Jabine, King, and Petroni (1990); Kalton et al. (2000); and U.S. Energy Information Administration, 1996. Kasprzyk and Kalton (2001) review the use of quality profiles in U.S. statistical agencies and discuss their strengths and weaknesses for survey improvement purposes.

OCR for page 153
160 ESTIMATING THE INCIDENCE OF RAPE AND SEXUAL ASSAULT RECOMMENDATION 9-3 The Bureau of Justice Statistics should clearly lay out its research agenda on the agency’s website and link the agenda topics to completed reports. For research in progress, the website should include information about what is being done and when a report is expected. The NCVS and this report do not address the victimization of children under the age of 12 years, a serious and neglected problem in the overall measurement of sexual victimization. There is a partial accounting of the victimization of children because several of the public health–oriented sur- veys that measure lifetime prevalence of rape and sexual assault have asked respondents their age when first victimized. For example, the National Intimate Partner and Sexual Violence Survey estimated that of individuals who reported having been raped in their lifetime, 12.3 percent of females and 27.8 percent of males were first raped before the age of 12 (Black et al., 2011). The panel believes that the victimization of young children is a serious problem that requires further attention. No ongoing survey captures this information, with child protective services providing the major source of information on these victimizations. Unfortunately, this situation is similar to when the Uniform Crime Reports was the single source reporting on the sexual victimization of adults, with resulting concerns about underreporting to authorities. However, the issues associated with interviewing children mean that the measurement of this type of victimization needs a special- ized approach—it cannot be measured through the same vehicle and the same methods that are used for adults and older children. The panel did not have the time and resources to pursue this extensive area of research. Instead, it recommends that BJS explore options to measure these sexual victimizations in the future. This might include the inclusion of retrospec- tive questions to adults about childhood victimizations. RECOMMENDATION 9-4: The Bureau of Justice Statistics should begin research to explore options for measuring the incidence of rape and sexual assault of children younger than 12.