National Academies Press: OpenBook

Estimating the Incidence of Rape and Sexual Assault (2014)

Chapter: 9 Synopsis of Potential Errors in the National Crime Victimization Survey

« Previous: 8 Potential Sources of Error: Nonresponse, Specification, and Measurement
Suggested Citation:"9 Synopsis of Potential Errors in the National Crime Victimization Survey." National Research Council. 2014. Estimating the Incidence of Rape and Sexual Assault. Washington, DC: The National Academies Press. doi: 10.17226/18605.
×

9


Synopsis of Potential Errors in the
National Crime Victimization Survey

The mission of the Bureau of Justice Statistics (BJS) is “to collect, analyze, publish, and disseminate information on crime, criminal offenders, victims of crime, and the operation of justice systems at all levels of government” (Bureau of Justice Statistics, n.d.-c). A major tool in accomplishing this mission is the National Crime Victimization Survey (NCVS), which provides a rich source of information on criminal victimizations, the victims, and the specifics of their criminal victimizations. In a 2008 report, the National Research Council found that the NCVS provided the only annual source of national data on criminal victimization with detail at the incident level. The report’s first recommendation (3-1) was that “BJS must ensure that the nation has quality annual estimates of levels and changes in criminal victimization” (National Research Council, 2008, p. 3). This panel strongly concurs with that recommendation. To further that mission, this panel has focused specifically on the quality of the measurements of rape and sexual assault with the purpose of helping improve quality in BJS’s estimates of those types of criminal victimizations.

The NCVS has both sampling and nonsampling error, as does every survey. The panel examined the error structure of the NCVS (see Chapters 7 and 8) in considerable detail with the intent of identifying areas that might be particular problems. We reiterate that the NCVS is an omnibus survey with the primary goal of estimating many types of criminal victimization. Its survey design is geared toward this broad goal, and it does not and cannot incorporate separate features for measuring each different type of victimization. Yet the survey’s basic goal and design create problems for measuring rape and sexual assault. This chapter summarizes the discussions

Suggested Citation:"9 Synopsis of Potential Errors in the National Crime Victimization Survey." National Research Council. 2014. Estimating the Incidence of Rape and Sexual Assault. Washington, DC: The National Academies Press. doi: 10.17226/18605.
×

in Chapters 7 and 8 that the panel believes are most problematic for measuring rape and sexual assault. The chapter concludes with several specific recommendations. The bulk of the panel’s recommendations are included in Chapter 10.

POTENTIAL SOURCES OF ERROR: SUMMARY

The NCVS uses a classical area sampling design based on selection in multiple stages to create a sample of housing units. This basic design is used for a number of general household surveys conducted by the Census Bureau, and it is a good design for an omnibus type survey. For estimates of rape and sexual assault, however, a limitation of the NCVS sample design is its current inability to control the sampling error for estimates of these low-incidence events. Although the sample size of the NCVS is large, rape and sexual assault are low-incidence events and the size of the estimated coefficients of variation (CVs)1 for those estimates at the national level allows considerable year-to-year fluctuation (see Chapter 7). In addition, also because of the low incidence of rape and sexual assault, BJS has to pool multiple years of data to get estimates of rape and sexual assault for important subpopulations (e.g., by race, gender, marital status, or age group).

In 2011, BJS changed its process for counting series victimizations. The effect of the change was substantial for estimates of rape and sexual assault: these victimizations were undercounted in the past, but series victimizations (based on only a few reports) now account for almost 40 percent of the national estimates of rape and sexual assault. The handling of these outlier reports has created additional concern about year-to-year fluctuation in the estimates of rape and sexual assault.

The overall response rates on the NCVS have remained fairly high, and there is little item nonresponse coded for specific rape and sexual assault screener questions. However, the panel is concerned about panel attrition, particularly because those individuals most likely to drop out—younger people and individuals not living as a couple—may be at greater risk for rape and sexual assault. The panel’s judgment is that individuals who respond on the NCVS but do not want to report a specific victimization will refuse the question indirectly by responding that no victimization occurred. This item nonresponse would then be coded as a legitimate zero.

Although BJS clearly defines what it intends to measure regarding rape and sexual assault, because of the omnibus design of the screener, these concepts are poorly translated through the data collection instruments. The in-

________________

1The CV is the standard error of a survey estimate divided by the estimate itself (expressed as a percent). It provides a relative measure of the sampling error associated with survey estimates.

Suggested Citation:"9 Synopsis of Potential Errors in the National Crime Victimization Survey." National Research Council. 2014. Estimating the Incidence of Rape and Sexual Assault. Washington, DC: The National Academies Press. doi: 10.17226/18605.
×

struments use oversimplified terms that can be interpreted in different ways. This situation creates the potential for specification error in the NCVS.

Several issues with the NCVS could lead to measurement error associated with the questionnaire, the data collection mode, and the interaction between interviewers and respondents. These issues include comprehension of key questions and respondents’ willingness to answer certain questions. The latter could be affected by the mode of data collection and interviewer gender. There continues to be inadequate training of interviewers and monitoring of the interviews on the NCVS.

All surveys have error associated with their processing. BJS publishes little information about its processing methods or errors found. The lack of such transparency makes it difficult for NCVS data users to fully understand NCVS estimates and their limitations.

OBSTACLES TO HIGH-QUALITY ESTIMATES

Based on the potential errors in the NCVS discussed in Chapters 7 and 8 and summarized above, the panel identified four major obstacles for accurately estimating incidences of rape and sexual assault.

1.   a sample design that is inefficient for measuring these low-incidence events,

2.   the context of “crime” that defines the survey,

3.   a lack of privacy for respondents in completing the survey, and

4.   the use of words with ambiguous meaning for key measures in the questionnaire.

These obstacles form the basis for the panel recommendations in Chapter 10 to create a separate survey to measure rape and sexual assault.

Inefficiency of the Sample Design

The NCVS is a general criminal victimization survey that targets the noninstitutionalized population of the United States with questions about different types of criminal victimizations, including crimes against property and people. Of the 19 million criminal victimizations identified through the NCVS in 2011, 64 percent were property crimes, which are dominated by theft; 29 percent were violent crimes (against people) but not classified as serious; and the remaining 7 percent of criminal victimizations were classified as serious violent crimes,2 of which only 1 of the percentage points

________________

2Serious violent crimes include robbery and aggravated assault, but not murder and kidnapping, which are not estimated through the NCVS.

Suggested Citation:"9 Synopsis of Potential Errors in the National Crime Victimization Survey." National Research Council. 2014. Estimating the Incidence of Rape and Sexual Assault. Washington, DC: The National Academies Press. doi: 10.17226/18605.
×

image

FIGURE 9-1 Criminal victimizations by type, NCVS 2011.
SOURCE: Data from Criminal Victimization 2011 (Bureau of Justice Statistics, 2012a).

was rape and sexual assault (see Figure 9-1). Thus, from a statistical perspective, rape and sexual assaults are statistically rare incidents.

The rarity of an attribute in a population presents unique challenges to efficient and effective sampling to estimate the proportion of the population with that attribute. In this situation, the difficulty can be seen by observing the CVs for various types of criminal victimizations, with a lower number meaning a more precise estimate (see footnote 1). Nationally, in 2011, the CV for property crime was 2 percent, for serious violent crime it was 7 percent, and for rape and sexual assault it was 14 percent (Bureau of Justice Statistics, 2012a). The NCVS sample design is not well suited to measure the uncommon attribute of having experienced a rape or sexual assault in the past 6 months (see Conclusion 7-1 in Chapter 7).

This concern is intrinsic to the basic design of the NCVS. BJS and the Census Bureau could address this issue by implementing a very large increase to the existing sample size of the NCVS or by modifying the NCVS sample design to focus more efficiently on subpopulations at greater risk for sexual victimization. (See Chapter 10 for a discussion of such a focused design.) However, the NCVS is an omnibus victimization survey whose goal is to measure a broad range of criminal victimizations. If it were to focus on the subpopulation at risk for a specific type of victimization (rape and sexual assault), then it would likely make the survey design much less efficient for measuring other types of criminal victimizations across the whole survey population.

Suggested Citation:"9 Synopsis of Potential Errors in the National Crime Victimization Survey." National Research Council. 2014. Estimating the Incidence of Rape and Sexual Assault. Washington, DC: The National Academies Press. doi: 10.17226/18605.
×

Criminal Context of the Survey

The NCVS is a criminal victimization survey. It is introduced that way to household members, and the questionnaire goes through a listing of different types of crimes, asking each respondent whether any of these crimes has happened to him or her. As discussed in Chapter 8, victims may not always think of a rape or sexual assault as a crime, particularly if the respondent knows the offender. In addition, some victims may fear disclosure to police and may associate a government crime survey too closely with law enforcement. The panel concludes that the context of a criminal victimization survey is a major barrier to accurate reporting of incidents of sexual victimizations (see Conclusion 8-7 in Chapter 8).

BJS could address this issue by moving the measurement of rape and sexual assault to a separate survey (as recommended in Chapter 10) or by changing the entire context of the NCVS to something more neutral. The modification of the NCVS into a survey that has a “neutral context” would entail a major redesign of that survey (including its name), and such a change would undermine other goals of the survey. The context of criminal victimization appears to work well for reporting other, less sensitive, victimizations. A redesign of the NCVS to remove the “criminal context” on this large omnibus survey would make it difficult to effectively communicate the purpose of the survey to respondents, and it would likely negatively affect the reporting of many other types of victimizations.

Lack of Privacy in Responding

Providing a respondent with privacy is an important prerequisite for any survey that deals with sensitive questions. Privacy, specifically from other household members, is critical for accurately responding to inquiries about rape and sexual assault, in part because the victim often knows the offender. In fact, the offender may be a household member. The current NCVS data collection protocols do not provide sufficient privacy (see Conclusion 8-8 in Chapter 8). Some privacy gains would be possible on the NCVS by switching to a self-administered mode of data collection. This switch would be a major change for the survey, and it would likely have both beneficial and detrimental effects on collecting information about other types of criminal victimizations.

The panel concludes that such a change of data collection mode is important to make, but that this change alone would still not provide the level of privacy that is needed for reporting sexual victimizations. As long as multiple members of the household are interviewed with an identical questionnaire, an adequate level of privacy is unlikely to be obtained. Each household member knows the questions that are asked on the survey and

Suggested Citation:"9 Synopsis of Potential Errors in the National Crime Victimization Survey." National Research Council. 2014. Estimating the Incidence of Rape and Sexual Assault. Washington, DC: The National Academies Press. doi: 10.17226/18605.
×

the duration of the interview based upon their own responses. An individual household respondent may continue to have concerns about reporting an incident of rape or sexual assault under those circumstances. To achieve the privacy level needed for questions regarding rape and sexual assault, the survey protocol would also need to change to targeting only a single person in each household or to deploy a “matrix” design in which each household member would be screened for different types of criminal victimizations. If implemented for the NCVS as a whole, then either approach would severely reduce the number of responses obtained for each specific type of victimization. Without an accompanying increase in sample size, this change would negatively affect the precision for estimates of all types of victimizations.

Ambiguous Terms in the Questionnaire

The current NCVS screening questionnaire uses a series of cues to aid respondents in remembering past victimizations. As part of those cues, it uses the terms rape and sexual assault without any further elaboration as to what those terms mean. Although BJS has precise meanings for these terms, they are not provided to respondents. It is not reasonable to assume that all individual respondents will interpret these words identically or as BJS anticipates (see Conclusion 8-6 in Chapter 8). More behaviorally specific words that describe a specific set of actions would lead to more accurate responses on these questions.

Such a behavioral approach is not completely alien to the NCVS. Some of the screening questions for assault do use some behaviorally specific language: a respondent is not asked whether he or she has been assaulted, but is asked questions that describe specific behaviors—“Has anyone attacked or threatened you … with anything like a baseball bat, frying pan, scissors, or stick?” It would be possible to make behaviorally specific changes on the current NCVS regarding rape and sexual assault with only a modest effect on other parts of the questionnaire. However, there would need to be specific screening for rape and sexual assault that is separate from and in addition to the more generalized cuing sequences currently used in the screener. Another difficulty is that the NCVS interviews children as young as 12 years of age, and there may be parental objections to the behaviorally specific questions. With a separate survey as recommended by the panel, there would be more options available to BJS on handling respondents between the ages of 12 and 17.

GENERAL RECOMMENDATIONS

The majority of the panel’s recommendations specifically address new directions for measuring rape and sexual assault (see Chapter 10). In this

Suggested Citation:"9 Synopsis of Potential Errors in the National Crime Victimization Survey." National Research Council. 2014. Estimating the Incidence of Rape and Sexual Assault. Washington, DC: The National Academies Press. doi: 10.17226/18605.
×

section, the panel offers several general recommendations to BJS regarding improved documentation and future research that grew out of the panel’s review of the NCVS. With regard to research, we consider the unaddressed topic of child victimization.

The error review contained in Chapters 7 and 8 and summarized above identifies potential errors on the NCVS. The panel did not have the time or resources to conduct a complete error profile of the NCVS, which would include measuring the levels of error and their effects on the estimation of rape and sexual assault. The panel recommends this action to BJS.3

RECOMMENDATION 9-1 The Bureau of Justice Statistics should conduct an in-depth total error profile of the National Crime Victimization Survey, specifically focusing on estimation of rape and sexual assault. This profile should be made available to public data users.

Throughout its work, the panel found that the publicly available documentation was sometimes difficult to locate on the BJS website, and the major methodology document was several years out of date. The age and lack of clarity of existing documentation of the NCVS inhibits a complete and accurate understanding of the NCVS methodology.

RECOMMENDATION 9-2 The Bureau of Justice Statistics should provide complete documentation of the methodology in its surveys, details of survey response, costs, and the individual components of total survey error. The documentation should be made publicly available and easy to access.

Although BJS has commissioned a number of research studies over the past 10 years and many of the final reports of those studies are on the BJS website, they are inadequately organized to allow data users easy access. An interested data user has to know that a particular research study was conducted and then do a “hit or miss” search to see whether a research report on that topic was posted. In addition, the document Survey Methodology for Criminal Victimization in the United States (Bureau of Justice Statistics, 2008b) has been periodically updated, but the most recent update appears to be 2008.

________________

3To provide guidance in implementing this recommendation, here are some examples of error profiles in the literature: Brooks and Bailar (1978); Chakrabarty and Torres (1996); Doyle and Clark (2001); Jabine, King, and Petroni (1990); Kalton et al. (2000); and U.S. Energy Information Administration, 1996. Kasprzyk and Kalton (2001) review the use of quality profiles in U.S. statistical agencies and discuss their strengths and weaknesses for survey improvement purposes.

Suggested Citation:"9 Synopsis of Potential Errors in the National Crime Victimization Survey." National Research Council. 2014. Estimating the Incidence of Rape and Sexual Assault. Washington, DC: The National Academies Press. doi: 10.17226/18605.
×

RECOMMENDATION 9-3 The Bureau of Justice Statistics should clearly lay out its research agenda on the agency’s website and link the agenda topics to completed reports. For research in progress, the website should include information about what is being done and when a report is expected.

The NCVS and this report do not address the victimization of children under the age of 12 years, a serious and neglected problem in the overall measurement of sexual victimization. There is a partial accounting of the victimization of children because several of the public health—oriented surveys that measure lifetime prevalence of rape and sexual assault have asked respondents their age when first victimized. For example, the National Intimate Partner and Sexual Violence Survey estimated that of individuals who reported having been raped in their lifetime, 12.3 percent of females and 27.8 percent of males were first raped before the age of 12 (Black et al., 2011).

The panel believes that the victimization of young children is a serious problem that requires further attention. No ongoing survey captures this information, with child protective services providing the major source of information on these victimizations. Unfortunately, this situation is similar to when the Uniform Crime Reports was the single source reporting on the sexual victimization of adults, with resulting concerns about underreporting to authorities. However, the issues associated with interviewing children mean that the measurement of this type of victimization needs a specialized approach—it cannot be measured through the same vehicle and the same methods that are used for adults and older children. The panel did not have the time and resources to pursue this extensive area of research. Instead, it recommends that BJS explore options to measure these sexual victimizations in the future. This might include the inclusion of retrospective questions to adults about childhood victimizations.

RECOMMENDATION 9-4: The Bureau of Justice Statistics should begin research to explore options for measuring the incidence of rape and sexual assault of children younger than 12.

Suggested Citation:"9 Synopsis of Potential Errors in the National Crime Victimization Survey." National Research Council. 2014. Estimating the Incidence of Rape and Sexual Assault. Washington, DC: The National Academies Press. doi: 10.17226/18605.
×
Page 153
Suggested Citation:"9 Synopsis of Potential Errors in the National Crime Victimization Survey." National Research Council. 2014. Estimating the Incidence of Rape and Sexual Assault. Washington, DC: The National Academies Press. doi: 10.17226/18605.
×
Page 154
Suggested Citation:"9 Synopsis of Potential Errors in the National Crime Victimization Survey." National Research Council. 2014. Estimating the Incidence of Rape and Sexual Assault. Washington, DC: The National Academies Press. doi: 10.17226/18605.
×
Page 155
Suggested Citation:"9 Synopsis of Potential Errors in the National Crime Victimization Survey." National Research Council. 2014. Estimating the Incidence of Rape and Sexual Assault. Washington, DC: The National Academies Press. doi: 10.17226/18605.
×
Page 156
Suggested Citation:"9 Synopsis of Potential Errors in the National Crime Victimization Survey." National Research Council. 2014. Estimating the Incidence of Rape and Sexual Assault. Washington, DC: The National Academies Press. doi: 10.17226/18605.
×
Page 157
Suggested Citation:"9 Synopsis of Potential Errors in the National Crime Victimization Survey." National Research Council. 2014. Estimating the Incidence of Rape and Sexual Assault. Washington, DC: The National Academies Press. doi: 10.17226/18605.
×
Page 158
Suggested Citation:"9 Synopsis of Potential Errors in the National Crime Victimization Survey." National Research Council. 2014. Estimating the Incidence of Rape and Sexual Assault. Washington, DC: The National Academies Press. doi: 10.17226/18605.
×
Page 159
Suggested Citation:"9 Synopsis of Potential Errors in the National Crime Victimization Survey." National Research Council. 2014. Estimating the Incidence of Rape and Sexual Assault. Washington, DC: The National Academies Press. doi: 10.17226/18605.
×
Page 160
Next: 10 New Directions for Measuring Rape and Sexual Assault »
Estimating the Incidence of Rape and Sexual Assault Get This Book
×
Buy Paperback | $54.00 Buy Ebook | $43.99
MyNAP members save 10% online.
Login or Register to save!
Download Free PDF

The Bureau of Justice Statistics' (BJS) National Crime Victimization Survey (NCVS) measures the rates at which Americans are victims of crimes, including rape and sexual assault, but there is concern that rape and sexual assault are undercounted on this survey. BJS asked the National Research Council to investigate this issue and recommend best practices for measuring rape and sexual assault on their household surveys. Estimating the Incidence of Rape and Sexual Assault concludes that it is likely that the NCVS is undercounting rape and sexual assault. The most accurate counts of rape and sexual assault cannot be achieved without measuring them separately from other victimizations, the report says. It recommends that BJS develop a separate survey for measuring rape and sexual assault. The new survey should more precisely define ambiguous words such as "rape," give more privacy to respondents, and take other steps that would improve the accuracy of responses. Estimating the Incidence of Rape and Sexual Assault takes a fresh look at the problem of measuring incidents of rape and sexual assault from the criminal justice perspective. This report examines issues such as the legal definitions in use by the states for these crimes, best methods for representing the definitions in survey instruments so that their meaning is clear to respondents, and best methods for obtaining as complete reporting as possible of these crimes in surveys, including methods whereby respondents may report anonymously.

Rape and sexual assault are among the most injurious crimes a person can inflict on another. The effects are devastating, extending beyond the initial victimization to consequences such as unwanted pregnancy, sexually transmitted infections, sleep and eating disorders, and other emotional and physical problems. Understanding the frequency and context under which rape and sexual assault are committed is vital in directing resources for law enforcement and support for victims. These data can influence public health and mental health policies and help identify interventions that will reduce the risk of future attacks. Sadly, accurate information about the extent of sexual assault and rape is difficult to obtain because most of these crimes go unreported to police. Estimating the Incidence of Rape and Sexual Assault focuses on methodology and vehicles used to measure rape and sexual assaults, reviews potential sources of error within the NCVS survey, and assesses the training and monitoring of interviewers in an effort to improve reporting of these crimes.

  1. ×

    Welcome to OpenBook!

    You're looking at OpenBook, NAP.edu's online reading room since 1999. Based on feedback from you, our users, we've made some improvements that make it easier than ever to read thousands of publications on our website.

    Do you want to take a quick tour of the OpenBook's features?

    No Thanks Take a Tour »
  2. ×

    Show this book's table of contents, where you can jump to any chapter by name.

    « Back Next »
  3. ×

    ...or use these buttons to go back to the previous chapter or skip to the next one.

    « Back Next »
  4. ×

    Jump up to the previous page or down to the next one. Also, you can type in a page number and press Enter to go directly to that page in the book.

    « Back Next »
  5. ×

    Switch between the Original Pages, where you can read the report as it appeared in print, and Text Pages for the web version, where you can highlight and search the text.

    « Back Next »
  6. ×

    To search the entire text of this book, type in your search term here and press Enter.

    « Back Next »
  7. ×

    Share a link to this book page on your preferred social network or via email.

    « Back Next »
  8. ×

    View our suggested citation for this chapter.

    « Back Next »
  9. ×

    Ready to take your reading offline? Click here to buy this book in print or download it as a free PDF, if available.

    « Back Next »
Stay Connected!