Rape and sexual assault are highly injurious victimizations, and accurate information about them is difficult to obtain because they are seriously underreported to law enforcement. The National Crime Victimization Survey (NCVS) measures these victimization rates,1 along with details on the victims, as part of its overall mission of measuring all criminal victimizations. However, data users have expressed concern that rape and sexual assault appear to be undercounted on the NCVS.
The Bureau of Justice Statistics (BJS), which has responsibility for the NCVS, has committed to a multiyear project to better understand the reasons for the possible underestimation of rape and sexual assault on the NCVS. As part of this effort, BJS asked the National Research Council, through its Committee on National Statistics, to convene an expert panel
_____________
1Throughout the report, we use two specific terms to discuss victimization rates of rape and sexual assault: incidence and prevalence. The incidence rate refers to the measure of the total number of incidents (or events) that occurred in a given period. It counts the total number of incidents or victimizations; it does not count the number of individual victims. In epidemiology, this rate is often referred to as the “event rate.” Incidence rates are generally calculated over a specific time period, such as 12 months. The prevalence rate refers to the number of victims. It counts the number of individuals who have been victimized at least once; it does not count the total number of incidents. Thus, a lifetime prevalence rate measures the number of individuals who have been raped or sexually assaulted at least once in their lifetimes. A 12-month prevalence rate would measure the number of individuals who had been raped or sexually assaulted at least once in that 12-month period. In epidemiology, the term incidence rate is often used to measure the number of “first time events,” which is what we are calling the prevalence rate.
to investigate these issues and recommend best practices for measuring rape and sexual assault on the NCVS and other BJS household surveys.
There are two quite different perspectives for the measurement of rape and sexual assault—the criminal justice perspective and the public health perspective. These different perspectives have led to methodological differences in designing and implementing surveys, which, in turn, have resulted in different estimates of the incidence rates. The NCVS reflects the criminal justice perspective, and its purpose is to measure criminal victimizations: “point-in-time” events that are judged to be criminal. In contrast, surveys that reflect the public health perspective look at victimization as a condition that endures over a period of time, and may not necessarily be criminal. These surveys are less focused on identifying point-in-time events.
The panel was formally charged to “assess the quality and relevance of statistics on rape and sexual assault from the NCVS and other surveys contracted for by other federal agencies as well as surveys conducted by private organizations,” examining issues such as the “legal definitions in use by the states for these crimes, best methods for representing the definitions in survey instruments so that their meaning is clear to respondents, and best methods for obtaining as complete reporting as possible of these crimes in surveys, including methods whereby respondents may report anonymously.” Thus, the panel took a fresh look at the problem of measuring incidents of rape and sexual assault from the criminal justice perspective, but the panel was not constrained to fit this measurement within the NCVS or to restrict its recommendations to specific methodologies that BJS has used in the past.
MEASUREMENT
The first part of this report focuses on methodology and vehicles used to measure rape and sexual assault. Looking first at legal definitions for these crimes, the panel found that there are considerable differences across jurisdictions. The differences include basic terminology, the level of “force” required before the victimization becomes criminal, and the concepts of “lack of consent” and the “capacity to consent.” Chapter 2 of this report provides details of what the panel learned about these legal definitions.
Along with these differences, the measurement of rape and sexual assault has been implemented in different venues, in different ways, using different definitions and different methodologies. The result has been different levels in the estimates. Chapters 3 through 6 in this report describe these different venues, with highlights below.
The Federal Bureau of Investigation (FBI) annually publishes Crime in the United States, which includes statistics from the Uniform Crime Reports (UCR) Summary Reporting System (SRS). The UCR SRS is based on
monthly crime counts, by type of crime, from approximately 17,000 law enforcement agencies around the country.
Until 2013, the FBI directed law enforcement agencies to report rape crimes using a restrictive definition established in 1929, “carnal knowledge of a female forcibly and against her will,” with a further explanation that “carnal knowledge” is penetration (however slight) of a penis into a vagina. Although a new definition for rape was established in 2013 for purposes of reporting in the UCR that is better aligned with current state and federal laws, there are still questions about whether the UCR can accurately capture all kinds of rape and sexual assault incidents. Two major concerns are that rape and sexual assault are underreported to law enforcement and sometimes downgraded by police.
The NCVS was established in part to provide another source of crime statistics beyond those supplied by police reports. It is a national household survey with the goal of obtaining information on a broad set of criminal victimizations (including rape and sexual assault) from the victims rather than law enforcement. The Census Bureau conducts it on an ongoing basis for BJS. The NCVS selects housing units through a multistage design that uses the infrastructure built for the decennial census. Individuals (12 years of age and older) residing at the selected housing units are interviewed. In 2011, the NCVS had reports from approximately 143,000 household members. Each address remains in the sample for 3 years, with interviews every 6 months.
BJS has established its own definitions of rape and sexual assault for estimation with the NCVS, broader than the one previously used by the FBI. In the NCVS, respondents are not asked to judge whether or not a crime has taken place but to report incidents in a number of categories. The NCVS estimated 217,331 rapes and sexual assaults in 2011 in the United States.
Users of NCVS data have expressed concern about potential underestimation of rape and sexual assault on the NCVS, in part because a number of other surveys have measured higher levels of those victimizations. Those independent surveys include (but are not limited to) the National Women’s Study (1989-1991); the National Violence Against Women Study (1995-1996); the National College Women Sexual Victimization Study (1997); and the National Intimate Partner and Sexual Violence Survey (2010). The surveys differ in many ways, including the definition used for rape; context in which data are collected; target population, sampling frame, and sample size; and data collection mode, response rates, and adjustments for nonresponse. Given these many differences, it is not surprising that the resulting estimates of rape and sexual assault are substantially different.
The panel found that a comparison across these sources of estimates of rape was particularly problematic because of the differences in the popula-
tions targeted, the definitions used, the data collection methodology, and the survey timing. The panel determined that it could not scientifically conclude which source was overall better, and it does not recommend any source as the best or as a standard. However, in reviewing all of this material, the panel judges that it is likely that the NCVS is undercounting rape and sexual assault victimization.
ASSESSMENT OF THE NCVS
All surveys are subject to errors, and the NCVS is no exception. An assessment of the errors and potential errors in a survey is important to understanding the overall quality of the estimates from that survey and to initiate improvements. Total survey error is a concept that involves a holistic view of all potential errors in a survey program, including both sampling error and various forms of nonsampling error.
The panel undertook an examination of the total error structure of the NCVS with the intent of identifying areas that were particularly problematic and that could contribute to underestimation of rape and sexual assault. This review of potential sources of error covered sampling error, frame error, processing error, nonresponse error, specification error, and measurement error. The panel also assessed the training and monitoring of interviewers for the NCVS.
Although it has identified areas where errors seem likely to occur, the panel, with limited time and resources, was not able to conduct a complete error profile of the NCVS. Such an error profile would measure the actual level of error and its impact on the estimation of rape and sexual assault. The panel encourages BJS to conduct an in-depth total error profile of the NCVS, to update and complete its documentation, and to make its research agenda and results more accessible and transparent to the public (Recommendations 9-1, 9-2, and 9-3 in Chapter 9).
It is important to note that the panel did not perform the same in-depth examination of the error structure of the other surveys for measuring rape and sexual assault because of limitations of time and resources. Presenting findings focused on the NCVS does not imply that the panel believes that the other surveys have fewer errors: the panel did not examine them carefully and so cannot draw overall conclusions about their error structures.
Sampling Error
The target population of the NCVS is the noninstitutionalized population of the United States, 12 years of age and older. One measure of sampling error, the coefficients of variation (CVs) for the number of rape and sexual assault victimizations, is approximately 14 percent at the national
level, with considerable year-to-year variation. Furthermore, the sampling errors for estimates of important subpopulations are quite large. As a result, BJS does not provide estimates for rape and sexual assault for those subpopulations, instead providing estimates only for the more aggregated category of serious violent crimes.
CONCLUSION 7-1 The National Crime Victimization Survey, which is designed as an omnibus victimization survey, is efficient in measuring the many types of criminal victimizations across the United States, but it does not measure the low incidence events of rape and sexual assault with the precision needed for policy and research purposes. Comparisons across subgroups and years are particularly problematic.
BJS made a major methodological change in 2011 in how the NCVS handles “series victimization.” Series victimization is defined as when a single respondent reports six or more separate but similar criminal victimizations over the reference period but is unable to recall each event individually or describe each one in detail to the interviewer. The old methodology suppressed these reports of multiple victimizations, contributing to the underestimation of rape and sexual assault. In 2011, the agency changed these procedures to count the number of reported victimizations in the series up to a maximum of 10. From a statistical point of view, the new series victimization procedures give the weighted outliers a very large impact on the estimates. The effect of this change increased the estimates of the incidence rate of rape and sexual assault by 55 percent and created more year-to-year fluctuations.
CONCLUSION 7-2 Records identified as series victimizations create an outlier problem in the estimation process for the National Crime Victimization Survey. The current method for handling series victimization, though an improvement over the method used until 2011, allows these relatively rare reports to have a large impact on the national estimates of rape and sexual assault and creates large year-to-year volatility.
Frame and Processing Error
Conducting a major household survey is complex. In the NCVS, as in any major survey, errors may arise in the process of constructing, maintaining, or sampling from a frame. All large household surveys, including the NCVS, use complex processes to edit, summarize, and publish data and the calculated estimates. The NCVS also includes a complex process for the classification of each reported victimization by type of crime. The panel
concludes that these survey processes contribute to errors in the NCVS estimates and that there is a lack of transparency in the edit and processing procedures (see Conclusions 7-3, 7-4, and 7-5 in Chapter 7).
Response Error
With regard to survey response, the NCVS has maintained a moderately high level of overall survey response. In 2011, the person-level response rate2 was 88 percent. These response rates have decreased several percentage points over the previous decade, but not substantially. A major question with regard to response rates is whether the nonresponse on the NCVS causes a bias in the estimates. Even though one major analysis of potential bias in the NCVS in 2009 found little evidence for nonresponse bias, the panel has some reservations and concludes that the NCVS may have a nonresponse bias related to estimates of sexual victimization.
CONCLUSION 8-1 The overall unit response rates, as calculated, on the National Crime Victimization Survey are moderately high and have been reasonably stable over the past 10 years. Although an independent analysis concluded that the methods that the Bureau of Justice Statistics uses to adjust for nonresponse appear to provide a satisfactory correction for nonresponse bias at the unit level, our panel has reservations about that analysis and remains concerned that there may be a nonresponse bias related to sexual victimization.
Because the NCVS is a panel survey with seven waves of data collection over 3 years, the panel was concerned about panel attrition: whether household members were less likely to respond or more likely to completely drop out over time. Because BJS does not provide NCVS response rates by wave, the panel calculated unweighted person-level response rates by time-in-sample. We found that response over the survey’s seven waves differs by important subgroups. In particular, younger individuals participate in fewer waves, as do individuals who did not live as a “couple.” These results are of concern because some of the people (younger people and females not living as part of a couple) who participate less appear to be people who are more at risk for being victims of rape and sexual assault.
CONCLUSION 8-2 There appears to be notable panel attrition over the 3 years in the National Crime Victimization Survey. This attrition is particularly problematic for estimating rape and sexual assault because
_____________
2The person-level response rate is the percentage of household members in cooperating households who responded on an individual questionnaire.
some people at greater risk for being victimized by these crimes—young people and females not living as part of a couple—are also some of those most likely to drop out before the seven waves of the NCVS have been completed.
CONCLUSION 8-3 Although the Bureau of Justice Statistics publishes annual response rates for the National Crime Victimization Survey (NCVS), the published data do not include important details of response, such as mode of data collection and attrition rate. Such details are needed by data users for a thorough assessment of the quality of NCVS estimates.
Item nonresponse occurs when a respondent completes a substantial portion of a questionnaire (enough to count the interview as “complete”) but does not provide answers to certain key items. The panel’s opinion is that item “refusals” to questions about sexual victimization are difficult to identify. If a respondent does not want to report a rape or sexual assault, then he or she is more likely to answer that he or she was not victimized rather than by directly refusing to answer the question. Thus, what may be an item refusal is most likely counted as just a “no.”
CONCLUSION 8-4 The panel believes it is likely that item refusals on questions about sexual victimization on the National Crime Victimization Survey may be recorded as if they were a “no” response rather than item nonresponse when a respondent does not want to report a victimization. Another possibility is for a respondent to sometimes answer “no” on screening questions to avoid additional questions in the survey.
Specification Error
Specification error may occur when there is a mismatch between what the survey is measuring and what it is intended to measure. A critical concept for the NCVS is to identify whether and when a respondent has experienced a rape or sexual assault. However, the complex, multifaceted definitions of what is meant by rape and sexual assault are translated into a few simple words in the omnibus screening questionnaire such as rape, attempted rape, other type of sexual attack, and unwanted sexual activity. These words do not convey the complexity of the intended concepts.
CONCLUSION 8-5 There is serious specification error in the National Crime Victimization Survey measurement of rape and sexual assault. Although the Bureau of Justice Statistics has developed clear defini-
tions of the concepts, they are replaced in the omnibus screener by ambiguous wording that does not convey the multifaceted concepts to respondents.
Measurement Error
Measurement error includes a large family of errors that may occur when a response on a survey results in the collection of inaccurate or incomplete information. The panel identified three characteristics of the current NCVS procedures that foster measurement errors: the use of ambiguous terms, such as rape, in questions (see above), the overall context (crime) in which questions are asked, and the lack of privacy in responding to the survey’s questions.
CONCLUSION 8-6 Words, such as “rape” and “sexual assault,” on the National Crime Victimization Survey may not be consistently understood by survey respondents. Other surveys have used more behaviorally specific words to describe a specific set of actions. More specific wording of questions would be understood more consistently by all respondents and thus lead to more complete and accurate answers.
CONCLUSION 8-7 Questions about incidents of rape and sexual assault in the National Crime Victimization Survey are asked in the context of a criminal victimization survey and embedded within individual questions that describe other types of crimes. This context may inhibit reporting of incidents that the respondent does not think of as criminal, did not report to the police, or does not want to report to police.
CONCLUSION 8-8 The current data collection mode and methods of the National Crime Victimization Survey do not provide adequate privacy for collecting information on rape and sexual assault. This lack of privacy may be a major reason for underreporting of such incidents.
As part of examining measurement error, the panel identified several problems with the training provided to interviewers on the NCVS and the subsequent monitoring of the interview process. For 10 years, until recently, there was no refresher training for interviewers, and the reinstated training offered only limited focus on the special training needs for sensitive questions about sexual victimizations. The training on these questions is not reinforced through the day-to-day survey process because of the low incidence of such reports. In addition, there is inadequate monitoring of the field data collection processes, of both in-person and decentralized
telephone interviewing by field representatives, to ensure consistent quality (see Conclusions 8-9 and 8-10 in Chapter 8).
SUMMARY AND RECOMMENDATIONS
With respect to measuring rape and sexual assault, the panel identified four major barriers for quality measurement on the NCVS:
1. a sample design that is inefficient for measuring these low incidence events,
2. the context of “crime” that defines the survey,
3. a lack of privacy for respondents in completing the survey, and
4. the use of words with ambiguous meaning for key measures in the questionnaire.
The first three barriers are intrinsic to the basic structure and processes of the NCVS, which appear to work well for measuring other criminal victimizations. These barriers cannot be overcome by making modifications to the NCVS without potentially compromising the quality of the NCVS design for measuring other crime rates. Only the last one—use of ambiguous terms—could be readily addressed within the structure and operations of the current NCVS.
CONCLUSION 10-1 The best methods for measuring rape and sexual assault cannot be implemented without separating that measurement from the measurement of other criminal victimizations.
A New Survey and Improved Methods
RECOMMENDATION 10-1 The Bureau of Justice Statistics should develop an independent survey—separate from the National Crime Victimization Survey—for measuring rape and sexual assault.
The panel makes several recommendations regarding the design and implementation of the independent survey, including a multiple frame approach (see Chapter 10). They are adaptations to the sampling and measurement strategies currently in place for NCVS and focus on finding ways to isolate and strategically oversample segments of the population where the risk of rape and sexual assault victimization is relatively greater.
RECOMMENDATION 10-2 The Bureau of Justice Statistics should rigorously compare the relative cost-efficiency of alternative sample
designs for the recommended new survey to measure rape and sexual assault, including the multiple frame approach described in Chapter 10.
RECOMMENDATION 10-3 The Bureau of Justice Statistics should continue to publish annual estimates of rape and sexual assault criminal victimizations, using the recommended new survey to do so. However, if that is not possible, then the Bureau should conduct the recommended new survey on a fixed schedule, such as every 2 or 3 years, and use data from both the National Crime Victimization Survey and the new survey to calculate annual estimates of rape and sexual assault.
The panel supports the multiwave structure of the NCVS for the proposed stand-alone survey, and it endorses the use of bounded recall procedures to control telescoping effects. However, the panel stresses that more can be learned through research into ways to improve the quality of data obtained using bounded recall. In addition, the panel has serious concerns about the current adjustments to wave 1 data to compensate for potential telescoping.
RECOMMENDATION 10-4 The recommended new survey should have a longitudinal structure with at least two waves to allow the use of bounded recall. Research should be conducted to determine an optimal length of reference period specifically for reporting rape and sexual assault victimizations. The Bureau of Justice Statistics should reassess the methodology used to adjust for forward telescoping if data from the bounding interview are used in estimation.
The context in which survey questions are asked is a critical element in obtaining accurate responses in any survey; it is particularly critical for questions about incidents of rape and sexual assault. The panel believes that framing these questions within a criminal context limits accurate responses. In addition, the panel strongly supports the wording of survey questions so that they describe specific actions rather than the more ambiguous term “rape.”
RECOMMENDATION 10-5 The questionnaire and protocols for the recommended new survey should have a neutral context, such as a health survey. The Bureau of Justice Statistics should explore several neutral alternatives while continuing to use both a victimization screening questionnaire and an incident report. The questions on both of these instruments should be reworded to incorporate behaviorally specific questions.
The lack of respondent privacy in the interview setting is a critically important barrier to obtaining truthful response in screening survey respondents for rape and sexual assault victimization. Self-administration of the screener instrument and incident report by a single randomly chosen member of participating households is the best strategy for overcoming this barrier. Because there are other survey error implications to this approach (e.g., on sampling error), BJS needs to carefully consider the relative merits of this approach for obtaining the most accurate possible victimization rates.
RECOMMENDATION 10-6 The recommended new survey should be conducted in a self-administered mode. The wave 1 contact would involve a personal visit and audio computer-assisted self-administered interviewing technology. Only one individual in each selected household should be selected for this survey to increase the respondent’s privacy.
The panel believes that people who have been victimized while not having the capacity to consent to sexual actions may be undiscovered by the current survey instruments. This oversight should be directly addressed by expanding the conceptual definition of rape and sexual assault to explicitly include these victims and by adding questions that probe in this area.
RECOMMENDATION 10-7 The Bureau of Justice Statistics’ definition of rape and sexual assault should be expanded to include victimizations when the victim does not have the capacity to consent to the sexual actions of the offender.
The exact process of determining the “type of crime” classification of an incident reported on the NCVS based on responses provided by the respondent is not clearly portrayed for data users.
RECOMMENDATION 10-8 The Bureau of Justice Statistics should make more transparent the link between responses on the survey screener and incident reports and the final type of crime classification of those incidents of potential rape and sexual assault.
The NCVS produces estimates of victimization rates, but it should also enable data analysts to identify important predictors of victimization. A generous array of social-demographic covariates obtained for each respondent is needed to more fully realize this potential.
RECOMMENDATION 10-9 The recommended new survey should include a number of covariates to add to the richness of the dataset for analysis. The Bureau of Justice Statistics should hold an expert-user workshop as it develops the new survey. A major purpose of the workshop would be to obtain advice on the covariates that could best improve the usefulness of the dataset for research, advocacy, and policy purposes.
The current procedures for handling series victimizations create estimation problems for the NCVS, resulting in large increases in the estimates of rape and sexual assault and their accompanying standard errors, and much greater year-to-year fluctuation in the estimates.
RECOMMENDATION 10-10 The Bureau of Justice Statistics should reassess the methodological change made to the National Crime Victimization Survey regarding series victimization and should investigate alternative procedures that are more effective in respect to measurements of rape and sexual assault. This reassessment should involve formal input by experts on outlier adjustment techniques and by data users who can help assess the relative tradeoffs in quality.
Training
There are some straightforward steps that could be taken to improve the quality of training and supervision being offered to the Census Bureau field representatives who administer the NCVS, particularly as they relate to the sensitive nature of the topic of rape and sexual assault. The monitoring of the process can also be improved.
RECOMMENDATION 10-11 The Census Bureau and the Bureau of Justice Statistics should provide specialized training for field representatives on how to assist the respondents and answer questions on the sensitive subjects of rape and sexual assault. The interaction between respondents and field representatives should be recorded using computer-assisted recorded interviewing technology.
Ongoing Research Program
The panel recommends that BJS develop an ongoing program of research addressing a variety of design-related problems related to the recommendations mentioned previously. There are 11 key research topics to investigate (see Chapter 10):
1. the cost-efficiency of introducing disproportionate stratified sampling of those at higher risk for rape and sexual assault victimization;
2. the cost-efficiency of supplementing the standard area household sampling frame with one or more frames derived from administrative sources with higher concentrations of victims (e.g., college residence hall records, police files, emergency room records, etc.);
3. the best estimation approach to deal with telescoping effects arising from the use of bounded questions in a longitudinal setting;
4. the effect of changing the survey to have a more neutral context;
5. the effects of following a neutral/behavioral orientation for questions used to screen for rape and sexual assault victimization;
6. the joint sampling and measurement error implications of self-administration of a single respondent chosen in each participating household;
7. the effect of expanding the definition of rape and sexual assault to include those without the capacity to give their consent to the offender;
8. the error and cost implications of improved training and supervision of field representatives;
9. ways to improve estimation in the presence of series victimization;
10. effective models to estimate the underreporting of rape and sexual assault on the NCVS based on data from a periodic independent survey; and
11. issues related to collecting data on rape and sexual assault criminal victimization from adolescents (12-17 years of age) because of their relatively high risk of that victimization.
The panel recommends that this research be conducted in a coordinated manner because many of the issues to be investigated interrelate.
RECOMMENDATION 10-12 The Bureau of Justice Statistics should conduct a series of coordinated research investigations to enable it to resolve each of the preceding specific issues in developing the design for the recommended stand-alone survey on rape and sexual assault.
Enhanced Communication with Data Users
As BJS moves in new directions to improve its measurement of rape and sexual assault, the agency needs to embrace external advice from the data user and statistical communities, and set up mechanisms to ensure open, regular, and effective communication with these communities. We offer three recommendations to BJS to better enhance this communication: establishment of a permanent advisory committee, regular data user confer-
ences, and steps for major methodological changes (see Recommendations 10-13, 10-14, 10-15 in Chapter 10).
The panel applauds BJS for its openness in addressing ways to fulfill its important mission to provide estimates of rape and sexual assault and is confident that the analyses and recommendations in this report can contribute to improvements in measuring these injurious victimizations.
