The panel was charged with recommending best methods for obtaining survey statistics on rape and sexual assault on household surveys of the Bureau of Justice Statistics (BJS). In reviewing all of the material presented earlier in this report, the panel thinks that it is highly likely that the National Crime Victimization Survey (NCVS) is underestimating rape and sexual assault. The panel, with limited resources, was not able to measure the extent of such an undercount with statistical rigor.
The previous chapter discusses four major obstacles to quality on the current NCVS: (1) a sample design that is inefficient for measuring these low-incidence events, (2) the context of “crime” that defines the survey, (3) a lack of privacy for respondents in completing the survey, and (4) the use of words with ambiguous meaning for key measures in the questionnaire.
As detailed in Chapter 9, only one of the four obstacles, the last one—use of ambiguous terms—can be readily addressed within the structure and operations of the current NCVS without negatively impacting the estimation of other important types of criminal victimizations. These obstacles led to the conclusion that the needed changes could not be adequately implemented within the framework of the existing survey, and that best practices for measuring rape and sexual assault would require that these measurements be decoupled from the NCVS.
CONCLUSION 10-1 The best methods for measuring rape and sexual assault cannot be implemented without separating that measurement from the measurement of other criminal victimizations.
RECOMMENDATION 10-1 The Bureau of Justice Statistics should develop an independent survey—separate from the National Crime Victimization Survey—for measuring rape and sexual assault.
In the rest of this chapter, the panel presents its recommendations for developing the recommended new survey for estimating rape and sexual assault, for important features of that survey, and for research needed as part of that development. As a part of the entire program design, the panel also offers several recommendations regarding improved training and monitoring of interviewers and regarding obtaining useful input and feedback from data users and the broader statistical community.
Enhanced Multiple-Frame Design
The proportion of a population with a specific attribute (in this case, having been victimized by rape or sexual assault) can be estimated with greater precision by isolating population subgroups with relatively higher attribute rates and then sampling those subgroups more intensively than the rest of the target population. The higher the attribute rate in a subgroup, the greater potential gains in precision. The first challenge in this approach is to identify subgroups of people who are at higher risk of rape and sexual assault criminal victimizations than the general population. A second challenge is to identify such groups in a way that does not isolate previous victims or make them fearful or reintroduce the trauma of the incident.
In spite of the difficulty, the introduction of one or more additional sampling frames derived from sources likely to include persons victimized by rape and sexual assault will be a more cost-efficient alternative to the current NCVS frame. The panel strongly urges BJS to explore these opportunities.
RECOMMENDATION 10-2 The Bureau of Justice Statistics should rigorously compare the relative cost-efficiency of alternative sample designs for the recommended new survey to measure rape and sexual assault, including the multiple-frame approach described in this chapter.
An enhanced design would be anchored by a frame with coverage of the general population, such as the current NCVS frame. An initial step toward this enhanced design would be to more effectively target stratification in selecting the general-purpose sample to oversample areas where the income
and demographics characteristics would be likely to make household members more at risk for sexual victimization. A second step would be to sample the group quarters frame with relatively higher rates because many types of group quarters may include relatively high concentrations of victims of rape and sexual assault (e.g., college residence halls, group shelters, etc.).
This general-purpose frame would be enhanced with one or more additional frames that would focus on specific subgroups that are at higher risk for sexual assault but are broader than a list of known sexual assault victims. Sources that might be used as a sample frame include, but are not limited to
• female college students,
• women who use Indian health service facilities,
• assault cases known to law enforcement,
• people treated for trauma in hospital emergency rooms,
• people who have filed a police report for any type of serious violent crime,
• residents of shelters for abused and battered women, and
• outpatients from mental health clinics.
These sources are not lists of rape and sexual assault victims but would include a much higher percentage of victims than seen in the general population (providing a statistical advantage to using them). Specialized sample selection and recruitment strategies may need to be developed to minimize concerns from victims who are found through supplemental frame sources. Following those procedures, the sample of individuals would be interviewed using the same survey instrument regardless of the frame from which they were selected. Note that a respondent’s victimization status (for rape and sexual assault) would be unknown when an interview begins: it would be determined solely from answers given on the questionnaire. However, because of the addition of these specialized frames, the resulting sample is likely to have more positive responses to the questions on sexual victimization. This will make the estimates more precise, while the survey weights will ensure that this process does not artificially inflate the criminal victimization rate.
This approach has significant drawbacks. First, it will be very difficult to get access to some of the administrative sources for the supplemental frames. For example, police reports are not available to the general public, but they may be available in some form to a government agency. Second, HIPAA regulations1 may restrict access. Third, there is concern that a victim who has experienced sexual abuse may not want his or her name shared
1HIPAA is the federal Health Insurance Portability and Accountability Act of 1996.
with a government survey group, and a person who has sought help from a hospital, shelter, or health clinic may feel betrayed if that institution provides names to BJS. These are serious concerns that will need to be worked through, but there is prior experience in the survey field working with respondent recruitment from these types of sources.
A fourth drawback is that working with more frames and the overlap between frames would add complexity and costs for frame development, sampling, and recruitment (Hartley, 1962; Lohr, 2011). In addition, because one cannot know in advance how well this approach would work, there is a basic question of whether the precision gains from the design will outweigh the added complexity. The exact answer to this question can only be determined through specific research. However, the panel did a preliminary cost-efficiency analysis that was very encouraging; it merits a thorough examination by BJS.
The panel’s analysis looked at a simplified case of two overlapping sampling frames with simple random sampling in each frame.2 One frame (the administrative frame) is completely contained in the other more complete frame (the household frame). For example, individuals who were treated for trauma in an emergency room are also a part of the U.S. population. For simplification in this illustration, overlap between the two frames is handled by screening out members of the administrative frame from household members in the sample chosen from the household frame. Under this simplified case, the panel observed the relative precision of a cost-variance optimized dual-frame estimator (in comparison with a single-household frame) of the proportion of the population with the attribute. In a specific example, the dual-frame design had a variance that was 43-45 percent lower than the cost-equivalent variance for the single-household frame design (see Box 10-1 and more detail in Appendix E).
The panel’s analysis illustrates that the added cost of data collection on the administrative frame is not an important factor in this scenario. What does matter is the proportion of the population in the administrative frame that has the attribute (see Table 10-1). As the table shows, the relative efficiency of the dual frame decreases significantly as that proportion decreases. The variance reduction shown in this table would be somewhat offset with a more complex formula that takes into account the cluster effects of the sample.
Research is needed to identify specific administrative frames for which there is a higher risk of rape and sexual assault that may be used in this enhanced multiple-frame design without infringing on the privacy of victims. Research is also needed to further assess the effects of the cluster
2Details of the analysis can be found in Appendix E and in a paper presented at the recent Joint Statistical Meetings (Kalsbeek, Spencer, and House, 2013).
Comparing Precision from Dual-and Single-Frame Alternatives
In this scenario, police records were to be used to define an administrative frame of rape victims, of size NA ~ 140,000 (extrapolated) to the total U.S. population from 1997 data from the Uniform Crime Reports (UCR) summary. Based on National Crime Victimization Survey (NCVS) data (average per 1992-2000), there were 116,300 such criminal victimizations reported to police. Thus the proportion of the population within this administrative frame, experiencing this criminal victimization, PA, is estimated as:
PA = 116,300 / 140,000 = 0.83
The household frame (similar to that used by the NCVS) would be much larger, N = 250,000,000. The population proportion of the same attribute (based on Criminal Victimization, 2007) is much smaller,
P = 0.001
Costs for the NCVS (Rand, 2009) for fiscal year 2009 were $26 million to collect approximately 150,000 interviews. Thus the cost per completed interview,
CHH = $26 million / 150,000 = $173
Results for fixed total cost:
If the cost of data collection using the administrative frame is twice that of the household frame, then the optimal allocation would have a sample size of 148,380 from the household frame and 955 from the administrative frame. Comparing the variances between the estimates of population proportion from the dual frame and the single-household frame, the relative variance RV (dual-frame variance / household-frame variance) is,
RV = 0.549.
This implies that the variance for the dual-frame design with these assumptions is 45 percent lower than the cost-equivalent variance for the household-frame design.
Looking at the same scenario again, when the cost of data collection using the administrative frame is ten (10) times that of the household frame, then the relative variance is:
RV = 0.566,
showing very little difference from the above example.
TABLE 10-1 Effect of Population Proportion in Administrative Frame on Relative Variance
|Proportion of Population with Attribute, PA||Estimated Relative Variance, RV|
sampling and to estimate both the added costs and the relative efficiency of this proposed design. This research should also explore various practical strategies for integrating the samples chosen from the multiple frames. For instance, supplementary administrative lists could be developed within selected primary sampling units (PSUs; from the household frame) so that national administrative frames would not be necessary.
Implementation of a new independent survey to measure rape and sexual assault would have a major impact on the current BJS budget. The panel is sensitive to these budget concerns, particularly in the current fiscal climate. Consequently, the panel considered several variations that would be less expensive than a full implementation of its recommendations for “best practices,” while still making some improvements in the quality of the estimates. In doing so, we do not retreat from our charge to recommend a plan for best methods; rather, we present these lower-cost variations for BJS to consider within the realities of its budget priorities.
The panel first considered whether a traditional supplement would suffice in lieu of a completely independent survey.3 The NCVS currently has a number of supplements, such as the 2008 Identity Theft Supplement. The major cost advantage to a supplement is that the survey is “piggy-backed” on the NCVS, using the same sample (or a subsample), the same field representatives, and the same survey structure. Individuals selected for a supplement would have been respondents on the NCVS survey over several waves of data collection. The demographics of the households and
3A survey supplement uses a supplemental questionnaire that is administered to a subset of respondents to the full survey, in this case the NCVS, in a separate data collection activity or wave. To avoid affecting the regular estimates, such supplements are usually administered to a subset of respondents who have rotated off the base survey.
individuals would be known. However, by the time they were asked to complete the supplement, their responses would still be in the context of a criminal victimization survey and they are unlikely to move beyond that context regardless of how the supplement is structured. Thus, if a supplement is used, it would need to be administered on the initial wave rather than on the households that are rotating off the NCVS.
A major problem with developing a supplement is that it would be based on the NCVS sample (or a subsample as described in “Enhanced Multiple-Frame Design” above). The design of that survey is very inefficient for measuring rape and sexual assault. With an independent survey, the sample design could be developed to be more efficient for measuring these specific criminal victimizations.
The NCVS is only one of several large household surveys conducted by the Census Bureau. The new survey may work better as a supplement to one of these other surveys rather than to the NCVS. For example, the American Community Survey (ACS) may be an appropriate vehicle. Because of the continuous design of the ACS, the Census Bureau could select individual household members (by appropriate demographic and geographic characteristics) who are at higher risk for rape and sexual assault. Using the ACS as a base for the new survey would avoid the context of a “crime.”
Another lower-cost option would be to use the basic NCVS sample design (same PSUs and clusters) and select a large and separate sample for measuring rape and sexual assault. A household from this sample could be screened based on demographics of household members. Women ages 18-34 who have been shown to be of higher risk for rape and sexual assault would be oversampled as the respondents in this survey.
One way to reduce costs is to reduce the sample size or the frequency of data collection. BJS currently publishes annual estimates of the criminal victimizations of rape and sexual assault. The panel does not recommend less frequent data collection or publication, but one cost-saving variation would be to develop a formal rolling estimation process similar to that used for the ACS, which has a relatively modest annual sample size for the intent to produce small area estimates. This type of approach is currently used on an ad hoc basis on the NCVS for some special studies and research but not for annual estimates. The ACS uses a rolling estimation process in which multiple years of data (generally 3 years for most estimates) are pooled for estimation. Annually, data from the latest data collection are rolled into the estimates while data from the oldest year are rotated out. This process provides stable year-to-year estimates for small areas, even with modest sample sizes. Some more recent recommendations for the ACS are to use some time-series modeling across years rather than just a pooling of data. These ideas would also be useful to pursue with the estimates of rape and sexual assault.
Another variation would be for data from the NCVS to be used in conjunction with data from the recommended new survey to reduce overall data collection costs. The new survey could be viewed as a vehicle to measure the underestimation of rape and sexual assault on the NCVS. Data from both surveys could be modeled together to estimate this underestimation. The recommended new survey would be fielded on a fixed schedule, such as biennially or every 3 years. The NCVS could be modified to use behaviorally specific questions (as the panel recommends) and continue to be fielded annually. The NCVS estimates of rape and sexual assault would be adjusted for underestimation (based on the new survey) and continue to measure year-to-year change in criminal victimization rates. Considerable research would be needed to develop and calibrate these models.4
RECOMMENDATION 10-3 The Bureau of Justice Statistics should continue to publish annual estimates of rape and sexual assault criminal victimizations, using the recommended new survey to do so. However, if that is not possible, then the Bureau should conduct the recommended new survey on a fixed schedule, such as every 2 or 3 years, and use data from both the National Crime Victimization Survey and the new survey to calculate annual estimates of rape and sexual assault.
Other Recommended Design Characteristics
Longitudinal Component with Bounding
To measure the incidences of rape and sexual assault, the NCVS asks respondents to remember events in the past and to report only those that occurred within a specific reference period, currently 6 months. When a victim remembers a traumatic event such as a rape or sexual assault, he or she may remember the event as having occurred more recently than it actually did. (This could result in counting that event as part of a reference period when in reality it occurred before the beginning of that period, thus over-counting for that period.) This phenomenon is known as forward telescoping. A classic study examined the technique of “bounded recall”5 to prevent the shifting of recalled events in time. In an experimental design, the study looked at collecting recall of expenditures on alterations and repairs made by resident homeowners. The study found that “unbounded recall of
4This would be a difficult task. The underreporting might not be uniform by subgroups, and BJS may lack sufficient data to accurately adjust by subgroup. Unless these adjustments could be extended beyond the estimates to the microdata, the differences would create difficulties in the analysis of rape and sexual assault victimizations.
5The previous interview is used to “bound” the reference period for recall.
expenditures for the preceding month involved substantial net forward telescoping of jobs into the recall period” (Neter and Waksberg, 1964, p. 43). The authors also concluded that shorter reference periods (1 month rather than 3 months) produced higher estimates of expenditures.
Because of the potential for telescoping, the NCVS originally did not use the data collected in its first wave in the estimation of criminal victimization. It currently uses data from initial interviews, with an adjustment to minimize overreporting during these initial contacts. (The adjustments are described in the “Estimation and Products” section in Chapter 4.)
In a recent analysis of NCVS data to look at the issues of telescoping, the role of the bounding interview, recency, and time-in-sample, Fay and Li (2010) found mixed results. More crimes were reported as having occurred in the month immediately preceding the interview than in other months in the reference period. The authors found that this “recency effect” is greater for violent crime than property crime. They concluded (Fay and Li, 2010, p. 1698):
[T]he evidence presented here would encourage a re-examination of the issue of telescoping and the role of the bounding interview, … however, we recognize that BJS certainly has good reason to maintain the status quo for now, until other changes in the design are implemented in the future.
The panel strongly supports the need for further research. The appropriate reference period for recalling incidents of rape and sexual assault may be longer than for other criminal victimizations. Time-in-sample analysis may indicate the need for a survey with fewer waves.
Until more definitive work is done, the panel recommends the continuing use of bounded recall procedures. However, the panel has serious concerns about the current adjustments to first wave data to compensate for potential telescoping.
RECOMMENDATION 10-4 The recommended new survey should have a longitudinal structure with at least two waves to allow the use of bounded recall. Research should be conducted to determine an optimal length of reference period specifically for reporting rape and sexual assault victimizations. The Bureau of Justice Statistics should reassess the methodology used to adjust for forward telescoping if data from the bounding interview are used in estimation.
Neutral Context with Behaviorally Specific Wording
Concern about the context of crime in the NCVS and the use of terms such as “rape” and “sexual assault,” and their potential effect to inhibit
the reporting of incidents of rape and sexual assault, are discussed in-depth in Chapters 4 and 8. Some respondents may not view their victimization as criminal. Or they may have decided not to report the incident to police as a crime and now have concerns about reporting it on a government crime survey. When asked specifically about “rape” and “sexual assault,” survey respondents may not consistently or accurately understand those terms. Research has shown that a change to behaviorally specific questions increases reporting of the criminal victimizations (Fisher, 2009). As detailed in Chapter 8, the context of a crime survey is likely to inhibit positive responses (Conclusion 8-7 in Chapter 8), and the use of behaviorally specific questions would likely lead to more accurate responses (Conclusion 8-6 in Chapter 8).
RECOMMENDATION 10-5 The questionnaire and protocols for the recommended new survey should have a neutral context, such as a health survey. The Bureau of Justice Statistics should explore several neutral alternatives while continuing to use both a victimization screening questionnaire and an incident report. The questions on both of these instruments should be reworded to incorporate behaviorally specific questions.
Self-Administered Mode of Data Collection with One Household Member Selected
It is critically important to provide privacy to respondents when asking them to recall the details of a rape or sexual assault. The current NCVS does not provide this privacy, and this fact is one of the primary reasons for the panel’s recommendation to conduct a separate survey to measure incidence rates. The lack of privacy in the current NCVS is rooted in the fact that everyone in the household is interviewed and therefore knows the questions that are being asked, and there is an oral interview taking place that might be overheard. A second issue in dealing with these sensitive questions is that respondents may feel that reporting their criminal victimization to an interviewer may be socially undesirable, so they do not report accurately. (These issues are discussed in-depth in Chapter 8.)
These characteristics of the NCVS lead to the panel’s recommendation for a self-administered mode of collecting the information and a single-respondent design. There are significant cost considerations in moving to a survey with a single respondent per household. It takes considerable resources to make the initial contact with a household, and the current NCVS then has multiple respondents as a result of that contact. Thus, the sampling error will increase under a single-respondent design unless a larger sample of households is selected or multiple-frame sampling provides
efficiency in finding individuals at risk for this victimization (as described in “Enhanced Multiple-Frame Design” above). The panel is confident that having a single-respondent design would be less problematic in the single-purpose survey for rape and sexual assault than it would be for the entire NCVS, which is obtaining information on multiple types of victimizations from different family members. This is a topic for which additional research would be helpful.
The selection of a single respondent within a household should not be made with equal probabilities of selection. Instead, individuals whose demographics would put them at greater risk for sexual criminalization (females, certain age groups, etc.) would have higher probabilities of selection. This would be straightforward in a survey specifically designed for measuring rape and sexual assault. It would be very difficult to do for an omnibus survey because the demographics more “at risk” would be different for different types of victimizations.
Advances in technology have eliminated many of the past concerns about self-administered surveys. In particular, audio computer-assisted self-administered interviewing (ACASI) has evolved with the proliferation of computers and computer-assisted survey techniques. In an ACASI survey, the interviewer obtains the respondent’s consent and cooperation and provides basic instruction. The respondent is then seated in front of a laptop or tablet computer and puts on a set of earphones. The survey questions appear visually on the computer screen and orally through the earphones simultaneously. The respondent can control the speed of each input source and ask additional questions of the interviewer as needed.
The use of ACASI technology will increase privacy. However, the panel is very concerned that the use of this technology alone will not provide the privacy needed in households in which the victim is concerned about other household members knowing that he or she is answering questions about rape and sexual assault. Thus the panel continues to support the single-respondent design even with the use of ACASI technology.
The Internet offers another approach to self-administered surveys, but this mode lacks the presence of an interviewer to encourage response and answer questions. An Internet-based survey may be an acceptable variation for certain respondents on subsequent waves following an initial in-person contact.
RECOMMENDATION 10-6 The recommended new survey should be conducted in a self-administered mode. The wave 1 contact would involve a personal visit and audio computer-assisted self-administered interviewing technology. Only one individual in a selected household should be selected for this survey to increase the respondent’s privacy.
Modified Definition of Rape
BJS currently uses a fairly broad definition of rape and sexual assault (see Box 8-1 in Chapter 8). However, the definitions do not include the incapacity to consent to sexual activities. Under most current laws, the capacity to consent is affected by the age of the victim, mental capability of the victim, and intoxication (see Chapter 2). This component should be included in the basic definitions: examples of potential revised definitions are shown in Box 10-2. As with any major definitional change, this change is likely to affect the estimates. These effects should be investigated, and a bridge with the older definition should be used as the new definitions are phased in.
RECOMMENDATION 10-7 The Bureau of Justice Statistics’ definition of rape and sexual assault should be expanded to include victimizations when the victim does not have the capacity to consent to the sexual actions of the offender.
BJS publishes its definitions and questionnaires on its website. However, the link between the answers given to various questions and the overall classification of the incident as a crime is not clear. Such a link would allow a more transparent understanding of the classification of incidents reported on the survey. The flowchart presented in Figure 7-3 (in Chapter 7) may be useful as a starting point for this link.
Proposed Revisions to BJS Definitions of Rape and Sexual Assault
Rape—Forced sexual intercourse including psychological coercion, physical force, and the victim’s inability to consent. Forced sexual intercourse means vaginal, anal or oral penetration by the offender(s). This category also includes incidents where the penetration is from a foreign object such as a bottle. Includes attempted rapes, male as well as female victims, and both heterosexual and homosexual rape. Attempted rape includes verbal threats of rape.
Sexual Assault—A wide range of criminal victimizations, separate from rape or attempted rape. These crimes include attacks or attempted attacks generally involving unwanted sexual contact between victim and offender. Sexual assaults may or may not involve force and include such things as grabbing or fondling. Sexual assault also includes verbal threats and situations where the victim does not have the capability to consent.
NOTES: Strikethroughs are deletions. Text in boldface italics are additions.
SOURCE: Panel modification to current NCVS definitions (Bureau of Justice Statistics, n.d.-b).
RECOMMENDATION 10-8 The Bureau of Justice Statistics should make more transparent the link between responses on the survey screener and incident reports and the final type of crime classification of those incidents of potential rape and sexual assault.
The NCVS and the recommended new independent survey can provide data that increase understanding of the frequency and context under which criminal victimizations occur. This improved understanding often comes from the research performed by criminologists, sociologists, and others using data from the NCVS. Having demographic variables and other relevant covariates available in the datasets enhances the value of the criminal victimization data.
The design of the recommended new survey with a focus on sexual violence provides an opportunity to review and enhance the set of covariates that are collected along with the criminal victimization variables. Researchers who study criminal victimizations have identified certain types of information about people’s lifestyle and routine activities that can provide insight into, and perhaps lead to identifying causal relationships between, the context and likelihood of future victimization (Fisher et al., 1998; Schreck, Stewart, and Fisher, 2006). It will be important to involve data users to assist in identifying a useful set of covariates.
RECOMMENDATION 10-9 The recommended new survey should include a number of covariates to add to the richness of the data set for data analysis. The Bureau of Justice Statistics should hold an expert-user workshop as it develops the new survey. A major purpose of the workshop would be to obtain advice on the covariates that could best improve the usefulness of the dataset for research, advocacy, and policy purposes.
Alternative Procedures for Series Victimizations
Series victimization is defined on the NCVS as the situation when a single respondent reports six or more separate but similar victimizations over the reference period but is unable to recall these events individually or describe them in detail to the interviewer. In 2011, BJS implemented a procedural change in how these series victimizations are handled on the NCVS. Chapter 7 discusses the impact of this change on the estimation of rape and sexual assault. The impact of the resulting outliers was sizable, resulting in large increases in the estimates of rape and sexual assault and the standard errors, and much greater year-to-year fluctuation. With a new
survey that focuses on measuring rape and sexual assault, there is an opportunity to reexamine the procedures that should be used in this survey.
The panel sees a key first step is to better understand the subpopulation that is experiencing multiple rapes and sexual assaults within a short period of time. This understanding would include the demographic and lifestyle characteristics of these victims as well as the size of the subpopulation. A second step is to investigate whether series victimizations are reported with sufficient quality to use in the estimation process, given the concerns about response error and other measurement errors discussed in Chapter 8. If not, alternative data collection procedures should be developed based on research (including cognitive testing) designed to better understand how to retrieve information on series victimizations. This research could lead to both better data collection procedures as well as data adjustment techniques to improve the quality of data that are collected.
Once the data are determined to be of sufficient quality, thought should be given to what is the primary goal of the survey—estimation within a single year, estimation of totals or rates across several years, or estimation of the annual change in rates or totals. If the latter two goals are of primary interest, then consideration might be given to spreading outlier respondents’ estimation weights across several years. This would have an effect akin to using a running average for the outliers each year.
Here is an example of a possible procedure for smoothing the responses over 3 years. Let W denote the respondent’s survey weight, and let R denote the number of rape and sexual assault victimizations reported by the respondent. A cap, C, on R may be imposed, so that R is truncated to the cap. BJS currently uses C = 10. Let the capped value of R be denoted by S = min(R, C). Let D denote a threshold value. If the product WS (weight times [capped] data value) exceeds D, then the respondent is included in the current wave with a modified weight such as
W’ = (1/3)W + (2/3)(D/S)
A separate record is created for the respondent in the survey data for the wave 12 months later and for the wave 24 months later with the same data values and modified weight
W” = (1/3)W - (1/3)(D/S).
If the respondent is included in the sample in one of the latter waves, then the respondent would have more than one survey record for the wave. How to choose the number of years, the cap, and the threshold value are questions that would need to be researched. The procedure as described is fairly simple and could be refined.
The panel recommends continuing research into improved methods for accurately accounting for series victimizations while minimizing the year-to-year effect caused by the current outlier adjustment procedure.
RECOMMENDATION 10-10 The Bureau of Justice Statistics should reassess the methodological change made to the National Crime Victimization Survey regarding series victimization and should investigate alternative procedures that are more effective in respect to measurements of rape and sexual assault. This reassessment should involve formal input by experts on outlier adjustment techniques and by data users who can help assess the relative tradeoffs in quality.
There is a clear need for improved training for Census Bureau field representatives who administer the NCVS (see Chapter 8). Training needs to be a high-priority activity even in times of tight budgets. The development of a separate independent survey will provide an opportunity to have focused, specialized training for the field representatives working on this survey. Their role will be different with the use of ACASI technology. They will need to have training on how to introduce this survey and encourage response. They will need special training regarding the topics of rape and sexual assault, the words to use in answering questions, and the correct level of emotional support to show.
The panel also recommends that the interaction between the field representative and the respondent be recorded using CARI (computer-assisted recorded interviewing) technology. These recordings can be used effectively to monitor the interaction process and to identify areas where retraining (of individual interviewers or more generally) will be important. It is possible that the use of a recording device may affect survey response. Although existing research does not indicate that CARI reduces response rate (Biemer et al., 2000; Thissen et al., 2009), it would be prudent to consider this an open research question to be investigated as the survey design is developed.
RECOMMENDATION 10-11 The Census Bureau and the Bureau of Justice Statistics should provide specialized training to field representatives on how to assist the respondents and answer questions on the sensitive subjects of rape and sexual assault. The interaction between respondents and field representatives should be recorded using computer-assisted recorded interviewing technology.
Many of the recommendations presented in this report will need additional research and development before implementation.
RECOMMENDATION 10-12 The Bureau of Justice Statistics should conduct a series of coordinated research investigations to enable it to resolve each of the following specific issues in developing the design for the recommended stand-alone survey on rape and sexual assault.
The panel recommends that BJS develop an ongoing program of research addressing the following 11 topics.
1. The cost-efficiency of introducing disproportionate stratified sampling of those at higher risk for rape and sexual assault victimization. This includes determining (i) which variables to use for stratification at each stage of selection and (ii) the possible utility of disproportionate sample allocation, particularly in sampling residences from nonoverlapping residential listings.
2. The cost-efficiency of supplementing the standard area household sampling frame with one or more frames derived from administrative sources. This includes determining (i) whether it would be more cost-effective to select the sample from a single NCVS-like household sampling frame alone, or from a household sampling frame plus one or more overlapping frames from administrative sources of possible victims and (ii) regardless of the feasibility of the multiple-frame option, whether design integration with a household survey conducted by another federal statistical agency (e.g., the American Community Survey conducted by the Census Bureau) would be a statistically and practically feasible way to generate the household sample that will be needed for the recommended stand-alone survey.
3. The best estimation approach to deal with telescoping effects that arise from the use of bounded questions in a longitudinal setting. This research should include the examination of the potential to use a longer reference period (possibly 1 year) for the new survey.
4. The overall effect of changing the “context” of the survey from that of crime to a more neutral context.
5. The effects of following a neutral/behavioral orientation for questions used to screen for rape and sexual assault victimization. This research should also include the examination of potentially useful covariates for predicting rape and sexual assault criminal victimization.
6. The joint sampling and measurement error implications of self-administration of a single respondent chosen in each participating household. This research should also include the exploration of the use of the Web for self-administration after the first wave.
7. The effect of expanding the definition of rape and sexual assault to include those without the capacity to give their consent to the offender.
8. The error and cost implications of improved training and supervision of field representatives.
9. Ways to improve estimation in the presence of series victimization. This includes an examination of the characteristics of the subpopulation that is sexually victimized on a frequent basis (series victimization) to provide data to better represent this subpopulation in the overall victimization rates.
10. Determine whether effective models can be built to estimate the underreporting on the NCVS, using data from the new survey that is fielded on a periodic basis.
11. Conduct further research on issues related to collecting data on rape and sexual assault criminal victimization from adolescents (12-17 years of age) because of their relatively high risk of criminal victimization.
Principles and Practices for a Federal Statistical Agency identifies as one of its key principles the need to maintain credibility among data users. It states the following (National Research Council, 2013b, p. 13):
Credibility derives from the respect and trust of users in the statistical agency and its data. Such respect results not only from an agency’s production of data that merit acceptance as relevant, accurate, timely, and free from political and other undue external influence, but also from many aspects of an agency’s policies and practices. Key among these are wide dissemination of data on an equal basis to all users; openness about the sources and processes used to produce data and the limitations of the data; commitment to quality and professional practice; a strong internal and external evaluation program to assess and improve an agency’s data systems; a willingness to understand and strive to meet user needs, even though users may not clearly articulate their needs; and a posture of respect and trust in the users of an agency’s data.
In pursuing this general principle, Principles and Practices recommends that these agencies engage in these specific practices (National Research Council, 2013b, pp. 19-20):
• seek advice on data concepts, content, processing, estimation, products, and documentation from a wide spectrum of data users, as well as from professional and technical subject-matter and methodological experts, using a variety of formal and informal means of communication that are appropriate to the types of input sought;
• seek advice on its statistical programs and priorities from external groups, including those with relevant subject-matter and technical expertise;
• keep abreast of and use modern statistical theory and sound statistical practice in all technical work; and
• document concepts, definitions, data collection methodologies, and measures of uncertainty and discuss sources of error in reports and other data releases to the public.
In its review of U.S. justice statistics (Ensuring the Quality, Credibility, and Relevance of U.S. Justice Statistics) the National Research Council (2009, pp. 250-251) focused on these principles and practices and made several specific recommendations regarding the need for BJS to maintain both formal and informal communication with its data users.
• BJS should regularly convene ad hoc stakeholder workshops to suggest areas of immediate data needs (Recommendation 5.7), and
• BJS should establish an Advisory Group under the Federal Advisory Committee Act to provide guidance to BJS on the addition of new data collection efforts and the modification of current ones in light of needs identified by the group (Recommendation 5.8).
BJS has reached out to data users in a number of useful ways. Over the years, BJS has held expert workshops that were well received by data users. Unfortunately, the frequency of the workshops has decreased in recent years because of budget constraints. BJS has made use of expertise and advice from the American Statistical Association’s Committee on Law and Justice Statistics. BJS has also sponsored an annual 4-week summer workshop at the University of Michigan’s Inter-university Consortium for Political and Social Research to train data users on analyses techniques for their datasets. The panel commends BJS for those efforts and recommends more.
This panel endorses the recommendation from the previous NRC report discussed above and repeats it as a recommendation.
RECOMMENDATION 10-13 The Bureau of Justice Statistics should establish a permanent advisory committee under the Federal Advisory Committee Act to provide guidance on user issues, ongoing program and research priorities, and the implementation of new methodological advances. This committee should advise on all of the survey programs of
the Bureau of Justice Statistics and not be limited to the National Crime Victimization Survey or the measurement of rape and sexual assault.
Going beyond the formal advisory committee for its entire program, BJS should also convene a regular data user conference focused on the measurement of criminal victimizations, including the measurement of rape and sexual assault.
RECOMMENDATION 10-14 The Bureau of Justice Statistics should hold a regular (annual or biennial) data users’ conference for users of the Bureau of Justice Statistics criminal victimization data. The conference would have four objectives: (1) to provide an update on new or planned program changes, (2) to facilitate informal communication between data users and the Bureau of Justice Statistics staff on issues important to those data users, (3) to provide training on the use of publicly available microdata, and (4) to provide a vehicle for data users to showcase academic papers using these data. This data users’ conference might be held in conjunction with a professional association meeting.
Going beyond a regular data users’ conference, BJS should more closely follow the practice identified in Principles and Practices when contemplating a major methodological change in one of its major surveys (National Research Council, 2013b, p. 9, emphasis added):
[S]eek advice on data concepts, content, processing, estimation, products, and documentation from a wide spectrum of data users, as well as from professional and technical subject-matter and methodological experts, using a variety of formal and informal means of communication that are appropriate to the types of input sought.
Specifically, the panel recommends that BJS take the following steps before making a major change:
• develop the conceptual background for change, along with alternative sampling, estimation, and/or survey methodology changes that are being considered;
• formally review the issues with data users to decide on a strategic direction for the change—this could be done during a data users’ conference;
• formally review the proposed sampling, estimation, and/or survey methodology changes with the broader statistical community to obtain input on best methodologies for implementing the proposed changes;
• formally test and evaluate the selected new procedure(s); and
• clearly communicate with data users at each step.
RECOMMENDATION 10-15 The Bureau of Justice Statistics should follow five steps when contemplating a major methodological change in one of its major surveys: (1) develop the conceptual background for the need for change and alternative sampling, estimation, or survey methodologies; (2) formally review these concepts with a broad set of data users to decide on a strategic direction; (3) formally review the statistical and survey methodological issues and proposed changes with technical experts in the broader statistical community; (4) formally test and evaluate the new procedures, their feasibility, and their impact; and (5) clearly communicate with data users at each step.
The panel offers two concrete examples regarding the need for the above recommendation: the recent change to account for series victimization and an earlier decision to use data from the bounding interview in estimation. In both cases, BJS concluded that a methodology change was needed and had informal, if not formal, discussions with a number of data users who weighed in on the issues. However, the third step is also important, and it appears to have received less attention. In the example of series victimization, BJS appears not to have engaged fully with specialists on outliers in the statistical community to help evaluate a wider range of alternative procedures to adjust (or not adjust) for these outliers. In the example of the bounding interview, the current adjustments that involve weighting the first wave data down to the average level of waves 2-7 are suspect. As more recent analysis has shown, the “recency” effect, along with notable attrition rates over the life of the survey, raises the question of whether these adjustments to the data are the best approach. Again, engaging more statistical expertise to examine alternatives would have been appropriate when this adjustment procedure was considered.
Following the decision to change, communication with data users needs to be frequent and clear. For example the publication Criminal Victimizations, 2011 (Bureau of Justice Statistics, 2012a) stated that the series victimization change was being implemented in that, and future, publications. The panel did not find a notice that the change was being made retroactively on the online database. More and frequent communication is always a good policy.
BJS has a very important mission to provide estimates of criminal victimizations within the United States, both annual rates of those victimiza-
tions and the change between years. It employs the NCVS, a large omnibus victimization survey, as a critical tool for accomplishing that mission. Addressing data user questions about its current methodology for measuring the incidence of rape and sexual assault, the BJS sought advice from the National Research Council, which led to the creation of this panel. The panel concludes that “best practices” for measuring rape and sexual assault on BJS household surveys would involve a decoupling of that measurement from the NCVS, and it provided guidelines for making that change. The panel applauds BJS for its openness in addressing these issues and is confident that it can use the analyses and recommendations in this report to move toward continuous quality improvements in measuring these injurious victimizations.