Below is the uncorrected machine-read text of this chapter, intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text of each book. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.
Executive Summary T HE B UREAU OF J USTICE S TATISTICS (BJS) of the U.S. Department of Justice, Ofï¬ce of Justice Programs (OJP), requested that the Committee on National Statistics (in cooperation with the Commit- tee on Law and Justice) convene this Panel to Review the Programs of the Bureau of Justice Statistics. The panel has a broad charge to: examine the full range of programs of the Bureau of Justice Statistics (BJS) in order to assess and make recommendations for BJSâ priorities for data collection. The review will examine the ways in which BJS statistics are used by Congress, executive agencies, the courts, state and local agencies, and researchers in order to determine the impact of BJS programs and the means to enhance that impact. The review will assess the organization of BJS and its relationships with other data gathering entities in the Department of Justice, as well as with state and local governments, to determine ways to improve the relevance, quality, and cost-effectiveness of justice statistics. The review will consider priority uses for additional funding that may be obtained through budget ini- tiatives or reallocation of resources within the agency. A focus of the panelâs work will be to consider alternative options for conducting the National Crime Victimization Survey, which is the largest BJS program. The goal of the panelâs work will be to assist BJS to reï¬ne its priorities and goals, as embodied in its strategic plan, both in the short and longer terms. The panelâs recommendations will address ways to improve the impact and cost-effectiveness of the agencyâs statistics on crime and the criminal justice system. [emphasis added] BJS speciï¬cally requested that the panel begin its work by providing guid- ance on options for conducting the National Crime Victimization Survey (NCVS), one of many data series sponsored by BJS and one that consumes 1
2 SURVEYING VICTIMS a large share (as much as 60 percent) of the agencyâs annual appropriations. This interim report responds to this request. Since the survey began full-scale data collection in the early 1970s, the NCVS has become a major social indicator for the United States. Serving as a complement to the ofï¬cial measure of crimes reported to the police (the Uniform Crime Reporting [UCR] program administered by the Federal Bu- reau of Investigation), the NCVS has been the basis for better understanding the cost and context of criminal victimization. However, and particularly over the course of the last decade, the effectiveness of the NCVS has been undermined by the demands of conducting an increasingly expensive sur- vey in an effectively ï¬at-line budgetary environment. In order to keep the survey going in light of tight resources, BJS has reduced the surveyâs sample size over time, and other design features have been altered. When the survey began in 1972, the sample of addresses for interviewing numbered 72,000; in 2005, the NCVS was administered in about 38,600 households, yielding interviews with 67,000 people. Although this sample size still qualiï¬es the NCVS as a large data collection program, occurrences of victimization are essentially a rare event relative to the whole population: many respondents to the survey do not have incidents to report when they are contacted by the survey. At present, the sample size is such that only a year-to-year change of 8 percent or more in the NCVS measure of violent crime can be deemed statistically to be signiï¬cantly different from no change at all. In its reports on the survey, BJS has to combine multiple years of data in order to com- ment on change over time, which is less desirable than an annual measure of year-to-year change. In approaching this work, the panel recognizes the ï¬scal constraints on the NCVS, but we do not intend to be either strictly limited by them or completely indifferent to them. Rather, our approach is to revisit the basic goals and objectives of the survey, to see how the current NCVS program meets those goals, and to suggest a range of alternatives and possibilities to match design features to desired sets of goals. PRESERVING THE VICTIMIZATION MEASURE There are no nationally available data on crime and victimizationâ collected at the incident level, with extensive detail on victims and the social context of the eventâexcept those collected by the NCVS. It is this basic fact that is the strongest argument for the continuation and maintenance of the survey. Certainly, one option for the future of the NCVSâand the ultimate cost-reducing optionâis to suspend or terminate the survey. It is an option that would have to be considered, if budget constraints require
EXECUTIVE SUMMARY 3 further reductions in sample size. To be clear, though, abandonment of the NCVS is not an option that we favor in any way. Annual national-level estimates from the NCVS are routinely used in conjunction with the UCR to describe the volume and nature of crime in the United States. There is great value in having two complementary but nonidentical systemsâthe NCVS and the UCRâaddressing the same phe- nomenon, for the basic reason that crime and victimization are topics that are too broad to be captured neatly by one measure. The police are not a disinterested party when it comes to characterizing the crime problem, and it is unwise to have data generated by the police as a sole measure of crime na- tionally. The UCR tells us little about the victims of crime; although its Na- tional Incident-Based Reporting System (NIBRS) has the potential to capture some of the detail currently measured by the NCVS, NIBRS has substantial limitations and remains incapable of providing national-level estimates after 20 years of implementation. Moreover, it is clear that a substantial pro- portion of crime is not reported fully and completely to law enforcement authorities. Thus, there remains a vital role for a survey-based measure that sheds light on unreported crime. Recommendation 3.1: BJS must ensure that the nation has quality annual estimates of levels and changes in criminal victim- ization. The current design of the NCVS has beneï¬ted from years of experience, methodological research, and evaluation; it is a good and useful model that has been adopted by international victimization surveys as well as subna- tional surveys within the United States. The principal fault of the current NCVS is not a design ï¬aw or methodological deï¬ciency, or even that the de- sign inherently costs too much to sustain, but ratherâsimplyâthat it costs more than is tenable under current budgetary priorities. In its present size and conï¬guration, the NCVS can permit insights into the dynamics of vic- timization. However, in our assessment, the current NCVS falls short of the vibrant measure of annual change in crime that was envisioned at the surveyâs outset. Finding 3.1: As currently conï¬gured and funded, the NCVS is not achieving and cannot achieve BJSâs legislatively mandated goal to âcollect and analyze data that will serve as a continuous and comparable national social indication of the prevalence, in- cidence, rates, extent, distribution, and attributes of crime . . .â (42 U.S.C. 3732(c)(3)). By several measuresâcomparison with the expenditures of foreign coun- tries for similar measurement efforts or with the cost of crime in the United
4 SURVEYING VICTIMS Statesâthe NCVS is underfunded. Accordingly, the panel recommends that BJS be afforded the budgetary resources necessary to generate accurate mea- sures of victimization, which are as important to understanding crime in the United States as the UCR measure of crimes reported to the police. Recommendation 3.2: Congress and the administration should ensure that BJS has a budget that is adequate to ï¬eld a survey that satisï¬es the goal in Recommendation 3.1. OVERALL GOAL AND DESIGN CONSIDERATIONS In considering historical goal statements of the NCVS, as well as new ones, we ï¬nd three basic goals to be particularly prevalent and important, in addition to the previously expressed goal of maintaining annual national- level estimates of victimization that are independent of ofï¬cial reports to the police: ⢠Flexibility, in terms of both content (capability to provide detail on the context and etiology of victimization and to assess emerging crime problems, such as identity theft, stalking, or violence against and in- volving immigrants) and analysis (providing informative metrics be- yond basic crime rates); ⢠Utility for gathering information on crimes that are not well reported to police or on hard-to-measure constructs (e.g., crimes against adoles- cents, family violence, and rape); and ⢠Small-domain estimation, including providing information on states or localities, which we think will be crucial to maximizing the utility of the NCVS and to building and maintaining constituencies for the survey. In this report, we describe various design possibilities and their implica- tions relative to these goals; however, we do not suggest one single path as the ideal for a redesigned NCVS. In part, this is because it is difï¬cult to jus- tify the case that our preferred set of NCVS goals is correct to the exclusion of all others; in part, it is because of the short time frame and the sequencing of this report (since it is inherently difï¬cult to try to consider NCVS in iso- lation from the balance of BJS programs). But in large part we refrain from expressing a single, unequivocal path because the potential effectiveness and cost implications of some major design choices are simply unknown at this time. We do think that it is critical to emphasize that even small changes to the design of a survey can have signiï¬cant impacts on resulting estimates and the errors associated with them. Design changes made in the name of
EXECUTIVE SUMMARY 5 ï¬scal expedience, without grounding in testing and evaluation, are highly inadvisable. They risk unexplained changes in the time series and confusion among users. Recommendation 4.1: BJS should carefully study changes in the NCVS survey design before implementing them. One potential cost-saving design choice is to change from asking respon- dents to recall and describe crime incidents in a 6-month window to using a 12-month window. This would entail contacting households once a year rather than twice (and, presumably, only 3 or 4 times if one chose to keep with the current regime of keeping households in the sample for 3.5 years). This would reduce the per-unit interviewing cost and free up resources to add additional sample addresses within each single year; 12 months is also the common reference period in victimization surveys in other countries. However, it could also increase problems of recall error by making re- spondents search their memories over a longer period. On its conceptual strengths and its use in comparable crime surveys in other western nations, we prefer a switch to a 12-month reference period as a cost-saving mecha- nism over options that would simply reduce the total sample size. That said, the empirical case for implementing this change is not completely clear and warrants up-to-date research. We note that such a move requires an overlap of designs over time to safely incorporate the change to 12 months. Recommendation 4.2: Changing from a 6-month reference pe- riod to a 12-month reference period has the potential for im- proving the precision per-unit cost in the NCVS framework, but the extent of loss of measurement quality is not clear from exist- ing research based on the post-1992-redesign NCVS instrument. BJS should sponsor additional researchâinvolving both exper- imentation as well as analysis of the timing of events in extant dataâto inform this trade-off. It is also the case that cost savings might be achieved by reï¬ning the NCVS sample stratiï¬cation schemes. The current multistage cluster design of the NCVS automatically includes households sampled from counties and other geographic regions with large population sizes, clustering the remain- ing geographic areas by social and demographic information to produce sim- ilar strata from which the remaining sample is drawn. The composition of the sample is relatively slow to change with each decennial census, although effort is made to include some new housing stock by sampling from housing permit data. If the NCVS continues to be conducted by the Census Bu- reau (see âCollecting the Data,â below), particular insight for altering the
6 SURVEYING VICTIMS basic sample design and modifying sample strata based on an up-to-date sampling frame could come from interaction with the new American Com- munity Survey (ACS). But, again, quantitative methodological research that could suggest exactly what beneï¬ts might or might not accrue is lacking. Recommendation 4.7: BJS should investigate changing the sam- ple design to increase efï¬ciency, thus allowing more precision for a given cost. Changes to investigate include: (i) changing the number or nature of the ï¬rst-stage sampling units; (ii) changing the stratiï¬cation of the primary sampling units; (iii) changing the stratiï¬cation of housing units; (iv) selecting housing units with unequal probabilities, so that probabilities are higher where victimization rates are higher; and (v) alternative person-level sampling schemes (sampling or subsampling persons within housing units). As early as 1980, the NCVS began the use of multiple response modes. Face-to-face personal interviews after the ï¬rst contact with a sample house- hold were replaced with interviews conducted by telephone, andâafter the 1992 implementation of the full NCVS redesignâsome interviewing be- gan to be done by Census Bureau computer-assisted telephone interviewing (CATI) centers using a fully automated survey instrument. The NCVS path to automation has been somewhat complicated: full conversion to nonpaper survey questionnaires was achieved only in 2006, andâas part of the most recent round of cost reductionsâBJS and the Census Bureau abandoned the use of the centralized CATI centers for NCVS interviews because antic- ipated cost savings never occurred. However, as redesign possibilities are considered, it is important that BJS continue to seek automation possibilities and not be limited to the NCVS traditional interview formats. A particular area of focus should be self-response options, such as computer-assisted self- interviewing (effectively, turning the interviewerâs laptop around so that the respondent answers questions directly) or Internet response for interviews after several visits. As with the central CATI centers, cost savings from new modes of data collection are not guaranteed, but they may put the survey in good stead for implementing new topical modules and promoting high respondent cooperation. They can also serve to reduce overall respondent burden. Recommendation 4.8: BJS should investigate the introduction of mixed mode data collection designs (including self-administered modes) into the NCVS.
EXECUTIVE SUMMARY 7 The NCVS is subject to the same pressures facing all household surveys in modern times, whether federal or private. It is increasingly difï¬cult (and expensive) to obtain survey responses from persons or households in an age of cell phones, call waiting, and Internet chat. A signiï¬cant fraction of survey costs are incurred to contact the most hard-to-ï¬nd respondents. In consid- ering design possibilities, it is important that BJS try to develop schemes that are relatively robust to declines in response rate, as such declines are virtually certain. Recommendation 4.9: The falling response rates of NCVS are likely to continue, with attendant increasing ï¬eld costs to avoid their decline. BJS should sponsor nonresponse bias studies, fol- lowing current OMB guidelines, to guide trade-off decisions among costs, response rates, and nonresponse error. BUILDING AND REINFORCING CONSTITUENCIES A continuing challenge for the NCVS is the development of constituen- cies with a strong interest in the data and their quality. The public is aware of the NCVS mainly due to one regular constituencyâthe mediaâand the spate of crime uptick or downtick stories that accompanies each yearâs re- lease of NCVS and UCR estimates. Likewise, ï¬ndings from topical supple- ments (such as racial dimensions of trafï¬c stops, measured by the Police- Public Contact Survey supplement) typically get prominent press coverage. Ofï¬cial statistics, like other societal infrastructures, are often highly valued but rarely passionately promoted by day-to-day users. However, the long- term viability of the survey depends crucially on building and shoring up constituencies for NCVS products and on cultivating the surveyâs user base among researchers. Small-Domain Estimates The world has changed since the mid-1970sâcomputers are more pow- erful, data users are more sophisticated, and the demand for small-area ge- ographic data is more insatiable. It is too strong to say that the NCVS can remain relevant only if it provides estimates for areas or populations smaller than the nation as a whole: state and local governments, which are among the most prodigious of NCVS users, continue to ï¬nd national benchmarks very valuable. However, the survey will increasingly grow out of step with potential constituencies if it cannot be used to provide estimates for smaller areas.
8 SURVEYING VICTIMS Recommendation 4.5: BJS should investigate the use of model- ing NCVS data to construct and disseminate subnational esti- mates of major crime and victimization rates. This recommendation runs counter to the principal effect of one of our predecessor National Research Council (1976b) panelâs recommendationsâ that the separate âimpact cityâ victimization surveys that were originally part of the National Crime Surveys suite should be terminated. However, it is very much consistent with that previous recommendationâs focus on an integrated set of estimates, including subnational geographies. These sub- national estimates need not be exhaustive: expanding the sample to sup- port estimates for the largest metropolitan statistical areas is a more sensible and cost-effective approach than a system for generating estimates for all 50 states. But they should permit insight on victimization for some smaller units than the nation as a whole. Small-domain estimates also refer to estimates by other social or demographic constructs, such as urbanicity (urban, subur- ban, or rural), in addition to the basic disaggregation by major race-ethnicity groups that is currently done. With particular regard to the generation of small-domain estimates, it should be noted that enhancing the NCVS to better serve constituencies is not strictly a process of addition, in terms of sample size or implemen- tation of a full supplemental questionnaire. In some important respects, user constituencies may best be served by more creative use of the current NCVS design. In the years since National Research Council (1976b) advo- cated eliminating the city surveys, statistical developments in small-domain estimation techniques have been considerable; hence, some small-domain estimates may be possible through modest investment by BJS in technical infrastructure for statistical modeling tasks. In addition to small-domain modeling using NCVS data, it may also be useful to explore ways to strengthen victimization surveys conducted by states and localities. Currently, BJS operates a program under which it devel- ops victimization survey software and provides it to interested local agencies; however, those agencies must supply all the resources (funds and manpower) to conduct a survey. An approach to strengthen this program would be to make use of BJSâs organizational position within the U.S. Department of Justice. The bureau is housed in the Ofï¬ce of Justice Programs, the core mission of which is to provide assistance to state and local law enforcement agencies; it does so through the technical research of the National Institute of Justice and the grant programs of the Bureau of Justice Assistance (BJA), among others. We suggest that OJP consider ways of dedicating fundsâlike BJA grants, but separate from BJS appropriationsâfor helping states and localities bolster their crime information infrastructures through the estab- lishment and regular conduct of state or regional victimization surveys. Such
EXECUTIVE SUMMARY 9 surveys would most likely involve cooperative arrangements with research organizations or local universities and make use of the existing BJS statistical analysis center infrastructure. This approach is analogous to the Behavioral Risk Factor Surveillance System (BRFSS) of the Centers for Disease Con- trol and Prevention, and it is similar in its partnership arrangements to the Federal-State Cooperative Program for Population Estimates (FSCPE) of the Census Bureau. Recommendation 4.6: BJS should develop, promote, and coor- dinate subnational victimization surveys through formula grants funded from state-local assistance resources. We discuss an extreme interpretation of this approachâwherein the ânationalâ victimization survey would be effectively be the combination of the subnational surveysâin Chapter 4. However, we emphasize that we suggest that this BRFSS/FSCPE approach should be considered independent of (and as a complement to) the chosen design of the NCVS. Topic Constituencies The NCVS ï¬rst added a topic supplement to the survey questionnaire in 1977, querying respondents on their perceptions of the severity of crime. Particularly since 1989, supplements have been an irregular part of the NCVS structure; the School Crime Supplement on school safety has been repeated six times and the Police-Public Contact Survey three times, with other supplements being (to date) one-time efforts. A strong program of topic supplements is an important part of the NCVS, both because of the breadth of topics that may be handled and because the ability to quickly ï¬eld questions on new topics of interest is a key advantage of survey-based collection compared with ofï¬cial records. Recommendation 4.3: BJS should make supplements a regular feature of the NCVS. Procedures should be developed for solic- iting ideas for supplements from outside BJS and for evaluating these supplements for inclusion in the survey. What is necessary regarding NCVS supplements is a more structured plan for their implementation, better exploration (and marketing) of sponsorship opportunities by other state and federal agencies, and greater transparency in real costs of conducting a supplement. Regardless of the overall design of the NCVS, the British Crime Survey offers an attractive model: a stream- lined core set of questions combined with a planned, regular slot for topical content.
10 SURVEYING VICTIMS Recommendation 4.4: BJS should maintain the core set of screening questions in the NCVS but should consider streamlin- ing the incident form (either by eliminating items or by changing their periodicity). This would reduce respondent burden and allow additional ï¬exibility for adding items to broaden and deepen information about prevalent crimes. ATTENTION TO DATA QUALITY AND ACCESS We make a series of recommendations that are agency-level in focus, aimed at better equipping BJS to understand its own products and to interact with its users. They are presented here in initial form because they are pertinent to the NCVS. We expect to expand on them in our ï¬nal report on the full suite of BJS programs and products. First, BJS currently receives periodic advice from the Committee on Law and Justice Statistics of the American Statistical Association (ASA). Al- though this input is certainly valuable, we think that BJSâand the NCVS in particularâwould beneï¬t from the commissioning of an ongoing scientiï¬c technical advisory board, such as is in place for other statistical agencies. This board should include subject matter, survey methodological, and sta- tistical expertise; spots on the board are also a vehicle for strengthening stakeholder constituencies for the NCVS. Recommendation 5.1: BJS should establish a scientiï¬c advisory board for the agencyâs programs; a particular focus should be on maintaining and enhancing the utility of the NCVS. Several of our recommendations listed earlier identify gaps in exist- ing research that must be ï¬lled to accurately inform trade-offs in design choices. More generally, the NCVS developmental work in the 1970s and the research conducted as part of the 1980s redesign effort are extensive, but we think that there is a paucity of recent methodological research mak- ing use of the post-1992-redesign NCVS instrument and techniques. BJS has already made some strides in fostering methodological research with its fellowship program, operated in conjunction with the ASA. We urge BJS to continue this work and to explore other creative ways to foster internal and extramural research using the NCVS and other BJS data sets, including grad- uate fellowships, as part of continuous efforts to assess the quality of NCVS estimates. Recommendation 5.3: BJS should undertake research to contin- uously evaluate and improve the quality of NCVS estimates.
EXECUTIVE SUMMARY 11 Conceptually, the survey-based NCVS is ideally suited (as the ofï¬cial record-based UCR is not) to study the dynamics of crimes that are emotion- ally or psychologically sensitive, such as violence against women, violence against adolescents, and stalking or harassment. We urge BJS to develop lines of research to ensure that such crimes are accurately measured on the NCVS instrument; these might include the testing of self-response options, such as audio computer-assisted interviewing. Recommendation 3.3: BJS should continue to use the NCVS to assess crimes that are difï¬cult to measure and poorly reported to police. Special studies should be conducted periodically in the context of the NCVS program to provide more accurate mea- surement of such events. The quality of NCVS data and its scientiï¬c rigor in measuring crime should always be the surveyâs primary goal and acknowledged as its principal beneï¬t. However, for the purpose of cultivating constituencies and users for the survey, attention to the accessibility and the ease of use of NCVS data is also vitally important. Part of this work involves reevaluation of basic products and reports from the NCVS and expansion of the range of analyses based on the data, and it involves both in-house research by BJS and effective ties with other users and researchers. Recommendation 5.2: BJS should perform additional and ad- vanced analysis of NCVS data. To do so, BJS should expand its capacity in the number and training of personnel and the ability to let contracts. A necessary consequence of this recommendation is that the agency must expand its capacity, both in the number and training of personnel and the agencyâs ability to let contracts for external research. Recommendation 5.4: BJS should continue to improve the avail- ability of NCVS data and estimates in ways that facilitate user access. Recommendation 5.5: The Census Bureau and BJS should en- sure that geographically identiï¬ed NCVS data are available to qualiï¬ed researchers through the Census Bureauâs research data centers, in a manner that ensures proper privacy protection. In the case of this last recommendation, we understand that arrange- ments to place detailed NCVS data at the research data centers are under development; we state it here as encouragement to ï¬nalize the work.
12 SURVEYING VICTIMS COLLECTING THE DATA It is important to note that some of the resource constraints on the NCVS are common to those on other important federal surveys, which have faced difï¬culties carrying out basic maintenance tasks like updating samples to reï¬ect new census and address list information. The country needs a mech- anism to alert itself to budget cuts that undermine the basic purposes of key federal statistical products. Recommendation 5.6: The Statistical Policy Ofï¬ce of the U.S. Ofï¬ce of Management and Budget is uniquely positioned to identify instances in which statistical agencies have been unable to perform basic sample or survey maintenance functions. For example, BJS was unable to update the NCVS household sample to reï¬ect population and household shifts identiï¬ed in the 2000 census until 2007. The Statistical Policy Ofï¬ce should note such breakdowns in basic survey maintenance functions in its annual report Statistical Programs of the United States Government. Any review of a major survey programâparticularly one carried out with an eye toward cost reductionâmust inevitably raise the question of the agent that collects the data: could survey operations be made better, faster, or cheaper by getting some other organization to carry out the survey? In this case, the U.S. Census Bureauâs involvement with the NCVS predates the formal establishment of the survey, as the Census Bureau convened planning discussions and conducted NCVS pilot work. The optimal decision on who should do the data collection for the NCVS will depend on the weight that one puts on desired objectives for the survey. For instance, an extremely strong weight on ï¬exibility and quick response to emerging trends might argue against the Census Bureau, where imple- mentation of a supplement can be made time-consuming through detailed cognitive testing (which ultimately improves the quality of the questions but can be slow) and passage through bureaucratic channels (e.g., clearance by the Ofï¬ce of Management and Budget, as required of all federal surveys). However, dominant weight on maintaining high response rates and draw- ing from the experience of other large, ongoing surveys would suggest that staying with the Census Bureau is the best course. Just as we do not offer a single design path for the NCVS, we do not ï¬nd justiï¬cation for offering a conclusion on âCensus Bureauâ or ânot Census Bureau.â Based on the ad- vantages and disadvantages, we suggest that âprivatizingâ the NCVS is not the panacea for high survey costs that some may believe it is. We have been provided no way of estimating the various costs associated with switching NCVS data collection agents; however, it is altogether appropriate to con- sider means of getting detailed and speciï¬c answers to these questions.
EXECUTIVE SUMMARY 13 In the interim, we suggest that the Census Bureau would beneï¬t both BJS and itself itself by providing greater transparency in true survey costs. Recommendation 5.7: Because BJS is currently receiving inad- equate information about the costs of the NCVS, the Census Bureau should establish a data-based, data-driven survey cost and information system. We further suggest that BJS consider a design competitionâproviding some funds for bidders to specify in detail how they would conduct a vic- timization survey. This design competition would effectively compensate bidders for their time in developing proposal speciï¬cations, but it should be run with a statement that a formal request for proposals may result from the competition (and not that it will deï¬nitely occur). Recommendation 5.8: BJS should consider a survey design com- petition in order to get a more accurate reading of the feasibility of alternative NCVS redesigns. The design competition should be administered with the assistance of external experts, and the competition should include private organizations under contract and the Census Bureau under an interagency agreement.