John V. Pepper and Carol V. Petrie
It is axiomatic that accurate and valid data and research information on both crime and victimization are critical for an understanding of crime in the United States and for any assessment of the quality of the activities and programs of the criminal justice system. In July 2000 the Committee on Law and Justice and the Committee on National Statistics of the National Research Council convened a workshop to examine an array of measurement issues in the area of crime victimization and offending and to explore possible areas for future research to improve measurement methods. This report provides information that was presented at the workshop.
TWO MAJOR DATA SOURCES
Most measurement of crime in this country emanates from two major data sources. For almost seven decades, the FBI’s Uniform Crime Reports (UCR) has collected information on crimes known to the police and arrests from local and state jurisdictions throughout the country. The National Crime Victimization Survey (NCVS), a general population survey designed to discover the extent, nature, and consequences of criminal victimization, has been conducted annually since the early 1970s. Other national surveys that focus on specific problems, such as delinquency, violence against women, and child abuse, also provide important data on crime, victims, and offenders.
These data collection systems utilize different methods of measuring criminal behavior. The UCR relies on official data that have been collected and reported by law enforcement agencies. The NCVS and other surveys discussed in this report are large-scale social surveys that rely on self-reports of offenses or victimization.
Although these data collection systems do many things right, they are, like any such system, beset with the methodological problems of surveys in general as well as particular problems associated with measuring illicit, deviant, and deleterious activities. Such problems include nonreporting and false reporting, nonstandard definitions of events, difficulties associated with asking sensitive questions, sampling problems such as coverage and nonresponse, and an array of other factors involved in conducting surveys of individuals and implementing official data reporting systems.
Compounding these problems are the recent interest in rare crime events, such as violent crimes committed by youth and hate crimes; the need for attention to vulnerable subpopulations, such as very young and school-age children and disabled, elderly, and immigrant populations; and a focus on small- or local-area estimates of crime and victimization. Congress periodically requires the U.S. Department of Justice to develop new research or data collection efforts to measure crime victimization in specific populations and for small areas. Understanding victimization and offending in these subgroups, however, can be particularly difficult.
In general, criminal victimization is a relatively rare event—that is, in any given reference period, the majority of respondents do not report any victimization. Very large general population samples are therefore required to accurately characterize the population of offenders and victims, and detailed subgroup analyses can be problematic. Some important subgroups may not be covered at all (e.g., homeless people), and smaller research studies of crimes against these subgroups often have problems of statistical power because of small sample sizes in most cases. For many hard-to-identify subpopulations, such as people with disabilities and abused children, there is no large, well-defined group from which to draw a sample for measuring victimization—in other words, a sampling frame. This, as well as more conventional problems associated with interviewing crime victims, presents substantial design and analytical difficulties.
Official data such as UCR arrest data have a different set of problems. Foremost among them is that most crimes are not reported to the police, and only a small proportion of those that are reported result in an arrest. Increases or decreases in reports or in arrests for certain offenses, such as
burglary or auto theft, can therefore result in large differences in outcomes and misleading conclusions about crime trends.
The accuracy of official data is also compromised by differences in the definitions of crimes and reporting protocols. Most national-level official data are compiled through the voluntary reporting of local-level agencies— 17,000 law enforcement agencies nationwide for arrests; a sample of 300 prosecutors’ offices nationwide for prosecution data, for example. However, these agencies do not always file reports as called for in the reporting protocol. A review of the 1999 UCR data posted on the FBI’s web site indicates that six states—Illinois, Kansas, Kentucky, Maine, Montana, and New Hampshire—report only limited data. In Illinois, for example, only six cities with populations of 10,000 or more report arrest data. Rape data were unavailable for two states because the state reporting agencies did not follow the national UCR guidelines (available: <http://www.FBI.gov/ucr/99cius.htm> [8/15/01]).
There has been a significant lack of governmental and private interest and investment in research aimed at solving these kinds of problems. It has now been more than 30 years since large-scale data systems on crime victimization have been under way and almost that long since research questions have been included as part of the structure of the NCVS. A major research effort undertaken in the early 1980s as part of the redesign of the NCVS addressed many important sampling questions, but (according to several workshop participants) for social and political reasons, much of this information did not make it into the survey.1
Today, in fact, the problems may be growing worse because of eroding federal investment in data systems and social science research on crime and victimization. The sample in the NCVS has shrunk because of flat funding over the past 25 years, from 60,000 households (in 1974) to the current level of 42,000 households (Patsy Klaus, senior statistician, Bureau of Justice Statistics, personal communication, 7/24/02). The promise of improvements in data on reported crimes through conversion to an incident-based reporting system has not been realized because of a lack of funding to
support the necessary changes at the state and local levels. Except for modest new funds to study violence against women, the federal budget for social science research on crime and victimization also has remained flat for two decades. Available funds generally are reserved for studies that potentially have a direct impact on policy; support for longitudinal or methodological studies for the most part simply is not there.
SCOPE OF THE REPORT
The workshop was designed to consider similarities and differences in the methodological problems encountered by the survey and criminal justice research communities and, given the paucity of available funding, to consider on what the academic community might best focus its research efforts. Participants represented a range of interests and academic disciplines, including criminology, econometrics, law, psychology, public policy, and statistics. Representatives of two federal agencies, the Bureau of Justice Statistics and the National Institute of Justice, also participated. The Workshop Agenda (Appendix A) and List of Workshop Participants (Appendix B) appear at the end of this workshop summary.
In addition to comparing and contrasting the methodological issues associated with self-report surveys and official records, the workshop explored methods for obtaining accurate self-reports on sensitive questions about stigmatized (crime) events, estimating crime and victimization in rural counties and townships, and developing unbiased prevalence and incidence rates for rare events among population subgroups. There also was a discussion of how to define and measure new types of crime: street crimes, such as carjacking, which combines car theft with robbery, assault, and sometimes homicide; and cyber crime, which combines ordinary forms of white-collar crime, such as fraud, with technology-based offenses, such as hacking.
The discussions prompted participants to think about the importance of good crime statistics, which according to some participants have never gotten much beyond measures of prevalence and incidence. How might more accurate figures serve the public interest? These data collection systems and measures are routinely used to talk about crime trends and the effect of interventions on crime rates and specific crime types, but because of what is known about the errors in them and what is tacitly assumed about the unknown errors—that they are even greater—researchers generally seem uncomfortable about using the available data to draw conclusions
about the scope and nature of crime problems. Participants discussed the need to prioritize these measurement problems, to think carefully about which ones are most important, and to focus on the two or three of primary concern.
To facilitate the workshop discussion, four papers were commissioned in advance, and they form the basis of this report. Following the workshop, the authors were invited to revise their papers in response to the workshop discussion.
Two chapters follow this introduction. Chapter 2, a paper by Roger Tourangeau and Madeline E. McNeeley, is on methodological issues in measuring crime and crime victimization. One discussant summarized this paper as an excellent exposition on why context matters. Chapter 3, a paper by Terence P. Thornberry and Marvin D. Krohn, is on the strengths and weaknesses of the self-report method in studies of offenders, and how to make improvements. A brief summary of the topics addressed in the other two commissioned papers appears below. It discusses particular issues that can arise when using surveys to draw inferences on specific subgroups, and methodological issues associated with measuring crimes in small geographic areas, from papers respectively by Richard McCleary, Douglas Wiebe, and David Turbow and by T.E. Raghunathan.
SUBGROUP AND SMALL-AREA ESTIMATION OF CRIME AND VICTIMIZATION
Public attention to crime and victimization often focuses on particular subgroups in which deviant behavior may be most troublesome. For example, hate crimes, crimes committed by youth, and crimes committed against vulnerable subpopulations including children, the elderly, and people with disabilities have all been the subject of recent investigations and legislation. Similarly, there has been increased focus in local-area criminal activity.
Interest in evaluating small subgroups of the population, however, results in especially difficult data and methodological problems. As noted above, crime and victimization are rare events. Thus, drawing precise conclusions about small subgroups requires that careful attention be paid to the sampling frame and the methods required to link the data to the subgroups of interest. Workshop participants spent some time discussing these issues, stressing the need for both reliable data and credible assumptions. Particular attention was focused on well-known but difficult problems for
drawing inferences on selected subpopulations, screening bias, and small-area estimation.
“Screening” refers to the basic survey methods used to identify cases or subgroups of interest (Morrison, 1985; Thorner and Remein, 1961; U.S. Commission on Chronic Illness, 1957). While the basic idea is both simple and appealing, screening for low-incidence events or cases can lead to substantial and systematic misreporting errors.
In Chapter 2, for example, screening errors are discussed in the context of false positive reports of defensive gun uses. A second example, which was explored in detail by McCleary and colleagues, centers on the victimization of persons with disabilities, and particularly persons with cognitive or developmental disabilities, and the extent to which a high percentage of false positives—persons identifying themselves as disabled when they are not— from general population samples can distort any estimates of the correlates of that particular category.2
False positive reports of rare events or subpopulations of interest can lead to substantial biases. In their paper, McCleary, Wiebe, and Turbow also demonstrated the potential biases created by screening respondents to select out rare subgroups of interest. These biases, the authors argue, should be sufficient to rule out any attempt to estimate low-probability events with a sample survey screening design, such as the National Crime Victimization Survey, and to instead develop estimates using a sampling frame that targets the specific group of interest. The authors also recommend using state-of-the-art survey methodologies that apply what is known about cognitive processes to interviewing techniques (also discussed by Tourangeau and McNeeley in Chapter 2).
For many research and policy questions, it is important to analyze crime data at a relatively low level of aggregation, such as county or even census tract. The problem, however, is that many of the national surveys, such as the National Victimization Survey, though large enough to yield national
estimates, or even state-level estimates, with adequate precision, are inadequate to yield reliable estimates for small areas. The raw rates are unstable, due to small numerators and/or denominators. Unstable rates are problematic because a change of only a few events in the numerator can result in large changes in rates. Furthermore, extreme rates (high or low) are often the result of inherent variability (noise) rather than true extremes in the phenomenon. The areas with the smallest populations will have the extreme values and will dominate a map or a statistical analysis. The least precise rates thus have the most influence.
In his paper, T.E. Raghunathan developed a framework for obtaining small or local-area estimates of victimization or crime rates by combining information from surveys and administrative data sources. Using these data, the measures created from the administrative data sources can be treated as predictors and the victimization rates computed from the surveys as dependent variables. This approach depends on the availability of a good set of predictors—that is, detailed data—and the credibility of the assumptions used in the prediction model.
PROGRESS AND CONTINUED PROBLEMS
As the workshop participants acknowledged, tremendous progress has been made in the past 50 years in our ability to measure, describe, and evaluate both victimization and criminal behavior. The development and refinement of a number of administrative and survey data systems have had and continue to play a central role in the ability of the nation to understand crime. However, many difficult barriers remain to drawing credible inferences using these data systems. The classic survey sampling problems of response and coverage are of special concern when attempting to measure stigmatized and other deleterious events. The transitory nature of the different types of criminal activity and of the interests of policy makers presents unique problems for survey designers and researchers. Likewise, continued interest in evaluating specific subpopulations and geographic areas requires improved data and research methodologies.
Workshop participants recognized the need to be pragmatic. The federal budget for data on crime and justice is modest and has not grown in nearly two decades. Without additional resources devoted to improving the data on crime and justice, many of the problems discussed at the workshop are likely to persist. In fact, to some degree, all of the issues discussed have been studied and described for over two decades without resolution.
The participants focused on the importance of two overriding issues: (1) there are significant and substantive measurement problems with the existing surveys that are likely to remain unresolved without additional funding support; (2) priorities need to be established so that the available resources devoted to these issues can be used most effectively.
In addition to discussions of these general points, much of the workshop focused on measurement problems that arise from the existing surveys and, to some degree, on future directions for research to resolve them. The measurement issues discussed include, but certainly are not limited to, the following:
Improving reliability. Many participants suggested that additional attention should be focused on improving the reliability and validity of self-report surveys, rather than simply assessing these characteristics. For example, Thornberry and Krohn (Chapter 3) argue that “it is likely that both validity and reliability would be improved if we experimented with alternative items for measuring the same behavior and identified the strongest ones.” McCleary, Wiebe, and Turbow suggest using multiple data sources and questions on the same item to “develop an estimate of the prevalence and magnitude of the bias for an item and adjust for it statistically.”
Evaluating the impact of nontraditional survey methodologies. Much of the knowledge about reporting errors in crime surveys comes from crosssectional studies. Using self-reports in longitudinal studies, especially ones that cover major portions of the life course, creates a new set of challenges. Likewise, Thornberry and Krohn argue for the need to better understand the effects of self-administration (e.g., computer-assisted self-administered interviews) on reporting errors in surveys of crime.
Response errors across self-report and administrative surveys. Many participants argued for a more systematic analysis comparing and contrasting the understanding of crime and victimization in self-report and official data.
Survey packaging design. Tourangeau and McNeeley (Chapter 2) wonder whether “the apparent topic of the survey, the survey’s sponsorship, the organization responsible for collecting the data, the letterhead used on advance letters, and similar procedural details will affect respondents’ views about the need for accuracy in reporting and about the type of incidents they are supposed to report.”
In addition to focusing on improving the validity and reliability of surveys and administrative data, there was some discussion at the workshop about the need to develop and apply credible models. Even perfectly valid and reliable data cannot completely address the many questions of interest. This was particularly obvious in Dr. Raghunathan’s presentation on developing small-area estimates and in the more general discussion that followed. Evaluations of subgroups or geographic areas that may not be adequately represented or covered by the sampling scheme invariably require researchers to make assumptions. The credibility of empirical findings, however, depends on the validity of the maintained assumptions. Some participants expressed concern that little thought was being paid to the assumptions used to derive inferences on crime and victimization, especially for those regarding specialized populations or geographic areas.
The variety of ideas, concerns, and recommendations raised by the papers and commentary at the workshop resulted in a rich discussion of crime measurement and research issues. Public opinion polls have repeatedly demonstrated that crime is a policy issue of pre-eminent concern to the American public. The goal of the Committee on Law and Justice and the Committee on National Statistics in convening the workshop, commissioning the papers, and issuing this report is to stimulate further discussion and eventually a greater focus on the importance of improving data collection systems and measurement of crime.
McCleary, R., D. Wiebe, and R. Turbow 2000 Screening Bias. Paper commissioned for the Committee on Law and Justice, Workshop on Measurement Problems in Criminal Justice Research, July 2000, National Research Council, Washington, DC.
Morrison, A.S. 1985 Screening in Chronic Disease. Cary, NC: Oxford University Press.
Raghunathan, T.E. 2000 Combining Information from Multiple Sources for Small Area Estimation of Victimization Rates. Paper commissioned for the Committee on Law and Justice, Workshop on Measurement Problems in Criminal Justice Research, July 2000, National Research Council, Washington, DC.
Thorner, R.M., and Q.R. Remein 1961 Principles and Procedure in the Evaluation of Screening for Disease. Public Health Monograph #67. Washington, DC: U.S. Government Printing Office.
U.S. Commission on Chronic Illness 1957 Chronic Illness in the United States. Cambridge: Harvard University Press.