Below are the first 10 and last 10 pages of uncorrected machine-read text (when available) of this chapter, followed by the top 30 algorithmically extracted key phrases from the chapter as a whole.
Intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text on the opening pages of each chapter. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.
Do not use for reproduction, copying, pasting, or reading; exclusively for search engines.
OCR for page 47
Ensuring the Quality, Credibility, and Relevance of U.S. Justice Statistics –2– Measurement in the Justice System IRONICALLY, IT IS ONE OF THE SHORTER ENTRIES in the statutory duties of the Bureau of Justice Statistics (BJS) that assigns the agency its most gargantuan task: Point 4 in Box 1-2 obliges BJS to collect and analyze data “concerning the operations of the criminal justice system” at all levels of government. The complication, of course, is that the “criminal justice system” is not a simple construct. Defining all the parts of a “criminal justice system” is difficult, much less organizing coherent techniques to measure aspects of said system, and different conceptual approaches lead to starkly different vantages of the system. An organizational theory perspective might focus on the interactions of government bodies within the system and on measures of throughput and effectiveness (e.g., police success rates in clearing cases and apprehending suspects and caseload processing rates by the courts); a person-level approach from the victim’s perspectives might emphasize the availability and effectiveness of victim support and compensation programs whereas an offender-based approach would give higher prominence to physical and social conditions in correctional facilities as well as parole and prison reentry programs. Measurement in the justice system is also complicated by the highly varied units of measurement that obtain throughout the process. The most basic of violent interpersonal crimes involves a triad of units—the victim, the offender, and the incident—each of which evinces a distinct geography, history, and set of circumstances and contexts, and the study of each of which may lead to different conclusions. However, crime has many types; “victims” and “offenders” need not be individual humans (they may, for instance, be businesses or corporations) and “incidents” need not be one-time acts of
OCR for page 48
Ensuring the Quality, Credibility, and Relevance of U.S. Justice Statistics violence. Police deal with “suspects” who may or may not be the actual perpetrators of crimes; courts deal with “cases” or “defendants,” each of which involves one or more specific “charges,” and the connection between these labels and the individual “prisoners” who serve correctional sentences may be lost when authority for them transfers from the courts to the corrections system. An immediate consequence of this unit-of-measurement problem, which we discuss in this chapter and elsewhere in the report, is that measuring the flow of individuals through the justice system is extremely complicated. As a general concept, though, it is important to bear in mind that an approach to the “justice system” that focuses on tracking the experience of individual persons as they move through the system will present a different picture of justice processing than inferences drawn from one that focuses on the progression of “cases” through the set of discrete operations that either move them forward toward resolution or divert them out of the system. There is no single way, and certainly no uniquely correct way, to conceptualize the criminal justice system. Hence, BJS’s task is to straddle a wide range of perspectives in selecting and defining its measurement programs, and to do so while ensuring collection at all (and widely varying, in themselves) levels of government. In this chapter, we discuss the general challenge of measurement in the justice system. We begin by describing the basic model that we use as an orienting framework—the “funnel” model first developed by the President’s Commission on Law Enforcement and Administration of Justice (1967) (Section 2–A)—and discuss how BJS’s data collections roughly conform to that model (Section 2–B). Analysis of the funnel suggests some major gaps in U.S. justice statistics—and BJS’s statistical coverage in particular—which we describe in Section 2–C; these include difficulty in addressing new and emerging types of crime and contextual factors that apply to a wide range of criminal activities. 2–A THE “FUNNEL” MODEL OF FLOWS IN THE CRIMINAL JUSTICE SYSTEM Many date the emergence of criminal justice research on the public scene with the release of the 1967 report of the President’s Commission on Law Enforcement and Administration of Justice, The Challenge of Crime in a Free Society. The report and its many companion volumes summarized what was known about criminal justice and called in virtually every section for more information about how the system operated. The first chapter of the main report introduced as its organizing metaphor a flowchart presenting “a simple yet comprehensive view of the movement of cases through the criminal justice system” (President’s Com-
OCR for page 49
Ensuring the Quality, Credibility, and Relevance of U.S. Justice Statistics mission on Law Enforcement and Administration of Justice, 1967:8–9). The chart was meant to emphasize that “a study of the system must begin by examining it as a whole,” that “the criminal process … is not a hodgepodge of random actions” but rather “a continuum—an orderly progression of events” (President’s Commission on Law Enforcement and Administration of Justice, 1967:7). BJS continues to publish the chart—essentially identical in content to the 1967 version—because the model remains popular and useful for studying flows in the criminal justice system; a current version of BJS’s publication of the chart is shown in Figure 2-1. Moving from “crime” through the many paths by which accused persons might eventually progress “out of system,” this chart illustrated some of the report’s main points. One was that the major institutions that make up the criminal justice system—the police, the courts, and corrections—are interdependent (President’s Commission on Law Enforcement and Administration of Justice, 1967:7) What each one does and how it does it has a direct effect on the work of the others. The courts must deal, and can only deal, with those whom the police arrest; the business of corrections is with those delivered to it by the courts. How successfully corrections reforms convicts determines whether they will once again become police business and influences the sentences the judges pass; police activities are subject to court scrutiny and are often determined by court decisions. And so reforming or reorganizing any part or procedure of the system changes other parts or procedures. Another was the complexity of the system, which the report contrasted to “the popular, or even the lawbook, theory of everyday criminal process” (President’s Commission on Law Enforcement and Administration of Justice, 1967:7). This complexity was illustrated by the many forks in the road along which individuals might travel, some of which led to a quick exit whereas others promised to involve them for years to come. Finally, the report expressed concern about the fairness as well as the effectiveness of the system that was on view. It noted that, “throughout the system the importance of individual judgment and discretion, as distinguished from stated rules and procedures, has increased.” It concluded that “a consideration of the changes needed to make it more effective and fair must focus on the extent to which invisible, administrative procedures depart from visible, traditional ones” (President’s Commission on Law Enforcement and Administration of Justice, 1967:10). Practically, the chart also served as a starting point for the commission to discuss what was not known then—and, in some cases, what is still not known—from data on the justice system. As the chart’s legend notes, “the differing weights of line [in the chart] indicate the relative volume of cases disposed of at various points in the system, but this is only suggestive since
OCR for page 50
Ensuring the Quality, Credibility, and Relevance of U.S. Justice Statistics Figure 2-1 Sequence of events in the criminal justice system SOURCES: Adapted from President’s Commission on Law Enforcement and Administration of Justice ( 1967: 8–9) in 1997; posted at http://www.ojp.usdoj.gov/bjs/flowchart.htm.
OCR for page 51
Ensuring the Quality, Credibility, and Relevance of U.S. Justice Statistics no nationwide data of this sort exists” (President’s Commission on Law Enforcement and Administration of Justice, 1967:8). Our review of BJS’s statistical programs generally follows the flow of this historic chart, for many of them are also organized around the institutions and organizations that make up the justice system. The review also reflects many of the commission’s original concerns. Like the commission’s report, we emphasize the importance of understanding the interface between the parts of the system as well as the decisions that structure the flow of individuals within each of its components and their eventual exit from the system. Reflecting continued concern about the fairness issues raised by the commission, we examine the utility of the data for assessing the distribution of the outcomes of these decisions, decisions that constitute “justice” for those who are subject to them. 2–B BJS DATA COLLECTIONS IN THE CONTEXT OF THE CRIMINAL JUSTICE SYSTEM For the panel’s benefit, BJS staff provided a listing of its varied data collections, indicating their approximate coverage of various steps relative to a stylized version of the crime sequence model (Figure 2-1) that more graphically resembles a “funnel.” The resulting diagram is shown in Figure 2-2. The diagram shows individual BJS data series and so does not explicitly mention BJS’s grantmaking functions, which would show up more directly in a listing of BJS programs. For instance, “Firearm Inquiry Statistics” are a basic summary of use of the National Instant Criminal Background Check System, which BJS does not directly administer but which it provides grants to local agencies to improve. The “ranges” of coverage indicated on the diagram are approximate in nature. This particular schematic underplays the potential range of the National Crime Victimization Survey (NCVS) and its supplements. Because its primary focus is on victimization incidents, it makes most sense to place the NCVS at the leftmost point on the diagram, corresponding to the occurrence of crime. However, the NCVS includes victimization incidents that are not reported to police and hence do not start the criminal process, as suggested by the diagram; part of the NCVS’s unique value is its ability to provide information on the characteristics of these incidents that “leak” out of the system in the earliest stages. Moreover, the NCVS is flexible enough to provide information on incidents of interpersonal violence that may not formally be “crime,” and it can also speak to individuals’ experiences with other, later parts of the system, such as contacts with police that do not result in arrest or experiences with court proceedings. Likewise, the coverage range may overstate the scope of the Justice Sys-
OCR for page 52
Ensuring the Quality, Credibility, and Relevance of U.S. Justice Statistics Figure 2-2 Mapping of Bureau of Justice Statistics data series to sequence of events in the criminal justice system SOURCE: Adapted from chart provided by the Bureau of Justice Statistics to the panel for its first meeting, February 2, 2007.
OCR for page 53
Ensuring the Quality, Credibility, and Relevance of U.S. Justice Statistics tem Expenditure and Employment Extracts, which appear to be the widest-ranging and, seemingly, most comprehensive of the data series. This measure of local governments’ reported spending on justice-related activities dates back to a special study in the late 1960s in which the Census Bureau compiled spending totals on police, corrections, and judicial processing for some of the large jurisdictions in their annual finance and employment surveys. Based on this initial work, a supplemental mail survey of local governments specifically on justice expenditures was first developed and fielded by the Census Bureau in 1969 and repeated until 1979. Though sample size was boosted in 1979 (with the passage of the Justice System Improvement Act that created BJS in its present form), it was canceled 1 year later for budgetary purposes. An expenditure survey has subsequently been conducted on a sporadic basis (albeit not since 1997). For regular collection of expenditure data, BJS has since adopted the original strategy for the collection: The annual Justice System Expenditure and Employment Extracts are derived from data collected by the Census Bureau’s Governments Division.1 The extracts are comprehensive relative to the system because they attempt to capture expenditures for the system’s major components; however, they suffer from being less substantive (comporting with the categories in the Census Bureau’s finance surveys) and less geographically detailed than might be possible with a survey-based collection (U.S. House of Representatives, Committee on the Judiciary, 1991:160–161). Tabulations from the Justice System Expenditure and Employment Extracts are currently released in electronic-only format as a set of spreadsheet files (e.g., Bureau of Justice Statistics, 2007a); the most recent report summarized fiscal year 2003 data and was released in 2006 (Hughes, 2006). 2–C GAPS IN THE COVERAGE OF CRIME AND JUSTICE STATISTICS Study and comparison of the “funnel” model of flows through the criminal justice system (Figure 2-1) and the illustration of BJS data series that correspond to those flows (Figure 2-2) suggest a few fundamental issues in BJS’s statistical coverage. One such issue is that the stylized BJS mapping (Figure 2-2) emphasizes a forward flow through the system from stage to stage. However, in doing so, it understates a key element that the fuller conceptual model of Figure 2-1 depicts: There are significant “leaks” in the funnel structure. These leaks, or diversions from the forward flow of the process, include matters that are of 1 Specifically, the data are derived from the Census Bureau’s Annual Government Finance Survey and Annual Survey of Public Employment. The Governments Division is part of the Census Bureau’s economic directorate, and the Census Bureau’s government programs were recently reviewed in National Research Council (2007).
OCR for page 54
Ensuring the Quality, Credibility, and Relevance of U.S. Justice Statistics keen interest for understanding the system as a whole: for instance, acts of violence that never enter the system because they are not reported to police, dismissals of charges in the early stages of prosecution, alternative court resolutions that prevent trials from following the usual channel, and plea agreements that circumvent parts of the process. These leaks are inherently difficult to measure because, by their nature, they may not show up in the extant data resources at any particular stage in the process. Another issue with the forward-flow sieve model is that it tends to cast the late stages of the process (e.g., corrections) as terminal or ending states. Accordingly, the model understates issues of reentry into the community from corrections, as well as recidivism. Figure 2-1 generally shows the last stage in the process as an individual being “out of system”; it does not explicitly show the trajectory that would place individuals back in custody (e.g., a “feedback loop” all the way to the start of the process and the possibility of either committing a new offense or being victimized again). Any search for gaps in BJS’s program coverage—types of crime that are not described well or at all in current collections or additional frontiers in the understanding of crime and justice on which statistical information would be valuable—will inevitably generate a considerable list. The topic area of crime and justice is sufficiently broad that is relatively easy to rattle off long lists of important and interesting topics that a better funded BJS could take on; the perennial problem is reconciling that “ideal” list of desired knowledge with realistic resource assignments. Singling out “gaps”—major or minor—in BJS’s coverage is not meant as a criticism of BJS in any way, but rather a reflection of practical realities and a suggestion of possibilities. Indeed, BJS itself goes through this exercise, recognizing gaps in its coverage. In a 2007 meeting with the executive committee of the Justice Research and Statistics Association, BJS staff articulated a “top 20” list of data needs and information gaps that it had developed as part of the process of preparing its budget requests (Sedgwick and Ramker, 2007:3): Statistics on the extent and usage of private security services; Expanded analysis of elderly victims of crime, including data on the prosecution of elder abuse and mistreatment; Data on crimes of human trafficking; Development of a statistical system to study outcomes of ex-offender employment programs, and prisoner reentry in general; Standardization of the Record of Arrest and Prosecution (“rap sheet”) criminal histories maintained by law enforcement and accelerated adoption of new standards for rap sheets; Information on the characteristics of “frequent fliers” who are incarcerated in local jails on a repeated basis;
OCR for page 55
Ensuring the Quality, Credibility, and Relevance of U.S. Justice Statistics Development of an infrastructure to study recidivism using criminal history record databases; National data on the use of lethal force by police; Data on the nature and extent of citizen complaints concerning behavior of local law enforcement; Information on the sentencing of felony identity theft offenders; Inventory of law enforcement “cold case” forensic units; Systematic collection of data on security threats and conditions in state and local courthouses; Fuller collection of contextual information on weapon usage, gang involvement, and drug involvement in incidents included in the National Incident-Based Reporting System (NIBRS); Data on predation and exploitation of children via the Internet; Data on juveniles processed as felony defendants in (adult) criminal courts; Creation of an establishment survey of U.S. businesses on computer security; Expanding participation by law enforcement agencies in common information-sharing databases; Fuller data on law enforcement use of excessive (but nonlethal) force; Fuller data on electronic crime and identify theft; and Articulation of particular challenges to law enforcement (including staffing and resources) in a post-9/11, heightened security environment. In the balance of this section, we discuss four major “gaps” in BJS’s data collection portfolio—things that are conspicuously absent in Figure 2-2. Two of these gaps are related because, as noted earlier, the “funnel” model of the justice system is perhaps too easily equated with the processing of violent crime; other classes of crime, such as white-collar offenses, and civil judicial proceedings are not well captured by current systems. A third gap is that contextual factors associated with crime are inherently difficult to describe—and even characterize consistently—at all steps in the criminal justice system; we describe the relation of drugs and crime as an example. A fourth major gap that is plainly apparent from a comparison of Figure 2-2 with the general framework of Figure 2-1 is the processing of juvenile offenders and victims. In discussing this topic, we also consider the relationship between BJS and one of its sister data-gathering units in the Department of Justice, the Office of Juvenile Justice and Delinquency Prevention (OJJDP). Our assessment of
OCR for page 56
Ensuring the Quality, Credibility, and Relevance of U.S. Justice Statistics what should be done concerning each of these gaps is similar, and so we close the section and chapter with a common discussion. We defer our commentary on a fifth—and possibly most severe—“gap” or flaw in BJS’s existing coverage until the next chapter; this gap is the concept that the justice system is more than the sum of its parts, and that longitudinal flows throughout the system as a whole are not well measured at present. However, this topic is best considered after we have discussed more of the content of BJS’s portfolio in Chapter 3, and so we defer the discussion of longitudinal structures until Section 3–F.1. 2–C.1 White-Collar Crime The degree to which the measurement of “crime” is primarily focused on violent or street crime and major property crimes such as theft and arson is due, at least in part, to the long-standing definition of “Type I crimes” in the Federal Bureau of Investigation’s (FBI’s) Uniform Crime Reporting (UCR) program (which we describe in more detail in Section 4–C.1). The major definitions and conceptions of crime from the UCR were carried over to BJS’s NCVS, and other programs. Yet both the UCR and the NCVS—the nation’s two principal indicators of crime—have been critiqued for being slow to catch up to new crime types. A report of the FBI’s Criminal Justice Information Services Division (Barnett, 2000:2) conceded that “it is well documented that the major limitation of the [UCR] Summary Reporting System is its failure to keep up with the changing face of crime and criminal activity.” Likewise, our panel’s interim report discussed the challenges involved in achieving the NCVS’s full flexibility in studying new types of victimization (National Research Council, 2008b:Sec. 3–C.1). A focus on certain forms of violent and property crime does not account for all important types of crime, or crime types that emerge with the introduction and maturation of new technologies. A particular gap is in the measurement of many forms of what could loosely be labeled as white-collar crime. The term “white-collar crime” has been in currency since sociologist and criminologist Edwin Sutherland introduced it in 1939. Some interpret the term narrowly (e.g., Sutherland’s early focus on crimes committed by a person of high responsibility in the pursuit of his or her occupation) whereas others interpret it more broadly. For instance, the Federal Bureau of Investigation (1989:3) has defined white-collar crime as: those illegal acts which are characterized by deceit, concealment, or violation of trust and which are not dependent upon the application or threat of physical force or violence. Individuals and organizations commit these acts to obtain money, property, or services; to avoid the payment or loss of money or services; or to secure personal or business advantage.
OCR for page 57
Ensuring the Quality, Credibility, and Relevance of U.S. Justice Statistics We use the term more broadly to refer to crimes such as corporate fraud, health care fraud, financial institution fraud, money laundering, government fraud, consumer fraud, public corruption, and Internet crimes. Recent survey evidence suggests that the public views white-collar crime at least as seriously as traditional forms of crime, and offenses committed by organizations or by higher-status persons as more serious than those committed by individuals or lower-status persons. With sponsorship from the Bureau of Justice Assistance, the National White Collar Crime Center (NW3C) conducted a “National Public Survey on White Collar Crime” in 2005. Based on the survey, the NW3C concluded that the general public views white-collar crime as seriously as traditional crime types. In addition, the survey also yielded the finding that (Kane and Wall, 2006:3): Crimes involving physical harm are seen as significantly more serious than those crimes that incur a monetary loss only; organizational offenses are viewed more harshly than those committed by individual offenders; and crimes committed by high-status offenders (those in a position of trust) are seen as more severe than those crimes committed by [non-high-status] persons. There is also evidence that white-collar crime is a concern to law enforcement. Amidst its recent major change in strategic priorities to the deterrence of terrorism, the Federal Bureau of Investigation (2004a) prominently included a white-collar crime plank in its strategic goals for 2004–2009: IIH.1 Reduce levels of corporate fraud by targeting those groups or individuals engaged in major corporate fraud schemes that significantly impact the investing public and financial markets. IIH.2 Reduce the incidence of large scale health care frauds, involving both government-sponsored and private insurer programs. IIH.3 Reduce fraud perpetrated by criminal enterprises targeting financial institutions. IIH.4 Disrupt and dismantle the most significant money laundering institutions and facilities. IIH.5 Reduce the impact of telemarketing, insurance, and investment fraud on businesses and individuals, particularly schemes originating from outside the United States. IIH.6 Address those investigative matters which represent the most significant economic losses within federally-funded procurement, contract, and entitlement programs, environmental crimes, bankruptcy fraud, and anti-trust offenses. In its strategic plan, the FBI further argues that “the ability of the U.S. Government and industry to function effectively [is] threatened by complex frauds” and that continuance of its “successful efforts in the white collar crime arena” is important to “ensure the integrity of government expenditures of taxpayer funds [and] protect individuals and businesses from catastrophic economic loss.”
OCR for page 64
Ensuring the Quality, Credibility, and Relevance of U.S. Justice Statistics [OJJDP] in developing and implementing programs in the juvenile justice and delinquency field” (42 USC § 3789i) In comparing OJJDP’s statistical collections on juveniles with BJS’s series for adults, it is important to keep in mind that the two agencies have quite different missions. BJS’s duties under the law (Box 1-2) are primarily oriented toward data collection, with some provisions for assistance to state and local authorities. By comparison, the stated purposes in OJJDP’s enabling legislation (42 USC § 5602) are exclusively concerned with programmatic support for state and local assistance: The purposes of this section of [this section of law] are— to support State and local programs that prevent juvenile involvement in delinquent behavior; to assist State and local governments in promoting public safety by encouraging accountability for acts of juvenile delinquency; and to assist State and local governments in addressing juvenile crime through the provision of technical assistance, research, training, evaluation, and the dissemination of information on effective programs for combating juvenile delinquency. The agency’s formal mission statement is likewise geared toward influencing policy at the state and local levels: OJJDP “provides national leadership, coordination, and resources to prevent and respond to juvenile delinquency and victimization” and “supports states and communities in their efforts to develop and implement effective and coordinated prevention and intervention programs” (http://ojjdp.ncjrs.org/about/missionstatement.html). That said, in carrying out its mission, OJJDP develops and sponsors a series of data collection programs—in some cases developing operations parallel to BJS work for adult facilities and processes, and in others making use of BJS data. Its efforts in data gathering and coordination are routinely made available in an online “Statistical Briefing Book” (http://ojjdp.ncjrs.gov/ojstatbb/index.html) that is maintained by the Pittsburgh-based National Center for Juvenile Justice (NCJJ) through a grant from OJJDP. OJJDP also regularly produces a “National Report” in its Juvenile Offenders and Victims series (Snyder and Sickmund, 2006). Regarding original data, OJJDP sponsors the collection of primary data in three areas: (1) juvenile facilities and juveniles in residential placement; (2) juvenile court statistics; and, to a more limited extent (3) juvenile victimization and offending. On the first of these, corrections and residential placement, OJJDP has sponsored a person-based Census of Juveniles in Residential Placement (CJRP) in odd-numbered years and a facility-based Juvenile Residential Facility Census (JRFC) in even-numbered years since the late 1990s.4 In their structure, OJJDP’s collections mirror the strategy used by 4 Typically, the reference date for the CJRP is in October of an odd-numbered year; “how
OCR for page 65
Ensuring the Quality, Credibility, and Relevance of U.S. Justice Statistics BJS in its series on adult prisons and jails, alternating between inmate-based and facility-based data; they also bear similarities because OJJDP employs the Census Bureau as its data collection agent as is true of the adult series. However, the OJJDP person-based series differ from their adult counterparts in that they rely entirely on facility records and reporting rather than direct personal interviewing. Collectively, the CJRP and JRFC provide information on facilities (type, physical layout, counts, use of locked doors/gates) and individual juveniles held in residential facilities (demographic characteristics, placement authority, most serious offense, adjudication status, security status, etc.). Because they use juvenile-specific facilities as their frame, the OJJDP collections do not capture those juveniles held in adult prisons or jails; mental health and drug treatment facilities are also not a part of the CJRP or JRFC frames, and so juveniles housed there are not determined either. Although the OJJDP Statistical Briefing Book provides a tool for generating national and state summaries from the CJRP,5 more detailed data from the CJRP are available to researchers only on a case-by-case basis. Through NCJJ, OJJDP makes available juvenile court statistics with information on the activities of juvenile courts in the United States and the cases disposed of in these courts. Similar to the FBI’s UCR program, data in this archive seem to be based upon voluntary submissions by juvenile courts. Thus, the number of participating courts varies each year; in addition, some courts provide detailed information for each case whereas others provide only aggregate counts. These data are used to provide national portraits of juvenile offenders and juvenile court activity, including information on court caseloads, variation in delinquency cases by demographic characteristics of youth involved, detention, disposition of cases, and the flow of cases through the juvenile justice system. Status offense cases are also considered, but to a lesser degree, and juveniles waived to adult courts are not followed beyond the waiver decision. Finally, with regard to juvenile victimization and offending, OJJDP does not routinely sponsor supplements to BJS’s NCVS. However, it sponsored a module on crime, delinquency, and arrest for inclusion in the National ever, a set of unforseen circumstances prevented the 2005 mailout from taking place in October of that year.” The mailout was pushed to February 2006, hence an apparent deviation in OJJDP’s annual collection strategy (http://www.ojjdp.ncjrs.gov/ojstatbb/cjrp/asp/methods.asp). The CJRP and JRFC continue and extend an earlier data collection fielded by the Justice Department since the early 1970s: the Census of Public and Private Juvenile Detention, Correctional, and Shelter Facilities, better known as the Children in Custody census. The innovation in creating the two newer, separate series was the addition of individual-level characteristics in the CJRP rather than summary-level counts. 5 As part of its strategy to protect confidentiality and prevent individual juveniles from being identified from such tabulations, “OJJDP has adopted a policy that requires all published table cells be rounded to the nearest multiple of three” (http://www.ojjdp.ncjrs.gov/ojstatbb/cjrp/asp/methods.asp).
OCR for page 66
Ensuring the Quality, Credibility, and Relevance of U.S. Justice Statistics Longitudinal Survey of Youth 1997 (NLSY97), conducted by the Bureau of Labor Statistics.6 The NLSY97 is based on a national sample of youth who were 12–16 years old as of December 31, 1996; they are interviewed annually, with the tenth round in 2006 reaching 7,559 of the original respondents (http://www.nlsinfo.org/nlsy97/nlsdocs/nlsy97/97sample/introsample.html). Through the OJJDP module, and related questions on general risk behaviors among youth, the NLSY97 includes questions on substance use, delinquency and deviance (e.g., status offenses, gang membership, arrests, property offenses, carrying guns), as well as incidents of criminal victimization. In addition to its own sponsored data sets, OJJDP’s Statistical Briefing Book compiles information from a wide variety of agencies and organizations to permit assessments of juvenile victimization, offending, and law enforcement experiences. Specifically, the briefing book includes results from, among others: The NCVS; The FBI’s UCR and, for participating jurisdictions, the National Incident-Based Reporting System; The Department of Health and Human Services’ National Child Abuse and Neglect Data System Child File (NCANDS); The National Institute of Drug Abuse’s Monitoring the Future Survey; and The Centers for Disease Control and Prevention’s National Youth Risk Behavior Survey (NYRBS). These other data sources provide more richness in detail for particular incident types than the NCVS can provide alone. For example, the NCANDS allows assessments of victim, caretaker, and perpetrator characteristics, and responses to abused or neglected children, and so is useful for studying specific kinds of victimization of youth. The NYRBS is a school-based sample (9th to 12th graders) that allows for additional information on victimization and participation in additional types of crimes and errant behaviors such as fights, suicide attempts, and alcohol and drug use. Considering both OJJDP and BJS’s portfolios, there remain important gaps in the coverage of youths’ activities. A major one is that most of the available data collections—particularly the NCVS and the high school–based NYRBS—provide reasonable coverage of adolescents but not the complete juvenile population. Through its focus on caretakers and perpetrators, the NCANDS is unusual in its potential coverage of all ages. Further, whatever the age range covered by the data series, the data are typically aggregated 6 See http://www.nlsinfo.org/nlsy97/nlsdocs/nlsy97/97sample/introsample.html on the credit for OJJDP’s sponsorship of the questions.
OCR for page 67
Ensuring the Quality, Credibility, and Relevance of U.S. Justice Statistics and reported for broad age categories (e.g., under age 12, 12–15, over 15, or simply under 18). Thus, it is not always possible to assess activities and situations for young people across the youthful life course. Importantly, too, some youthful populations are simply not considered. For example, OJJDP reports do not typically consider youth in adult courts or custodial facilities, beyond documenting who is waived. BJS gives attention to this group, but these data are not linked with the records from juvenile courts and facilities. Consequently, meaningful assessments of which types of youth are handled in the juvenile justice versus the criminal justice system are not readily available. Nor is there much information on how youth fare who transition out of the juvenile court to the adult courts. Ideally, record and data-reporting systems regarding juveniles under age 18 and adults over 18 would not be quite so hard and fast because the division at 18 does not represent hard and fast differences in maturation of young people. 2–C.4 Drugs and Crime Amidst growing expenditures in the federal government’s “war on drugs” policies, reauthorization legislation enacted in 1988 added three specific clauses on data collection on drugs and crime to BJS’s statutory mandates, as shown in items 16–18 in Box 1-2. The new clauses referred to the “drug crisis” and the “overall national anti-drug strategy,” and tasked BJS with establishing a “a national clearinghouse for the gathering of data generated by Federal, State, and local criminal justice agencies on their drug enforcement activities,” particularly data that could be used to demonstrate the efficacy of programs and intervention efforts. By this language, Congress and the administration—not for the first time, and not for the last—assigned BJS a task that was difficult under the best of circumstances and virtually impossible given tight fiscal resources. Contextual factors associated with crime pose particular difficulties for measurement. They can be difficult to define in general and can be particularly difficult to define consistently across multiple data sources or throughout the different steps in justice system processing. In addition, different data collection types—for instance, personal survey interviewing versus coding from written police reports—may be especially strong at capturing some factors but weak at others. The kinds of contextual factors we refer to here include what Cook (1991) has described as “the technology of personal violence”—the use of weaponry, particularly firearms, in crime—which can affect the probability of success of the crime, the consequences to the victim, the responses of law enforcement, and the implications for punishment in the court system. The geography of crime—more than just latitude and longitude, including social and physical conditions and community resources in an area—is another crucial and challenging contextual factor that has
OCR for page 68
Ensuring the Quality, Credibility, and Relevance of U.S. Justice Statistics grown into a particularly vibrant area of criminological research in recent decades. In this section, we focus on another important crosscutting contextual factor—the interplay between drugs and crime—as an example principally because of the explicit references in BJS’s enabling legislation. Drug crimes—offenses involving the possession and sale of controlled substances—are a major area of criminal activity that resists an easy fit with the conceptual model of the crime funnel. Drug offenses account for about 30 percent of admissions to state prisons, but arrests for drug offenses tend not to follow the initial sequence of events in the funnel where a crime (or victimization) leads to a report or complaint to police and, in turn, to an arrest. Rather, the pattern of drug-specific arrests is more closely linked to targeted enforcement efforts which vary greatly across jurisdictions. In addition, within the court processing steps of the funnel, drug cases resist simple categorization because laws regulating drugs vary across states and within states, and change over time. Thus possession of a small quantity of drugs, for example, may be only a minor violation in some jurisdictions, but a felony in others. Over and above the difficulties involved in mapping drugs and drug crime to the funnel framework, drug-related activities are generally difficult to measure. The sensitive nature of inquiries on drug use make it a behavior that can be challenging to measure accurately through self-report techniques such as survey interviewing. Those data resources that do cover aspects of drug use and drug-related criminal and violent behavior include both probability surveys and records-based series, and each has unique strengths and weaknesses. A partial list of these resources—past and present, conveying the range of data collections that have touched on drug-related issues—includes the following: The National Survey on Drug Use and Health (NSDUH; formerly the National Household Survey on Drug Abuse) is sponsored by the Substance Abuse and Mental Health Services Administration (SAMHSA). It has been in operation since 1971 and has utilized RTI International, Inc., as its data collection agent since 1988; it is perhaps the main source of statistical information on the use of illegal drugs by the U.S. population. The Monitoring the Future Survey has been conducted by the Institute for Social Research at the University of Michigan since 1975, asking secondary school students about drug use as well as other risk behaviors. Both of these surveys share a public health orientation, asking about drug use and dependency, and offer relatively little information to distinguish criminal drug activities. The Arrestee Drug Abuse Monitoring (ADAM) program (formerly the Drug Use Forecasting [DUF] program) was a data collection sponsored by the National Institute of Justice that produced data between 1987 and 2003. In select, participating sites, new arrestees were in-
OCR for page 69
Ensuring the Quality, Credibility, and Relevance of U.S. Justice Statistics terviewed within 48 hours and asked a battery of questions on arrest history, drug use patterns, drug acquisition, and prior participation in treatment programs. Survey results were combined with—and could be compared against—the results of urinalysis to detect the presence of 10 drugs (but focusing in particular on cocaine, marijuana, methamphetamine, opiates, and phencyclidine [PCP]). A wide-ranging redesign in 1999 gave ADAM a sounder basis in probability sampling, started a “calendaring” routine in the questionnaire to cue arrestees to document drug use patterns over longer period of times, and positioned ADAM to have relatively easy “crosswalk” connections for linkage to other data collections such as NSDUH (National Institute of Justice, 2003:4, 13). DUF and ADAM data were subject to criticism over their representativeness and because they could function only as an indirect indicator of drug market activity (Caulkins, 2000:397–398); ultimately, the major flaw of the program is that it became unsustainable in light of constrained budgetary resources. Data collection was suspended in 2004 and has not since been reactivated. Like the ADAM program, the Drug Abuse Warning Network (DAWN) avoids the use of self-reporting in measuring drug use. Instead, this SAMHSA-funded surveillance system collects data from hospital emergency departments (on drug-related visits) and medical examiner offices (on drug-related deaths). DAWN suffered from well-documented problems and key limitations (e.g., Caulkins et al., 1995), prompting a major redesign effort between 1997 and 2003 (Substance Abuse and Mental Health Services Administration, 2002). As of November 2008, links to publications based on “New DAWN” data on the program’s website (http://dawninfo.samhsa.gov), from 2006 onward, all include a “caution” note that “SAMHSA is currently reviewing the estimates in this report and expects to publish revised estimates at a future date,” suggesting potential instability in the redesigned program’s estimation routines. The Drug Enforcement Administration’s (DEA) System to Retrieve Information from Drug Evidence (STRIDE) compiles information from drug sales to undercover federal agents. By their nature, STRIDE data are uniquely positioned to provide information on the prices paid for drugs in those undercover transactions and the quality (purity) of the purchased drugs. However, the reliability of these data for economic and policy analyses—for instance, how closely the prices paid in the transactions logged by undercover federal agents track with prices in the broader illegal drug market, and the degree to which they represent federal (DEA) interdiction priorities rather than local-level activities—was ruled to be inadequate by the National Research Council (2001a).
OCR for page 70
Ensuring the Quality, Credibility, and Relevance of U.S. Justice Statistics That previous National Research Council (2001a) panel reviewed the extant data sources on drugs, drug markets, and the connection between drugs and crimes, and provides fuller descriptions of these and other data programs. In policy analysis, public conversation, and research, measures of drug crime tend to be proxied by measures of enforcement, though the empirical relationship between drug crime and drug arrests is poorly understood. Drug arrest rates grew substantially through the 1980s, for example, while self-reported drug use among high school seniors was falling. Still, patterns such as these have multiple interpretations. Enforcement efforts may have been moving in the opposite direction of trends in use. Drug enforcement may have been reducing drug use. Increased enforcement may also have reduced survey respondents’ propensity to report drug use. The summary of the state of quantitative knowledge of drugs and crime by Caulkins (2000:394, 395) remains apt: In the drug policy arena we have an abundance of numbers, but the glass of insight is at best half full.… We know quite a bit about drug offenders within the criminal justice system but much less about their activities on the street. We know quite a bit about how many drug users there are but little about why there are so many. In contrast, we understand why people sell drugs but know little about how many upper level dealers there are, let alone how they operate…. More generally, existing data systems are reasonably adequate for describing patterns and trends but generally are incapable of explaining them, in part because opportunistic instead of random samples and the absence of control groups makes it difficult to tease out causal relationships. Though BJS refers to some of the data sources listed above on its website, it generally attempts no analytical work based on those data. “Drugs and crime” is a top-level link on the BJS home page (http://www.ojp.usdoj.gov/bjs/drugs.htm), the principal link on which is to the electronic BJS publication Drugs & Crime Facts (Dorsey et al., 2004); that report does briefly attempt to pull together a series of findings about drugs from a number of BJS and non-BJS data sets. A number of BJS analyses have also summarized findings of drug-related questions in BJS’s standard data series; for instance, Wilson (2000) summarizes both the Annual Survey of Jails’ information on drug services provided by facilities and the Survey of Inmates in Local Jails’ queries on arrests for drug offenses and prior drug use (these two collections are summarized in Section 3–B.2). Still, in a section of BJS’s strategic plan that briefly itemizes data collections under major section headings, the section on drug crime statistics notes only that “many ongoing BJS statistical series collect and analyze data on drugs and crime” (Bureau of Justice Statistics, 2005a:Fig. 1); in the absence of a coherent overview of what is and is not known from existing data, the interactions of drugs and crime must be considered a gap in the coverage of BJS’s overall portfolio.
OCR for page 71
Ensuring the Quality, Credibility, and Relevance of U.S. Justice Statistics 2–D ASSESSMENT: FILLING THE GAPS In the panel’s assessment, the four topics we have described in Section 2–C constitute clear gaps in BJS’s statistical coverage of the events and interactions inherent in the criminal justice funnel model of Figure 2-1: Finding 2.1: The data on crime currently collected by BJS are primarily focused on street crime. This focus on certain forms of violent and property crime does not account for important or emerging types of crime—notably, many forms of white-collar crime such as corporate fraud, health care fraud, financial institution fraud, money laundering, government fraud, consumer fraud, public corruption, and Internet crimes. The broad area of civil justice proceedings—distinct from criminal justice—is represented by one principal data series in BJS’s extensive portfolio, and is limited by its construction to cover only completed court cases (and not out-of-court settlements). BJS’s slate of cross-sectional series also does not readily provide for comprehensive analyses of contextual factors such as drugs and their impact on crime and violence. Finding 2.2: Responsibility within the U.S. Department of Justice for coordinating and organizing data collections on juveniles is generally assumed by the Office of Juvenile Justice and Delinquency Prevention (OJJDP), instead of BJS. Though BJS’s series do cover some segments of the juvenile population (e.g., juveniles housed in adult correctional facilities), the results of BJS and OJJDP studies are not well integrated. Within both BJS’s and OJJDP’s statistical coverage, there remain substantial gaps in data for juvenile offenders and victims with respect to their processing through the justice system “funnel.” Clear though these gaps are, it is equally clear that considerable care and caution are in order when suggesting what to do about them. The four topics we have profiled share the basic quality that they are massive and complex, and that crafting full and effective data collection strategies for them would require major innovations in BJS’s current concepts and protocols. A full focus on white-collar crime would require BJS to shift from its historical norm of using either people or a relatively limited number of establishments (e.g., correctional agencies) as their unit of analysis and grapple with the unique problems of businesses as a unit of study. BJS coverage of white-collar crime would ultimately benefit the large number of agencies that are already involved to varying degrees in the monitoring of such crimes, but would also require extensive coordi-
OCR for page 72
Ensuring the Quality, Credibility, and Relevance of U.S. Justice Statistics nation with those agencies; this would likely involve the need for an ongoing interagency advisory or coordinating board.7 Civil trials are a large part of the overall justice system, posing harms to both persons and businesses, and so fuller knowledge of civil justice would be highly beneficial. However, civil justice is also replete with serious definitional issues (hard-to-define concepts, exacerbated by extensive state-by-state variation in legal standards); even defining the range of possible ADRs and determining their applicability in various states is difficult, much less generating reliable counts of their use. Further, attempts to make the measurement problem more tractable by focusing on filed cases that enter the system miss the large fraction of potential “cases” that are resolved in private. As with white-collar crime, effective expansion of civil justice data collection would necessitate involvement and coordination with a number of other actors, including the Administrative Office of the U.S. Courts, the individual state court systems and NCSC, and the American Bar Foundation. It is complicated, or impossible, to reach the youngest end of the juvenile population through traditional self-interviewing methods, because of their age and cognitive development. The NCVS limits itself to respondents ages 12 and older and other existing data series similarly focus on adolescents and teens; although it is possible that age levels might feasibly be pushed lower (e.g., to age 10 or 11), we know of no evidence of the possible effect of such a switch on the accuracy of self-reports. Collection and release of data on juveniles are also subject to a wide array of legal and ethical restrictions, and studies of juvenile justice raise special sensitivities and heighten the involvement of intensely interested interest groups. Measuring the entry of juveniles into supervision or residential placement is somewhat complicated because referrals come from a variety of sources other than the police, including family members or guardians and state child welfare bureaus. OJJDP’s person-level measure of juveniles in correctional facilities relies on indirect responses, through reference to facility records and contact with administrators. A fuller assessment of the quality and coverage of data that may be available in school or juvenile facility records would have to accompany expanded data collections on the juvenile population. To be clear, these conceptual and operational complexities are only one part of the difficulty in suggesting that new data collections be developed to 7 These agencies include, at a minimum, the Federal Trade Commission, the Federal Deposit Insurance Corporation, the Federal Reserve Board, the Office of the Comptroller of Currency, the Office of Thrift Supervision, the National Credit Union Administration, the Mortgage Bankers Association, the U.S. Department of the Treasury, the National Center for Missing and Exploited Children, the FBI, the U.S. Department of Justice Criminal Division and Tax Division, the U.S. Postal Inspection Service, and the Securities and Exchange Commission.
OCR for page 73
Ensuring the Quality, Credibility, and Relevance of U.S. Justice Statistics fill these gaps. The other—and more acute—difficulty is that none of these gaps can be filled without extensive planning, new financial resources, and additional personnel. We think that it is clear that these areas of study are things with which the principal statistical agency of the U.S. Department of Justice should be concerned, but it is equally clear that it is unreasonable to expect that major progress could be made on any of these gaps within what is effectively BJS’s current flat-funding situation. Addressing any of these gaps would require commitment and a sense of high priority from the Justice Department, the administration, and Congress. Given these complexities, our basic suggestion on how to proceed borrows from two sources: the “clearinghouse” role defined by law for BJS on data on drugs and crime (point 17, Box 1-2) and OJJDP’s detailed online Statistical Briefing Book. In any of these gap areas, the necessary first step is a structured accounting of what is and is not known from existing data resources, both internal and external to BJS. BJS’s Drugs & Crime Facts is a first step toward such an accounting, as is the compilation of data from a wider variety of sources in the online Sourcebook of Criminal Justice Statistics. Still, these gap areas would benefit from a more analytical approach and more complete exploration of existing data sets (and their limitations); BJS’s website (and reports) should more completely catalog external data sources and research, particularly for subject areas where BJS’s own collections are limited. Through such a mapping of problem areas—and the more refined list of specific information needs that the mapping would suggest—BJS would have a more useful template for soliciting input on new data collections, should commitment of resources be secured. Recommendation 2.1: Consistent with its legal mandate to collect, analyze, and disseminate statistical information on all aspects of the justice system, BJS should (a) document and organize the available statistics on forms of crime not covered by the NCVS, the FBI’s UCR and NIBRS data systems, and other major data series maintained by other statistical agencies, (b) pursue research on what new statistics could be feasibly and usefully developed, and (c) propose such new data collections as the research suggests to be both feasible and useful. BJS should strive to function as a clearinghouse of justice-related statistical information, including reference to data not directly collected by BJS. Given our panel’s charge to consider BJS’s relationship to other data-gathering entities within the Department of Justice, we think that the gap of coverage of the juvenile population warrants its own specific recommendation. Having concluded that the funnel model of the justice system (Figure 2-1) is a useful and sound one, and given the numerous references to juvenile delinquency in BJS’s legal mandate, we think it odd that BJS’s ced-
OCR for page 74
Ensuring the Quality, Credibility, and Relevance of U.S. Justice Statistics ing of a complete branch of the funnel to another Justice Department entity is as complete as it is. In short, the measurement of juvenile justice is something in which the principal statistical agency of the U.S. Department of Justice should be fully engaged: Recommendation 2.2: In line with its original charge and to better document and understand the contribution of juveniles to street crime and violence, the victimization of youth, and the consequences for youth and society of their victimization and offending, BJS should develop juvenile victimization, crime, and justice statistical series suitable for describing the patterns of offending and victimization of youth, longitudinal progression of youth through the juvenile and criminal justice systems, and reentry into the community and criminal system. Taking on this responsibility would require additional resources. We hasten to add, however, that this recommendation should not be construed as saying that BJS should necessarily usurp (or “reclaim”) data collection functions from OJJDP. Like BJS, OJJDP has invested considerable time and effort in developing its relationships with its data collection providers, and upending those relationships should not be taken lightly. What we do envision through this recommendation is BJS-OJJDP collaboration on research on the full juvenile population, including, at a minimum, fuller study of juveniles processed by adult courts and correctional facilities.