Ensuring the Quality, Credibility, and Relevance of U.S. Justice Statistics (2009)

Below is the uncorrected machine-read text of this chapter, intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text of each book. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.

Summary T HE B UREAU OF J USTICE S TATISTICS (BJS) of the U.S. Department of Justice (DOJ) requested that the Committee on National Statistics of the National Academies (in cooperation with the Committee on Law and Justice) convene this Panel to Review the Programs of the Bureau of Justice Statistics. The panel has a broad charge to: examine the full range of programs of the Bureau of Justice Statistics (BJS) in order to assess and make recommendations for BJS’ priorities for data collection. The review will examine the ways in which BJS statistics are used by Congress, executive agencies, the courts, state and local agencies, and researchers in order to determine the impact of BJS programs and the means to enhance that impact. The review will assess the organization of BJS and its relationships with other data gathering entities in the Department of Justice, as well as with state and local governments, to determine ways to improve the relevance, quality, and cost-effectiveness of justice statistics. The review will consider priority uses for additional funding that may be obtained through budget ini- tiatives or reallocation of resources within the agency. A focus of the panel’s work will be to consider alternative options for conducting the National Crime Victimization Survey, which is the largest BJS program. The goal of the panel’s work will be to assist BJS to refine its priorities and goals, as embodied in its strategic plan, both in the short and longer terms. The panel’s recommendations will address ways to improve the impact and cost-effectiveness of the agency’s statistics on crime and the criminal justice system. This is the panel’s second and final report. At BJS’s request, the panel devoted the first phase of its work to considering broad options for conduct- ing the National Crime Victimization Survey (NCVS), arguably the agency’s flagship data collection and certainly its most expensive. We described such options in detail in our interim report, National Research Council (2008b), 5 

6 JUSTICE STATISTICS and the summary of that report is included in this volume as Appendix B. In this final report, we complement our work on the NCVS by considering the balance of BJS’s data collection series, its work in providing assistance to state and local authorities, and the functions and priorities of the agency as a whole. Established in its present form in 1979, BJS is one of the smallest of the principal agencies in the U.S. federal statistical system, yet it shoulders one of the most expansive and detailed legal mandates among the statistical agencies. Formally, BJS is tasked to provide information on crime and the operation of the justice system, across all levels of government. Like its ad- ministrative parent, the Office of Justice Programs (OJP), BJS was originally created as part of the former Law Enforcement Assistance Administration (LEAA); accordingly, BJS also inherits the LEAA’s strong mandate to assist state and local authorities—both financially and technically—in developing justice information systems. ASSESSMENT OF THE BJS DATA COLLECTION PORTFOLIO In our review of BJS programs, we have used as a guide a model of the se- ries of events in the justice system that was first developed by the President’s Commission on Law Enforcement and Administration of Justice (1967). Graphically, this model resembles a funnel, in which large numbers of crimes and victimization incidents are processed—in decreasing numbers—through major parts of the system, including police work, prosecution and adjudica- tion, and correctional supervision. Through our review of existing BJS data collections and the manner in which they map to the funnel model of the justice system, we conclude that BJS’s data collection portfolio is a solid body of work, generally well justified by public information needs or legal requirements. It represents a better-than-good-faith effort to marshal data relevant to an astoundingly large mandate, given that fiscal resources have typically been less than com- mensurate. Within its resources and the topics it has chosen to address, BJS has done well in the sense that nothing in its portfolio is obviously frivolous, wasteful, or inconsistent with its legal mandates. Our basic observations about BJS’s portfolio include: • BJS’s key victimization series, the NCVS, is its most expensive, most flexible, and most scrutinized collection. It is also, arguably, the agency’s most underutilized collection, undercut by scarce resources, diminishing sample size, and—to a degree—a lack of innovation in analyzing and promoting the data. • BJS’s corrections data series is a good example of a well-designed and integrated system of collections, wisely using different methods (per- 

SUMMARY 7 sonal interviewing and facility or administrative records) at certain time intervals for a range of facilities (prisons, jails, support agencies). The set of collections is designed such that censuses build the frame for subsequent, more detailed surveys. Going forward, the challenge for BJS’s efforts in the general area of corrections is to expand its cov- erage to include prisoner reentry issues, broadly, and to improve on its previous solid (but infrequent) studies of recidivism. • BJS’s work in law enforcement is hindered by an overly restrictive fo- cus on management and administrative issues, with little direct con- nection to data on crime, much less providing the basis for assess- ing the effectiveness of police programs. It is also in the area of law enforcement—with the proliferation of numerous special-agency cen- suses and little semblance of a fixed schedule or interconnectedness of series—where the need for refining the conceptual framework for multiple data collections is most evident. • Our critique of BJS’s work in adjudication is more a reflection on the general difficulty of measurement in the justice system than a criti- cism of BJS. Information systems in state court systems—and, indeed, the structure and jurisdiction of those courts—vary strongly in their accessibility and sophistication. The dominant impression is of the agency (with its data collection partners) doing the best it can with what it has. That said, there are numerous areas where improvement is needed: bolstering the adjudication series’ basis in statistical sampling and patching important gaps in statistical coverage of the justice sys- tem “funnel” (particularly declinations to prosecute and out-of-court settlements) would dramatically upgrade the relevance and utility of BJS’s data series. Through comparison to the funnel model, we identify four major sub- stantive gaps in BJS’s portfolio: • White-collar crime, including various types of fraud, public corrup- tion, and Internet crimes; • Civil justice (as opposed to criminal justice) matters—ranging from prosecution of nonviolent crimes to property disputes to divorce and custody arrangements—which are currently covered by only a single BJS collection (the Civil Justice Survey of State Courts) that, by design, is not able to collect information on out-of-court settlements; • Juvenile justice, authority for data collection of which is largely ceded to BJS’s fellow OJP bureau, the Office of Juvenile Justice and Delin- quency Prevention (OJJDP); and • Contextual factors such as the interaction between drugs and crime 

8 JUSTICE STATISTICS that cut across various parts of the system and are thus difficult to measure completely and consistently. These are important topics and ones on which the principal statistical agency of DOJ should be able to speak authoritatively. However, each of these areas involves major methodological complexities; to fill any of these gaps in BJS’s portfolio would require increased and sustained support from Congress and the administration in terms of staff and fiscal resources. In the interim, we recommend (Recommendation 2.1) that BJS play a stronger “clearinghouse” role related to these and other substantive gaps, documenting and organizing those statistics that are available and pursuing research on what new statistics could be feasibly and usefully developed. Our review of BJS programs leads us to suggest four broad strategic goals for the agency over the coming years. A STRONG POSITION OF INDEPENDENCE Strategic Goal 1: To establish and maintain a strong position of independence as a statistical agency; to serve as an independent and objective source of statistical information on crime and the administration of justice. BJS generally espouses the expected principles and practices of a federal statistical agency, but it has sustained major challenges to its independence as a national statistical resource in recent years. These include: • An attempt by Justice Department officials to alter the content and substantive emphases of a statistical press release announcing new es- timates on police-public contact during traffic stops, an incident that led to the dismissal of a BJS director; • The legal imposition, through the Prison Rape Elimination Act of 2003 (PREA), of reporting requirements on sexual violence in correctional facilities that oblige BJS to play a regulatory role, holding individual facilities up for sanction; and • Legislative changes that have the effect of tethering BJS to the policy objectives of its immediate administrative parent, OJP and DOJ gen- , erally. BJS’s uniquely precarious position—as a statistical agency nested within a program agency (OJP) that is dedicated to furthering policy objectives, nested in turn within the nation’s principal law enforcement department— means that its independent function as a source of independent and objec- tive information is under constant threat. For this reason, we identify the building of a position of independence as a statistical agency as BJS’s chief strategic goal. The panel concludes that this goal requires strong corrective 

SUMMARY 9 actions: changing BJS’s administrative placement within DOJ and the term of service of the BJS director. Organizational Changes There exists no organizational arrangement that can completely shield a statistical agency from threats to its independence, guaranteeing complete freedom from political or structural interference or the appearance thereof. That said, the measures we recommend are strong and should reduce the likelihood that BJS and its officials are inappropriately treated. Recommendation 5.3: BJS should be administratively moved out of the Office of Justice Programs, reporting to the attorney general or deputy attorney general. Recommendation 5.4: Congress and the administration should make the BJS director a fixed-term presidential appointee with the advice and consent of the Senate. To insulate the BJS director from political interference, the term of service should be no less than 4 years. By these recommendations and the suggested strategic goal statement, we also suggest that preserving a strong position of independence should be BJS’s first criterion for undertaking new data collections or revising ex- isting ones. Because of the experience with PREA reporting requirements, we recommend (Recommendation 5.1) that BJS should not provide or be required to provide individually identified data in support of functions that compromise BJS’s role—functions that involve collecting and analyzing sta- tistical data for policy-furthering, tactical, regulatory, and operational pur- poses. We further reinforce guidance issued by the U.S. Office of Manage- ment and Budget in the wake of the press release incident described above, recommending that any Justice Department review of BJS statistical prod- ucts and related communications preserve the content, the release schedule, and the mode of dissemination planned by BJS (Recommendation 5.2). BJS should position itself as a statistical resource to DOJ, not an “arm” for the furtherance of any policy objectives. To this end, reciprocal outreach and cooperation are necessary: The BJS director needs to reach out to other agencies within DOJ, forming partnerships to propose information collec- tion initiatives (Recommendation 5.5), and DOJ needs to build provisions for BJS collection of statistical information—the raw material for monitor- ing and evaluation—into its program initiatives aimed at crime reduction (Recommendation 5.6). 

10 JUSTICE STATISTICS Ensuring the Quality of the NCVS Statistical agencies can safeguard their independence through the quality and visibility of their products. They are better suited to ward off threats to independence if their data are essential indicators of national well-being; this produces support by a vocal set of stakeholders who are committed to their objectivity and accuracy. In our assessment, the NCVS can and should be positioned as such a critical social indicator, with its unique ability to provide insight into the etiology of crime and the characteristics of crimes not reported to police. As such, the NCVS should have clear primacy in BJS’s resource allocations. It is sufficiently core to BJS’s legally mandated duties and its basic function as a statistical agency that it is difficult to imagine an effective BJS without a strong and continuing NCVS. Accordingly, we reiterate in this report two recommendations that we first offered in our interim report: Recommendation 6.1: BJS must ensure that the nation has qual- ity annual estimates of levels and changes in criminal victimiza- tion. Recommendation 6.2: Congress and the administration should ensure that BJS has a budget that is adequate to field a survey that satisfies the goal in Recommendation 6.1. We further recommend (Recommendation 6.4) that any additional re- sources made available for the NCVS should be used not only to increase the reliability of annual estimates (i.e., rebuild the NCVS sample size) but also to supplement the survey in ways that increase understanding of criminal victimization and keep the content of the survey fresh. This includes seeking ongoing topic supplements such as the School Crime Supplement, with the regular support and cooperation of other federal agencies. It also includes fuller use of the NCVS to measure citizen involvement with other parts of the system (e.g., experiences with adjudication procedures and law enforce- ment). Given the utility of NCVS information on the needs of victims and the compensation and assistance goals of the Justice Department’s Office of Victims of Crime—we also suggest that Congress permit the use of funds obtained through the Victims of Crime Act to provide some funding for the collection and improvement of victimization data (Recommendation 6.3). Continuing and Strengthening State Justice Statistics Program and SAC Partnerships Through its State Justice Statistics program, BJS supports the operation of Statistical Analysis Centers (SACs) in states and several U.S. territories. The SAC network includes a mix of organizational arrangements—some 

SUMMARY 11 SACs function as independent state agencies, whereas others are affiliated with state justice or law enforcement departments or academic institutions. In the panel’s assessment, the BJS-state SAC network is a strong exam- ple of the federal-state cooperative systems that are important to statistical agency functioning. Support for the SAC network stands as a relatively low- cost activity on BJS’s part with great dividends in terms of outreach and feedback, as well as dissemination of data and products to state policy mak- ers. Moreover, the SAC network is mutually beneficial—states gain from the technical assistance that might otherwise be unavailable or cost-prohibitive to obtain and from national-level benchmarking that is useful for framing policy developments in the states, and BJS benefits from the inventiveness of research performed by its SAC partners. As a vital part of BJS’s opera- tions, we urge continued strengthening of the relationship between BJS and its SAC partners: Recommendation 4.1: Through its Statistical Analysis Center and State Justice Statistics programs, BJS should continue to de- velop its ties with the states, and more fully exploit the potential for using states as partners in data collections. We further suggest that the experience of BJS’s SAC partners be tapped as BJS pursues methodological improvements, including developments toward longitudinal and small-area measurement systems (Recommendation 4.2). BUILDING STATISTICAL SYSTEMS AND CONCEPTUAL FRAMEWORKS Strategic Goal 2: To build, maintain, and utilize statistical sys- tems that describe the extent and characteristics of crime in our nation and the status and response of the justice system. More than a basic statement of topic, the language of this goal suggests attention to “statistical systems” in two basic respects: first, ensuring that BJS’s data collection portfolio has a greater sense of coherence and inter- connectedness than it does at present and, second, making use of comput- erized record systems—including those supported by BJS’s grants for system development—to develop new sources of data. Conceptual Frameworks Based on our review of past and current BJS data collections, several of our recommendations concern the refinement of conceptual frameworks for what BJS does, with respect not only to the whole structure of BJS’s portfolio, but also to the design of BJS collections in major topic areas. 

12 JUSTICE STATISTICS With respect to the whole portfolio, we recommend that BJS’s strategic plan articulate a blueprint of interrelated data collection and product activ- ities that could then be used to evaluate new opportunities (Recommenda- tion 3.7). The BJS-developed mapping of data series to the criminal justice system funnel model is an important first step in this regard; our concern is that the unique characteristics, shared areas of substantive overlap, and interrelationships of BJS’s current set of collections are not always immedi- ately clear. We find this lack of an overall framework to be most striking in BJS’s law enforcement data collections, where the agency’s large number of special-agency censuses has an unmistakable scattershot appearance. In particular, we recommend that BJS develop an integrated conceptual plan for the periodicity of these law enforcement agency censuses and surveys, abandoning those with no meaningful prospect of being repeated (Recom- mendation 3.10). New one-time collections should have a clearly defined role within the broader portfolio. Within BJS’s major survey vehicles, we urge the implementation of a core-supplement framework, where the base content of the survey is re- duced to a simplified common “core” of recurring content and coupled with recurring, structured topic supplements. We recommend such a structure for the NCVS in Recommendation 3.8 and for the Law Enforcement Man- agement and Administrative Survey (LEMAS) in Recommendation 3.9. We think that such a structure is an important part of overcoming limitations in BJS’s current holdings in the area of law enforcement generally, which are currently principally limited to the organizational and administrative fo- cus of LEMAS. We suggest the use of LEMAS topic supplements to expand the substantive scope of that survey and the analysis of LEMAS-type data in combination with crime data (thus coming closer to informing assessment of police policy effectiveness). We further recommend (Recommendation 3.11) that the NCVS and its supplements be used more effectively as a tool for studying law enforcement, both in terms of the types of crime that are reported (and not reported) to police and the action that results from the reporting of a crime (such as has been done by the current Police-Public Contact Survey supplement). As noted above, the panel’s general impression of BJS’s data collections in the area of adjudication is one of the agency doing the best it can with what it has—making good use of data collection partners but facing the ba- sic problem of wide variation in the operational structure of state courts, in their levels of automation, and their use of computerized case management systems. Thus, it is more a statement on the condition of court information systems than on BJS’s efforts that we conclude that BJS’s adjudication se- ries lack an effective basis in sampling. We recommend (Recommendation 3.12) that BJS work to implement more rigorous methods of probability sampling in its adjudication series as court records become more accessible 

SUMMARY 13 through computerized case management systems; its recent steps to redesign the State Court Processing Statistics collection exemplify developments in this regard. We further recommend that BJS develop a research program to build representative samples of courts and to assess strategies for collec- tion of case records for even such a small, but representative, sample (Rec- ommendation 3.13). One major omission in BJS’s current work in court statistics is information on declinations to prosecute and the use of alterna- tive dispute resolution techniques; collecting these data in full detail would be a major enterprise, but we suggest that one low-cost way to get at least some information on this missing piece would be to add some basic ques- tions on aggregate case processing to the National Prosecutors Survey, which currently has a purely administrative focus. Making Use of Record Systems Interpreting “statistical systems” more technically, this strategic goal statement also includes two recommendations regarding the use of comput- erized record systems. First, it is part of BJS’s explicit legal mandate to provide both technical and financial assistance to state and local authorities for purposes of devel- oping information resources. Much of this work has been done under BJS’s National Criminal History Improvement Program (NCHIP), under which BJS provides grants to local authorities to help their development of crim- inal history record databases. These local databases are used, in turn, to populate national-level record databases maintained by the Federal Bureau of Investigation (FBI) that are used, among other things, to conduct back- ground checks for firearm purchases. From the purely statistical standpoint, BJS’s NCHIP work is ripe for criticism because BJS’s role is strictly a money- transfer operation; although it pays to develop the local databases, BJS has no access to the resulting data for research purposes. Other than generating rough summary statistics of firearm-purchase background checks, BJS has not been able to utilize the criminal history record data that its grant monies help to develop at the state and local levels. This failure has occurred despite a formal set-aside of funds in all of those grants for BJS evaluation purposes. Very recently, however, the FBI has begun to permit BJS the same access to criminal history records that is available to law enforcement agencies. Access to compiled history record data for research purposes opens very exciting avenues for study, which BJS intends to first explore in studies of prisoner reentry to the community and recidivism. We encourage BJS to pursue this work and expand it to include broader studies of criminal behavior: Recommendation 4.3: BJS should actively utilize the NCHIP program to improve criminal history records necessary for lon- gitudinal studies of crime. 

14 JUSTICE STATISTICS Second, we suggest that BJS study the feasibility of a records-based data collection that would be capable of providing a timely indicator of major crime trends. In the panel’s assessment, the panel is well served by hav- ing multiple indicators of crime and justice in the United States; the FBI’s Uniform Crime Reporting (UCR) program has the weight of being an of- ficial measure of crimes reported to police while BJS’s NCVS has unique strengths in describing the etiology and contexts of crime and violence (in- cluding those incidents not reported to police). However, both series suffer in public perception from lengthy lag times between data collection and re- lease of results. It has been argued that the UCR program would be better ad- ministered if it were transferred from the FBI to BJS; we do not recommend such a transfer at this time, arguing that the move would pick unnecessary turf fights and would incur a much more intensive—and costly—redesign than the simple organizational switch suggests at first glance. However, we suggest that BJS study the feasibility of an entirely different approach that is better in some respects, in order to determine whether crime incident data collected by local police departments on their own (and, in some cases, published on their websites) can be compiled on a probability sample basis. This new collection is not intended to be duplicative of the UCR, because it puts the burden of structuring the sample and compiling the data on BJS rather than requiring local departments to compile and record the counts in the UCR format; rather, it is simply intended to gain leverage from the availability of existing local data. Recommendation 4.4: To improve the timeliness of crime statis- tics, BJS should explore the development of a crime reporting system based on a probability sample of police administrative records. The goals of such a system would be national repre- sentativeness, high response, high data quality, timeliness and flexibility in terms of crime classification and analysis, and na- tional statistics for the monitoring of crime trends. IMPROVING COVERAGE OF THE JUSTICE SYSTEM Strategic Goal 3: To provide comprehensive statistical coverage of all parts of the criminal justice system, including the longi- tudinal flow of persons and cases through the system and their return to the community. Through this goal statement, we return to our analysis of the funnel model for the criminal justice system sequence of events and our conclu- sions from that analysis. Accepting the funnel model as a premise, this goal might be restated as saying that data collections should rise and fall in im- portance relative to their proximity to the funnel: The further a specific 

SUMMARY 15 phenomenon is from the activities described by the funnel model, the less central it is to the mission of BJS. By its nature, the funnel model has strong implications for priority setting. First, it suggests that BJS should focus on activities that affect the most people and affect them most extensively. By definition, this means the incidence of crime and victimization, which the funnel graphically describes as the most prevalent phenomenon in the crim- inal justice system. Following this logic, and all other things being equal, describing crime, victimization, and the immediate consequences should be the principal focus of BJS. Second, however, the nature of a funnel also di- rects attention to the late stages of the process. As offenders make their way through the funnel, their numbers decrease but the consequences of justice system decisions increase, in terms of impacts on lives and public budgets. At the far end of the criminal justice system, in the correctional area, the effects of criminal justice decisions are so extensive that obtaining adequate data on these populations has importance well beyond their numbers. This principle for establishing the importance of any given activity need not correspond to the allocation of resources on that activity because some data can be col- lected more cheaply than others. Nonetheless, it should guide decisions as to where to put the intellectual energy of the agency if not in a one-to-one correspondence with its fiscal resources. A logical decision process may be to separately value activities by their proximity to the funnel, and then assess their relative cost; the final portfolio is a function of both importance and cost. The word “longitudinal” in this goal statement is particularly critical. As the mapping of BJS data series to the funnel makes clear, BJS generally has good coverage of all the steps in the criminal justice process but, almost exclusively, this coverage is cross-sectional in nature. BJS’s steps in true lon- gitudinal measurement of persons and cases through multiple steps of the criminal justice system have been rare; in large part, this is due to the struc- tural problem that there is no common identifier attached to a person (or case) that follows the person (or case) through all major steps (police, court processing, entry into corrections, and reentry into the community) to fa- cilitate easy longitudinal measurement. We see a longitudinal approach as essential to study of the performance of the justice system as a whole, and we recommend a variety of strategies for improving longitudinal designs. Some of these are low-cost techniques that simply build on longitudinal structures within existing data resources such as building methods for record linkage into existing and emerging correctional data collections; one exam- ple is linking data from recidivism studies to the Census of Adult Correc- tional Facilities to understand how correctional facilities and programming affect recidivism (Recommendation 3.3). We also recommend that BJS work to emphasize transitions and flows in prison and jail populations, whereas BJS’s current publications tend to emphasize current “stocks” or fixed-time 

16 JUSTICE STATISTICS counts of inmates. We suggest that BJS’s regular correctional data products include both stock (Recommendation 3.1) and flow (Recommendation 3.2) information capable of disaggregation and cross-tabulation by state, offense category, and demographic group. We recognize that other suggestions are more costly, but would ultimately be beneficial in enhancing understanding of the justice system: • Conduct research on the “common identifier” problem discussed above, determining the feasibility of measuring individuals’ experi- ences in the justice system on a prospective, longitudinal basis, be- ginning as early as practicable in the process (arrest) and ending with their eventual exit (ranging from early dismissal of charge through completion of sentence) (Recommendation 3.4); • Develop an approach to measure the victimization of experiences of individuals (as contrasted with the NCVS focus on households) on a longitudinal basis, beginning from a focal victimization and following the victim forward in time, measuring subsequent victimizations (Rec- ommendation 3.5); and • Develop a panel survey of people under correctional supervision to un- derstand how individuals move between institutional and community settings, and to understand the social contexts of correctional supervi- sion (Recommendation 3.6). To extend this last point, BJS and the Justice Department should look for opportunities to leverage studies called for by the Second Chance Act of 2007 (enacted in April 2008) to build an active program on recidivism and reentry. Specifically, we recommend that BJS mount a feasibility study of the flow of individuals between correctional supervision and community settings. Repeated interviews of samples of about-to-be-released prisoners that track their successes and failures in reintegrating with the community would enhance understanding of this critical policy issue (Recommendation 3.14). We also suggest that BJS explore ways to reactivate its studies of probationers and parolees. Under this general heading of justice system coverage, we further recom- mend that BJS continue to develop and study the measurement of emerging or hard-to-reach groups and develop more appropriate approaches to sam- pling and measurement of these populations (Recommendation 5.14). In addition, BJS (in conjunction with OJJDP) should develop juvenile victim- ization, crime, and justice statistical series suitable for describing the patterns of offending and victimization of youth, as well as studies of the longitudi- nal progression of youth through the juvenile and criminal justice systems (Recommendation 2.2). 

SUMMARY 17 FACILITATING ACCESS AND IMPROVING DISSEMINATION AND OUTREACH Strategic Goal 4: To ensure access to statistics and data on crime and justice by the American public, the U.S. Congress, the U.S. Department of Justice and other executive agencies, and state and local government agencies. Outreach and dissemination are areas that already play considerable roles in BJS operations and in which BJS has made laudable strides. We have al- ready noted and endorsed BJS’s formation of a strong communications and contact network with the states through its SACs. BJS also deserves credit for the extensive backfile of reports and summary tabulations that are avail- able from the agency’s website and the dissemination of its reports through the OJP-administered National Criminal Justice Reference Service. High- end users are well served by BJS’s microdata holdings that are available for download through the National Archive of Criminal Justice Data. Through this goal statement, we encourage still further developments along these lines as BJS works to increase its public profile and its relevance in policy debates, while maintaining the high quality standards it has set for itself. In the panel’s assessment, there is value in BJS pursuing both formal and informal mechanisms for obtaining user input and feedback and for shaping possible new data collections. The law that created BJS in 1979 also created a formal advisory board for the agency, a provision that was later omitted in reauthorization language in 1984. We recommend that such an ongoing ad- visory group would be a useful vehicle for obtaining recommendations from diverse perspectives; specifically, we recommend that BJS establish an Advi- sory Group under the Federal Advisory Committee Act, the membership of which should include—at a minimum—leaders and practitioners from each of the major subject matters covered by BJS data, as well as those with sta- tistical and other types of academic expertise (Recommendation 5.8). In the past, BJS has also made good use of ad hoc user and stakeholder workshops on targeted issues; the agency convened a very useful data user conference in February 2008 in partial support of this panel’s activities. We recommend that BJS continue to hold such ad hoc stakeholder workshops to suggest ar- eas of immediate needs or to get input on contemporaneous topics of interest (Recommendation 5.7). The data extracts directly available from the BJS website tend to be se- lected tabulations and spreadsheets. As BJS works to enhance its website presence, it should explore methods for making more of its data available for direct tabulation, analysis, and—perhaps—mapping on the website. We also recommend that BJS take care in managing how some of its data col- lections (gathered by external data collection agents) are housed on non-BJS websites; BJS should articulate and describe the process by which links to 

18 JUSTICE STATISTICS external sites are allowed and used, and should also articulate and justify the use of its insignia—which carries with it important quality connotations—on external websites (Recommendation 5.12). From the numerous references to BJS data and findings in legislative text and debate records, it is clear that BJS has a key and receptive audience in the U.S. Congress and its staff. However, from the types of legislative demands that are sometimes put on BJS (e.g., the PREA reporting require- ments) and past difficulties BJS has experienced in securing funding, it is not quite as clear that the dialogue between BJS and Congress is effective. (At present, of course, a major reason for this problem in communication is that liaison with Congress is done principally by and through OJP; for instance, BJS directors do not directly testify to appropriations committees on agency needs.) Accordingly, we recommend that—regardless of the organizational structure of BJS and OJP—BJS should take a stronger role in cultivating a relationship with Congress: Recommendation 5.9: DOJ should take steps to ensure that con- gressional staff are aware of BJS data that could be used in devel- oping legislation; DOJ and BJS should learn from congressional staff how their data are needed to inform/support legislation so that they can improve the utility of their current data and so that they can develop new datasets that could enhance policy development. From the technical standpoint, an important means for BJS to increase its public relevance is to find ways to address long-standing concerns about the agency’s data products: though they are generally held in high regard, they are frequently seen as lacking timeliness and, particularly for the NCVS, subnational geographic detail. One approach that we recommend to deal with this concern is that BJS evaluate each of its data programs to ascertain whether more timely estimates might be obtained by (a) making discrete data collections into more continuous operations and (b) issuing preliminary estimates, to be followed by final estimates (Recommendation 5.11). More fundamentally, and consistent with the expected principles and practices of a federal statistical agency, we believe that the cultivation of an active and continuous research program is essential to an agency’s progress. For BJS, such research should include work on statistical modeling to im- prove the temporal and spatial resolution of estimates (a program for which BJS could partner with other federal statistical agencies, most of which face similar pressures from their user constituencies). A vigorous research pro- gram could also suggest ways to better and more efficiently make use of state and local database resources, such as through steps to increase the automa- tion and consistency of BJS’s National Corrections Reporting Program data (Recommendation 5.10). The presence of a solid research program would 

SUMMARY 19 also allow BJS to do something that sounds obvious—carefully study changes in the NCVS design before implementing them (Recommendation 5.13)— but that is difficult to accomplish in a climate of constrained resources. To carry out a research program and continually improve the agency’s data col- lection holdings, we recommend that BJS improve the technical skill mix of its staff, including mathematical statisticians, computer scientists, survey methodologists, and criminologists (Recommendation 5.15). PRIORITY SETTING AND CONSTRAINED RESOURCES A possible criticism of this report is that it tends to suggest adding to BJS’s inventory of collections rather than subtracting. The recommenda- tions are generally geared to improvements within BJS’s various existing data collections—for instance, ensuring a high-quality independent measure of crime in the NCVS, emphasizing conceptual frameworks in BJS’s adjudi- cation and law enforcement collections, and expanding its corrections series to study prisoner reentry into society. As a panel, we grappled with ques- tions regarding this: How should these improvements be paid for? Should some existing series be cut to make ways for new ones? And, given the disproportionate share of BJS’s current resources consumed by the NCVS, should some smaller collections be stopped to free up additional resources for the NCVS or should some NCVS resources be steered to other areas? The reason why we rejected explicitly suggesting that some data series be cut in order to pay for others is certainly not that the current collections are perfect. The improvements we suggest in the recommendations are tes- timony to that. Rather, we believe that BJS cannot achieve its legislated goals by cutting programs. In our assessment, we think it can be stated as a fact: BJS has been given more responsibilities than can be achieved with current resources. The resources provided to BJS to carry out its work are not commensurate with the breadth—and importance—of the responsibili- ties assigned to the agency by its authorizing legislation. Because of this, the agency has for some years walked a tight line of small cuts of sample or measurement, short delays of publications, and tem- porary hiring freezes—each of these tolerable in itself, but cumulating over the years such that core functions have broken down. On a routine ba- sis, BJS must make decisions about addressing certain responsibilities and not others, trading off periodicity and completeness of data collections, and balancing continuity of existing data collections against the need to comply with directives from Congress and DOJ. Such decisions are hardly unique to BJS—at some point, all organizations must make such trade-offs—but BJS’s mismatch between resources and responsibility makes the decisions particu- larly difficult. 

20 JUSTICE STATISTICS Thus, in setting priorities, BJS directors have perforce had a short time horizon—responding to a certain set of demands even though those deci- sions may have negative long-term consequences for individual data collec- tions and the health of the agency. Certainly, in the midst of year-to-year juggling of data series in order to keep production moving, longer-term in- vestments in research and innovation become difficult or impossible to make. The most striking example of the consequences of this extremely tough cli- mate is the current state of the NCVS: what was once, clearly, the best vic- timization survey in the world is now unable to satisfy its basic function of providing annual estimates of level and change in common-law crime. This decay happened gradually as BJS administrators were attempting to respond to immediate exigencies, aggravated by an overly broad mandate. Each sin- gle cut in sample size, or other cost-cutting measure, was justifiable given then-present alternatives. Cumulatively—as demonstrated most vividly by the declared “break in series” with the 2006 NCVS data—they lead to our conclusion in our interim report that “the [current] NCVS is not achieving and cannot achieve BJS’s legislatively mandated goals” (National Research Council, 2008b:Finding 3.1). Statistical infrastructure (data collections and record series) shares with physical infrastructure (such as roads, bridges, sewers, and cable) the funda- mental problem that it lacks glamour and can be difficult to champion as a government spending priority. Yet it is undeniably important to the public knowledge and the public good; BJS’s purview includes topics important to general welfare, spanning the measurement of interpersonal violence, the function and magnitude of law enforcement and corrections, and the oper- ations of the judicial branch of government. The problem is not that there are parts of BJS’s legal mandate that are unimportant or unworthy, but that its current resources do not permit it to cover its mandates as effectively as possible. Given this finding, we are loath to construct a list of series to terminate or reduce out of concern that the agency fail even more visibly to fulfill the charge given by its legislation. Another reason for not considering a specific ranking of data collections in order to suggest possible cuts is consistency with the approach we took in our interim report on options for the NCVS. In that report, we presented an array of possible options and described how specific design choices cor- responded to particular goals for the survey. However, we made a point of “not suggest[ing] one single path as the ideal for a redesigned NCVS” (Na- tional Research Council, 2008b:4). Different people and decision makers do not necessarily put the exact same weights on the goals and objectives of a program such as the NCVS, and we did not want to presume “that our pre- ferred set of NCVS goals is correct to the exclusion of all others” (National Research Council, 2008b:4). The same logic applies to considering other collections in the BJS portfolio: deeming one collection more worthy than 

SUMMARY 21 another involves complicated value judgments, not science. Specific con- stituencies for BJS data that are not represented on the panel or by groups who have spoken before the panel could make eloquent and compelling cases for their particular favored set of statistics; again, we do not wish to suggest that any weighting we could suggest is somehow paramount. Although hoping that tight fiscal constraints may be alleviated somewhat in coming years, we cannot assume infinite resources either; we have not interpreted our task as assembling a “wish list” for everything that a justice statistics agency could do, but rather think that out suggestions are a mix of short- and long-term, low- and higher-cost ideas for improving the sta- tistical evidence with which crime and justice policy in the United States is developed. Though we do not explicitly rank BJS’s data collections, our suggested strategic goals provide BJS with a set of principles against which its data collection portfolio can be assessed. BJS, which marks its thirtieth anniversary in its present form this year, can rightly look back on a solid body of accomplishment. But our recommendations in this report suggest that there are many directions for continued improvement; much work re- mains to be done to ensure the quality, credibility, and relevance of statistics on justice in the United States—to achieve the BJS the country deserves.