Below are the first 10 and last 10 pages of uncorrected machine-read text (when available) of this chapter, followed by the top 30 algorithmically extracted key phrases from the chapter as a whole.
Intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text on the opening pages of each chapter. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.
Do not use for reproduction, copying, pasting, or reading; exclusively for search engines.
OCR for page 211
Strengthening the National Institute of Justice 7 Recommendations The National Institute of Justice (NIJ) is the only federal research agency devoted to the control of crime and the administration of justice, which are a central component of national identity. Since its establishment as part of the emerging federal role in assisting state and local officials in these important issues, NIJ has been the central research arm of the U.S. Department of Justice (DOJ). The charge to this committee, detailed in Chapter 1, was to assess the performance of this agency and to identify changes in the agency that would allow it to better achieve its mission. Our review has demonstrated that NIJ is central to the better understanding of how to control crime and its impact on citizens and communities and how to ensure that the system of justice is fair and effective. No other federal, state, local, or private organization can do what NIJ was created to do. During the past 40 years, no other agency has contributed more to the understanding of crime and justice efforts than NIJ. This report has documented many of these accomplishments, including in the areas of crime mapping and hot spot policing, violence against women, the role of firearms and drugs in crime, drug courts, and forensic DNA analysis, as well as the success of the data archives, fellowships, and other outreach programs. Despite the many shortcomings of NIJ that are demonstrated in this report, the nation needs an NIJ—but one that functions better than the current one. We are convinced that a “new” NIJ, defined by the recommendations that follow, will better serve the nation and instill increased confidence in its justice system operations. We encourage Congress and the executive branch to provide the funding that the tasks assigned to NIJ require.
OCR for page 212
Strengthening the National Institute of Justice As this report demonstrates, few major initiatives in criminal justice during the past 40 years have occurred without significant involvement of research developed and sponsored by NIJ. These contributions are in part noteworthy because they have occurred in an agency that, as documented in this report, has lacked the level of independence, appropriate leadership, funding, and operational practices that characterize much more successful federal research agencies. The appropriate level of independence is important to ensure that a research agency has the authority to set agendas through planned deliberations with the research and practitioner communities. Otherwise, its mission, decisions, and results can be influenced by nonscientific goals and objectives, and it loses credibility with these communities. Although NIJ has had periods of good leadership, it has never had leadership with experience and prestige in the research community. This not only reinforces the problems associated with limited independence, but it also results in leadership that cannot exercise independent scientific judgment and credibly involve the research community in its work. We have demonstrated that, given its mission, the funding of NIJ has been seriously inadequate, far too variable, and too subject to earmarks that remove funding decisions from peer review and scientific standards. We also have demonstrated that basic operations at NIJ—that is, agenda setting, peer review, report review and assessment—have not been consistent with those at other successful federal research agencies. These aspects of NIJ are unlikely to be greatly improved without attention to the issues of independence and resources. The committee gave considerable attention to the issue of how best to achieve the appropriate level of independence for NIJ. Two basic approaches were considered: (1) moving NIJ out of the Office of Justice Programs (OJP) and (2) retaining NIJ in OJP and giving it a level of independence similar to other federal research agencies. In considering these options, the committee reviewed other federal research agencies, consulted with former directors of NIJ and OJP, and raised the issue with many others who offered guidance to the committee.1 We also included a question about NIJ’s independence on our survey of researchers and practitioners. In addition, the committee considered the recommendations of two other NRC committees that issued reports on related topics during our deliberations—the report on the needs of the forensic science community (National Research Council, 2009c) and the report on BJS (National Research Council, 2009a). After careful consideration of all the evidence the committee concluded that maintaining NIJ in OJP but with substantially increased levels of independence secured by Congress offered the greatest likelihood of NIJ achieving its mission. The 1 Most instructive were the views of Jeremy Travis and James K. Stewart, former directors of NIJ (Travis, 2008; Stewart, 2009).
OCR for page 213
Strengthening the National Institute of Justice committee considered whether the independence NIJ needs to be an effective science agency can be achieved in other ways (e.g., an attorney general order or an OJP policy) and concluded that nonlegislative options could not be as effective. Only Congress can provide the requisites of increased independence and the necessary oversight to ensure that specific authorities cannot be easily retracted or eroded. Strengthening Forensic Science in the United States: A Path Forward recommended that a new federal agency should be developed outside DOJ to address all the needs of the forensic science community. The primary rationale for this approach was the belief that placing such activities in a law enforcement agency could result in science being actually biased or perceived to be biased towards law enforcement agencies.2 In Ensuring the Quality, Credibility, and Relevance of U.S. Justice Statistics, the recommendation was to move the Bureau of Justice Statistics (BJS) out of OJP into a new agency in DOJ that would report directly to the Attorney General. This recommendation was based on the conclusion that political influences from OJP had and could in the future bias BJS findings and reports. The committee concluded that the concerns that prompted these recommendations were not major reasons for the problems we have described at NIJ. We also concluded that forensic science research fits within NIJ’s research mission. Furthermore, the committee received information that convinced us that having NIJ report directly to the attorney general was not likely to reduce potential political influence; that it is unlikely that a new research and statistical agency in DOJ would be established given the costs of doing so; that moving NIJ out of OJP would make linking research and action programs less likely; and that such a move would make it less likely for Congress and the administration to move funds from action agencies to NIJ as has happened in the past and is specified in the OJP current appropriations bill. The committee concluded that keeping NIJ in OJP but with increased independence and greater involvement of the research and practitioner communities has a better chance to result in an agency that can gain the trust and confidence of Congress, the administration, and the criminal justice community. If the changes we recommend are not implemented within 5 years or if they are and the problems we have identified persist then we recommend that the idea of moving this research function should be carefully revisited.3 Furthermore, the committee concluded there are a number of substan- 2 As described in Chapter 3, should Congress adopt the recent recommendation of the National Research Council (NRC) (2009c) and establish a new independent federal entity responsible for forensic science activities, NIJ’s future role in sponsoring forensic science research and development activities would need to be defined accordingly. 3 Currently, Congress is considering establishing a national crime commission. If it is formed, then it would be the natural body to conduct this review.
OCR for page 214
Strengthening the National Institute of Justice tive reasons for retaining NIJ within OJP. Foremost is that this relationship allows NIJ to be organizationally closest to those for whom its research is eventually intended. This does not mean that specific research projects will have immediately useful findings. However, unlike other federal research agencies, such as the National Institutes of Health (NIH) institutes and the National Science Foundation (NSF), NIJ’s mission includes a greater focus on development and evaluation activities designed to bridge research and practice. Being in OJP provides an opportunity for NIJ to advise and participate in the development of OJP-sponsored programs and to be in a position to evaluate those programs. Because criminal justice policy makers and practitioners are much more closely tied to OJP than NIJ, NIJ can use its OJP relationship as a bridge to the field. NIJ can then use these connections to better target its research to the needs of the field, to involve practitioners and their agencies more directly in research activities, and to increase the likelihood that its research will eventually have an impact on criminal justice practices and policies. Because of the importance of the overall OJP program to the field and to DOJ, as well as the unprecedented levels of federal funding that have been made available in recent years, the OJP Assistant Attorney General is in a position to assist NIJ in making a compelling case for a well-designed and funded research program with the Office of Management and Budget (OMB), DOJ, and Congress. The need for research is particularly apparent given the expanded scope of OJP and DOJ program efforts, their partnerships with state and local criminal justice agencies and the expectations of Congress for well-designed, evidence-based programming. Of course, leaving NIJ in OJP means that the relationships between these agencies need to be clearly specified in the authorizing legislation for each. NIJ should be able to set its own research agenda, determine the awards it makes, set monitoring and oversight procedures for its grantees, control publication and dissemination decisions, hire and supervise its allotted personnel, present its budget to the Attorney General and Congress, and interact with appropriate DOJ and congressional entities. When OJP provides some limited centralized services, it should be done in full cooperation with NIJ and without the level of interference in research decisions and operations that have occurred too often during the history of NIJ. To address these issues the committee makes five recommendations: (1) to ensure NIJ’s independence and improve governance, (2) to strengthen its research mission, (3) to bolster the crime and justice research infrastructure, (4) to enhance the scientific integrity and transparency of its operations, and (5) to establish a culture of self-assessment. Our analysis strongly suggests that if these recommendations are implemented, NIJ can be a great leader in national efforts to reduce crime and improve the administration of justice.
OCR for page 215
Strengthening the National Institute of Justice ENSURE INDEPENDENCE AND IMPROVE GOVERNANCE RECOMMENDATION 1: The committee recommends that Congress provide for the requisite independence and authority of the National Institute of Justice (NIJ) while retaining its organizational placement within the Office of Justice Programs and the U.S. Department of Justice. Among the key issues to be considered in pursuit of this goal are a statutory advisory board, a set term of office and minimum qualifications for the NIJ director, and clear authority for NIJ to make awards and control its budget and resources. NIJ’s efforts to fulfill its research mission to build a research infrastructure and produce a body of knowledge that can contribute to a greater understanding of crime and society’s response have met with mixed success. Much of this is due to the deficiencies that have continued to persist since the last review. To become a more effective agency, NIJ needs control over its research and development (R&D) management operations. Like the prior review by the the NRC (1977), we recognize the need for independence, improved governance, and authority to conduct its work. Need for Independence The issue of independence for research and statistical agencies has been hotly debated for years. The 1931 Wickersham Commission, the 1967 President’s Crime Commission on Law Enforcement and Administration of Justice, and participants at the 30th Anniversary Symposium of the President’s Crime Commission held in 1997 all grappled with it. The 1977 NRC report pointed out that the location of NIJ in the Law Enforcement Assistance Administration had not served it well. Its lack of independence and political constraints prevented the institute from fulfilling its mission of building knowledge about crime. The report warned that unless its present form was changed to an independent research agency within DOJ, with final approval authority over all awards as well as control over its administrative budget, personnel, and detailed program review, it was unlikely to become a “significant and quality-oriented research agency” (National Research Council, 1977, p. 3). In many respects, this prophecy could be made today exactly as it was more than 30 years ago. As this report shows, the history of NIJ reflects diminishing authority and resources, not only as a result of congressional action, but also from actions taken by its oversight agency, OJP. From our review of other federal research agencies, we are well aware that independence is an essential attribute shared by all of them. Conducting rigorous and objective research requires freedom from control by policy
OCR for page 216
Strengthening the National Institute of Justice makers and others who have a stake in the research findings and their implications for policy, programs, and practice. Independence does not necessarily imply that all federal research organizations need to be free-standing, like NSF, whose single purpose is “to promote the progress of science.” What it does imply is that clear lines of separation need to be made from the policy-making organization, with a balance maintained between independence and responsiveness to problems a parent agency is addressing. This appears to be best accomplished in mission-oriented agencies when program services (funding of direct or indirect services) are separated but related to the research unit. In addition, in these situations, the research agency should report to the parent agency’s top leadership to ensure that it receives the best scientific advice on issues of concern. An effective research organization needs to have the independence to conduct its work. The key components of independence include authority to (1) establish research and evaluation priorities; (2) control the grant-making process, from solicitation through grant approval; (3) make budgetary recommendations to the highest level of the parent agency; (4) make staffing decisions, including recruitment and hiring; and (5) issue reports and other dissemination activities. An important aspect of these powers is the ability to avoid earmarks in the budgeting process. In order to achieve the level of independence necessary for a trusted crime and justice research agency, the new NIJ that we propose will need a strong advisory board, terms of service for a qualified director, and grant-making, budgetary, and staffing authority. Governance Advisory Board While advisory boards have been used in different ways over the years at NIJ, the committee concludes that they have never functioned as a scientific advisory board should—to set agendas, review the integrity of the research operations, and assess the accomplishments of the agency with the standards of science as the guiding principles. As described in Chapter 2, these groups were a disappointment to their respective NIJ directors. Neither was able to build support for NIJ by gaining the attention of Congress, nor was it able to serve as a buffer against what was regarded as a highly politicized environment in which everyone had a different take on crime and what was needed to combat it. There was never any sustained attempt to involve the members in a policy-making or priority-setting role. Although scientific advisory boards are used in different ways by different federal research organizations, they appear to be central to the effective operation of these agencies. Advisory boards provide oversight, assist in
OCR for page 217
Strengthening the National Institute of Justice setting research agendas, may have approval authority for certain awards, and conduct reviews of agency operations. Federal research agencies may have one or more boards; if they have more than one, a distinction is made between ongoing oversight boards and those created periodically to assess performance. In all instances that the committee reviewed, scientific advisory boards are overwhelmingly comprised of research scientists but also may include a small number (perhaps 25 percent of membership) of those practicing in the field. Boards are usually nominated by the research agency and approved by the head of the agency that houses the research agency or a designee. They report only to the director of the research agency or if they are advisors to a subunit, to the head of that research component. The experience with the NIJ advisory board differs markedly from that of other research agencies, including many with an operational mission. For example, the Environmental Protection Agency (EPA) has an overall Science Advisory Board to guide its various science programs as well as individual program boards, such as the Clean Air Scientific Advisory Committee. These science boards are distinct from policy boards and each has a separate role to play in guiding the agency’s operational and technology programs. The U.S. Department of Education (DoEd) has a 15-member research advisory board that works with the assistant secretary for educational research and improvement to develop the agency’s long-term research and development agenda. The board’s membership includes some policy officials, especially local school officials, but most of its members are scientists. The two most prominent research agencies, NSF and NIH, have advisory groups that play a major role advising on policies and a wide range of research activities, approving priorities, soliciting advice from the field, suggesting how the research can be strengthened, and assessing the work of their respective organizations. The members of both these boards are appointed by the president, have set terms that are staggered, and are selected for their preeminence in research, a particular field of scientific endeavor, or public service. Among the 27 institutes and centers that constitute NIH, 20 of them have some kind of advisory council, committee of counselors, or panels, and several have more than one kind of advisory group.4 The most common form of advisory group is the advisory council composed of 12-18 members, who are appointed by the secretary of health and human services in consultation with the institute director and serve 4-year terms. The National Advisory Council on Drug Abuse has 18 members, of whom 4 For example, the National Cancer Institute has five advisory groups: (1) the President’s Cancer Panel, (2) the National Cancer Advisory Board, (3) the Director’s Consumer Liaison Group, (4) the Board of Scientific Advisors, and (5) the Board of Scientific Counselors (see http://getinvolved.nih.gov/advisory_councils.asp [accessed March 17, 2010]).
OCR for page 218
Strengthening the National Institute of Justice 12 are experts in scientific fields and 6 are knowledgeable members of the public.5 The National Institute on Alcohol Abuse and Alcoholism (NIAAA) has 15 members, of whom 10 represent health and scientific fields (including not less than 2 individuals who are leaders in the fields of public health and the behavioral or social sciences) relevant to the activities of NIAAA.6 The remaining one-third are selected from the public and include leaders in the fields of public policy, law, health policy, economics, and management. These councils approve scientific review group or peer-review recommendations. They also provide advice on research opportunities, guidance on ways to improve approaches to stimulating research, and ideas for working more closely with constituent groups and consumers of the research. Although these councils do not make policy, there is a tradition of their advice being heeded by the institute administrators. Similarly, in addition to the presidentially appointed National Science Board, which provides overall guidance and oversight of the National Science Foundation, each of its seven directorates also has its own advisory committee. The Directorate of Social, Behavioral & Economic Sciences (SBE) is the principal source of federal support for fundamental research on human cognition and behavior and social structures and social interaction, as well as research on the intellectual and social contexts that govern the development and use of science and technology. It has an 18-member advisory committee that meets semiannually to review and advise on the impact of programs in the disciplines and fields encompassed by the directorate, provides oversight of overall program management and performance, and advises on the impact of NSF-wide policies on the SBE scientific community. A typical agenda includes briefings on recent scientific discoveries and activities, review of human capital and succession planning issues, a discussion with representatives of professional research member organizations, and review of program assessment reports produced by committees of visitors (described in Chapter 6).7 Our committee discussed whether the Office of Research and Evaluation (ORE) and the Office of Science and Technology (OST) should also have advisory boards and concluded that NIJ is currently too small to warrant such an elaborate advisory board structure. However, subcommittees of the proposed NIJ advisory board could be designated for more specific oversight of ORE and OST, for specific grants or contracts, or for both. In proposing an advisory board for NIJ, the committee considered 5 Available: http://www.nida.nih.gov/about/organization/NACDA/NACDAHome.html [accessed March 17, 2010]. 6 Available: http://www.niaaa.nih.gov/AboutNIAAA/AdvisoryCouncil/default.htm [accessed March 17, 2010]. 7 Available: http://www.nsf.gov/sbe/advisory.jsp [accessed March 17, 2010].
OCR for page 219
Strengthening the National Institute of Justice the number of NIJ constituencies and decided 15 members was a reasonable number to cover all functions and perspectives. Given the scope of the agency and the responsibilities we have outlined for the advisory committee, it would be difficult to have a smaller membership. In determining who should be the appointing authority for the advisory board, the committee carefully considered three alternatives: appointment by the president, appointment by the attorney general, and appointment by the NIJ director. NIJ’s size (organization as well as resources) seemed to preclude it as being suitable for a presidential appointed advisory board. The committee thought that giving the NIJ director authority to appoint might provide too much power and give the perception that the members were in some way beholden to the director. The prior experience of having the attorney general as the appointing authority seemed to work well. Previous advisory boards appointed by the attorney general contained a mix of highly regarded and nationally recognized researchers, practitioners, and policy makers who were familiar with NIJ and supportive of its mission. Using the experiences of these other federal agencies as a guide, the committee suggests the following structure for the NIJ advisory board: Membership 15 members appointed by the attorney general for 5-year staggered terms. A slate of nominees will be recommended to the attorney general by the NIJ director following consultation with institutional groups representing important research and practitioner communities. Interested parties may also recommend nominees either via the NIJ director or to the attorney general directly. Ten board members will be experienced researchers; five members will be from the practitioner and policy communities. Their interests and expertise should be broadly representative of criminology and criminal justice and of issues of importance to both NIJ’s social science and technology research programs. Additional persons from other federal research agencies as well as NIJ leadership may be invited by the advisory board to serve as ex officio members as appropriate. Governance and Scheduling The chair and vice chair should be elected by the board for terms of 2 years. The board should meet at least semiannually. The board will report directly to the director, NIJ.
OCR for page 220
Strengthening the National Institute of Justice Powers and Responsibilities: Advise on policies involving grant selection and administration. Advise on the development and implementation of research priorities. Review and advise on appropriate procedures for technical and scientific peer review. Review and advise on NIJ’s dissemination strategy. Review and advise on NIJ’s quality assurance activities. Review all grants and contracts exceeding $1 million prior to their award. Biannually prepare a report to the attorney general on NIJ’s progress. Given the powers and responsibilities described above, the committee suggests in Box 7-1 an array of topics that might conceivably be included in meeting agendas. Leadership Since 1979, the NIJ director has been a presidential appointee. But the original legislation says little about the requirements for that position, does not set terms, and does not provide for a careful transition from one director to another. Traditionally this role has been seen as one requiring familiarity with legal institutions, crime trends, crime policy, and the criminal justice system but not experience in directing or conducting crime and BOX 7-1 Possible NIJ Advisory Board Meeting Topics Critical research gaps relevant to NIJ’s mission Development of a cumulative research strategy Need and realities of balancing basic and applied research NIJ’s research activities vis-à-vis the work of other federal agencies Creation of an advisory infrastructure Feasibility of expanding efforts to build research infrastructure (e.g., graduate fellowships or the data archive) Transparency in the peer review and grant award processes NIJ’s role in translating knowledge to policy and practice (e.g., training, commercializing new technologies, and/or dissemination products) Objective performance metrics for measuring strategic performance
OCR for page 221
Strengthening the National Institute of Justice justice research. In short, there has never been a premium placed on having as the director of NIJ a criminal justice researcher and in its 40-year history no NIJ director can be described as such. Successful research agencies are characterized by leaders who are recognized scientists in their field and staff who are researchers with experience and expertise in areas of their specialization. Success in the application of the science or in related areas cannot substitute for scientific accomplishments. In situations in which the research unit is in a parent organization dealing with intense political issues, terms of appointment that are fixed and that minimize the influence of political factors in retention decisions are critical (e.g., a fixed term of appointment that overlaps presidential elections). This is in sharp contrast to other federal research agencies, such as the Institute of Education Sciences (IES), in which the director is appointed for a 6-year term designed to overlap presidential administrations. The most common terms of office are 4- and 6-year terms. The director of NSF also has a 6-year term. The head of Congress’s Office of Technology Assessment has a 6-year term, and the head of the U.S. Government Accountability Office has a 12-year term. Terms of office do contribute to a longer period of sustained professional management and decrease the likelihood of shifting priorities and interests that a new director might bring to the job. The NRC has conducted numerous assessments of federal research programs similar to this one (e.g., National Research Council, 1992b, 2000, 2008c). These assessments have been unequivocal about the need for a top science official. In a study of EPA’s Office of Research Development, the report noted “the lack of a top science official is a formula for weak scientific performance in the agency and poor scientific credibility outside the agency” (National Research Council, 2000, p. 5). The leaders of the NIH institutes, the NSF directorates and divisions, and IES are all drawn from the scientific community. In the case of the latter, this requirement is legislatively mandated and calls for the director to be “[a] highly qualified authorit[y] in the fields of scientifically valid research, statistics or evaluation in education as well as management within such areas” (U.S. House of Representatives, 2002). For these reasons, we recommend that Congress should set science requirements for the director and a renewable fixed term of service of 6 years. An appropriate director for NIJ shall have had experience in directing crime and justice research, be recognized as a highly qualified authority in the fields of crime and justice research (including evaluation research), and have demonstrated success in managing substantial crime and justice research efforts. A term of 6 years means that appointments will overlap presidential elections but allows for change in leadership as the requirements of the position may change. Also, during the most recent congres-
OCR for page 228
Strengthening the National Institute of Justice NIJ should revise its program planning and grant selection processes in ways that reflect adherence to established criteria and well-established scientific practices. The committee urges the new NIJ to establish a clear and timely planning process that includes the following elements: A structured agenda-setting process with full participation of staff and broad outreach to researchers in the planning process for both social science and technology research. Engagement of practitioners in the planning process to identify needs and to build support for the conduct of research as well as the adoption of findings by criminal justice agencies. More open, transparent, and collaborative approach to identifying areas and topics for long-term funding. Development of a strategic plan that identifies these areas and topics. Making the strategic plan publicly available in a timely way. Consistent and extensive oversight by external, independent, scientific reviewers. There are numerous models of agenda setting employed by other federal research agencies that NIJ can turn to for guidance. The individual institutes of NIH engage in a very structured process that is well documented, involves the full participation of staff, and engages the research community through their participation as members of standing advisory committees and as experts in particular areas. Although there is considerable variety in the specifics, the overall process is similar and is designed to result in a strategic plan. The purpose of the strategic plan is to announce to researchers the near-term priorities—priorities that focus on “the greatest need and promise for scientific advancement” (National Institute on Alcohol Abuse and Alcoholism, 2006). Several NIH institute division directors met with the committee, shared information about their planning processes, and described the various roles of senior leadership, program staff, and researchers. Detailed written information was also received from the Division of Epidemiology and Prevention Research (DEPR) at the National Institute on Alcohol Abuse and Alcoholism (NIAAA) that proved very helpful in fleshing out the particulars of its planning experience (see Box 7-2). Two things stand out about the DEPR process. There was broad input from the science field at every stage, including the final review meeting of NIAAA’s Extramural Advisory Council. Ad hoc experts from several federal agencies, universities, research centers, and medical centers attended this meeting, commented on the priorities, and responded to questions. There
OCR for page 229
Strengthening the National Institute of Justice BOX 7-2 Planning at the Division of Epidemiology and Prevention Research One of four divisions of NIAAA, the Division of Epidemiology and Prevention Research was newly created in 2004. Its responsibilities are similar to those of NIJ in that it plans, conducts, and supports epidemiological studies; it supports clinical programs; it analyzes ecological and situational factors related to its particular subject area; it collaborates with outside organizations; it collaborates on studies with a broad array of organizations; it sponsors scientific conferences; and it supports a wide range of grants, including research and research training. As part of its 2-year strategic planning process, division staff developed a mission statement; identified critical issues; reported on trends and related areas impacting on the problem being studied; reviewed relevant research findings; analyzed its current portfolio; described pertinent methodological issues; developed contextual background information and information on other relevant initiatives sponsored by the broader agency; and undertook a financial analysis of funding allocations within the division. This planning process culminated with the adoption of its strategic plan that received approval by the NIAAA National Advisory Council. was also full documentation of the process, of the analyses that had been undertaken, and of decisions reached. Upon review by NIAAA’s National Advisory Council in February 2007, the strategic plan and all written materials were posted on the NIAAA website, where it remains today. In comparison to the DEPR description of a formal process, NIJ’s agenda planning activities have varied over time and, particularly in the case of ORE, are not well developed or formalized. NIJ should take immediate steps to develop a formal, unified, and structured planning process. It should include broad participation of the research community at various stages and in different capacities. NIJ should base its agenda on a thorough understanding of the state of the art, an analysis of its own past and current research portfolio, and extensive consultation with the science community. Only by having such a process is NIJ likely to encourage creative and innovative thinking about both research issues and methodologies.
OCR for page 230
Strengthening the National Institute of Justice Signaling of Research Solicitations NIJ has not been consistent in signaling the research community about its solicitations and proposed submission dates. NIJ should provide more predictable levels and timing of funding and as much advance notice as possible about solicitations. Other federal research agencies, such as NSF, provide notice of solicitations well in advance of their deadline dates. In the case of the Law and Science Program at NSF, submission dates occur semiannually at the same time of the year. NIJ should aim for regularly timed solicitation cycles. This makes it much easier for applicants to plan around these deadlines and, in the event they apply and are not successful, to have sufficient time to revise their proposal and resubmit for the next round of funding. Peer Review and Grant Selection Process Much has been written about peer review and the requisites of a high-quality scientific peer-review system (National Research Council, 1992b, 2000; U.S. General Accounting Office, 1999; U.S. Office of Management and Budget, 2004). In a review of peer-review processes being used by agencies that fund education research, NRC identified the following attributes (National Research Council, 2004d): flexibility in designing peer-review systems to meet the individual needs of an agency; independence of the project officer from the peer-review process; nonparticipation of practitioners in assessment of research expertise; if included, their role is to assess relevance, significance, applicability, and impact of the proposal; a strong infrastructure: knowledgeable staff, systems for managing the logistics of peer review, technologies to support review and discussion of proposals, clear mechanisms for providing feedback, and standing panels when research priorities are clearly stable; predictable levels and timing of funding for the process itself; clearly delineated scoring systems and the meaning of the scoring levels defined and understood; training on scoring for reviewers as well as training on what the peer-review process entails and how to be an effective reviewer; qualified peer reviewers who have the research experience and expertise to judge the theoretical and technical merits of the proposals they review; and systematic evaluation of peer review based on intended outcomes and the public reporting of the evaluation.
OCR for page 231
Strengthening the National Institute of Justice Measured against these attributes, the NIJ peer-review process is very weak. The committee notes the current practice of having both researchers and practitioners serve as peer reviewers and assess proposals using the same criteria. We view this as highly inappropriate given that the purpose of peer review is to assess scientific merit and arrive at consensus. NIJ’s ability to adapt or change its peer-review process to meet its own needs is constrained because peer-review services are managed and overseen centrally under OJP. Scoring systems are not understood and do not appear to have a scientific basis. NIJ has not undertaken a systematic evaluation of its peer-review processes. Having a good scientific peer review and grant selection process in place is a critical requirement for a high-quality research program. In the federal government, although peer review is used for many purposes, its primary use is to assess the scientific merit of competitive and noncompetitive proposals. There is no written government-wide definition of peer review, but there is general acceptance that it is an “independent assessment of the technical, scientific merit of research by peers who are scientists in knowledge and expertise equal to that of the researchers whose work they review” (U.S. General Accounting Office, 1999). Proper peer review is integral to the work and reputation of a science agency. NIJ’s authority to conduct and manage its own peer review should be clearly established and protected. NIJ should follow standards of practice for peer review reflective of a science agency. NSF and NIH are excellent models of scientific peer review. These models have been in place for many years, and although the NIH peer-review process has recently undergone a major overhaul to improve what was perceived as its shortcomings, it still retains the basic infrastructure and approach (see Box 7-3). Grant Award Process During the course of our study, four different persons served as the final arbiter of grant award decisions at NIJ, and it is possible that each one used different criteria and a different in-house process to make the decision. NIJ staff themselves seemed unclear as to how decisions were made. According to the perceptions of Congress reflected in recent appropriation bills and the majority of researchers who responded to the committee’s survey, the NIJ process appears to be flawed. A research institute’s reputation is based not only on the quality of its work but also on the perception that its grant award process is fair, consistent, and transparent. NIJ needs to reestablish its credibility by signaling clearly to the research community as much information as it can regarding its funding, the amount it anticipates providing, the selection criteria it will use, and the role of the various participants in the process—in short, the basis on which research support is being provided. Documenting and
OCR for page 232
Strengthening the National Institute of Justice BOX 7-3 Peer-Review Models The NIH Peer-Review Model Responsibility for peer review across NIH resides with the Center for Scientific Review (CSR), an organization under the Office of the Director of NIH. CSR is usually the initial point of contact for applications, and it also manages the review process for most of the grant applications from the 27 NIH institutes. For grant applications, there is a two-step review process: (1) a review by a Scientific Review Group (SRG) under CSR’s direction and then (2) assignment to at least one NIH institute for potential funding. This SRG usually contains 15-18 scientific experts who meet three times a year. One of the members, designated as the chair, leads the discussion. The initial or first-level review evaluates the scientific and technical merits of the application. The second review is done by the national advisory council for the particular institute. These councils are composed of 12 or more members drawn from the scientific and lay communities. They review the initial judgments of the proposals and make recommendations for funding in light of relevance to the program and to the institute’s overall mission. In practice, the councils make few changes in the priority rankings from the initial review. Sometimes applications for funding bypass the CSR review process and are submitted directly to the individual institutes, where they undergo review by standing committees that are formed for a very specific purpose. In this case, they fall under an institute’s individual Office of External Research. Once through this stage of review, they go on to the particular national advisory council of that institute for a funding decision. It is important to note that this process involves not only grants and cooperative agreements but also contracts. NIH is currently in the implementation phase of a multiyear process to improve its peer-review process. The goals of this effort are to engage the best reviewers; to improve the quality and transparency of review; to ensure balanced and fair reviews across scientific fields and career stages and reduce administrative burden; and to provide for a continuous review of peer review. It is an activity that NIJ would do well to emulate as it seeks improvements to its own peer-review process. The NSF Model Peer review is conducted differently at NSF than at NIH but it reflects the same core values of scientific competence, fairness, timeliness, and integrity. The division within NSF that most closely approximates NIJ in
OCR for page 233
Strengthening the National Institute of Justice terms of the kinds of research it sponsors is the Law and Social Science program, one of 16 individual programs with the Social and Economic Sciences (SES). The SES is located within SBE, one of seven Directorates at NSF. Proposals received by the NSF Proposal Processing Unit within the Division of Grants and Agreements are assigned to the appropriate NSF program for acknowledgement and, if they meet NSF requirements, for review. All proposals are carefully reviewed by a scientist, engineer, or educator serving as an NSF Program Officer, and usually by three to ten other persons outside NSF who are experts in the particular fields represented by the proposal. Proposers are invited to suggest names of persons they believe are especially well qualified to review the proposal and/or persons they would prefer not review the proposal. These suggestions may serve as one source in the reviewer selection process at the program officer’s discretion. Program Officers may also obtain comments from assembled review panels or from site visits before recommending final action on proposals. Senior NSF staff further review recommendations for awards. Each Division has its own Advisory Panel that is comprised of scholars from multiple disciplines (anthropology, criminology, economics, political science, psychology, sociology). In the case of the Law and Social Science Advisory Panel, eleven members serve a 2-year rotation and provide written reviews that supplement the external reviews. The Panel meets to discuss proposals and propose the ranking and within that a further breakdown for actual funding. It provides advice as to whether the proposal is competitive. The program officer makes the recommendation to the Division Director as to which ones to recommend and can disagree with the Panel’s guidance. Normally final programmatic approval occurs at the division level. NSF relies on the judgment of program officers to make funding recommendations that address NSF strategic goals. One important way in which the NSF peer-review system differs from that of the NIH is the amount of discretion the program officers have. Peer review at NSF still receives high marks because of other features it has and its overall transparency. For example, all applicants have access to an electronic system called FastLane in which they can track the progress of their applications and can view the individual peer reviews and scores as well as any summary reports, site visits, or other documents that constitute their application file. Second, NSF program officers are highly qualified researchers and academicians who see their role as encouraging good research that is responsive to NSF priorities and who work with applicants to shape them accordingly. Staff also communicate with applicants in a timely way regarding their submission and provide useful feedback that often allows unsuccessful applicants to revise pro
OCR for page 234
Strengthening the National Institute of Justice posals in time for the next round of applications. Third, the system is perceived as responsive to researcher needs. Detailed information on the process is available to any applicant on the NSF website. The NSF electronic system can accommodate changes in an application prior to the final submission date. Submission dates for many programs are set in advance, frequently on a semiannual schedule. Finally, there is a check on peer review in the form of program reviews that are undertaken periodically by the Committee of Visitors. This quality assurance process examines technical as well as managerial oversight of the program. In doing so, it holds program officers’ accountable for their recommendations but also serves as check on the quality of reviews, the basis for the program officers’ recommendations. maintaining consistency in the award decision process is an important first step to rebuilding trust and confidence in NIJ. Report Review Although NIJ regards the dissemination of research findings as a part of its mission, the review process that supports its own dissemination process is very weak. There seems to be a lack of clarity about what NIJ expects from its grantees regarding final reports. Like agenda setting and peer review, the report review process is not well understood by researchers, who are critical of the amount of time it takes for NIJ to release research reports or research findings publicly. This situation prevails despite the fact that interest in making scientific research available to the public has never been greater. Recently, particular attention has focused on the ability of scientists working in the federal government to communicate with the media and the public. The Union of Concerned Scientists, a public interest advocacy group that investigates the extent to which scientists have been thwarted in their efforts to release their findings, has been very involved in bringing to the public’s attention various cases in which research findings had been suppressed or altered (Union of Concerned Scientists, 2007). It also proposes a model media policy that outlines responsibilities for public affairs officers as well as researchers and other staff. This policy provides that employees have the right to review, approve, and comment publicly on the final version of any proposed publication that significantly relies on their research, identifies them as an author or contributor, or purposes to represent their scientific opinion (Union of
OCR for page 235
Strengthening the National Institute of Justice Concerned Scientists, 2008). GAO has also assessed several research agencies’ practices regarding release of information and recommended that policies be in place to guide researchers and public affairs officials in their efforts to disseminate research to the scientific community (U.S. Government Accountability Office, 2007). Our examination of other federal research agencies indicates that they either have complete authority over reports issued in their name or they rely on researchers publishing independently. NIJ needs to improve its research report review process as well as to support efforts of researchers to publish in peer-reviewed journals. Achieving Transparency A common theme running through our assessment of NIJ’s research operations is the importance of transparency. Transparency implies openness, communication, and accountability. It means that the decisions of research agencies regarding agendas, award decisions, and impact (in the form of research findings and reports) should be apparent to the research community and the public. In our examination of NIJ’s research operations and comparison with the practices of other federal research agencies, the committee found a lack of transparency, which is reflected in the limitations of the information made available to grant applicants, grant awardees, and the public. It is also reflected in our interviews with NIJ staff, in which many of them explained that they did not know how funding decisions are made or the basis for them. The lack of transparency and quality of information were apparent to the committee as we attempted to access directly or be provided with certain information for this study. Despite the facts that our study meets the necessary privacy and confidentiality requirements and it was requested by NIJ itself, our efforts to gather data were often unsuccessful, either because the data did not exist, they existed in an unusable form, or barriers to obtaining the data could not be overcome by NIJ staff or leadership. Several members of Congress have voiced concerns about the lack of transparency, raising questions about conflicts of interest and questionable practices. As mentioned in Chapter 4, the most severe criticism has come from the Department of Justice’s Office of Inspector General, which concluded that NIJ’s failure to document key aspects of its preselection and grant award process called into question the fairness and openness of its competitive grant-making process (U.S. Department of Justice, 2009b). The lack of transparency is also reflected in the views of criminal justice researchers we surveyed. In the absence of information and feedback on how decisions are made and the criteria that are used to reach them, it is reasonable to attribute these decisions to political considerations.
OCR for page 236
Strengthening the National Institute of Justice In order to achieve transparency, a research organization needs the capability, authority, and resources to maintain records of its own activities that are accurate, easily retrievable, and accessible to the public within limits of the law. The committee acknowledges that NIJ is very dependent on OJP’s centralized grant management information system and the resources provided by the Office of the Chief Information Officer. However, NIJ should take responsibility for documenting its decision-making processes, and in instances in which the information is not being generated by OJP, it should develop its own documentation and records. At a minimum these records should include Information on the life cycle of a grant, from award to final report and resultant publications. At a minimum, titles, grant amounts, principal investigators, and grant abstracts should be a part of this information. Information on individuals who have received funding and their research records, products, citations, and professional recognition. Financial information for all NIJ organizational units that is consistent and includes appropriated funds and expenditures and breakdowns between various categories of restricted and discretionary funds, and program topics. This information should be compiled in a way that it can be used to present a historical picture of funding. Information on new staff hires and vacancies that indicates grade level, position, and education levels. Such information should be compiled annually. Involving the Research and Practitioner Communities Another theme throughout our deliberations on NIJ’s research operations is the need to clarify the important but separate roles that the research and practitioner communities should play in the research program. These roles are reflected in the proposed composition of the NIJ advisory board, in the qualifications of the NIJ director, and in the improvements that the committee recommends regarding agenda setting and planning, peer review, and report review. NIJ should increase its efforts to involve researchers and to seek their advice in the development, implementation, and assessment of its research activities. Their advice is critical to identifying and shaping the kind of science needed to accumulate enough knowledge to answer critical policy questions. The question “What works?” requires a multilayered response and a probing into what led to success as well as what led to failure. Researchers are uniquely trained to pose these questions and seek the answers.
OCR for page 237
Strengthening the National Institute of Justice Practitioners also have an important role to play and through their training and expertise can provide broad policy direction to address research concerns. As consumers of the research, practitioners can also advise as to its need and relevance. ESTABLISH A CULTURE OF SELF-ASSESSMENT RECOMMENDATION 5: The National Institute of Justice should measure the influence of its programs on research and practice and assess the quality of operations and program-level technical and managerial matters. NIJ’s efforts to promote or conduct assessment activities have been extremely limited. With the exception of the 1977 NRC study and this current one, there has been no other independent review of its body of work. Research on particular topics has been discussed as part of more general substantive NRC reviews (see Appendix E), but, with the exception of the NRC report on violence against women (National Research Council, 1996b, 2004a), NIJ’s own research has generally not been singled out. The committee saw little evidence of regularly scheduled systematic reviews of the ongoing research portfolio as part of a planning process or ongoing monitoring activities. Ensuring quality in a research organization through careful monitoring is a particularly complex undertaking. It needs to take into account the quality of the research as well as the many different processes that support the research enterprise. At its highest level, assessment activities can be directed to assessing the broad influence of a program or its major achievements or to more narrow programmatic goals or benchmarks. They can include assessments of various processes and of the products themselves as well as customer satisfaction. They can also include assessment of staff performance. NIJ will not be able to conduct these kinds of assessments without better records and a willingness to provide access. Throughout this report, we note difficulties in obtaining documentation of research operations as well as other kinds of administrative information. Some of the problem lies with the centralization of records under OJP, which is not tailored to the needs of NIJ. NIJ should take responsibility for creating record systems that will allow for detailed analyses of administrative and program funding, administrative and personnel matters, and all programmatic activities. These records need to be consistent across all NIJ offices. Most critically, it should create a basic mechanism to track usage and influence of funded research in scholarship and practice. Considering how other federal research agencies address the issue of
OCR for page 238
Strengthening the National Institute of Justice assessment is instructive. First, many research agencies struggle with defining quality and then constructing a process to assess it. This is especially true in assessing the impact of research or research outcomes as a measure of quality (National Research Council, 2008b). Second, scientists outside the agency are heavily relied on, either individually or as peer reviewers, to assess quality. Peer review has many other purposes than proposal review. Many federal research agencies rely on individual peer reviewers or peer-review panels to set priorities, develop programs, and even conduct personnel evaluations. Third, individual processes, such as peer review of proposals and the report review process, should include an evaluation component that provides useful feedback on efficiency and quality. Like other well-managed federal agencies, NIJ should establish self-assessment as an ongoing activity and use it to constantly improve its operations. Furthermore, to ensure transparency, it should make the results of such assessments publicly available. This is unlikely to happen, however, unless our recommendations on leadership and independence are implemented. NIJ is the only federal agency devoted to crime and justice research. Despite the problems noted in this report, it has accomplished much in its short history. Our assessment of NIJ supports its continuation and growth, but with legislation and actions by the executive and congressional branches that will allow it to fulfill its role as the nation’s center for improvements in crime control and prevention. When those changes happen, we are confident that NIJ’s self-assessments and periodic external assessments will describe an agency that is science driven and that is making even greater contributions to the understanding of crime and the improvement of justice.