National Academies Press: OpenBook

Strengthening the National Institute of Justice (2010)

Chapter: 4 Research Operations and Staffing Resources

« Previous: 3 The Research Program Offices
Suggested Citation:"4 Research Operations and Staffing Resources." National Research Council. 2010. Strengthening the National Institute of Justice. Washington, DC: The National Academies Press. doi: 10.17226/12929.
×

4
Research Operations and Staffing Resources

The history and broad outline of the research portfolio of the National Institute of Justice (NIJ) described in this report reflect frequent changes in leadership, an increased oversight role for the Office of Justice Programs (OJP), several upswings and downturns in funding, and a shift from support for a predominantly social science research program to one in which the majority of funding is expended on technology-related activities.

In an effort to better understand these changes and their ramifications, the committee examined NIJ’s research operations and staffing resources. In this chapter, we first describe how NIJ plans its agenda, develops solicitations, conducts peer review, awards grants, and reviews its research reports. We examined these processes in light of three overarching issues: (1) the transparency of the processes, (2) the relationship between the research and practitioner communities and NIJ staff, and (3) the relationship between NIJ and OJP. When documentation is available, we describe how the processes changed over time.

We then describe NIJ’s staffing resources and analyze staff turnover and other staffing characteristics that have affected how NIJ conducts its business. Taken together, this analysis provides an important context for understanding and assessing the current program and its evolution into the research organization of today.

Suggested Citation:"4 Research Operations and Staffing Resources." National Research Council. 2010. Strengthening the National Institute of Justice. Washington, DC: The National Academies Press. doi: 10.17226/12929.
×

RESEARCH OPERATIONS

Consistency, openness, efficiency, and fairness are characteristics associated with good research and development (R&D) project management (National Advisory Committee on Criminal Justice Standards and Goals, 1976). They are also useful as a yardstick against which one can measure the quality of a research agency’s operations and determine how well they contribute to fulfilling its mission. For our analysis of research operations, the committee reviewed written OJP guidelines and policies (Office of Justice Programs, 2007c) and also relied heavily on interviews with former and current staff as to how these processes worked in reality and over time. Anyone familiar with bureaucracies knows that often there is a divide between what an official policy or process proposes and how things are actually done. In this report, we have found it necessary to distinguish the two.

We have also tried to describe the differences between the way the Office of Research and Evaluation (ORE) and the Office of Science and Technology (OST) handle various functions. As described earlier, until the late 1990s, the social science research activities dominated the science and technology ones. Consequently, the background information on operations primarily describes how social science research activities are handled. After the mid-1990s, when the OST program greatly expanded, these processes were either adopted by OST or were modified to reflect its own statutory requirements and programmatic needs. We have tried to indicate the nature of these changes and when and how they occurred.

Planning

Past Planning Activities

Traditionally, planning at NIJ has been a bottom-up process, with staff playing a major role in determining research priorities. Although there had been some attempt to use an advisory board (see Chapter 2) to solicit input from a broader audience and build support for research activities, there is no documentation that the NIJ Advisory Committee in the 1970s or the NIJ Advisory Board in the 1980s contributed to a formal research plan.

Rather than rely on an overall advisory board to advise it on long-range planning and annual priorities or on an advisory infrastructure for specific program areas, NIJ has convened topical meetings and workshops to solicit input from practitioners and researchers1 on its social science research ac-

1

Throughout our report, we use the terms “researcher” and “practitioner” to refer broadly to two groups of people who generally differ from each other in their training, expertise, and

Suggested Citation:"4 Research Operations and Staffing Resources." National Research Council. 2010. Strengthening the National Institute of Justice. Washington, DC: The National Academies Press. doi: 10.17226/12929.
×

tivities. One example of meetings aimed exclusively at practitioner groups is the 21st-century working groups convened during 1991-1993. Three separate groups of law enforcement, courts, and corrections practitioners provided informal guidance about the research agenda. The law enforcement group consisted of newly appointed and progressive police chiefs who met three or four times a year. The courts and corrections groups were convened less frequently but also assisted in identifying critical research needs.2

Through the years, NIJ staff have also participated heavily in conferences and meetings held by national criminal justice practitioner organizations, such as the International Association of Chiefs of Police and the American Correctional Association; research organizations, such as the American Society of Criminology and the Justice Research and Statistics Association; and meetings convened by other federal agencies, such as the Centers for Disease Control and Prevention. NIJ relies heavily on these meetings to exchange information and to generate interest in its activities.3

Until the mid-1990s, planning was conducted and coordinated by a director of planning who headed the Analysis, Planning and Management staff and reported directly to the NIJ director. The staff prepared budget requests for the Office of Management and Budget (OMB) and the U.S. Department of Justice, developed budgets, tracked spending, handled personnel matters, and oversaw an institute-wide planning process. The director of planning was responsible for communicating with outside groups, soliciting their input, and working closely with NIJ management and staff to develop an annual program plan.

Strategic Planning

NIJ undertook development of a long-range strategic plan in the early 1990s (National Institute of Justice, 1993). Six long-range goals were identified: (1) reduce violent crimes and their consequences; (2) reduce drug-related crimes; (3) reduce the consequences of crimes for individuals, households, organizations, and communities; (4) develop household, school, business, workplace, and community crime prevention programs; (5) improve the effectiveness of the criminal justice and service systems in responding to offenses, offending, and victimization; and (6) develop and evaluate information for criminal justice responses to changing and emerg-

occupation. The term “researcher” is inclusive of all people whose primary training, expertise, and occupation involve research activities regardless of where they work.

2

E-mail from Craig Uchida, former director ORE (August 20, 2009).

3

Memorandum to David W. Hagy, director of NIJ, from Thomas Feucht, assistant director of ORE, on practitioner consultations, August 2, 2007.

Suggested Citation:"4 Research Operations and Staffing Resources." National Research Council. 2010. Strengthening the National Institute of Justice. Washington, DC: The National Academies Press. doi: 10.17226/12929.
×

ing crime patterns and for use of new technologies. The idea was to build a body of cumulative research—both basic and applied—and to develop practitioner-oriented activities around these as well. The result of this planning was the development of a substantial number of solicitations that were issued in fiscal year (FY) 1994. A matrix was developed that could be used to monitor progress in conducting the research over a 10-year period. With the change in leadership in 1994, the plan was relegated to an open solicitation and its use was abandoned.4

In FY 2002, NIJ developed a 3-5 year strategic plan.5 This plan, referenced in NIJ’s submission to OMB’s assessment process in FY 2005, identifies 7 broad strategic goals and 10 program areas. These same goals appear in NIJ’s 2002 Annual Report to Congress and represent a change from mission statements that appear in its 1999-2001 annual reports. After 2002, no similar statement of mission or strategic goals appears in NIJ’s annual report to Congress. More recently, NIJ took steps to develop a strategic plan and to obtain some feedback on its research priorities, in requesting this study from the National Research Council (NRC).

The lack of a structured strategic planning process makes NIJ even more susceptible to having its priorities set by others.6 Sometimes these priorities are supportive of NIJ’s mission, but sometimes they are questionable. On the science and technology side, for example, one program that was developed at the request of the attorney general was research on new technologies to detect concealed weapons at a distance. This line of research had never received much support from practitioners, but NIJ staff agreed that it would be useful and subsequently requested and received additional monies from the attorney general’s office. A solicitation was issued and, according to NIJ staff, high-quality proposals were received. In a similar way, the school safety technology program became a priority because of congressional and departmental interest and continues to be supported today. But this priority did not engender the same kind of support from senior managers, who questioned whether the investment in new technology was an appropriate response to the spike in school homicides.

4

E-mail from Carol Petrie, former acting director of NIJ, August 7, 2009.

5

In response to the committee’s request for this plan, we received a memorandum dated August 15, 2002 prepared by Sarah Hart, then director of NIJ, to Deborah Daniels, assistant attorney general. According to the memo, NIJ’s initial attempt to develop a strategic plan was insufficient, and the revised plan was being submitted as part of the document. No other documentation regarding a strategic plan was uncovered by the committee.

6

With the exception of the planning efforts by OST, there is little documentation regarding the priority-setting process across NIJ. The information on concealed weapons and the school safety program that follows was taken from staff interviews.

Suggested Citation:"4 Research Operations and Staffing Resources." National Research Council. 2010. Strengthening the National Institute of Justice. Washington, DC: The National Academies Press. doi: 10.17226/12929.
×
Annual Research Plans

Intermittently during the 1980s and 1990s, an annual NIJ research prospectus or research plan was published. It contained general descriptions of “strategic challenges” and focused almost exclusively on social science research activities, with increasing mention in the 1990s of other activities, such as crime mapping, DNA analysis, and standards development (National Institute of Justice, 1987, 1991, 1992, 1996, 1997, 1998, 1999). The documents also contained information on targeted or “directed” solicitations—for example, solicitations linked to 1994 Crime Act initiatives. After 2000, this kind of annual document signaling NIJ priorities to the research community was discontinued, until 2009, when NIJ released High Priority Criminal Justice Technology Needs (National Institute of Justice, 2009a), a document that describes how NIJ sets its research agenda. Unlike its predecessors, it focuses exclusively on the programs of OST.

Although no research prospectus for NIJ itself was produced during the period 2001-2008, the Violence and Victimization Research Division of ORE prepared a detailed divisional strategic plan in 2005. Covering FY 2006 “and beyond,” the plan built on an earlier planning document written in 2001. It reflected the deliberations from a two-day planning meeting as well as ideas from an earlier Violence Against Women research agenda. The strategic plan contained descriptions and justifications for more than a dozen future solicitation topics.

Currently, agenda setting and planning is decentralized, and ORE and OST present their plans separately to the director. Consultation on program priorities between the director and the senior management staff is conducted on an ad hoc basis. Although several people have had responsibility for planning in the director’s office since 2000, the current Planning, Budget, Management, and Administration Division does not appear to have major responsibility or oversight for institute-wide programmatic planning. Its responsibilities appear to be exclusively confined to providing financial oversight and monitoring activities, such as responding to audits and other financial inquiries, overseeing the packaging of grant proposals, and managing contracts.

Office of Research and Evaluation

Planning and agenda setting is handled differently by ORE and OST. ORE uses an ad hoc process that varies from year to year and from division to division. It relies heavily on staff expertise and consultation, primarily with practitioner groups, and policy makers through participation at professional meetings. In a listing of consultations with practitioners and policy makers for 2006-2007 prepared by ORE staff, approximately a dozen

Suggested Citation:"4 Research Operations and Staffing Resources." National Research Council. 2010. Strengthening the National Institute of Justice. Washington, DC: The National Academies Press. doi: 10.17226/12929.
×

meetings were held for the purpose of planning research activities. Also listed were NIJ staff participation in research advisory committee meetings of the International Association of Chiefs of Police, and other meetings involving groups of related programs, such as the Project Safe Neighborhoods, the Multi-jurisdictional Task Force Cluster, and the Serious Violent Offender Re-entry Initiative. There were also meetings to plan research workshops (e.g., on elder abuse, sexual violence).

As mentioned earlier, ORE also sponsors periodic workshops and meetings on particular topics as the need arises. According to NIJ, it has held approximately two dozen meetings or workshops (it is unclear what distinguishes one from the other) since 2000, an average of about two a year. NIJ relies heavily on these “strategic planning” meetings to articulate new or emerging knowledge needs; to assess and evaluate a body of recently matured research findings; or to describe options and priorities for further research in a given area.7 The list of various meetings covering 2001-2008 is available online, along with many of the meeting agendas, participant lists, and summaries.8 Of the 27 meetings listed, 12 focused on sexual violence, victimization, and violence theory; 4 were on policing; 4 were related to forensic science, including 2 on DNA evidence; 2 were on drugs and crime; 2 were on terrorism; and single meetings were held on pretrial research, hate crime, and human trafficking. With one or two exceptions, most of the meetings involved a mix of researchers and practitioners, with the number of practitioners dominating a majority of the meetings. In general, the meeting materials reflect a broad overview of the subject, with a general listing of issues or questions needing research. With few exceptions, there appeared to be little emphasis on the science but more on what information practitioners need to do their jobs better. No further information on these meetings exists, and with few exceptions it was not possible for the committee to determine the relationship between research ideas presented at these meetings and any action taken by NIJ in response to them.

There are no standing advisory committees for ORE as a whole or for any of its divisions. Occasionally NIJ-wide working groups have been formed, but these are strictly composed of NIJ and other OJP staff with an interest in the particular area. In 2007, according to ORE, there were at least eight of these working groups, as well as other internal NIJ working groups that participate in an activity known collectively as the Innovative Assessment Program. These internal NIJ staff groups meet for the purpose

7

Memorandum to David W. Hagy, director of NIJ, from Thomas Feucht, assistant director of ORE, on practitioner consultations, August 2, 2007.

8

A list of NIJ’s research meetings and workshops with links to available background materials is available online at http://www.ojp.usdoj.gov/nij/events/research-meetings.htm [accessed October 15, 2009].

Suggested Citation:"4 Research Operations and Staffing Resources." National Research Council. 2010. Strengthening the National Institute of Justice. Washington, DC: The National Academies Press. doi: 10.17226/12929.
×

of identifying, verifying, and assessing innovative criminal justice programs and policies.9

Once research topics are identified by the ORE divisions, they wind their way up to the director, following a series of meetings between staff and the ORE director. The process is not a formal one (i.e., no written guidelines or requirements). Similar meetings are held between the NIJ director and the OST director. The NIJ director then makes decisions on what particular research and programs will help accomplish the missions, with some consultation with the office directors and to some extent from senior management of OJP and the offices of the attorney general.

The interviews with current NIJ staff reveal considerable differences in their knowledge of the steps in the planning process as well as the quality of the process itself. The overriding sense of these interviews was that the process was a haphazard one. Some described the process as “intermittent,” with no focus on long-range efforts. Generally, there was a sense that “hot topics” rule, and some areas are completely neglected due to scarce resources. Some staff attributed the ad hoc nature of the process to the fact that senior managers are not provided with budget numbers around which to plan. Others expressed the view that, with such limited resources, there was no incentive to plan.

Ongoing consultation with the broader research community outside more formal planned meetings or workshops appears to be very limited and dependent on individual staff initiative. According to the staff interviews, some staff contact researchers directly to obtain feedback on particular issues and to keep abreast, and other staff depend on meetings with “experts,” presentations at professional association meetings, and contact with relevant Justice Department offices to provide opportunities to engage the research community.

Office of Science and Technology

In contrast to ORE, OST has had a history of working with a large and complex set of advisory and working group entities. It has relied on these groups to provide feedback on its overall program, to identify practitioner needs, and to coordinate with others conducting technology R&D (National Institute of Justice, 2009a).10 Currently, OST relies on the Law

9

Memorandum to David W. Hagy, director of NIJ, from Thomas Feucht, assistant director of ORE, on practitioner consultations, August 2, 2007.

10

From 1994 to 2003, Vice-Admiral Al Burkhalter, Jr. USN (Ret.), chaired an executive panel of members with science backgrounds that advised OST and provided detailed reviews of particular programs.

Suggested Citation:"4 Research Operations and Staffing Resources." National Research Council. 2010. Strengthening the National Institute of Justice. Washington, DC: The National Academies Press. doi: 10.17226/12929.
×

BOX 4-1

Technical Working Groups (TWGs)

  • Biometrics

  • Body Armor

  • Communications Technologies

  • Community Corrections

  • DNA Forensics

  • Electronic Crime

  • Explosives

  • General Forensics

  • Geospatial Technologies

  • Information-Led Policing

  • (Institutional) Corrections

  • Less-Lethal Technologies

  • Modeling and Simulation

  • Personnel Protection

  • Pursuit Management

  • School Safety

  • Sensors and Surveillance

Enforcement and Corrections Technology Advisory Council (LECTAC),11 a group of approximately 40 senior-level representatives from federal, state, and local criminal justice agencies; labor organizations; and international criminal justice organizations. They meet once a year to review field needs developed from various technical working groups (TWGs) and make a list of the top-10 R&D priorities. Through this list of priorities, NIJ uses practitioner-based input to help shape the activities of its S&T portfolio. Although the number of active TWGs has fluctuated, in 2008, there were 17 TWGs developing high-priority needs and reporting them to LECTAC (Law Enforcement and Corrections Technology Advisory Council, 2008). See Box 4-1 for the core technology portfolios represented by these TWGs.

11

A law enforcement advisory body, LECTAC has its roots dating back nearly 30 years before the establishment of the current TWG process (Law Enforcement and Corrections Technology Advisory Council, 2008). The Technology Assessment Program Advisory Council (TAPAC), established in the mid-1980s and consisting of more than 80 senior federal, state, and local law enforcement officials from the United States and Canada, preceded LECTAC (National Institute of Justice, 1994a). TAPAC was preceded by the National Advisory Committee for Law Enforcement Equipment and Technology, established in 1977 under NILECJ. According to David Boyd, former director of OST, LECTAC played a larger role in planning, and the TWGs began as subgroups of LECTAC.

Suggested Citation:"4 Research Operations and Staffing Resources." National Research Council. 2010. Strengthening the National Institute of Justice. Washington, DC: The National Academies Press. doi: 10.17226/12929.
×

As their titles indicate, some focus on specific technologies (e.g., body armor), some on criminal justice areas (e.g., community corrections), and others on crime problems (e.g., school safety).

Most, if not all, TWG members are criminal justice practitioners appointed by the OST deputy director and approved by the NIJ director. TWGs do not have formal policies or procedures regarding their size, management, or activities. There are no term limits on membership; members simply have to be actively employed. The TWGs meet twice a year to develop, rank, and finalize a list of priority technology needs in their respective areas. Each TWG generates about 50 priority needs, which results in a long list of roughly 900 technology needs. OST estimates that it is able to address one-sixth of them through competitive R&D and another one-sixth through operational testing and demonstration.12 The needs that are addressed depend on budget and directives as well as gap analyses conducted by OST staff to determine that no other federal agency is investing in similar technologies.

The OST process is well regarded by the OST staff and by many practitioners who are familiar with it through their association on the TWGs. However, the current process reflects numerous deficiencies. Planning by OST does not appear to include systematic or formal outreach to the research or academic communities. Researchers are seldom included as members of either the technical working groups that develop the initial list of “technology priority needs” or on LECTAC, which selects the overall priorities. Although one might argue that technology priority needs are not meant to be the same thing as research priorities, the technology priority needs are the basis for the operational requirements that NIJ uses to provide direction for its research and development activities. According to NIJ, its solicitations for research and development of new tools and technologies for criminal justice practice are advised by TWGs and LECTAC recommendations (National Institute of Justice, 2009a).

Although the TWGs play an important role in identifying practitioner needs, and through those needs identify priorities, they do not operate like formal scientific advisory groups. With the exception of the DNA Forensics TWG, their membership does not include scientists who can use their expertise to assess the state of the art or the methodology required to move the technology forward. The absence of scientific or technical advice means that needs are developed in a vacuum, without input as to what is possible. As a result, opportunities may be missed. TWG meetings are not open, nor are their deliberations made public. The Homeland Security Act of 2002 specifically exempts the TWGs from the Federal Advisory Committee Act (FACA) rules which require notification of meetings and access by the

12

E-mail from George Tillery, OST deputy director, May 16, 2008.

Suggested Citation:"4 Research Operations and Staffing Resources." National Research Council. 2010. Strengthening the National Institute of Justice. Washington, DC: The National Academies Press. doi: 10.17226/12929.
×

public to documents and meetings. As a result, they do not seek or elicit guidance from the broader scientific and practitioner community.

The current TWG structure is defined around the OST R&D portfolio areas and is not intended to be static. The TWGs are expected to adapt as priorities in the field change or as new technological capabilities emerge (National Institute of Justice, 2009a). However, the committee noted at least one example of a TWG (Corrections TWG) continuing to meet despite the absence of funded research awards in their respective portfolios over a number of years. OST’s investment portfolios are likely to change first, because of limits on available funds or office reorganization, before the TWGs are restructured. As such, the advice coming from the TWGs often falls into a void, and it is unclear to what extent their advice and recommendations can and do result in solicitations and subsequent awards. According to OST’s research, development, testing, and evaluation process (described in Chapter 3), if practitioner needs are not addressed in research and development programs because potential solutions already exist (or perhaps development is already sponsored by another agency), these needs are then addressed in Phase 4 of the process, in which NIJ considers supporting demonstrations and operational evaluations of potential solutions, or in Phase 5, in which NIJ considers publishing resource guides or standards documents that may be useful to the practitioner community. The elaborate advisory structure is used more often to inform the broader federal research community of the technology needs of criminal justice practitioners and to define NIJ’s dissemination and outreach practices regarding technological capabilities than to define its technology research and development agenda.

Thus, while the OST process is standardized and more formalized than the ORE process, the OST process shares some major deficiencies. It is unclear what guidance and advice either office receives from the research community. In the case of ORE, there is some attempt to document researcher input by posting agendas and summary minutes of meetings and workshops on specific research topics on the NIJ website. But it is very unclear how final decisions about priorities are made or how these decisions in turn influence which solicitations are released. It is clear that both offices rely almost exclusively on staff judgments as to what the state of the art is, what is needed, what is feasible, and how a particular research technology or question should be approached. The committee has doubts as to whether the scientific and technical qualifications of the staff are up to the task. Consequently, the committee thinks that the advice and guidance of researchers need to be made a meaningful part of a sustained agenda-setting process.

Suggested Citation:"4 Research Operations and Staffing Resources." National Research Council. 2010. Strengthening the National Institute of Justice. Washington, DC: The National Academies Press. doi: 10.17226/12929.
×

Solicitations

Directed and Field-Initiated Solicitations

The committee heard brief descriptions of NIJ’s solicitations in presentations by staff and was told that, to understand NIJ’s research priorities, we should look at their solicitations. NIJ maintains an archive of solicitations from 1996 to the present.13 From this archive, the committee reviewed the types of solicitations over the years, changes in topics, and changes in deadlines. In addition, we received from NIJ a list of titles for applications received and subsequent awards by solicitation for the period 2004 to 2007.

ORE and OST issue different kinds of solicitations to the field. ORE funds solicitations that are “directed” and describe the scope of the research or evaluation and the nature of the proposed research activity. It also issues an annual “open” or “field-initiated” solicitation. This kind of solicitation allows the researcher creative license to propose research studies on a broad array of topics. Typically $500,000 to $1.5 million is set aside for each directed solicitation and from $2.1 to $2.8 million for the annual field-initiated solicitation. However, in FY 2008, $5.4 million was awarded under the field-initiated Crime and Justice Research solicitation.

OST issues solicitations dealing with a specific technology area (e.g., communications, less lethal technologies, biometrics) that call for research studies and other kinds of activities to solve problems related to adoption and use of new technologies by criminal justice agencies. The solicitations are usually structured in two phases: submission of a concept paper, followed by a full proposal if the concept paper is approved. Concept papers under OST solicitations, which were introduced in 2005, differ from full proposals in that they are much shorter and do not contain detailed budgets. All concept papers and subsequent proposals are subject to review by a panel comprised of expert practitioners and technologists14 (National Institute of Justice, 2006b).

An average of $1 million is designated for grants funded under each solicitation for targeted technology development. For some solicitations such as Personal Protective Equipment and Corrections Technology, $250,000 is set aside while up to $3 million is set aside for the Forensic DNA Research solicitation. OST also issues a second kind of solicitation, which calls for capacity-building activities. These are issued for such programs as the Fo-

13

See http://www.ojp.usdoj.gov/nij/funding/expired-96-07.htm and http://www.ojp.usdoj.gov/nij/funding/expired.htm [accessed November 24, 2009].

14

NIJ uses the term “technologist” to refer to scientists and engineers; it can include academicians. Technologists are often program managers from one of NIJ’s federal partner agencies.

Suggested Citation:"4 Research Operations and Staffing Resources." National Research Council. 2010. Strengthening the National Institute of Justice. Washington, DC: The National Academies Press. doi: 10.17226/12929.
×

rensic DNA Backlog Reduction Program and the Paul Coverdell Forensic Science Improvement Program (see Chapter 5 for more details). Typically over $90 million is designated per year for grants through the eight solicitations for DNA backlog reductions and forensic lab improvements.

Between 1996 and 2009, only nine solicitations have been released regularly each year for a period of 6 years or more:

  1. Crime and Justice Research (and previously titled Investigator Initiated);

  2. Data Resources Program: Funding for the Analysis of Existing Data;

  3. Research and Evaluation on Violence Against Women;

  4. Forensic DNA Research and Development;

  5. Electronic Crime and Digital Evidence Recovery;

  6. Paul Coverdell Forensic Science Improvement Program (and its predecessor Forensic DNA Lab Improvement Program);

  7. Forensic Casework DNA Backlog Reduction Program;

  8. W.E.B. Du Bois Fellowship Program; and

  9. Graduate Research Fellowship.

This list reflects a mix of sustained effort in which the criminal justice field can expect NIJ to support: (a) field-generated research, (b) knowledge accumulation in a few substantive areas (violence against women, forensic DNA analysis, and electronic crime), (c) forensic capacity building, and (d) development of the research infrastructure through fellowships (examined in Chapter 5). NIJ has regularly released about 40 solicitations since 2005, which is a doubling of the 20 solicitations released each year in the period 1996-2004.

Most of the research, development, testing, and evaluation solicitations (70 percent) in substantive areas, however, have a shelf life of 1 or 2 years. A few have been continued for 4 or 5 years, such as research on elderly abuse, terrorism, and the impact of forensic science as well as technology development in communications, information-led policing, less lethal weapons, sensors and surveillance, biometrics, and general forensics, although half of these areas have been discontinued as of the 2009 solicitations.

As noted earlier, the annual release of NIJ’s research prospectus or research plan was discontinued in 2000. No other document or mechanism has filled its place that would adequately signal to the research community NIJ’s upcoming research priorities. The lack of a research plan, in combination with sporadically appearing and unannounced solicitations, leaves the

Suggested Citation:"4 Research Operations and Staffing Resources." National Research Council. 2010. Strengthening the National Institute of Justice. Washington, DC: The National Academies Press. doi: 10.17226/12929.
×

research community unprepared to respond competently to NIJ’s changing priorities.

Annual solicitations are released in the period from October of the preceding year to May-June of the fiscal year. From 2005 to 2009, the average number of days from solicitation release to due date has remained fairly constant, 55-68 days. Solicitations available for a shorter period, 29-46 days, tend to be from OST, both for technology development and forensic capacity building. A smaller window for these kinds of grants is probably justified on the basis that technology development solicitations involve submission of a concept paper stage months prior to the solicitation deadline for full proposals and the forensic capacity-building program is a formula grant award program that requires more of a pro forma application. ORE solicitations tend to be posted for 60-100 days. For the most part, the same type of solicitation is released at a regular time of the solicitation cycle.

For example, OST calls for technology development are released around October, and its solicitations for new national technology centers were released in May. There have been a few examples of a regular solicitation lagging from its regular release date: the Crime and Justice Research solicitation release date has varied from October to February; the Data Resources Program solicitation release date has varied from November to April; and the Social Science Research in Forensic Science solicitation release date has varied from November to February.

In the absence of a publicly available research agenda, the timing of solicitation announcements and their posting times are critically important factors in determining the level of response from researchers. This is particularly true for studies that are either appropriate for or call for building collaborative relationships between researchers and practitioners.

In spite of a narrow window for preparation, NIJ received an average of 1,200 applications a year through its competitive solicitations for the period 2004-2007. Of these, roughly 35 percent were submitted in response to the capacity-building solicitations; many of these (58 percent) were funded. In contrast, only 15 percent of the applications under solicitations for substantive areas of research, development, or evaluation are funded. These are not unusual trends when comparing awards for formula grants with awards for research grants. However, NIJ’s rate of funding research applications is noticeably lower than the 20-25 percent rate reported in recent years by other federal agencies, despite the recent financial squeezes they have experienced (Chase, 2009; Powell, 2009). With the exception of OST’s capacity-building solicitations, most (about 70 percent) of NIJ’s solicitations have resulted in only 1-3 funded awards per solicitation.

Suggested Citation:"4 Research Operations and Staffing Resources." National Research Council. 2010. Strengthening the National Institute of Justice. Washington, DC: The National Academies Press. doi: 10.17226/12929.
×

In FY 2008, ORE issued 11 directed solicitations15 in addition to its field-initiated solicitation. OST issued 13 solicitations for targeted technology development.16 Most of these solicitations were supported with funds transferred from other federal agencies (such as the Office for Victims of Crime, the Department of Homeland Security, and the Federal Bureau of Investigation) or were the result of congressional priorities (such as the Prison Rape Elimination Act, the President’s DNA Initiative, the Violence Against Women Act). It is also noteworthy that in 2008 10 of these solicitations did not result in funded awards.

According to ORE, directed solicitations, the pillars of its strategic research program, are used to build empirical knowledge over a number of years.17 Although directed solicitations end up reflecting NIJ’s research priorities, for the most part they are not independently arrived at as part of a strategic planning process, and few are supported by NIJ base funds. Instead, they depend on transferred funds and the interests of the funding agencies, Congress, or both. NIJ thus plays a minor role in determining how its directed solicitations will build knowledge across projects and over time.

The annual field-initiated solicitation is intended to fund numerous exploratory research activities. These awards for the period 2005-2008 address a range of issues, from understanding criminal behavior and victimization to examining the effects of a state’s program or policy. NIJ argues that open research solicitations create the most diversity and allow for innovative research and unanticipated research knowledge.18 This objective appeared to be reflected in the FY 2008 Crime and Justice Research solicitation, which called for “any social and behavioral research and evaluation topic relevant to State and/or local criminal and juvenile justice policy and

15

These included Sexual Violence in Prisons, Criminal Justice Technology Evaluation, Social Science in Forensics, Justice Systems Reponses to Violence Against Women, American Indian/Alaskan Native Criminal Justice Systems and Criminal Justice Technology, Crime Prevention and Gangs, Pretrial and Jail Research and Evaluation, Research on Trafficking in Persons, Policing/Public Safety Research and Evaluation, Terrorism Research, and Elder Abuse Research and Evaluation.

16

These included Biometric Technologies; Body Armor for Law Enforcement and Corrections; Communications Technology; Electronic Crime and Digital Evidence Recovery; Forensic DNA Research and Development; Geospatial Technology; Information-Led Policing Research, Technology Development, Testing, and Evaluation; Less Lethal Technologies; Personal Protection Equipment; Research and Development in Forensic Anthropology and Forensic Odontology; Research and Development in the Area of Controlled Substances Detection and Analysis; Research and Development in the Forensic Analysis of Fire and Arson Evidence; and Sensors and Surveillance Technology.

17

Unpublished paper by Thomas E. Feucht, June 2008, drafted in response to questions from the committee on how ORE selects and organizes its research programs and research investments.

18

Unpublished paper by Thomas E. Feucht, June 2008.

Suggested Citation:"4 Research Operations and Staffing Resources." National Research Council. 2010. Strengthening the National Institute of Justice. Washington, DC: The National Academies Press. doi: 10.17226/12929.
×

practice” (National Institute of Justice, 2007b). However, two grants NIJ funded under its FY 2008 solicitation included a $1.9 million longitudinal study of policing19 and a $995,707 grant to continue a longitudinal study of desistance by juvenile offenders in two states.20 Changing course with the FY 2009 solicitation, NIJ directed the field to focus on the following topics: predicting crime, terrorism, human trafficking, gangs, prisoner reentry, improving the justice system, and reducing the prison population. As of this writing, it is not known whether the resulting grants will reflect these priorities.

Approval Process for Solicitations

Through the 1990s, the approval process for solicitations was a fairly straightforward one. Solicitations were signed off by the NIJ director, prepared for publication, and then disseminated by the National Criminal Justice Reference Service using its mailing lists. In some years, the solicitations were published in full or as a summary announcement in the Federal Register.

Currently, preparation of solicitations consumes a great deal of staff time in terms of developing them, getting them through the review process, and readying them for posting on the NIJ website. Currently, solicitations go through a lengthy review process that involves three offices in OJP: (1) the Office of the Chief Financial Officer Budget Division (OCFO-BD), (2) the Office of the General Counsel (OCG), and (3) the Office of the Assistant Attorney General. OJP guidelines advise its bureaus to allow 47 days for review and final sign-off by the assistant attorney general, OJP.

The process includes multiple reviews by OCFO-BD and OGC and the assistant attorney general, OJP. It includes an initial review by OCFO-BD prior to its being signed off by the NIJ director. We were told that the purpose of this early review is to check whether appropriate performance measures are contained in the solicitations. Since a check of solicitations from 2007, 2008, and 2009 showed that they contain standardized language regarding performance measures, the nature of the review is unclear. Once approved by the NIJ director, it is sent to OGC, where, until spring 2009, it was reviewed by one of two staff attorneys specifically assigned to NIJ.

NIJ staff explained that OGC checks to be sure the solicitation is in keeping with NIJ’s legislative mission and that NIJ is not requesting an

19

An award of $1.9 million was made to the University of Illinois, Chicago, for the project Advancing Knowledge and Practice in Policing: A Longitudinal Platform for National Research.

20

The Pathways to Desistance study has been an ongoing effort since 1999 at the University of Pittsburgh. It was originally funded by the Office of Juvenile Justice and Delinquency Prevention and the MacArthur Foundation.

Suggested Citation:"4 Research Operations and Staffing Resources." National Research Council. 2010. Strengthening the National Institute of Justice. Washington, DC: The National Academies Press. doi: 10.17226/12929.
×

activity to be undertaken that may pose later legal issues. However, NIJ staff also reported that OGC staff also review for programmatic content and frequently raise questions or offer comments about the research objectives or activities being proposed. NIJ staff reported that on at least one occasion, an OGC attorney raised questions about the need for a particular methodological approach in a technology research solicitation. Once comments are made, staff must respond to them, and if changes are made, the documents are recirculated to the OJP offices for re-review. The whole process is handled electronically, and, in the final review stage, the various offices can sign off simultaneously.

The committee found this process to be an onerous one, and we question why this level of oversight and programmatic review by others outside NIJ is necessary or appropriate.

Peer Review and Grant Award Selection

Peer Review

NIJ developed its first peer review process in direct response to the recommendations of the earlier NRC committee (National Research Council, 1977). From the beginning the process was managed by an outside contractor that had responsibility for maintaining a list of qualified reviewers, identifying appropriate reviewers in response to a staff request, arranging for the logistics and payments associated with in-house peer-review meetings, and managing the transmission of the reviews to NIJ staff and then storing them. Numerous contractors have performed these services over the past 30 years; there has been some variation in how each contractor handled these tasks, but the basic tasks have remained the same.


Centralizing Peer Review in OJP. Over time, other program offices in OJP determined that they should also use peer-review processes for their competitive programs. To meet this need, the various offices transferred funds to NIJ in order to use its contractor. Eventually, this arrangement became an administrative burden, so the individual bureaus contracted for their own services.

In 2004, OJP determined that the peer-review process would be centralized. This decision resulted from recommendations of the OJP Business Process Improvement initiative, which was created to improve the OJP grant application process. According to the OJP order, the new policies are intended to increase the efficiency, integrity, and overall quality of the OJP peer review process.

Today, all applications for OJP competitive discretionary funding are subject to peer review, and one contractor serves the needs of all the bu-

Suggested Citation:"4 Research Operations and Staffing Resources." National Research Council. 2010. Strengthening the National Institute of Justice. Washington, DC: The National Academies Press. doi: 10.17226/12929.
×

reaus. The peer-review contract is overseen by an OJP contracting officer’s technical representative. For each solicitation, there is a subject-matter expert, usually a program manager assigned to a particular solicitation.


Description of the Process. A directive that is available on the OJP website describes responsibilities of the bureau program staff who oversee the work of the peer reviewers as well as various administrative requirements (Office of Justice Programs, 2007b). For example, it requires that peer review panels be conducted by either telephone or video conference unless a waiver is sought and approved for an in-person peer review.21 The order also describes requirements for evaluating peer reviewer performance, the process for changing peer-review panel dates, and a standardized methodology for scoring applications. The order establishes certain guidelines regarding how often the same peer reviewers may be used and requires that program offices track the composition of peer-review panels on a yearly basis in order to “promote unique panel compositions for each review cycle; track the individual service of peer reviewers; and analyze the use of peer reviewers by skill or primary discipline.” It also requires formal orientations and ongoing training for all peer reviewers. Steps in the review process include

  • individual scoring by the peer reviewer prior to the convening of the panel,

  • discussion of each applicant’s individual score by the group,

  • preparation of a final recommendation memorandum by the NIJ staff, and

  • presentation of the scores and recommendation memorandum to the program office leaders with any additional analysis performed by lead program staff.

Guidance is also provided by NIJ to its peer reviewers.22 This guidance describes in more detail the various steps leading up to the preparation of a final recommendation memorandum and includes a description of specific responsibilities before and during the peer-review panel, of the lead

21

NIJ argued against this requirement when it was proposed on the grounds that video- or teleconferenced peer reviews were not as effective in discussing research applications and arriving at consensus around complex issues (memorandum to Regina B. Schofield from Glenn Schmitt, acting director, NIJ, December 21, 2005). NIJ was denied approval to hold in-person meetings from 2006 through 2008 and was required to seek a waiver for each in-person review it sought. The committee could not find out how many waivers were requested or approved. In January 2009, the acting assistant attorney general, OJP, rescinded this policy but the OJP order has not been revised as of this writing.

22

Instructions to Reviewers for Peer Review Panels, ORE, NIJ, U.S. Department of Justice (February 2007).

Suggested Citation:"4 Research Operations and Staffing Resources." National Research Council. 2010. Strengthening the National Institute of Justice. Washington, DC: The National Academies Press. doi: 10.17226/12929.
×

technical reviewer as well as the other technical reviewers and practitioner reviewers and the NIJ program manager. Ethics and integrity, conflict of interest, and confidentiality issues are also spelled out.

According to the contracting officer’s technical representative for the peer-review contract, assessment is ongoing and includes monthly monitoring of the frequency of individual peer reviewers’ participation and use of first-time reviewers. Ad hoc reviews of peer-reviewer evaluations as well as the peer reviews themselves are also conducted.23

The committee was very interested in the details of NIJ’s peer-review process because of its importance to the research enterprise. We were particularly interested in the various steps of the review process and the extent to which the recommendations of peer-review panels are followed at the division level and to what extent the recommendations are used by the NIJ director. This decision process is not detailed in the OJP grants management system, although it does exist manually in the documents forwarded to the NIJ director.

The committee did not have access to any documents associated with the review or award of grants. We were unable to assess how closely these written policies are followed by NIJ or other program offices; what, if any, latitude the program offices actually have in modifying or changing any of these policies; whether quality reviews are being undertaken; and what role the numerical rankings play in funding decisions. The last question was of particular interest, since the OJP order clearly states that “final decision makers of awards may consider application scores generated by peer-review panels in their review of applications but shall not be bound by them.”24 The OJP order explains that numerical scores are considered pre-decisional, for internal information only, and are not “normally” released to the applicants.

On the basis of staff interviews and the memoranda we examined, we think that centralization of peer review in OJP, as well as some of the specific practices associated with it, are problematic for NIJ and raise serious questions about the quality and integrity of the process. The U.S. Government Accountability Office (GAO) has noted that there is no uniform federal policy for conducting peer reviews and cites officials at the White House’s Office of Science and Technology who believe that peer-review practices should not be dictated uniformly for every agency or for all types of federally funded research (U.S. General Accounting Office, 1999). This view was also expressed in an OMB guidance document to the effect that

23

Telephone interview with Amy Callaghan, contracting officer’s technical representative, OJP, August 7, 2009.

24

See the section above on the grant award process for a discussion of recent findings of the U.S. Department of Justice audit of NIJ’s practices.

Suggested Citation:"4 Research Operations and Staffing Resources." National Research Council. 2010. Strengthening the National Institute of Justice. Washington, DC: The National Academies Press. doi: 10.17226/12929.
×

a one-size-fits-all approach is not appropriate for agencies with different missions (U.S. Office of Management and Budget, 2004).


Issues Arising from Centralization. On the basis of our review of how solicitations are developed and applications are reviewed, the committee thinks that a centralized process limits NIJ’s ability to tailor peer review to its own needs and mission. Its discretionary grants are very different from those of other OJP bureaus, which provide support for program services, training, or equipment. Applications requesting programmatic support require a different kind of expertise than applications proposing to conduct research. The former usually have more definitive goals and activities and may involve less complicated issues than research applications in which there is a need to discuss the feasibility and appropriateness of a particular research design and methodology and the need to meet accepted standards of rigor. Furthermore, the peer review for an NIJ research application may serve a more expansive purpose than one produced for another OJP program: it may be used by NIJ to modify the application, to provide guidance once the applicant is funded, and to inform the program staff of potential issues with the newly funded research, among other issues.

Because peer review is so fundamental to the scientific process, NIJ needs to have a full say in all aspects of its peer review, including how the panels are conducted, the selection of reviewers, the composition of the panel, the review criteria, and other aspects of the process. NIJ also needs the ability to make the reviews available in a timely way to the applicants and to conduct its own assessment of the quality of the process. It needs to be able to explain the process accurately and fully to all applicants. In our review of the OJP order and NIJ guidelines, the committee notes several features that deserve further scrutiny: the use of practitioners on panels, the scoring and ranking system, and the process for making individual peer reviews available. These are discussed further below.

The role of practitioners as peer reviewers is an issue that NIJ needs to address, but it will be difficult to do if the peer-review system remains centralized. Although this has been a long-standing practice of NIJ (certainly since the 1980s), the committee questions whether practitioners should be using the same review criteria as technical reviewers and should be participating in equal numbers on larger panels. (Apparently on panels of four persons or larger, the number of reviewers is evenly divided between practitioners and researchers.) Although there is no written government-wide definition of peer review, GAO points out there is concurrence among officials at research agencies that peer review is a “process that includes an independent assessment of the technical, scientific merit of research by peers who are scientists with knowledge and expertise equal to that of the researchers whose work they review” (U.S. General Accounting Office,

Suggested Citation:"4 Research Operations and Staffing Resources." National Research Council. 2010. Strengthening the National Institute of Justice. Washington, DC: The National Academies Press. doi: 10.17226/12929.
×

1999). The committee acknowledges the importance of soliciting input from practitioners as to the need and impact of the proposed research—but we suggest that such input can best be obtained as part of a two-step process or, if practitioners are included as part of the same panel meeting, they should be using different criteria to score the proposals. The practice of using practitioners on peer review panels is not mentioned in the formal guidelines and appears to have originated with NIJ directors. According to NIJ staff, some directors were more insistent on it and, in fact, one recent NIJ director believed that practitioners should not be excluded from serving as the lead technical reviewer, the person responsible for drafting the consensus review. According to staff, this was tried once or twice but proved unworkable and was abandoned.

Informing applicants of peer review results and providing them with an electronic copy of the consensus report or review summary is the responsibility of the OJP contractor after the NIJ staff person signs off on the consensus review.25 The OJP order setting forth this policy prohibits the release of any consensus report until successful applicants receive notification of their awards. Depending on the specific solicitation, NIJ advises applicants to expect funding decisions in 4 months (e.g., DNA backlog) to 9 months (e.g., Crime and Justice Research solicitation). There is seldom if any feedback provided directly to applicants on their applications. All applicants, whether their applications are accepted or rejected, are notified simultaneously. They receive electronic copies of the consensus review and not the individual reviews. This practice contrasts with that of the National Science Foundation (NSF) and the National Institutes of Health (NIH), which make available electronically to all applicants the individual reviews of their applications, along with a final decision from the program officers. This kind of transparency allows the applicant to see the full range of reviews as well as to understand more fully the basis for the funding decision; this kind of visibility may also result in better quality reviews.

The committee also learned that it is not unusual for an NSF program officer to discuss peer reviews informally with applicants whose applications will not be approved but appear promising, so they can benefit from the comments and resubmit for the next known funding cycle. NIH applicants above a certain score are also informed that they should come back and resubmit for possible funding. NIJ’s award cycle for all solicitations is on an annual basis, so there is no possibility that an application will be reconsidered in another year. Furthermore, a totally different panel reviews the next year’s applications and does not have access to the earlier reviews.

One final point regarding the OJP process bears mentioning. In talking with NIJ staff and others who have been NIJ applicants, the committee

25

E-mail from Angela Moore Parmley, acting director, ORE (October 22, 2009).

Suggested Citation:"4 Research Operations and Staffing Resources." National Research Council. 2010. Strengthening the National Institute of Justice. Washington, DC: The National Academies Press. doi: 10.17226/12929.
×

heard concerns about the standardized scoring and ranking system used by the OJP peer review process. It is not easily understood and does not appear to have a scientific basis—that is, how well scores obtained through this system reflect the overall quality of the proposal or the importance of the component parts. On one hand, the criteria are very simple and broad to accommodate any kind of proposal; on the other hand, research applications must provide very specific kinds of information that need to be assessed fully and weighted appropriately. The current scoring and ranking system does provide for different weightings of the criteria, but the committee could not determine how often this option is used or on what basis changes in weightings are made.

Although the committee is not prepared to make specific recommendations regarding the individual elements of peer review, we raise these issues to emphasize the importance of a research agency’s having the authority to manage its own peer review and assess its own process to determine whether changes are warranted. It then needs to have the authority to implement them. Currently, a centrally managed peer review process under OJP does not provide NIJ with the flexibility to make changes in line with its own research program needs.

Grant Award Process

As mentioned previously, the committee was unable to view any documents relating to the peer review or grant award process, including any review memoranda proposing certain applications for award. During the period of this study, four different persons served as the final arbiter of grant award decisions at NIJ, and it is possible that each one used different criteria and a different in-house process to make his or her decision.

An example of the decision-making process was provided by an NIJ director who explained that in the FY 2008 funding cycle, he lumped all the applications from several solicitations that were recommended for funding consideration into one large pool and selected those from the pool that he had sufficient funding for and felt were most appropriate for funding.26 Obviously, this kind of selection process sends a very mixed and confusing message to the research community, which is under the impression that it is competing against similarly themed research projects rather than against an entire pool of candidates. It also runs counter to the notion that a research institute’s practices should be fair, transparent, and consistent. NIJ’s grant award process should be followed for every solicitation, with full disclosure

26

Remarks by David Hagy, director, NIJ, to the Committee on Law and Justice (October 3, 2009).

Suggested Citation:"4 Research Operations and Staffing Resources." National Research Council. 2010. Strengthening the National Institute of Justice. Washington, DC: The National Academies Press. doi: 10.17226/12929.
×

of how the process works and the roles that NIJ staff, peer reviewers, and NIJ and the OJP leadership play.

NIJ’s peer-review and grant-selection processes were the focus of a recent audit by the Office of Inspector General of the Justice Department, which was requested by Senator Richard Shelby (R-AL). Released in September 2009, the audit report findings are highly critical of NIJ’s grant award practices. They cite NIJ for failing to maintain adequate preaward records, for not screening persons with potential conflict of interest from peer-review panels, for having inadequate documentation of the application review process, and for failing to ensure that potential applicants disclose lobbying activities. Most importantly, the report concludes that these problems raise concerns about the fairness and openness of NIJ’s competitive grant-making process. The report went on to recommend that NIJ take steps to establish policies and procedures for documenting key aspects of the preaward and award process (U.S. Department of Justice, 2009b).

Report Review

The NIJ report review process brings to full circle the research endeavor that began with agenda planning and priority setting. Grantees are required to submit a report on their research that provides a full description of the research, including the research questions, the methodology, and findings. NIJ heavily relies on final research reports to ascertain what has been accomplished and to pave the way for understanding the implications of the research for policy and practice. It relies on peer review of these reports for its assessment, in terms of both quality of the research and its utility. Peer review of ORE final reports generally consists of two outside reviews—one by a practitioner and one by a researcher, although occasionally only technical reviews will be requested. OST final reports generally receive three reviews by a combination of practitioners and technologists (refer to footnote 14 in this chapter). On the ORE side, there are separate review questions for practitioners and researchers. Most questions are common to both except that researchers are asked to comment on whether the findings are supported by the research and the appropriateness of methodology and practitioners are asked about the utility of the report.

The NIJ program monitor must also complete a review. Once the reviews are completed and a decision is made to recommend some form of dissemination other than archiving the report, the report is sent to the NIJ Editorial Review Board. A recommendation is then made to the NIJ director, who has the final approving authority. (See Chapter 5 for further discussion of NIJ’s dissemination efforts.)

The report review process is straightforward up to the point at which

Suggested Citation:"4 Research Operations and Staffing Resources." National Research Council. 2010. Strengthening the National Institute of Justice. Washington, DC: The National Academies Press. doi: 10.17226/12929.
×

a recommendation is made to the director. From that point on, depending on the recommendation, the director may become more or less involved in further discussions around the final production of the report and the selection of a specific kind of format. The committee heard from two NIJ directors that they were involved in editing publications. NIJ staff and former grantees also commented on the amount of time it took to produce a report once the actual peer-review process had been completed. However, the committee was not able to document the length of this process or to assess the rationale behind or the nature of the director’s involvement in the process. Here it suffices to say that different directors have had their own ideas as to their level of involvement in review and dissemination of reports.

The report review process can be a critical component of a research agency’s efforts to ensure that research of high quality is produced. It therefore deserves careful scrutiny in terms of the role the staff plays in assessing research reports, the quality of and utility of the peer reviews, the nature of the review itself, and the feedback that is provided to the grantee. The committee did not see actual reviews by peer-review consultants, but we were able to review the consultant task agreement that peer reviewers must sign. The committee notes the following areas of concern. Neither group receives background information on what was proposed, other than the report itself. The questions to the researcher are very general and do not get at such issues as data quality and utility. It is also unclear whether researcher and practitioner views are treated equally. In any event, such a review falls short of what would constitute a review by a peer-reviewed journal. The 1977 NRC report pointed out the inadequacy of a mail review to assess research findings (National Research Council, 1977). NIJ should assess its report review process in an effort to explore these issues. As with other research operations, NIJ needs to ask the questions: How can this process be used by NIJ to ensure quality in its research? Does this process contribute to NIJ’s stature as a science agency?

Availability of Final Reports

At a minimum, all final grantee reports are supposed to be stored with NCJRS for public access. The committee undertook an analysis to determine how many final reports are actually available at NCJRS. This was a tedious task. Neither NIJ nor NCJRS could provide the committee with records that adequately identified the number of grants producing final reports or whether these final reports have been received.

However, NCJRS did provide the committee with a list of all publications received from NIJ since 1995 linked to their grant numbers. We matched these grant numbers to those in our award archive database and

Suggested Citation:"4 Research Operations and Staffing Resources." National Research Council. 2010. Strengthening the National Institute of Justice. Washington, DC: The National Academies Press. doi: 10.17226/12929.
×

TABLE 4-1 Analysis of Final Research Reports

Fiscal Year

Number of Research and Evaluation Grantsa

Percent with a Final Report at NCJRS by December 2009

Year by Which All (or all but one) Currently Available Reports Had Been Submitted to NCJRS

1997

142

75

2004

1998

197

67.5

2004

2002

92

62

2005

2003

108

64

2009

NOTE: NCJRS = National Criminal Justice Reference Service.

aThis number is less than what is presented in Figure 3-2. For this analysis, we removed all the Interagency Agreements (because we were sure we didn’t have them all represented in our database) and all the awards continued from an earlier year.

considered the availability of final reports for 4 years: 1997, 1998, 2002, and 2003. We chose these years to allow enough time for a final report to be available and to compare years with different funding levels. We also considered only research grants (both social science and science and technology) and evaluation awards that had the potential to produce final research reports or equivalent articles in peer-reviewed journals. These grants had been awarded to a range of academic institutions, research foundations, and, to a lesser extent, local criminal justice agencies. We found the trends quite similar across each of the years. Our results are presented in Table 4-1.

The data show that 25 to 38 percent of the anticipated reports were not available in NCJRS and that this rate has increased in recent years. We were unable to determine whether missing reports were the result of grantees not turning in reports to NIJ or of NIJ not submitting final reports to NCJRS. Either way, we think NIJ should examine its final report submission processes in order to increase the percentage of final research reports that become publicly available. The average time of 3 years from grant award to availability of final report also needs consideration, especially when this average represents turnarounds from 1 to 7 years. It is noteworthy that NIJ was quite successful at short turnarounds with its 2002 grants. It is also interesting that its 1998 grants resulted in a higher number of missing reports in social science research areas. These trends can be understood only if the topics of specific grants and the context in which they were administered are examined, a task that was beyond the available resources of the committee.

Suggested Citation:"4 Research Operations and Staffing Resources." National Research Council. 2010. Strengthening the National Institute of Justice. Washington, DC: The National Academies Press. doi: 10.17226/12929.
×

Average Time from Year of Award to Year Submitted to NCJRS

Number of Grants Without a Report in NCJRS

Number of Those Missing a Report Considered Social Science Research

Number Considered Technology Development and/or Demonstration or Forensic Science Research

3.04

35

9

26

3.47

64

53

11

2.79

35

10

25

2.96

39

14

25

STAFFING RESOURCES

Dichotomy of Staff Responsibilities

Well-qualified and trained staff are important for good performance. It is also critical that they have a clear understanding of their roles and responsibilities. Similar to those of other science agencies that fund research, NIJ staff responsibilities fall into two broad and simplified categories: (1) knowledge development and (2) grant management. Knowledge development includes the activities involved in developing a research agenda and portfolio, such as identifying research needs, articulating research questions, identifying appropriate methodologies, preparing solicitations, providing input to the researcher on substantive issues, and assessing the quality of the research. Among the skills that are required to perform these tasks are substantive knowledge of a research area and research methodology. Because NIJ is a funding agency, program staff are also required to oversee the grant application and the review and award processes and, once grants are funded, to ensure that grantees comply with administrative and programmatic requirements. These kinds of activities require that staff have administrative and communication skills as well as relevant substantive knowledge of the work to be performed. There is considerable overlap between the two kinds of responsibilities required by program staff throughout the research grant process. By no means are they mutually exclusive. From management’s vantage point, the ideal staff person will have a mix of skills and will handle both kinds of responsibilities equally well.

Tension arises when responsibility for developing and implementing a research portfolio becomes overwhelmed or supplanted by grant adminis-

Suggested Citation:"4 Research Operations and Staffing Resources." National Research Council. 2010. Strengthening the National Institute of Justice. Washington, DC: The National Academies Press. doi: 10.17226/12929.
×

trative responsibilities. To attract and retain staff with substantive expertise who have a primary interest in knowledge development, a research institute will need to offer a different set of responsibilities and rewards than it will need to attract persons who see themselves as good grant managers and communicators. An appropriate balance between the two kinds of roles needs to be maintained if staff who have a primary interest in knowledge development are not to be turned off by what they perceive as unnecessarily time-consuming responsibilities that are tangential to their main interests and expertise.

This dichotomy constantly asserted itself as a backdrop to the committee’s review of staffing levels, staff characteristics, and organizational culture. It was very much reflected in a September 2007 NIJ staff survey conducted in-house by NIJ (results of the survey presented at the NIJ forum, January 10, 2008) as well as interviews with former and current NIJ staff conducted for this study. NIJ’s staffing resources are described in the following sections and serve as an important context for this report.27

Staff Size Compared with Budget

NIJ’s level of staffing resources has shifted dramatically in the past 10 years. From FY 1994 to FY 2000, the size of NIJ’s staff mirrored the rise and fall of its budget. But when one looks at the individual program offices—ORE and OST—the size of these offices has not necessarily reflected the rise and fall of their individual budgets (see Figures 4-1 and 4-2). When one compares budget size with staffing resources for ORE and OST, the staffing of OST has remained relatively constant during a period of great influx of funding, while the staffing of ORE has continued to

27

Information in this section has been pieced together from various sources. We relied heavily on information for the period 1994-2008 provided by the National Finance Center (NFC), the federal employee records center, which listed the names of individual staff, their position titles, and grade levels. In December 2007, ORE gave the committee a breakdown of numbers of full-time equivalents by year for the period 1994-2007, but comparable information was not available from OST. The ORE staffing information was relied on only for trend analysis. We were also given three signed NIJ organizational charts listing organizational units, names of staff (for one of the years), position titles, grade levels, and office designations. Two of these charts listed vacancies. Information on NIJ losses by separation or transfer was provided by the OJP Human Resources Division and covered the period from January 1, 2002, through November 24, 2007. We also conducted interviews with 26 current and former staff and were provided with a summary of findings from a survey of 45 nonsupervisory NIJ staff conducted in October 2007.

Suggested Citation:"4 Research Operations and Staffing Resources." National Research Council. 2010. Strengthening the National Institute of Justice. Washington, DC: The National Academies Press. doi: 10.17226/12929.
×
FIGURE 4-1 ORE employees and budget (in constant 2008 dollars).

FIGURE 4-1 ORE employees and budget (in constant 2008 dollars).

SOURCE: Adapted from figures sent from National Finance Center (for staff totals) and National Institute of Justice (for budget totals).

FIGURE 4-2 OST employees and budget (in constant 2008 dollars).

FIGURE 4-2 OST employees and budget (in constant 2008 dollars).

SOURCE: Adapted from figures sent from National Finance Center (for staff totals) and National Institute of Justice (for budget totals).

Suggested Citation:"4 Research Operations and Staffing Resources." National Research Council. 2010. Strengthening the National Institute of Justice. Washington, DC: The National Academies Press. doi: 10.17226/12929.
×
FIGURE 4-3 Number of ORE and OST employees for budget years 1995, 1998, 2001, 2004, and 2008 (in constant 2008 dollars).

FIGURE 4-3 Number of ORE and OST employees for budget years 1995, 1998, 2001, 2004, and 2008 (in constant 2008 dollars).

SOURCE: Adapted from figures sent from National Finance Center (for budget totals) and National Institute of Justice (for staff totals).

decline, whether funds allocated to this office have increased or decreased (see Figure 4-3).28

Trends in Staffing

In FY 1994, the NIJ staff consisted of approximately 40 people who oversaw a total budget of $33 million (National Institute of Justice, 1994b). Of these, approximately 18 were professional staff handling nonadministrative research grant responsibilities. After the passage of the Crime Act in 1994, a large influx of transferred funds for evaluation and other specific

28

These figures are approximate, since it is difficult to compare individual office staffing levels over time. For example, to obtain an accurate picture of staffing for OST and ORE in 2008 in comparison to the numbers provided for 1998 and 2001, one would need to count people assigned to the Planning, Budgeting, Management, and Administration Division (in the Office of the Director), who work with OST and ORE on administrative matters. As described earlier, we have relied on annual lists of NIJ employees compiled by the NFC for much of this analysis. Because we do not know whether the NFC lists of NIJ employees were compiled at the same time each year, there could be considerable variation in total numbers for any given year, depending on when people were hired or resigned.

Suggested Citation:"4 Research Operations and Staffing Resources." National Research Council. 2010. Strengthening the National Institute of Justice. Washington, DC: The National Academies Press. doi: 10.17226/12929.
×

research purposes came into NIJ. Beginning in 1995, the NIJ director recruited heavily, adding approximately 21 new program staff in 1995, 12 in 1996, and 16 in 1997. By 1998, the size of the NIJ staff had more than doubled (from 44 to 101) and the size of its total program budget had increased sixfold to $176 million. By 1999, the number of NIJ staff had reached its highest level of 111.29

Within two years and a changed administration, the size of staff had begun to shrink dramatically, even as funding increased as a result of the President’s DNA Initiative. Between the years of 2002 and 2003, NIJ experienced its biggest turnover, with 18 staff departing. This decrease continued (except in 2005) until 2007, when approximately 58 people were identified as NIJ employees. Since 2008, the size of NIJ’s staff has increased slightly.

Use of Contractors

In addition to the individuals listed in the staff rolls, NIJ relies on contractors to meet its responsibilities. Their involvement with various grant-related matters was mentioned frequently during site visits with OST grantees. Limited information was available on the extent to which contractors were used and the nature and scope of their responsibilities.30 We learned that contractors have been heavily used since 2000 by OST for a variety of technology and research-related tasks, in addition to grant-processing activities, and to a lesser degree by other offices at NIJ for primarily administrative and grant-processing activities. The committee’s strong sense is that NIJ relies on contractors for a level of experience and expertise that is not available from its staff.

For example, in 2001, NIJ oversaw the work of 45 contractors as part of what it called its Systems Engineering and Technical Assistance (SETA) staff. Approximately 25 of these were assigned to OST divisions and projects, and another 8 worked somewhere in NIJ, 3 were assigned to work offsite on National Institute of Standards and Technology activities funded

29

These numbers do not include other nonfederal employees working at NIJ, who would have been hired under the Intergovernmental Personnel Act Mobility Program. This program provides for the temporary assignment of personnel between the federal government and state and local governments, colleges and universities, Indian tribal governments, federally funded research and development centers, and other eligible organizations. These individuals serve as visiting scientists and in some cases have managed specific programs, such as NIJ’s International Center and OST’s forensic science activities. A typical term of appointment is 1-3 years, and, unlike contractors, they can participate fully in all aspects of NIJ’s work. OST has had as many as three visiting scientists on board for most of the years described.

30

Presentation by David Boyd, deputy director, OST, to Glen Schmitt, deputy director, NIJ (August 2, 2001).

Suggested Citation:"4 Research Operations and Staffing Resources." National Research Council. 2010. Strengthening the National Institute of Justice. Washington, DC: The National Academies Press. doi: 10.17226/12929.
×

by NIJ, and 3 were assigned to the Bureau of Justice Assistance. NIJ continues to make use of SETA contractors to provide scientific and analytic services; they are used to create an expanded work capacity and quick turnaround to meet immediate program requirements. They also free up program managers, who can then respond quickly and proactively to new developments. SETA contractors are subject to certain restrictions.

Continued reliance on contractors may weaken NIJ in the short and long term. Contractors are not treated like federal employees. They cannot speak for NIJ at meetings or conferences and are not allowed to access certain types of information (e.g., the electronic grant management information system). This restriction was noted by NIJ grantees, who complained that the SETA contractors with whom they consulted were limited in their ability to handle certain grant monitoring functions. As a result, the needs of researchers may not be getting met. Also when contracts expire and are not renewed, the resulting turnover may be detrimental to NIJ’s stability. In FY 2008, 75 percent of the contractors supported the Forensic Science Division, the one that handles approximately 75 percent of the NIJ budget.31

Staff Turnover and Vacancies

As indicated above, the past decade has been one of staff growth and decline, with very high turnover occurring at all levels of the organization. Looking at the period 1998-2008—only 21 of the original 111 NIJ staff from 1998 remained in 2008.32 Of these 21, 15 individuals continued to have responsibility for research-related activities. Comparing two points in time—2002 and 2008—63 individuals (out of 93) who appeared on the 2002 rolls were gone by 2008, and 41 new staff were hired or transferred to NIJ in the interim and on board in 2008. According to a different source, information on NIJ staffing reflects that, during a 5-year period from January 1, 2002, to November 24, 2007, approximately 76 persons separated or transferred out of NIJ, many of whom held management positions.33 The 76 included 12 division director positions distributed among 9 different divisions, 2 deputy director positions (OST and ORE), an NIJ director, and an NIJ deputy director.34

Turnover is to be expected during changes of administration. But personal loyalties do not explain such a great exodus by career staff, many of

31

E-mail from George Tillery, acting director, OST (June 27, 2008).

32

This figure is based on data from the NFC on NIJ staff for the years 1994-2008.

33

The list of OJP losses by separation or transfer was provided by the OJP Office of Audit, Assessment, and Management.

34

During our own brief tenure, the committee witnessed the resignation or transfer of one NIJ director, one deputy director, one ORE director, and one OST director. Several division chiefs have also been shifted within NIJ to vacated management positions.

Suggested Citation:"4 Research Operations and Staffing Resources." National Research Council. 2010. Strengthening the National Institute of Justice. Washington, DC: The National Academies Press. doi: 10.17226/12929.
×

whom took lateral positions at other agencies or research organizations. Of the 15 former staff whom the committee interviewed, all had left NIJ during the period 1998-2006. Among the reasons they cited for leaving were lack of leadership, lack of opportunity to use their methodological expertise, and lack of a culture in which substantive expertise was highly valued and could be nurtured (see the discussion on intramural research later in this chapter).

A presidential initiative intended to restructure the federal workforce and streamline federal agency operations may also have affected staff turnover at NIJ. Known as the A-76 strategy, after the OMB Circular A-76 for which it was named (U.S. Office of Management and Budget, 1983), it involved classifying positions as “inherently governmental” or “commercial” and opening the latter to competition. The process essentially requires that private companies and in-house personnel engaged in winner-take-all competitions for the tasks in hand. Despite numerous arguments being made by OJP offices as to certain positions being intimately related to the public interest and as such must be performed by government employees, by February 2003, 79 of 98 positions at NIJ were deemed to be “commercial” and therefore open to competition. The NIJ positions included 61 of 79 research grant-monitoring positions, as well as administrative support and information technology jobs (Consortium of Social Science Associations, 2003). In September 2004, the procurement process was initiated, and, for the next 2 years, many staff believed their jobs were in jeopardy pending the outcome of the competition. In September 2006, the Department of Justice cancelled the OJP procurement, after the congressional appropriators directed that any A-76 action related to OJP needed to be brought to them in advance and special justification submitted for any program changes that reduced the personnel of an agency by 10 percent or more (U.S. House of Representatives, 2005). But by then many senior staff had departed.

It is obviously difficult to project whether or not such a high rate of turnover will continue and what the impact might be. However, one possible indicator of future turnover among senior staff is the number of staff eligible for retirement. As of April 30, 2008, NIJ had 12 employees eligible for retirement, 7 at the grade service (GS)-14/15 level and 1 at the executive-service level.

Contrasted with these numbers is NIJ’s authorized staffing allocation of 87. On February 14, 2008, 21 NIJ positions remained vacant, almost one-fourth of its full-time equivalent positions.35 These vacancies included seven

35

The committee obtained OJP-approved NIJ organization charts dated September 28, 2006; November 21, 2007; and February 14, 2008. Each showed an approved allocation of 87. Information for earlier years was unavailable. Only the organization chart dated February 14, 2008, showed vacancies.

Suggested Citation:"4 Research Operations and Staffing Resources." National Research Council. 2010. Strengthening the National Institute of Justice. Washington, DC: The National Academies Press. doi: 10.17226/12929.
×

supervisory positions (either a chief or deputy chief of every ORE division, chief of the Planning, Budget, Management, and Administration Division, and chief of one OST division). In addition, there were two senior-level social science analyst positions at the GS-14/15 level and a physical scientist at the GS-14/15 level. Prior to filling a vacancy, the NIJ director must seek approval from the assistant attorney general, OJP, through the Human Resources Division, Office of Administration. NIJ managers informed us that they were not allowed to move ahead with vacancies. It is possible that some vacancies were advertised but not filled, and we know of one position in the Office of the Director for which this was true. The committee surmises that the sheer number of vacancies reflects NIJ’s inability to obtain approval to move ahead and do what is necessary to fill them.

There was consensus among the former and current staff who spoke to us that NIJ is understaffed. While many factors affect how many staff a specific program will require, the committee is unaware of any formal standards by which to judge the staffing levels of federal agencies. But agencies do use some kind of yardstick to determine approved staffing allocations; given budget constraints, one can assume that these allocations will be based on conservative estimates of what is needed. Certainly for the past three years, based on the approved allocation and the number of positions that remain unfilled, NIJ appears to be severely understaffed.

Staff Qualifications

Education Levels

Given that NIJ staff are responsible for developing and managing a research portfolio, the committee was interested in learning about the staff’s educational training and whether it reflects substantive knowledge of relevant science as well as training in research methodologies. We were also interested in understanding the responses to our survey of researchers and practitioners, in which only 57 percent of the respondents were satisfied with the qualifications of staff and researchers were less satisfied than practitioners (53 versus 64 percent). (See Appendix B for a discussion of the survey and its results.)

Information on individuals’ educational attainment proved almost impossible to obtain. The committee was informed that educational degrees are considered private information. The committee notes that this is not the case at selective NIH institutes that provide full staff listings including bios and degrees on their web pages.36 The committee was provided with

Suggested Citation:"4 Research Operations and Staffing Resources." National Research Council. 2010. Strengthening the National Institute of Justice. Washington, DC: The National Academies Press. doi: 10.17226/12929.
×

a breakdown of educational attainment by division, which was compiled by NIJ from information provided by NFC.37 Although questions remain about the data, one can conclude that the proportionate number of programmatic staff with advanced degrees (doctorates and postdoctorates) has declined since 1995 in both ORE and OST, most dramatically in OST. In 2008, only 3 persons out of 20 with programmatic responsibilities in OST had doctorate degrees or postdoctoral training. In ORE 9 persons out of 23 had advanced degrees, a lower percentage than in 1995 or 2001. The rise in the number with advanced degrees in ORE in 2001 dovetails with the large numbers of staff recruited during the period 1995-2000. Given the high staff attrition after 2001, it is not surprising that the number of staff with advanced science training also declined.

The committee recognizes that educational attainment provides a very limited view of staff qualifications, particularly since we were unable to obtain information on the fields in which degrees were granted. Nevertheless, a research agency needs to be staffed by persons with strong academic and research credentials. Experience and freshness of that experience are also important. One specific program that several committee members reviewed and discussed with NIJ staff was the centers of excellence. On the basis of discussions with NIJ staff, review of the applicants’ proposals, and meetings with center staff, the committee questions whether OST staff have the technical experience and expertise to evaluate the applicants’ qualifications or the work of the centers.

Not all jobs or all responsibilities will require the kind of academic training that a doctorate or postdoctorate signifies. However, a critical mass is needed of staff with advanced training in the sciences that can be called on to provide the kind of scientific advice and guidance required and to signal to the research community that NIJ staff fully understand the nature of the research enterprise. One strategy for attracting and retaining this kind of expertise is to offer opportunities to conduct intramural research. The committee notes that other science agencies, including NIH and the National Institute of Standards and Technology, have staff actively engaged in conducting research. In the 1970s and 1980s, NIJ had a division specifically devoted to intramural research. But the function was eliminated at some point during an NIJ reorganization. In 1994, an

37

NIJ worked from NFC lists of staff that divided staff education levels into 22 codes. NIJ staff worked with OJP’s Office of the General Counsel to group those codes into broader categories because of concern that if the cells were too small, individual staff might be identified. NIJ also eliminated staff that were at the GS-7 level or lower and staff at the GS-9 or GS-10 level, as well as anyone who was a “technician” of some sort. People who were budget analysts or program analysts in high grades remained. No further clarification from NIJ was forthcoming regarding the specific codes and the kinds of degrees included in them. Since we were not allowed to see the raw data, we were unable to verify the information.

Suggested Citation:"4 Research Operations and Staffing Resources." National Research Council. 2010. Strengthening the National Institute of Justice. Washington, DC: The National Academies Press. doi: 10.17226/12929.
×

intramural research program was resurrected, and staff members with the interest and qualifications to conduct intramural research were specifically recruited. By 1996, 16 NIJ staff members were identified as authors of papers and reports emanating from the intramural research program (National Institute of Justice, 1996). Since then, some staff members have continued to engage in intramural research, but this activity is not offered as a selling point to attract qualified staff, nor does there appear to be any incentives for it. NIJ should consider the role of a formal intramural research program as a way to attract talented criminologists who can bring new energy and expertise to its organization. Such a program can also serve as a strong incentive to retain staff who are well qualified to manage the extramural research.

Positions and Grade Levels

Most civil service employees hold a position that is part of a job series and a grade level. Exceptions to the broad generalizations here include among others the military, commissioned officers of the Public Health Service, Bureau of Veteran Affairs medical personnel, “special government employees” who serve on committees and in other ways, persons hired through the Intergovernmental Personnel Act. Each job series contains one or more position descriptions, and each position carries with it a set of requisite skills. The grade level determines the rate of pay and amount of authority. The lowest level is GS-1 (an abbreviation for grade service-1) All jobs at GS-7 or above require some form of specialized training, by either government or the private sector. Those entering the job market who have master’s degrees may qualify for GS-9 and those with a doctorate may qualify for a GS-11 position.

The majority of staff in ORE fall under the 101 job series that includes social science analysts or social science program specialists. Supervisors and persons with lead responsibility have the title of “supervisory social science analyst” or “lead social science analyst.” This was the prevailing category in 1998 as well as in 2008. Typically NIJ staff are recruited for these positions at the GS-12 or GS-13 level and compete for a GS-14 level position when one is advertised, but this occurs infrequently and is in fact a source of great frustration to the career staff.

OST, however, has undergone a major change in the various job series of its staff. In 1998, OST had six staff members who held GS-14 and GS-15 positions.38 Of these, four were classified as operational research analysts and fell under the 1515 series, which emphasizes having “competence in rigorous methods of scientific inquiry and analysis” and

38

Supervisory positions can be at either at the GS-14 or the GS-15 level.

Suggested Citation:"4 Research Operations and Staffing Resources." National Research Council. 2010. Strengthening the National Institute of Justice. Washington, DC: The National Academies Press. doi: 10.17226/12929.
×

is geared toward those with an operations research education.39 In 2008, eight OST staff persons were classified as GS-14 and GS-15. Only one of these is classified as a 1515 operational research analyst position; four are in the 1301 physical scientists series, one is in the 801 general engineering series, and two fall into the administrative and management 340 series. The physical scientist 1301 series is a catchall series and includes work in the physical sciences that is either not classified elsewhere or that combines physical science fields, with no single one predominant.40 It does not include a research competency qualification. Although it is difficult to say with certainty, without knowing more about people’s backgrounds and training, it does appear that there has been a shift in the staffing of OST from persons with greater research expertise to persons with more generalized science and technology knowledge or administrative experience.

It is difficult to draw exact comparisons with other federal research agencies. Both NSF and NIH have the authority under United States Code Title 42 to appoint scientific positions without regard to the provisions of Title 5, which governs federal appointments. This allows the directors considerable discretion in setting requirements for scientific positions and salaries. Typically, the health scientist administrator position at NIH, a position that most closely approximates the responsibilities of the social science analyst position at NIJ, is targeted at the GS-13 and GS-14 level and specifies a Ph.D. education level; the director has the ability to hire at the GS-14 level. The net result is that NSF and NIH have much more flexibility in recruiting specifically for persons with advanced degrees and scientific skills and in setting pay levels higher than what could be typically done by NIJ.

Organizational Culture

Staff resources serve as the foundation of any organization and can be quantified to provide a rough picture of how well equipped an organization is to carry out its mission. There are also many other nonquantifiable elements that can affect the quality of the workplace and the work that is performed, and organizational culture is certainly one of them. Although it is difficult to document the psychology, experiences, and values that comprise organizational culture, a fairly consistent picture emerged from our interviews with former and current NIJ staff as well as from a summary of findings from an in-house survey of NIJ’s staff conducted in September

Suggested Citation:"4 Research Operations and Staffing Resources." National Research Council. 2010. Strengthening the National Institute of Justice. Washington, DC: The National Academies Press. doi: 10.17226/12929.
×

2007. Taken together, these reflect an organization in which there is considerable confusion and disagreement as to the roles and responsibilities of the top leadership, the mid-level managers, and line staff. With few exceptions, the staff faults the failure of top leadership to provide direction and sufficient responsibility to the staff. Mid-level managers are viewed as undeserving of their salaries when others are doing the “real” work of the agency. Finally, the program or line staff most heavily represented in the interviews and surveys expressed very different opinions as to what aspects of their work are or should be most important. According to the NIJ in-house survey summary of 35 current employees, respondents are described as falling into two groups: (1) those who believe program administration is at the heart of their work and most important and (2) those who believe that the substantive work of knowledge development, dissemination, and other service to the criminal justice field is most important.

In contrast, former and current NIJ staff persons who were interviewed did not express views that would identify themselves as persons strongly committed to either knowledge development or to program administration. These interviewees assigned great importance to the need for strong substantive and communication skills and the need to staff the agency with persons with strong subject-matter expertise and R&D (particularly on the technology side) experience. One plausible explanation for this difference may be that, of the 26 interviewees, 15 were former staff and 11 were current staff, and the median length of service for both groups was fairly long at 9 years for former staff and 7 for current staff. None mentioned a similar kind of conflict, nor did anyone describe day-to-day activities in the same way.

One area in which those interviewed and surveyed appeared to be in agreement was that attention to the mundane operations exceeded that spent on substantive issues. Examples of such unrelated research tasks include resolving issues of language with the NIJ website, estimating quarterly travel expenses, responding to short turnaround times on reviews of OJP documents, and resolving personnel issues.

In any assessment of an agency that involves interviews with former and current staff, one expects to hear criticism, but the extent and frequency of the critical comments went well beyond what would be considered normal complaints about a bureaucracy. Taken together, these views reflect an organization that is urgently in need of strong leadership that can resolve many staffing issues that prevent it from having a more positive work environment.

Suggested Citation:"4 Research Operations and Staffing Resources." National Research Council. 2010. Strengthening the National Institute of Justice. Washington, DC: The National Academies Press. doi: 10.17226/12929.
×

CONCLUSIONS

Three conclusions emerge from the committee’s analysis of NIJ research operations and staffing resources.

  • Researchers currently play a limited role in NIJ’s research operations.

In reviewing NIJ’s research operations, the committee was struck by the limited role that researchers currently play in the planning, peer-review, and report review processes. NIJ has never used an advisory infrastructure to help with planning or to provide feedback in a focused and consistent way. The lack of researcher involvement in current planning processes is particularly evident in the technology research as well in the social science research areas. Even when the advice of researchers is solicited, as it is with peer review of grant applications and report reviews, the scientific judgments of researchers do not take precedence over the judgments of practitioners, who are also being asked to assess scientific quality and to numerically rank applications using the same criteria. In the case of peer-reviewed reports, it is not clear whether scientific merit trumps other criteria, such as relevance and utility, and, unlike peer review of applications, practitioners and researchers are not brought together to discuss their concerns and to arrive at a consensus.

The committee is not aware of any science agency in which the research community is less involved in providing guidance and feedback or in which practitioners play a coequal role in assessing research proposals and research reports. One possible reason for this is the perceived need, on the part of NIJ or of OJP, to equalize the contributions made by practitioners and researchers in order to ensure that NIJ research is policy relevant. Whatever the reason, NIJ appears to have devalued the contribution of researchers and distanced itself from the research community, even to the extent that the term “researcher” is not used to describe the participants in the peer-review process.41 This distancing is also reflected in NIJ’s organizational culture, in which contact with researchers other than grantees appears very limited or avoided, on the grounds that such contact may create a competitive advantage or encourage an “old boy” network.

41

“Proposals received under a solicitation are reviewed by independent peer panels, comprised of technologists (emphasis added) from academia, industry, and government organizations, along with practitioners from Federal, State and local agencies. Once reviewers have completed evaluations, NIJ Program Managers recommend individual proposals to the NIJ Director, who makes final award decisions” (from the NIJ website at http://www.ojp.usdoj.gov/nij/funding/proposal-review.htm [last updated November 15, 2007] [accessed December 10, 2009]).

Suggested Citation:"4 Research Operations and Staffing Resources." National Research Council. 2010. Strengthening the National Institute of Justice. Washington, DC: The National Academies Press. doi: 10.17226/12929.
×
  • There is a lack of transparency and consistency regarding NIJ’s research operations.

Nothing affects the stature of a research institute more than a lack of transparency as to how funding priorities are arrived at and funding decisions are made. Because the grant award process is a highly competitive one and researchers must invest considerable time and effort in preparing applications, researchers will not seek support from an institution they believe is arbitrary in its decision making. The committee heard firsthand from the NIJ leadership that the grant award process was altered from one in which individual applications on a particular topic competed against each other to one in which all applications for numerous solicitations were grouped together and considered for the same pool of funds. The change in the grant award process was not announced to applicants prior to their submitting an application, nor were NIJ staff aware that this might occur. Not being clear on how decisions are made or, worse, changing the rules in the middle of the game, has serious implications for researchers’ perceptions of the fairness and consistency of NIJ’s practices. Once these perceptions are formed, they are difficult to change.

The committee also encountered considerable difficulty in obtaining documentation for NIJ’s research operations as well as other kinds of administrative information. While we note that most programmatic as well as administrative information is maintained by OJP, there are steps NIJ can take to document its research processes, improve its record-keeping, and become a more transparent operation.

One of these is to improve access to data through the NIJ website. Despite numerous improvements that have been made during the course of our study, the committee found it difficult to determine whether research grants had resulted in available final reports or to trace titles of existing reports or publications back to grants. Many websites of research agencies42 have a searchable online database of research awards that allows users to search for awards by topic, principal investigator, or grantee institution. In addition, these agencies provide funding amounts, names of principal investigators, and abstracts for each award. NIJ does have an archive list of awards dating back to 1995 online that are sorted by topic and, in recent years, by solicitation as well; funding amounts and grantee institution are provided for each award listed. However, names of principal investigators ceased to be included after 2003, there are no abstracts or descriptions available for

42

See, for example, the Institute for Education Sciences at http://ies.ed.gov/funding/grantsearch/index.asp and NSF at http://www.nsf.gov/awardsearch/ [accessed July 28, 2009].

Suggested Citation:"4 Research Operations and Staffing Resources." National Research Council. 2010. Strengthening the National Institute of Justice. Washington, DC: The National Academies Press. doi: 10.17226/12929.
×

the awards, and there is no mechanism to distinguish research grants from other types of capacity-building or technology assistance grants.

Recognizing the importance of transparency and conducting a thorough assessment of its research operations, financial, and administrative records to determine how it might be achieved are important steps for NIJ to take.

  • OJP’s oversight of various aspects of NIJ’s research operations and administrative functions is incompatible with NIJ’s role as an independent research organization.

The committee identified at least two research operations in which OJP plays a heavy hand: the approval of solicitations and the peer-review process. In the case of the solicitation review process, OJP appears to have created a lengthy and convoluted process in which NIJ solicitations undergo extensive review by numerous OJP offices, and the focus of the review appears to go beyond what are strictly legal or financial issues. The centralization of peer review has also resulted in several requirements that may not be best suited for a research organization. All solicitations are not equal, and NIJ needs the authority to assess the peer review process in light of its own needs and to make changes it deems necessary. Under the current system, this is difficult to do.

NIJ’s staffing resources have also been greatly impacted by OJP’s oversight and control. The analysis of NIJ’s staffing resources paints a picture of an organization that has experienced great variations in staffing size and composition caused by high turnover during the period 1994-2008. For many of these years, NIJ has been badly understaffed because it was prevented by the OJP leadership from recruiting against its approved staffing allocation. Management positions have been particularly affected, and with two or three exceptions there is almost no one currently in a management position who held that same position at NIJ as recently as two years ago. There also appears to be a shift from NIJ staff with strong science backgrounds to those with a more generalized knowledge of technology and a continued reliance on contractors to compensate for this lack of research experience and technology expertise. While it is difficult to show cause and effect, the committee thinks that these changes have taken a great toll on NIJ’s current staff, as reflected by its current organizational culture. NIJ staff interviews also reflect frustration with the lack of administrative support, and the nonresearch-related demands growing out of OJP oversight.

While remaining very understaffed, NIJ has a cadre of experienced people who expressed a strong desire to produce high-quality research and to be of service to the criminal justice field. For this to occur, NIJ needs control over its own operations, needs to undertake a thorough self-examination

Suggested Citation:"4 Research Operations and Staffing Resources." National Research Council. 2010. Strengthening the National Institute of Justice. Washington, DC: The National Academies Press. doi: 10.17226/12929.
×

in an effort to improve its operations and become more transparent, and needs the authority and capacity to make the changes. Key among these is for NIJ to ensure the proper role of science in its decisions and to involve the research community in a more meaningful way in accomplishing its mission.

Suggested Citation:"4 Research Operations and Staffing Resources." National Research Council. 2010. Strengthening the National Institute of Justice. Washington, DC: The National Academies Press. doi: 10.17226/12929.
×
Page 105
Suggested Citation:"4 Research Operations and Staffing Resources." National Research Council. 2010. Strengthening the National Institute of Justice. Washington, DC: The National Academies Press. doi: 10.17226/12929.
×
Page 106
Suggested Citation:"4 Research Operations and Staffing Resources." National Research Council. 2010. Strengthening the National Institute of Justice. Washington, DC: The National Academies Press. doi: 10.17226/12929.
×
Page 107
Suggested Citation:"4 Research Operations and Staffing Resources." National Research Council. 2010. Strengthening the National Institute of Justice. Washington, DC: The National Academies Press. doi: 10.17226/12929.
×
Page 108
Suggested Citation:"4 Research Operations and Staffing Resources." National Research Council. 2010. Strengthening the National Institute of Justice. Washington, DC: The National Academies Press. doi: 10.17226/12929.
×
Page 109
Suggested Citation:"4 Research Operations and Staffing Resources." National Research Council. 2010. Strengthening the National Institute of Justice. Washington, DC: The National Academies Press. doi: 10.17226/12929.
×
Page 110
Suggested Citation:"4 Research Operations and Staffing Resources." National Research Council. 2010. Strengthening the National Institute of Justice. Washington, DC: The National Academies Press. doi: 10.17226/12929.
×
Page 111
Suggested Citation:"4 Research Operations and Staffing Resources." National Research Council. 2010. Strengthening the National Institute of Justice. Washington, DC: The National Academies Press. doi: 10.17226/12929.
×
Page 112
Suggested Citation:"4 Research Operations and Staffing Resources." National Research Council. 2010. Strengthening the National Institute of Justice. Washington, DC: The National Academies Press. doi: 10.17226/12929.
×
Page 113
Suggested Citation:"4 Research Operations and Staffing Resources." National Research Council. 2010. Strengthening the National Institute of Justice. Washington, DC: The National Academies Press. doi: 10.17226/12929.
×
Page 114
Suggested Citation:"4 Research Operations and Staffing Resources." National Research Council. 2010. Strengthening the National Institute of Justice. Washington, DC: The National Academies Press. doi: 10.17226/12929.
×
Page 115
Suggested Citation:"4 Research Operations and Staffing Resources." National Research Council. 2010. Strengthening the National Institute of Justice. Washington, DC: The National Academies Press. doi: 10.17226/12929.
×
Page 116
Suggested Citation:"4 Research Operations and Staffing Resources." National Research Council. 2010. Strengthening the National Institute of Justice. Washington, DC: The National Academies Press. doi: 10.17226/12929.
×
Page 117
Suggested Citation:"4 Research Operations and Staffing Resources." National Research Council. 2010. Strengthening the National Institute of Justice. Washington, DC: The National Academies Press. doi: 10.17226/12929.
×
Page 118
Suggested Citation:"4 Research Operations and Staffing Resources." National Research Council. 2010. Strengthening the National Institute of Justice. Washington, DC: The National Academies Press. doi: 10.17226/12929.
×
Page 119
Suggested Citation:"4 Research Operations and Staffing Resources." National Research Council. 2010. Strengthening the National Institute of Justice. Washington, DC: The National Academies Press. doi: 10.17226/12929.
×
Page 120
Suggested Citation:"4 Research Operations and Staffing Resources." National Research Council. 2010. Strengthening the National Institute of Justice. Washington, DC: The National Academies Press. doi: 10.17226/12929.
×
Page 121
Suggested Citation:"4 Research Operations and Staffing Resources." National Research Council. 2010. Strengthening the National Institute of Justice. Washington, DC: The National Academies Press. doi: 10.17226/12929.
×
Page 122
Suggested Citation:"4 Research Operations and Staffing Resources." National Research Council. 2010. Strengthening the National Institute of Justice. Washington, DC: The National Academies Press. doi: 10.17226/12929.
×
Page 123
Suggested Citation:"4 Research Operations and Staffing Resources." National Research Council. 2010. Strengthening the National Institute of Justice. Washington, DC: The National Academies Press. doi: 10.17226/12929.
×
Page 124
Suggested Citation:"4 Research Operations and Staffing Resources." National Research Council. 2010. Strengthening the National Institute of Justice. Washington, DC: The National Academies Press. doi: 10.17226/12929.
×
Page 125
Suggested Citation:"4 Research Operations and Staffing Resources." National Research Council. 2010. Strengthening the National Institute of Justice. Washington, DC: The National Academies Press. doi: 10.17226/12929.
×
Page 126
Suggested Citation:"4 Research Operations and Staffing Resources." National Research Council. 2010. Strengthening the National Institute of Justice. Washington, DC: The National Academies Press. doi: 10.17226/12929.
×
Page 127
Suggested Citation:"4 Research Operations and Staffing Resources." National Research Council. 2010. Strengthening the National Institute of Justice. Washington, DC: The National Academies Press. doi: 10.17226/12929.
×
Page 128
Suggested Citation:"4 Research Operations and Staffing Resources." National Research Council. 2010. Strengthening the National Institute of Justice. Washington, DC: The National Academies Press. doi: 10.17226/12929.
×
Page 129
Suggested Citation:"4 Research Operations and Staffing Resources." National Research Council. 2010. Strengthening the National Institute of Justice. Washington, DC: The National Academies Press. doi: 10.17226/12929.
×
Page 130
Suggested Citation:"4 Research Operations and Staffing Resources." National Research Council. 2010. Strengthening the National Institute of Justice. Washington, DC: The National Academies Press. doi: 10.17226/12929.
×
Page 131
Suggested Citation:"4 Research Operations and Staffing Resources." National Research Council. 2010. Strengthening the National Institute of Justice. Washington, DC: The National Academies Press. doi: 10.17226/12929.
×
Page 132
Suggested Citation:"4 Research Operations and Staffing Resources." National Research Council. 2010. Strengthening the National Institute of Justice. Washington, DC: The National Academies Press. doi: 10.17226/12929.
×
Page 133
Suggested Citation:"4 Research Operations and Staffing Resources." National Research Council. 2010. Strengthening the National Institute of Justice. Washington, DC: The National Academies Press. doi: 10.17226/12929.
×
Page 134
Suggested Citation:"4 Research Operations and Staffing Resources." National Research Council. 2010. Strengthening the National Institute of Justice. Washington, DC: The National Academies Press. doi: 10.17226/12929.
×
Page 135
Suggested Citation:"4 Research Operations and Staffing Resources." National Research Council. 2010. Strengthening the National Institute of Justice. Washington, DC: The National Academies Press. doi: 10.17226/12929.
×
Page 136
Suggested Citation:"4 Research Operations and Staffing Resources." National Research Council. 2010. Strengthening the National Institute of Justice. Washington, DC: The National Academies Press. doi: 10.17226/12929.
×
Page 137
Suggested Citation:"4 Research Operations and Staffing Resources." National Research Council. 2010. Strengthening the National Institute of Justice. Washington, DC: The National Academies Press. doi: 10.17226/12929.
×
Page 138
Suggested Citation:"4 Research Operations and Staffing Resources." National Research Council. 2010. Strengthening the National Institute of Justice. Washington, DC: The National Academies Press. doi: 10.17226/12929.
×
Page 139
Suggested Citation:"4 Research Operations and Staffing Resources." National Research Council. 2010. Strengthening the National Institute of Justice. Washington, DC: The National Academies Press. doi: 10.17226/12929.
×
Page 140
Suggested Citation:"4 Research Operations and Staffing Resources." National Research Council. 2010. Strengthening the National Institute of Justice. Washington, DC: The National Academies Press. doi: 10.17226/12929.
×
Page 141
Suggested Citation:"4 Research Operations and Staffing Resources." National Research Council. 2010. Strengthening the National Institute of Justice. Washington, DC: The National Academies Press. doi: 10.17226/12929.
×
Page 142
Suggested Citation:"4 Research Operations and Staffing Resources." National Research Council. 2010. Strengthening the National Institute of Justice. Washington, DC: The National Academies Press. doi: 10.17226/12929.
×
Page 143
Suggested Citation:"4 Research Operations and Staffing Resources." National Research Council. 2010. Strengthening the National Institute of Justice. Washington, DC: The National Academies Press. doi: 10.17226/12929.
×
Page 144
Next: 5 Building a Research Infrastructure and Guiding Policy and Practice »
Strengthening the National Institute of Justice Get This Book
×
Buy Paperback | $92.00 Buy Ebook | $74.99
MyNAP members save 10% online.
Login or Register to save!
Download Free PDF

The National Institute of Justice (NIJ) is the nation's primary resource for advancing scientific research, development, and evaluation on crime and crime control and the administration of justice in the United States. Headed by a presidentially appointed director, it is one of the major units in the Office of Justice Programs (OJP) of the U.S. Department of Justice. Under its authorizing legislation, NIJ awards grants and contracts to a variety of public and private organizations and individuals.

At the request of NIJ, Strengthening the National Institute of Justice assesses the operations and quality of the full range of its programs. These include social science research, science and technology research and development, capacity building, and technology assistance.

The book concludes that a federal research institute such as NIJ is vital to the nation's continuing efforts to control crime and administer justice. No other federal, state, local, or private organization can do what NIJ was created to do. Forty years ago, Congress envisioned a science agency dedicated to building knowledge to support crime prevention and control by developing a wide range of techniques for dealing with individual offenders, identifying injustices and biases in the administration of justice, and supporting more basic and operational research on crime and the criminal justice system and the involvement of the community in crime control efforts. As the embodiment of that vision, NIJ has accomplished a great deal. It has succeeded in developing a body of knowledge on such important topics as hot spots policing, violence against women, the role of firearms and drugs in crime, drug courts, and forensic DNA analysis. It has helped build the crime and justice research infrastructure. It has also widely disseminated the results of its research programs to help guide practice and policy. But its efforts have been severely hampered by a lack of independence, authority, and discretionary resources to carry out its mission.

  1. ×

    Welcome to OpenBook!

    You're looking at OpenBook, NAP.edu's online reading room since 1999. Based on feedback from you, our users, we've made some improvements that make it easier than ever to read thousands of publications on our website.

    Do you want to take a quick tour of the OpenBook's features?

    No Thanks Take a Tour »
  2. ×

    Show this book's table of contents, where you can jump to any chapter by name.

    « Back Next »
  3. ×

    ...or use these buttons to go back to the previous chapter or skip to the next one.

    « Back Next »
  4. ×

    Jump up to the previous page or down to the next one. Also, you can type in a page number and press Enter to go directly to that page in the book.

    « Back Next »
  5. ×

    Switch between the Original Pages, where you can read the report as it appeared in print, and Text Pages for the web version, where you can highlight and search the text.

    « Back Next »
  6. ×

    To search the entire text of this book, type in your search term here and press Enter.

    « Back Next »
  7. ×

    Share a link to this book page on your preferred social network or via email.

    « Back Next »
  8. ×

    View our suggested citation for this chapter.

    « Back Next »
  9. ×

    Ready to take your reading offline? Click here to buy this book in print or download it as a free PDF, if available.

    « Back Next »
Stay Connected!