Appendix B
Survey of Researchers and Practitioners
The Committee to Assess the Research Program of the National Institute of Justice (NIJ) asked HCM Marketing Research to conduct a survey in order to learn about the views of criminal justice researchers and practitioners. The committee wanted to know how familiar they were with NIJ’s activities and what they thought about the quality and impact of these activities. The committee was also interested in overall perceptions of NIJ as an independent science agency.
METHODOLOGY
Between November 20 and December 4, 2008, a total of 509 self-administered questionnaires were completed using an online survey technique. There were 347 questionnaires completed by researchers and 162 completed by practitioners. The sample, provided by the committee, included 2,603 e-mail addresses supplied by professional organizations.
The target researcher sample consisted of members of the American Society of Criminology. The target practitioner sample consisted of leaders and key staff in well-known organizations with an interest in criminal justice issues: the American Academy of Forensic Scientists, the American Correctional Association, the American Probation and Parole Association, the International Association of Chiefs of Police, the Justice Research Statistics Association, the National Association of Drug Court Professionals, the National Association of Sentencing Commissions, the National Center for State Courts, the National Criminal Justice Association, the National District Attorneys Association, and the Police Executive Research Forum. The
sample pool was such that a small percentage of “practitioner” respondents (14 percent) would have been considered researchers if it had been possible to separate them out. They were affiliated with the target practitioner organizations but work as researchers in a government agency.
Quotas for the study were set proportional to the number of researchers and practitioners provided in the sample. A quota of 500 completed interviews had been set in advance, but a light excess was due to multiple respondents completing the survey at the same time. An initial email invitation was sent on November 19, 2008, and one reminder email was sent during the course of the data collection period. Excluding 212 email addresses that bounced back as undeliverable, the overall response rate for the Internet survey is 21 percent, with nearly equal response among researchers (21 percent) and practitioners (22 percent). In order to qualify for the survey, respondents had to have some level of familiarity with NIJ. Only 13 respondents did not qualify and were terminated from continuing the survey. In addition, 73 respondents dropped out of the survey midway; had these surveys been completed, the response rate would have been 23 percent.
Data from the online interviews were captured using Kinesis interviewing software. The data were cleaned and responses to open-ended questions were coded and classified by similar responses. Cross-tabulations of the data were prepared showing replies to all questions for the entire sample as well as by multiple subgroups, including researchers, practitioners, years in field, current position, type of contact with NIJ, years of association with NIJ, field of training, respondent age, and gender. Since the cross-tabulations contain more information than can be readily assimilated, it should be regarded as the database for the study, of which this appendix is a summary.
When a question was answered by a small number of respondents (30 or less), results are not as statistically reliable as those answered by larger groups, and results should be viewed with caution or for directional purposes only.
RESULTS
Familiarity with NIJ
NIJ is well known to criminal justice researchers and practitioners. Most of those interviewed (70 percent) said that they are very familiar with NIJ; only 13 of the potential respondents were dropped because they were unfamiliar with NIJ. This high level of familiarity underscores the importance of NIJ to the field and lends credibility to the survey findings.
Familiarity with NIJ was higher among researchers than practitioners:
74 percent of the researchers said they were very familiar with NIJ compared with 61 percent of the practitioners. Many researchers actively use NIJ information resources, cite NIJ research findings in their work, and participate in NIJ’s grant process.
-
Nearly all researchers (98 percent) have made use of or have cited NIJ-sponsored research in their own work, including 87 percent who have done so three or more times.
-
Among available sources of research funding, researchers most frequently named state or local governments (43 percent), followed by NIJ (31 percent), as their source of funding within the past 5 years.
The types of contact with NIJ varied among respondents. A majority (75 percent) have used NIJ’s products and services. Over half said they had attended a NIJ conference or workshop or applied for a research grant (53 percent each). Roughly two in five have applied for and received a grant (43 percent). A third (33 percent) have been a peer reviewer, and over a quarter (26 percent) have been a participant in an advisory group. Only 8 percent said they have not been associated with NIJ in any of these types of roles. Of the 92 percent of respondents who have been associated with NIJ in some type of role, nearly all (91 percent) reported contact with NIJ since 2001.
Image and Perceptions of NIJ
Respondents believe that NIJ is an important government agency dedicated to the funding and dissemination of research on crime control and criminal justice issues, and many feel the need for NIJ has become more critical over the past decade. Furthermore, they believe that NIJ has impacted the policy and practices in many areas of criminal justice.
-
Nearly all (99 percent) of respondents believe it is important to have a government agency, such as NIJ, dedicated to funding and disseminating research on crime control and criminal justice issues.
-
More researchers than practitioners believe such an agency is very important (94 versus 85 percent).
-
Two-thirds (67 percent) of respondents feel the need for NIJ has become more critical over the past 10 years. Nearly a quarter (23 percent) feel the need has remained the same, and 5 percent feel the need has become less critical over the past 10 years. Another 5 percent were unsure.
While deemed important and influential, some respondents questioned whether the agency has the necessary independence required to operate objectively, consistently, and fairly to best meet field needs.
-
Respondents were equally likely to say that NIJ does not have the independence necessary to be a research agency as to say that it does (37 percent doesn’t have independence versus 36 percent has independence). Over a quarter are not sure if NIJ does or not (27 percent).
-
More practitioners than researchers believe NIJ has the necessary independence (44 versus 32 percent).
-
-
Areas in which researchers believe political considerations have impacted NIJ are setting research priorities (85 percent), selecting proposals for funding (73 percent), and disseminating research findings (60 percent). Open-ended comments indicated that respondents believe that political considerations, whether external or internal to the agency, are too influential in establishing research priorities and in selecting proposals for funding.
-
NIJ commitment to fairness and openness in disseminating research findings was rated positively by 65 percent of the researchers, but only half as many researchers gave positive ratings to NIJ commitment to fairness and openness in establishing research priorities (37 percent) and selecting proposals for funding (36 percent).
One measure of NIJ influence is the percentage of respondents saying that NIJ has an impact on policies and practices. Ratings of NIJ influence range from moderate to low for each of 15 areas listed in the questionnaire, as described below.
-
Areas in NIJ seen as having the greatest impact are crime mapping (75 percent), law enforcement (66 percent), forensics and investigative science (66 percent), forensic laboratory enhancement (62 percent), and program evaluation (60 percent).
-
Other areas of impact reported by more that half of the respondents include technology research and development (59 percent), courts (59 percent), violence against women and family violence (59 percent), corrections (59 percent), crime prevention (53 percent), victims (53 percent), juvenile justice (51 percent), and drugs and crime (51 percent).
-
Over a quarter (28 percent) indicated that NIJ has impacted policies and practices in areas other than these 15.
Performance on Key Measures
Overall satisfaction is a key measure of how respondents feel about their experience with NIJ. Satisfaction was rated on a five-point scale from very positive to very negative with the midpoint of three indicating neither satisfied nor dissatisfied (answer wording varied by question). These scores were reclassified to indicate the percentage with positive or very positive satisfaction ratings. Based on other studies of performance using similar measures, scores in the 90 percent or higher range are considered outstanding. Organizations with scores at this level are usually growing and have a high level of retention. Scores in the 70 to 80 percent range indicate some good points and some areas for improvement. Scores below 70 percent are indicative of more serious problems.
The results reveal opportunities for improvements to increase satisfaction with NIJ. Fewer than 60 percent of all respondents (57 percent) rated NIJ’s overall performance positively. This includes 19 percent who rated NIJ overall performance as excellent. About a quarter (23 percent) gave a neutral rating and 20 percent gave a low rating, including 7 percent who rated it poor. Practitioners are more satisfied than researchers with NIJ’s performance (69 versus 50 percent).
Respondents were asked to rate five key aspects on their importance to NIJ performance. Staff qualifications, adequacy of resources, and consultation with the researcher community were rated as important by over 90 percent of the respondents. A large majority also rated leadership (89 percent) and consultation with the practitioner community (87 percent) as important.
However, respondent satisfaction with NIJ performance for the same five key aspects is low. Over half of respondents are satisfied with the qualifications of the staff (57 percent) and consultation with the researcher and practitioner communities (51 percent each). And 40 percent are satisfied with NIJ leadership and 27 percent with adequate resources. Researchers are less satisfied than practitioners in several areas: qualifications of the staff (53 versus 64 percent), NIJ leadership (33 versus 56 percent), adequate resources (22 versus 37 percent), and consultation with the researcher community (47 versus 60 percent).
The open-ended question elicited comments from over a third (34 percent) of the 509 respondents. Areas of concern described in the mostly negative comments include inappropriate political influence on NIJ (6 percent), lack of continued research funding (4 percent), the need for NIJ to operate independently (3 percent), the desire for NIJ to develop an unbiased grant process (3 percent), and an interest in diversifying the research to include topics other than DNA, technology, and terrorism (3 percent).
Separate questions for researchers and practitioners further probed for
opinions on NIJ performance related to their areas of interest, as described below.
Researcher Satisfaction with Key Measures
Researchers reported low to moderate levels of satisfaction with NIJ performance.
-
Dissemination of findings to the research community was rated positively by nearly two-thirds (64 percent) of researchers, while over half (56 percent) are satisfied with the dissemination of research findings to policy makers and practitioners.
-
Levels of researchers’ satisfaction are lower in funding high-quality research (44 percent), establishing research priorities that are policy relevant (44 percent), encouraging top flight researchers to apply for funding (37 percent), communicating research priorities to the field (37 percent), developing affordable and effective tools and technologies (36 percent), establishing research priorities that are coherent, important, and cumulative (33 percent), and developing and financially supporting future researchers (25 percent).
Recipients of NIJ grants (44 percent of the researchers interviewed) expressed low levels of satisfaction with the grant process and project monitoring, with the exception of some moderate levels of satisfaction with NIJ staff.
-
Just over half (53 percent) of grantees are satisfied with NIJ’s monitoring of research activities.
-
Less than half are satisfied with the review of research products (46 percent), ease of applying (44 percent), dissemination of research products (43 percent), and the quality of feedback/reviews (41 percent).
-
Fewer grantees are satisfied with the quality of funding decisions (38 percent) and transparency of the award process (28 percent).
-
Levels of researchers’ satisfaction are moderate for the staff’s responsiveness (70 percent), fairness (69 percent), and competence (66 percent) but lower for the staff’s scientific knowledge (56 percent).
Unsuccessful grant applicants (56 percent of researchers interviewed who had ever been denied a grant after applying) rated NIJ somewhat lower than grantees. The percentage satisfied with NIJ was quite low in the areas of ease of applying (44 percent), quality of feedback/reviews (33 percent),
quality of funding decisions (24 percent), and transparency of the award process (22 percent).
Over a third (35 percent) of researchers had never applied to NIJ for a grant for a variety of reasons. The most frequently given reasons were that they thought it was unlikely that they would get funding (55 percent), research opportunities were not in their field of expertise or interest (45 percent), and they had insufficient notice to prepare an application (24 percent).
Practitioner Satisfaction with Key Measures
Practitioner satisfaction with NIJ performance in four key areas is moderate:
-
Dissemination of relevant research knowledge to practitioners and policy makers (72 percent).
-
Excellence and integrity in the conduct of NIJ activities (68 percent).
-
Identification of research and technology needs of criminal justice agencies and practitioners (66 percent).
-
Commitment to fairness and openness in practices (61 percent).
Practitioner satisfaction with NIJ performance for six other areas is low:
-
Development of developing affordable and effective tools and technologies (55 percent).
-
Improvement of forensic laboratories (53 percent).
-
Technical assistance (51 percent).
-
Testing of existing and new technologies (51 percent).
-
Development of equipment standards (48 percent).
-
Training of new scientists (28 percent).
NIJ Information Resources
NIJ supports a number of activities designed to communicate information on criminal justice research to researchers and practitioners. NIJ data resources are used more widely by researchers, while NIJ sponsored workshops and conferences are attended more widely by practitioners. Many of those who use these resources find them to be useful.
-
The NIJ website has been used by nearly all (98 percent) and is useful to three-quarters of them (73 percent).
-
The National Criminal Justice Reference Service has been accessed by 90 percent of the researchers and is useful to most (89 percent) of them.
-
The University of Michigan data archive has been used by 75 percent of the researchers and 90 percent of them find it useful.
-
NIJ-sponsored national conferences have been attended by about half of the researchers (51 percent) and about two-thirds (62 percent) of attendees found them useful.
-
NIJ-sponsored workshops have been attended by more practitioners (65 percent) than researchers (43 percent), and more practitioner participants (83 percent) than researcher participants (60 percent) found them useful.
-
NIJ Criminal Justice Research and Evaluation Annual Conferences have been attended by more than a third (37 percent) of the practitioners, and over three-quarters (77 percent) of attendees found the conference useful.
-
NIJ Annual Technology Conferences have been attended by 27 percent of practitioners, and nearly two-thirds (62 percent) of attendees found them useful.
-
The NIJ Critical Incidents Conference was attended by 5 percent of practitioners, and half (50 percent) of the attendees found it useful.
Researcher and Practitioner Profiles and Demographics
The practitioner sample consisted of leaders and key staff in wellknown organizations with an interest in criminal justice issues. Most of these respondents work full-time (89 percent) and are very experienced, with most (70 percent) having more than 20 years in the field. While nearly half (44 percent) were trained in social sciences, a number were trained in law (23 percent) and science and technology (17 percent). Nearly a quarter are currently criminal justice practitioners (24 percent) or government officials (24 percent), and 1 in 7 is a researcher in a government agency (14 percent). Demographically, practitioners are middle-aged with a median age of 52, more likely to be male than female (73 versus 27 percent), and to have an advanced degree (40 percent master’s, 21 percent M.D./J.D./D.D.S., 21 percent doctorate).
The researcher sample consists of members of the American Society of Criminology and includes a broader range of age groups and experience levels. Nearly all researchers work full-time (97 percent) and are quite experienced, with half (47 percent) having more than 20 years in
the field. Nearly all (92 percent) were trained in social sciences, and most (80 percent) currently work as an academic. Compared with practitioners, researchers are younger, with a median age of 48, more likely to be female (37 percent), and more likely to have a doctorate degree (86 percent).
Corrections and program evaluation were the topics most often named by researchers and practitioners as an area of interest. The primary area of interest was corrections and law enforcement for both researchers and practitioners. In addition, a fair number of practitioners focus on forensics and investigative sciences and courts.
Respondent Segment Differences
Differences in satisfaction found in the examination of the detailed tabulations are summarized below.
-
Respondents with longer experience in criminal justice had more different kinds of contact with NIJ and are more satisfied with its performance than respondents who have worked in the field for shorter periods. Research applicants with over 10 years in the field who had ever been denied a grant were significantly less satisfied with the grant process than denied applicants who are less experienced.
-
Opinions and perceptions of NIJ of those who have been in contact with NIJ since 2001 were compared with those who have not been in contact as recently. Both groups rated overall satisfaction with NIJ and the importance of NIJ similarly; however, respondents with recent contact were less satisfied with the leadership and having adequate resources.
-
Compared with researchers with only earlier contact, researchers with recent contact since 2001 are more satisfied with the NIJ grant process, but are less satisfied with some other service aspects of NIJ.
-
Researcher satisfaction with the quality of funding decisions during the grant process and the quality of feedback and reviews if denied a grant is higher among those with recent contact than among those with only earlier contact.
-
Researcher satisfaction with NIJ’s leadership and resources is lower among those with recent contact than among those with only earlier contact.
-
Researcher satisfaction with the coherence, importance, and cumulativeness of research priorities is lower among those with recent contact than among those with only earlier contact.
-
-
Practitioners with recent contact since 2001 appear less satisfied with NIJ than those who have not been in contact as recently.
-
Practitioner satisfaction with excellence and integrity in the conduct of NIJ activities, the dissemination of research findings, and the utility of NIJ-sponsored activities is lower among those with more recent contact with the agency than those with only earlier contact.
-
-
There are some differences in NIJ ratings between men and women. Female respondents place greater importance on NIJ and feel its impact on policy and practice has been greater in some areas than do men. Ratings for overall performance are similar among men and women, but women are more likely than men to believe the NIJ staff is qualified. Female grantees gave much higher ratings to the grant process than male grantees, but ratings among those denied a grant were similar between the two genders.