Appendix B
Survey of Researchers and Practitioners

The Committee to Assess the Research Program of the National Institute of Justice (NIJ) asked HCM Marketing Research to conduct a survey in order to learn about the views of criminal justice researchers and practitioners. The committee wanted to know how familiar they were with NIJ’s activities and what they thought about the quality and impact of these activities. The committee was also interested in overall perceptions of NIJ as an independent science agency.

METHODOLOGY

Between November 20 and December 4, 2008, a total of 509 self-administered questionnaires were completed using an online survey technique. There were 347 questionnaires completed by researchers and 162 completed by practitioners. The sample, provided by the committee, included 2,603 e-mail addresses supplied by professional organizations.

The target researcher sample consisted of members of the American Society of Criminology. The target practitioner sample consisted of leaders and key staff in well-known organizations with an interest in criminal justice issues: the American Academy of Forensic Scientists, the American Correctional Association, the American Probation and Parole Association, the International Association of Chiefs of Police, the Justice Research Statistics Association, the National Association of Drug Court Professionals, the National Association of Sentencing Commissions, the National Center for State Courts, the National Criminal Justice Association, the National District Attorneys Association, and the Police Executive Research Forum. The



The National Academies | 500 Fifth St. N.W. | Washington, D.C. 20001
Copyright © National Academy of Sciences. All rights reserved.
Terms of Use and Privacy Statement



Below are the first 10 and last 10 pages of uncorrected machine-read text (when available) of this chapter, followed by the top 30 algorithmically extracted key phrases from the chapter as a whole.
Intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text on the opening pages of each chapter. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.

Do not use for reproduction, copying, pasting, or reading; exclusively for search engines.

OCR for page 263
Appendix B Survey of Researchers and Practitioners T he Committee to Assess the Research Program of the National Insti- tute of Justice (NIJ) asked HCM Marketing Research to conduct a survey in order to learn about the views of criminal justice research- ers and practitioners. The committee wanted to know how familiar they were with NIJ’s activities and what they thought about the quality and impact of these activities. The committee was also interested in overall perceptions of NIJ as an independent science agency. METHODOLOgY Between November 20 and December 4, 2008, a total of 509 self- administered questionnaires were completed using an online survey tech- nique. There were 347 questionnaires completed by researchers and 162 completed by practitioners. The sample, provided by the committee, in- cluded 2,603 e-mail addresses supplied by professional organizations. The target researcher sample consisted of members of the American Society of Criminology. The target practitioner sample consisted of lead- ers and key staff in well-known organizations with an interest in criminal justice issues: the American Academy of Forensic Scientists, the American Correctional Association, the American Probation and Parole Association, the International Association of Chiefs of Police, the Justice Research Statis- tics Association, the National Association of Drug Court Professionals, the National Association of Sentencing Commissions, the National Center for State Courts, the National Criminal Justice Association, the National Dis- trict Attorneys Association, and the Police Executive Research Forum. The 

OCR for page 263
 STRENGTHENING THE NATIONAL INSTITUTE OF JUSTICE sample pool was such that a small percentage of “practitioner” respondents (14 percent) would have been considered researchers if it had been possible to separate them out. They were affiliated with the target practitioner orga- nizations but work as researchers in a government agency. Quotas for the study were set proportional to the number of research- ers and practitioners provided in the sample. A quota of 500 completed interviews had been set in advance, but a light excess was due to multiple respondents completing the survey at the same time. An initial email in- vitation was sent on November 19, 2008, and one reminder email was sent during the course of the data collection period. Excluding 212 email addresses that bounced back as undeliverable, the overall response rate for the Internet survey is 21 percent, with nearly equal response among researchers (21 percent) and practitioners (22 percent). In order to qualify for the survey, respondents had to have some level of familiarity with NIJ. Only 13 respondents did not qualify and were terminated from continuing the survey. In addition, 73 respondents dropped out of the survey midway; had these surveys been completed, the response rate would have been 23 percent. Data from the online interviews were captured using Kinesis interview- ing software. The data were cleaned and responses to open-ended questions were coded and classified by similar responses. Cross-tabulations of the data were prepared showing replies to all questions for the entire sample as well as by multiple subgroups, including researchers, practitioners, years in field, current position, type of contact with NIJ, years of association with NIJ, field of training, respondent age, and gender. Since the cross- tabulations contain more information than can be readily assimilated, it should be regarded as the database for the study, of which this appendix is a summary. When a question was answered by a small number of respondents (30 or less), results are not as statistically reliable as those answered by larger groups, and results should be viewed with caution or for directional pur- poses only. RESuLTS Familiarity with NIJ NIJ is well known to criminal justice researchers and practitioners. Most of those interviewed (70 percent) said that they are very familiar with NIJ; only 13 of the potential respondents were dropped because they were unfamiliar with NIJ. This high level of familiarity underscores the impor- tance of NIJ to the field and lends credibility to the survey findings. Familiarity with NIJ was higher among researchers than practitioners:

OCR for page 263
 APPENDIX B 74 percent of the researchers said they were very familiar with NIJ com- pared with 61 percent of the practitioners. Many researchers actively use NIJ information resources, cite NIJ research findings in their work, and participate in NIJ’s grant process. • early all researchers (98 percent) have made use of or have cited N NIJ-sponsored research in their own work, including 87 percent who have done so three or more times. • mong available sources of research funding, researchers most A frequently named state or local governments (43 percent), followed by NIJ (31 percent), as their source of funding within the past 5 years. The types of contact with NIJ varied among respondents. A majority (75 percent) have used NIJ’s products and services. Over half said they had attended a NIJ conference or workshop or applied for a research grant (53 percent each). Roughly two in five have applied for and received a grant (43 percent). A third (33 percent) have been a peer reviewer, and over a quarter (26 percent) have been a participant in an advisory group. Only 8 percent said they have not been associated with NIJ in any of these types of roles. Of the 92 percent of respondents who have been associated with NIJ in some type of role, nearly all (91 percent) reported contact with NIJ since 2001. Image and Perceptions of NIJ Respondents believe that NIJ is an important government agency dedi- cated to the funding and dissemination of research on crime control and criminal justice issues, and many feel the need for NIJ has become more critical over the past decade. Furthermore, they believe that NIJ has im- pacted the policy and practices in many areas of criminal justice. • early all (99 percent) of respondents believe it is important to N have a government agency, such as NIJ, dedicated to funding and disseminating research on crime control and criminal justice issues. • ore researchers than practitioners believe such an agency is very M important (94 versus 85 percent). • wo-thirds (67 percent) of respondents feel the need for NIJ has T become more critical over the past 10 years. Nearly a quarter (23 percent) feel the need has remained the same, and 5 percent feel the need has become less critical over the past 10 years. Another 5 percent were unsure.

OCR for page 263
 STRENGTHENING THE NATIONAL INSTITUTE OF JUSTICE While deemed important and influential, some respondents questioned whether the agency has the necessary independence required to operate objectively, consistently, and fairly to best meet field needs. • espondents were equally likely to say that NIJ does not have R the independence necessary to be a research agency as to say that it does (37 percent doesn’t have independence versus 36 percent has independence). Over a quarter are not sure if NIJ does or not (27 percent). o ore practitioners than researchers believe NIJ has the neces- M sary independence (44 versus 32 percent). • reas in which researchers believe political considerations have A impacted NIJ are setting research priorities (85 percent), selecting proposals for funding (73 percent), and disseminating research findings (60 percent). Open-ended comments indicated that re- spondents believe that political considerations, whether external or internal to the agency, are too influential in establishing research priorities and in selecting proposals for funding. • IJ commitment to fairness and openness in disseminating research N findings was rated positively by 65 percent of the researchers, but only half as many researchers gave positive ratings to NIJ commit- ment to fairness and openness in establishing research priorities (37 percent) and selecting proposals for funding (36 percent). One measure of NIJ influence is the percentage of respondents saying that NIJ has an impact on policies and practices. Ratings of NIJ influence range from moderate to low for each of 15 areas listed in the questionnaire, as described below. • reas in NIJ seen as having the greatest impact are crime map- A ping (75 percent), law enforcement (66 percent), forensics and investigative science (66 percent), forensic laboratory enhancement (62 percent), and program evaluation (60 percent). • ther areas of impact reported by more that half of the respon- O dents include technology research and development (59 percent), courts (59 percent), violence against women and family violence (59 percent), corrections (59 percent), crime prevention (53 per- cent), victims (53 percent), juvenile justice (51 percent), and drugs and crime (51 percent). • ver a quarter (28 percent) indicated that NIJ has impacted poli- O cies and practices in areas other than these 15.

OCR for page 263
 APPENDIX B Performance on Key Measures Overall satisfaction is a key measure of how respondents feel about their experience with NIJ. Satisfaction was rated on a five-point scale from very positive to very negative with the midpoint of three indicating neither satisfied nor dissatisfied (answer wording varied by question). These scores were reclassified to indicate the percentage with positive or very positive satisfaction ratings. Based on other studies of performance using similar measures, scores in the 90 percent or higher range are considered outstand- ing. Organizations with scores at this level are usually growing and have a high level of retention. Scores in the 70 to 80 percent range indicate some good points and some areas for improvement. Scores below 70 percent are indicative of more serious problems. The results reveal opportunities for improvements to increase satisfac- tion with NIJ. Fewer than 60 percent of all respondents (57 percent) rated NIJ’s overall performance positively. This includes 19 percent who rated NIJ overall performance as excellent. About a quarter (23 percent) gave a neutral rating and 20 percent gave a low rating, including 7 percent who rated it poor. Practitioners are more satisfied than researchers with NIJ’s performance (69 versus 50 percent). Respondents were asked to rate five key aspects on their importance to NIJ performance. Staff qualifications, adequacy of resources, and con- sultation with the researcher community were rated as important by over 90 percent of the respondents. A large majority also rated leadership (89 percent) and consultation with the practitioner community (87 percent) as important. However, respondent satisfaction with NIJ performance for the same five key aspects is low. Over half of respondents are satisfied with the quali- fications of the staff (57 percent) and consultation with the researcher and practitioner communities (51 percent each). And 40 percent are satisfied with NIJ leadership and 27 percent with adequate resources. Researchers are less satisfied than practitioners in several areas: qualifications of the staff (53 versus 64 percent), NIJ leadership (33 versus 56 percent), adequate resources (22 versus 37 percent), and consultation with the researcher com- munity (47 versus 60 percent). The open-ended question elicited comments from over a third (34 percent) of the 509 respondents. Areas of concern described in the mostly negative comments include inappropriate political influence on NIJ (6 per- cent), lack of continued research funding (4 percent), the need for NIJ to operate independently (3 percent), the desire for NIJ to develop an unbiased grant process (3 percent), and an interest in diversifying the research to include topics other than DNA, technology, and terrorism (3 percent). Separate questions for researchers and practitioners further probed for

OCR for page 263
 STRENGTHENING THE NATIONAL INSTITUTE OF JUSTICE opinions on NIJ performance related to their areas of interest, as described below. Researcher Satisfaction with Key Measures Researchers reported low to moderate levels of satisfaction with NIJ performance. • issemination of findings to the research community was rated D positively by nearly two-thirds (64 percent) of researchers, while over half (56 percent) are satisfied with the dissemination of re- search findings to policy makers and practitioners. • evels of researchers’ satisfaction are lower in funding high-quality L research (44 percent), establishing research priorities that are policy relevant (44 percent), encouraging top flight researchers to apply for funding (37 percent), communicating research priorities to the field (37 percent), developing affordable and effective tools and technologies (36 percent), establishing research priorities that are coherent, important, and cumulative (33 percent), and developing and financially supporting future researchers (25 percent). Recipients of NIJ grants (44 percent of the researchers interviewed) ex- pressed low levels of satisfaction with the grant process and project moni- toring, with the exception of some moderate levels of satisfaction with NIJ staff. • ust over half (53 percent) of grantees are satisfied with NIJ’s moni- J toring of research activities. • ess than half are satisfied with the review of research products (46 L percent), ease of applying (44 percent), dissemination of research products (43 percent), and the quality of feedback/reviews (41 percent). • ewer grantees are satisfied with the quality of funding decisions F (38 percent) and transparency of the award process (28 percent). • evels of researchers’ satisfaction are moderate for the staff’s re- L sponsiveness (70 percent), fairness (69 percent), and competence (66 percent) but lower for the staff’s scientific knowledge (56 percent). Unsuccessful grant applicants (56 percent of researchers interviewed who had ever been denied a grant after applying) rated NIJ somewhat lower than grantees. The percentage satisfied with NIJ was quite low in the areas of ease of applying (44 percent), quality of feedback/reviews (33 percent),

OCR for page 263
 APPENDIX B quality of funding decisions (24 percent), and transparency of the award process (22 percent). Over a third (35 percent) of researchers had never applied to NIJ for a grant for a variety of reasons. The most frequently given reasons were that they thought it was unlikely that they would get funding (55 percent), research opportunities were not in their field of expertise or interest (45 percent), and they had insufficient notice to prepare an application (24 percent). Practitioner Satisfaction with Key Measures Practitioner satisfaction with NIJ performance in four key areas is moderate: 1. Dissemination of relevant research knowledge to practitioners and policy makers (72 percent). 2. Excellence and integrity in the conduct of NIJ activities (68 percent). 3. Identification of research and technology needs of criminal justice agencies and practitioners (66 percent). 4. Commitment to fairness and openness in practices (61 percent). Practitioner satisfaction with NIJ performance for six other areas is low: 1. Development of developing affordable and effective tools and tech- nologies (55 percent). 2. Improvement of forensic laboratories (53 percent). 3. Technical assistance (51 percent). 4. Testing of existing and new technologies (51 percent). 5. Development of equipment standards (48 percent). 6. Training of new scientists (28 percent). NIJ Information Resources NIJ supports a number of activities designed to communicate informa- tion on criminal justice research to researchers and practitioners. NIJ data resources are used more widely by researchers, while NIJ sponsored work- shops and conferences are attended more widely by practitioners. Many of those who use these resources find them to be useful.

OCR for page 263
0 STRENGTHENING THE NATIONAL INSTITUTE OF JUSTICE • he NIJ website has been used by nearly all (98 percent) and is T useful to three-quarters of them (73 percent). • he National Criminal Justice Reference Service has been accessed T by 90 percent of the researchers and is useful to most (89 percent) of them. • he University of Michigan data archive has been used by 75 per- T cent of the researchers and 90 percent of them find it useful. • IJ-sponsored national conferences have been attended by about N half of the researchers (51 percent) and about two-thirds (62 per- cent) of attendees found them useful. • IJ-sponsored workshops have been attended by more practitio- N ners (65 percent) than researchers (43 percent), and more practi- tioner participants (83 percent) than researcher participants (60 percent) found them useful. • IJ Criminal Justice Research and Evaluation Annual Conferences N have been attended by more than a third (37 percent) of the prac- titioners, and over three-quarters (77 percent) of attendees found the conference useful. • IJ Annual Technology Conferences have been attended by 27 N percent of practitioners, and nearly two-thirds (62 percent) of at- tendees found them useful. • he NIJ Critical Incidents Conference was attended by 5 percent T of practitioners, and half (50 percent) of the attendees found it useful. Researcher and Practitioner Profiles and Demographics The practitioner sample consisted of leaders and key staff in well- known organizations with an interest in criminal justice issues. Most of these respondents work full-time (89 percent) and are very experienced, with most (70 percent) having more than 20 years in the field. While nearly half (44 percent) were trained in social sciences, a number were trained in law (23 percent) and science and technology (17 percent). Nearly a quarter are currently criminal justice practitioners (24 percent) or government of- ficials (24 percent), and 1 in 7 is a researcher in a government agency (14 percent). Demographically, practitioners are middle-aged with a median age of 52, more likely to be male than female (73 versus 27 percent), and to have an advanced degree (40 percent master’s, 21 percent M.D./J.D./D.D.S., 21 percent doctorate). The researcher sample consists of members of the American Society of Criminology and includes a broader range of age groups and experi- ence levels. Nearly all researchers work full-time (97 percent) and are quite experienced, with half (47 percent) having more than 20 years in

OCR for page 263
 APPENDIX B the field. Nearly all (92 percent) were trained in social sciences, and most (80 percent) currently work as an academic. Compared with practitioners, researchers are younger, with a median age of 48, more likely to be female (37 percent), and more likely to have a doctorate degree (86 percent). Corrections and program evaluation were the topics most often named by researchers and practitioners as an area of interest. The primary area of interest was corrections and law enforcement for both researchers and practitioners. In addition, a fair number of practitioners focus on forensics and investigative sciences and courts. Respondent Segment Differences Differences in satisfaction found in the examination of the detailed tabulations are summarized below. • espondents with longer experience in criminal justice had more R different kinds of contact with NIJ and are more satisfied with its performance than respondents who have worked in the field for shorter periods. Research applicants with over 10 years in the field who had ever been denied a grant were significantly less sat- isfied with the grant process than denied applicants who are less experienced. • pinions and perceptions of NIJ of those who have been in contact O with NIJ since 2001 were compared with those who have not been in contact as recently. Both groups rated overall satisfaction with NIJ and the importance of NIJ similarly; however, respondents with recent contact were less satisfied with the leadership and hav- ing adequate resources. • ompared with researchers with only earlier contact, researchers C with recent contact since 2001 are more satisfied with the NIJ grant process, but are less satisfied with some other service aspects of NIJ. o esearcher satisfaction with the quality of funding decisions R during the grant process and the quality of feedback and reviews if denied a grant is higher among those with recent contact than among those with only earlier contact. o esearcher satisfaction with NIJ’s leadership and resources is R lower among those with recent contact than among those with only earlier contact. o esearcher satisfaction with the coherence, importance, and R cumulativeness of research priorities is lower among those with recent contact than among those with only earlier contact.

OCR for page 263
 STRENGTHENING THE NATIONAL INSTITUTE OF JUSTICE • ractitioners with recent contact since 2001 appear less satisfied P with NIJ than those who have not been in contact as recently. o ractitioner satisfaction with excellence and integrity in the P conduct of NIJ activities, the dissemination of research findings, and the utility of NIJ-sponsored activities is lower among those with more recent contact with the agency than those with only earlier contact. • here are some differences in NIJ ratings between men and women. T Female respondents place greater importance on NIJ and feel its impact on policy and practice has been greater in some areas than do men. Ratings for overall performance are similar among men and women, but women are more likely than men to believe the NIJ staff is qualified. Female grantees gave much higher ratings to the grant process than male grantees, but ratings among those denied a grant were similar between the two genders.