4

NIDRR’s Peer Review Process

This chapter addresses the following key study question:

Key Question #2. To what extent are peer reviews of grant applications done in such a way as to enhance the quality of final results?

The National Institute on Disability and Rehabilitation Research’s (NIDRR’s) peer review process encompasses recruiting and training reviewers, conducting the review, and approving the awards. In the context of this study, as with the priority-setting process (Chapter 3), it is challenging to link the peer review process directly with specific output quality because the quality of grant outputs is the product of multiple complex factors, including the priority-setting process, funding levels, the peer review process, and the scientific quality of grantees. However, it is clear that the peer review process used by NIDRR contributes significantly to the success of the grant award program and the quality of the resulting outputs. Moreover, as described in The Future of Disability (Institute of Medicine, 2007), significant efforts to enhance the quality of NIDRR’s portfolio by strengthening the peer review process were implemented during the past decade.

This chapter begins by describing NIDRR’s peer review process. It then presents results of the committee’s assessment of the process. Finally, the chapter offers the committee’s conclusions and recommendations on this aspect of its evaluation.



The National Academies | 500 Fifth St. N.W. | Washington, D.C. 20001
Copyright © National Academy of Sciences. All rights reserved.
Terms of Use and Privacy Statement



Below are the first 10 and last 10 pages of uncorrected machine-read text (when available) of this chapter, followed by the top 30 algorithmically extracted key phrases from the chapter as a whole.
Intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text on the opening pages of each chapter. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.

Do not use for reproduction, copying, pasting, or reading; exclusively for search engines.

OCR for page 83
4 NIDRR’s Peer Review Process This chapter addresses the following key study question: Key Question #2. To what extent are peer reviews of grant applications done in such a way as to enhance the quality of final results? The National Institute on Disability and Rehabilitation Research’s (NIDRR’s) peer review process encompasses recruiting and training reviewers, conducting the review, and approving the awards. In the context of this study, as with the priority-setting process (Chapter 3), it is chal- lenging to link the peer review process directly with specific output quality because the quality of grant outputs is the product of multiple complex fac- tors, including the priority-setting process, funding levels, the peer review process, and the scientific quality of grantees. However, it is clear that the peer review process used by NIDRR contributes significantly to the success of the grant award program and the quality of the resulting outputs. More- over, as described in The Future of Disability (Institute of Medicine, 2007), significant efforts to enhance the quality of NIDRR’s portfolio by strength- ening the peer review process were implemented during the past decade. This chapter begins by describing NIDRR’s peer review process. It then presents results of the committee’s assessment of the process. Finally, the chapter offers the committee’s conclusions and recommendations on this aspect of its evaluation. 83

OCR for page 83
84 REVIEW OF DISABILITY AND REHABILITATION RESEARCH DESCRIPTION OF NIDRR’S PEER REVIEW PROCESS This description of NIDRR’s peer review process was compiled from ex- isting documentation, such as legislation, the Federal Register, NIDRR and the U.S. Department of Education (ED) policies and procedures, NIDRR’s Long-Range Plan (LRP), and notices inviting applications (NIAs). In addi- tion, the committee interviewed NIDRR and ED management to obtain a more thorough and cohesive understanding of the process.1 Legislative and Departmental Foundation Title II, section 202, of the Rehabilitation Act (1973, as amended) states that NIDRR will perform scientific peer review of all applications for research, training, and demonstration projects. The peer review is to “be conducted by scientists or other experts in the rehabilitation field, including knowledgeable individuals with disabilities, and the individuals’ represen- tatives” (p. 98). Federal employees are not allowed to be peer reviewers. NIDRR is to provide training for peer reviewers as is deemed necessary and appropriate. Title 34 of the Code of Federal Regulations (Disability and Rehabilita- tion Research Projects and Centers Program, 2009) states, “The purpose of peer review is to insure that activities supported by NIDRR are of the high- est scientific, administrative, and technical quality, and include all appropri- ate target populations and rehabilitation problems” (p. 217). Applications for awards of $60,000 or more must be reviewed by a peer review panel, with the exception of applications related to evaluation, dissemination of information, or conferences. In addition, NIDRR follows the peer review requirements of ED. In accordance with ED’s Handbook for the Discretionary Grant Process (ED Handbook), NIDRR annually reviews and updates its procedures in ED’s Application Technical Review Plan (a description of the processes for iden- tifying and involving reviewers, resolving conflicts of interest, working with the review panels, and selecting applications for funding) and maintains Grant Program Competition Files (a collection of all information, decisions, and documentation related to a competition) (U.S. Department of Educa- tion, 2009). Key Personnel in NIDRR’s Peer Review Process Key personnel in NIDRR’s peer review process include the competition manager, the panel monitor, and the agency’s peer review contractor. 1 The committee conducted interviews with NIDRR and ED management in four sessions during summer 2010 and one session in spring 2011.

OCR for page 83
85 NIDRR’S PEER REVIEW PROCESS Competition Manager Once an application kit2 has been published, NIDRR assigns a competi- tion manager—a NIDRR staff member who is responsible for all aspects of the review process (generally the individual who wrote the description of the priority area; see Chapter 3) (National Institute on Disability and Reha- bilitation Research, 2010b, 2010c). The competition manager arranges for the participation of additional NIDRR staff as necessary, recruits reviewers, confirms receipt of all applications, and performs a final screen of eligibility and responsiveness. In accordance with the Education Department General Administrative Regulations (EDGAR) (2008),3 NIDRR generally errs on the side of inclusivity, ruling out applications that are ineligible or nonre- sponsive and allowing peer reviewers to judge the merit of all remaining applications. Panel Monitor According to NIDRR management, the competition manager may also be the panel monitor. Duties of the panel monitor include managing the lo- gistics of panel review with assistance from NIDRR’s peer review contractor (see below), monitoring the progress of individual reviews, and overseeing the panel discussion. Competitions involving multiple panels typically em- ploy additional panel monitors from NIDRR, but may include panel moni- tors drawn from across the Office of Special Education and Rehabilitative Services (OSERS). Contractor Support NIDRR uses a contractor to provide support for the grant applica- tion and review process (Synergy Enterprises, Inc., 2008). The peer review contractor performs an initial screen of the eligibility and responsiveness of applications prior to the competition manager’s final screen, provides logis- tical support for the panel discussions, administers the postmeeting survey of the reviewers, compiles reports as requested, and provides other support as required. Additional detail on the role of the peer review contractor is provided later in the chapter. 2 An application kit is a package containing application forms, the notice of final priority, the NIA, salient regulations, and the peer review criteria for a competition. 3 Available: http://www2.ed.gov/policy/fund/reg/edgarReg/edgar.html [November 22, 2011].

OCR for page 83
86 REVIEW OF DISABILITY AND REHABILITATION RESEARCH Stages in NIDRR’s Peer Review Process NIDRR’s grant selection and peer review process consists of 12 stages: 1. Determine peer review criteria 2. Peer review kick-off meeting 3. Recruiting of peer reviewers 4. Preapplication meeting with potential applicants 5. Peer reviewer orientation 6. Prepanel correspondence 7. Panel discussion 8. Site visits 9. Prefunding meeting 10. Preparation and finalization of slate 11. Slate review 12. Slate approval and award The process takes approximately 4-6 months. The stages of the process are described below. Determine Peer Review Criteria Selection criteria applied by peer reviewers to assess and rate appli- cations are drawn from Title 34 of the Code of Federal Regulations and matched to the requirements of the competition. Each competition includes 100 possible points allocated across the criteria and subcriteria. With the exception of Spinal Cord Injury Model System (SCIMS), for which the point allocation is prespecified, the distribution of points across the selected criteria is determined by NIDRR staff. Criteria related to the quality of the proposed research or development are always allocated a substantial per- centage of the points (National Institute on Disability and Rehabilitation Research, 2010c). Past performance as a NIDRR grantee is not considered in the criteria for peer review, but is considered during the prefunding meet- ing (discussed below). The ED Handbook instructs reviewers to consider only the merit of the application itself. Additional knowledge of the field or the applicant is not to influence the review. Annex 4-1 at the end of this chapter provides more detail on the grant selection criteria, as well as an example of the selection criteria for a Disability and Rehabilitation Research Project-General (DRRP) competition. Kick-Off Meeting After publication of an application kit the competition manager con- venes a kick-off meeting with the contractor. During the kick-off meeting,

OCR for page 83
87 NIDRR’S PEER REVIEW PROCESS NIDRR staff determine the dates of panel discussions and other key dates leading up to the competition and discuss the division of labor for recruit- ing peer reviewers (National Institute on Disability and Rehabilitation Research, 2010c). Recruiting of Peer Reviewers NIDRR establishes peer review panels of five to seven members to re- view each submitted grant application (National Institute on Disability and Rehabilitation Research, 2006, 2009a, 2009b). The panel size depends on the size of the grants to be reviewed and the expertise needed. NIDRR uses standing panels—consisting of seven reviewers who serve as peer reviewers for up to 3 consecutive years following their initial appointment—for Field Initiated Project (FIP) competitions.4 Ad hoc panels are formed for all other competitions (National Institute on Disability and Rehabilitation Research, 2010b). According to NIDRR management, for Advanced Rehabilitation Research Training (ARRT), Small Business Innovation Research (SBIR), and Switzer Fellowship competitions, NIDRR draws on reviewers who have previ- ously been supported by these program mechanisms and who have relevant knowledge and expertise in these program areas. The competition manager tailors the composition of each review panel to competition requirements to ensure that the panel includes the expertise needed for the review (National Institute on Disability and Rehabilita- tion Research, 2010b). Competition managers identify potential reviewers through the Peer Review System (PRS), a searchable database containing information and resumes for thousands of potential peer reviewers main- tained at the OSERS level, as well as through literature searches, networking at conferences, and personal connections (National Institute on Disability and Rehabilitation Research, 2010b). As part of the recruiting process, the competition manager screens potential reviewers for conflicts of interest and often is forced to rule out many qualified individuals. NIDRR management stated that it is not uncommon for competition managers to make 50 or more recruitment calls in order to find five reviewers. Additionally, conflicts of interest can develop after the initial screening, requiring that reviewers be replaced (sometimes at the last minute). Furthermore, delays in the approval and publication of NIAs often leave NIDRR staff with shortened timelines in which to recruit peer reviewers and hold the panel discussion. NIDRR also strives to include qualified individuals with disabilities or their authorized representatives on review panels, as well as individuals from underrepresented populations. Since the number of individuals with disabili- 4 ED has strict rules related to conflict of interest, which impact the formation of NIDRR standing panels. FIP competitions are large enough to be exempt from the particular ED rules on conflict of interest.

OCR for page 83
88 REVIEW OF DISABILITY AND REHABILITATION RESEARCH ties who have the scientific credentials to conduct reviews is quite small, it can be difficult to represent the views of the various disability constituencies. At times, NIDRR will include individuals with disabilities without scientific expertise on review panels to lend the perspective of consumers5 if particu- larly relevant constituencies would otherwise not be included. NIDRR also produces a general list of all reviewers who have served in a given year (National Institute on Disability and Rehabilitation Research, 2009b). Per ED policy, the list does not identify the specific competitions in which the reviewers participated and is made available upon request (U.S. Department of Education, 2009). Preapplication Meeting with Potential Applicants Several weeks after an NIA is published in the Federal Register, NIDRR arranges and publicizes a conference call to provide guidance on the peer review process and technical assistance to potential applicants (National Institute on Disability and Rehabilitation Research, 2010c; also noted by NIDRR management). During the call, NIDRR staff provide guidance on the application process but do not provide advice related to the content of potential applications. NIDRR staff also generally make time for one-on- one consultation if it is requested. Peer Reviewer Orientation The competition manager conducts a competition-specific orientation session for all reviewers (National Institute on Disability and Rehabilitation Research, 2010b). The session is conducted via telephone within a few days of reviewers’ receipt of applications and review materials. The session is set up by the peer review contractor and generally lasts 1 hour. It includes an overview of the review process, a review of the selection criteria to be used in evaluating each application, a review of the online system, a discussion of reviewers’ responsibilities, tips for conducting a good review, and inqui- ries to determine whether any reviewer has developed a conflict of interest. Prepanel Correspondence After the training session and prior to the review, the competition man- ager and/or panel monitor will correspond with the reviewers (National Institute on Disability and Rehabilitation Research, 2010c; also noted by NIDRR management). The correspondence is intended to ensure that re- 5 Consumers are defined in this report as individuals with disabilities and their family mem- bers and/or authorized representatives.

OCR for page 83
89 NIDRR’S PEER REVIEW PROCESS viewers have everything they need to complete the review, that they are progressing through their initial reading of the applications, and that they are entering their initial scores and comments into the e-Reader system. Panel Discussion6 The technical review of applications consists of two parts: individual review of all applications, followed by panel review (National Institute on Disability and Rehabilitation Research, 2009a, 2009b). The panel review generally takes place via teleconference and e-Reader over 2-3 days. Individ- ual written reviews from each member of the review panel and a summary of the panel review documenting an application’s strengths and weaknesses are required before a grant can be awarded. NIDRR has conducted review meetings exclusively via teleconference for more than 5 years. NIDRR management noted that in the past there was some resistance to conducting review meetings by teleconference as opposed to in person. However, NIDRR believes that the benefits of teleconferences, including reduced cost for the agency and reduced time commitment for reviewers (which has resulted in more experienced researchers agreeing to participate), far outweigh the drawbacks, such as a loss of rapport among reviewers and between NIDRR staff and reviewers. Additionally, NIDRR has noticed that reviewers with mobility impairments benefit greatly from teleconference reviews, although reviewers with vision and hearing dis - abilities find the teleconference reviews more challenging. NIDRR provides additional support as necessary in the form of interpreters, communication access realtime translation (CART) services, alternative-format materials, and other personal assistance to allow reviewers with disabilities to partici- pate fully in the review. Grant applications are mailed to reviewers at least 3 weeks in advance of the review whenever possible (U.S. Department of Education, 2009). Re- viewers independently score and comment on each application using techni- cal review forms, which are accessed and saved electronically via e-Reader. Scores (whole numbers only) are assigned to each factor of each criterion. Peer reviewers may adjust their own scores before or immediately following the review teleconference. A score of less than the maximum point value must be accompanied by a written rationale. A maximum score does not require a written rationale, but reviewers are encouraged to include com- ments. As described by NIDRR management, the number of applications 6 Panel discussion procedures described here are a synthesis of information from written sources provided by NIDRR (National Institute on Disability and Rehabilitation Research, 2009a, 2009b); interviews with NIDRR management; and direct observation of panel discus- sions by committee members Thubi Kolobe and Pamela Loprest and co-study director Jeanne Rivard.

OCR for page 83
90 REVIEW OF DISABILITY AND REHABILITATION RESEARCH each panel reviews and the size of the applications vary greatly by program mechanism. On one end of the spectrum, center grant panel reviews (such as RRTC and RERC) generally include 2 or 3 applications with a maximum recommended length of 125 pages each (or 375 total pages maximum). Although center grant competitions usually receive only a few applications, each application is highly complex and technical. Additionally, many ap- plications are longer than the maximum recommended length. On the other end of the spectrum, FIP applications are shorter (50 pages) and not as technical as center grant applications, but a single panel is likely to review 20 applications totaling 1,000 pages minimum. In addition to the general review of all applications, each panel mem- ber is assigned to be either the primary or secondary reviewer for certain applications. The primary reviewer presents the application for discussion and writes a summary of the discussion. The secondary reviewer provides commentary on the application and assists the primary reviewer in writing the summary. All panel members participate in the discussion of each proposal (Na- tional Institute on Disability and Rehabilitation Research, 2009a, 2009b). Each application is discussed in turn, with each reviewer, beginning with the primary reviewer, presenting the scores and rationales for each criterion. Differences in scores among reviewers are discussed. If panel members’ scores are very different, the primary reviewer submits a description, taken from the discussion, of why this is the case. During the teleconference, the panel monitor oversees the discussion; helps the panel maintain consistency from criterion to criterion and applica- tion to application; reviews scores, comments, and summaries for adequacy and accuracy; and provides information concerning policy, regulations, selection criteria, technical review forms, conflicts of interest, and confiden- tiality. The panel monitor does not participate in the substantive discussion of applications or related research issues. NIDRR provides peer reviewers an honorarium of $200 a day, gener- ally for 1 day of preparation and 3 days of reviewing.7 NIDRR monitors the compensation for peer reviewers provided by other federal agencies and believes its rates are competitive. Site Visits Title II of the Rehabilitation Act requires a preaward 1-day site visit for those competitions in which an award or awards of more than $500,000 will be made. NIDRR management stated that the site visit is considered a 7 Doris Werwie, personal communication, National Institute on Disability and Rehabilitation Research, April 14, 2011.

OCR for page 83
91 NIDRR’S PEER REVIEW PROCESS part of the peer review process, with a visit being conducted for the highest rated applicant. Multiple site visits may be made if the highest rated appli- cants are within one point of each other. Site visits are conducted shortly af- ter the review and include one member of the review panel and one NIDRR staff member (National Institute on Disability and Rehabilitation Research, 2010c). Shortly before the visit, the NIDRR staff member submits questions to the applicant developed by the peer reviewers and by NIDRR staff. Ap- plicants respond to the questions in writing prior to and during the visit. Prefunding Meeting Following peer review, NIDRR holds a prefunding meeting involving the NIDRR Director, the Deputy Director, the two division Directors, the agency’s scientific advisor, the competition manager, and interested NIDRR staff to develop specific funding recommendations (National Institute on Disability and Rehabilitation Research, 2009a). At the meeting, the panel monitor and/or the competition manager presents the rank order of the applications as well as summary information on the peer review process, including information from the site visit if applicable, with emphasis on the peer reviewer comments (National Institute on Disability and Rehabilita- tion Research, 2010b). Additionally, applicants’ proposed project activities, budgets, and past performance are discussed. From this discussion, program staff develop specific funding recommendations. According to NIDRR man- agement, only in rare cases do the recommendations not follow the rank order established in peer review. Preparation and Finalization of Slate Through Award After the prefunding meeting, the competition manager transfers the recommendations for funding into a departmental format called a slate (National Institute on Disability and Rehabilitation Research, 2010c). The slate is then reviewed by Research Division management and approved by the NIDRR Director. It then must undergo an OSERS and ED clearance pro- cess, similar to proposed priorities (see Chapter 3). After approval of a slate by the Office of the Secretary of Education, NIDRR’s Program, Budget, and Evaluation Division obligates the funds to the new grantee. Additionally, NIDRR provides comments and suggestions for improvement to unsuccess- ful applicants following a review. NIDRR Competitions from Fiscal Years 2006 to 2009 NIDRR provided the committee with general data on the competi- tions held from fiscal years (FY) 2006 through 2009, including the num-

OCR for page 83
92 REVIEW OF DISABILITY AND REHABILITATION RESEARCH ber of competitions held for each program mechanism, applications re- ceived per competition, applications reviewed per competition, and awards made per competition (National Institute on Disability and Rehabilitation Research, 2010a). Table 4-1 summarizes these data. Each year over this 4-year time span, NIDRR held an average of 25 competitions and received an average of 492 applications. NIDRR reviewed between 48 percent (RERC 2006 competition) and 100 percent (13 different competitions) of applications received for each competition, and awarded grants to between 6 percent (Field Initiated Project-Development [FID] 2006 competition) and 83 percent (Burn Model System [BMS] 2007 competition) of applica- tions reviewed for each competition. However, the numbers of submitted applications, reviewed applications, and awards appear to vary greatly across years within the various program mechanisms. FIPs for research or development (FIR and FID) are by far the most competitive of the mecha- nisms, having the smallest proportion of grants awarded relative to number of grants reviewed (6 percent to 11 percent over the 4 years). The BMS and Disability and Business Technical Assistance Center (DBTAC) mechanisms (two mechanisms for which competitions were held for only a year of the analyzed data) appear to be the least competitive. Five out of the six BMS applications that were reviewed received awards; half of the DBTAC ap- plications reviewed received awards. Data Collection and Analysis by the Peer Review Contractor NIDRR’s peer review contractor collects and manages the data from and about peer reviews, including the peer review scores themselves and peer reviewer feedback on the process. In 2008, NIDRR asked the contractor to analyze the scoring data it had collected for 18 of the competitions that occurred in 2007 (Synergy Enterprises, Inc., 2008). The contractor drew three notable conclusions. First, there appeared to be no bias as to the types of individuals and organizations that received NIDRR funding, although institutes of higher education were being funded slightly more often than other types of organizations. Second, some competitions, such as those un- der the DRRP and RERC program mechanisms, had a notably higher rate of ineligible applications. Finally, while all funded applications received an overall score of at least 77, the contractor observed a lack of consistency in the language used for the scoring criteria for each program mechanism and no consistency in the number of points assigned to each scoring criterion within a mechanism. NIDRR’s peer review contractor surveys peer reviewers for feedback following every panel using the OSERS Panel Review Logistics Evaluation Form (Synergy Enterprises, Inc., 2010). Peer reviewers are asked to provide feedback on the prereview and review process, logistical support provided

OCR for page 83
93 NIDRR’S PEER REVIEW PROCESS by the contractor, special needs (only if any special accommodation was received), and suggestions for future reviews. Reviewers use a 5-point scale from poor (1) to excellent (5) to rate dimensions of the first three areas and provide comments on all areas. NIDRR provided the National Research Council (NRC) with data and a summary of the data collected from 147 of the 163 panel members participating in fiscal year 2008 to 2009 peer reviews. Response forms indicated the 18 specific competitions to which they related, but reviewer names were not included. Of note, 5 panels included fewer reviewers than are recommended by NIDRR procedure. To supplement NIDRR’s summary, NRC staff conducted a reanalysis of the data on the prereview and review process, special needs, and suggestions for future reviews. Data on the prereview and review process cover five dimensions: (1) completeness of materials, (2) quality of materials, (3) time allowed for initial review, (4) assistance provided by staff, and (5) participation by staff. The average rating for all dimensions was between excellent (5) and very good (4) except for the dimension time allowed, which was rated between very good (4) and good (3). The average ratings of the process across com- petitions for all dimensions ranged between 3.3 and 5, again except for time allowed, for which the ratings ranged from 1.7 to 4.8 and for which six ratings were lower than the lowest rating (3.3) for any of the other four dimensions. Comments on the prereview and review process also indicated that peer reviewers spent an average of 27 hours preparing for the reviews and an average of 20 hours participating. Combined preparation and participation time ranged from a low of an average of 15 hours to a high of an average of 60 hours. It should be noted that some peer reviewers both reported less time spent preparing and gave low ratings to time allowed for initial review, indicating they had less time to prepare than they wished. The last question about the prereview and review process asked review- ers to indicate whether the total of preparation and participation time was more than, less than, or about as much time as they expected to spend. Fifty-five percent of reviewers indicated they spent about as much time as they expected, 42 percent that they spent more time than they expected, and 3 percent that they spent less time then they expected. The section of the Panel Review Logistics Evaluation Form on special needs includes space to rate interpreter services, CART services, alternative- format materials, readers or scribes, and other personal assistance if any of these were requested. Only six reviewers used this section of the form; five rated alternative-format materials, readers or scribes, and other personal assistance as excellent, and one rated alternative-format materials as fair. Finally, many reviewers provided suggestions for future reviews. The most common suggestion by far was to reduce reviewers’ time commitment—

OCR for page 83
117 NIDRR’S PEER REVIEW PROCESS reviews of breast cancer research proposals on review panels that included 11-17 scientists and 2 lay consumers. The authors found little difference in proposal scores of the nonscientist consumers and the scientists. Pre- and post-panel opinion questionnaires concerning consumer involvement in the scientific review process showed significantly greater positive post-panel opinions of consumer involvement than negative opinions. Furthermore, the use of consumers in peer review processes is extensive in many other agencies. Following are examples of models used by other agencies to involve consumers in peer review which NIDRR might wish to review and consider for future use. (These examples are not intended to be exhaustive.) The Office of Congressionally Directed Medical Research Programs (CDMRP), located in the Department of Defense, fully integrates consumers and scientists on peer review panels. According to CDMRP (2011), consum- ers “add perspective, passion, and a sense of urgency that ensures the human dimension is incorporated in the program policy, investment strategy, and research focus.” CDMRP employs a two-tiered system of review, involving first a scientific review by a peer review panel and then a programmatic re- view by an integration panel. Consumers are fully integrated in both panels. Additionally, the National Institute of Mental Health (NIMH) at times includes consumers without scientific expertise in peer review. NIMH also uses a two-tiered peer review process. The first tier involves assessment of grant applications by review committees, which are comprised of scientist reviewers and sometimes reviewers who are members of the general public, including consumers (National Institute of Mental Health, 2011a). The NIMH website (2011b) states that, “The role of public reviewers is to bring critical perspectives from individuals and family members who have been directly affected . . . and to enhance the capability of the review commit- tee to evaluate the ‘real world’ relevance and practicality of each research application.” Public reviewers are instructed to focus their review on par- ticular aspects of the grant applications, such as public health significance, feasibility, outreach, and protection of human subjects (National Institute of Mental Health, 2011a). Similarly, NIDRR might identify which of its review criteria are most relevant to consumers without scientific expertise, and then ask consumer reviewers to rate only these criteria. The second tier in the NIMH process involves review by the NIMH Advisory Council, which is also composed of both scientist and lay members. Finally, the Juvenile Diabetes Research Foundation (2011) also utilizes a two-tiered system of review. The first tier is scientific review, during which “each individual project should be evaluated for its standalone scientific merit as well as its potential contribution to the whole program.” This phase of the process involves panels made up only of scientists. The second tier is lay review, during which a lay review committee uses its consumer experi-

OCR for page 83
118 REVIEW OF DISABILITY AND REHABILITATION RESEARCH ence and the results of the scientific review to determine which applications are likely to have the greatest impact. REFERENCES Andejeski, Y., Bisceglio, I.T., Dickersin, K., Johnson, J.E., Robinson, S.I., Smith, H S., Visco, F.M., and Rich, I.M. (2002). Quantitative impact of including consumers in the scientific review of breast cancer research proposals. Journal of Women’s Health and Gender-Based Medicine, 11(4), 379-388. Disability and Rehabilitation Research Projects and Centers Program, 34 C.F.R. pt. 350 (2009) Education Department General Administrative Regulations, 34 C.F.R. pts. 74-86 and 97-99 (2008). Available: http://www2.ed.gov/policy/fund/reg/edgarReg/edgar.html [November 22, 2011] . Institute of Medicine. (2007). The future of disability in America. Washington, DC: The Na- tional Academies Press. Ismail, S., Farrands, A., and Wooding, S. (2009). Evaluating grant peer review in the health sciences. Cambridge, England: RAND Europe. Juvenile Diabetes Research Foundation. (2011). Information for reviewers. Available: http:// www.jdrf.org/index.cfm?page_id=103243 [June 4, 2011]. National Institute of Mental Health. (2011a). Review process. Available: http://www.nimh.nih. gov/research-funding/grants/review-process.shtml [October 12, 2011]. National Institute of Mental Health. (2011b). Role of public participants in NIMH grant reviews. Available: http://www.nimh.nih.gov/research-funding/grants/role-of-public- participants-in-nimh-grant-reviews.shtml [October 12, 2011]. National Institute on Disability and Rehabilitation Research. (2006). Department of Education: National Institute on Disability and Rehabilitation Research—Notice of Final Long- Range Plan for Fiscal Years 2005–2009. Federal Register, 71(31), 8,166-8,200. National Institute on Disability and Rehabilitation Research. (2009a). Briefing book for The National Academies. Unpublished document. Washington, DC: National Institute on Disability and Rehabilitation Research. National Institute on Disability and Rehabilitation Research. (2009b). Peer reviewer instruc- tions. Unpublished document. Washington, DC: National Institute on Disability and Rehabilitation Research. National Institute on Disability and Rehabilitation Research. (2010a). 2006-2009 applications received, applications reviewed, and applications funded. Unpublished document. Wash- ington, DC: National Institute on Disability and Rehabilitation Research. National Institute on Disability and Rehabilitation Research. (2010b). Application technical re- view plan. Washington, DC: National Institute on Disability and Rehabilitation Research. National Institute on Disability and Rehabilitation Research. (2010c). Overview of the NIDRR’s peer review process (what happens between OPP’s priority review and OPP’s slate review). Unpublished document. Washington, DC: National Institute on Disability and Rehabilitation Research. National Institutes of Health. (2008). 2007-2008 peer review self-study. Washington, DC: National Institutes of Health. National Science Foundation. (2011). Committee of Visitors (COV). Available: http://www. nsf.gov/od/oia/activities/cov/ [April 5, 2011]. Office of Congressionally Directed Medical Research Programs. (2011). Consumer involve- ment. Available: http://cdmrp.army.mil/cwg/default.shtml [June 4, 2011]. The Rehabilitation Act of 1973, as amended. Pub. L. No. 93-112. Available: http://www2. ed.gov/policy/speced/reg/narrative.html [January 21, 2011].

OCR for page 83
119 NIDRR’S PEER REVIEW PROCESS Synergy Enterprises, Inc. (2008). Draft task 4 analysis. Washington, DC: National Institute on Disability and Rehabilitation. Synergy Enterprises, Inc. (2010). Analysis of peer review process—Synergy survey. Unpublished document. Washington, DC: National Institute on Disability and Rehabilitation. U.S. Department of Education. (2009). Handbook for the discretionary grant process. Wash- ington, DC: U.S. Department of Education. ANNEX 4-1 SELECTION CRITERIA Selection criteria are used by peer reviewers in assessing and rating applications submitted by researchers for funding. Title 34 of the Code of Federal Regulations (CFR)9 provides guidance for NIDRR’s peer review process, as well as selection criteria. Part 350 of the CFR outlines the selec- tion criteria for the competitions administered through the DRRP primary mechanisms, including DRRP-General, DBTAC, KT, Section 21, BMS, and TBIMS, as well as for the program mechanisms ARRT, FIP, RERC, and RRTC. Part 356 provides selection criteria for Switzer Fellowship. Part 359 provides selection criteria for SCIMS. Part 75 provides selection criteria for SBIR. Each competition includes 100 possible points allocated across criteria and subcriteria. With the exception of Part 359, governing SCIMS, where the points are prespecified, the distribution of points across the selected criteria is determined by NIDRR staff. All criteria are displayed in Table A4-1. The term “absolute priority” refers to those requirements that appli- cants must address to demonstrate their responsiveness to the requirements of the program mechanism (e.g., DRRP) or to the specific topic (e.g., telere- habilitation). The term “competitive priority” refers to requirements that can result in competitive preference, either by awarding extra points based on the extent to which the application meets the priority or by selecting an application that meets the priority over a similarly reviewed application that does not. An example is additional points being awarded to an application that includes effective strategies for employing and advancing in employ- ment qualified individuals with disabilities. Competitions under Parts 350 and 75 are not required to use all of the criteria, as certain criteria are not relevant to some competitions. NIDRR staff select the relevant criteria from the list provided in the CFR. As defined in the CFR, each criterion in Parts 350 and 75 contains subcriteria. As part 9 The electronic Code of Federal Regulations can be accessed at: http://ecfr.gpoaccess.gov/ cgi/t/text/text-idx?c=ecfr&tpl=%2Findex.tpl [January 4, 2012].

OCR for page 83
120 REVIEW OF DISABILITY AND REHABILITATION RESEARCH TABLE A4-1 Selection Criteria from Title 34, Code of Federal Regulations Title 34, Part 350 DBTAC DRRP KT Burn Title 34, Part 350 Importance of the problem x x x x Responsiveness to an absolute or competitive x x x x priority Design of research activities x x x x Design of development activities x x x x Design of demonstration activities x x x x Design of training activities x x x x Design of dissemination activities x x x x Design of utilization activities x x x x Design of technical assistance activities x x x x Plan of operation x x x x Collaboration x x x x Adequacy and reasonableness of the budget x x x x Plan of evaluation x x x x Project staff x x x x Adequacy and reasonableness of resources x x x x Title 34, Part 356 Quality and level of formal education Previous work experience Recommendations Quality of a research proposal The research hypothesis, methodology, and design Resources, equipment, institutional support

OCR for page 83
121 NIDRR’S PEER REVIEW PROCESS Part 356 Part 359 Part 75 TBI 21 FIP RRTC RERC ARRT Switzer SCI SBIR x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x continued

OCR for page 83
122 REVIEW OF DISABILITY AND REHABILITATION RESEARCH TABLE A4-1 Continued Title 34, Part 350 DBTAC DRRP KT Burn Title 34, Part 359 Project design (20 points) Service comprehensiveness (20 points) Plan of operation (15 points) Quality of key personnel (10 points) Adequacy of resources (10 points) Budget/ cost effectiveness (10 points) Dissemination/ Utilization (5 points) Evaluation plan (10 points) Title 34, Part 75 Need for project Significance Quality of the project design Quality of project services Quality of project personnel Adequacy of resources Quality of the management plan Quality of the project evaluation SOURCE: Generated by the committee based on the CFR, Title 34. of recommending criteria for a competition, NIDRR staff also recommend which subcriteria are relevant. For each competition, points out of 100 are distributed across the chosen criteria. The points assigned to each criterion are then divided among the subcriteria for purposes of scoring. Box A4-1 contains an example of the selection criteria for a DRRP competition. Part 350 also establishes additional considerations for FIP. Before fund- ing is awarded, the Secretary of Education considers the extent to which applications that have been awarded 80 percent or more of the maximum possible points meet one or both of the following conditions: represent a unique opportunity to advance rehabilitation knowledge and/or comple- ment current research or address such research in a promising new way. Part 75 does not include any additional considerations. The criteria in Part 356 governing Switzer do not contain subcriteria. Based on peer review scores, the Secretary grades applicants as outstanding (5), superior (4), satisfactory (3), marginal (2), or poor (1). The Secretary

OCR for page 83
123 NIDRR’S PEER REVIEW PROCESS Part 356 Part 359 Part 75 TBI 21 FIP RRTC RERC ARRT Switzer SCI SBIR x x x x x x x x x x x x x x x x funds some or all of the applications that have been awarded a rating of superior or better (4-5). In making a final selection, the Secretary considers the extent to which outstanding or superior applicants present a unique op- portunity to effect a major advance in knowledge, address critical problems in innovative ways, present proposals that are consistent with the NIDRR’s Long-Range Plan, build research capacity within the field, or complement and significantly increase the potential value of already planned research and related activities. Unlike the criteria in the other parts, Part 359 criteria governing SCIMS include point values (as can be seen in Table A4-1). The criteria in Part 359 do contain subcriteria for reviewers to consider, but the subcriteria are not scored; only the main criteria receive a score. In determining which applications to fund under this program, the Secretary also considers the proposed location of any project in order to achieve, to the extent possible, a geographic distribution of projects.

OCR for page 83
124 REVIEW OF DISABILITY AND REHABILITATION RESEARCH BOX A4-1 Example of Selection Criteria for Disability and Rehabilitation Research Project: Center on the Effective Delivery of Rehabilitation Technology by State Vocational Rehabilitation Agencies to Improve Employment Outcomes (CFDA Number 84.133A-4) Requirement for DRRP Projects: To meet this priority, the Disability and Rehabilitation Research Projects (DRRP) must— (a) Coordinate on research projects of mutual interest with relevant NIDRR-funded projects, as identified through consultation with the NIDRR project officer; (b) Involve individuals with disabilities in planning and implementing the DRRP’s research, training, and dissemination activities, and in evaluating its work; and (c) Identify anticipated outcomes (i.e., advances in knowledge or changes and improvements in policy, practice, behavior, and system capacity) that are linked to the applicant’s stated grant objectives. Specific Criteria for This Competition: The following selection criteria are used to evaluate applications under the DRRP program. The maximum score for all of these criteria is 100 points. The maximum score for each criterion is indicated in parentheses. (a) Importance of the problem. (8 points total). (1) The Secretary considers the importance of the problem. (2) In determining the importance of the problem, the Secretary considers the following factors: (i) The extent to which the applicant clearly describes the need and target population (4 points). (ii) The extent to which the proposed project will have a beneficial impact on the target population (4 points). (b) Responsiveness to an absolute or competitive priority (8 points total). (1) The Secretary considers the responsiveness of the application to an absolute or competitive priority published in the Federal Register. (2) In determining the application’s responsiveness to the absolute or com- petitive priority, the Secretary considers the following factors: (i) The extent to which the applicant addresses all requirements of the ab- solute or competitive priority (4 points). (ii) The extent to which the applicant’s proposed activities are likely to achieve the purposes of the absolute or competitive priority (4 points). (c) Design of research activities (40 points total). (1) The Secretary considers the extent to which the design of research activi- ties is likely to be effective in accomplishing the objectives of the project.

OCR for page 83
125 NIDRR’S PEER REVIEW PROCESS (2) In determining the extent to which the design is likely to be effective in accomplishing the objectives of the project, the Secretary considers the following factors: (i) The extent to which the research activities constitute a coherent, sustained approach to research in the field, including a substantial addition to the state-of-the-art (6 points). (ii) The extent to which the methodology of each proposed research activity is meritorious, including consideration of the extent to which— (A) The proposed design includes a comprehensive and informed review of the current literature, demonstrating knowledge of the state-of-the-art (5 points). (B) Each research hypothesis is theoretically sound and based on current knowledge (5 points). (C) Each sample population is appropriate and of sufficient size (8 points). (D) The data collection and measurement techniques are appropriate and likely to be effective (8 points). (E) The data analysis methods are appropriate (8 points). (d) Design of dissemination activities (8 points total). (1) The Secretary considers the extent to which the design of dissemination activities is likely to be effective in accomplishing the objectives of the project. (2) In determining the extent to which the design is likely to be effective in accomplishing the objectives of the project, the Secretary considers the following factors: (i) The extent to which the methods for dissemination are of sufficient quality, intensity, and duration (4 points). (ii) The extent to which the information to be disseminated will be accessible to individuals with disabilities (4 points). (e) Plan of operation (6 points total). (1) The Secretary considers the quality of the plan of operation. (2) In determining the quality of the plan of operation, the Secretary considers the following factor: (i) The adequacy of the plan of operation to achieve the objectives of the proposed project on time and within budget, including clearly defined responsibilities, and timelines for accomplishing project tasks (6 points). (f) Collaboration (4 points total). (1) The Secretary considers the quality of collaboration. (2) In determining the quality of collaboration, the Secretary considers the following factor: (i) The extent to which the applicant’s proposed collaboration with one or more agencies, organizations, or institutions is likely to be effective in achieving the relevant proposed activities of the project (4 points). continued

OCR for page 83
126 REVIEW OF DISABILITY AND REHABILITATION RESEARCH BOX A4-1 Continued (g) Adequacy and reasonableness of the budget (4 points total). (1) The Secretary considers the adequacy and the reasonableness of the proposed budget. (2) In determining the adequacy and the reasonableness of the proposed budget, the Secretary considers the following factors: (i) The extent to which the costs are reasonable in relation to the proposed project activities (2 points). (ii) The extent to which the budget for the project, including any subcontracts, is adequately justified to support the proposed project activities (2 points). (h) Plan of evaluation (8 points total). (1) The Secretary considers the quality of the plan of evaluation. (2) In determining the quality of the plan of evaluation, the Secretary consid- ers the following factors: (i) The extent to which the plan of evaluation provides for periodic assess- ment of progress toward— (A) Implementing the plan of operation (4 points); and (B) Achieving the project’s intended outcomes and expected impacts (4 points). (i) Project staff (10 points total). (1) The Secretary considers the quality of the project staff. (2) In determining the quality of the project staff, the Secretary considers the extent to which the applicant encourages applications for employment from persons who are members of groups that have traditionally been underrepresented based on race, color, national origin, gender, age, or disability (4 points).

OCR for page 83
127 NIDRR’S PEER REVIEW PROCESS (3) In addition, the Secretary considers the following factors: (i) The extent to which the key personnel and other key staff have appropri- ate training and experience in disciplines required to conduct all proposed activities (3 points). (ii) The extent to which the commitment of staff time is adequate to accom- plish all the proposed activities of the project (3 points). (j) Adequacy and accessibility of resources (4 points). (1) The Secretary considers the adequacy and accessibility of the applicant’s resources to implement the proposed project. (2) In determining the adequacy and accessibility of resources, the Secretary considers the following factors: (i) The extent to which the applicant is committed to provide adequate facili- ties, equipment, other resources, including administrative support, and laboratories, if appropriate (2 points). (ii) The extent to which the facilities, equipment, and other resources are appropriately accessible to individuals with disabilities who may use the facilities, equipment, and other resources of the project (2 points). NOTE: After the substantive review by the committee, but before publication of this report, NIDRR changed the function of Part 350 subcriteria. Selection criteria from Part 350 continue to use subcriteria but no longer include a breakdown of main criteria point values across the subcriteria (similar to the criteria from Part 359). SOURCE: NIDRR 2009 Application Kit for DRRP 84.133A-4, Center on the Effective Delivery of Rehabilitation Technology by State Vocational Rehabilitation Agencies to Improve Employ- ment Outcomes.