Below are the first 10 and last 10 pages of uncorrected machine-read text (when available) of this chapter, followed by the top 30 algorithmically extracted key phrases from the chapter as a whole.
Intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text on the opening pages of each chapter.
Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.
Do not use for reproduction, copying, pasting, or reading; exclusively for search engines.
OCR for page 83
4
NIDRR’s Peer Review Process
This chapter addresses the following key study question:
Key Question #2. To what extent are peer reviews of grant applications
done in such a way as to enhance the quality of final results?
The National Institute on Disability and Rehabilitation Research’s
(NIDRR’s) peer review process encompasses recruiting and training
reviewers, conducting the review, and approving the awards. In the context
of this study, as with the priority-setting process (Chapter 3), it is chal-
lenging to link the peer review process directly with specific output quality
because the quality of grant outputs is the product of multiple complex fac-
tors, including the priority-setting process, funding levels, the peer review
process, and the scientific quality of grantees. However, it is clear that the
peer review process used by NIDRR contributes significantly to the success
of the grant award program and the quality of the resulting outputs. More-
over, as described in The Future of Disability (Institute of Medicine, 2007),
significant efforts to enhance the quality of NIDRR’s portfolio by strength-
ening the peer review process were implemented during the past decade.
This chapter begins by describing NIDRR’s peer review process. It then
presents results of the committee’s assessment of the process. Finally, the
chapter offers the committee’s conclusions and recommendations on this
aspect of its evaluation.
83
OCR for page 84
84 REVIEW OF DISABILITY AND REHABILITATION RESEARCH
DESCRIPTION OF NIDRR’S PEER REVIEW PROCESS
This description of NIDRR’s peer review process was compiled from ex-
isting documentation, such as legislation, the Federal Register, NIDRR and
the U.S. Department of Education (ED) policies and procedures, NIDRR’s
Long-Range Plan (LRP), and notices inviting applications (NIAs). In addi-
tion, the committee interviewed NIDRR and ED management to obtain a
more thorough and cohesive understanding of the process.1
Legislative and Departmental Foundation
Title II, section 202, of the Rehabilitation Act (1973, as amended)
states that NIDRR will perform scientific peer review of all applications for
research, training, and demonstration projects. The peer review is to “be
conducted by scientists or other experts in the rehabilitation field, including
knowledgeable individuals with disabilities, and the individuals’ represen-
tatives” (p. 98). Federal employees are not allowed to be peer reviewers.
NIDRR is to provide training for peer reviewers as is deemed necessary and
appropriate.
Title 34 of the Code of Federal Regulations (Disability and Rehabilita-
tion Research Projects and Centers Program, 2009) states, “The purpose of
peer review is to insure that activities supported by NIDRR are of the high-
est scientific, administrative, and technical quality, and include all appropri-
ate target populations and rehabilitation problems” (p. 217). Applications
for awards of $60,000 or more must be reviewed by a peer review panel,
with the exception of applications related to evaluation, dissemination of
information, or conferences.
In addition, NIDRR follows the peer review requirements of ED. In
accordance with ED’s Handbook for the Discretionary Grant Process (ED
Handbook), NIDRR annually reviews and updates its procedures in ED’s
Application Technical Review Plan (a description of the processes for iden-
tifying and involving reviewers, resolving conflicts of interest, working with
the review panels, and selecting applications for funding) and maintains
Grant Program Competition Files (a collection of all information, decisions,
and documentation related to a competition) (U.S. Department of Educa-
tion, 2009).
Key Personnel in NIDRR’s Peer Review Process
Key personnel in NIDRR’s peer review process include the competition
manager, the panel monitor, and the agency’s peer review contractor.
1 The committee conducted interviews with NIDRR and ED management in four sessions
during summer 2010 and one session in spring 2011.
OCR for page 85
85
NIDRR’S PEER REVIEW PROCESS
Competition Manager
Once an application kit2 has been published, NIDRR assigns a competi-
tion manager—a NIDRR staff member who is responsible for all aspects of
the review process (generally the individual who wrote the description of
the priority area; see Chapter 3) (National Institute on Disability and Reha-
bilitation Research, 2010b, 2010c). The competition manager arranges for
the participation of additional NIDRR staff as necessary, recruits reviewers,
confirms receipt of all applications, and performs a final screen of eligibility
and responsiveness. In accordance with the Education Department General
Administrative Regulations (EDGAR) (2008),3 NIDRR generally errs on
the side of inclusivity, ruling out applications that are ineligible or nonre-
sponsive and allowing peer reviewers to judge the merit of all remaining
applications.
Panel Monitor
According to NIDRR management, the competition manager may also
be the panel monitor. Duties of the panel monitor include managing the lo-
gistics of panel review with assistance from NIDRR’s peer review contractor
(see below), monitoring the progress of individual reviews, and overseeing
the panel discussion. Competitions involving multiple panels typically em-
ploy additional panel monitors from NIDRR, but may include panel moni-
tors drawn from across the Office of Special Education and Rehabilitative
Services (OSERS).
Contractor Support
NIDRR uses a contractor to provide support for the grant applica-
tion and review process (Synergy Enterprises, Inc., 2008). The peer review
contractor performs an initial screen of the eligibility and responsiveness of
applications prior to the competition manager’s final screen, provides logis-
tical support for the panel discussions, administers the postmeeting survey
of the reviewers, compiles reports as requested, and provides other support
as required. Additional detail on the role of the peer review contractor is
provided later in the chapter.
2 An application kit is a package containing application forms, the notice of final priority,
the NIA, salient regulations, and the peer review criteria for a competition.
3 Available: http://www2.ed.gov/policy/fund/reg/edgarReg/edgar.html [November 22, 2011].
OCR for page 86
86 REVIEW OF DISABILITY AND REHABILITATION RESEARCH
Stages in NIDRR’s Peer Review Process
NIDRR’s grant selection and peer review process consists of 12 stages:
1. Determine peer review criteria
2. Peer review kick-off meeting
3. Recruiting of peer reviewers
4. Preapplication meeting with potential applicants
5. Peer reviewer orientation
6. Prepanel correspondence
7. Panel discussion
8. Site visits
9. Prefunding meeting
10. Preparation and finalization of slate
11. Slate review
12. Slate approval and award
The process takes approximately 4-6 months. The stages of the process are
described below.
Determine Peer Review Criteria
Selection criteria applied by peer reviewers to assess and rate appli-
cations are drawn from Title 34 of the Code of Federal Regulations and
matched to the requirements of the competition. Each competition includes
100 possible points allocated across the criteria and subcriteria. With the
exception of Spinal Cord Injury Model System (SCIMS), for which the
point allocation is prespecified, the distribution of points across the selected
criteria is determined by NIDRR staff. Criteria related to the quality of the
proposed research or development are always allocated a substantial per-
centage of the points (National Institute on Disability and Rehabilitation
Research, 2010c). Past performance as a NIDRR grantee is not considered
in the criteria for peer review, but is considered during the prefunding meet-
ing (discussed below). The ED Handbook instructs reviewers to consider
only the merit of the application itself. Additional knowledge of the field
or the applicant is not to influence the review. Annex 4-1 at the end of this
chapter provides more detail on the grant selection criteria, as well as an
example of the selection criteria for a Disability and Rehabilitation Research
Project-General (DRRP) competition.
Kick-Off Meeting
After publication of an application kit the competition manager con-
venes a kick-off meeting with the contractor. During the kick-off meeting,
OCR for page 87
87
NIDRR’S PEER REVIEW PROCESS
NIDRR staff determine the dates of panel discussions and other key dates
leading up to the competition and discuss the division of labor for recruit-
ing peer reviewers (National Institute on Disability and Rehabilitation
Research, 2010c).
Recruiting of Peer Reviewers
NIDRR establishes peer review panels of five to seven members to re-
view each submitted grant application (National Institute on Disability and
Rehabilitation Research, 2006, 2009a, 2009b). The panel size depends on
the size of the grants to be reviewed and the expertise needed. NIDRR uses
standing panels—consisting of seven reviewers who serve as peer reviewers
for up to 3 consecutive years following their initial appointment—for Field
Initiated Project (FIP) competitions.4 Ad hoc panels are formed for all other
competitions (National Institute on Disability and Rehabilitation Research,
2010b). According to NIDRR management, for Advanced Rehabilitation
Research Training (ARRT), Small Business Innovation Research (SBIR), and
Switzer Fellowship competitions, NIDRR draws on reviewers who have previ-
ously been supported by these program mechanisms and who have relevant
knowledge and expertise in these program areas.
The competition manager tailors the composition of each review panel
to competition requirements to ensure that the panel includes the expertise
needed for the review (National Institute on Disability and Rehabilita-
tion Research, 2010b). Competition managers identify potential reviewers
through the Peer Review System (PRS), a searchable database containing
information and resumes for thousands of potential peer reviewers main-
tained at the OSERS level, as well as through literature searches, networking
at conferences, and personal connections (National Institute on Disability
and Rehabilitation Research, 2010b). As part of the recruiting process, the
competition manager screens potential reviewers for conflicts of interest and
often is forced to rule out many qualified individuals. NIDRR management
stated that it is not uncommon for competition managers to make 50 or
more recruitment calls in order to find five reviewers. Additionally, conflicts
of interest can develop after the initial screening, requiring that reviewers be
replaced (sometimes at the last minute). Furthermore, delays in the approval
and publication of NIAs often leave NIDRR staff with shortened timelines
in which to recruit peer reviewers and hold the panel discussion.
NIDRR also strives to include qualified individuals with disabilities or
their authorized representatives on review panels, as well as individuals from
underrepresented populations. Since the number of individuals with disabili-
4 ED has strict rules related to conflict of interest, which impact the formation of NIDRR
standing panels. FIP competitions are large enough to be exempt from the particular ED rules
on conflict of interest.
OCR for page 88
88 REVIEW OF DISABILITY AND REHABILITATION RESEARCH
ties who have the scientific credentials to conduct reviews is quite small, it
can be difficult to represent the views of the various disability constituencies.
At times, NIDRR will include individuals with disabilities without scientific
expertise on review panels to lend the perspective of consumers5 if particu-
larly relevant constituencies would otherwise not be included.
NIDRR also produces a general list of all reviewers who have served in
a given year (National Institute on Disability and Rehabilitation Research,
2009b). Per ED policy, the list does not identify the specific competitions in
which the reviewers participated and is made available upon request (U.S.
Department of Education, 2009).
Preapplication Meeting with Potential Applicants
Several weeks after an NIA is published in the Federal Register, NIDRR
arranges and publicizes a conference call to provide guidance on the peer
review process and technical assistance to potential applicants (National
Institute on Disability and Rehabilitation Research, 2010c; also noted by
NIDRR management). During the call, NIDRR staff provide guidance on
the application process but do not provide advice related to the content of
potential applications. NIDRR staff also generally make time for one-on-
one consultation if it is requested.
Peer Reviewer Orientation
The competition manager conducts a competition-specific orientation
session for all reviewers (National Institute on Disability and Rehabilitation
Research, 2010b). The session is conducted via telephone within a few days
of reviewers’ receipt of applications and review materials. The session is set
up by the peer review contractor and generally lasts 1 hour. It includes an
overview of the review process, a review of the selection criteria to be used
in evaluating each application, a review of the online system, a discussion
of reviewers’ responsibilities, tips for conducting a good review, and inqui-
ries to determine whether any reviewer has developed a conflict of interest.
Prepanel Correspondence
After the training session and prior to the review, the competition man-
ager and/or panel monitor will correspond with the reviewers (National
Institute on Disability and Rehabilitation Research, 2010c; also noted by
NIDRR management). The correspondence is intended to ensure that re-
5 Consumers are defined in this report as individuals with disabilities and their family mem-
bers and/or authorized representatives.
OCR for page 89
89
NIDRR’S PEER REVIEW PROCESS
viewers have everything they need to complete the review, that they are
progressing through their initial reading of the applications, and that they
are entering their initial scores and comments into the e-Reader system.
Panel Discussion6
The technical review of applications consists of two parts: individual
review of all applications, followed by panel review (National Institute on
Disability and Rehabilitation Research, 2009a, 2009b). The panel review
generally takes place via teleconference and e-Reader over 2-3 days. Individ-
ual written reviews from each member of the review panel and a summary
of the panel review documenting an application’s strengths and weaknesses
are required before a grant can be awarded.
NIDRR has conducted review meetings exclusively via teleconference
for more than 5 years. NIDRR management noted that in the past there was
some resistance to conducting review meetings by teleconference as opposed
to in person. However, NIDRR believes that the benefits of teleconferences,
including reduced cost for the agency and reduced time commitment for
reviewers (which has resulted in more experienced researchers agreeing to
participate), far outweigh the drawbacks, such as a loss of rapport among
reviewers and between NIDRR staff and reviewers. Additionally, NIDRR
has noticed that reviewers with mobility impairments benefit greatly from
teleconference reviews, although reviewers with vision and hearing dis -
abilities find the teleconference reviews more challenging. NIDRR provides
additional support as necessary in the form of interpreters, communication
access realtime translation (CART) services, alternative-format materials,
and other personal assistance to allow reviewers with disabilities to partici-
pate fully in the review.
Grant applications are mailed to reviewers at least 3 weeks in advance
of the review whenever possible (U.S. Department of Education, 2009). Re-
viewers independently score and comment on each application using techni-
cal review forms, which are accessed and saved electronically via e-Reader.
Scores (whole numbers only) are assigned to each factor of each criterion.
Peer reviewers may adjust their own scores before or immediately following
the review teleconference. A score of less than the maximum point value
must be accompanied by a written rationale. A maximum score does not
require a written rationale, but reviewers are encouraged to include com-
ments. As described by NIDRR management, the number of applications
6 Panel discussion procedures described here are a synthesis of information from written
sources provided by NIDRR (National Institute on Disability and Rehabilitation Research,
2009a, 2009b); interviews with NIDRR management; and direct observation of panel discus-
sions by committee members Thubi Kolobe and Pamela Loprest and co-study director Jeanne
Rivard.
OCR for page 90
90 REVIEW OF DISABILITY AND REHABILITATION RESEARCH
each panel reviews and the size of the applications vary greatly by program
mechanism. On one end of the spectrum, center grant panel reviews (such
as RRTC and RERC) generally include 2 or 3 applications with a maximum
recommended length of 125 pages each (or 375 total pages maximum).
Although center grant competitions usually receive only a few applications,
each application is highly complex and technical. Additionally, many ap-
plications are longer than the maximum recommended length. On the other
end of the spectrum, FIP applications are shorter (50 pages) and not as
technical as center grant applications, but a single panel is likely to review
20 applications totaling 1,000 pages minimum.
In addition to the general review of all applications, each panel mem-
ber is assigned to be either the primary or secondary reviewer for certain
applications. The primary reviewer presents the application for discussion
and writes a summary of the discussion. The secondary reviewer provides
commentary on the application and assists the primary reviewer in writing
the summary.
All panel members participate in the discussion of each proposal (Na-
tional Institute on Disability and Rehabilitation Research, 2009a, 2009b).
Each application is discussed in turn, with each reviewer, beginning with
the primary reviewer, presenting the scores and rationales for each criterion.
Differences in scores among reviewers are discussed. If panel members’
scores are very different, the primary reviewer submits a description, taken
from the discussion, of why this is the case.
During the teleconference, the panel monitor oversees the discussion;
helps the panel maintain consistency from criterion to criterion and applica-
tion to application; reviews scores, comments, and summaries for adequacy
and accuracy; and provides information concerning policy, regulations,
selection criteria, technical review forms, conflicts of interest, and confiden-
tiality. The panel monitor does not participate in the substantive discussion
of applications or related research issues.
NIDRR provides peer reviewers an honorarium of $200 a day, gener-
ally for 1 day of preparation and 3 days of reviewing.7 NIDRR monitors
the compensation for peer reviewers provided by other federal agencies and
believes its rates are competitive.
Site Visits
Title II of the Rehabilitation Act requires a preaward 1-day site visit for
those competitions in which an award or awards of more than $500,000
will be made. NIDRR management stated that the site visit is considered a
7 Doris Werwie, personal communication, National Institute on Disability and Rehabilitation
Research, April 14, 2011.
OCR for page 91
91
NIDRR’S PEER REVIEW PROCESS
part of the peer review process, with a visit being conducted for the highest
rated applicant. Multiple site visits may be made if the highest rated appli-
cants are within one point of each other. Site visits are conducted shortly af-
ter the review and include one member of the review panel and one NIDRR
staff member (National Institute on Disability and Rehabilitation Research,
2010c). Shortly before the visit, the NIDRR staff member submits questions
to the applicant developed by the peer reviewers and by NIDRR staff. Ap-
plicants respond to the questions in writing prior to and during the visit.
Prefunding Meeting
Following peer review, NIDRR holds a prefunding meeting involving
the NIDRR Director, the Deputy Director, the two division Directors, the
agency’s scientific advisor, the competition manager, and interested NIDRR
staff to develop specific funding recommendations (National Institute on
Disability and Rehabilitation Research, 2009a). At the meeting, the panel
monitor and/or the competition manager presents the rank order of the
applications as well as summary information on the peer review process,
including information from the site visit if applicable, with emphasis on the
peer reviewer comments (National Institute on Disability and Rehabilita-
tion Research, 2010b). Additionally, applicants’ proposed project activities,
budgets, and past performance are discussed. From this discussion, program
staff develop specific funding recommendations. According to NIDRR man-
agement, only in rare cases do the recommendations not follow the rank
order established in peer review.
Preparation and Finalization of Slate Through Award
After the prefunding meeting, the competition manager transfers the
recommendations for funding into a departmental format called a slate
(National Institute on Disability and Rehabilitation Research, 2010c). The
slate is then reviewed by Research Division management and approved by
the NIDRR Director. It then must undergo an OSERS and ED clearance pro-
cess, similar to proposed priorities (see Chapter 3). After approval of a slate
by the Office of the Secretary of Education, NIDRR’s Program, Budget, and
Evaluation Division obligates the funds to the new grantee. Additionally,
NIDRR provides comments and suggestions for improvement to unsuccess-
ful applicants following a review.
NIDRR Competitions from Fiscal Years 2006 to 2009
NIDRR provided the committee with general data on the competi-
tions held from fiscal years (FY) 2006 through 2009, including the num-
OCR for page 92
92 REVIEW OF DISABILITY AND REHABILITATION RESEARCH
ber of competitions held for each program mechanism, applications re-
ceived per competition, applications reviewed per competition, and awards
made per competition (National Institute on Disability and Rehabilitation
Research, 2010a). Table 4-1 summarizes these data. Each year over this
4-year time span, NIDRR held an average of 25 competitions and received
an average of 492 applications. NIDRR reviewed between 48 percent
(RERC 2006 competition) and 100 percent (13 different competitions) of
applications received for each competition, and awarded grants to between
6 percent (Field Initiated Project-Development [FID] 2006 competition)
and 83 percent (Burn Model System [BMS] 2007 competition) of applica-
tions reviewed for each competition. However, the numbers of submitted
applications, reviewed applications, and awards appear to vary greatly
across years within the various program mechanisms. FIPs for research or
development (FIR and FID) are by far the most competitive of the mecha-
nisms, having the smallest proportion of grants awarded relative to number
of grants reviewed (6 percent to 11 percent over the 4 years). The BMS and
Disability and Business Technical Assistance Center (DBTAC) mechanisms
(two mechanisms for which competitions were held for only a year of the
analyzed data) appear to be the least competitive. Five out of the six BMS
applications that were reviewed received awards; half of the DBTAC ap-
plications reviewed received awards.
Data Collection and Analysis by the Peer Review Contractor
NIDRR’s peer review contractor collects and manages the data from
and about peer reviews, including the peer review scores themselves and peer
reviewer feedback on the process. In 2008, NIDRR asked the contractor
to analyze the scoring data it had collected for 18 of the competitions that
occurred in 2007 (Synergy Enterprises, Inc., 2008). The contractor drew
three notable conclusions. First, there appeared to be no bias as to the types
of individuals and organizations that received NIDRR funding, although
institutes of higher education were being funded slightly more often than
other types of organizations. Second, some competitions, such as those un-
der the DRRP and RERC program mechanisms, had a notably higher rate
of ineligible applications. Finally, while all funded applications received an
overall score of at least 77, the contractor observed a lack of consistency in
the language used for the scoring criteria for each program mechanism and
no consistency in the number of points assigned to each scoring criterion
within a mechanism.
NIDRR’s peer review contractor surveys peer reviewers for feedback
following every panel using the OSERS Panel Review Logistics Evaluation
Form (Synergy Enterprises, Inc., 2010). Peer reviewers are asked to provide
feedback on the prereview and review process, logistical support provided
OCR for page 93
93
NIDRR’S PEER REVIEW PROCESS
by the contractor, special needs (only if any special accommodation was
received), and suggestions for future reviews. Reviewers use a 5-point scale
from poor (1) to excellent (5) to rate dimensions of the first three areas and
provide comments on all areas.
NIDRR provided the National Research Council (NRC) with data
and a summary of the data collected from 147 of the 163 panel members
participating in fiscal year 2008 to 2009 peer reviews. Response forms
indicated the 18 specific competitions to which they related, but reviewer
names were not included. Of note, 5 panels included fewer reviewers than
are recommended by NIDRR procedure. To supplement NIDRR’s summary,
NRC staff conducted a reanalysis of the data on the prereview and review
process, special needs, and suggestions for future reviews.
Data on the prereview and review process cover five dimensions: (1)
completeness of materials, (2) quality of materials, (3) time allowed for
initial review, (4) assistance provided by staff, and (5) participation by staff.
The average rating for all dimensions was between excellent (5) and very
good (4) except for the dimension time allowed, which was rated between
very good (4) and good (3). The average ratings of the process across com-
petitions for all dimensions ranged between 3.3 and 5, again except for
time allowed, for which the ratings ranged from 1.7 to 4.8 and for which
six ratings were lower than the lowest rating (3.3) for any of the other four
dimensions.
Comments on the prereview and review process also indicated that peer
reviewers spent an average of 27 hours preparing for the reviews and an
average of 20 hours participating. Combined preparation and participation
time ranged from a low of an average of 15 hours to a high of an average
of 60 hours. It should be noted that some peer reviewers both reported less
time spent preparing and gave low ratings to time allowed for initial review,
indicating they had less time to prepare than they wished.
The last question about the prereview and review process asked review-
ers to indicate whether the total of preparation and participation time was
more than, less than, or about as much time as they expected to spend.
Fifty-five percent of reviewers indicated they spent about as much time as
they expected, 42 percent that they spent more time than they expected, and
3 percent that they spent less time then they expected.
The section of the Panel Review Logistics Evaluation Form on special
needs includes space to rate interpreter services, CART services, alternative-
format materials, readers or scribes, and other personal assistance if any of
these were requested. Only six reviewers used this section of the form; five
rated alternative-format materials, readers or scribes, and other personal
assistance as excellent, and one rated alternative-format materials as fair.
Finally, many reviewers provided suggestions for future reviews. The
most common suggestion by far was to reduce reviewers’ time commitment—
OCR for page 117
117
NIDRR’S PEER REVIEW PROCESS
reviews of breast cancer research proposals on review panels that included
11-17 scientists and 2 lay consumers. The authors found little difference in
proposal scores of the nonscientist consumers and the scientists. Pre- and
post-panel opinion questionnaires concerning consumer involvement in the
scientific review process showed significantly greater positive post-panel
opinions of consumer involvement than negative opinions.
Furthermore, the use of consumers in peer review processes is extensive
in many other agencies. Following are examples of models used by other
agencies to involve consumers in peer review which NIDRR might wish to
review and consider for future use. (These examples are not intended to be
exhaustive.)
The Office of Congressionally Directed Medical Research Programs
(CDMRP), located in the Department of Defense, fully integrates consumers
and scientists on peer review panels. According to CDMRP (2011), consum-
ers “add perspective, passion, and a sense of urgency that ensures the human
dimension is incorporated in the program policy, investment strategy, and
research focus.” CDMRP employs a two-tiered system of review, involving
first a scientific review by a peer review panel and then a programmatic re-
view by an integration panel. Consumers are fully integrated in both panels.
Additionally, the National Institute of Mental Health (NIMH) at times
includes consumers without scientific expertise in peer review. NIMH also
uses a two-tiered peer review process. The first tier involves assessment of
grant applications by review committees, which are comprised of scientist
reviewers and sometimes reviewers who are members of the general public,
including consumers (National Institute of Mental Health, 2011a). The
NIMH website (2011b) states that, “The role of public reviewers is to bring
critical perspectives from individuals and family members who have been
directly affected . . . and to enhance the capability of the review commit-
tee to evaluate the ‘real world’ relevance and practicality of each research
application.” Public reviewers are instructed to focus their review on par-
ticular aspects of the grant applications, such as public health significance,
feasibility, outreach, and protection of human subjects (National Institute of
Mental Health, 2011a). Similarly, NIDRR might identify which of its review
criteria are most relevant to consumers without scientific expertise, and then
ask consumer reviewers to rate only these criteria. The second tier in the
NIMH process involves review by the NIMH Advisory Council, which is
also composed of both scientist and lay members.
Finally, the Juvenile Diabetes Research Foundation (2011) also utilizes
a two-tiered system of review. The first tier is scientific review, during which
“each individual project should be evaluated for its standalone scientific
merit as well as its potential contribution to the whole program.” This phase
of the process involves panels made up only of scientists. The second tier is
lay review, during which a lay review committee uses its consumer experi-
OCR for page 118
118 REVIEW OF DISABILITY AND REHABILITATION RESEARCH
ence and the results of the scientific review to determine which applications
are likely to have the greatest impact.
REFERENCES
Andejeski, Y., Bisceglio, I.T., Dickersin, K., Johnson, J.E., Robinson, S.I., Smith, H S., Visco,
F.M., and Rich, I.M. (2002). Quantitative impact of including consumers in the scientific
review of breast cancer research proposals. Journal of Women’s Health and Gender-Based
Medicine, 11(4), 379-388.
Disability and Rehabilitation Research Projects and Centers Program, 34 C.F.R. pt. 350 (2009)
Education Department General Administrative Regulations, 34 C.F.R. pts. 74-86 and 97-99
(2008). Available: http://www2.ed.gov/policy/fund/reg/edgarReg/edgar.html [November
22, 2011] .
Institute of Medicine. (2007). The future of disability in America. Washington, DC: The Na-
tional Academies Press.
Ismail, S., Farrands, A., and Wooding, S. (2009). Evaluating grant peer review in the health
sciences. Cambridge, England: RAND Europe.
Juvenile Diabetes Research Foundation. (2011). Information for reviewers. Available: http://
www.jdrf.org/index.cfm?page_id=103243 [June 4, 2011].
National Institute of Mental Health. (2011a). Review process. Available: http://www.nimh.nih.
gov/research-funding/grants/review-process.shtml [October 12, 2011].
National Institute of Mental Health. (2011b). Role of public participants in NIMH grant
reviews. Available: http://www.nimh.nih.gov/research-funding/grants/role-of-public-
participants-in-nimh-grant-reviews.shtml [October 12, 2011].
National Institute on Disability and Rehabilitation Research. (2006). Department of Education:
National Institute on Disability and Rehabilitation Research—Notice of Final Long-
Range Plan for Fiscal Years 2005–2009. Federal Register, 71(31), 8,166-8,200.
National Institute on Disability and Rehabilitation Research. (2009a). Briefing book for The
National Academies. Unpublished document. Washington, DC: National Institute on
Disability and Rehabilitation Research.
National Institute on Disability and Rehabilitation Research. (2009b). Peer reviewer instruc-
tions. Unpublished document. Washington, DC: National Institute on Disability and
Rehabilitation Research.
National Institute on Disability and Rehabilitation Research. (2010a). 2006-2009 applications
received, applications reviewed, and applications funded. Unpublished document. Wash-
ington, DC: National Institute on Disability and Rehabilitation Research.
National Institute on Disability and Rehabilitation Research. (2010b). Application technical re-
view plan. Washington, DC: National Institute on Disability and Rehabilitation Research.
National Institute on Disability and Rehabilitation Research. (2010c). Overview of the
NIDRR’s peer review process (what happens between OPP’s priority review and OPP’s
slate review). Unpublished document. Washington, DC: National Institute on Disability
and Rehabilitation Research.
National Institutes of Health. (2008). 2007-2008 peer review self-study. Washington, DC:
National Institutes of Health.
National Science Foundation. (2011). Committee of Visitors (COV). Available: http://www.
nsf.gov/od/oia/activities/cov/ [April 5, 2011].
Office of Congressionally Directed Medical Research Programs. (2011). Consumer involve-
ment. Available: http://cdmrp.army.mil/cwg/default.shtml [June 4, 2011].
The Rehabilitation Act of 1973, as amended. Pub. L. No. 93-112. Available: http://www2.
ed.gov/policy/speced/reg/narrative.html [January 21, 2011].
OCR for page 119
119
NIDRR’S PEER REVIEW PROCESS
Synergy Enterprises, Inc. (2008). Draft task 4 analysis. Washington, DC: National Institute on
Disability and Rehabilitation.
Synergy Enterprises, Inc. (2010). Analysis of peer review process—Synergy survey. Unpublished
document. Washington, DC: National Institute on Disability and Rehabilitation.
U.S. Department of Education. (2009). Handbook for the discretionary grant process. Wash-
ington, DC: U.S. Department of Education.
ANNEX 4-1
SELECTION CRITERIA
Selection criteria are used by peer reviewers in assessing and rating
applications submitted by researchers for funding. Title 34 of the Code of
Federal Regulations (CFR)9 provides guidance for NIDRR’s peer review
process, as well as selection criteria. Part 350 of the CFR outlines the selec-
tion criteria for the competitions administered through the DRRP primary
mechanisms, including DRRP-General, DBTAC, KT, Section 21, BMS,
and TBIMS, as well as for the program mechanisms ARRT, FIP, RERC,
and RRTC. Part 356 provides selection criteria for Switzer Fellowship.
Part 359 provides selection criteria for SCIMS. Part 75 provides selection
criteria for SBIR. Each competition includes 100 possible points allocated
across criteria and subcriteria. With the exception of Part 359, governing
SCIMS, where the points are prespecified, the distribution of points across
the selected criteria is determined by NIDRR staff. All criteria are displayed
in Table A4-1.
The term “absolute priority” refers to those requirements that appli-
cants must address to demonstrate their responsiveness to the requirements
of the program mechanism (e.g., DRRP) or to the specific topic (e.g., telere-
habilitation). The term “competitive priority” refers to requirements that
can result in competitive preference, either by awarding extra points based
on the extent to which the application meets the priority or by selecting an
application that meets the priority over a similarly reviewed application that
does not. An example is additional points being awarded to an application
that includes effective strategies for employing and advancing in employ-
ment qualified individuals with disabilities.
Competitions under Parts 350 and 75 are not required to use all of the
criteria, as certain criteria are not relevant to some competitions. NIDRR
staff select the relevant criteria from the list provided in the CFR. As defined
in the CFR, each criterion in Parts 350 and 75 contains subcriteria. As part
9 The electronic Code of Federal Regulations can be accessed at: http://ecfr.gpoaccess.gov/
cgi/t/text/text-idx?c=ecfr&tpl=%2Findex.tpl [January 4, 2012].
OCR for page 120
120 REVIEW OF DISABILITY AND REHABILITATION RESEARCH
TABLE A4-1 Selection Criteria from Title 34, Code of Federal Regulations
Title 34, Part 350
DBTAC DRRP KT Burn
Title 34, Part 350
Importance of the problem x x x x
Responsiveness to an absolute or competitive x x x x
priority
Design of research activities x x x x
Design of development activities x x x x
Design of demonstration activities x x x x
Design of training activities x x x x
Design of dissemination activities x x x x
Design of utilization activities x x x x
Design of technical assistance activities x x x x
Plan of operation x x x x
Collaboration x x x x
Adequacy and reasonableness of the budget x x x x
Plan of evaluation x x x x
Project staff x x x x
Adequacy and reasonableness of resources x x x x
Title 34, Part 356
Quality and level of formal education
Previous work experience
Recommendations
Quality of a research proposal
The research hypothesis, methodology, and
design
Resources, equipment, institutional support
OCR for page 121
121
NIDRR’S PEER REVIEW PROCESS
Part 356 Part 359 Part 75
TBI 21 FIP RRTC RERC ARRT Switzer SCI SBIR
x x x x x x
x x x x x x
x x x x x x
x x x x x x
x x x x x x
x x x x x x
x x x x x x
x x x x x x
x x x x x x
x x x x x x
x x x x x x
x x x x x x
x x x x x x
x x x x x x
x x x x x x
x
x
x
x
x
x
continued
OCR for page 122
122 REVIEW OF DISABILITY AND REHABILITATION RESEARCH
TABLE A4-1 Continued
Title 34, Part 350
DBTAC DRRP KT Burn
Title 34, Part 359
Project design (20 points)
Service comprehensiveness (20 points)
Plan of operation (15 points)
Quality of key personnel (10 points)
Adequacy of resources (10 points)
Budget/ cost effectiveness (10 points)
Dissemination/ Utilization (5 points)
Evaluation plan (10 points)
Title 34, Part 75
Need for project
Significance
Quality of the project design
Quality of project services
Quality of project personnel
Adequacy of resources
Quality of the management plan
Quality of the project evaluation
SOURCE: Generated by the committee based on the CFR, Title 34.
of recommending criteria for a competition, NIDRR staff also recommend
which subcriteria are relevant. For each competition, points out of 100 are
distributed across the chosen criteria. The points assigned to each criterion
are then divided among the subcriteria for purposes of scoring. Box A4-1
contains an example of the selection criteria for a DRRP competition.
Part 350 also establishes additional considerations for FIP. Before fund-
ing is awarded, the Secretary of Education considers the extent to which
applications that have been awarded 80 percent or more of the maximum
possible points meet one or both of the following conditions: represent a
unique opportunity to advance rehabilitation knowledge and/or comple-
ment current research or address such research in a promising new way.
Part 75 does not include any additional considerations.
The criteria in Part 356 governing Switzer do not contain subcriteria.
Based on peer review scores, the Secretary grades applicants as outstanding
(5), superior (4), satisfactory (3), marginal (2), or poor (1). The Secretary
OCR for page 123
123
NIDRR’S PEER REVIEW PROCESS
Part 356 Part 359 Part 75
TBI 21 FIP RRTC RERC ARRT Switzer SCI SBIR
x
x
x
x
x
x
x
x
x
x
x
x
x
x
x
x
funds some or all of the applications that have been awarded a rating of
superior or better (4-5). In making a final selection, the Secretary considers
the extent to which outstanding or superior applicants present a unique op-
portunity to effect a major advance in knowledge, address critical problems
in innovative ways, present proposals that are consistent with the NIDRR’s
Long-Range Plan, build research capacity within the field, or complement
and significantly increase the potential value of already planned research
and related activities.
Unlike the criteria in the other parts, Part 359 criteria governing SCIMS
include point values (as can be seen in Table A4-1). The criteria in Part
359 do contain subcriteria for reviewers to consider, but the subcriteria
are not scored; only the main criteria receive a score. In determining which
applications to fund under this program, the Secretary also considers the
proposed location of any project in order to achieve, to the extent possible,
a geographic distribution of projects.
OCR for page 124
124 REVIEW OF DISABILITY AND REHABILITATION RESEARCH
BOX A4-1
Example of Selection Criteria for Disability and
Rehabilitation Research Project:
Center on the Effective Delivery of Rehabilitation
Technology by State Vocational Rehabilitation Agencies to
Improve Employment Outcomes
(CFDA Number 84.133A-4)
Requirement for DRRP Projects:
To meet this priority, the Disability and Rehabilitation Research Projects (DRRP)
must—
(a) Coordinate on research projects of mutual interest with relevant NIDRR-funded
projects, as identified through consultation with the NIDRR project officer;
(b) Involve individuals with disabilities in planning and implementing the DRRP’s
research, training, and dissemination activities, and in evaluating its work; and
(c) Identify anticipated outcomes (i.e., advances in knowledge or changes and
improvements in policy, practice, behavior, and system capacity) that are linked
to the applicant’s stated grant objectives.
Specific Criteria for This Competition:
The following selection criteria are used to evaluate applications under the DRRP
program. The maximum score for all of these criteria is 100 points. The maximum
score for each criterion is indicated in parentheses.
(a) Importance of the problem. (8 points total).
(1) The Secretary considers the importance of the problem.
(2) In determining the importance of the problem, the Secretary considers
the following factors:
(i) The extent to which the applicant clearly describes the need and target
population (4 points).
(ii) The extent to which the proposed project will have a beneficial impact on
the target population (4 points).
(b) Responsiveness to an absolute or competitive priority (8 points total).
(1) The Secretary considers the responsiveness of the application to an
absolute or competitive priority published in the Federal Register.
(2) In determining the application’s responsiveness to the absolute or com-
petitive priority, the Secretary considers the following factors:
(i) The extent to which the applicant addresses all requirements of the ab-
solute or competitive priority (4 points).
(ii) The extent to which the applicant’s proposed activities are likely to achieve
the purposes of the absolute or competitive priority (4 points).
(c) Design of research activities (40 points total).
(1) The Secretary considers the extent to which the design of research activi-
ties is likely to be effective in accomplishing the objectives of the project.
OCR for page 125
125
NIDRR’S PEER REVIEW PROCESS
(2) In determining the extent to which the design is likely to be effective in
accomplishing the objectives of the project, the Secretary considers the
following factors:
(i) The extent to which the research activities constitute a coherent, sustained
approach to research in the field, including a substantial addition to the
state-of-the-art (6 points).
(ii) The extent to which the methodology of each proposed research activity
is meritorious, including consideration of the extent to which—
(A) The proposed design includes a comprehensive and informed review of
the current literature, demonstrating knowledge of the state-of-the-art (5
points).
(B) Each research hypothesis is theoretically sound and based on current
knowledge (5 points).
(C) Each sample population is appropriate and of sufficient size (8 points).
(D) The data collection and measurement techniques are appropriate and
likely to be effective (8 points).
(E) The data analysis methods are appropriate (8 points).
(d) Design of dissemination activities (8 points total).
(1) The Secretary considers the extent to which the design of dissemination
activities is likely to be effective in accomplishing the objectives of the
project.
(2) In determining the extent to which the design is likely to be effective in
accomplishing the objectives of the project, the Secretary considers the
following factors:
(i) The extent to which the methods for dissemination are of sufficient quality,
intensity, and duration (4 points).
(ii) The extent to which the information to be disseminated will be accessible
to individuals with disabilities (4 points).
(e) Plan of operation (6 points total).
(1) The Secretary considers the quality of the plan of operation.
(2) In determining the quality of the plan of operation, the Secretary considers
the following factor:
(i) The adequacy of the plan of operation to achieve the objectives of the
proposed project on time and within budget, including clearly defined
responsibilities, and timelines for accomplishing project tasks (6 points).
(f) Collaboration (4 points total).
(1) The Secretary considers the quality of collaboration.
(2) In determining the quality of collaboration, the Secretary considers the
following factor:
(i) The extent to which the applicant’s proposed collaboration with one or
more agencies, organizations, or institutions is likely to be effective in
achieving the relevant proposed activities of the project (4 points).
continued
OCR for page 126
126 REVIEW OF DISABILITY AND REHABILITATION RESEARCH
BOX A4-1 Continued
(g) Adequacy and reasonableness of the budget (4 points total).
(1) The Secretary considers the adequacy and the reasonableness of the
proposed budget.
(2) In determining the adequacy and the reasonableness of the proposed
budget, the Secretary considers the following factors:
(i) The extent to which the costs are reasonable in relation to the proposed
project activities (2 points).
(ii) The extent to which the budget for the project, including any subcontracts,
is adequately justified to support the proposed project activities (2 points).
(h) Plan of evaluation (8 points total).
(1) The Secretary considers the quality of the plan of evaluation.
(2) In determining the quality of the plan of evaluation, the Secretary consid-
ers the following factors:
(i) The extent to which the plan of evaluation provides for periodic assess-
ment of progress toward—
(A) Implementing the plan of operation (4 points); and
(B) Achieving the project’s intended outcomes and expected impacts
(4 points).
(i) Project staff (10 points total).
(1) The Secretary considers the quality of the project staff.
(2) In determining the quality of the project staff, the Secretary considers the
extent to which the applicant encourages applications for employment
from persons who are members of groups that have traditionally been
underrepresented based on race, color, national origin, gender, age, or
disability (4 points).
OCR for page 127
127
NIDRR’S PEER REVIEW PROCESS
(3) In addition, the Secretary considers the following factors:
(i) The extent to which the key personnel and other key staff have appropri-
ate training and experience in disciplines required to conduct all proposed
activities (3 points).
(ii) The extent to which the commitment of staff time is adequate to accom-
plish all the proposed activities of the project (3 points).
(j) Adequacy and accessibility of resources (4 points).
(1) The Secretary considers the adequacy and accessibility of the applicant’s
resources to implement the proposed project.
(2) In determining the adequacy and accessibility of resources, the Secretary
considers the following factors:
(i) The extent to which the applicant is committed to provide adequate facili-
ties, equipment, other resources, including administrative support, and
laboratories, if appropriate (2 points).
(ii) The extent to which the facilities, equipment, and other resources are
appropriately accessible to individuals with disabilities who may use the
facilities, equipment, and other resources of the project (2 points).
NOTE: After the substantive review by the committee, but before publication of this report,
NIDRR changed the function of Part 350 subcriteria. Selection criteria from Part 350 continue
to use subcriteria but no longer include a breakdown of main criteria point values across the
subcriteria (similar to the criteria from Part 359).
SOURCE: NIDRR 2009 Application Kit for DRRP 84.133A-4, Center on the Effective Delivery
of Rehabilitation Technology by State Vocational Rehabilitation Agencies to Improve Employ-
ment Outcomes.
