Below are the first 10 and last 10 pages of uncorrected machine-read text (when available) of this chapter, followed by the top 30 algorithmically extracted key phrases from the chapter as a whole.
Intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text on the opening pages of each chapter.
Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.
Do not use for reproduction, copying, pasting, or reading; exclusively for search engines.
OCR for page 1
Summary
The National Institute on Disability and Rehabilitation Research
(NIDRR) is one of the principal federal agencies supporting applied re-
search, training, and development to improve the lives of individuals with
disabilities. NIDRR’s mission is to generate new knowledge and promote its
effective use in improving the ability of persons with disabilities to perform
activities of their choice in the community, as well as to expand society’s
capacity to provide full opportunities and accommodations for its citizens
with disabilities. Located within the Office of Special Education and Reha-
bilitative Services in the U.S. Department of Education (ED), NIDRR has
an annual budget for grants and contracts of approximately $109 million,
awarded through 14 separate program mechanisms that result in 1- to
5-year awards ranging in size from less than $100,000 to several million
dollars. NIDRR has the largest budget of the three primary federal agencies
with disability and rehabilitation as part of their mandate but also has the
broadest mandate. NIDRR aims to reach all disability types and age groups,
and its mission is tied to long-term outcomes such as independence, com-
munity participation, and employment.
Assessing the outcomes of research is a complex undertaking that can
variably take into account the stated goals of the research, the contribution
to the relevant field of research, the impact on the well-being of a particu-
lar population, or other related issues. The Government Performance and
Results Act has led to a particular emphasis on establishing specific perfor-
mance measures assessing the outputs of research programs.
NIDRR takes pride in proactively establishing program performance
measures focused on the quality of its grantee outputs, developing account-
1
OCR for page 2
2 REVIEW OF DISABILITY AND REHABILITATION RESEARCH
ability data systems to track the results of those measures, and developing
internal management systems to facilitate quality research. In 2009, NIDRR
requested that the National Research Council form a committee to conduct
a “process evaluation” of aspects of its grantmaking and a “summative
evaluation” of the quality of grantee outputs. In addition, the committee
was charged with assessing the methods it developed for conducting the
summative evaluation and making recommendations for the conduct of
future evaluations.1 The requested study was the most recent effort in a
series of NIDRR-funded activities aimed at assessing and improving the
agency’s performance.
PROCESS EVALUATION
NIDRR posed three questions specific to the process evaluation aimed
at assessing the process used for priority writing, practices for peer review
of grant applications, and the planning and budgetary processes used by
grantees. The development of priorities determines the areas of emphasis
for research and the specific topics to be targeted by potential applicants,
while peer review is a fundamental component of the grant selection process.
Although it is not possible to establish a clear causal link, these NIDRR
processes, as well as planning and budgetary processes used by grantees,
can influence the quality of the work produced by grantees.
To address these questions, the committee reviewed existing docu-
ments (e.g., legislation, Federal Register notices, NIDRR and ED policies
and procedures) and interviewed NIDRR management to obtain a more
thorough and cohesive understanding of these processes. The committee
gained additional insight into NIDRR’s peer review process by listening to
teleconferences held by three panels as they conducted their reviews of dif-
ferent grant competitions. In addition, the committee collected original data
through surveys of NIDRR staff, stakeholder organizations (other federal
agencies, professional associations, and advocacy organizations), NIDRR
peer reviewers, and principal investigators of NIDRR grants.
Priority Setting
To what extent is NIDRR’s priority-writing process conducted in such
a way as to enhance the quality of the final results?
As used in the study question, the term “priority-writing process” en-
compasses many aspects of priority setting, including gathering input from
1 This aspect of the committee’s charge was summarized in a letter report provided to NIDRR
in July 2011 and is also addressed in Chapter 6 of this report.
OCR for page 3
3
SUMMARY
multiple sources (e.g., the field, stakeholder organizations, grantees, other
agencies, and persons with disabilities and their families), identifying poten-
tial topics and determining priorities for funding, writing the proposed pri-
orities and having them cleared for release, and publishing notices inviting
applications (NIAs) on these priorities. The term “priority setting” is used
synonymously with “priority writing” in this report to clarify that the focus
of the committee’s evaluation included this larger priority-setting process.
The committee concluded that NIDRR’s long-range planning and
priority-setting processes are successful in producing grants that are aligned
with its mission and that stakeholders value as unique. Nonetheless, ar-
eas for improvement were identified. First, the committee concluded that
NIDRR needs to do more to inform and engage stakeholders with respect
to its long-range planning and priority-setting processes. Although the re-
sults of the stakeholder survey were generally positive, the transparency of
the processes, responsiveness to stakeholder feedback, and use of NIDRR
grantee products could be improved. Second, priority setting not only
reflects the agency’s intent to influence the advancement of research in tar-
geted areas but also offers specific funding opportunities for potential grant
applications to the agency. The announced priorities should therefore be
developed and communicated in a manner that attracts the best researchers
to participate in disability and rehabilitation research. Attracting the larg-
est pool of applications from which to select grantees increases the chances
for the highest quality outputs. When establishing its priorities, the agency
needs to consider continuity from one funding cycle to another, as well as
identify future research challenges and societal needs. The committee offers
recommendations in four areas to strengthen NIDRR’s long-range planning
and priority-setting processes.
Formation of an Advisory Council
NIDRR has a broad and diverse mission that makes it challenging to
set priorities that are responsive to the current state of the science and the
needs of the stakeholder community. Currently, NIDRR relies on staff, the
portfolio of existing projects, recent findings from completed grants, and
the current research literature, as well as guidance from federal partners, for
input to the priority-setting process. NIDRR’s statute directs it to establish
a standing Rehabilitation Research Advisory Council to advise the director
of the agency on research priorities and the development of the agency’s
Long-Range Plans. While NIDRR has formed ad hoc advisory bodies to
support the development of its Long-Range Plans, a standing body has never
been formed. Given NIDRR’s mandate, the council should be tasked with
providing advice on both disability and rehabilitation research.
OCR for page 4
4 REVIEW OF DISABILITY AND REHABILITATION RESEARCH
Recommendation 3-12: NIDRR should fulfill the statutory mandate
to form and utilize a standing disability and rehabilitation research
advisory council to advise on the priority-setting process and pro-
vide input for priority setting.
Most federal funding agencies, including the National Institutes of
Health (NIH), the National Science Foundation (NSF), and the National In-
stitute for Occupational Safety and Health (NIOSH), use standing advisory
bodies. A standing advisory body is likely to add stability and continuity
to both NIDRR’s long-range planning and its priority setting. The commit-
tee recognizes that NIDRR, like other federal research agencies, will face
challenges in capturing the broad diversity of perspectives held by its many
stakeholders. However, the committee feels strongly that, like other federal
research agencies, NIDRR can meet these challenges.
Strategic Planning
NIDRR’s multiple stakeholders include persons with disabilities and
their families, the scientific community, professional associations, and advo-
cacy organizations representing a variety of disability groups. In the face of
this diversity, it is important for the agency to have a consistent mechanism
for gathering information and input to inform the strategic planning process
beyond the input that will be possible through an advisory council. NIDRR
utilizes input from multiple sources, such as its stakeholders, other federal
agencies, the Interagency Committee on Disability Research (ICDR), the
current literature, and state-of-the-science conferences. However, the pro-
cesses for gathering input and developing proposed Long-Range Plans have
varied from one plan to another. Negative comments from the field gener-
ated by the last draft Long-Range Plan, coupled with the plan’s subsequent
delay, which caused NIDRR to operate under the prior plan for several years
beyond its intended time frame, suggest a breakdown in NIDRR’s priority-
setting process. The lack of a permanent director also hampers and delays
the agency’s priority-setting process.3
Recommendation 3-2: NIDRR should use a structured, consistent,
and inclusive strategic planning process to develop its Long-Range
Plans and priorities.
NIDRR might consider the long-range planning and priority setting pro-
cesses of other funding agencies, including NIH, NSF, and NIOSH, which
2 The
committee’s recommendations are numbered according to the chapter of the report in
which they appear.
3 At the time of this writing, a permanent NIDRR Director had been recently hired.
OCR for page 5
5
SUMMARY
have sought to integrate long-range planning and priority-setting processes
through specific initiatives such as the NIH Roadmap, the NSF Strategic
Plan, and the National Occupational Research Agenda.
Establishment of a Standard Calendar
For many program mechanisms, NIDRR has not established a regular
schedule for drafting and approving priorities and NIAs and disseminating
them to the field. ED has a lengthy review and approval process for obtain-
ing clearance for the release of priorities and NIAs. The variability in the
length of the clearance process may be an important factor, among others,
that impacts the timing of the release of NIAs. The irregular or delayed
release of NIAs may affect NIDRR’s ability to provide individuals sufficient
notice of grant opportunities or an optimal amount of time to complete ap-
plications. An irregular schedule may discourage the best investigators from
submitting applications. Additionally, certain program mechanisms (such as
Model Systems) include collaboration between institutions. Irregular post-
ing and shortened response times hamper the ability of applicants to identity
and recruit appropriate collaborators. These factors are likely to limit the
number of investigators who apply and adversely affect the quality of the
applications they submit. Additionally, young investigators less familiar with
NIDRR are more likely to pursue grants from other agencies.
Recommendation 3-3: NIDRR should utilize a standard calendar
for the setting of priorities, publication of notices inviting appli-
cations, submission of applications, and peer review meetings to
improve the efficiency of the process.
NIDRR has made efforts to standardize the schedule for NIAs. The
committee suggests that program mechanisms competed on a yearly basis
have a consistent annual schedule for the submission and review of applica-
tions. For multiyear grants, the committee recommends that NIDRR estab-
lish a long-range operational plan listing projected future grant application
submission dates, pending funding availability in that fiscal year.
Soliciting Applications
Like other federal agencies, NIDRR makes its NIAs available at Grants.
gov in addition to publishing them in the Federal Register. The agency
also uses a contractor to notify former grantees and others who, via their
webpage, express an interest in receiving NIAs. NIDRR would benefit from
more active efforts to solicit interest in its funding announcements.
OCR for page 6
6 REVIEW OF DISABILITY AND REHABILITATION RESEARCH
Recommendation 3-4: NIDRR should expand its efforts to dis-
seminate notices inviting applications to new potential applicants,
including developing a communication strategy to ensure that the
notices reach new audiences.
To expand dissemination, notices should be sent to the disability and re-
habilitation professional and research organizations that make up NIDRR’s
stakeholder network and to university departments and offices of sponsored
research. The latter could perhaps be accomplished through collaboration
with other federal research programs that regularly send funding notices to
universities.
Peer Review
To what extent are peer reviews of grant applications done in such a
way as to enhance the quality of final results?
NIDRR’s peer review process encompasses recruiting and training re-
viewers, conducting the review, and approving the awards. As with priority
setting, it is challenging to link the peer review process directly to specific re -
sults because the quality of the portfolio, grants, and outputs emerging from
the process is the product of multiple complex factors. It is clear, however,
that the peer review process used by NIDRR contributes significantly to the
success of the grant award program and the quality of the outputs produced.
The responses to the committee’s peer review survey were largely posi-
tive, including peer reviewers’ responses related to their experiences with
NIDRR’s peer review process and how it compares with the processes used
by other federal research agencies. While the committee concluded that
NIDRR’s peer review process is generally good, there are opportunities for
improvement that would likely enhance the process and the quality of final
results. The committee offers three recommendations to this end.
Enhancements to the Peer Review Process
The committee concluded that NIDRR’s peer review process is ham-
pered by a limited pool of potential reviewers. NIDRR’s competition
managers take great care to assemble and facilitate qualified review panels
and spend considerable time recruiting and screening potential reviewers.
Competition managers regularly must manage potential conflicts of inter-
est and rule out qualified reviewers. Despite these staff efforts, however,
the committee found evidence that a number of panels are smaller than
NIDRR’s recommended size, reviewers are added so close to the meeting
OCR for page 7
7
SUMMARY
date that they have inadequate time to prepare, and reviewers lacking neces-
sary scientific expertise may be participating in reviews.
The formation of formal “cohorts” of reviewers with particular areas
of expertise would reduce the recruiting burden on NIDRR staff in locat-
ing reviewers needed for individual competitions with specific targeted
expertise. In addition, reviewers surveyed by the committee reported that
the quality of the training they received was inconsistent; enhancing this
training would be a simple and effective way to improve the quality of the
review process. Finally, considerable variation exists among competitions in
the way NIDRR staff facilitate panel discussions. The result is variation in
the quality of the discussions; such inconsistency also can result in confusion
and negatively influence overall quality.
Recommendation 4-1: NIDRR should further strengthen the peer
review infrastructure by expanding the pool of high-quality review-
ers; establishing standing panels, or formal cohorts of peer review-
ers with specialized knowledge and expertise as appropriate for the
program mechanisms; enhancing reviewer training; and improving
the consistency of NIDRR staff facilitation of panel meetings and
the quality of feedback provided to grantees.
Reducing Reviewer Burden
Participating in NIDRR’s peer review process is a significant burden for
a large percentage of reviewers. Many reviewers spend more time than they
would like preparing, and the review days are long and intense. This sig-
nificant time commitment makes it less likely that qualified and experienced
reviewers will participate. Reviewers surveyed also reported sometimes hav-
ing insufficient time to review proposals, which could affect the quality of
the review discussions. The committee concluded that the review process is
so burdensome to peer reviewers as to threaten the quality of the process.
Recommendation 4-2: NIDRR should streamline the review process
in order to reduce the burden on peer reviewers.
Use of Consumer Peer Reviewers
To address its mission, NIDRR makes concerted efforts to include both
scientists with disabilities and consumers without scientific expertise in the
peer review process. Consumers can represent the experiences and views
of their particular disability communities and can evaluate applications for
relevance to their community’s needs and concerns.
OCR for page 8
8 REVIEW OF DISABILITY AND REHABILITATION RESEARCH
All reviewers, including researchers and consumers, should have the
appropriate expertise to review those elements of proposals to which they
are assigned. If consumers are to review scientific aspects of proposals, they
should have the relevant expertise, or NIDRR should consider providing
them with relevant methodological training. NIDRR should review and
monitor the role of consumers and researchers in peer review to ensure that
quality is not compromised.
Recommendation 4-3: NIDRR should continue to have consumer
representation in the peer review process and establish procedures
to guide the participation of those without scientific expertise.
Many federal research programs involve consumers without scientific
expertise in peer review. NIDRR may want to examine such practices at
NIH, Congressionally Directed Medical Research Programs, Juvenile Dia-
betes Research Foundation, and other agencies to inform its own approach
to including nonresearchers in peer review.
Grant Management
What planning and budgetary processes does the grantee use to pro-
mote high-quality outputs?
The committee assessed grantee planning and budgetary processes in the
larger context of NIDRR’s structure and processes supporting grant man-
agement. To perform this assessment, the committee (1) reviewed existing
documentation on the grant management and monitoring processes of ED
and NIDRR, (2) gathered information from principal investigators about
the processes they use for managing grants, and (3) interviewed NIDRR staff
to obtain their perspectives on how grant monitoring facilitates grantees’
efforts to manage their grants for successful results.
NIDRR appears to have developed a good plan for upgrading its routine
monitoring of grants and for identifying and monitoring grants that are at
risk of noncompliance with ED or NIDRR requirements and performance
expectations. On the whole, grantees appear to appreciate aspects of NI-
DRR’s grant management processes that facilitate their own grant manage-
ment strategies. While grantees generally commented that NIDRR’s grant
management processes were effective in facilitating their own grant manage-
ment processes, they offered some suggestions for improvement that would
help them further. NIDRR staff also offered suggestions for improvement,
focused on strengthening their capacity to monitor grants and help grant-
ees stay on course in implementing their grants and meeting performance
expectations. Among other suggestions, they expressed the need for smaller
OCR for page 9
9
SUMMARY
grant caseloads, additional travel funds for on-site monitoring of grants that
require higher levels of technical assistance, more training for new project
officers to promote consistency and quality in the monitoring process, and a
freer flow of communication between project officers and NIDRR planning
staff with respect to financial information.
Recommendation 5-1: NIDRR should continue to focus efforts on
improving its grant monitoring procedures and specific elements
of its overall grant management system that impact grantee-level
planning, budgets, and the quality of outputs.
From its interviews with grantees, the committee also learned that some
grants focused on developing technology innovations may not accord well
with a management template that calls for strict up-front planning and
adherence to original designs and timetables. Similarly, a grant funding a
large multisite study may require more or different supervision, monitor-
ing, and technical assistance than a more focused or limited study. Grantees
expressed the need for greater flexibility in grant management so they can
stay on the cutting edge of technology or adapt more easily to changing
needs of multisite research projects.
Recommendation 5-2: NIDRR should review the requirements
placed upon technical innovation grants and large multisite studies
to ensure that planning, reporting, supervisory, and technical as-
sistance requirements fit their particular circumstances.
To what extent are the results of the reviewed research and develop-
ment outputs used to inform new projects by both the grantee and NIDRR?
To assess how research and development outputs inform new projects,
the committee (1) reviewed information from NIDRR management about
how they use the results of their grantees’ research and (2) reviewed infor-
mation from grantees about new projects that have been generated from
their grants. The committee concluded that research and development
outputs are used to generate new projects by grantees to a great extent and
lead to substantial numbers of new collaborations with other researchers
and organizations, as well as transfers of data, instruments, or models to
other projects, and commercialization of technology products.
SUMMATIVE EVALUATION
The summative evaluation, designed to inform NIDRR’s performance
monitoring and reporting, involved assessing the quality of outputs pro-
OCR for page 10
10 REVIEW OF DISABILITY AND REHABILITATION RESEARCH
duced by a sample of 30 NIDRR grantees.4 These grants were drawn from
nine NIDRR program mechanisms: Burn Model System, Traumatic Brain
Injury Model System, Spinal Cord Injury Model System, Rehabilitation
Research and Training Center, Rehabilitation Engineering Research Cen-
ter, Disability and Rehabilitation Research Project-General, Field Initiated
Project, Small Business Innovation Research II, and Switzer Fellowship. The
committee reviewed four different types of outputs, as defined by NIDRR:
publications; tools, measures, and intervention protocols; technology prod-
ucts and devices; and informational products.
The committee developed and used four criteria to assess quality: (1)
technical quality; (2) advancement of knowledge or the field; (3) likely or
demonstrated impact (on science, persons with disabilities and their families,
provider practice, health and social systems, social and health policy, and
the private sector/commercialization); and (4) dissemination according to
principles of appropriate knowledge translation.
A total of 148 outputs produced by the 30 grantees were rated on each
criterion, using a 7-point scale, where 1 indicated poor quality, 4 indicated
good quality, and 7 indicated excellent quality. Ratings on each of the four
criteria were distributed fairly symmetrically along the scale, with the larg-
est proportion of scores falling at the midpoint of 4 and with most being
slightly skewed toward the higher end of the scale. Although close to 75
percent of the outputs rated fell in the “good to excellent” range of the
quality scale (i.e., mean ratings of 4 or greater on the 7-point quality scale),
25 percent of the outputs fell in the lower quality range (1 or “poor” to
3 or “below good”) across all four criteria. The committee offers NIDRR
two recommendations for assisting grantees in continuously improving the
quality of their outputs.
First, the quality of outputs is the product of multiple complex factors
that involve the priority-setting process, the funding level, the peer review
process, the quality of the proposed science/research and the grantees, and
ultimately the quality of the research findings. For grantees that are not
performing optimally, NIDRR may conduct ongoing formative reviews with
experts to identify strategies for improvement, increase its grant monitor-
ing activities, and require additional grantee reporting. Grantees generally
report that NIDRR’s oversight and reporting functions foster successful
grants and high-quality outputs by assisting them in adhering to their budget
and timeline, providing an external quality assurance mechanism for their
project management, and prompting them to maintain their focus on project
goals for high-quality products.
4 The committee performed a random sampling of grants at the level of program mechanism.
Five mechanisms were excluded in consultation with NIDRR.
OCR for page 11
11
SUMMARY
Recommendation 6-1: Although close to 75 percent of outputs were
rated as “good to excellent” (i.e., 4 or higher on the 7-point quality
scale), NIDRR should make it clear that it expects all grantees to
produce the highest quality outputs.
The intent of this recommendation is for NIDRR to encourage all of its
grantees to publish in peer-reviewed journals, present at national meetings,
publish/disseminate materials, and bring technology solutions to market
while producing these outputs at the highest levels of quality. To this end,
NIDRR should push forward by establishing clear and consistent expecta-
tions for grantees to publish in higher-impact journals as one indicator of
higher quality. For outputs other than publications, NIDRR should establish
standards for quality to be achieved and adopt appropriate metrics to as-
sess adherence to these standards. One way of setting the quality bar higher
would be to encourage grantees to use standardized reporting forms and
checklists5 for reviewing the technical quality of their own work before
subjecting it to external review.
Second, despite limitations in the use of bibliometrics,6 they are a valu-
able and objective set of measures that can be used in combination with
other assessment strategies. NIDRR has conducted bibliometric analyses
in the past, but has not routinely incorporated use of these metrics into its
performance measurement process.
Recommendation 6-2: NIDRR should consider undertaking biblio-
metric analyses of its grantees’ publications as a routine component
of performance measurement.
Bibliometric analyses would take advantage of an existing data source
for periodic measurement of the scientific impact of NIDRR grantee publi-
cations, and would provide an indicator of the extent to which these grant
outputs are being disseminated and used. This type of metric is being rec-
ommended for use in combination with other measures, just as it was used
in the committee’s evaluation along with expert review and supplemental
evidence of the impact an article may have had on science, persons with
disabilities and their families, provider practice, health and social systems,
social and health policy, and the private sector/commercialization.
5 Seehttp://www.equator-network.org/ for examples.
6 Common bibliometric measures include the impact factor of journals in which articles are
published and the number of times an article is cited in other articles.
OCR for page 12
12 REVIEW OF DISABILITY AND REHABILITATION RESEARCH
SELF-ASSESSMENT OF THE COMMITTEE’S METHODS
FOR OUTPUT REVIEW AND RECOMMENDATIONS
FOR FUTURE EVALUATIONS
The committee developed and implemented an evaluation process for
assessing the outputs of NIDRR’s grantees and identifying the various levels
of quality and characteristics of those outputs. The committee spent con-
siderable time selecting and refining the criteria used to assess the quality
of outputs. While some variation was evident in the independent scoring
among the committee members, it was rarely extreme, particularly after
group discussions. However, as summarized below, the committee encoun-
tered several challenges and obstacles during the course of its work that
limited the generalization of its findings and restricted what could be said
about the totality of outputs generated by all NIDRR grantees.
Defining Future Evaluation Objectives
The primary focus of the committee’s summative evaluation was on
assessing the quality of outputs produced by grantees; the evaluation did
not include in-depth examination or comparison of the larger contexts of
the funding programs, grants, or projects within which the outputs were
produced. However, the committee was asked to formulate an overall rating
for each grant based on the outputs reviewed and the information avail-
able about the grant from the Annual Performance Report (APR). Results
at the grant level were subject to limitations resulting from the general lack
of information about how the outputs did or did not interrelate; whether,
and if so how, grant objectives were accomplished; and the relative priority
placed on the various outputs. In addition, for larger, more complex grants,
such as center grants, a number of expectations for the grants, such as ca-
pacity building, dissemination, outreach, technical assistance, and training,
are unlikely to be adequately reflected in the committee’s approach, which
focused exclusively on specific outputs. The relationship of outputs to grants
is more complex than this approach could address.
Recommendation 6-3: NIDRR should determine whether as-
sessment of the quality of outputs should be the sole evaluation
objective.
Strengthening the Output Assessment
The committee was able to develop and implement a quantifiable expert
review process for evaluating the outputs of NIDRR grantees that was based
on criteria used in assessing research programs in both the United States and
other countries. With refinements, this method could be applied to the evalu-
OCR for page 13
13
SUMMARY
ation of future outputs even more effectively. Nonetheless, in implementing
this method, the committee encountered challenges and issues related to the
diversity of outputs, the timing of evaluations, sources of information, and
reviewer expertise.
Diversity of Outputs
There were acknowledged limitations in conducting the summative
evaluation, such as the inability to generalize the results because of the small
sample size, the need for more testing of the quality rating scale developed,
and possible biases that could have arisen from sampling and measure-
ment methods. In spite of these limitations, the quality rating system used
for the committee’s summative evaluation worked well for publications
in particular, which made up 70 percent of the outputs reviewed. Using
the four criteria outlined above, the reviewers were able to identify and
describe varying levels of quality and the characteristics associated with
each. However, the committee’s quality criteria were not as easily applied
to outputs such as websites, conferences, and interventions; these outputs
require more individualized criteria for assessing specialized technical ele-
ments, and sometimes more in-depth evaluation methods. Applying one set
of criteria, even though broad and flexible, could not guarantee sufficient
or appropriate applicability to every type of output.
Timing of Evaluations
The question arises of when best to perform an evaluation of outputs.
Evaluation of outputs during the final year of an award may not allow
sufficient time for the outputs to have full impact. For example, some pub-
lications will be forthcoming at this point, and others will not have had
sufficient time to have full impact. The trade-off of waiting a year or more
after the end of a grant before performing the evaluation is the likelihood
that staff involved with the original grant may not be available, recollec-
tion of grant activities may be compromised, and engagement or interest
in demonstrating results may be reduced. However, publications can be
tracked regardless of access to the grantee. Outputs other than publications,
such as technology products, could undergo an interim evaluation to enable
examination of the development and evolution of outputs.
Sources of Information
In addition to reviewing outputs directly, committee members consid-
ered information from two other sources in rating the quality of outputs:
information submitted through the grantee’s APR and information provided
OCR for page 14
14 REVIEW OF DISABILITY AND REHABILITATION RESEARCH
in a supplemental questionnaire developed by the committee. It is important
to note that both of these sources involved grantee self-reports, which could
be susceptible to social desirability bias. Moreover, the APR is designed as a
grant monitoring tool rather than as a source of information for a program
evaluation. Because the information supplied on the APR and the question-
naire was not always sufficient to inform the quality ratings, additional
methods are needed to ensure complete information for such reviews.
Reviewer Expertise
The committee was directed to assess the quality of four types of pre-
specified outputs. While the most common output type was publications,
NIDRR grants produce a variety of other outputs, including tools and
measures, technology devices and standards, and informational products.
These outputs vary widely in their complexity and the investment needed
to produce them. The criteria used by the committee to assess the quality
of outputs were based on the cumulative literature reviewed and the com-
mittee members’ own research expertise in diverse areas of disability and
rehabilitation research, medicine, and engineering, as well as their expertise
in evaluation, economics, knowledge translation, and policy. However, the
committee’s combined expertise did not include every possible content area
in the broad field of disability and rehabilitation research.
Recommendation 6-4: If future evaluations of output quality are
conducted, the process developed by the committee should be
implemented with refinements to strengthen the design related to
the diversity of outputs, timing of evaluations, sources of informa-
tion, and reviewer expertise.
Improving Use of the Annual Performance Report
The APR data set provided to the committee by NIDRR at the outset of
the evaluation was helpful in profiling the grants for sampling and in listing
all of the grantees’ projects and outputs. In addition, the narrative infor-
mation provided in the reports was useful to the committee in compiling
descriptions of the grants; however, they varied with respect to the quality
of the information they contained.
Recommendation 6-5: NIDRR should consider revising its APR to
better capture information needed to routinely evaluate the qual-
ity and impacts of outputs, grants, or program mechanisms. They
might consider efforts such as consolidating existing data elements
OCR for page 15
15
SUMMARY
or adding new elements to capture the quality criteria and dimen-
sions used in the present summative evaluation.
Recommendation 6-6: NIDRR should investigate ways to work
with grantees to ensure the completeness and consistency of infor-
mation provided in the APR.
CONCLUSION
In summary, the committee concluded that NIDRR grants have pro-
duced valuable research, tools, and other outputs for advancing the field
of disability and rehabilitation research in line with the agency’s mandate.
Improvements to NIDRR’s priority-setting, peer review, and grant manage-
ment processes, as well as consideration of alternative evaluation goals and
strategies, would further enhance the quality of these processes, their results,
and the agency’s efforts to improve the lives of individuals with disabilities.
OCR for page 16
