Below are the first 10 and last 10 pages of uncorrected machine-read text (when available) of this chapter, followed by the top 30 algorithmically extracted key phrases from the chapter as a whole.
Intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text on the opening pages of each chapter.
Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.
Do not use for reproduction, copying, pasting, or reading; exclusively for search engines.
OCR for page 17
1
Introduction
The National Institute on Disability and Rehabilitation Research
(NIDRR) is located within the Office of Special Education and Rehabilita-
tive Services in the U.S. Department of Education. NIDRR is one of mul-
tiple federal agencies that invest in disability and rehabilitation research,
but it has a broader mandate than those other agencies. Established by the
1978 amendments to the Rehabilitation Act of 1973,1 NIDRR’s mission
“is to generate new knowledge and promote its effective use to improve
the abilities of people with disabilities to perform activities of their choice
in the community, and also to expand society’s capacity to provide full
opportunities and accommodations for its citizens with disabilities” (Na-
tional Institute on Disability and Rehabilitation Research, 2006c, p. 8,167).
The agency pursues this mission by funding research and related activities
focused on maximizing the full inclusion, social integration, employment,
and independent living of individuals of all ages with disabilities (National
Institute on Disability and Rehabilitation Research, 2009a).
NIDRR prides itself on being proactive in establishing program perfor-
mance measures and developing accountability data systems to track the
progress of its grantees. An electronic annual reporting system is used to
collect data from grantees on many aspects of grant operation and outputs.
Various formative and summative evaluation approaches have been used to
assess the quality of the performance and results of the agency’s research
portfolio and its grantees. Prompted by the need to provide more data on
1 NIDRR was originally called the National Institute on Handicapped Research in the 1978
amendments to the act; its name was changed to its present form by the 1986 amendments.
17
OCR for page 18
18 REVIEW OF DISABILITY AND REHABILITATION RESEARCH
its program results, in 2009 NIDRR requested that the National Research
Council (NRC) conduct an external evaluation of some of the agency’s key
processes and assess the quality of outputs produced by NIDRR grantees
(National Institute on Disability and Rehabilitation Research, 2009a). This
report presents the results of that evaluation.
This chapter introduces the report by first explaining the charge to
the committee that conducted the evaluation. It then provides background
information on NIDRR, including its unique legislative mandate, the types
of research it funds and its grant funding mechanisms, its role related to
the Interagency Committee on Disability Research, and its budget and staff.
The third section summarizes approaches NIDRR has used in the past to
evaluate its grantees and distinguishes them from the methods used by the
committee. The final section provides an overview of the remaining chapters
of the report.
CHARGE TO THE COMMITTEE
This ad hoc committee, with oversight by the NRC’s Board on Human-
Systems Integration, was charged with developing and implementing a
framework and evaluation design for the purpose of (1) reviewing NIDRR’s
priority-writing and grant review processes (“process evaluation”) and (2)
assessing the quality of grantee outputs for a sample of grants representing
the NIDRR portfolio (“summative evaluation”). Additionally, the com -
mittee was charged with assessing the design and implementation of its
summative evaluation process and making recommendations for additional
evaluations that might follow this effort. The evaluation was to be con-
ducted over a period of 2 years between October 2009 and September 2011.
The results of this evaluation are intended to provide NIDRR with a better
understanding of the quality of its grantees’ outputs and how the agency
can best manage an important and evolving research portfolio that meets
its strategic goals and objectives while regularly assessing and improving
its performance as required by the Government Performance and Results
Act (GPRA) of 1993 (U.S. Office of Management and Budget, 2009a). The
specific questions guiding the evaluation were as follows:
Process Evaluation
1. To what extent is NIDRR’s priority-writing process conducted
in such a way as to enhance the quality of the final results?
2. To what extent are the peer reviews of grant applications done
in such a way as to enhance the quality of the final results?
3. What planning and budgetary processes does the grantee use to
promote high-quality outputs?
OCR for page 19
19
INTRODUCTION
Summative Evaluation
4. To what extent are the final outputs from NIDRR grants of high
quality?
5. To what extent are the results of the reviewed research and devel-
opment outputs used to inform new projects by both the grantee
and NIDRR?
This external evaluation was designed and conducted independently by
the NRC committee. However, it was funded by NIDRR, and the contract
stipulated in advance certain features and parameters for the design of the
summative evaluation component (study questions 4 and 5) as follows:
Level of analysis—The level of analysis was specified to be grantee
•
outputs. The outputs to be assessed included four types as defined
by NIDRR: (1) publications; (2) tools, measures, and intervention
protocols; (3) technology products and devices; and (4) informa-
tional products.
Sampling of grants—The sample for the evaluation was to include
•
a minimum of 30 grants. The selection criteria were to include rep-
resentation across all program mechanisms and grants in their last
year of funding.
Number of outputs to be reviewed—Two outputs were to be re-
•
viewed for each project being carried out under each grant selected
for the evaluation. The number of projects depended on the size of
the grant and varied from 1 for small investigator-initiated grants
to 10 for large center grants.
For NIDRR, the ultimate objective of the output review was to gain
an understanding of what was being produced by the time grants came to
an end with respect to value added for the disability community and new
knowledge produced for the field. Because the focus of the summative
evaluation was primarily on the quality of outputs produced by the sampled
grants, it is important to clarify what was not included in the committee’s
charge. The charge did not include a larger focus on how the grants were
implemented with respect to their original objectives and whether the ob-
jectives were achieved, how the outputs of the various projects were linked
and how they developed over time, and how the quality of outputs differed
across the various NIDRR program mechanisms. The charge also did not
include an assessment of the long-term impact of outputs on persons with
disabilities. However, suggestions are made by the committee on the impor-
tance of these types of evaluation foci in Chapter 6 of the report.
OCR for page 20
20 REVIEW OF DISABILITY AND REHABILITATION RESEARCH
BACKGROUND ON NIDRR
NIDRR’s purpose is to
. . . provide for research, demonstration projects, training, and related activi-
ties to maximize the full inclusion and integration into society, employment,
independent living, family support, and economic and social self-sufficiency of
individuals with disabilities of all ages . . .; promote the transfer of rehabilitation
technology to individuals with disabilities through research and demonstration
projects . . .; ensure the widespread distribution, in usable formats, of practical
scientific and technological information . . .; identify effective strategies that
enhance the opportunities of individuals with disabilities to engage in employ-
ment . . .; and increase opportunities for researchers who are members of
traditionally underserved populations, including researchers who are members
of minority groups and researchers who are individuals with disabilities (The
Rehabilitation Act of 1973, as amended).
NIDRR’s Unique Role
The multidimensional challenges faced by individuals with disabilities in
employment, housing, public accommodations, education, transportation,
communication, recreation, health services, institutionalization, voting, and
public services, as outlined in the Rehabilitation Act of 1973 as amended,
necessitate a comprehensive, holistic approach to NIDRR’s mission (Na-
tional Institute on Disability and Rehabilitation Research, 2009a). Across
NIDRR’s agenda, the central focus is on the whole person with a disability,
whose ability to function and quality of life are dependent on the complex
interactions among personal, societal, and environmental factors.
NIDRR is one of three major disability-focused research sponsors in the
federal government,2 but it plays a unique role in that its target population
includes all disability types and all age groups. (See the Rehabilitation Act
of 1973 as amended for a complete list of NIDRR’s mandated responsibili-
ties.) While other federal research entities fund prevention, cure, and acute
rehabilitation research, NIDRR also invests in rehabilitation research that
is tied more closely to longer-term outcomes, such as independence, com-
munity participation, and employment (National Institute on Disability
2 One of the other major sponsors is the National Center for Medical and Rehabilitation
Research (NCMRR), which is located within the National Institute of Child Health and Human
Development of the U.S. Department of Health and Human Services. NCMRR funds research
aimed at developing scientific knowledge to enhance the health, productivity, independence,
and quality of life of people with disabilities. The other major sponsor is the Department of
Veterans Affairs, which, through its Veterans Administration Rehabilitation Research and De-
velopment Service, funds extramural basic and biomedical research and an intramural research
program in 13 rehabilitation centers.
OCR for page 21
21
INTRODUCTION
and Rehabilitation Research, 2009b). The World Health Organization’s
International Classification of Functioning, Disability and Health (ICF)
(World Health Organization, 2001) provides context for understanding
NIDRR-supported research and development. The ICF classifies disability
and health along a continuum from body function and structure to activities
and participation, while accounting for environmental and personal factors.
In these terms, NIDRR funds primarily research and development grants
that are aimed at increasing functional abilities to facilitate greater self-
determination and participation of individuals with disabilities in the home,
community, school, and workplace as defined in the ICF (National Institute
on Disability and Rehabilitation Research, 2009b).
NIDRR’s Research Domains
Table 1-1 summarizes the various types of research that NIDRR funds
in five outcome domains: (1) employment, (2) participation and community
living, (3) health and function, (4) technology for access and function, and
(5) disability demographics (National Institute on Disability and Rehabili-
tation Research, 2009a, pp. 3-6). Although most grants address more than
TABLE 1-1 NIDRR Research Domains and Topics
Number of
Research Awards as of
Domain January 2009 Research Topics
Employment 16 (7%) • areer planning; job entry, advancement,
C
and retention; transitions in moving from
financial dependency to self-sufficiency or from
underemployment into work that is consistent with
the individual’s strengths, abilities, and interests.
• ethods for integrating the unique needs of
M
employers and disability populations to improve
employment outcomes across the life span, such
as methods, costs, and results of services by
rehabilitation programs or supported employment,
including studies of natural supports at work as
they relate to employment outcomes.
• he role of personal assistance services in the
T
workplace and the application of rehabilitation
technology, universal design principals,
environmental adaptations, and engineering
solutions to enhance personal function and address
barriers confronted in employment by people with
disabilities.
continued
OCR for page 22
22 REVIEW OF DISABILITY AND REHABILITATION RESEARCH
TABLE 1-1 Continued
Number of
Research Awards as of
Domain January 2009 Research Topics
Participation 39 (17%) • evelopment and evaluation of strategies for
D
and Community services, interventions, products, and modifications
Living to the built and social environments that would
allow individuals with all types of disabilities to
live and participate in their communities.
• evelopment of new and improved theories about
D
and measures of participation and community
living that will enable the impact of specific
strategies and interventions to be determined more
accurately.
Health and 93 (39%) Individual level:
Function
• ngoing research and clinical efforts to produce
O
a wide variety of programs, interventions, and
products aimed at enhancing the health and
function of individuals with disabilities.
• tudy of new technologies that can improve the
S
diagnosis and measurement of disabling conditions
and study of devices to support enhanced function.
• esearch that can help prevent secondary conditions
R
from developing among people with disabilities and
explore the implications of nonmedical interventions,
such as exercise, in this context.
Systems level:
• tudy of system-level policies and practices
S
that exacerbate or ameliorate disability-related
disparities in access to health care services.
• ccurate assessment of the health status
A
of individuals with disabilities to increase
understanding of the impact of the health care
delivery system on their health and wellness.
• tudies to classify specific interventions in medical
S
rehabilitation so as to better define and measure
the effectiveness of the multitude of rehabilitation
interventions.
OCR for page 23
23
INTRODUCTION
TABLE 1-1 Continued
Number of
Research Awards as of
Domain January 2009 Research Topics
Technology 81 (34%) Individual level:
for Access and
Function • ocuses primarily on assistive technology devices
F
that enhance the physical, sensory, and cognitive
abilities of people with disabilities and assist them
in participating and functioning more independently
in the home, at work, in recreational settings, and at
cultural and community events.
Systems level:
• ocuses on applying technology research and
F
development in ways that enhance community
integration, independence, productivity,
competitiveness, and equal opportunity by mitigating
or eliminating barriers found in large social systems,
such as public transportation, telecommunications,
information technology, and the built environment.
• xploration of how recent, breakthrough advances
E
in biomaterials research, composite technologies,
information and telecommunication technologies,
nanotechnologies, micro electro-mechanical systems
(MEMS), sensor technologies, and the neurosciences
can be incorporated into future rehabilitation-related
technology research.
Disability 8 (3%) • tudies to generate new information that can
S
Demographics be used by individuals with disabilities, service
providers, policy makers, and others working to
identify and eliminate disparities in employment,
participation and community living, and health
and function.
• tudies that mine data to address the full range of
S
social, health, and economic facets of disability
and that compare findings across data sources.
• esearch to understand the variances in levels of
R
participation by individuals with disabilities and to
evaluate strategies or interventions that may help
bridge the gap between preference and feasibility in
an existing environment.
• stablishment of a center on disability
E
demographics and statistics.
Total Number 237
of Awards as of
January 2009
SOURCE: National Institute on Disability and Rehabilitation Research (2009a, pp. 3-6).
OCR for page 24
24 REVIEW OF DISABILITY AND REHABILITATION RESEARCH
one domain, the second column of the table notes the number of grants in
NIDRR’s portfolio as of January 2009 whose primary research domain was
in each of these five areas.
NIDRR Grant Mechanisms
NIDRR has eight primary mechanisms for awarding grants defined
by the Catalog of Federal Domestic Assistance (CFDA) (National Institute
on Disability and Rehabilitation Research, 2009a, pp. 7-29). Using these
primary mechanisms, funds are distributed through 14 separate program
mechanisms, which are described below.
1. Disability and Rehabilitation Research Project (DRRP; CFDA #84.133A).
DRRPs are awarded through six separate program mechanisms:
• Disability and Business Technical Assistance Center (DBTAC),
which comprises a network of 10 regional centers and one Coor-
dination Outreach and Research Center that provide information
and referral, technical assistance, public awareness, and training on
all aspects of the Americans with Disabilities Act (ADA).3
• Traumatic Brain Injury Model System (TBIMS), whose aim is to
demonstrate the benefits of a coordinated system of neurotrauma
and rehabilitation care and to conduct innovative research on all
aspects of care for those who sustain traumatic brain injuries. The
mission of the TBIMS program is to improve the lives of persons
who experience traumatic brain injury by creating and disseminat-
ing new knowledge about the course, treatment, and outcomes
relating to their condition.
• Disability and Rehabilitation Research Project-General (DRRP),
which funds projects with special emphasis on research, demonstra-
tions, training, dissemination, utilization, and technical assistance.
Projects may include combinations of these activities. These projects
may develop methods, procedures, and rehabilitation technology
to assist in achieving the full inclusion and integration into society,
employment, independent living, family support, and economic
and social self-sufficiency of individuals with disabilities, especially
individuals with the most severe disabilities, or to improve the ef-
fectiveness of services authorized under the Rehabilitation Act.
• Knowledge Translation (KT), whose projects range from inves-
tigating models, methods, strategies, and mechanisms that could
3 Although still funded as DBTACs, these centers are currently referred to as ADA National
Network Centers.
OCR for page 25
25
INTRODUCTION
contribute to the successful knowledge translation of disability and
rehabilitation research and products to synthesizing, disseminat-
ing, and promoting the use of existing knowledge and products to
improve the lives of individuals with disabilities.
• Burn Model System (BMS), which comprises centers that establish
innovative projects for the delivery, demonstration, and evaluation
of comprehensive medical, vocational, and other rehabilitation
services to meet the wide range of needs of individuals with burns.
• Section 21, which focuses on research capacity building for minor-
ity entities such as historically black colleges and universities and
institutions serving primarily Hispanic, Asian, and American Indian
students and nonminority entities with an interest in improving
understanding about the needs and outcomes of individuals with
disabilities from minority populations. Program activities include
assisting minority entities with networking that supports enhanced
collaboration between minority and nonminority entities and the
exchange of expertise and advanced training across program areas.
2. Rehabilitation Research and Training Center (RRTC; CFDA #84.133B).
RRTCs conduct coordinated and integrated advanced research aimed
at alleviating or stabilizing disabling conditions, promoting maxi -
mum social and economic independence of people with disabilities,
or improving rehabilitation methodology or service delivery systems.
RRTCs operate in collaboration with institutions of higher education
and providers of rehabilitation services and serve as national centers of
excellence in rehabilitation research.
3. Rehabilitation Engineering Research Center (RERC; CFDA #84.133E).
RERCs conduct programs of advanced engineering and technical re-
conduct re-
search designed to apply technology, scientific achievement, and psy-
chological and social knowledge to solve rehabilitation problems and
remove environmental barriers. RERCs are affiliated with institutions
of higher education or nonprofit organizations.
4. Switzer Fellowship (CFDA #84.133F). The Switzer program gives
individual researchers an opportunity to develop new ideas and gain
research experience. Fellows design and work for 1 year on an indepen-
dent research project.
5. Field Initiated Project (FIP; CFDA #84.133G). The FIP program pro-
vides funding to individual researchers to address rehabilitation issues
in promising and innovative ways. FIPs are of two types—Field Initiated
Projects-Research (FIR) and Field Initiated Projects-Development (FID).
6. Spinal Cord Injury Model System (SCIMS) (CFDA #84.133N). SCIMS
centers study the course of recovery and outcomes following the delivery
of a coordinated system of care for individuals with spinal cord injuries.
OCR for page 26
26 REVIEW OF DISABILITY AND REHABILITATION RESEARCH
The centers under this program provide comprehensive rehabilitation
services to individuals with such injuries and conduct spinal cord re-
search, including clinical research.
7. Advanced Rehabilitation Research Training (ARRT; CFDA #84.133P).
The ARRT program provides funding to institutions of higher education
to recruit qualified postdoctoral candidates with clinical, management,
basic, or engineering research experience and prepare them to conduct
independent research on disability and rehabilitation issues.
8. Small Business Innovation Research (SBIR; CFDA #84.133S). SBIR
grants, administered by NIDRR as a part of the larger mandatory SBIR
program, help support the production of new assistive and rehabilita-
tion technology. This two-phase program takes a rehabilitation-related
product from development to market readiness. Phase I grants are de-
signed to demonstrate proof of concept, while Phase II grants develop
products envisioned in Phase I grants.
In addition to these program mechanisms, contracts are awarded to
provide technical support for NIDRR’s internal management, research, and
knowledge translation activities.
Interagency Collaboration
Efforts have been made to develop and coordinate a coherent program
of disability and rehabilitation research across the many federal agencies
involved in such research. Yet coordination is difficult to accomplish because
of differences in agency missions and organizational cultures (e.g., medical
model versus social or environmental model), competitive budget processes,
weak to nonexistent incentives for coordination and collaboration, and
separately constructed long-range strategic plans within each agency (Insti-
tute of Medicine, 2007).
The Rehabilitation Act of 1973 authorized the Interagency Committee
on Disability Research (ICDR), a federal interagency committee chaired by
the NIDRR Director. The ICDR is mandated to promote coordination and
cooperation among the many federal departments and agencies conducting
disability and rehabilitation research programs. The committee comprises
presidential designees, including the following (or their designees): the
Director of NIDRR, the Commissioner of the Rehabilitation Services
Administration, the Assistant Secretary for the Office of Special Educa-
tion and Rehabilitative Services, the Secretary of Education, the Secretary
of Veterans Affairs, the Director of the National Institutes of Health, the
Director of the National Institute of Mental Health, the Administrator
of the National Aeronautics and Space Administration, the Secretary of
Transportation, the Assistant Secretary of the Interior for Indian Affairs,
OCR for page 27
27
INTRODUCTION
the Director of the Indian Health Service, and the Director of the National
Science Foundation. The role of ICDR is to identify and provide oppor-
tunities for information sharing and partnerships on various initiatives in
which the agencies are engaged. The ICDR works to cross-fertilize ideas
and promote dialogue, but does not prescribe activities to its members.4
Its activities are organized and undertaken through the structure of five
subcommittees addressing medical rehabilitation, employment, technol-
ogy, education, and disability statistics.
NIDRR Budget
The Rehabilitation Act states that NIDRR’s Director will lay out fund-
ing priorities and covered activities in a 5-year plan that will be published
in the Federal Register and that the plan will dedicate at least 90 percent
of NIDRR funds to extramural research. Additionally, Section 21 of the
Rehabilitation Act mandates that 1 percent of NIDRR appropriations be
set aside to address traditionally underserved populations. Finally, the SBIR
Development Act of 2000 requires agencies with research and development
budgets in excess of $100 million, such as NIDRR, to set aside a certain
percentage of their funds for SBIR. The current set-aside is 2.5 percent of
the research and development budget.
NIDRR’s 2009 budget for grants and contracts was $109 million. Of
this total, $99,904,000 (U.S. Department of Education, 2009) was dedi-
cated to grants, with the remainder going to contracts and project support.
Figure 1-1 illustrates NIDRR’s distribution of its funds for grants across all
of the program mechanisms in 2009 (National Institute on Disability and
Rehabilitation Research, 2009a).
NIDRR’s annual budget for research and development grants has re-
mained fairly constant in nominal terms during the last decade. As stated
above, the budget for grants in 2009 was nearly $100 million; in 2008
and 2007 it was approximately $97 million (U.S. Department of Educa-
tion, 2008, 2009). Previously, a report of the Institute of Medicine (2007)
showed that the agency’s overall funding had been fairly steady since 2002.
Although inflation has been reasonably low in recent years, steady funding
in nominal terms implies a reduction in the research that can be conducted.
NIDRR Staff
NIDRR’s most recent organizational chart (National Institute on Dis-
ability and Rehabilitation Research, 2011) shows an acting Director,5 an
4 S. Swenson and C. Pledger, personal communication, April 2011.
5 At the time of this writing, a permanent NIDRR Director had been recently hired.
OCR for page 28
28 REVIEW OF DISABILITY AND REHABILITATION RESEARCH
ARRT- 2%
BMS- 2%
Switzer- 1%
Section 21- 1% TBIMS- 10%
DBTAC- 12%
SCIMS- 9%
DRRP- 7%
SBIR- 4%
FIP- 13%
RRTC- 18%
RERC- 17%
KT- 4%
FIGURE 1-1 Distribution of NIDRR funds across program mechanisms, fiscal year
2009.
NOTE: ARRT = Advanced Rehabilitation Research Training, BMS = Burn Model
System, DBTAC = Disability and Business Technical Assistance Center, DRRP =
Disability and Rehabilitation Research Project-General, FIP = Field Initiated Project,
KT = Knowledge Translation, RERC = Rehabilitation Engineering Research Center,
RRTC = Rehabilitation Research and Training Center, SBIR = Small Business
Innovation Research, SCIMS = Spinal Cord Injury Model System, TBIMS = Traumatic
Brain Figure 1-1, editable
Injury Model System.
SOURCE: National Institute on Disability and Rehabilitation Research (2009a, p. 32).
acting Deputy Director, and two main divisions—Research Sciences and
Program, Budget, and Evaluation. Both divisions have directors and su-
pervisors. In the Research Sciences Division, 14 positions are classified as
rehabilitation specialists; these are the project officers who interact directly
with grantees and perform grant monitoring functions. The organizational
chart shows four of these positions to be vacant. The Program, Budget, and
Evaluation Division has 10 other staff positions.
OCR for page 29
29
INTRODUCTION
HISTORY OF NIDRR ASSESSMENTS
Assessments of NIDRR have included a 2005 performance assessment
under the GPRA and Annual Performance Assessment Expert Reviews
conducted in 2005 and 2006.
NIDRR Performance Assessment Under GPRA
In compliance with the GPRA (U.S. Office of Management and Budget,
2009b), NIDRR developed the following performance measures to assess its
key outcomes (National Institute on Disability and Rehabilitation Research,
2009a, pp. 78-79):
• The percentage of NIDRR-supported fellows, post-doctoral trainees, and
doctoral students who publish results of NIDRR-sponsored research in
refereed journals
• The number of accomplishments (new or improved tools, methods, discov-
eries, standards, interventions, programs, or devices) developed or tested
with NIDRR funding that has been judged by expert panels to be of high
quality and to advance the field
• The percentage of NIDRR-funded grant applications that receive an aver-
age peer review score of 85 or higher
• The percentage of NIDRR grants that assess the effectiveness of interven-
tions, programs, and devices using rigorous methods
• The number of new or improved NIDRR-funded assistive and universally
designed technologies, products, and devices, transferred to industry for
potential commercialization
• The average number of publications per award based on NIDRR-funded
R&D activities in refereed journals
• The percentage of NIDRR competitions announced by October 1
• The percentage of NIDRR grant awards issued within 6 months of the
competition closing date
• The percentage of NIDRR-funded research projects identified as having
an employment focus (This is a measure required by the Department of
Education, Office of Special Education and Rehabilitative Services, not a
GPRA measure)
Under the GPRA requirements at the time,6 NIDRR’s 2005 Program
Assessment Rating Tool (PART) review showed excellent scores on Pro-
6 PART was developed and used by the Office of Management and Budget to assess the per-
formance of federal programs and to identify actions that could improve results (http://www.
whitehouse.gov/omb/performance_past [October 24, 2011]). Expectations for performance
measurement are currently being amended by the GPRA Modernization Act of 2010 (http://
www.whitehouse.gov/omb/performance/gprm-act [October 24, 2011]).
OCR for page 30
30 REVIEW OF DISABILITY AND REHABILITATION RESEARCH
gram Purpose and Design (100 percent), Strategic Planning (90 percent),
and Program Management (90 percent). However, Program Results and
Accountability was rated ineffective (42 percent).7
NIDRR’s Annual Performance Assessment Expert Reviews
NIDRR’s Annual Performance Assessment Expert Review process was
implemented in 2005 and 2006 to evaluate the agency’s portfolio of grants
in three areas: (1) health and function, (2) employment, and (3) technology.
The objectives of the review were to provide an independent assessment
of the quality and relevance of NIDRR-funded research and the extent to
which the research outputs and outcomes were contributing to the agency’s
long-term performance goals and measures; the quality and relevance of the
agency’s management of research directions and award decisions; and the
strengths and weaknesses of the three research portfolios, including recom-
mendations to ensure the accomplishment of NIDRR’s goals and objec-
tives (National Institute on Disability and Rehabilitation Research, 2006a,
2006b, 2007). To conduct the review, NIDRR assembled three panels of
expert reviewers, one to review each portfolio.
As with the present evaluation, grantee outputs were reviewed as part
of the portfolio evaluation. Grantees submitted up to five accomplishments,
defined as “outputs” or “outcomes,” that they considered to be the most
important that occurred or matured during the current reporting period.
“Outputs” included publications or presentations of significant findings;
products, including tools, devices, and written products; and services com-
pleted as part of training or capacity building. “Outcomes” could be either
short term (referring to documented changes or advances in knowledge, un-
derstanding, or skills) or intermediate (referring to documented changes in
policy, practice, behavior, or systems capacity). The accomplishments were
scored on a scale of 1 to 3, with 1 being “little-to-no contribution to the
field,” 2 being a “substantial contribution,” and “3” being an “outstanding
contribution.” A rating of 0 was used for “unable to determine.”
Key findings from the three reports include the following:
Health and function (October 2006) (National Institute on Disabil-
•
ity and Rehabilitation Research, 2007) (included multiple program
mechanisms): The panel commended NIDRR on its long-standing
productive portfolio in health and function research, and recog-
nized the agency’s success in building infrastructure and capacity
7 ExpectMore.gov: National Institute on Disability and Rehabilitation Research, page 8 of 15
(http://www.whitehouse.gov/omb/expectmore/detail/10001041.2005.html [April 30, 2010]).
OCR for page 31
31
INTRODUCTION
for disability and rehabilitation research. The panel noted several
achievements of grantees, but felt limited in making judgments
based on information covering only 1 year or less of multiyear
awards. The panel rated 67 percent of the grantee accomplishments
in this area as substantial or outstanding.
Employment (September 2005) (National Institute on Disability
•
and Rehabilitation Research, 2006a) (included the RRTC, DRRP,
and FIP program mechanisms): Overall, the panel concluded that
the portfolio contained very high-quality research ideas. However,
the panel expressed four concerns: “(1) the apparent lack of sci-
entific rigor behind the identified outputs, (2) the lack of sufficient
information on the methodologies used by grantees, (3) the lack
of evidence supporting many of the claims made by grantees in
their Supplemental Information Reports, and (4) the lack of peer-
reviewed publications” (p. 4). The panel was unable to rate the
grantee accomplishments given the limited information it received.
Technology (October 2005) (National Institute on Disability and
•
Rehabilitation Research, 2006b) (included the RERC, DRRP, and
FIP program mechanisms): The panel identified significant accom-
plishments in each cluster area. However, given NIDRR’s level of
investment, the panel stated that the overall number of high-quality
research and development outputs and outcomes that advanced
scientific knowledge was inadequate. The panel also noted that
many of the claims made in the reports it reviewed lacked sufficient
supporting evidence. The panel rated 54 percent of the grantee ac-
complishments as substantial or outstanding.
Methods used in the summative component of the present evaluation
were somewhat similar to those used in the Annual Portfolio Assessment
Expert Reviews with regard to the assessment of accomplishments, but
can be distinguished from the latter in several ways. First, this committee
reviewed only the category of “outputs.” Second, the committee focused on
the quality of outputs. Therefore, instead of assigning one rating for the ex-
tent to which the outputs contributed to the field as was done in the Annual
Portfolio Assessment Expert Reviews, the committee examined and rated
each output in a multidimensional way using four criteria: technical qual-
ity, advancement of knowledge or the field, likely or demonstrated impact,
and dissemination. Third, the committee itself served as the expert panel of
reviewers. Finally, the committee assessed the methods used in its summative
evaluation and made recommendations for improving future evaluations of
outputs from NIDRR research. A full description of the methods used to
assess outputs is presented in Chapter 6.
OCR for page 32
32 REVIEW OF DISABILITY AND REHABILITATION RESEARCH
ORGANIZATION OF THE REPORT
Chapter 2 describes the scope of the committee’s evaluation and the meth-
ods used. Chapters 3, 4, and 5, respectively, present results of the evaluation
of NIDRR’s priority-setting, peer review, and grant management processes.
The chapters summarize NIDRR’s policies and procedures with regard to
these key processes and present the committee’s findings based on information
gathered from such sources as interviews with NIDRR staff; questionnaires
completed by grantees; and surveys of NIDRR stakeholder organizations,
other federal agencies, and NIDRR peer reviewers. Chapter 6 summarizes the
findings of the evaluation of grantee outputs, based on a sample of 30 grants,
and presents the assessment of the committee’s summative evaluation process.
Conclusions and recommendations for improving NIDRR’s key processes
and the quality of grantee outputs and for conducting future evaluations are
included at the end of Chapters 3 through 6.
The report’s appendixes provide background information. Appendix A
contains summaries of the 30 grants whose outputs were evaluated by the
committee; these summaries include descriptions of the outputs reviewed.
Appendix B contains the questionnaires and rating sheets used in the com-
mittee’s process and summative evaluations. Appendix C lists the acronyms
used in this report. Finally, Appendix D contains biographical sketches of
the committee members and NRC staff who participated in the study.
REFERENCES
Institute of Medicine. (2007). The future of disability in America. Washington, DC: The Na-
tional Academies Press.
National Institute on Disability and Rehabilitation Research. (2006a). Annual portfolio as-
sessment expert review pilot employment portfolio panel summary report. Available:
http://www.naric.com/nidrr/APAER_Employ_Final-Revised1-4-06.pdf [March 21, 2011].
National Institute on Disability and Rehabilitation Research. (2006b). Annual portfolio
assessment expert review pilot technology portfolio panel summary report. Available:
http://www.naric.com/nidrr/APAERTechnologyReport-FinalRevised-1-4-06.pdf [March
21, 2011].
National Institute on Disability and Rehabilitation Research. (2006c). Department of Edu-
cation: National Institute on Disability and Rehabilitation Research—Notice of Final
Long-Range Plan for Fiscal Years 2005–2009. Federal Register, 71(31), 8,166-8,200.
National Institute on Disability and Rehabilitation Research. (2007). Annual performance
assessment expert review health and function research. Available: http://www.ncddr.
org/new/announcements/apaer/hf/HF-APAER2006RptPub7-5-07.pdf [March 21, 2011].
National Institute on Disability and Rehabilitation Research. (2009a). Briefing book for The
National Academies. Unpublished document. Washington, DC: National Institute on
Disability and Rehabilitation Research.
National Institute on Disability and Rehabilitation Research. (2009b). Department of Educa-
tion: National Institute on Disability and Rehabilitation Research—Notice of Proposed
Long-Range Plan for Fiscal Years 2010–2014. Federal Register, 74(10), 2,564-2,569.
OCR for page 33
33
INTRODUCTION
National Institute on Disability and Rehabilitation Research. (2011). NIDRR organizational
chart. Unpublished document. Washington, DC: National Institute on Disability and
Rehabilitation Research.
The Rehabilitation Act of 1973, as amended. Pub. L. No. 93-112. Available: http://www2.
ed.gov/policy/speced/reg/narrative.html [January 21, 2011].
U.S. Department of Education, Office of Special Education, Rehabilitative Services, Reha-
bilitation Services Administration. (2008). Annual report on federal activities under the
Rehabilitation Act, Fiscal Year 2008. Washington, DC: U.S. Department of Education.
U.S. Department of Education, Office of Special Education, Rehabilitative Services, Reha-
bilitation Services Administration. (2009). Annual report on federal activities under the
Rehabilitation Act, Fiscal Year 2009. Washington, DC: U.S. Department of Education.
U.S. Office of Management and Budget. (2009a). Government Performance and Result Act.
Available: http://www.whitehouse.gov/omb/mgmt-gpra/gplaw2m.html [March 20, 2009].
U.S. Office of Management and Budget. (2009b). Assessing program performance. Available:
http://whitehouse.gov/omb/part/ [March 20, 2009].
World Health Organization. (2001). ICF: International Classification of Functioning, Disabil-
ity and Health. Geneva, Switzerland: World Health Organization.
