Below are the first 10 and last 10 pages of uncorrected machine-read text (when available) of this chapter, followed by the top 30 algorithmically extracted key phrases from the chapter as a whole.
Intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text on the opening pages of each chapter.
Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.
Do not use for reproduction, copying, pasting, or reading; exclusively for search engines.
OCR for page 56
3
NIDRR’s Priority-Setting Processes
This chapter addresses the following key study question:
Key Question #1. To what extent is NIDRR’s priority-writing process
conducted in such a way as to enhance the quality of the final results?
As used in the study question, the term “priority-writing” process en-
compasses many aspects of priority setting, including gathering input from
multiple sources (e.g., the field, stakeholder organizations, grantees, other
agencies, and persons with disabilities and their families), identifying poten-
tial topics and determining priorities for funding, writing the proposed pri-
orities and having them cleared for release, and publishing notices inviting
applications (NIAs)1 on these priorities.2 The term “priority setting” is used
synonymously with “priority writing” in this report to clarify that the focus
of the committee’s evaluation was on this larger priority-setting process.
Priority setting not only reflects the National Institute on Disability and
Rehabilitation Research’s (NIDRR’s) intent to influence the advancement of
research in targeted areas but also offers specific funding opportunities for
potential grant applications to the agency. The announced priorities should
therefore be developed and communicated in a manner that attracts the
best researchers and encourages participation in disability and rehabilita-
1A “notice inviting applications” (NIA) is NIDRR’s announcement to the field of the open-
ing of a grant competition. It is similar to the terms “request for proposals” and “request for
applications” used by other agencies.
2 “Priority” in this context means the topic of the grant competition for which scientists
submit applications to investigate.
56
OCR for page 57
57
NIDRR’S PRIORITY-SETTING PROCESSES
tion research. Attracting the largest pool of applications from which to
select grantees increases the chances for the highest-quality outputs. When
establishing its priorities, the agency needs to consider continuity from one
funding cycle to another, as well as identify future research challenges and
societal needs.
In the context of this committee’s work, it is challenging to link priority
setting directly to specific output quality. The quality of NIDRR’s research
portfolio, grants, and outputs is the product of multiple complex factors, in-
cluding the priority-setting process, funding levels, the peer review process,
and the scientific quality of the grantees. It is clear, however, that NIDRR’s
priority-setting process has a positive impact on the quality of the final out-
puts. In this chapter, the committee reviews and assesses that process and
makes recommendations for its improvement.
The chapter has three major sections. The first describes NIDRR’s
priority-setting process based on existing documentation and interviews
with NIDRR executives. The second presents an assessment of the process,
based on data gathered from NIDRR staff, grantees, and stakeholder orga-
nizations. The final section offers the committee’s conclusions and recom-
mendations with respect to NIDRR’s priority-setting process.
DESCRIPTION OF NIDRR’S PRIORITY-SETTING PROCESS
The following description is based on existing documentation, such as
legislation, the Federal Register, NIDRR and the U.S. Department of Educa-
tion (ED) policies and procedures, NIDRR’s Long-Range Plan (LRP), and
NIAs, as well as interviews with NIDRR and ED management.3
Legislative Foundation
The Rehabilitation Act (1973, as amended) authorizes the formula grant
programs of vocational rehabilitation, supported employment, independent
living, and client assistance. Title II4 of the act—Research and Training—au-
thorizes the majority of NIDRR’s research activities, while Section 21 of the
Act and the Small Business Innovation Research (SBIR) Development Act
(2000) require NIDRR to reserve small portions of its budget for those two
specific program mechanisms. Section 202 of Title II states that NIDRR’s
Director will lay out funding priorities and covered activities in a 5-year plan
3 The committee conducted interviews with NIDRR and ED management in four sessions
during summer 2010 and one session in spring 2011.
4 The other six titles are Title I, Vocational Rehabilitation Services; Title III, Professional
Development and Special Projects and Demonstrations; Title IV, National Council on Disabil-
ity; Title V, Rights and Advocacy; Title VI, Employment Opportunities for Individuals with
Disabilities; and Title VII, Independent Living Services and Centers for Independent Living.
OCR for page 58
58 REVIEW OF DISABILITY AND REHABILITATION RESEARCH
that will be published in the Federal Register. The plan will utilize at least
90 percent of NIDRR funds for extramural research. Chapter 1 provides a
description of NIDRR’s program mechanisms.
The Long-Range Plan
Title II, section 202(h), of the Rehabilitation Act states that, starting in
October 1998 and every subsequent fifth October, the NIDRR Director will
develop and publish in the Federal Register for public comment a draft LRP
outlining NIDRR’s priorities, explaining the basis for those priorities, and
providing a broad framework for the funding of research aimed at achiev-
ing the priorities. Every fifth subsequent June, after consideration of public
comments, the final LRP with revisions will be submitted to the appropriate
members of Congress. The LRP is required to
• I
dentify any covered activity that should be conducted under section 202
(NIDRR) and section 204 (Research and Other Covered Activities) re-
specting the full inclusion and integration into society of individuals with
disabilities, especially in the area of employment;
• D
etermine the funding priorities for covered activities to be conducted
under section 202 and section 204; and
• S
pecify appropriate goals and timetables for covered activities to be con-
ducted under this section and section 204.
The LRP is the foundation that guides the development of grant priorities.
In developing the 5-year LRP, the Director must consider input from the
Commissioner of the Rehabilitation Services Administration, the Commis-
sioner of the Administration on Developmental Disabilities, the National
Council on Disability, the Interagency Committee on Disability Research
(ICDR), consumers, organizations representing people with disabilities,
researchers, service providers, and other appropriate entities. Additionally,
Title II, section 205, states that, subject to the availability of appropria-
tions, a 12-member standing disability and rehabilitation research advisory
council should advise the Director on the development and revision of each
LRP; however, NIDRR has never formed such a council.
According to NIDRR management, the agency’s use of long-range plans
extends back to the development of the first such plan, which spanned 1981
to 1985. Based on information gathered from 4,000 agencies and organiza-
tions, this plan laid out a comprehensive landscape of research and develop-
ment needs and topics across all disability and age groups. More recently,
the LRP for 1999 to 2003 was developed in 1998 by key management staff
following the reauthorization of the Rehabilitation Act. The content of the
plan was drawn from staff member contributions, and commissioned papers
helped inform the process. The major innovations in the 1999 to 2003 LRP
OCR for page 59
59
NIDRR’S PRIORITY-SETTING PROCESSES
included the “paradigm of disability” (which emphasizes the contextual
nature of disability as a product of individual and societal factors) and
the five research domains (Employment Outcomes, Health and Function,
Technology for Access and Function, Independent Living and Community
Integration, and Associated Disability Research Areas). This plan also en-
couraged participatory action research, which generated a great deal of
positive feedback. According to NIDRR management, the agency was able
to accomplish the vast majority of the research goals outlined in this LRP.
During the next presidential administration (in 2000), however, the LRP was
criticized for not being written in outcome-oriented terms.
The development of the next LRP, covering 2005 to 2009, differed
somewhat from that of the prior LRP. Agency staff utilized considerable
outside assistance to gather input from the public. These efforts included
hiring a contractor to support teleconferences linking NIDRR staff with
centers around the country and forming a steering committee. Subteams
of the staff wrote sections of the plan corresponding to the research do-
mains and capacity building. The major innovation in the LRP for 2005 to
2009 was a logic model that supported and provided a structure for more
outcome-oriented work.
The development of the LRP for 2010 to 2014 was impacted by the
change in presidential administrations, the departure of NIDRR’s Director,
and a shorter time frame in which to develop the plan. Input was solicited
through a national teleconference, but the development of this plan was
driven more by department leadership. During the public comment phase
of the plan’s development, close to 100 comments were received, many of
which were critical of the increased focus on employment included in this
plan. As a result, the plan was put on hold, and NIDRR is still operating
under the LRP for 2005 to 2009.5 At the time of this writing, a new LRP was
under development but had not yet been published for public comment. In
addition, after several years of the position being filled only on a temporary
basis, a new permanent Director has recently been hired.
Priority-Setting Stages in the Grant Competition Process
To meet the objectives laid out in the LRP, NIDRR formulates priori-
ties for research grants and development projects aimed at generating new
knowledge and products, along with supporting knowledge translation and
capacity-building activities (National Institute on Disability and Rehabilita-
tion Research, 2006). The priority-setting process generally starts 2 years
before funding announcements are published and involves several stages (see
5 NIDRR’s 1999-2003 LRP, 2005-2009 LRP, and information about the development of the
2010-2014 LRP can be found on the Publications and Products page of NIDRR’s website,
http://www2.ed.gov/rschstat/research/pubs/index.html [January 4, 2012].
OCR for page 60
60 REVIEW OF DISABILITY AND REHABILITATION RESEARCH
Months
Priority-Setting Stages 1 2 3 4 5 6 7 8 9 10 11 12 13 14
Identify grant priorities
(3-5 months)
Develop and publish notice of
proposed priority (2-4 months)
Public comment (1-2 months)
Develop notice of final priority (1-2
months)
Develop notice inviting
applications (1-2 months)
Develop application kit
(1-2 months)
Publish notice of final priority and
notice inviting applications in
Federal Register (1 month)
FIGURE 3-1 Timeline for NIDRR’s priority-setting process.
SOURCE: Committee developed from National Institute on Disability and
Rehabilitation Research (2009).
Figure 3-1 for an illustration of the general timeline). According to NIDRR
management, whileeditablehas standard procedures for the various stages
Figure 3-1, NIDRR
of priority setting, described below, the length of time required for new
proposed priorities to obtain ED approval and clearance varies.
Identification of Grant Priorities
In planning its priorities for funding, NIDRR considers changes to its
budget, as well as the funding level of expiring grants, and determines the
amount of money that will be available for new grant competitions (Na-
tional Institute on Disability and Rehabilitation Research, 2009). Looking
at current priority areas, NIDRR determines whether grantees have been
productive, and whether more work remains to be done or a topic has been
exhausted. NIDRR management stated that new priority areas may be
added if there is additional funding, if new technology or science suggests
OCR for page 61
61
NIDRR’S PRIORITY-SETTING PROCESSES
new areas for research, if experts from the field suggest a new area, or if the
administration suggests new priorities.
To inform the identification and development of potential priority top-
ics, NIDRR’s Research Sciences Division (RSD) considers the LRP, other
strategic and performance goals, the portfolio of existing projects, recent
findings from completed grants, and the current research literature, as well
as key guidance documents from federal partners, professional associations,
and consumer organizations (National Institute on Disability and Rehabili-
tation Research, 2006, 2009). Contractors may be asked to conduct specific
literature searches and factor in feedback from the field in response to prior
grant cycles. New priority areas may also be generated by NIDRR-funded
state-of-the-science conferences. The ICDR helps facilitate information
sharing and partnerships that can contribute to the identification of new
priority areas. In addition, the NIDRR Director and the Assistant Secretary
of the Office of Special Education and Rehabilitative Services often identify
potential priority topics of high importance (National Institute on Disability
and Rehabilitation Research, 2009).
NIDRR staff, as key sources of input into potential priority topics,
continually monitor the literature in their areas of expertise to stay cur-
rent, to identify potential areas for funding, to determine field capacity,
and also to identify active researchers who might serve as peer reviewers
of proposals. When resources are available to support travel, staff also stay
current with the state of the science through attending meetings of orga-
nizations such as the Rehabilitation Engineering and Assistive Technol-
ogy Society of North America, the American Congress of Rehabilitation
Medicine, the American Public Health Association, the Gerontological
Society of America, or the National Council of Rehabilitation Educators.
As priority topics are being considered, NIDRR assesses their alignment
with the following criteria (National Institute on Disability and Rehabilita-
tion Research, 2009, p. 58). Research priorities must
• R
eflect and incorporate state-of-the-science in a specific area
• F
ulfill a demonstrated need for new knowledge
• B
uild upon prior research-based knowledge and advance the stage of
knowledge development in specific areas
• H
ave real-world relevance to individuals with disabilities
• P
rioritize meeting the needs of individuals with disabilities from minority
backgrounds
• S
upport the NIDRR mission
• R
eflect the thinking and guidance of the current LRP
Considering all of the possible priority topics, emerging opportunities,
and ongoing needs, NIDRR management then asks staff to utilize their
knowledge of the relevant literature and the field in developing succinct,
OCR for page 62
62 REVIEW OF DISABILITY AND REHABILITATION RESEARCH
one-paragraph descriptions of the most promising potential topics, along
with rationales and recommendations for which topics should be selected.
The descriptions address the following areas (National Institute on Disabil-
ity and Rehabilitation Research, 2009):
• I
ncidence and prevalence of the condition to be targeted
• P
roblem or need addressed, in terms of improved policy, practice,
behavior, or system capacity
• H
ow the priority topic builds upon the state of the science in this
topic area
• N
IDRR’s historical investment in this topic area, key findings from
NIDRR investments in this area, and how the proposed priority
topic builds upon these investments
• H
ow the priority topic relates to the work of NIDRR’s relevant
federal partners, or other organizations that fund research
• R
eal-world relevance of this topic to people with disabilities
• H
ow this priority topic addresses the needs of individuals with dis-
abilities from minority backgrounds
• H
ow the priority topic fits under NIDRR LRP goals, objectives, and
strategies
The written descriptions of the potential topics are discussed with the
NIDRR Director who then approves topics to be published as priorities and
presents these to the Assistant Secretary of the Office of Special Education
and Rehabilitative Services. The process of identifying funding priorities
generally takes 3 to 5 months or longer.
Notices of Proposed Priority
Once priority topics have been indentified, the second stage in NIDRR’s
priority-setting process is to draft and publish notices of proposed priority
(NPPs). According to NIDRR management, NIDRR staff follow a template
in drafting selected topics into NPPs, which are NIDRR’s first notice to the
research community about each grant solicitation. The NPP explains the
nature of the problem and describes the requirements of the priority (e.g.,
for Burn Injury Model System grants, establish a multidisciplinary system
that begins with acute care and encompasses rehabilitation services specifi-
cally designed to meet the needs of individuals with burn injuries). NIDRR’s
style of NPPs has shifted over time from more to less prescriptive.
Each NPP is reviewed and cleared at multiple levels within NIDRR, the
Office of Special Education and Rehabilitative Services, and ED (including
the Office of the General Counsel, the Budget Office, the Deputy Secretary,
and the Executive Secretary) (National Institute on Disability and Rehabili-
OCR for page 63
63
NIDRR’S PRIORITY-SETTING PROCESSES
tation Research, 2009). The NPPs must also be reviewed and approved by
the Office of Management and Budget. According to NIDRR management,
multiple rounds of comments and revisions are not uncommon. The result-
ing NPP is published in the Federal Register. During a response period,
the public may submit comments on the proposed priority. The NPP de-
velopment and clearance process often takes 2-4 months, while the public
comment period lasts for 1-2 months, which includes the period of time for
NIDRR to respond to public comments.
Notices of Final Priority
After the close of the public comment period, NIDRR staff review
comments on the NPP, respond to the comments, and develop a notice of
final priority (NFP) (National Institute on Disability and Rehabilitation
Research, 2009). The draft NFP is reviewed and cleared at several levels
within NIDRR, the Office of Special Education and Rehabilitative Services,
and ED, much as is done with the NPP. Again, multiple revisions are not
uncommon. Development of the NFP generally takes 1-2 months, and when
approved it is published in the Federal Register.
Notices Inviting Applications and Application Kits
Concurrently with the NFP, NIDRR drafts the NIA and develops an
application kit for each priority (development of these materials generally
takes 1-2 months) (National Institute on Disability and Rehabilitation Re-
search, 2009). The NIA contains information about the grant competition,
such as submission deadlines, dollar amounts, and the procedure for submit-
ting an application. The NIA goes through the same clearance process as the
NFP and is published with the NFP in the Federal Register. NIAs are also
published on Grants.gov. In addition, NIDRR uses a contractor to notify
former grantees and others who, via the contractor’s webpage, express an
interest in receiving NIAs.
Application kits contain application forms, the NFP and NIA, regula-
tions, and the peer review criteria. The application kits are available on
Grants.gov and NIDRR’s website. The standard period between publica-
tion of the NFP and NIA and the application deadline is 60 days, although
NIDRR management stated that the period may be shortened to 45 days or
less when needed to enable awards before the end of the fiscal year.
For each competition, NIDRR convenes a preapplication technical as-
sistance meeting for potential applicants (National Institute on Disability
and Rehabilitation Research, 2010; also noted by NIDRR management).
Upon request, staff members are also available for individual consultation.
The staff can discuss only the application process with potential applicants,
OCR for page 64
64 REVIEW OF DISABILITY AND REHABILITATION RESEARCH
however—not application content. NIDRR does not track whether those
who participate in assistance events go on to apply for grants. Every NIDRR
competition has received at least one application, which is all that is required
to conduct the peer review process. However, not every competition has
resulted in a grant being funded. (Chapter 4 provides information on the
number of competitions between 2006 and 2009.)
RESULTS OF THE ASSESSMENT OF NIDRR’S
PRIORITY-SETTING PROCESS
This section summarizes the results of interviews and surveys of NIDRR
staff, grantees, and other stakeholder organizations concerning their views
of NIDRR’s priority-setting process. See Chapter 2 for a description of the
data collection and analysis methods used by the committee.
Perspectives of NIDRR Staff
In personal interviews, 16 NIDRR staff members were asked open-
ended questions about their roles in the priority-setting process, their per-
spectives on the quality of the process, and suggestions for improvement.
Two-thirds of the interviewees were project officers or direct supervisors of
project officers; the remaining held administrative positions. The perspec-
tives of staff coalesced around the processes of identifying priorities, moving
priorities into the funding announcement phase, and disseminating priorities
to the field.
Identification of Priority Topics
As stated earlier in the chapter, potential priority topics are identified
through several sources. Staff comments focused on several of these sources.
With regard to the roles of project officers, some play a very active part in
proposing priority topics, conducting the literature reviews for proposed
priorities, and writing the proposed priorities. Other project officers com-
mented that their role in priority setting is fairly minimal. This difference
appears to be based on individuals’ areas of expertise and allocation of time
among their other duties (e.g., monitoring grants; organizing and leading
peer review activities; managing program areas, such as Field Initiated Proj-
ect [FIP] grants or Traumatic Brain Injury center or research grants). Staying
well informed and current with the relevant literature entails a substantial
time commitment on the part of project officers. This demand was noted
as particularly challenging considering that staffing levels at NIDRR have
fallen over the years.
OCR for page 65
65
NIDRR’S PRIORITY-SETTING PROCESSES
Another source of potential priority topics is the results of previously
funded grants. The Annual Performance Reports submitted by grantees are
used to feed into priority setting more now than in the past. Staff mentioned
that this is an example of how NIDRR’s processes have become more stan-
dardized and data driven. However, it was suggested that more systematic
information should be collected on the stages of grant development, move-
ment rates, and life cycles. Such information could potentially inform suc-
cessive generations of priorities that would build on each other.
Staff also commented on other forces that influence the selection of
priority topics. The broad scope of NIDRR’s mission, which includes all dis-
abilities and all ages, relative to its limited budget sets a base of competing
priorities to meet the needs of multiple target populations. While NIDRR
is striving toward more scientific rigor in the research it funds, tensions still
arise among differing paradigms that call for varying levels of scientific rigor
to achieve research and development goals. There are also tensions at play
between proponents of the continuance of large center grant priorities and
those of smaller, flexible, more problem-focused or field-driven priorities. It
was suggested that NIDRR should gather information from the field more
regularly to inform priority setting and that NIDRR could benefit from
the use of a national advisory board with more diverse disability research
expertise to inform its priority-setting process.
Moving Priorities into the Funding Announcement Phase
Several comments were made about the challenges inherent in moving
proposed priorities through the various organizational clearance levels to
the final grant announcement phase. This process sometimes causes delays
as questions about proposed priorities arise at different levels, and new jus-
tifications for or changes to proposed priorities must be made. More barriers
are encountered with smaller, innovative grants than with the larger center
grants. Delays in the clearance process impact the regularity of the timing
of grant announcements and competitions.
Dissemination and Response
Staff commented that once grants have been announced, the specialty
nature of certain priorities (e.g., for Model System grants) limits the number
of researchers eligible to apply, whereas programs with field-driven priorities
(e.g., FIP grants) have larger applicant pools. Concern was expressed that
grant announcements are not reaching the largest intended audience and
that there needs to be broader dissemination of priorities through listservs
and routine dissemination to universities.
OCR for page 66
66 REVIEW OF DISABILITY AND REHABILITATION RESEARCH
Perspectives of NIDRR Grantees
As part of the questionnaire dealing primarily with outputs that grant-
ees submitted to the committee for review, grantees were asked to share
their perspectives on NIDRR’s priority-setting process. Comments on this
topic were received from 14 of the 30 grantees that were involved in the
output assessment.
Some comments applauded NIDRR’s periodic formulation of 5-year
LRPs, suggesting that significant field and community input increases the
likelihood that the priorities established and resulting outputs can have
the greatest potential impact. Consistent with a point made earlier in this
chapter, it was commented that priorities are less prescriptive now than in
the past, making it possible to explore areas of importance that would not
otherwise have been addressed. It was also noted that NIDRR is able to
fund studies that would not have been funded by other agencies in the early
stages. As a result, important developmental studies can be incubated before
other agencies will assume and extend them.
Some concern was expressed about the nature of priorities, which may
shift from cycle to cycle. Grantees suggested that this can inhibit certain
lines of research from continuing, affect the quality of research, and reduce
the potential pool of applicants. Difficulties also were noted with respect
to certain changes that have occurred in the course of grants (e.g., require-
ments for dissemination) that, although ultimately good, affected grantee
activities and budgets.
These respondents also commented on some areas in which they thought
priorities should be focused, including treatment and intervention studies
in the rehabilitation process, which have received little attention; economic
empowerment; and individual fellowship grants (similar to the National
Institutes of Health [NIH] F series for predoctoral and postdoctoral train-
ing) for capacity building in many research areas covered by NIDRR that
do not map neatly to NIH priorities. One respondent suggested that NIDRR
should continue to focus on projects that go beyond what is encompassed
by the medical rehabilitation research funded by NIH.
Perspectives of NIDRR Stakeholder Organizations
Because stakeholder input is a fundamental aspect of priority set-
ting, the committee gathered information on stakeholder perspectives on
NIDRR’s long-range planning and priority-setting processes by surveying
three key stakeholder groups. First, in NIDRR’s most immediate network
of stakeholders are other federal agencies with which it interacts and col-
laborates in order to achieve its mission. Second, professional associations
are key stakeholders that represent the professional base of providers and
researchers that work in the field. Third, the intended ultimate beneficiaries
OCR for page 72
72
TABLE 3-2 Distribution of Ratings of the Extent to Which NIDRR’s Grants Are Perceived as Beneficial and Their
Products Are Used (72 respondents)
Scale
Number of
Number of “Not Number Not at Less Than More Than
Total “Don’t Know” Applicable” of Valid All Somewhat Somewhat Somewhat Very Much
Responses Responses Responses Responses (%) (%) (%) (%) (%)
Question
Are grants 72 14 3 55 4 16 39 27 14
beneficial?
Are products 72 13 2 57 9 30 33 14 14
used?
SOURCE: Generated by the committee based on data from the stakeholder questionnaire.
OCR for page 73
73
NIDRR’S PRIORITY-SETTING PROCESSES
The synergy of the two agencies working together for 30 years may be a study of
braiding funding and staff which is unique to the Federal Government. Sharing
similar values and principles that reflect our individual Departments and what
the field reflects is unusual.
We have extensive scientific and strategic interaction with NIDRR. We have
coordinated research priorities with NIDRR plans and given presentations at
each other’s venues. We and NIDRR officials have served on each other’s tech-
nical working groups, [and] shared technical information and expertise that is
relevant for both groups.
Respondents from 3 professional associations and advocacy organiza-
tions remarked that they have met with NIDRR staff to prepare materials
to be used in advocating with Congress for NIDRR funding. Respondents
from 6 of these organizations stated that NIDRR staff have attended their
meetings and forums to inform members about program mechanisms and
research priorities.
Uniqueness and strengths of NIDRR’s mission and research Respondents
from 7 federal agencies emphasized the benefits of NIDRR’s specialized
mission as the touchstone for all of the agency’s program mechanisms and
grants. They commented that NIDRR’s unique mission among all federal
agencies requires grantees to use the state-of-the-art International Classifica-
tion of Functioning, Disability and Health (ICF) as a framework for all re-
search, training, and demonstration projects. They emphasized the benefits
of NIDRR’s specialized scope of research, which includes people of all ages
with psychiatric disabilities and comparative effectiveness research focused
on health services for persons with disabilities. Respondents from 12 of the
other stakeholder organizations and 8 federal agencies identified specific
program mechanisms beneficial to their organization: Burn Model System
(BMS), Traumatic Brain Injury Model System (TBIMS), Spinal Cord Injury
Model System (SCIMS), FIP, Rehabilitation Research and Training Center
(RRTC), Rehabilitation Engineering Research Center (RERC), and Dis-
ability and Rehabilitation Research Project-General (DRRP). Respondents
from 7 professional associations and advocacy organizations applauded
the funding of grants aimed at increasing accessibility for people with dis-
abilities, as exemplified by grants related to communication for the deaf and
hard-of-hearing; the availability of assistive technology; and physical and
programmatic access to health care, education, housing, and employment.
Respondents from 12 of these same organizations also emphasized the ben-
efits of grants focused on community participation, such as employment and
delivery of services to people with disabilities who are seeking employment;
information technology issues that prevent these people from getting jobs;
outcome research related to community participation, personal assistants,
OCR for page 74
74 REVIEW OF DISABILITY AND REHABILITATION RESEARCH
and aging with developmental disabilities; and education and workforce
development.
Ways of enhancing collaboration with federal agencies Respondents from
four professional associations and advocacy organizations commented on
the need to develop stronger collaborations with currently collaborating
federal agencies and entities and to expand the number of new potential
collaborators (e.g., more coordination of priorities with NIH institutes and
the Department of Veterans Affairs). Respondents from two federal agencies
suggested that NIDRR make more effective use of ICDR members in the
strategic planning and priority-setting processes—for example, by delivering
presentations to the group with specific review requests to members. Three
federal agency respondents commented on the need to expand the depth and
breadth of collaboration with agencies and co-funding of research projects
with new agencies. Four federal agency respondents suggested that an im-
portant quality improvement initiative would be for NIDRR staff to become
more knowledgeable about the relevant research efforts and products of
other agencies so they could seek opportunities and avoid duplication of
effort. Respondents from eight other stakeholder organizations suggested
establishing quality improvement initiatives related to priority setting by,
for example, investigating how other federal agencies establish priorities,
communicating the details of the process, and incorporating feedback.
Strengths and needs related to outreach and dissemination Although re-
spondents from four of the stakeholder organizations stated that they were
unaware of available products of NIDRR grants, respondents from 8 profes-
sional and advocacy organizations and from 11 federal agencies commented
that NIDRR’s website and other linked sources (e.g., National Rehabilitation
Information Center [NARIC], the Center for International Rehabilitation
Research Information and Exchange, the Repository of Recovery Resources,
and RERC and RRTC grant websites) provide easy access to information,
resources, technical assistance, and research results in topic areas such as: asis-
tive technology use, evidence-based information on employment, disability
statistics, guidelines for accessible digital media and guidelines, information
on health promotion for persons with disabilities, and technical assistance
related to the Americans with Disabilities Act.
Several suggestions also were made for ways to enhance NIDRR’s
outreach and dissemination activities to inform the agency’s planning and
priority setting. For example, respondents from 5 federal agencies recom-
mended expanding and refining NIDRR’s outreach plan by including experts
from a wider variety of disciplines and heads of research and evaluation
departments of other federal agencies. For greater outreach, 4 federal agency
respondents suggested increasing stakeholders’ awareness of NIDRR ac-
OCR for page 75
75
NIDRR’S PRIORITY-SETTING PROCESSES
complishments by initiating a new public relations campaign using “brag-
sheets” and disseminating information about companies and vendors that
use products based on the results of NIDRR-funded projects. Seven profes-
sional association and advocacy organization respondents commented on
the need to develop more user-friendly ways for NIDRR to receive feedback
from stakeholders (e.g., provide summaries of larger planning documents,
develop electronic capacity to insert comments directly into LRPs undergo-
ing public review and comment). In addition, 12 respondents from these
same stakeholder organizations encouraged NIDRR to make more effective
use of the disability communities through enhanced communication ap-
proaches, such as webinars and presentations at existing annual meetings.
Policies and practices related to priority setting Finally, four federal agency
respondents suggested that NIDRR should review and adapt its current poli-
cies and practices that guide the long-range planning and priority-setting
processes to include standardized timelines and a review of progress half-
way through the 5-year LRPs. One federal agency respondent proposed a
specific set of guidelines for priority setting that would
(a) point to a national vision for research that is aspirational; (b) capture the
imagination and support of the disability community at-large; (c) [be] based
on the best available evidence and information; (d) require projects to produce
measurable outcomes and results; (e) promote initiatives that are supported
over an appropriate timeframe; (f) [be] multidisciplinary in nature; and (g)
stimulate a collaborative approach to solutions among stakeholders.
Respondents from three other federal agencies suggested in general that
NIDRR should increase funding for planning and priority setting. With
regard to how grants are structured, respondents from four professional
associations and advocacy organizations remarked that NIDRR should re-
fine the inclusion of policies and practices in funded studies and projects to
ensure that grantees planning to include people with disabilities as collabo-
rators or partners actually follow through in doing so. In this same vein, one
federal agency respondent commented that the “diversity” criteria NIDRR
uses to score applications should be critically reviewed so as to consider
requirements for including consumers in the planning and implementation
phases of all projects.
CONCLUSIONS AND RECOMMENDATIONS
NIDRR has a legislatively based process for formulating 5-year LRPs
and for establishing its funding priorities. In proposing priority topics,
NIDRR has specific criteria that must be met and subjects potential pri-
orities to key questions. It also has written procedures for staff to follow
in proposing priorities that will be recommended to the Office of Special
OCR for page 76
76 REVIEW OF DISABILITY AND REHABILITATION RESEARCH
Education and Rehabilitative Services for funding. Processes for gathering
initial input on LRPs and priorities appear to be inclusive but vary from
cycle to cycle. However, formal public comments on proposed LRPs and
priorities are consistently obtained via the Federal Register. NIDRR has
not established a standing disability and rehabilitation research advisory
council, which is included in its legislation. As of the time of this writing,
the agency is still operating under the LRP for 2005 to 2009 because of
critical input from formal public comments and the long-term absence of
a permanent Director. However, a permanent Director was recently hired.
Seventy-one percent of stakeholder organizations surveyed were “more
than somewhat” or “very much” familiar with NIDRR. Attributes of
NIDRR’s priority-setting process that were most endorsed included rele-
vance of the process to their organization, responsiveness to emerging issues
in disability and rehabilitation research, and degree to which the process is
publicized. Respondents thought NIDRR’s priority-setting process least re-
flected the attribute of responsiveness to stakeholder organization feedback.
In open-ended remarks, stakeholder organizations emphasized the benefits
of NIDRR’s specialized mission and scope of research, its specific program
mechanisms, and funding of grants aimed at increasing accessibility and
community participation for people with disabilities. Respondents com-
mented on the need to develop stronger collaborations with federal agencies
and entities. Although stakeholder organizations commented on the benefits
of NIDRR’s website and linked sources, suggestions were made for ways
to enhance the agency’s outreach and dissemination activities. Respondents
suggested that NIDRR should review and adapt its current policies and
practices that guide the long-range planning and priority-setting processes
to include standardized timelines.
The committee offers recommendations regarding NIDRR’s priority-
setting process in four areas: formation of an advisory council, strategic
planning, establishment of a standard calendar, and expanded dissemination
of NIAs.
Formation of an Advisory Council
NIDRR has a broad and diverse mission that makes it challenging to set
priorities that are responsive to the current state of the science and the needs
of the stakeholder community. Currently, NIDRR relies on staff, the portfo-
lio of existing projects, recent findings from completed grants, and the cur-
rent research literature, as well as guidance from federal partners, for input
to the priority-setting process. Title II, section 205, of the Rehabilitation Act
instructs NIDRR, subject to the availability of appropriations, to establish a
12-member standing disability and rehabilitation research advisory council
to support its priority setting. While NIDRR formed a steering committee
OCR for page 77
77
NIDRR’S PRIORITY-SETTING PROCESSES
to support the development of the LRP for 2005 to 2009, a standing body
has never been formed.
Recommendation 3-16: NIDRR should fulfill the statutory mandate
to form and utilize a standing disability and rehabilitation research
advisory council to advise on the priority-setting process and pro-
vide input for priority setting.
In the committee’s view, it is somewhat anomalous that NIDRR does
not have a standing advisory body. Most federal funding agencies, includ-
ing NIH, the National Science Foundation (NSF), and the National Insti-
tute for Occupational Safety and Health (NIOSH), benefit from the use of
standing advisory bodies. A standing advisory body is likely to add stability
and continuity to both NIDRR’s long-range planning and priority-setting
processes.
Additionally, a disability and rehabilitation research advisory council
would be an efficient way to obtain expertise from the scientific community,
as well as input from members of NIDRR’s diverse constituency of stake-
holder organizations and consumers. As documented in the survey data
presented in this chapter, a number of relevant stakeholder organizations,
including professional associations, advocacy associations, and federal agen-
cies, are less familiar with NIDRR than they might be expected to be. An
advisory council could provide another way for these stakeholder organiza-
tions to interact meaningfully with NIDRR.
Furthermore, an advisory council would represent an important, regu-
lar forum through which consumers could interact with NIDRR. Accord-
ing to Title II, at least half of the council members are to be consumers.
Consumer participation on the council should help build and strengthen
productive partnerships between NIDRR and the populations it serves.
Participation would allow consumers to have direct input into the advice
that guides NIDRR’s work, which is vital for those affected by the research
the agency funds (Ahmed and Palermo, 2010). Such input could benefit
NIDRR in many ways. Consumer input could educate scientists about the
research being conducted. As the Director’s Consumer Liaison Group of
the National Cancer Institute (2011) states, consumer input can “improve
research outcomes by identifying new approaches, promoting innovation,
recognizing unforeseen risks or barriers, and identifying unintended conse-
quences.” According to the NIH Director’s Council of Public Representa-
tives, through participation on the advisory board, consumers may have the
ability to represent their communities to research funders such as NIDRR
6 The committee’s recommendations are numbered according to the chapter of the report in
which they appear.
OCR for page 78
78 REVIEW OF DISABILITY AND REHABILITATION RESEARCH
(National Institutes of Health, 2011). At the same time, they can gain a
more informed understanding of NIDRR and the research being conducted.
This understanding in turn can be used to better inform their communities
about NIDRR’s work.
Title II states that, in addition to including six consumers, the mem-
bership of the council should represent the community of rehabilitation
professionals and the community of rehabilitation researchers. Additional
populations that NIDRR regularly tries to engage include national, state,
and local rehabilitation agencies and facilities; administrators and practi-
tioners in agencies serving persons with disabilities; other federal agencies;
educators of rehabilitation professionals and their students; and the general
public (National Institute on Disability and Rehabilitation Research, 2011).
The committee recognizes that NIDRR, like other federal research agencies,
will face challenges in capturing the broad diversity of perspectives held by
this large collection of different populations. However, the committee feels
strongly that, like other federal research agencies, NIDRR can meet these
challenges.
Strategic Planning
The delay in NIDRR’s LRP for 2010 to 2014, resulting from the nega-
tive reaction to the plan by the field as well as the absence of a Director to
guide the process, suggests a breakdown in the LRP process. As the LRP
is the foundation of priority setting, the priority-setting process is likewise
affected. Long-range planning and priority-setting processes may be en-
hanced in ways that are likely to help NIDRR avoid such a breakdown in
the future.
In its review of NIDRR’s long-range planning process, the committee
was never presented by NIDRR with a document explaining the steps in
the process from start to finish. Instead, the process has varied from the
creation of one LRP to the next. It is the Committee’s viewpoint that LRP
processes are better conceived as a documented series of logical stages. The
methods used to gather input from stakeholders also have varied during the
development of each plan. In addition, NIDRR documentation (discussed
above) indicated that input into the creation of NPPs is drawn from such
sources as experts in the field, the current literature, and state-of-the-science
conferences. The committee applauds these efforts but notes that this input
is somewhat informal and the process is not systematic. The first formal
input is not received until after potential topics have been narrowed down
and the NPPs have been published for comment in the Federal Register. The
committee believes more could be done to involve stakeholders earlier in the
process of identifying potential priority topics.
OCR for page 79
79
NIDRR’S PRIORITY-SETTING PROCESSES
Recommendation 3-2: NIDRR should use a structured, consistent,
and inclusive strategic planning process to develop its Long-Range
Plans and priorities.
To begin implementing this recommendation, NIDRR should first for-
malize and document the structure of its long-range planning process. The
advisory council recommended above could assist in this effort. Once the
structure of the process has been documented, NIDRR will be able to follow
it consistently in the development of future LRPs. Subsequent changes to
the process, and the rationale behind them, should be documented as well.
NIDRR should also establish a regular form of interaction with stake-
holders in the long-range planning process, and might also make efforts to
expand the stakeholder groups that are included in the process. The recom-
mended advisory council might assist in regularizing and expanding the in-
clusivity of the process. With regard to priority setting, NIDRR should seek
more formal input from the field on potential priority topics earlier in the
process and pursue maximum participation from stakeholders. While incor-
porating a broad range of stakeholder views can be a slow, arduous process,
the added structure and consistency, along with the regular influence of an
advisory council, will allow the process to be as efficient as possible.
One source NIDRR might consider in planning a more structured, con-
sistent, and inclusive process is the Canadian Institutes of Health Research’s
Knowledge Translation module on Deliberative Priority Setting (Campbell,
2010). Included in Campbell’s work, derived from an extensive literature
review, is a description of seven ideal elements of agenda-setting and/or
priority-setting processes:
• p
lanning of the process, including anticipating the needs, barriers,
and challenges for all the remaining steps and identifying the leader-
ship that will guide the process;
• s
takeholder identification, analysis, and engagement, which in-
volves purposefully identifying the proper stakeholders to include
in the process, understanding both the opportunities and risks as-
sociated with including stakeholders, and formulating appropriate
engagement strategies;
• k
nowledge management, which ensures that all stakeholders have
the same information, and any stakeholders with less technical
expertise are supported such that they can understand all the
information;
• i
nterpretive workshops, which involve gathering with stakeholders
to define criteria, establish weights, and then apply the criteria to
identify the most relevant issues or topics;
OCR for page 80
80 REVIEW OF DISABILITY AND REHABILITATION RESEARCH
• t
ranslation of the issues or topics identified in the workshops into
actionable research agendas and priorities;
• p
ublication and validation, which involve making the draft long-
range plans and/or proposed priorities available to all stakeholders
to ensure alignment with the goals of the communities and clarity
of the expected outcomes; and
• r
evision or an appeal mechanism, the presence of which is neces-
sary to allow stakeholders to communicate disagreement with draft
long-range plans and/or proposed priorities constructively and
through which such concerns can be comprehensively addressed.
The document also contains an extensive literature review on different types
of priority-setting processes.
Additionally, NIDRR might consider the long-range planning and prior-
ity setting of other funding agencies, including NIH, the National Science
Foundation, and the National Institute for Occupational Safety and Health,
which have sought to integrate long-range planning and priority-setting
processes through specific initiatives such as the NIH Roadmap, the NSF
Strategic Plan, and the National Occupational Research Agenda.
Establishment of a Standard Calendar
For many program mechanisms, NIDRR has not established a regular
schedule for drafting and approving priorities and NIAs and disseminating
them to the field. In exploring this issue further, the committee reviewed
Federal Register publication dates of NIAs for all of the program mecha-
nisms over the last 5 years. There appeared to be no regular timing pattern
of the publication of NIAs either within or across program mechanisms.
The Department of Education has a lengthy review and approval process for
obtaining clearance for the release of priorities and NIAs. The variability in
the length of the clearance process may be an important factor, among oth-
ers, that impacts the timing of the release of NIAs. The irregular or delayed
release of NIAs may affect NIDRR’s ability to provide individuals sufficient
notice of grant opportunities or an optimal amount of time to complete ap-
plications. An irregular schedule may discourage the best investigators from
submitting applications. Additionally, certain program mechanisms (such as
Model Systems) include collaboration between institutions. Irregular post-
ing and shortened response times hamper the ability of applicants to identity
and recruit appropriate collaborators. These factors are likely to limit the
number of investigators who apply and adversely affect the quality of the
applications they submit. Additionally, young investigators less familiar with
NIDRR are more likely to pursue grants from other agencies.
OCR for page 81
81
NIDRR’S PRIORITY-SETTING PROCESSES
Recommendation 3-3: NIDRR should utilize a standard calendar
for the setting of priorities, publication of notices inviting appli-
cations, submission of applications, and peer review meetings to
improve the efficiency of the process.
NIDRR has made efforts to standardize the schedule for NIAs for its
various program mechanisms, such as FIP, SBIR, Advanced Rehabilitation
Research Training (ARRT), and Switzer Fellowship. However, NIDRR has
been unable to standardize the schedule for most of its program mecha-
nisms. Even within the constraints of its lengthy review process for publish-
ing NIAs, a standardized calendar should be developed. The committee sug-
gests that program mechanisms competed on a yearly basis have a consistent
annual schedule for the submission and review of applications. For multi-
year grants, the committee recommends that NIDRR establish a long-range
operational plan listing projected future grant application submission dates,
pending funding availability in that fiscal year. With a submission timeline,
NIDRR could establish when NPPs would have to be published for public
comment and also when NFPs would have to be published. In addition to
better supporting applicants, establishing a standard calendar would reduce
some of the burden on NIDRR staff caused by the current unanticipatable
priority-setting timeline. Delays in priority setting also often cause delays
and/or shortened timelines in the peer review process (see Chapter 4). A
standard calendar could potentially help address those delays as well.
Expanded Dissemination of Notices Inviting Application
While creating and utilizing a standard calendar is likely to increase
the number of researchers already familiar with NIDRR who will apply for
grants, the committee thinks more effort needs to be made to expand this
pool. NIDRR publishes NIAs in the Federal Register and on Grants.gov,
and also uses a contractor to notify former grantees and others who, via the
contractor’s webpage, express an interest in receiving NIAs. Given the vast
number of scientists whose work is relevant to disability and rehabilitation
research, however, NIDRR would benefit from more active efforts to solicit
interest in its funding announcements.
Recommendation 3-4: NIDRR should expand its efforts to dis-
seminate notices inviting applications to new potential applicants,
including developing a communication strategy to ensure that the
notices reach new audiences.
Increasing the pool of applicants may ultimately increase the quality
of the work NIDRR funds. A logical means of expanding dissemination is
OCR for page 82
82 REVIEW OF DISABILITY AND REHABILITATION RESEARCH
through sending notices to the disability-relevant professional organizations
and agencies and disability and rehabilitation research organizations that
make up NIDRR’s stakeholder network. NIDRR should also make efforts to
expand the network to organizations not yet familiar with the agency. Ad-
ditionally, NIDRR should begin sending notices to university departments
and offices of sponsored research. This could perhaps be accomplished
through collaboration with other federal research programs that regularly
send funding notices to universities. Increasing the number of potential ap-
plicants may also contribute to increasing the pool of reviewers, addressed
by Recommendation 4-1 in the next chapter.
REFERENCES
Ahmed, S.M., and Palermo, A.-G.S. (2010). Community engagement in research: Frameworks
for education and peer review. American Journal of Public Health, 100(8), 1,380-1,387.
Baruch, Y., and Holtom, B.C. (2008). Survey levels and trends in organizational research. Hu-
man Relations, 61(8), 1,139-1,160.
Campbell, S. (2010). Deliberative priority setting. Ottawa, Canada: Canadian Institutes of
Health Research.
National Cancer Institute. (2011). About the Director’s Consumer Liaison Group. Available:
http://dclg.cancer.gov/about [June 4, 2011].
National Institute on Disability and Rehabilitation Research. (2006). Department of Education:
National Institute on Disability and Rehabilitation Research—Notice of Final Long-
Range Plan for Fiscal Years 2005–2009. Federal Register, 71(31), 8,166–8,200.
National Institute on Disability and Rehabilitation Research. (2009). Briefing book for The
National Academies. Unpublished document. Washington, DC: National Institute on
Disability and Rehabilitation Research.
National Institute on Disability and Rehabilitation Research. (2010). Overview of the NIDRR’s
peer review process (what happens between OPP’s priority review and OPP’s slate re-
view). Unpublished document. Washington, DC: National Institute on Disability and
Rehabilitation Research.
National Institute on Disability and Rehabilitation Research. (2011). Target populations for
NARIC. Unpublished document. Washington, DC: National Institute on Disability and
Rehabilitation Research.
National Institutes of Health. (2011). About Council of Public Representatives. Available:
http://copr.nih.gov/secondary/about/ [June 4, 2011].
The Rehabilitation Act of 1973, as amended. Pub. L. No. 93-112. Available: http://www2.
ed.gov/policy/speced/reg/narrative.html [January 21, 2011].
The Small Business Innovation Research Development Act, Public. L. No. 97-219, 106 Stat.
2965 (2000).
