This chapter addresses the following key study question:
Key Question #1. To what extent is NIDRR’s priority-writing process conducted in such a way as to enhance the quality of the final results?
As used in the study question, the term “priority-writing” process encompasses many aspects of priority setting, including gathering input from multiple sources (e.g., the field, stakeholder organizations, grantees, other agencies, and persons with disabilities and their families), identifying potential topics and determining priorities for funding, writing the proposed priorities and having them cleared for release, and publishing notices inviting applications (NIAs)1 on these priorities.2 The term “priority setting” is used synonymously with “priority writing” in this report to clarify that the focus of the committee’s evaluation was on this larger priority-setting process.
Priority setting not only reflects the National Institute on Disability and Rehabilitation Research’s (NIDRR’s) intent to influence the advancement of research in targeted areas but also offers specific funding opportunities for potential grant applications to the agency. The announced priorities should therefore be developed and communicated in a manner that activities in a 5-year planattracts the best researchers and encourages participation in disability and rehabilitation
1 A “notice inviting applications” (NIA) is NIDRR’s announcement to the field of the opening of a grant competition. It is similar to the terms “request for proposals” and “request for applications” used by other agencies.
2 “Priority” in this context means the topic of the grant competition for which scientists submit applications to investigate.
research. Attracting the largest pool of applications from which to select grantees increases the chances for the highest-quality outputs. When establishing its priorities, the agency needs to consider continuity from one funding cycle to another, as well as identify future research challenges and societal needs.
In the context of this committee’s work, it is challenging to link priority setting directly to specific output quality. The quality of NIDRR’s research portfolio, grants, and outputs is the product of multiple complex factors, including the priority-setting process, funding levels, the peer review process, and the scientific quality of the grantees. It is clear, however, that NIDRR’s priority-setting process has a positive impact on the quality of the final outputs. In this chapter, the committee reviews and assesses that process and makes recommendations for its improvement.
The chapter has three major sections. The first describes NIDRR’s priority-setting process based on existing documentation and interviews with NIDRR executives. The second presents an assessment of the process, based on data gathered from NIDRR staff, grantees, and stakeholder organizations. The final section offers the committee’s conclusions and recommendations with respect to NIDRR’s priority-setting process.
DESCRIPTION OF NIDRR’S PRIORITY-SETTING PROCESS
The following description is based on existing documentation, such as legislation, the Federal Register, NIDRR and the U.S. Department of Education (ED) policies and procedures, NIDRR’s Long-Range Plan (LRP), and NIAs, as well as interviews with NIDRR and ED management.3
The Rehabilitation Act (1973, as amended) authorizes the formula grant programs of vocational rehabilitation, supported employment, independent living, and client assistance. Title II4 of the act—Research and Training—authorizes the majority of NIDRR’s research activities, while Section 21 of the Act and the Small Business Innovation Research (SBIR) Development Act (2000) require NIDRR to reserve small portions of its budget for those two specific program mechanisms. Section 202 of Title II states that NIDRR’s Director will lay out funding priorities and covered activities in a 5-year plan
3 The committee conducted interviews with NIDRR and ED management in four sessions during summer 2010 and one session in spring 2011.
4 The other six titles are Title I, Vocational Rehabilitation Services; Title III, Professional Development and Special Projects and Demonstrations; Title IV, National Council on Disability; Title V, Rights and Advocacy; Title VI, Employment Opportunities for Individuals with Disabilities; and Title VII, Independent Living Services and Centers for Independent Living.
that will be published in the Federal Register. The plan will utilize at least 90 percent of NIDRR funds for extramural research. Chapter 1 provides a description of NIDRR’s program mechanisms.
The Long-Range Plan
Title II, section 202(h), of the Rehabilitation Act states that, starting in October 1998 and every subsequent fifth October, the NIDRR Director will develop and publish in the Federal Register for public comment a draft LRP outlining NIDRR’s priorities, explaining the basis for those priorities, and providing a broad framework for the funding of research aimed at achieving the priorities. Every fifth subsequent June, after consideration of public comments, the final LRP with revisions will be submitted to the appropriate members of Congress. The LRP is required to
• Identify any covered activity that should be conducted under section 202 (NIDRR) and section 204 (Research and Other Covered Activities) respecting the full inclusion and integration into society of individuals with disabilities, especially in the area of employment;
• Determine the funding priorities for covered activities to be conducted under section 202 and section 204; and
• Specify appropriate goals and timetables for covered activities to be conducted under this section and section 204.
The LRP is the foundation that guides the development of grant priorities.
In developing the 5-year LRP, the Director must consider input from the Commissioner of the Rehabilitation Services Administration, the Commissioner of the Administration on Developmental Disabilities, the National Council on Disability, the Interagency Committee on Disability Research (ICDR), consumers, organizations representing people with disabilities, researchers, service providers, and other appropriate entities. Additionally, Title II, section 205, states that, subject to the availability of appropriations, a 12-member standing disability and rehabilitation research advisory council should advise the Director on the development and revision of each LRP; however, NIDRR has never formed such a council.
According to NIDRR management, the agency’s use of long-range plans extends back to the development of the first such plan, which spanned 1981 to 1985. Based on information gathered from 4,000 agencies and organizations, this plan laid out a comprehensive landscape of research and development needs and topics across all disability and age groups. More recently, the LRP for 1999 to 2003 was developed in 1998 by key management staff following the reauthorization of the Rehabilitation Act. The content of the plan was drawn from staff member contributions, and commissioned papers included helped inform the process. The major innovations in the 1999 to 2003 LRP
included the “paradigm of disability” (which emphasizes the contextual nature of disability as a product of individual and societal factors) and the five research domains (Employment Outcomes, Health and Function, Technology for Access and Function, Independent Living and Community Integration, and Associated Disability Research Areas). This plan also encouraged participatory action research, which generated a great deal of positive feedback. According to NIDRR management, the agency was able to accomplish the vast majority of the research goals outlined in this LRP. During the next presidential administration (in 2000), however, the LRP was criticized for not being written in outcome-oriented terms.
The development of the next LRP, covering 2005 to 2009, differed somewhat from that of the prior LRP. Agency staff utilized considerable outside assistance to gather input from the public. These efforts included hiring a contractor to support teleconferences linking NIDRR staff with centers around the country and forming a steering committee. Subteams of the staff wrote sections of the plan corresponding to the research domains and capacity building. The major innovation in the LRP for 2005 to 2009 was a logic model that supported and provided a structure for more outcome-oriented work.
The development of the LRP for 2010 to 2014 was impacted by the change in presidential administrations, the departure of NIDRR’s Director, and a shorter time frame in which to develop the plan. Input was solicited through a national teleconference, but the development of this plan was driven more by department leadership. During the public comment phase of the plan’s development, close to 100 comments were received, many of which were critical of the increased focus on employment included in this plan. As a result, the plan was put on hold, and NIDRR is still operating under the LRP for 2005 to 2009.5 At the time of this writing, a new LRP was under development but had not yet been published for public comment. In addition, after several years of the position being filled only on a temporary basis, a new permanent Director has recently been hired.
Priority-Setting Stages in the Grant Competition Process
To meet the objectives laid out in the LRP, NIDRR formulates priorities for research grants and development projects aimed at generating new knowledge and products, along with supporting knowledge translation and capacity-building activities (National Institute on Disability and Rehabilitation Research, 2006). The priority-setting process generally starts 2 years before funding announcements are published and involves several stages (see
5 NIDRR’s 1999-2003 LRP, 2005-2009 LRP, and information about the development of the 2010-2014 LRP can be found on the Publications and Products page of NIDRR’s website, http://www2.ed.gov/rschstat/research/pubs/index.html [January 4, 2012].
FIGURE 3-1 Timeline for NIDRR’s priority-setting process.
SOURCE: Committee developed from National Institute on Disability and Rehabilitation Research (2009).
Figure 3-1 for an illustration of the general timeline). According to NIDRR management, while NIDRR has standard procedures for the various stages of priority setting, described below, the length of time required for new proposed priorities to obtain ED approval and clearance varies.
Identification of Grant Priorities
In planning its priorities for funding, NIDRR considers changes to its budget, as well as the funding level of expiring grants, and determines the amount of money that will be available for new grant competitions (National Institute on Disability and Rehabilitation Research, 2009). Looking at current priority areas, NIDRR determines whether grantees have been productive, and whether more work remains to be done or a topic has been exhausted. NIDRR management stated that new priority areas may be added if there is additional funding, if new technology or science suggests
new areas for research, if experts from the field suggest a new area, or if the administration suggests new priorities.
To inform the identification and development of potential priority topics, NIDRR’s Research Sciences Division (RSD) considers the LRP, other strategic and performance goals, the portfolio of existing projects, recent findings from completed grants, and the current research literature, as well as key guidance documents from federal partners, professional associations, and consumer organizations (National Institute on Disability and Rehabilitation Research, 2006, 2009). Contractors may be asked to conduct specific literature searches and factor in feedback from the field in response to prior grant cycles. New priority areas may also be generated by NIDRR-funded state-of-the-science conferences. The ICDR helps facilitate information sharing and partnerships that can contribute to the identification of new priority areas. In addition, the NIDRR Director and the Assistant Secretary of the Office of Special Education and Rehabilitative Services often identify potential priority topics of high importance (National Institute on Disability and Rehabilitation Research, 2009).
NIDRR staff, as key sources of input into potential priority topics, continually monitor the literature in their areas of expertise to stay current, to identify potential areas for funding, to determine field capacity, and also to identify active researchers who might serve as peer reviewers of proposals. When resources are available to support travel, staff also stay current with the state of the science through attending meetings of organizations such as the Rehabilitation Engineering and Assistive Technology Society of North America, the American Congress of Rehabilitation Medicine, the American Public Health Association, the Gerontological Society of America, or the National Council of Rehabilitation Educators.
As priority topics are being considered, NIDRR assesses their alignment with the following criteria (National Institute on Disability and Rehabilitation Research, 2009, p. 58). Research priorities must
• Reflect and incorporate state-of-the-science in a specific area
• Fulfill a demonstrated need for new knowledge
• Build upon prior research-based knowledge and advance the stage of knowledge development in specific areas
• Have real-world relevance to individuals with disabilities
• Prioritize meeting the needs of individuals with disabilities from minority backgrounds
• Support the NIDRR mission
• Reflect the thinking and guidance of the current LRP
Considering all of the possible priority topics, emerging opportunities, and ongoing needs, NIDRR management then asks staff to utilize their knowledge of the relevant literature and the field in developing succinct,
one-paragraph descriptions of the most promising potential topics, along with rationales and recommendations for which topics should be selected. The descriptions address the following areas (National Institute on Disability and Rehabilitation Research, 2009):
• Incidence and prevalence of the condition to be targeted
• Problem or need addressed, in terms of improved policy, practice, behavior, or system capacity
• How the priority topic builds upon the state of the science in this topic area
• NIDRR’s historical investment in this topic area, key findings from NIDRR investments in this area, and how the proposed priority topic builds upon these investments
• How the priority topic relates to the work of NIDRR’s relevant federal partners, or other organizations that fund research
• Real-world relevance of this topic to people with disabilities
• How this priority topic addresses the needs of individuals with disabilities from minority backgrounds
• How the priority topic fits under NIDRR LRP goals, objectives, and strategies
The written descriptions of the potential topics are discussed with the NIDRR Director who then approves topics to be published as priorities and presents these to the Assistant Secretary of the Office of Special Education and Rehabilitative Services. The process of identifying funding priorities generally takes 3 to 5 months or longer.
Notices of Proposed Priority
Once priority topics have been indentified, the second stage in NIDRR’s priority-setting process is to draft and publish notices of proposed priority (NPPs). According to NIDRR management, NIDRR staff follow a template in drafting selected topics into NPPs, which are NIDRR’s first notice to the research community about each grant solicitation. The NPP explains the nature of the problem and describes the requirements of the priority (e.g., for Burn Injury Model System grants, establish a multidisciplinary system that begins with acute care and encompasses rehabilitation services specifically designed to meet the needs of individuals with burn injuries). NIDRR’s style of NPPs has shifted over time from more to less prescriptive.
Each NPP is reviewed and cleared at multiple levels within NIDRR, the Office of Special Education and Rehabilitative Services, and ED (including the Office of the General Counsel, the Budget Office, the Deputy Secretary, and the Executive Secretary) (National Institute on Disability and Rehabilitation
Research, 2009). The NPPs must also be reviewed and approved by the Office of Management and Budget. According to NIDRR management, multiple rounds of comments and revisions are not uncommon. The resulting NPP is published in the Federal Register. During a response period, the public may submit comments on the proposed priority. The NPP development and clearance process often takes 2-4 months, while the public comment period lasts for 1-2 months, which includes the period of time for NIDRR to respond to public comments.
Notices of Final Priority
After the close of the public comment period, NIDRR staff review comments on the NPP, respond to the comments, and develop a notice of final priority (NFP) (National Institute on Disability and Rehabilitation Research, 2009). The draft NFP is reviewed and cleared at several levels within NIDRR, the Office of Special Education and Rehabilitative Services, and ED, much as is done with the NPP. Again, multiple revisions are not uncommon. Development of the NFP generally takes 1-2 months, and when approved it is published in the Federal Register.
Notices Inviting Applications and Application Kits
Concurrently with the NFP, NIDRR drafts the NIA and develops an application kit for each priority (development of these materials generally takes 1-2 months) (National Institute on Disability and Rehabilitation Research, 2009). The NIA contains information about the grant competition, such as submission deadlines, dollar amounts, and the procedure for submitting an application. The NIA goes through the same clearance process as the NFP and is published with the NFP in the Federal Register. NIAs are also published on Grants.gov. In addition, NIDRR uses a contractor to notify former grantees and others who, via the contractor’s webpage, express an interest in receiving NIAs.
Application kits contain application forms, the NFP and NIA, regulations, and the peer review criteria. The application kits are available on Grants.gov and NIDRR’s website. The standard period between publication of the NFP and NIA and the application deadline is 60 days, although NIDRR management stated that the period may be shortened to 45 days or less when needed to enable awards before the end of the fiscal year.
For each competition, NIDRR convenes a preapplication technical assistance meeting for potential applicants (National Institute on Disability and Rehabilitation Research, 2010; also noted by NIDRR management). Upon request, staff members are also available for individual consultation. The staff can discuss only the application process with potential applicants,
however—not application content. NIDRR does not track whether those who participate in assistance events go on to apply for grants. Every NIDRR competition has received at least one application, which is all that is required to conduct the peer review process. However, not every competition has resulted in a grant being funded. (Chapter 4 provides information on the number of competitions between 2006 and 2009.)
RESULTS OF THE ASSESSMENT OF NIDRR’S PRIORITY-SETTING PROCESS
This section summarizes the results of interviews and surveys of NIDRR staff, grantees, and other stakeholder organizations concerning their views of NIDRR’s priority-setting process. See Chapter 2 for a description of the data collection and analysis methods used by the committee.
Perspectives of NIDRR Staff
In personal interviews, 16 NIDRR staff members were asked openended questions about their roles in the priority-setting process, their perspectives on the quality of the process, and suggestions for improvement. Two-thirds of the interviewees were project officers or direct supervisors of project officers; the remaining held administrative positions. The perspectives of staff coalesced around the processes of identifying priorities, moving priorities into the funding announcement phase, and disseminating priorities to the field.
Identification of Priority Topics
As stated earlier in the chapter, potential priority topics are identified through several sources. Staff comments focused on several of these sources. With regard to the roles of project officers, some play a very active part in proposing priority topics, conducting the literature reviews for proposed priorities, and writing the proposed priorities. Other project officers commented that their role in priority setting is fairly minimal. This difference appears to be based on individuals’ areas of expertise and allocation of time among their other duties (e.g., monitoring grants; organizing and leading peer review activities; managing program areas, such as Field Initiated Project [FIP] grants or Traumatic Brain Injury center or research grants). Staying well informed and current with the relevant literature entails a substantial time commitment on the part of project officers. This demand was noted as particularly challenging considering that staffing levels at NIDRR have fallen over the years.
Another source of potential priority topics is the results of previously funded grants. The Annual Performance Reports submitted by grantees are used to feed into priority setting more now than in the past. Staff mentioned that this is an example of how NIDRR’s processes have become more standardized and data driven. However, it was suggested that more systematic information should be collected on the stages of grant development, movement rates, and life cycles. Such information could potentially inform successive generations of priorities that would build on each other.
Staff also commented on other forces that influence the selection of priority topics. The broad scope of NIDRR’s mission, which includes all disabilities and all ages, relative to its limited budget sets a base of competing priorities to meet the needs of multiple target populations. While NIDRR is striving toward more scientific rigor in the research it funds, tensions still arise among differing paradigms that call for varying levels of scientific rigor to achieve research and development goals. There are also tensions at play between proponents of the continuance of large center grant priorities and those of smaller, flexible, more problem-focused or field-driven priorities. It was suggested that NIDRR should gather information from the field more regularly to inform priority setting and that NIDRR could benefit from the use of a national advisory board with more diverse disability research expertise to inform its priority-setting process.
Moving Priorities into the Funding Announcement Phase
Several comments were made about the challenges inherent in moving proposed priorities through the various organizational clearance levels to the final grant announcement phase. This process sometimes causes delays as questions about proposed priorities arise at different levels, and new justifications for or changes to proposed priorities must be made. More barriers are encountered with smaller, innovative grants than with the larger center grants. Delays in the clearance process impact the regularity of the timing of grant announcements and competitions.
Dissemination and Response
Staff commented that once grants have been announced, the specialty nature of certain priorities (e.g., for Model System grants) limits the number of researchers eligible to apply, whereas programs with field-driven priorities (e.g., FIP grants) have larger applicant pools. Concern was expressed that grant announcements are not reaching the largest intended audience and that there needs to be broader dissemination of priorities through listservs and routine dissemination to universities.
Perspectives of NIDRR Grantees
As part of the questionnaire dealing primarily with outputs that grantees submitted to the committee for review, grantees were asked to share their perspectives on NIDRR’s priority-setting process. Comments on this topic were received from 14 of the 30 grantees that were involved in the output assessment.
Some comments applauded NIDRR’s periodic formulation of 5-year LRPs, suggesting that significant field and community input increases the likelihood that the priorities established and resulting outputs can have the greatest potential impact. Consistent with a point made earlier in this chapter, it was commented that priorities are less prescriptive now than in the past, making it possible to explore areas of importance that would not otherwise have been addressed. It was also noted that NIDRR is able to fund studies that would not have been funded by other agencies in the early stages. As a result, important developmental studies can be incubated before other agencies will assume and extend them.
Some concern was expressed about the nature of priorities, which may shift from cycle to cycle. Grantees suggested that this can inhibit certain lines of research from continuing, affect the quality of research, and reduce the potential pool of applicants. Difficulties also were noted with respect to certain changes that have occurred in the course of grants (e.g., requirements for dissemination) that, although ultimately good, affected grantee activities and budgets.
These respondents also commented on some areas in which they thought priorities should be focused, including treatment and intervention studies in the rehabilitation process, which have received little attention; economic empowerment; and individual fellowship grants (similar to the National Institutes of Health [NIH] F series for predoctoral and postdoctoral training) for capacity building in many research areas covered by NIDRR that do not map neatly to NIH priorities. One respondent suggested that NIDRR should continue to focus on projects that go beyond what is encompassed by the medical rehabilitation research funded by NIH.
Perspectives of NIDRR Stakeholder Organizations
Because stakeholder input is a fundamental aspect of priority setting, the committee gathered information on stakeholder perspectives on NIDRR’s long-range planning and priority-setting processes by surveying three key stakeholder groups. First, in NIDRR’s most immediate network of stakeholders are other federal agencies with which it interacts and collaborates in order to achieve its mission. Second, professional associations are key stakeholders that represent the professional base of providers and researchers that work in the field. Third, the intended ultimate beneficiaries
of NIDRR research and development activities are persons with disabilities and their families. Advocacy organizations were surveyed to gather the perspectives of consumers, who are defined in this report as individuals with disabilities and their family members and/or authorized representatives.
Responding representatives of stakeholder organizations were asked closed- and open-ended questions about their role in the NIDRR planning and priority-setting processes, their perspectives on the processes, benefits their organization derived from NIDRR grants or outputs, and suggestions for enhancements to the priority-setting process.
As discussed in Chapter 2, the committee sent invitations to participate in the survey to executive directors of 130 professional and advocacy organizations identified by NIDRR as stakeholders and to organizations that are statutory members of the ICDR or nonstatutory members that have participated in ICDR special committees, as identified through a list provided by the ICDR. Seventy-two agencies responded to the survey with a response rate of 55 percent, an acceptable response rate for a questionnaire targeting executive directors (Baruch and Holtom, 2008). Of the 72 organizations responding to the survey, 26 percent described themselves as professional associations, 31 percent as advocacy organizations, and 43 percent as federal agencies. The questionnaire is included in Appendix B.
Responses to the quantitative survey items, summarized first, illuminate these stakeholder organizations’ relationships with NIDRR, how they were involved in NIDRR’s priority-setting process and their perspectives on the process, and the extent to which NIDRR grants are perceived as beneficial and their products are used. Responses to the open-ended questions are summarized next, organized by the major topics that emerged.
Summary of Quantitative Responses
Relationships with NIDRR Many of the respondents have interacted with NIDRR in multiple ways, but not all have the same level of familiarity with the agency. On a 5-point scale from 1 (not at all) to 5 (very much), 28 percent of the stakeholder organizations stated that they were “somewhat” familiar with NIDRR, 31 percent that they were “more than somewhat” familiar, and 40 percent that they were “very much” familiar. More than half of the organizations reported having interacted with the agency in the following ways: using the NIDRR website or related information sources to search for information, tools, or resources for working with consumers (61 percent) or to search for information about grants (56 percent); speaking with NIDRR representatives about professional issues (63 percent); and attending
NIDRR trainings sessions, workshops, or conferences (58 percent). With regard to collaboration, 51 percent reported having participated in general planning and special-purpose meetings convened by NIDRR, 38 percent having collaborated with NIDRR as a member of a consortium or professional association, and 29 percent having coordinated activities with NIDRR to support joint priorities and/or avoid duplication of effort.
Involvement in and perspectives on NIDRR priority setting Stakeholders were asked about their involvement in NIDRR’s priority-setting process through submission of comments and review of the LRP and/or funding priorities. A majority (63 percent) of the responding organizations stated that they had had opportunities to review and comment on NIDRR’s LRP or funding priorities. Approximately one-third of respondents reported that their organization had submitted comments to NIDRR.
Table 3-1 presents stakeholders’ perspectives on the extent to which they think that NIDRR’s long-range planning and priority-setting processes reflect important attributes of priority setting. Results include 72 respondents. However, no attribute was rated by all respondents, so the number of total responses (shown in the shaded first column of the table) for each attribute is less than 72. A relatively large number of respondents indicated that they did not know how to rate some of the attributes, which in part reflects the 28 percent of respondents stating that they were only somewhat familiar with NIDRR and the 37 percent stating that they had not had an opportunity to review or comment on NIDRR’s LRP or funding priorities (see above). The number of “don’t know” responses for each attribute is shown in the shaded column to the right of the “Total Responses” column. The number of valid responses for each attribute—responses that rated the attribute on the scale—is shown in the unshaded column to the right of the “don’t know” responses. The “don’t know” responses plus the valid responses equal the total responses for each attribute. The proportion of valid responses that characterized NIDRR’s priority setting at the various points along the five-point scale for each attribute is shown to the right of the “Valid Responses” column.
The attribute “relevant to your organization” had the highest number of favorable responses, with 67 percent of organizations responding “more than somewhat” or “very much.” The next two highest ratings were for the attributes “responsive to emerging issues in disability and rehabilitation/research” (51 percent) and “publicized” (48 percent). The priority-setting attribute that respondents thought NIDRR’s process least reflected was “responsive to stakeholder organization feedback”; only 29 percent responded “more than somewhat” or “very much” on this attribute, and 26 percent responded “less than somewhat.” A large number (28 out of 70, or 40 percent)
TABLE 3-1 Distribution of Ratings of the Extent to Which NIDRR’s Process Reflects Important Attributes of Priority Setting (72 respondents)
|Priority-Setting Attributes||Total* Responses||Number of “Don’t Know” Responses||Number of Valid Responses||Not at All (%)||Less Than Somewhat (%)||Somewhat (%)||More Than Somewhat (%)||Very Much (%)|
|Relevant to your organization||68||10||58||2||10||21||31||36|
|Responsive to emerging issues in disability and rehabilitation/research||71||15||56||—||13||36||31||20|
|Welcoming of stakeholder organization feedback||70||22||48||—||6||50||23||21|
|Responsive to stakeholder organization feedback||70||28||42||—||16||45||19||10|
of respondents answered they did not know for this attribute, a larger proportion than for the other attributes.
Another question on the survey asked how NIDRR’s long-range planning and priority-setting processes compare with those of other federal research agencies. With only 37 (42 percent) organizations responding to this question, 40 percent stated that the processes were about the same, 38 percent that NIDRR’s processes were stronger, and 22 percent that they were weaker.
Extent to which grants are perceived as beneficial and products are used To assess the extent to which NIDRR’s priority-setting process enhances the quality of final products, the survey asked stakeholders whether they perceive NIDRR grants as beneficial and whether they use the resulting products. Responses are summarized in Table 3-2. As in Table 3-1, the results in Table 3-2 include 72 respondents, as indicated in the shaded “Total Responses” column at the left of the table. Within the 72 responses, the numbers of “don’t know” and “not applicable” responses are indicated in the next two shaded columns to the right of the “Total Responses” column. The number of remaining valid responses to the questions is indicated in the unshaded column to the right of the “Not Applicable Responses” column. The proportion of valid responses that characterized NIDRR grants as beneficial and NIDRR products as useful along the five-point scale is shown to the right of the “Valid Responses” column.
The first row of Table 3-2 shows that 39 percent of respondents from stakeholder organizations indicated that the grants NIDRR funds are “somewhat” advancing the field in a direction that is beneficial for their organization and the members and consumers they represent, while 41 percent stated that the grants are “more than somewhat” to “very much” beneficial.
The survey also asked stakeholder organizations to rate the extent to which grant products (e.g., publications, websites, training materials, tools, devices, measures, interventions) are used. Thirty-three percent of respondents reported that the products of NIDRR-funded grants are used “somewhat,” while 28 percent stated they are used “more than somewhat” to “very much.”
The relatively large number of “don’t know” responses across several items was somewhat surprising in a sample of NIDRR-identified stakeholder organizations. A post hoc analysis of the results by type of stakeholder organization (professional association, advocacy organization, federal agency) showed that the “don’t know” responses could not be attributed systematically to any one type of stakeholder organization, although respondents from professional associations tended to give somewhat greater proportions of this response on several items. Results of the survey may indicate that further work is needed to inform and engage stakeholders with respect to the
process. However, it is also possible that the large number of “don’t know” responses is due to high turnover in the leadership positions the survey was targeting. Professional associations, advocacy organizations, and federal agencies often experience high turnover in their executive positions, so it is possible that some of these respondents were less than familiar with NIDRR because they were relatively new to their job.
Summary of Qualitative Responses
Four open-ended questions asked respondents to elaborate on their responses to the quantitative questions and additionally to offer suggestions for enhancing NIDRR’s long-range planning and priority-setting processes:
• Please use the space below to comment on the above, or any other types of interactions, your organization has had with NIDRR.
• What types of grants have served your organization, and the members and consumers that you represent?
• What are the three most important NIDRR-funded products used by your organization?
• What three things might NIDRR do to enhance its long-range planning and priority-setting processes?
One or more of the four open-ended questions were answered by 76 percent of respondents. As discussed in Chapter 2, responses to these questions were analyzed using standard qualitative methods. The results of the analysis were organized into five major topics: (1) the types of interactions the stakeholder organizations have had with NIDRR, (2) the uniqueness and strengths of NIDRR’s mission and research, (3) ways of enhancing collaboration with federal agencies, (4) strengths and needs related to outreach and dissemination, and (5) policies and practices related to priority setting. A summary of the stakeholders’ narrative responses is presented below under each of these topic headings.
Types of interactions with NIDRR Respondents from five federal agencies remarked that they have had strong collaboration with NIDRR over the years on such activities as coordinating research priorities, co-funding grants, participating in working groups on topics of mutual interest, and sharing information and expertise. Respondents from nine federal agencies stated that they participated in the work of the ICDR. Respondents from seven federal agencies stated that they have participated in other types of working groups, panels, and meetings to give NIDRR feedback about research priorities and LRPs. For example, representatives from two federal agencies commented on the strength of their collaboration with NIDRR:
TABLE 3-2 Distribution of Ratings of the Extent to Which NIDRR’s Grants Are Perceived as Beneficial and Their Products Are Used (72 respondents)
|Total* Responses||Number of “Don’t Know” Responses||Number of Valid Responses||Not at All (%)||Less Than Somewhat (%)||Somewhat (%)||More Than Somewhat (%)||Very Much (%)|
|Ate grants beneficial?||72||14||3||55||4||16||39||27||14|
|Ate products used?||72||13||2||57||9||30||33||14||14|
The synergy of the two agencies working together for 30 years may be a study of braiding funding and staff which is unique to the Federal Government. Sharing similar values and principles that reflect our individual Departments and what the field reflects is unusual.
We have extensive scientific and strategic interaction with NIDRR. We have coordinated research priorities with NIDRR plans and given presentations at each other’s venues. We and NIDRR officials have served on each other’s technical working groups, [and] shared technical information and expertise that is relevant for both groups.
Respondents from 3 professional associations and advocacy organizations remarked that they have met with NIDRR staff to prepare materials to be used in advocating with Congress for NIDRR funding. Respondents from 6 of these organizations stated that NIDRR staff have attended their meetings and forums to inform members about program mechanisms and research priorities.
Uniqueness and strengths of NIDRR’s mission and research Respondents from 7 federal agencies emphasized the benefits of NIDRR’s specialized mission as the touchstone for all of the agency’s program mechanisms and grants. They commented that NIDRR’s unique mission among all federal agencies requires grantees to use the state-of-the-art International Classification of Functioning, Disability and Health (ICF) as a framework for all research, training, and demonstration projects. They emphasized the benefits of NIDRR’s specialized scope of research, which includes people of all ages with psychiatric disabilities and comparative effectiveness research focused on health services for persons with disabilities. Respondents from 12 of the other stakeholder organizations and 8 federal agencies identified specific program mechanisms beneficial to their organization: Burn Model System (BMS), Traumatic Brain Injury Model System (TBIMS), Spinal Cord Injury Model System (SCIMS), FIP, Rehabilitation Research and Training Center (RRTC), Rehabilitation Engineering Research Center (RERC), and Disability and Rehabilitation Research Project-General (DRRP). Respondents from 7 professional associations and advocacy organizations applauded the funding of grants aimed at increasing accessibility for people with disabilities, as exemplified by grants related to communication for the deaf and hard-of-hearing; the availability of assistive technology; and physical and programmatic access to health care, education, housing, and employment. Respondents from 12 of these same organizations also emphasized the benefits of grants focused on community participation, such as employment and delivery of services to people with disabilities who are seeking employment; information technology issues that prevent these people from getting jobs; outcome research related to community participation, personal assistants,
and aging with developmental disabilities; and education and workforce development.
Ways of enhancing collaboration with federal agencies Respondents from four professional associations and advocacy organizations commented on the need to develop stronger collaborations with currently collaborating federal agencies and entities and to expand the number of new potential collaborators (e.g., more coordination of priorities with NIH institutes and the Department of Veterans Affairs). Respondents from two federal agencies suggested that NIDRR make more effective use of ICDR members in the strategic planning and priority-setting processes—for example, by delivering presentations to the group with specific review requests to members. Three federal agency respondents commented on the need to expand the depth and breadth of collaboration with agencies and co-funding of research projects with new agencies. Four federal agency respondents suggested that an important quality improvement initiative would be for NIDRR staff to become more knowledgeable about the relevant research efforts and products of other agencies so they could seek opportunities and avoid duplication of effort. Respondents from eight other stakeholder organizations suggested establishing quality improvement initiatives related to priority setting by, for example, investigating how other federal agencies establish priorities, communicating the details of the process, and incorporating feedback.
Strengths and needs related to outreach and dissemination Although respondents from four of the stakeholder organizations stated that they were unaware of available products of NIDRR grants, respondents from 8 professional and advocacy organizations and from 11 federal agencies commented that NIDRR’s website and other linked sources (e.g., National Rehabilitation Information Center [NARIC], the Center for International Rehabilitation Research Information and Exchange, the Repository of Recovery Resources, and RERC and RRTC grant websites) provide easy access to information, resources, technical assistance, and research results in topic areas such as: asistive technology use, evidence-based information on employment, disability statistics, guidelines for accessible digital media and guidelines, information on health promotion for persons with disabilities, and technical assistance related to the Americans with Disabilities Act.
Several suggestions also were made for ways to enhance NIDRR’s outreach and dissemination activities to inform the agency’s planning and priority setting. For example, respondents from 5 federal agencies recommended expanding and refining NIDRR’s outreach plan by including experts from a wider variety of disciplines and heads of research and evaluation departments of other federal agencies. For greater outreach, 4 federal agency respondents suggested increasing stakeholders’ awareness of NIDRR
accomplishments by initiating a new public relations campaign using “bragsheets” and disseminating information about companies and vendors that use products based on the results of NIDRR-funded projects. Seven professional association and advocacy organization respondents commented on the need to develop more user-friendly ways for NIDRR to receive feedback from stakeholders (e.g., provide summaries of larger planning documents, develop electronic capacity to insert comments directly into LRPs undergoing public review and comment). In addition, 12 respondents from these same stakeholder organizations encouraged NIDRR to make more effective use of the disability communities through enhanced communication approaches, such as webinars and presentations at existing annual meetings.
Policies and practices related to priority setting Finally, four federal agency respondents suggested that NIDRR should review and adapt its current policies and practices that guide the long-range planning and priority-setting processes to include standardized timelines and a review of progress halfway through the 5-year LRPs. One federal agency respondent proposed a specific set of guidelines for priority setting that would
(a) point to a national vision for research that is aspirational; (b) capture the imagination and support of the disability community at-large; (c) [be] based on the best available evidence and information; (d) require projects to produce measurable outcomes and results; (e) promote initiatives that are supported over an appropriate timeframe; (f) [be] multidisciplinary in nature; and (g) stimulate a collaborative approach to solutions among stakeholders.
Respondents from three other federal agencies suggested in general that NIDRR should increase funding for planning and priority setting. With regard to how grants are structured, respondents from four professional associations and advocacy organizations remarked that NIDRR should refine the inclusion of policies and practices in funded studies and projects to ensure that grantees planning to include people with disabilities as collaborators or partners actually follow through in doing so. In this same vein, one federal agency respondent commented that the “diversity” criteria NIDRR uses to score applications should be critically reviewed so as to consider requirements for including consumers in the planning and implementation phases of all projects.
CONCLUSIONS AND RECOMMENDATIONS
NIDRR has a legislatively based process for formulating 5-year LRPs and for establishing its funding priorities. In proposing priority topics, NIDRR has specific criteria that must be met and subjects potential priorities to key questions. It also has written procedures for staff to follow in proposing priorities that will be recommended to the Office of Special
Education and Rehabilitative Services for funding. Processes for gathering initial input on LRPs and priorities appear to be inclusive but vary from cycle to cycle. However, formal public comments on proposed LRPs and priorities are consistently obtained via the Federal Register. NIDRR has not established a standing disability and rehabilitation research advisory council, which is included in its legislation. As of the time of this writing, the agency is still operating under the LRP for 2005 to 2009 because of critical input from formal public comments and the long-term absence of a permanent Director. However, a permanent Director was recently hired.
Seventy-one percent of stakeholder organizations surveyed were “more than somewhat” or “very much” familiar with NIDRR. Attributes of NIDRR’s priority-setting process that were most endorsed included relevance of the process to their organization, responsiveness to emerging issues in disability and rehabilitation research, and degree to which the process is publicized. Respondents thought NIDRR’s priority-setting process least reflected the attribute of responsiveness to stakeholder organization feedback. In open-ended remarks, stakeholder organizations emphasized the benefits of NIDRR’s specialized mission and scope of research, its specific program mechanisms, and funding of grants aimed at increasing accessibility and community participation for people with disabilities. Respondents commented on the need to develop stronger collaborations with federal agencies and entities. Although stakeholder organizations commented on the benefits of NIDRR’s website and linked sources, suggestions were made for ways to enhance the agency’s outreach and dissemination activities. Respondents suggested that NIDRR should review and adapt its current policies and practices that guide the long-range planning and priority-setting processes to include standardized timelines.
The committee offers recommendations regarding NIDRR’s prioritysetting process in four areas: formation of an advisory council, strategic planning, establishment of a standard calendar, and expanded dissemination of NIAs.
Formation of an Advisory Council
NIDRR has a broad and diverse mission that makes it challenging to set priorities that are responsive to the current state of the science and the needs of the stakeholder community. Currently, NIDRR relies on staff, the portfolio of existing projects, recent findings from completed grants, and the current research literature, as well as guidance from federal partners, for input to the priority-setting process. Title II, section 205, of the Rehabilitation Act instructs NIDRR, subject to the availability of appropriations, to establish a 12-member standing disability and rehabilitation research advisory council to support its priority setting. While NIDRR formed a steering committee
to support the development of the LRP for 2005 to 2009, a standing body has never been formed.
Recommendation 3-16: NIDRR should fulfill the statutory mandate to form and utilize a standing disability and rehabilitation research advisory council to advise on the priority-setting process and provide input for priority setting.
In the committee’s view, it is somewhat anomalous that NIDRR does not have a standing advisory body. Most federal funding agencies, including NIH, the National Science Foundation (NSF), and the National Institute for Occupational Safety and Health (NIOSH), benefit from the use of standing advisory bodies. A standing advisory body is likely to add stability and continuity to both NIDRR’s long-range planning and priority-setting processes.
Additionally, a disability and rehabilitation research advisory council would be an efficient way to obtain expertise from the scientific community, as well as input from members of NIDRR’s diverse constituency of stakeholder organizations and consumers. As documented in the survey data presented in this chapter, a number of relevant stakeholder organizations, including professional associations, advocacy associations, and federal agencies, are less familiar with NIDRR than they might be expected to be. An advisory council could provide another way for these stakeholder organizations to interact meaningfully with NIDRR.
Furthermore, an advisory council would represent an important, regular forum through which consumers could interact with NIDRR. According to Title II, at least half of the council members are to be consumers. Consumer participation on the council should help build and strengthen productive partnerships between NIDRR and the populations it serves. Participation would allow consumers to have direct input into the advice that guides NIDRR’s work, which is vital for those affected by the research the agency funds (Ahmed and Palermo, 2010). Such input could benefit NIDRR in many ways. Consumer input could educate scientists about the research being conducted. As the Director’s Consumer Liaison Group of the National Cancer Institute (2011) states, consumer input can “improve research outcomes by identifying new approaches, promoting innovation, recognizing unforeseen risks or barriers, and identifying unintended consequences.” According to the NIH Director’s Council of Public Representatives, through participation on the advisory board, consumers may have the ability to represent their communities to research funders such as NIDRR
6 The committee’s recommendations are numbered according to the chapter of the report in which they appear.
(National Institutes of Health, 2011). At the same time, they can gain a more informed understanding of NIDRR and the research being conducted. This understanding in turn can be used to better inform their communities about NIDRR’s work.
Title II states that, in addition to including six consumers, the membership of the council should represent the community of rehabilitation professionals and the community of rehabilitation researchers. Additional populations that NIDRR regularly tries to engage include national, state, and local rehabilitation agencies and facilities; administrators and practitioners in agencies serving persons with disabilities; other federal agencies; educators of rehabilitation professionals and their students; and the general public (National Institute on Disability and Rehabilitation Research, 2011). The committee recognizes that NIDRR, like other federal research agencies, will face challenges in capturing the broad diversity of perspectives held by this large collection of different populations. However, the committee feels strongly that, like other federal research agencies, NIDRR can meet these challenges.
The delay in NIDRR’s LRP for 2010 to 2014, resulting from the negative reaction to the plan by the field as well as the absence of a Director to guide the process, suggests a breakdown in the LRP process. As the LRP is the foundation of priority setting, the priority-setting process is likewise affected. Long-range planning and priority-setting processes may be enhanced in ways that are likely to help NIDRR avoid such a breakdown in the future.
In its review of NIDRR’s long-range planning process, the committee was never presented by NIDRR with a document explaining the steps in the process from start to finish. Instead, the process has varied from the creation of one LRP to the next. It is the Committee’s viewpoint that LRP processes are better conceived as a documented series of logical stages. The methods used to gather input from stakeholders also have varied during the development of each plan. In addition, NIDRR documentation (discussed above) indicated that input into the creation of NPPs is drawn from such sources as experts in the field, the current literature, and state-of-the-science conferences. The committee applauds these efforts but notes that this input is somewhat informal and the process is not systematic. The first formal input is not received until after potential topics have been narrowed down and the NPPs have been published for comment in the Federal Register. The committee believes more could be done to involve stakeholders earlier in the process of identifying potential priority topics.
Recommendation 3-2: NIDRR should use a structured, consistent, and inclusive strategic planning process to develop its Long-Range Plans and priorities.
To begin implementing this recommendation, NIDRR should first formalize and document the structure of its long-range planning process. The advisory council recommended above could assist in this effort. Once the structure of the process has been documented, NIDRR will be able to follow it consistently in the development of future LRPs. Subsequent changes to the process, and the rationale behind them, should be documented as well.
NIDRR should also establish a regular form of interaction with stakeholders in the long-range planning process, and might also make efforts to expand the stakeholder groups that are included in the process. The recommended advisory council might assist in regularizing and expanding the inclusivity of the process. With regard to priority setting, NIDRR should seek more formal input from the field on potential priority topics earlier in the process and pursue maximum participation from stakeholders. While incorporating a broad range of stakeholder views can be a slow, arduous process, the added structure and consistency, along with the regular influence of an advisory council, will allow the process to be as efficient as possible.
One source NIDRR might consider in planning a more structured, consistent, and inclusive process is the Canadian Institutes of Health Research’s Knowledge Translation module on Deliberative Priority Setting (Campbell, 2010). Included in Campbell’s work, derived from an extensive literature review, is a description of seven ideal elements of agenda-setting and/or priority-setting processes:
• planning of the process, including anticipating the needs, barriers, and challenges for all the remaining steps and identifying the leadership that will guide the process;
• stakeholder identification, analysis, and engagement, which involves purposefully identifying the proper stakeholders to include in the process, understanding both the opportunities and risks associated with including stakeholders, and formulating appropriate engagement strategies;
• knowledge management, which ensures that all stakeholders have the same information, and any stakeholders with less technical expertise are supported such that they can understand all the information;
• interpretive workshops, which involve gathering with stakeholders to define criteria, establish weights, and then apply the criteria to identify the most relevant issues or topics;
• translation of the issues or topics identified in the workshops into actionable research agendas and priorities;
• publication and validation, which involve making the draft long-range plans and/or proposed priorities available to all stakeholders to ensure alignment with the goals of the communities and clarity of the expected outcomes; and
• revision or an appeal mechanism, the presence of which is necessary to allow stakeholders to communicate disagreement with draft long-range plans and/or proposed priorities constructively and through which such concerns can be comprehensively addressed.
The document also contains an extensive literature review on different types of priority-setting processes.
Additionally, NIDRR might consider the long-range planning and priority setting of other funding agencies, including NIH, the National Science Foundation, and the National Institute for Occupational Safety and Health, which have sought to integrate long-range planning and priority-setting processes through specific initiatives such as the NIH Roadmap, the NSF Strategic Plan, and the National Occupational Research Agenda.
Establishment of a Standard Calendar
For many program mechanisms, NIDRR has not established a regular schedule for drafting and approving priorities and NIAs and disseminating them to the field. In exploring this issue further, the committee reviewed Federal Register publication dates of NIAs for all of the program mechanisms over the last 5 years. There appeared to be no regular timing pattern of the publication of NIAs either within or across program mechanisms. The Department of Education has a lengthy review and approval process for obtaining clearance for the release of priorities and NIAs. The variability in the length of the clearance process may be an important factor, among others, that impacts the timing of the release of NIAs. The irregular or delayed release of NIAs may affect NIDRR’s ability to provide individuals sufficient notice of grant opportunities or an optimal amount of time to complete applications. An irregular schedule may discourage the best investigators from submitting applications. Additionally, certain program mechanisms (such as Model Systems) include collaboration between institutions. Irregular posting and shortened response times hamper the ability of applicants to identity and recruit appropriate collaborators. These factors are likely to limit the number of investigators who apply and adversely affect the quality of the applications they submit. Additionally, young investigators less familiar with NIDRR are more likely to pursue grants from other agencies.
Recommendation 3-3: NIDRR should utilize a standard calendar for the setting of priorities, publication of notices inviting applications, submission of applications, and peer review meetings to improve the efficiency of the process.
NIDRR has made efforts to standardize the schedule for NIAs for its various program mechanisms, such as FIP, SBIR, Advanced Rehabilitation Research Training (ARRT), and Switzer Fellowship. However, NIDRR has been unable to standardize the schedule for most of its program mechanisms. Even within the constraints of its lengthy review process for publishing NIAs, a standardized calendar should be developed. The committee suggests that program mechanisms competed on a yearly basis have a consistent annual schedule for the submission and review of applications. For multiyear grants, the committee recommends that NIDRR establish a long-range operational plan listing projected future grant application submission dates, pending funding availability in that fiscal year. With a submission timeline, NIDRR could establish when NPPs would have to be published for public comment and also when NFPs would have to be published. In addition to better supporting applicants, establishing a standard calendar would reduce some of the burden on NIDRR staff caused by the current unanticipatable priority-setting timeline. Delays in priority setting also often cause delays and/or shortened timelines in the peer review process (see Chapter 4). A standard calendar could potentially help address those delays as well.
Expanded Dissemination of Notices Inviting Application
While creating and utilizing a standard calendar is likely to increase the number of researchers already familiar with NIDRR who will apply for grants, the committee thinks more effort needs to be made to expand this pool. NIDRR publishes NIAs in the Federal Register and on Grants.gov, and also uses a contractor to notify former grantees and others who, via the contractor’s webpage, express an interest in receiving NIAs. Given the vast number of scientists whose work is relevant to disability and rehabilitation research, however, NIDRR would benefit from more active efforts to solicit interest in its funding announcements.
Recommendation 3-4: NIDRR should expand its efforts to disseminate notices inviting applications to new potential applicants, including developing a communication strategy to ensure that the notices reach new audiences.
Increasing the pool of applicants may ultimately increase the quality of the work NIDRR funds. A logical means of expanding dissemination is
through sending notices to the disability-relevant professional organizations and agencies and disability and rehabilitation research organizations that make up NIDRR’s stakeholder network. NIDRR should also make efforts to expand the network to organizations not yet familiar with the agency. Additionally, NIDRR should begin sending notices to university departments and offices of sponsored research. This could perhaps be accomplished through collaboration with other federal research programs that regularly send funding notices to universities. Increasing the number of potential applicants may also contribute to increasing the pool of reviewers, addressed by Recommendation 4-1 in the next chapter.
Ahmed, S.M., and Palermo, A.-G.S. (2010). Community engagement in research: Frameworks for education and peer review. American Journal of Public Health, 100(8), 1,380-1,387. Baruch, Y., and Holtom, B.C. (2008). Survey levels and trends in organizational research. Human Relations, 61(8), 1,139-1,160.
Campbell, S. (2010). Deliberative priority setting. Ottawa, Canada: Canadian Institutes of Health Research.
National Cancer Institute. (2011). About the Director’s Consumer Liaison Group. Available: http://dclg.cancer.gov/about [June 4, 2011].
National Institute on Disability and Rehabilitation Research. (2006). Department of Education:
National Institute on Disability and Rehabilitation Research—Notice of Final Long-Range Plan for Fiscal Years 2005–2009. Federal Register, 71(31), 8,166–8,200.
National Institute on Disability and Rehabilitation Research. (2009). Briefing book for The National Academies. Unpublished document. Washington, DC: National Institute on Disability and Rehabilitation Research.
National Institute on Disability and Rehabilitation Research. (2010). Overview of the NIDRR’s peer review process (what happens between OPP’s priority review and OPP’s slate review). Unpublished document. Washington, DC: National Institute on Disability and Rehabilitation Research.
National Institute on Disability and Rehabilitation Research. (2011). Target populations for NARIC. Unpublished document. Washington, DC: National Institute on Disability and Rehabilitation Research.
National Institutes of Health. (2011). About Council of Public Representatives. Available: http://copr.nih.gov/secondary/about/ [June 4, 2011].
The Rehabilitation Act of 1973, as amended. Pub. L. No. 93-112. Available: http://www2.ed.gov/policy/speced/reg/narrative.html [January 21, 2011].
The Small Business Innovation Research Development Act, Public. L. No. 97-219, 106 Stat. 2965 (2000).