Below are the first 10 and last 10 pages of uncorrected machine-read text (when available) of this chapter, followed by the top 30 algorithmically extracted key phrases from the chapter as a whole.
Intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text on the opening pages of each chapter. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.
Do not use for reproduction, copying, pasting, or reading; exclusively for search engines.
OCR for page 56
3 NIDRR’s Priority-Setting Processes This chapter addresses the following key study question: Key Question #1. To what extent is NIDRR’s priority-writing process conducted in such a way as to enhance the quality of the final results? As used in the study question, the term “priority-writing” process en- compasses many aspects of priority setting, including gathering input from multiple sources (e.g., the field, stakeholder organizations, grantees, other agencies, and persons with disabilities and their families), identifying poten- tial topics and determining priorities for funding, writing the proposed pri- orities and having them cleared for release, and publishing notices inviting applications (NIAs)1 on these priorities.2 The term “priority setting” is used synonymously with “priority writing” in this report to clarify that the focus of the committee’s evaluation was on this larger priority-setting process. Priority setting not only reflects the National Institute on Disability and Rehabilitation Research’s (NIDRR’s) intent to influence the advancement of research in targeted areas but also offers specific funding opportunities for potential grant applications to the agency. The announced priorities should therefore be developed and communicated in a manner that attracts the best researchers and encourages participation in disability and rehabilita- 1A “notice inviting applications” (NIA) is NIDRR’s announcement to the field of the open- ing of a grant competition. It is similar to the terms “request for proposals” and “request for applications” used by other agencies. 2 “Priority” in this context means the topic of the grant competition for which scientists submit applications to investigate. 56
OCR for page 57
57 NIDRR’S PRIORITY-SETTING PROCESSES tion research. Attracting the largest pool of applications from which to select grantees increases the chances for the highest-quality outputs. When establishing its priorities, the agency needs to consider continuity from one funding cycle to another, as well as identify future research challenges and societal needs. In the context of this committee’s work, it is challenging to link priority setting directly to specific output quality. The quality of NIDRR’s research portfolio, grants, and outputs is the product of multiple complex factors, in- cluding the priority-setting process, funding levels, the peer review process, and the scientific quality of the grantees. It is clear, however, that NIDRR’s priority-setting process has a positive impact on the quality of the final out- puts. In this chapter, the committee reviews and assesses that process and makes recommendations for its improvement. The chapter has three major sections. The first describes NIDRR’s priority-setting process based on existing documentation and interviews with NIDRR executives. The second presents an assessment of the process, based on data gathered from NIDRR staff, grantees, and stakeholder orga- nizations. The final section offers the committee’s conclusions and recom- mendations with respect to NIDRR’s priority-setting process. DESCRIPTION OF NIDRR’S PRIORITY-SETTING PROCESS The following description is based on existing documentation, such as legislation, the Federal Register, NIDRR and the U.S. Department of Educa- tion (ED) policies and procedures, NIDRR’s Long-Range Plan (LRP), and NIAs, as well as interviews with NIDRR and ED management.3 Legislative Foundation The Rehabilitation Act (1973, as amended) authorizes the formula grant programs of vocational rehabilitation, supported employment, independent living, and client assistance. Title II4 of the act—Research and Training—au- thorizes the majority of NIDRR’s research activities, while Section 21 of the Act and the Small Business Innovation Research (SBIR) Development Act (2000) require NIDRR to reserve small portions of its budget for those two specific program mechanisms. Section 202 of Title II states that NIDRR’s Director will lay out funding priorities and covered activities in a 5-year plan 3 The committee conducted interviews with NIDRR and ED management in four sessions during summer 2010 and one session in spring 2011. 4 The other six titles are Title I, Vocational Rehabilitation Services; Title III, Professional Development and Special Projects and Demonstrations; Title IV, National Council on Disabil- ity; Title V, Rights and Advocacy; Title VI, Employment Opportunities for Individuals with Disabilities; and Title VII, Independent Living Services and Centers for Independent Living.
OCR for page 58
58 REVIEW OF DISABILITY AND REHABILITATION RESEARCH that will be published in the Federal Register. The plan will utilize at least 90 percent of NIDRR funds for extramural research. Chapter 1 provides a description of NIDRR’s program mechanisms. The Long-Range Plan Title II, section 202(h), of the Rehabilitation Act states that, starting in October 1998 and every subsequent fifth October, the NIDRR Director will develop and publish in the Federal Register for public comment a draft LRP outlining NIDRR’s priorities, explaining the basis for those priorities, and providing a broad framework for the funding of research aimed at achiev- ing the priorities. Every fifth subsequent June, after consideration of public comments, the final LRP with revisions will be submitted to the appropriate members of Congress. The LRP is required to • I dentify any covered activity that should be conducted under section 202 (NIDRR) and section 204 (Research and Other Covered Activities) re- specting the full inclusion and integration into society of individuals with disabilities, especially in the area of employment; • D etermine the funding priorities for covered activities to be conducted under section 202 and section 204; and • S pecify appropriate goals and timetables for covered activities to be con- ducted under this section and section 204. The LRP is the foundation that guides the development of grant priorities. In developing the 5-year LRP, the Director must consider input from the Commissioner of the Rehabilitation Services Administration, the Commis- sioner of the Administration on Developmental Disabilities, the National Council on Disability, the Interagency Committee on Disability Research (ICDR), consumers, organizations representing people with disabilities, researchers, service providers, and other appropriate entities. Additionally, Title II, section 205, states that, subject to the availability of appropria- tions, a 12-member standing disability and rehabilitation research advisory council should advise the Director on the development and revision of each LRP; however, NIDRR has never formed such a council. According to NIDRR management, the agency’s use of long-range plans extends back to the development of the first such plan, which spanned 1981 to 1985. Based on information gathered from 4,000 agencies and organiza- tions, this plan laid out a comprehensive landscape of research and develop- ment needs and topics across all disability and age groups. More recently, the LRP for 1999 to 2003 was developed in 1998 by key management staff following the reauthorization of the Rehabilitation Act. The content of the plan was drawn from staff member contributions, and commissioned papers helped inform the process. The major innovations in the 1999 to 2003 LRP
OCR for page 59
59 NIDRR’S PRIORITY-SETTING PROCESSES included the “paradigm of disability” (which emphasizes the contextual nature of disability as a product of individual and societal factors) and the five research domains (Employment Outcomes, Health and Function, Technology for Access and Function, Independent Living and Community Integration, and Associated Disability Research Areas). This plan also en- couraged participatory action research, which generated a great deal of positive feedback. According to NIDRR management, the agency was able to accomplish the vast majority of the research goals outlined in this LRP. During the next presidential administration (in 2000), however, the LRP was criticized for not being written in outcome-oriented terms. The development of the next LRP, covering 2005 to 2009, differed somewhat from that of the prior LRP. Agency staff utilized considerable outside assistance to gather input from the public. These efforts included hiring a contractor to support teleconferences linking NIDRR staff with centers around the country and forming a steering committee. Subteams of the staff wrote sections of the plan corresponding to the research do- mains and capacity building. The major innovation in the LRP for 2005 to 2009 was a logic model that supported and provided a structure for more outcome-oriented work. The development of the LRP for 2010 to 2014 was impacted by the change in presidential administrations, the departure of NIDRR’s Director, and a shorter time frame in which to develop the plan. Input was solicited through a national teleconference, but the development of this plan was driven more by department leadership. During the public comment phase of the plan’s development, close to 100 comments were received, many of which were critical of the increased focus on employment included in this plan. As a result, the plan was put on hold, and NIDRR is still operating under the LRP for 2005 to 2009.5 At the time of this writing, a new LRP was under development but had not yet been published for public comment. In addition, after several years of the position being filled only on a temporary basis, a new permanent Director has recently been hired. Priority-Setting Stages in the Grant Competition Process To meet the objectives laid out in the LRP, NIDRR formulates priori- ties for research grants and development projects aimed at generating new knowledge and products, along with supporting knowledge translation and capacity-building activities (National Institute on Disability and Rehabilita- tion Research, 2006). The priority-setting process generally starts 2 years before funding announcements are published and involves several stages (see 5 NIDRR’s 1999-2003 LRP, 2005-2009 LRP, and information about the development of the 2010-2014 LRP can be found on the Publications and Products page of NIDRR’s website, http://www2.ed.gov/rschstat/research/pubs/index.html [January 4, 2012].
OCR for page 60
60 REVIEW OF DISABILITY AND REHABILITATION RESEARCH Months Priority-Setting Stages 1 2 3 4 5 6 7 8 9 10 11 12 13 14 Identify grant priorities (3-5 months) Develop and publish notice of proposed priority (2-4 months) Public comment (1-2 months) Develop notice of final priority (1-2 months) Develop notice inviting applications (1-2 months) Develop application kit (1-2 months) Publish notice of final priority and notice inviting applications in Federal Register (1 month) FIGURE 3-1 Timeline for NIDRR’s priority-setting process. SOURCE: Committee developed from National Institute on Disability and Rehabilitation Research (2009). Figure 3-1 for an illustration of the general timeline). According to NIDRR management, whileeditablehas standard procedures for the various stages Figure 3-1, NIDRR of priority setting, described below, the length of time required for new proposed priorities to obtain ED approval and clearance varies. Identification of Grant Priorities In planning its priorities for funding, NIDRR considers changes to its budget, as well as the funding level of expiring grants, and determines the amount of money that will be available for new grant competitions (Na- tional Institute on Disability and Rehabilitation Research, 2009). Looking at current priority areas, NIDRR determines whether grantees have been productive, and whether more work remains to be done or a topic has been exhausted. NIDRR management stated that new priority areas may be added if there is additional funding, if new technology or science suggests
OCR for page 61
61 NIDRR’S PRIORITY-SETTING PROCESSES new areas for research, if experts from the field suggest a new area, or if the administration suggests new priorities. To inform the identification and development of potential priority top- ics, NIDRR’s Research Sciences Division (RSD) considers the LRP, other strategic and performance goals, the portfolio of existing projects, recent findings from completed grants, and the current research literature, as well as key guidance documents from federal partners, professional associations, and consumer organizations (National Institute on Disability and Rehabili- tation Research, 2006, 2009). Contractors may be asked to conduct specific literature searches and factor in feedback from the field in response to prior grant cycles. New priority areas may also be generated by NIDRR-funded state-of-the-science conferences. The ICDR helps facilitate information sharing and partnerships that can contribute to the identification of new priority areas. In addition, the NIDRR Director and the Assistant Secretary of the Office of Special Education and Rehabilitative Services often identify potential priority topics of high importance (National Institute on Disability and Rehabilitation Research, 2009). NIDRR staff, as key sources of input into potential priority topics, continually monitor the literature in their areas of expertise to stay cur- rent, to identify potential areas for funding, to determine field capacity, and also to identify active researchers who might serve as peer reviewers of proposals. When resources are available to support travel, staff also stay current with the state of the science through attending meetings of orga- nizations such as the Rehabilitation Engineering and Assistive Technol- ogy Society of North America, the American Congress of Rehabilitation Medicine, the American Public Health Association, the Gerontological Society of America, or the National Council of Rehabilitation Educators. As priority topics are being considered, NIDRR assesses their alignment with the following criteria (National Institute on Disability and Rehabilita- tion Research, 2009, p. 58). Research priorities must • R eflect and incorporate state-of-the-science in a specific area • F ulfill a demonstrated need for new knowledge • B uild upon prior research-based knowledge and advance the stage of knowledge development in specific areas • H ave real-world relevance to individuals with disabilities • P rioritize meeting the needs of individuals with disabilities from minority backgrounds • S upport the NIDRR mission • R eflect the thinking and guidance of the current LRP Considering all of the possible priority topics, emerging opportunities, and ongoing needs, NIDRR management then asks staff to utilize their knowledge of the relevant literature and the field in developing succinct,
OCR for page 62
62 REVIEW OF DISABILITY AND REHABILITATION RESEARCH one-paragraph descriptions of the most promising potential topics, along with rationales and recommendations for which topics should be selected. The descriptions address the following areas (National Institute on Disabil- ity and Rehabilitation Research, 2009): • I ncidence and prevalence of the condition to be targeted • P roblem or need addressed, in terms of improved policy, practice, behavior, or system capacity • H ow the priority topic builds upon the state of the science in this topic area • N IDRR’s historical investment in this topic area, key findings from NIDRR investments in this area, and how the proposed priority topic builds upon these investments • H ow the priority topic relates to the work of NIDRR’s relevant federal partners, or other organizations that fund research • R eal-world relevance of this topic to people with disabilities • H ow this priority topic addresses the needs of individuals with dis- abilities from minority backgrounds • H ow the priority topic fits under NIDRR LRP goals, objectives, and strategies The written descriptions of the potential topics are discussed with the NIDRR Director who then approves topics to be published as priorities and presents these to the Assistant Secretary of the Office of Special Education and Rehabilitative Services. The process of identifying funding priorities generally takes 3 to 5 months or longer. Notices of Proposed Priority Once priority topics have been indentified, the second stage in NIDRR’s priority-setting process is to draft and publish notices of proposed priority (NPPs). According to NIDRR management, NIDRR staff follow a template in drafting selected topics into NPPs, which are NIDRR’s first notice to the research community about each grant solicitation. The NPP explains the nature of the problem and describes the requirements of the priority (e.g., for Burn Injury Model System grants, establish a multidisciplinary system that begins with acute care and encompasses rehabilitation services specifi- cally designed to meet the needs of individuals with burn injuries). NIDRR’s style of NPPs has shifted over time from more to less prescriptive. Each NPP is reviewed and cleared at multiple levels within NIDRR, the Office of Special Education and Rehabilitative Services, and ED (including the Office of the General Counsel, the Budget Office, the Deputy Secretary, and the Executive Secretary) (National Institute on Disability and Rehabili-
OCR for page 63
63 NIDRR’S PRIORITY-SETTING PROCESSES tation Research, 2009). The NPPs must also be reviewed and approved by the Office of Management and Budget. According to NIDRR management, multiple rounds of comments and revisions are not uncommon. The result- ing NPP is published in the Federal Register. During a response period, the public may submit comments on the proposed priority. The NPP de- velopment and clearance process often takes 2-4 months, while the public comment period lasts for 1-2 months, which includes the period of time for NIDRR to respond to public comments. Notices of Final Priority After the close of the public comment period, NIDRR staff review comments on the NPP, respond to the comments, and develop a notice of final priority (NFP) (National Institute on Disability and Rehabilitation Research, 2009). The draft NFP is reviewed and cleared at several levels within NIDRR, the Office of Special Education and Rehabilitative Services, and ED, much as is done with the NPP. Again, multiple revisions are not uncommon. Development of the NFP generally takes 1-2 months, and when approved it is published in the Federal Register. Notices Inviting Applications and Application Kits Concurrently with the NFP, NIDRR drafts the NIA and develops an application kit for each priority (development of these materials generally takes 1-2 months) (National Institute on Disability and Rehabilitation Re- search, 2009). The NIA contains information about the grant competition, such as submission deadlines, dollar amounts, and the procedure for submit- ting an application. The NIA goes through the same clearance process as the NFP and is published with the NFP in the Federal Register. NIAs are also published on Grants.gov. In addition, NIDRR uses a contractor to notify former grantees and others who, via the contractor’s webpage, express an interest in receiving NIAs. Application kits contain application forms, the NFP and NIA, regula- tions, and the peer review criteria. The application kits are available on Grants.gov and NIDRR’s website. The standard period between publica- tion of the NFP and NIA and the application deadline is 60 days, although NIDRR management stated that the period may be shortened to 45 days or less when needed to enable awards before the end of the fiscal year. For each competition, NIDRR convenes a preapplication technical as- sistance meeting for potential applicants (National Institute on Disability and Rehabilitation Research, 2010; also noted by NIDRR management). Upon request, staff members are also available for individual consultation. The staff can discuss only the application process with potential applicants,
OCR for page 64
64 REVIEW OF DISABILITY AND REHABILITATION RESEARCH however—not application content. NIDRR does not track whether those who participate in assistance events go on to apply for grants. Every NIDRR competition has received at least one application, which is all that is required to conduct the peer review process. However, not every competition has resulted in a grant being funded. (Chapter 4 provides information on the number of competitions between 2006 and 2009.) RESULTS OF THE ASSESSMENT OF NIDRR’S PRIORITY-SETTING PROCESS This section summarizes the results of interviews and surveys of NIDRR staff, grantees, and other stakeholder organizations concerning their views of NIDRR’s priority-setting process. See Chapter 2 for a description of the data collection and analysis methods used by the committee. Perspectives of NIDRR Staff In personal interviews, 16 NIDRR staff members were asked open- ended questions about their roles in the priority-setting process, their per- spectives on the quality of the process, and suggestions for improvement. Two-thirds of the interviewees were project officers or direct supervisors of project officers; the remaining held administrative positions. The perspec- tives of staff coalesced around the processes of identifying priorities, moving priorities into the funding announcement phase, and disseminating priorities to the field. Identification of Priority Topics As stated earlier in the chapter, potential priority topics are identified through several sources. Staff comments focused on several of these sources. With regard to the roles of project officers, some play a very active part in proposing priority topics, conducting the literature reviews for proposed priorities, and writing the proposed priorities. Other project officers com- mented that their role in priority setting is fairly minimal. This difference appears to be based on individuals’ areas of expertise and allocation of time among their other duties (e.g., monitoring grants; organizing and leading peer review activities; managing program areas, such as Field Initiated Proj- ect [FIP] grants or Traumatic Brain Injury center or research grants). Staying well informed and current with the relevant literature entails a substantial time commitment on the part of project officers. This demand was noted as particularly challenging considering that staffing levels at NIDRR have fallen over the years.
OCR for page 65
65 NIDRR’S PRIORITY-SETTING PROCESSES Another source of potential priority topics is the results of previously funded grants. The Annual Performance Reports submitted by grantees are used to feed into priority setting more now than in the past. Staff mentioned that this is an example of how NIDRR’s processes have become more stan- dardized and data driven. However, it was suggested that more systematic information should be collected on the stages of grant development, move- ment rates, and life cycles. Such information could potentially inform suc- cessive generations of priorities that would build on each other. Staff also commented on other forces that influence the selection of priority topics. The broad scope of NIDRR’s mission, which includes all dis- abilities and all ages, relative to its limited budget sets a base of competing priorities to meet the needs of multiple target populations. While NIDRR is striving toward more scientific rigor in the research it funds, tensions still arise among differing paradigms that call for varying levels of scientific rigor to achieve research and development goals. There are also tensions at play between proponents of the continuance of large center grant priorities and those of smaller, flexible, more problem-focused or field-driven priorities. It was suggested that NIDRR should gather information from the field more regularly to inform priority setting and that NIDRR could benefit from the use of a national advisory board with more diverse disability research expertise to inform its priority-setting process. Moving Priorities into the Funding Announcement Phase Several comments were made about the challenges inherent in moving proposed priorities through the various organizational clearance levels to the final grant announcement phase. This process sometimes causes delays as questions about proposed priorities arise at different levels, and new jus- tifications for or changes to proposed priorities must be made. More barriers are encountered with smaller, innovative grants than with the larger center grants. Delays in the clearance process impact the regularity of the timing of grant announcements and competitions. Dissemination and Response Staff commented that once grants have been announced, the specialty nature of certain priorities (e.g., for Model System grants) limits the number of researchers eligible to apply, whereas programs with field-driven priorities (e.g., FIP grants) have larger applicant pools. Concern was expressed that grant announcements are not reaching the largest intended audience and that there needs to be broader dissemination of priorities through listservs and routine dissemination to universities.
OCR for page 66
66 REVIEW OF DISABILITY AND REHABILITATION RESEARCH Perspectives of NIDRR Grantees As part of the questionnaire dealing primarily with outputs that grant- ees submitted to the committee for review, grantees were asked to share their perspectives on NIDRR’s priority-setting process. Comments on this topic were received from 14 of the 30 grantees that were involved in the output assessment. Some comments applauded NIDRR’s periodic formulation of 5-year LRPs, suggesting that significant field and community input increases the likelihood that the priorities established and resulting outputs can have the greatest potential impact. Consistent with a point made earlier in this chapter, it was commented that priorities are less prescriptive now than in the past, making it possible to explore areas of importance that would not otherwise have been addressed. It was also noted that NIDRR is able to fund studies that would not have been funded by other agencies in the early stages. As a result, important developmental studies can be incubated before other agencies will assume and extend them. Some concern was expressed about the nature of priorities, which may shift from cycle to cycle. Grantees suggested that this can inhibit certain lines of research from continuing, affect the quality of research, and reduce the potential pool of applicants. Difficulties also were noted with respect to certain changes that have occurred in the course of grants (e.g., require- ments for dissemination) that, although ultimately good, affected grantee activities and budgets. These respondents also commented on some areas in which they thought priorities should be focused, including treatment and intervention studies in the rehabilitation process, which have received little attention; economic empowerment; and individual fellowship grants (similar to the National Institutes of Health [NIH] F series for predoctoral and postdoctoral train- ing) for capacity building in many research areas covered by NIDRR that do not map neatly to NIH priorities. One respondent suggested that NIDRR should continue to focus on projects that go beyond what is encompassed by the medical rehabilitation research funded by NIH. Perspectives of NIDRR Stakeholder Organizations Because stakeholder input is a fundamental aspect of priority set- ting, the committee gathered information on stakeholder perspectives on NIDRR’s long-range planning and priority-setting processes by surveying three key stakeholder groups. First, in NIDRR’s most immediate network of stakeholders are other federal agencies with which it interacts and col- laborates in order to achieve its mission. Second, professional associations are key stakeholders that represent the professional base of providers and researchers that work in the field. Third, the intended ultimate beneficiaries
OCR for page 72
72 TABLE 3-2 Distribution of Ratings of the Extent to Which NIDRR’s Grants Are Perceived as Beneficial and Their Products Are Used (72 respondents) Scale Number of Number of “Not Number Not at Less Than More Than Total “Don’t Know” Applicable” of Valid All Somewhat Somewhat Somewhat Very Much Responses Responses Responses Responses (%) (%) (%) (%) (%) Question Are grants 72 14 3 55 4 16 39 27 14 beneficial? Are products 72 13 2 57 9 30 33 14 14 used? SOURCE: Generated by the committee based on data from the stakeholder questionnaire.
OCR for page 73
73 NIDRR’S PRIORITY-SETTING PROCESSES The synergy of the two agencies working together for 30 years may be a study of braiding funding and staff which is unique to the Federal Government. Sharing similar values and principles that reflect our individual Departments and what the field reflects is unusual. We have extensive scientific and strategic interaction with NIDRR. We have coordinated research priorities with NIDRR plans and given presentations at each other’s venues. We and NIDRR officials have served on each other’s tech- nical working groups, [and] shared technical information and expertise that is relevant for both groups. Respondents from 3 professional associations and advocacy organiza- tions remarked that they have met with NIDRR staff to prepare materials to be used in advocating with Congress for NIDRR funding. Respondents from 6 of these organizations stated that NIDRR staff have attended their meetings and forums to inform members about program mechanisms and research priorities. Uniqueness and strengths of NIDRR’s mission and research Respondents from 7 federal agencies emphasized the benefits of NIDRR’s specialized mission as the touchstone for all of the agency’s program mechanisms and grants. They commented that NIDRR’s unique mission among all federal agencies requires grantees to use the state-of-the-art International Classifica- tion of Functioning, Disability and Health (ICF) as a framework for all re- search, training, and demonstration projects. They emphasized the benefits of NIDRR’s specialized scope of research, which includes people of all ages with psychiatric disabilities and comparative effectiveness research focused on health services for persons with disabilities. Respondents from 12 of the other stakeholder organizations and 8 federal agencies identified specific program mechanisms beneficial to their organization: Burn Model System (BMS), Traumatic Brain Injury Model System (TBIMS), Spinal Cord Injury Model System (SCIMS), FIP, Rehabilitation Research and Training Center (RRTC), Rehabilitation Engineering Research Center (RERC), and Dis- ability and Rehabilitation Research Project-General (DRRP). Respondents from 7 professional associations and advocacy organizations applauded the funding of grants aimed at increasing accessibility for people with dis- abilities, as exemplified by grants related to communication for the deaf and hard-of-hearing; the availability of assistive technology; and physical and programmatic access to health care, education, housing, and employment. Respondents from 12 of these same organizations also emphasized the ben- efits of grants focused on community participation, such as employment and delivery of services to people with disabilities who are seeking employment; information technology issues that prevent these people from getting jobs; outcome research related to community participation, personal assistants,
OCR for page 74
74 REVIEW OF DISABILITY AND REHABILITATION RESEARCH and aging with developmental disabilities; and education and workforce development. Ways of enhancing collaboration with federal agencies Respondents from four professional associations and advocacy organizations commented on the need to develop stronger collaborations with currently collaborating federal agencies and entities and to expand the number of new potential collaborators (e.g., more coordination of priorities with NIH institutes and the Department of Veterans Affairs). Respondents from two federal agencies suggested that NIDRR make more effective use of ICDR members in the strategic planning and priority-setting processes—for example, by delivering presentations to the group with specific review requests to members. Three federal agency respondents commented on the need to expand the depth and breadth of collaboration with agencies and co-funding of research projects with new agencies. Four federal agency respondents suggested that an im- portant quality improvement initiative would be for NIDRR staff to become more knowledgeable about the relevant research efforts and products of other agencies so they could seek opportunities and avoid duplication of effort. Respondents from eight other stakeholder organizations suggested establishing quality improvement initiatives related to priority setting by, for example, investigating how other federal agencies establish priorities, communicating the details of the process, and incorporating feedback. Strengths and needs related to outreach and dissemination Although re- spondents from four of the stakeholder organizations stated that they were unaware of available products of NIDRR grants, respondents from 8 profes- sional and advocacy organizations and from 11 federal agencies commented that NIDRR’s website and other linked sources (e.g., National Rehabilitation Information Center [NARIC], the Center for International Rehabilitation Research Information and Exchange, the Repository of Recovery Resources, and RERC and RRTC grant websites) provide easy access to information, resources, technical assistance, and research results in topic areas such as: asis- tive technology use, evidence-based information on employment, disability statistics, guidelines for accessible digital media and guidelines, information on health promotion for persons with disabilities, and technical assistance related to the Americans with Disabilities Act. Several suggestions also were made for ways to enhance NIDRR’s outreach and dissemination activities to inform the agency’s planning and priority setting. For example, respondents from 5 federal agencies recom- mended expanding and refining NIDRR’s outreach plan by including experts from a wider variety of disciplines and heads of research and evaluation departments of other federal agencies. For greater outreach, 4 federal agency respondents suggested increasing stakeholders’ awareness of NIDRR ac-
OCR for page 75
75 NIDRR’S PRIORITY-SETTING PROCESSES complishments by initiating a new public relations campaign using “brag- sheets” and disseminating information about companies and vendors that use products based on the results of NIDRR-funded projects. Seven profes- sional association and advocacy organization respondents commented on the need to develop more user-friendly ways for NIDRR to receive feedback from stakeholders (e.g., provide summaries of larger planning documents, develop electronic capacity to insert comments directly into LRPs undergo- ing public review and comment). In addition, 12 respondents from these same stakeholder organizations encouraged NIDRR to make more effective use of the disability communities through enhanced communication ap- proaches, such as webinars and presentations at existing annual meetings. Policies and practices related to priority setting Finally, four federal agency respondents suggested that NIDRR should review and adapt its current poli- cies and practices that guide the long-range planning and priority-setting processes to include standardized timelines and a review of progress half- way through the 5-year LRPs. One federal agency respondent proposed a specific set of guidelines for priority setting that would (a) point to a national vision for research that is aspirational; (b) capture the imagination and support of the disability community at-large; (c) [be] based on the best available evidence and information; (d) require projects to produce measurable outcomes and results; (e) promote initiatives that are supported over an appropriate timeframe; (f) [be] multidisciplinary in nature; and (g) stimulate a collaborative approach to solutions among stakeholders. Respondents from three other federal agencies suggested in general that NIDRR should increase funding for planning and priority setting. With regard to how grants are structured, respondents from four professional associations and advocacy organizations remarked that NIDRR should re- fine the inclusion of policies and practices in funded studies and projects to ensure that grantees planning to include people with disabilities as collabo- rators or partners actually follow through in doing so. In this same vein, one federal agency respondent commented that the “diversity” criteria NIDRR uses to score applications should be critically reviewed so as to consider requirements for including consumers in the planning and implementation phases of all projects. CONCLUSIONS AND RECOMMENDATIONS NIDRR has a legislatively based process for formulating 5-year LRPs and for establishing its funding priorities. In proposing priority topics, NIDRR has specific criteria that must be met and subjects potential pri- orities to key questions. It also has written procedures for staff to follow in proposing priorities that will be recommended to the Office of Special
OCR for page 76
76 REVIEW OF DISABILITY AND REHABILITATION RESEARCH Education and Rehabilitative Services for funding. Processes for gathering initial input on LRPs and priorities appear to be inclusive but vary from cycle to cycle. However, formal public comments on proposed LRPs and priorities are consistently obtained via the Federal Register. NIDRR has not established a standing disability and rehabilitation research advisory council, which is included in its legislation. As of the time of this writing, the agency is still operating under the LRP for 2005 to 2009 because of critical input from formal public comments and the long-term absence of a permanent Director. However, a permanent Director was recently hired. Seventy-one percent of stakeholder organizations surveyed were “more than somewhat” or “very much” familiar with NIDRR. Attributes of NIDRR’s priority-setting process that were most endorsed included rele- vance of the process to their organization, responsiveness to emerging issues in disability and rehabilitation research, and degree to which the process is publicized. Respondents thought NIDRR’s priority-setting process least re- flected the attribute of responsiveness to stakeholder organization feedback. In open-ended remarks, stakeholder organizations emphasized the benefits of NIDRR’s specialized mission and scope of research, its specific program mechanisms, and funding of grants aimed at increasing accessibility and community participation for people with disabilities. Respondents com- mented on the need to develop stronger collaborations with federal agencies and entities. Although stakeholder organizations commented on the benefits of NIDRR’s website and linked sources, suggestions were made for ways to enhance the agency’s outreach and dissemination activities. Respondents suggested that NIDRR should review and adapt its current policies and practices that guide the long-range planning and priority-setting processes to include standardized timelines. The committee offers recommendations regarding NIDRR’s priority- setting process in four areas: formation of an advisory council, strategic planning, establishment of a standard calendar, and expanded dissemination of NIAs. Formation of an Advisory Council NIDRR has a broad and diverse mission that makes it challenging to set priorities that are responsive to the current state of the science and the needs of the stakeholder community. Currently, NIDRR relies on staff, the portfo- lio of existing projects, recent findings from completed grants, and the cur- rent research literature, as well as guidance from federal partners, for input to the priority-setting process. Title II, section 205, of the Rehabilitation Act instructs NIDRR, subject to the availability of appropriations, to establish a 12-member standing disability and rehabilitation research advisory council to support its priority setting. While NIDRR formed a steering committee
OCR for page 77
77 NIDRR’S PRIORITY-SETTING PROCESSES to support the development of the LRP for 2005 to 2009, a standing body has never been formed. Recommendation 3-16: NIDRR should fulfill the statutory mandate to form and utilize a standing disability and rehabilitation research advisory council to advise on the priority-setting process and pro- vide input for priority setting. In the committee’s view, it is somewhat anomalous that NIDRR does not have a standing advisory body. Most federal funding agencies, includ- ing NIH, the National Science Foundation (NSF), and the National Insti- tute for Occupational Safety and Health (NIOSH), benefit from the use of standing advisory bodies. A standing advisory body is likely to add stability and continuity to both NIDRR’s long-range planning and priority-setting processes. Additionally, a disability and rehabilitation research advisory council would be an efficient way to obtain expertise from the scientific community, as well as input from members of NIDRR’s diverse constituency of stake- holder organizations and consumers. As documented in the survey data presented in this chapter, a number of relevant stakeholder organizations, including professional associations, advocacy associations, and federal agen- cies, are less familiar with NIDRR than they might be expected to be. An advisory council could provide another way for these stakeholder organiza- tions to interact meaningfully with NIDRR. Furthermore, an advisory council would represent an important, regu- lar forum through which consumers could interact with NIDRR. Accord- ing to Title II, at least half of the council members are to be consumers. Consumer participation on the council should help build and strengthen productive partnerships between NIDRR and the populations it serves. Participation would allow consumers to have direct input into the advice that guides NIDRR’s work, which is vital for those affected by the research the agency funds (Ahmed and Palermo, 2010). Such input could benefit NIDRR in many ways. Consumer input could educate scientists about the research being conducted. As the Director’s Consumer Liaison Group of the National Cancer Institute (2011) states, consumer input can “improve research outcomes by identifying new approaches, promoting innovation, recognizing unforeseen risks or barriers, and identifying unintended conse- quences.” According to the NIH Director’s Council of Public Representa- tives, through participation on the advisory board, consumers may have the ability to represent their communities to research funders such as NIDRR 6 The committee’s recommendations are numbered according to the chapter of the report in which they appear.
OCR for page 78
78 REVIEW OF DISABILITY AND REHABILITATION RESEARCH (National Institutes of Health, 2011). At the same time, they can gain a more informed understanding of NIDRR and the research being conducted. This understanding in turn can be used to better inform their communities about NIDRR’s work. Title II states that, in addition to including six consumers, the mem- bership of the council should represent the community of rehabilitation professionals and the community of rehabilitation researchers. Additional populations that NIDRR regularly tries to engage include national, state, and local rehabilitation agencies and facilities; administrators and practi- tioners in agencies serving persons with disabilities; other federal agencies; educators of rehabilitation professionals and their students; and the general public (National Institute on Disability and Rehabilitation Research, 2011). The committee recognizes that NIDRR, like other federal research agencies, will face challenges in capturing the broad diversity of perspectives held by this large collection of different populations. However, the committee feels strongly that, like other federal research agencies, NIDRR can meet these challenges. Strategic Planning The delay in NIDRR’s LRP for 2010 to 2014, resulting from the nega- tive reaction to the plan by the field as well as the absence of a Director to guide the process, suggests a breakdown in the LRP process. As the LRP is the foundation of priority setting, the priority-setting process is likewise affected. Long-range planning and priority-setting processes may be en- hanced in ways that are likely to help NIDRR avoid such a breakdown in the future. In its review of NIDRR’s long-range planning process, the committee was never presented by NIDRR with a document explaining the steps in the process from start to finish. Instead, the process has varied from the creation of one LRP to the next. It is the Committee’s viewpoint that LRP processes are better conceived as a documented series of logical stages. The methods used to gather input from stakeholders also have varied during the development of each plan. In addition, NIDRR documentation (discussed above) indicated that input into the creation of NPPs is drawn from such sources as experts in the field, the current literature, and state-of-the-science conferences. The committee applauds these efforts but notes that this input is somewhat informal and the process is not systematic. The first formal input is not received until after potential topics have been narrowed down and the NPPs have been published for comment in the Federal Register. The committee believes more could be done to involve stakeholders earlier in the process of identifying potential priority topics.
OCR for page 79
79 NIDRR’S PRIORITY-SETTING PROCESSES Recommendation 3-2: NIDRR should use a structured, consistent, and inclusive strategic planning process to develop its Long-Range Plans and priorities. To begin implementing this recommendation, NIDRR should first for- malize and document the structure of its long-range planning process. The advisory council recommended above could assist in this effort. Once the structure of the process has been documented, NIDRR will be able to follow it consistently in the development of future LRPs. Subsequent changes to the process, and the rationale behind them, should be documented as well. NIDRR should also establish a regular form of interaction with stake- holders in the long-range planning process, and might also make efforts to expand the stakeholder groups that are included in the process. The recom- mended advisory council might assist in regularizing and expanding the in- clusivity of the process. With regard to priority setting, NIDRR should seek more formal input from the field on potential priority topics earlier in the process and pursue maximum participation from stakeholders. While incor- porating a broad range of stakeholder views can be a slow, arduous process, the added structure and consistency, along with the regular influence of an advisory council, will allow the process to be as efficient as possible. One source NIDRR might consider in planning a more structured, con- sistent, and inclusive process is the Canadian Institutes of Health Research’s Knowledge Translation module on Deliberative Priority Setting (Campbell, 2010). Included in Campbell’s work, derived from an extensive literature review, is a description of seven ideal elements of agenda-setting and/or priority-setting processes: • p lanning of the process, including anticipating the needs, barriers, and challenges for all the remaining steps and identifying the leader- ship that will guide the process; • s takeholder identification, analysis, and engagement, which in- volves purposefully identifying the proper stakeholders to include in the process, understanding both the opportunities and risks as- sociated with including stakeholders, and formulating appropriate engagement strategies; • k nowledge management, which ensures that all stakeholders have the same information, and any stakeholders with less technical expertise are supported such that they can understand all the information; • i nterpretive workshops, which involve gathering with stakeholders to define criteria, establish weights, and then apply the criteria to identify the most relevant issues or topics;
OCR for page 80
80 REVIEW OF DISABILITY AND REHABILITATION RESEARCH • t ranslation of the issues or topics identified in the workshops into actionable research agendas and priorities; • p ublication and validation, which involve making the draft long- range plans and/or proposed priorities available to all stakeholders to ensure alignment with the goals of the communities and clarity of the expected outcomes; and • r evision or an appeal mechanism, the presence of which is neces- sary to allow stakeholders to communicate disagreement with draft long-range plans and/or proposed priorities constructively and through which such concerns can be comprehensively addressed. The document also contains an extensive literature review on different types of priority-setting processes. Additionally, NIDRR might consider the long-range planning and prior- ity setting of other funding agencies, including NIH, the National Science Foundation, and the National Institute for Occupational Safety and Health, which have sought to integrate long-range planning and priority-setting processes through specific initiatives such as the NIH Roadmap, the NSF Strategic Plan, and the National Occupational Research Agenda. Establishment of a Standard Calendar For many program mechanisms, NIDRR has not established a regular schedule for drafting and approving priorities and NIAs and disseminating them to the field. In exploring this issue further, the committee reviewed Federal Register publication dates of NIAs for all of the program mecha- nisms over the last 5 years. There appeared to be no regular timing pattern of the publication of NIAs either within or across program mechanisms. The Department of Education has a lengthy review and approval process for obtaining clearance for the release of priorities and NIAs. The variability in the length of the clearance process may be an important factor, among oth- ers, that impacts the timing of the release of NIAs. The irregular or delayed release of NIAs may affect NIDRR’s ability to provide individuals sufficient notice of grant opportunities or an optimal amount of time to complete ap- plications. An irregular schedule may discourage the best investigators from submitting applications. Additionally, certain program mechanisms (such as Model Systems) include collaboration between institutions. Irregular post- ing and shortened response times hamper the ability of applicants to identity and recruit appropriate collaborators. These factors are likely to limit the number of investigators who apply and adversely affect the quality of the applications they submit. Additionally, young investigators less familiar with NIDRR are more likely to pursue grants from other agencies.
OCR for page 81
81 NIDRR’S PRIORITY-SETTING PROCESSES Recommendation 3-3: NIDRR should utilize a standard calendar for the setting of priorities, publication of notices inviting appli- cations, submission of applications, and peer review meetings to improve the efficiency of the process. NIDRR has made efforts to standardize the schedule for NIAs for its various program mechanisms, such as FIP, SBIR, Advanced Rehabilitation Research Training (ARRT), and Switzer Fellowship. However, NIDRR has been unable to standardize the schedule for most of its program mecha- nisms. Even within the constraints of its lengthy review process for publish- ing NIAs, a standardized calendar should be developed. The committee sug- gests that program mechanisms competed on a yearly basis have a consistent annual schedule for the submission and review of applications. For multi- year grants, the committee recommends that NIDRR establish a long-range operational plan listing projected future grant application submission dates, pending funding availability in that fiscal year. With a submission timeline, NIDRR could establish when NPPs would have to be published for public comment and also when NFPs would have to be published. In addition to better supporting applicants, establishing a standard calendar would reduce some of the burden on NIDRR staff caused by the current unanticipatable priority-setting timeline. Delays in priority setting also often cause delays and/or shortened timelines in the peer review process (see Chapter 4). A standard calendar could potentially help address those delays as well. Expanded Dissemination of Notices Inviting Application While creating and utilizing a standard calendar is likely to increase the number of researchers already familiar with NIDRR who will apply for grants, the committee thinks more effort needs to be made to expand this pool. NIDRR publishes NIAs in the Federal Register and on Grants.gov, and also uses a contractor to notify former grantees and others who, via the contractor’s webpage, express an interest in receiving NIAs. Given the vast number of scientists whose work is relevant to disability and rehabilitation research, however, NIDRR would benefit from more active efforts to solicit interest in its funding announcements. Recommendation 3-4: NIDRR should expand its efforts to dis- seminate notices inviting applications to new potential applicants, including developing a communication strategy to ensure that the notices reach new audiences. Increasing the pool of applicants may ultimately increase the quality of the work NIDRR funds. A logical means of expanding dissemination is
OCR for page 82
82 REVIEW OF DISABILITY AND REHABILITATION RESEARCH through sending notices to the disability-relevant professional organizations and agencies and disability and rehabilitation research organizations that make up NIDRR’s stakeholder network. NIDRR should also make efforts to expand the network to organizations not yet familiar with the agency. Ad- ditionally, NIDRR should begin sending notices to university departments and offices of sponsored research. This could perhaps be accomplished through collaboration with other federal research programs that regularly send funding notices to universities. Increasing the number of potential ap- plicants may also contribute to increasing the pool of reviewers, addressed by Recommendation 4-1 in the next chapter. REFERENCES Ahmed, S.M., and Palermo, A.-G.S. (2010). Community engagement in research: Frameworks for education and peer review. American Journal of Public Health, 100(8), 1,380-1,387. Baruch, Y., and Holtom, B.C. (2008). Survey levels and trends in organizational research. Hu- man Relations, 61(8), 1,139-1,160. Campbell, S. (2010). Deliberative priority setting. Ottawa, Canada: Canadian Institutes of Health Research. National Cancer Institute. (2011). About the Director’s Consumer Liaison Group. Available: http://dclg.cancer.gov/about [June 4, 2011]. National Institute on Disability and Rehabilitation Research. (2006). Department of Education: National Institute on Disability and Rehabilitation Research—Notice of Final Long- Range Plan for Fiscal Years 2005–2009. Federal Register, 71(31), 8,166–8,200. National Institute on Disability and Rehabilitation Research. (2009). Briefing book for The National Academies. Unpublished document. Washington, DC: National Institute on Disability and Rehabilitation Research. National Institute on Disability and Rehabilitation Research. (2010). Overview of the NIDRR’s peer review process (what happens between OPP’s priority review and OPP’s slate re- view). Unpublished document. Washington, DC: National Institute on Disability and Rehabilitation Research. National Institute on Disability and Rehabilitation Research. (2011). Target populations for NARIC. Unpublished document. Washington, DC: National Institute on Disability and Rehabilitation Research. National Institutes of Health. (2011). About Council of Public Representatives. Available: http://copr.nih.gov/secondary/about/ [June 4, 2011]. The Rehabilitation Act of 1973, as amended. Pub. L. No. 93-112. Available: http://www2. ed.gov/policy/speced/reg/narrative.html [January 21, 2011]. The Small Business Innovation Research Development Act, Public. L. No. 97-219, 106 Stat. 2965 (2000).