1

Introduction

The National Institute on Disability and Rehabilitation Research (NIDRR) is located within the Office of Special Education and Rehabilitative Services in the U.S. Department of Education. NIDRR is one of multiple federal agencies that invest in disability and rehabilitation research, but it has a broader mandate than those other agencies. Established by the 1978 amendments to the Rehabilitation Act of 1973,1 NIDRR’s mission “is to generate new knowledge and promote its effective use to improve the abilities of people with disabilities to perform activities of their choice in the community, and also to expand society’s capacity to provide full opportunities and accommodations for its citizens with disabilities” (National Institute on Disability and Rehabilitation Research, 2006c, p. 8,167). The agency pursues this mission by funding research and related activities focused on maximizing the full inclusion, social integration, employment, and independent living of individuals of all ages with disabilities (National Institute on Disability and Rehabilitation Research, 2009a).

NIDRR prides itself on being proactive in establishing program performance measures and developing accountability data systems to track the progress of its grantees. An electronic annual reporting system is used to collect data from grantees on many aspects of grant operation and outputs. Various formative and summative evaluation approaches have been used to assess the quality of the performance and results of the agency’s research portfolio and its grantees. Prompted by the need to provide more data on

_______________

1 NIDRR was originally called the National Institute on Handicapped Research in the 1978 amendments to the act; its name was changed to its present form by the 1986 amendments.



The National Academies | 500 Fifth St. N.W. | Washington, D.C. 20001
Copyright © National Academy of Sciences. All rights reserved.
Terms of Use and Privacy Statement



Below are the first 10 and last 10 pages of uncorrected machine-read text (when available) of this chapter, followed by the top 30 algorithmically extracted key phrases from the chapter as a whole.
Intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text on the opening pages of each chapter. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.

Do not use for reproduction, copying, pasting, or reading; exclusively for search engines.

OCR for page 17
1 Introduction The National Institute on Disability and Rehabilitation Research (NIDRR) is located within the Office of Special Education and Rehabilita- tive Services in the U.S. Department of Education. NIDRR is one of mul- tiple federal agencies that invest in disability and rehabilitation research, but it has a broader mandate than those other agencies. Established by the 1978 amendments to the Rehabilitation Act of 1973,1 NIDRR’s mission “is to generate new knowledge and promote its effective use to improve the abilities of people with disabilities to perform activities of their choice in the community, and also to expand society’s capacity to provide full opportunities and accommodations for its citizens with disabilities” (Na- tional Institute on Disability and Rehabilitation Research, 2006c, p. 8,167). The agency pursues this mission by funding research and related activities focused on maximizing the full inclusion, social integration, employment, and independent living of individuals of all ages with disabilities (National Institute on Disability and Rehabilitation Research, 2009a). NIDRR prides itself on being proactive in establishing program perfor- mance measures and developing accountability data systems to track the progress of its grantees. An electronic annual reporting system is used to collect data from grantees on many aspects of grant operation and outputs. Various formative and summative evaluation approaches have been used to assess the quality of the performance and results of the agency’s research portfolio and its grantees. Prompted by the need to provide more data on 1 NIDRR was originally called the National Institute on Handicapped Research in the 1978 amendments to the act; its name was changed to its present form by the 1986 amendments. 17

OCR for page 17
18 REVIEW OF DISABILITY AND REHABILITATION RESEARCH its program results, in 2009 NIDRR requested that the National Research Council (NRC) conduct an external evaluation of some of the agency’s key processes and assess the quality of outputs produced by NIDRR grantees (National Institute on Disability and Rehabilitation Research, 2009a). This report presents the results of that evaluation. This chapter introduces the report by first explaining the charge to the committee that conducted the evaluation. It then provides background information on NIDRR, including its unique legislative mandate, the types of research it funds and its grant funding mechanisms, its role related to the Interagency Committee on Disability Research, and its budget and staff. The third section summarizes approaches NIDRR has used in the past to evaluate its grantees and distinguishes them from the methods used by the committee. The final section provides an overview of the remaining chapters of the report. CHARGE TO THE COMMITTEE This ad hoc committee, with oversight by the NRC’s Board on Human- Systems Integration, was charged with developing and implementing a framework and evaluation design for the purpose of (1) reviewing NIDRR’s priority-writing and grant review processes (“process evaluation”) and (2) assessing the quality of grantee outputs for a sample of grants representing the NIDRR portfolio (“summative evaluation”). Additionally, the com - mittee was charged with assessing the design and implementation of its summative evaluation process and making recommendations for additional evaluations that might follow this effort. The evaluation was to be con- ducted over a period of 2 years between October 2009 and September 2011. The results of this evaluation are intended to provide NIDRR with a better understanding of the quality of its grantees’ outputs and how the agency can best manage an important and evolving research portfolio that meets its strategic goals and objectives while regularly assessing and improving its performance as required by the Government Performance and Results Act (GPRA) of 1993 (U.S. Office of Management and Budget, 2009a). The specific questions guiding the evaluation were as follows: Process Evaluation 1. To what extent is NIDRR’s priority-writing process conducted in such a way as to enhance the quality of the final results? 2. To what extent are the peer reviews of grant applications done in such a way as to enhance the quality of the final results? 3. What planning and budgetary processes does the grantee use to promote high-quality outputs?

OCR for page 17
19 INTRODUCTION Summative Evaluation 4. To what extent are the final outputs from NIDRR grants of high quality? 5. To what extent are the results of the reviewed research and devel- opment outputs used to inform new projects by both the grantee and NIDRR? This external evaluation was designed and conducted independently by the NRC committee. However, it was funded by NIDRR, and the contract stipulated in advance certain features and parameters for the design of the summative evaluation component (study questions 4 and 5) as follows: Level of analysis—The level of analysis was specified to be grantee • outputs. The outputs to be assessed included four types as defined by NIDRR: (1) publications; (2) tools, measures, and intervention protocols; (3) technology products and devices; and (4) informa- tional products. Sampling of grants—The sample for the evaluation was to include • a minimum of 30 grants. The selection criteria were to include rep- resentation across all program mechanisms and grants in their last year of funding. Number of outputs to be reviewed—Two outputs were to be re- • viewed for each project being carried out under each grant selected for the evaluation. The number of projects depended on the size of the grant and varied from 1 for small investigator-initiated grants to 10 for large center grants. For NIDRR, the ultimate objective of the output review was to gain an understanding of what was being produced by the time grants came to an end with respect to value added for the disability community and new knowledge produced for the field. Because the focus of the summative evaluation was primarily on the quality of outputs produced by the sampled grants, it is important to clarify what was not included in the committee’s charge. The charge did not include a larger focus on how the grants were implemented with respect to their original objectives and whether the ob- jectives were achieved, how the outputs of the various projects were linked and how they developed over time, and how the quality of outputs differed across the various NIDRR program mechanisms. The charge also did not include an assessment of the long-term impact of outputs on persons with disabilities. However, suggestions are made by the committee on the impor- tance of these types of evaluation foci in Chapter 6 of the report.

OCR for page 17
20 REVIEW OF DISABILITY AND REHABILITATION RESEARCH BACKGROUND ON NIDRR NIDRR’s purpose is to . . . provide for research, demonstration projects, training, and related activi- ties to maximize the full inclusion and integration into society, employment, independent living, family support, and economic and social self-sufficiency of individuals with disabilities of all ages . . .; promote the transfer of rehabilitation technology to individuals with disabilities through research and demonstration projects . . .; ensure the widespread distribution, in usable formats, of practical scientific and technological information . . .; identify effective strategies that enhance the opportunities of individuals with disabilities to engage in employ- ment . . .; and increase opportunities for researchers who are members of traditionally underserved populations, including researchers who are members of minority groups and researchers who are individuals with disabilities (The Rehabilitation Act of 1973, as amended). NIDRR’s Unique Role The multidimensional challenges faced by individuals with disabilities in employment, housing, public accommodations, education, transportation, communication, recreation, health services, institutionalization, voting, and public services, as outlined in the Rehabilitation Act of 1973 as amended, necessitate a comprehensive, holistic approach to NIDRR’s mission (Na- tional Institute on Disability and Rehabilitation Research, 2009a). Across NIDRR’s agenda, the central focus is on the whole person with a disability, whose ability to function and quality of life are dependent on the complex interactions among personal, societal, and environmental factors. NIDRR is one of three major disability-focused research sponsors in the federal government,2 but it plays a unique role in that its target population includes all disability types and all age groups. (See the Rehabilitation Act of 1973 as amended for a complete list of NIDRR’s mandated responsibili- ties.) While other federal research entities fund prevention, cure, and acute rehabilitation research, NIDRR also invests in rehabilitation research that is tied more closely to longer-term outcomes, such as independence, com- munity participation, and employment (National Institute on Disability 2 One of the other major sponsors is the National Center for Medical and Rehabilitation Research (NCMRR), which is located within the National Institute of Child Health and Human Development of the U.S. Department of Health and Human Services. NCMRR funds research aimed at developing scientific knowledge to enhance the health, productivity, independence, and quality of life of people with disabilities. The other major sponsor is the Department of Veterans Affairs, which, through its Veterans Administration Rehabilitation Research and De- velopment Service, funds extramural basic and biomedical research and an intramural research program in 13 rehabilitation centers.

OCR for page 17
21 INTRODUCTION and Rehabilitation Research, 2009b). The World Health Organization’s International Classification of Functioning, Disability and Health (ICF) (World Health Organization, 2001) provides context for understanding NIDRR-supported research and development. The ICF classifies disability and health along a continuum from body function and structure to activities and participation, while accounting for environmental and personal factors. In these terms, NIDRR funds primarily research and development grants that are aimed at increasing functional abilities to facilitate greater self- determination and participation of individuals with disabilities in the home, community, school, and workplace as defined in the ICF (National Institute on Disability and Rehabilitation Research, 2009b). NIDRR’s Research Domains Table 1-1 summarizes the various types of research that NIDRR funds in five outcome domains: (1) employment, (2) participation and community living, (3) health and function, (4) technology for access and function, and (5) disability demographics (National Institute on Disability and Rehabili- tation Research, 2009a, pp. 3-6). Although most grants address more than TABLE 1-1 NIDRR Research Domains and Topics Number of Research Awards as of Domain January 2009 Research Topics Employment 16 (7%) • areer planning; job entry, advancement, C and retention; transitions in moving from financial dependency to self-sufficiency or from underemployment into work that is consistent with the individual’s strengths, abilities, and interests. • ethods for integrating the unique needs of M employers and disability populations to improve employment outcomes across the life span, such as methods, costs, and results of services by rehabilitation programs or supported employment, including studies of natural supports at work as they relate to employment outcomes. • he role of personal assistance services in the T workplace and the application of rehabilitation technology, universal design principals, environmental adaptations, and engineering solutions to enhance personal function and address barriers confronted in employment by people with disabilities. continued

OCR for page 17
22 REVIEW OF DISABILITY AND REHABILITATION RESEARCH TABLE 1-1 Continued Number of Research Awards as of Domain January 2009 Research Topics Participation 39 (17%) • evelopment and evaluation of strategies for D and Community services, interventions, products, and modifications Living to the built and social environments that would allow individuals with all types of disabilities to live and participate in their communities. • evelopment of new and improved theories about D and measures of participation and community living that will enable the impact of specific strategies and interventions to be determined more accurately. Health and 93 (39%) Individual level: Function • ngoing research and clinical efforts to produce O a wide variety of programs, interventions, and products aimed at enhancing the health and function of individuals with disabilities. • tudy of new technologies that can improve the S diagnosis and measurement of disabling conditions and study of devices to support enhanced function. • esearch that can help prevent secondary conditions R from developing among people with disabilities and explore the implications of nonmedical interventions, such as exercise, in this context. Systems level: • tudy of system-level policies and practices S that exacerbate or ameliorate disability-related disparities in access to health care services. • ccurate assessment of the health status A of individuals with disabilities to increase understanding of the impact of the health care delivery system on their health and wellness. • tudies to classify specific interventions in medical S rehabilitation so as to better define and measure the effectiveness of the multitude of rehabilitation interventions.

OCR for page 17
23 INTRODUCTION TABLE 1-1 Continued Number of Research Awards as of Domain January 2009 Research Topics Technology 81 (34%) Individual level: for Access and Function • ocuses primarily on assistive technology devices F that enhance the physical, sensory, and cognitive abilities of people with disabilities and assist them in participating and functioning more independently in the home, at work, in recreational settings, and at cultural and community events. Systems level: • ocuses on applying technology research and F development in ways that enhance community integration, independence, productivity, competitiveness, and equal opportunity by mitigating or eliminating barriers found in large social systems, such as public transportation, telecommunications, information technology, and the built environment. • xploration of how recent, breakthrough advances E in biomaterials research, composite technologies, information and telecommunication technologies, nanotechnologies, micro electro-mechanical systems (MEMS), sensor technologies, and the neurosciences can be incorporated into future rehabilitation-related technology research. Disability 8 (3%) • tudies to generate new information that can S Demographics be used by individuals with disabilities, service providers, policy makers, and others working to identify and eliminate disparities in employment, participation and community living, and health and function. • tudies that mine data to address the full range of S social, health, and economic facets of disability and that compare findings across data sources. • esearch to understand the variances in levels of R participation by individuals with disabilities and to evaluate strategies or interventions that may help bridge the gap between preference and feasibility in an existing environment. • stablishment of a center on disability E demographics and statistics. Total Number 237 of Awards as of January 2009 SOURCE: National Institute on Disability and Rehabilitation Research (2009a, pp. 3-6).

OCR for page 17
24 REVIEW OF DISABILITY AND REHABILITATION RESEARCH one domain, the second column of the table notes the number of grants in NIDRR’s portfolio as of January 2009 whose primary research domain was in each of these five areas. NIDRR Grant Mechanisms NIDRR has eight primary mechanisms for awarding grants defined by the Catalog of Federal Domestic Assistance (CFDA) (National Institute on Disability and Rehabilitation Research, 2009a, pp. 7-29). Using these primary mechanisms, funds are distributed through 14 separate program mechanisms, which are described below. 1. Disability and Rehabilitation Research Project (DRRP; CFDA #84.133A). DRRPs are awarded through six separate program mechanisms: • Disability and Business Technical Assistance Center (DBTAC), which comprises a network of 10 regional centers and one Coor- dination Outreach and Research Center that provide information and referral, technical assistance, public awareness, and training on all aspects of the Americans with Disabilities Act (ADA).3 • Traumatic Brain Injury Model System (TBIMS), whose aim is to demonstrate the benefits of a coordinated system of neurotrauma and rehabilitation care and to conduct innovative research on all aspects of care for those who sustain traumatic brain injuries. The mission of the TBIMS program is to improve the lives of persons who experience traumatic brain injury by creating and disseminat- ing new knowledge about the course, treatment, and outcomes relating to their condition. • Disability and Rehabilitation Research Project-General (DRRP), which funds projects with special emphasis on research, demonstra- tions, training, dissemination, utilization, and technical assistance. Projects may include combinations of these activities. These projects may develop methods, procedures, and rehabilitation technology to assist in achieving the full inclusion and integration into society, employment, independent living, family support, and economic and social self-sufficiency of individuals with disabilities, especially individuals with the most severe disabilities, or to improve the ef- fectiveness of services authorized under the Rehabilitation Act. • Knowledge Translation (KT), whose projects range from inves- tigating models, methods, strategies, and mechanisms that could 3 Although still funded as DBTACs, these centers are currently referred to as ADA National Network Centers.

OCR for page 17
25 INTRODUCTION contribute to the successful knowledge translation of disability and rehabilitation research and products to synthesizing, disseminat- ing, and promoting the use of existing knowledge and products to improve the lives of individuals with disabilities. • Burn Model System (BMS), which comprises centers that establish innovative projects for the delivery, demonstration, and evaluation of comprehensive medical, vocational, and other rehabilitation services to meet the wide range of needs of individuals with burns. • Section 21, which focuses on research capacity building for minor- ity entities such as historically black colleges and universities and institutions serving primarily Hispanic, Asian, and American Indian students and nonminority entities with an interest in improving understanding about the needs and outcomes of individuals with disabilities from minority populations. Program activities include assisting minority entities with networking that supports enhanced collaboration between minority and nonminority entities and the exchange of expertise and advanced training across program areas. 2. Rehabilitation Research and Training Center (RRTC; CFDA #84.133B). RRTCs conduct coordinated and integrated advanced research aimed at alleviating or stabilizing disabling conditions, promoting maxi - mum social and economic independence of people with disabilities, or improving rehabilitation methodology or service delivery systems. RRTCs operate in collaboration with institutions of higher education and providers of rehabilitation services and serve as national centers of excellence in rehabilitation research. 3. Rehabilitation Engineering Research Center (RERC; CFDA #84.133E). RERCs conduct programs of advanced engineering and technical re- conduct re- search designed to apply technology, scientific achievement, and psy- chological and social knowledge to solve rehabilitation problems and remove environmental barriers. RERCs are affiliated with institutions of higher education or nonprofit organizations. 4. Switzer Fellowship (CFDA #84.133F). The Switzer program gives individual researchers an opportunity to develop new ideas and gain research experience. Fellows design and work for 1 year on an indepen- dent research project. 5. Field Initiated Project (FIP; CFDA #84.133G). The FIP program pro- vides funding to individual researchers to address rehabilitation issues in promising and innovative ways. FIPs are of two types—Field Initiated Projects-Research (FIR) and Field Initiated Projects-Development (FID). 6. Spinal Cord Injury Model System (SCIMS) (CFDA #84.133N). SCIMS centers study the course of recovery and outcomes following the delivery of a coordinated system of care for individuals with spinal cord injuries.

OCR for page 17
26 REVIEW OF DISABILITY AND REHABILITATION RESEARCH The centers under this program provide comprehensive rehabilitation services to individuals with such injuries and conduct spinal cord re- search, including clinical research. 7. Advanced Rehabilitation Research Training (ARRT; CFDA #84.133P). The ARRT program provides funding to institutions of higher education to recruit qualified postdoctoral candidates with clinical, management, basic, or engineering research experience and prepare them to conduct independent research on disability and rehabilitation issues. 8. Small Business Innovation Research (SBIR; CFDA #84.133S). SBIR grants, administered by NIDRR as a part of the larger mandatory SBIR program, help support the production of new assistive and rehabilita- tion technology. This two-phase program takes a rehabilitation-related product from development to market readiness. Phase I grants are de- signed to demonstrate proof of concept, while Phase II grants develop products envisioned in Phase I grants. In addition to these program mechanisms, contracts are awarded to provide technical support for NIDRR’s internal management, research, and knowledge translation activities. Interagency Collaboration Efforts have been made to develop and coordinate a coherent program of disability and rehabilitation research across the many federal agencies involved in such research. Yet coordination is difficult to accomplish because of differences in agency missions and organizational cultures (e.g., medical model versus social or environmental model), competitive budget processes, weak to nonexistent incentives for coordination and collaboration, and separately constructed long-range strategic plans within each agency (Insti- tute of Medicine, 2007). The Rehabilitation Act of 1973 authorized the Interagency Committee on Disability Research (ICDR), a federal interagency committee chaired by the NIDRR Director. The ICDR is mandated to promote coordination and cooperation among the many federal departments and agencies conducting disability and rehabilitation research programs. The committee comprises presidential designees, including the following (or their designees): the Director of NIDRR, the Commissioner of the Rehabilitation Services Administration, the Assistant Secretary for the Office of Special Educa- tion and Rehabilitative Services, the Secretary of Education, the Secretary of Veterans Affairs, the Director of the National Institutes of Health, the Director of the National Institute of Mental Health, the Administrator of the National Aeronautics and Space Administration, the Secretary of Transportation, the Assistant Secretary of the Interior for Indian Affairs,

OCR for page 17
27 INTRODUCTION the Director of the Indian Health Service, and the Director of the National Science Foundation. The role of ICDR is to identify and provide oppor- tunities for information sharing and partnerships on various initiatives in which the agencies are engaged. The ICDR works to cross-fertilize ideas and promote dialogue, but does not prescribe activities to its members.4 Its activities are organized and undertaken through the structure of five subcommittees addressing medical rehabilitation, employment, technol- ogy, education, and disability statistics. NIDRR Budget The Rehabilitation Act states that NIDRR’s Director will lay out fund- ing priorities and covered activities in a 5-year plan that will be published in the Federal Register and that the plan will dedicate at least 90 percent of NIDRR funds to extramural research. Additionally, Section 21 of the Rehabilitation Act mandates that 1 percent of NIDRR appropriations be set aside to address traditionally underserved populations. Finally, the SBIR Development Act of 2000 requires agencies with research and development budgets in excess of $100 million, such as NIDRR, to set aside a certain percentage of their funds for SBIR. The current set-aside is 2.5 percent of the research and development budget. NIDRR’s 2009 budget for grants and contracts was $109 million. Of this total, $99,904,000 (U.S. Department of Education, 2009) was dedi- cated to grants, with the remainder going to contracts and project support. Figure 1-1 illustrates NIDRR’s distribution of its funds for grants across all of the program mechanisms in 2009 (National Institute on Disability and Rehabilitation Research, 2009a). NIDRR’s annual budget for research and development grants has re- mained fairly constant in nominal terms during the last decade. As stated above, the budget for grants in 2009 was nearly $100 million; in 2008 and 2007 it was approximately $97 million (U.S. Department of Educa- tion, 2008, 2009). Previously, a report of the Institute of Medicine (2007) showed that the agency’s overall funding had been fairly steady since 2002. Although inflation has been reasonably low in recent years, steady funding in nominal terms implies a reduction in the research that can be conducted. NIDRR Staff NIDRR’s most recent organizational chart (National Institute on Dis- ability and Rehabilitation Research, 2011) shows an acting Director,5 an 4 S. Swenson and C. Pledger, personal communication, April 2011. 5 At the time of this writing, a permanent NIDRR Director had been recently hired.

OCR for page 17
28 REVIEW OF DISABILITY AND REHABILITATION RESEARCH ARRT- 2% BMS- 2% Switzer- 1% Section 21- 1% TBIMS- 10% DBTAC- 12% SCIMS- 9% DRRP- 7% SBIR- 4% FIP- 13% RRTC- 18% RERC- 17% KT- 4% FIGURE 1-1 Distribution of NIDRR funds across program mechanisms, fiscal year 2009. NOTE: ARRT = Advanced Rehabilitation Research Training, BMS = Burn Model System, DBTAC = Disability and Business Technical Assistance Center, DRRP = Disability and Rehabilitation Research Project-General, FIP = Field Initiated Project, KT = Knowledge Translation, RERC = Rehabilitation Engineering Research Center, RRTC = Rehabilitation Research and Training Center, SBIR = Small Business Innovation Research, SCIMS = Spinal Cord Injury Model System, TBIMS = Traumatic Brain Figure 1-1, editable Injury Model System. SOURCE: National Institute on Disability and Rehabilitation Research (2009a, p. 32). acting Deputy Director, and two main divisions—Research Sciences and Program, Budget, and Evaluation. Both divisions have directors and su- pervisors. In the Research Sciences Division, 14 positions are classified as rehabilitation specialists; these are the project officers who interact directly with grantees and perform grant monitoring functions. The organizational chart shows four of these positions to be vacant. The Program, Budget, and Evaluation Division has 10 other staff positions.

OCR for page 17
29 INTRODUCTION HISTORY OF NIDRR ASSESSMENTS Assessments of NIDRR have included a 2005 performance assessment under the GPRA and Annual Performance Assessment Expert Reviews conducted in 2005 and 2006. NIDRR Performance Assessment Under GPRA In compliance with the GPRA (U.S. Office of Management and Budget, 2009b), NIDRR developed the following performance measures to assess its key outcomes (National Institute on Disability and Rehabilitation Research, 2009a, pp. 78-79): • The percentage of NIDRR-supported fellows, post-doctoral trainees, and doctoral students who publish results of NIDRR-sponsored research in refereed journals • The number of accomplishments (new or improved tools, methods, discov- eries, standards, interventions, programs, or devices) developed or tested with NIDRR funding that has been judged by expert panels to be of high quality and to advance the field • The percentage of NIDRR-funded grant applications that receive an aver- age peer review score of 85 or higher • The percentage of NIDRR grants that assess the effectiveness of interven- tions, programs, and devices using rigorous methods • The number of new or improved NIDRR-funded assistive and universally designed technologies, products, and devices, transferred to industry for potential commercialization • The average number of publications per award based on NIDRR-funded R&D activities in refereed journals • The percentage of NIDRR competitions announced by October 1 • The percentage of NIDRR grant awards issued within 6 months of the competition closing date • The percentage of NIDRR-funded research projects identified as having an employment focus (This is a measure required by the Department of Education, Office of Special Education and Rehabilitative Services, not a GPRA measure) Under the GPRA requirements at the time,6 NIDRR’s 2005 Program Assessment Rating Tool (PART) review showed excellent scores on Pro- 6 PART was developed and used by the Office of Management and Budget to assess the per- formance of federal programs and to identify actions that could improve results (http://www. whitehouse.gov/omb/performance_past [October 24, 2011]). Expectations for performance measurement are currently being amended by the GPRA Modernization Act of 2010 (http:// www.whitehouse.gov/omb/performance/gprm-act [October 24, 2011]).

OCR for page 17
30 REVIEW OF DISABILITY AND REHABILITATION RESEARCH gram Purpose and Design (100 percent), Strategic Planning (90 percent), and Program Management (90 percent). However, Program Results and Accountability was rated ineffective (42 percent).7 NIDRR’s Annual Performance Assessment Expert Reviews NIDRR’s Annual Performance Assessment Expert Review process was implemented in 2005 and 2006 to evaluate the agency’s portfolio of grants in three areas: (1) health and function, (2) employment, and (3) technology. The objectives of the review were to provide an independent assessment of the quality and relevance of NIDRR-funded research and the extent to which the research outputs and outcomes were contributing to the agency’s long-term performance goals and measures; the quality and relevance of the agency’s management of research directions and award decisions; and the strengths and weaknesses of the three research portfolios, including recom- mendations to ensure the accomplishment of NIDRR’s goals and objec- tives (National Institute on Disability and Rehabilitation Research, 2006a, 2006b, 2007). To conduct the review, NIDRR assembled three panels of expert reviewers, one to review each portfolio. As with the present evaluation, grantee outputs were reviewed as part of the portfolio evaluation. Grantees submitted up to five accomplishments, defined as “outputs” or “outcomes,” that they considered to be the most important that occurred or matured during the current reporting period. “Outputs” included publications or presentations of significant findings; products, including tools, devices, and written products; and services com- pleted as part of training or capacity building. “Outcomes” could be either short term (referring to documented changes or advances in knowledge, un- derstanding, or skills) or intermediate (referring to documented changes in policy, practice, behavior, or systems capacity). The accomplishments were scored on a scale of 1 to 3, with 1 being “little-to-no contribution to the field,” 2 being a “substantial contribution,” and “3” being an “outstanding contribution.” A rating of 0 was used for “unable to determine.” Key findings from the three reports include the following: Health and function (October 2006) (National Institute on Disabil- • ity and Rehabilitation Research, 2007) (included multiple program mechanisms): The panel commended NIDRR on its long-standing productive portfolio in health and function research, and recog- nized the agency’s success in building infrastructure and capacity 7 ExpectMore.gov: National Institute on Disability and Rehabilitation Research, page 8 of 15 (http://www.whitehouse.gov/omb/expectmore/detail/10001041.2005.html [April 30, 2010]).

OCR for page 17
31 INTRODUCTION for disability and rehabilitation research. The panel noted several achievements of grantees, but felt limited in making judgments based on information covering only 1 year or less of multiyear awards. The panel rated 67 percent of the grantee accomplishments in this area as substantial or outstanding. Employment (September 2005) (National Institute on Disability • and Rehabilitation Research, 2006a) (included the RRTC, DRRP, and FIP program mechanisms): Overall, the panel concluded that the portfolio contained very high-quality research ideas. However, the panel expressed four concerns: “(1) the apparent lack of sci- entific rigor behind the identified outputs, (2) the lack of sufficient information on the methodologies used by grantees, (3) the lack of evidence supporting many of the claims made by grantees in their Supplemental Information Reports, and (4) the lack of peer- reviewed publications” (p. 4). The panel was unable to rate the grantee accomplishments given the limited information it received. Technology (October 2005) (National Institute on Disability and • Rehabilitation Research, 2006b) (included the RERC, DRRP, and FIP program mechanisms): The panel identified significant accom- plishments in each cluster area. However, given NIDRR’s level of investment, the panel stated that the overall number of high-quality research and development outputs and outcomes that advanced scientific knowledge was inadequate. The panel also noted that many of the claims made in the reports it reviewed lacked sufficient supporting evidence. The panel rated 54 percent of the grantee ac- complishments as substantial or outstanding. Methods used in the summative component of the present evaluation were somewhat similar to those used in the Annual Portfolio Assessment Expert Reviews with regard to the assessment of accomplishments, but can be distinguished from the latter in several ways. First, this committee reviewed only the category of “outputs.” Second, the committee focused on the quality of outputs. Therefore, instead of assigning one rating for the ex- tent to which the outputs contributed to the field as was done in the Annual Portfolio Assessment Expert Reviews, the committee examined and rated each output in a multidimensional way using four criteria: technical qual- ity, advancement of knowledge or the field, likely or demonstrated impact, and dissemination. Third, the committee itself served as the expert panel of reviewers. Finally, the committee assessed the methods used in its summative evaluation and made recommendations for improving future evaluations of outputs from NIDRR research. A full description of the methods used to assess outputs is presented in Chapter 6.

OCR for page 17
32 REVIEW OF DISABILITY AND REHABILITATION RESEARCH ORGANIZATION OF THE REPORT Chapter 2 describes the scope of the committee’s evaluation and the meth- ods used. Chapters 3, 4, and 5, respectively, present results of the evaluation of NIDRR’s priority-setting, peer review, and grant management processes. The chapters summarize NIDRR’s policies and procedures with regard to these key processes and present the committee’s findings based on information gathered from such sources as interviews with NIDRR staff; questionnaires completed by grantees; and surveys of NIDRR stakeholder organizations, other federal agencies, and NIDRR peer reviewers. Chapter 6 summarizes the findings of the evaluation of grantee outputs, based on a sample of 30 grants, and presents the assessment of the committee’s summative evaluation process. Conclusions and recommendations for improving NIDRR’s key processes and the quality of grantee outputs and for conducting future evaluations are included at the end of Chapters 3 through 6. The report’s appendixes provide background information. Appendix A contains summaries of the 30 grants whose outputs were evaluated by the committee; these summaries include descriptions of the outputs reviewed. Appendix B contains the questionnaires and rating sheets used in the com- mittee’s process and summative evaluations. Appendix C lists the acronyms used in this report. Finally, Appendix D contains biographical sketches of the committee members and NRC staff who participated in the study. REFERENCES Institute of Medicine. (2007). The future of disability in America. Washington, DC: The Na- tional Academies Press. National Institute on Disability and Rehabilitation Research. (2006a). Annual portfolio as- sessment expert review pilot employment portfolio panel summary report. Available: http://www.naric.com/nidrr/APAER_Employ_Final-Revised1-4-06.pdf [March 21, 2011]. National Institute on Disability and Rehabilitation Research. (2006b). Annual portfolio assessment expert review pilot technology portfolio panel summary report. Available: http://www.naric.com/nidrr/APAERTechnologyReport-FinalRevised-1-4-06.pdf [March 21, 2011]. National Institute on Disability and Rehabilitation Research. (2006c). Department of Edu- cation: National Institute on Disability and Rehabilitation Research—Notice of Final Long-Range Plan for Fiscal Years 2005–2009. Federal Register, 71(31), 8,166-8,200. National Institute on Disability and Rehabilitation Research. (2007). Annual performance assessment expert review health and function research. Available: http://www.ncddr. org/new/announcements/apaer/hf/HF-APAER2006RptPub7-5-07.pdf [March 21, 2011]. National Institute on Disability and Rehabilitation Research. (2009a). Briefing book for The National Academies. Unpublished document. Washington, DC: National Institute on Disability and Rehabilitation Research. National Institute on Disability and Rehabilitation Research. (2009b). Department of Educa- tion: National Institute on Disability and Rehabilitation Research—Notice of Proposed Long-Range Plan for Fiscal Years 2010–2014. Federal Register, 74(10), 2,564-2,569.

OCR for page 17
33 INTRODUCTION National Institute on Disability and Rehabilitation Research. (2011). NIDRR organizational chart. Unpublished document. Washington, DC: National Institute on Disability and Rehabilitation Research. The Rehabilitation Act of 1973, as amended. Pub. L. No. 93-112. Available: http://www2. ed.gov/policy/speced/reg/narrative.html [January 21, 2011]. U.S. Department of Education, Office of Special Education, Rehabilitative Services, Reha- bilitation Services Administration. (2008). Annual report on federal activities under the Rehabilitation Act, Fiscal Year 2008. Washington, DC: U.S. Department of Education. U.S. Department of Education, Office of Special Education, Rehabilitative Services, Reha- bilitation Services Administration. (2009). Annual report on federal activities under the Rehabilitation Act, Fiscal Year 2009. Washington, DC: U.S. Department of Education. U.S. Office of Management and Budget. (2009a). Government Performance and Result Act. Available: http://www.whitehouse.gov/omb/mgmt-gpra/gplaw2m.html [March 20, 2009]. U.S. Office of Management and Budget. (2009b). Assessing program performance. Available: http://whitehouse.gov/omb/part/ [March 20, 2009]. World Health Organization. (2001). ICF: International Classification of Functioning, Disabil- ity and Health. Geneva, Switzerland: World Health Organization.