External Evaluation of the National Institute on Disability and Rehabilitation Research and Its Grantees Letter Report (2011) / Chapter Skim
Currently Skimming:
Letter Report
Letter Report
Pages 1-20
The Chapter Skim interface presents what we've algorithmically identified as the most significant single chunk of text within every page in the chapter.
Select key terms on the right to highlight them within pages of the chapter.
From page 1... ...
BACKGROUND ON NIDRR AND COMMITTEE CHARGE The National Institute on Disability and Rehabilitation Research is the principal federal agency that funds applied research and development to improve the lives and functioning of persons with disabilities (Office of Special Education and Rehabilitative Services, 2007)
|
From page 2... ...
The second component, termed the "summative evaluation," involved the assessment of grantee outputs. The key question of the summative evaluation was articulated by NIDRR as follows: To what extent are the final outputs from NIDRR grants of high quality?
|
From page 3... ...
Quality Criteria Development A key element of the summative evaluation was to respond to NIDRR's request to develop criteria for assessing the quality of its grantees' outputs.1 In developing the criteria, the committee drew on its own research expertise, recommendations of the external advisory group convened by NIDRR while planning this NRC evaluation (National Institute on Disability and Rehabilitation Research, 2008) , and methods used in other NRC and international studies that have evaluated federal research programs (see, e.g., Bernstein et al., 2007; Canadian Academy of Health Sciences, 2009; Chien, Chen, and Chen, 2009; Ismail, Tiessen, and Wooding, 2010; National Research Council, 2007; Wooding and Starkey, 2010; Wooding et al., 2009)
|
From page 4... ...
Grantee Questionnaire NIDRR supplied the committee with information gathered from grantees in their Annual Performance Reports (APRs) and final reports (Research Triangle International, 2009)
|
From page 5... ...
For supplemental information about dissemination, grantees were asked to describe the stage and scope (e.g., local, regional, national) of dissemination efforts, specific dissemination activities, any identification and tailoring of materials for particular audiences, efforts to collaborate with particular audiences or user communities to identify content and medium needs and preferences, and the delivery of information through multiple media types.
|
From page 6... ...
The committee used the smaller subset of all NIDRR grants as the sampling pool because of its charge and preliminary analysis of the data. The committee was directed by its charge to draw a sample of 30 grants ending in 2009 that reflected the range of work conducted across NIDRR's 14 program mechanisms.
|
From page 7... ...
Center Grants Rehabilitation Engineering and 12 4.84 5.7 55,816,980 17.98 0 0 0 8 4 Research Centers (RERC) Rehabilitation Research and 21 8.47 5.9 82,920,345 26.71 0 0 0 10 11 Training Centers (RRTC)
|
From page 8... ...
To put the outputs reviewed into the larger context of the outputs produced by grantees in the sampling pool of 111 grants, Table 2 also shows that the proportion of publications and other outputs (tools, technology, and information products) that were reviewed by the committee were relatively close to the proportions of the various output types produced by grantees in the larger sampling pool.
|
From page 9... ...
(MS-SCI) Center Grants Rehabilitation Research and 3 16 0 0 12 28 (19%)
|
From page 10... ...
Based on direct review of the output itself and supplemental information about the output provided in the APRs, final reports, and questionnaire responses from grantees, each subgroup member independently rated every output assigned to that subgroup, assigning a quality criteria score for each of the four quality criteria (technical quality, advancement of knowledge or the field, likely or actual impact, and dissemination) , as well as an overall score for the output and a rationale for the overall score.
|
From page 11... ...
In addition, conference calls with the full committee were held after each set of subgroup meetings to discuss the evaluation process and refine the methods. Lastly, during its final meeting, the committee devoted a half-day session to discussion of the strengths and weaknesses of the process and developing conclusions and recommendations for future evaluations.
|
From page 12... ...
Even though five program mechanisms were not included in the sampling pool, the number of grants 12
|
From page 13... ...
The committee was also asked to produce an overall grant rating based on the outputs reviewed and the information available about the grants from the APRs. Results at the grant level are subject to more limitations than those regarding outputs due to the general lack of information about how the outputs did or did not interrelate; whether, and if so, how grant objectives were accomplished; and the relative priority placed on the various outputs.
|
From page 14... ...
Diversity of Outputs The quality rating system used in the summative evaluation worked very well for publications in particular, which comprised 70 percent of the outputs reviewed. Using the four criteria developed by the committee, the reviewers were able to identify varying levels of quality and the characteristics associated with each of them.
|
From page 15... ...
Outputs other than publications, such as technology products, could be assessed in an interim evaluation. Sources of Information Committee members were provided with structured briefing books containing the outputs to be reviewed and supplemental information that members could draw on if additional information was needed to assign quality scores.
|
From page 16... ...
Diversity of Outputs The dimensions of the quality criteria should be tailored and appropriately operationalized for different types of outputs, such as devices, tools, and information products (including newsletters, conferences, and websites) and should be field tested with grants under multiple program mechanisms and refined as needed.
|
From page 17... ...
includes the grant abstract, funding information, descriptions of the research and development projects, and outcome domains targeted by projects, as well as a range of variables for reporting on the four different types of grantee outputs; see Table 3. The system is tailored to different program mechanisms as needed.
|
From page 18... ...
Whether output is described in a publication output and X X X indicate which one Key findings or lessons learned X How output is contributing to the outcome-oriented goal by X X X X solving a problem, closing an identified gap, or benefiting the target population a SOURCE: Using NIDRR APR report format for Rehabilitation Research and Training Centers as an example b Defined for grantees by NIDRR as "those that contributed most to achieving the outcome-oriented goals for the award by advancing knowledge, increasing capacity for research, training or knowledge translation; or facilitating changes in policy, practice, or system capacity." NIDRR also provided grantees' narrative APRs from the last year of the grants, as well as their final reports. These narratives were very useful to the committee for compiling descriptions of the grants.
|
From page 19... ...
Recommendation 3: NIDRR should consider revising its APR to better capture information needed to routinely evaluate the quality and impacts of outputs, grants, and funding mechanisms. They might consider such efforts as consolidating existing data elements or adding new elements to capture the quality criteria and dimensions used in the committee's summative evaluation.
|
From page 20... ...
If future evaluations of output quality are conducted, the process developed by the committee should be implemented with refinements to strengthen the design and process. Although assessing grantee outputs is of great value, the committee thinks that even greater value would come from assessing outputs in the context of a more comprehensive grant-level evaluation, which could yield broader implications for the value of grants, their impact, and future directions for NIDRR.
|
Key Terms
This material may be derived from roughly machine-read images, and so is provided only to facilitate research.
More
information on Chapter Skim is available.