National Academies Press: OpenBook
« Previous: Front Matter
Suggested Citation:"Summary." National Research Council. 2012. Review of Disability and Rehabilitation Research: NIDRR Grantmaking Processes and Products. Washington, DC: The National Academies Press. doi: 10.17226/13285.
×

Summary

The National Institute on Disability and Rehabilitation Research (NIDRR) is one of the principal federal agencies supporting applied research, training, and development to improve the lives of individuals with disabilities. NIDRR’s mission is to generate new knowledge and promote its effective use in improving the ability of persons with disabilities to perform activities of their choice in the community, as well as to expand society’s capacity to provide full opportunities and accommodations for its citizens with disabilities. Located within the Office of Special Education and Rehabilitative Services in the U.S. Department of Education (ED), NIDRR has an annual budget for grants and contracts of approximately $109 million, awarded through 14 separate program mechanisms that result in 1- to 5-year awards ranging in size from less than $100,000 to several million dollars. NIDRR has the largest budget of the three primary federal agencies with disability and rehabilitation as part of their mandate but also has the broadest mandate. NIDRR aims to reach all disability types and age groups, and its mission is tied to long-term outcomes such as independence, community participation, and employment.

Assessing the outcomes of research is a complex undertaking that can variably take into account the stated goals of the research, the contribution to the relevant field of research, the impact on the well-being of a particular population, or other related issues. The Government Performance and Results Act has led to a particular emphasis on establishing specific performance measures assessing the outputs of research programs.

NIDRR takes pride in proactively establishing program performance measures focused on the quality of its grantee outputs, developing accountability

Suggested Citation:"Summary." National Research Council. 2012. Review of Disability and Rehabilitation Research: NIDRR Grantmaking Processes and Products. Washington, DC: The National Academies Press. doi: 10.17226/13285.
×

data systems to track the results of those measures, and developing internal management systems to facilitate quality research. In 2009, NIDRR requested that the National Research Council form a committee to conduct a “process evaluation” of aspects of its grantmaking and a “summative evaluation” of the quality of grantee outputs. In addition, the committee was charged with assessing the methods it developed for conducting the summative evaluation and making recommendations for the conduct of future evaluations.1 The requested study was the most recent effort in a series of NIDRR-funded activities aimed at assessing and improving the agency’s performance.

PROCESS EVALUATION

NIDRR posed three questions specific to the process evaluation aimed at assessing the process used for priority writing, practices for peer review of grant applications, and the planning and budgetary processes used by grantees. The development of priorities determines the areas of emphasis for research and the specific topics to be targeted by potential applicants, while peer review is a fundamental component of the grant selection process. Although it is not possible to establish a clear causal link, these NIDRR processes, as well as planning and budgetary processes used by grantees, can influence the quality of the work produced by grantees.

To address these questions, the committee reviewed existing documents (e.g., legislation, Federal Register notices, NIDRR and ED policies and procedures) and interviewed NIDRR management to obtain a more thorough and cohesive understanding of these processes. The committee gained additional insight into NIDRR’s peer review process by listening to teleconferences held by three panels as they conducted their reviews of different grant competitions. In addition, the committee collected original data through surveys of NIDRR staff, stakeholder organizations (other federal agencies, professional associations, and advocacy organizations), NIDRR peer reviewers, and principal investigators of NIDRR grants.

Priority Setting

To what extent is NIDRR’s priority-writing process conducted in such a way as to enhance the quality of the final results?

As used in the study question, the term “priority-writing process” encompasses many aspects of priority setting, including gathering input from

_______________

1 This aspect of the committee’s charge was summarized in a letter report provided to NIDRR in July 2011 and is also addressed in Chapter 6 of this report.

Suggested Citation:"Summary." National Research Council. 2012. Review of Disability and Rehabilitation Research: NIDRR Grantmaking Processes and Products. Washington, DC: The National Academies Press. doi: 10.17226/13285.
×

multiple sources (e.g., the field, stakeholder organizations, grantees, other agencies, and persons with disabilities and their families), identifying potential topics and determining priorities for funding, writing the proposed priorities and having them cleared for release, and publishing notices inviting applications (NIAs) on these priorities. The term “priority setting” is used synonymously with “priority writing” in this report to clarify that the focus of the committee’s evaluation included this larger priority-setting process.

The committee concluded that NIDRR’s long-range planning and priority-setting processes are successful in producing grants that are aligned with its mission and that stakeholders value as unique. Nonetheless, areas for improvement were identified. First, the committee concluded that NIDRR needs to do more to inform and engage stakeholders with respect to its long-range planning and priority-setting processes. Although the results of the stakeholder survey were generally positive, the transparency of the processes, responsiveness to stakeholder feedback, and use of NIDRR grantee products could be improved. Second, priority setting not only reflects the agency’s intent to influence the advancement of research in targeted areas but also offers specific funding opportunities for potential grant applications to the agency. The announced priorities should therefore be developed and communicated in a manner that attracts the best researchers to participate in disability and rehabilitation research. Attracting the largest pool of applications from which to select grantees increases the chances for the highest quality outputs. When establishing its priorities, the agency needs to consider continuity from one funding cycle to another, as well as identify future research challenges and societal needs. The committee offers recommendations in four areas to strengthen NIDRR’s long-range planning and priority-setting processes.

Formation of an Advisory Council

NIDRR has a broad and diverse mission that makes it challenging to set priorities that are responsive to the current state of the science and the needs of the stakeholder community. Currently, NIDRR relies on staff, the portfolio of existing projects, recent findings from completed grants, and the current research literature, as well as guidance from federal partners, for input to the priority-setting process. NIDRR’s statute directs it to establish a standing Rehabilitation Research Advisory Council to advise the director of the agency on research priorities and the development of the agency’s Long-Range Plans. While NIDRR has formed ad hoc advisory bodies to support the development of its Long-Range Plans, a standing body has never been formed. Given NIDRR’s mandate, the council should be tasked with providing advice on both disability and rehabilitation research.

Suggested Citation:"Summary." National Research Council. 2012. Review of Disability and Rehabilitation Research: NIDRR Grantmaking Processes and Products. Washington, DC: The National Academies Press. doi: 10.17226/13285.
×

Recommendation 3-12: NIDRR should fulfill the statutory mandate to form and utilize a standing disability and rehabilitation research advisory council to advise on the priority-setting process and provide input for priority setting.

Most federal funding agencies, including the National Institutes of Health (NIH), the National Science Foundation (NSF), and the National Institute for Occupational Safety and Health (NIOSH), use standing advisory bodies. A standing advisory body is likely to add stability and continuity to both NIDRR’s long-range planning and its priority setting. The committee recognizes that NIDRR, like other federal research agencies, will face challenges in capturing the broad diversity of perspectives held by its many stakeholders. However, the committee feels strongly that, like other federal research agencies, NIDRR can meet these challenges.

Strategic Planning

NIDRR’s multiple stakeholders include persons with disabilities and their families, the scientific community, professional associations, and advocacy organizations representing a variety of disability groups. In the face of this diversity, it is important for the agency to have a consistent mechanism for gathering information and input to inform the strategic planning process beyond the input that will be possible through an advisory council. NIDRR utilizes input from multiple sources, such as its stakeholders, other federal agencies, the Interagency Committee on Disability Research (ICDR), the current literature, and state-of-the-science conferences. However, the processes for gathering input and developing proposed Long-Range Plans have varied from one plan to another. Negative comments from the field generated by the last draft Long-Range Plan, coupled with the plan’s subsequent delay, which caused NIDRR to operate under the prior plan for several years beyond its intended time frame, suggest a breakdown in NIDRR’s prioritysetting process. The lack of a permanent director also hampers and delays the agency’s priority-setting process.3

Recommendation 3-2: NIDRR should use a structured, consistent, and inclusive strategic planning process to develop its Long-Range Plans and priorities.

NIDRR might consider the long-range planning and priority setting processes of other funding agencies, including NIH, NSF, and NIOSH, which

_______________

2 The committee’s recommendations are numbered according to the chapter of the report in which they appear.

3 At the time of this writing, a permanent NIDRR Director had been recently hired.

Suggested Citation:"Summary." National Research Council. 2012. Review of Disability and Rehabilitation Research: NIDRR Grantmaking Processes and Products. Washington, DC: The National Academies Press. doi: 10.17226/13285.
×

have sought to integrate long-range planning and priority-setting processes through specific initiatives such as the NIH Roadmap, the NSF Strategic Plan, and the National Occupational Research Agenda.

Establishment of a Standard Calendar

For many program mechanisms, NIDRR has not established a regular schedule for drafting and approving priorities and NIAs and disseminating them to the field. ED has a lengthy review and approval process for obtaining clearance for the release of priorities and NIAs. The variability in the length of the clearance process may be an important factor, among others, that impacts the timing of the release of NIAs. The irregular or delayed release of NIAs may affect NIDRR’s ability to provide individuals sufficient notice of grant opportunities or an optimal amount of time to complete applications. An irregular schedule may discourage the best investigators from submitting applications. Additionally, certain program mechanisms (such as Model Systems) include collaboration between institutions. Irregular posting and shortened response times hamper the ability of applicants to identity and recruit appropriate collaborators. These factors are likely to limit the number of investigators who apply and adversely affect the quality of the applications they submit. Additionally, young investigators less familiar with NIDRR are more likely to pursue grants from other agencies.

Recommendation 3-3: NIDRR should utilize a standard calendar for the setting of priorities, publication of notices inviting applications, submission of applications, and peer review meetings to improve the efficiency of the process.

NIDRR has made efforts to standardize the schedule for NIAs. The committee suggests that program mechanisms competed on a yearly basis have a consistent annual schedule for the submission and review of applications. For multiyear grants, the committee recommends that NIDRR establish a long-range operational plan listing projected future grant application submission dates, pending funding availability in that fiscal year.

Soliciting Applications

Like other federal agencies, NIDRR makes its NIAs available at Grants. gov in addition to publishing them in the Federal Register. The agency also uses a contractor to notify former grantees and others who, via their webpage, express an interest in receiving NIAs. NIDRR would benefit from more active efforts to solicit interest in its funding announcements.

Suggested Citation:"Summary." National Research Council. 2012. Review of Disability and Rehabilitation Research: NIDRR Grantmaking Processes and Products. Washington, DC: The National Academies Press. doi: 10.17226/13285.
×

Recommendation 3-4: NIDRR should expand its efforts to disseminate notices inviting applications to new potential applicants, including developing a communication strategy to ensure that the notices reach new audiences.

To expand dissemination, notices should be sent to the disability and rehabilitation professional and research organizations that make up NIDRR’s stakeholder network and to university departments and offices of sponsored research. The latter could perhaps be accomplished through collaboration with other federal research programs that regularly send funding notices to universities.

Peer Review

To what extent are peer reviews of grant applications done in such a way as to enhance the quality of final results?

NIDRR’s peer review process encompasses recruiting and training reviewers, conducting the review, and approving the awards. As with priority setting, it is challenging to link the peer review process directly to specific results because the quality of the portfolio, grants, and outputs emerging from the process is the product of multiple complex factors. It is clear, however, that the peer review process used by NIDRR contributes significantly to the success of the grant award program and the quality of the outputs produced.

The responses to the committee’s peer review survey were largely positive, including peer reviewers’ responses related to their experiences with NIDRR’s peer review process and how it compares with the processes used by other federal research agencies. While the committee concluded that NIDRR’s peer review process is generally good, there are opportunities for improvement that would likely enhance the process and the quality of final results. The committee offers three recommendations to this end.

Enhancements to the Peer Review Process

The committee concluded that NIDRR’s peer review process is hampered by a limited pool of potential reviewers. NIDRR’s competition managers take great care to assemble and facilitate qualified review panels and spend considerable time recruiting and screening potential reviewers. Competition managers regularly must manage potential conflicts of interest and rule out qualified reviewers. Despite these staff efforts, however, the committee found evidence that a number of panels are smaller than NIDRR’s recommended size, reviewers are added so close to the meeting

Suggested Citation:"Summary." National Research Council. 2012. Review of Disability and Rehabilitation Research: NIDRR Grantmaking Processes and Products. Washington, DC: The National Academies Press. doi: 10.17226/13285.
×

date that they have inadequate time to prepare, and reviewers lacking necessary scientific expertise may be participating in reviews.

The formation of formal “cohorts” of reviewers with particular areas of expertise would reduce the recruiting burden on NIDRR staff in locating reviewers needed for individual competitions with specific targeted expertise. In addition, reviewers surveyed by the committee reported that the quality of the training they received was inconsistent; enhancing this training would be a simple and effective way to improve the quality of the review process. Finally, considerable variation exists among competitions in the way NIDRR staff facilitate panel discussions. The result is variation in the quality of the discussions; such inconsistency also can result in confusion and negatively influence overall quality.

Recommendation 4-1: NIDRR should further strengthen the peer review infrastructure by expanding the pool of high-quality reviewers; establishing standing panels, or formal cohorts of peer reviewers with specialized knowledge and expertise as appropriate for the program mechanisms; enhancing reviewer training; and improving the consistency of NIDRR staff facilitation of panel meetings and the quality of feedback provided to grantees.

Reducing Reviewer Burden

Participating in NIDRR’s peer review process is a significant burden for a large percentage of reviewers. Many reviewers spend more time than they would like preparing, and the review days are long and intense. This significant time commitment makes it less likely that qualified and experienced reviewers will participate. Reviewers surveyed also reported sometimes having insufficient time to review proposals, which could affect the quality of the review discussions. The committee concluded that the review process is so burdensome to peer reviewers as to threaten the quality of the process.

Recommendation 4-2: NIDRR should streamline the review process in order to reduce the burden on peer reviewers.

Use of Consumer Peer Reviewers

To address its mission, NIDRR makes concerted efforts to include both scientists with disabilities and consumers without scientific expertise in the peer review process. Consumers can represent the experiences and views of their particular disability communities and can evaluate applications for relevance to their community’s needs and concerns.

Suggested Citation:"Summary." National Research Council. 2012. Review of Disability and Rehabilitation Research: NIDRR Grantmaking Processes and Products. Washington, DC: The National Academies Press. doi: 10.17226/13285.
×

All reviewers, including researchers and consumers, should have the appropriate expertise to review those elements of proposals to which they are assigned. If consumers are to review scientific aspects of proposals, they should have the relevant expertise, or NIDRR should consider providing them with relevant methodological training. NIDRR should review and monitor the role of consumers and researchers in peer review to ensure that quality is not compromised.

Recommendation 4-3: NIDRR should continue to have consumer representation in the peer review process and establish procedures to guide the participation of those without scientific expertise.

Many federal research programs involve consumers without scientific expertise in peer review. NIDRR may want to examine such practices at NIH, Congressionally Directed Medical Research Programs, Juvenile Diabetes Research Foundation, and other agencies to inform its own approach to including nonresearchers in peer review.

Grant Management

What planning and budgetary processes does the grantee use to promote high-quality outputs?

The committee assessed grantee planning and budgetary processes in the larger context of NIDRR’s structure and processes supporting grant management. To perform this assessment, the committee (1) reviewed existing documentation on the grant management and monitoring processes of ED and NIDRR, (2) gathered information from principal investigators about the processes they use for managing grants, and (3) interviewed NIDRR staff to obtain their perspectives on how grant monitoring facilitates grantees’ efforts to manage their grants for successful results.

NIDRR appears to have developed a good plan for upgrading its routine monitoring of grants and for identifying and monitoring grants that are at risk of noncompliance with ED or NIDRR requirements and performance expectations. On the whole, grantees appear to appreciate aspects of NIDRR’s grant management processes that facilitate their own grant management strategies. While grantees generally commented that NIDRR’s grant management processes were effective in facilitating their own grant management processes, they offered some suggestions for improvement that would help them further. NIDRR staff also offered suggestions for improvement, focused on strengthening their capacity to monitor grants and help grantees stay on course in implementing their grants and meeting performance expectations. Among other suggestions, they expressed the need for smaller

Suggested Citation:"Summary." National Research Council. 2012. Review of Disability and Rehabilitation Research: NIDRR Grantmaking Processes and Products. Washington, DC: The National Academies Press. doi: 10.17226/13285.
×

grant caseloads, additional travel funds for on-site monitoring of grants that require higher levels of technical assistance, more training for new project officers to promote consistency and quality in the monitoring process, and a freer flow of communication between project officers and NIDRR planning staff with respect to financial information.

Recommendation 5-1: NIDRR should continue to focus efforts on improving its grant monitoring procedures and specific elements of its overall grant management system that impact grantee-level planning, budgets, and the quality of outputs.

From its interviews with grantees, the committee also learned that some grants focused on developing technology innovations may not accord well with a management template that calls for strict up-front planning and adherence to original designs and timetables. Similarly, a grant funding a large multisite study may require more or different supervision, monitoring, and technical assistance than a more focused or limited study. Grantees expressed the need for greater flexibility in grant management so they can stay on the cutting edge of technology or adapt more easily to changing needs of multisite research projects.

Recommendation 5-2: NIDRR should review the requirements placed upon technical innovation grants and large multisite studies to ensure that planning, reporting, supervisory, and technical assistance requirements fit their particular circumstances.

To what extent are the results of the reviewed research and development outputs used to inform new projects by both the grantee and NIDRR?

To assess how research and development outputs inform new projects, the committee (1) reviewed information from NIDRR management about how they use the results of their grantees’ research and (2) reviewed information from grantees about new projects that have been generated from their grants. The committee concluded that research and development outputs are used to generate new projects by grantees to a great extent and lead to substantial numbers of new collaborations with other researchers and organizations, as well as transfers of data, instruments, or models to other projects, and commercialization of technology products.

SUMMATIVE EVALUATION

The summative evaluation, designed to inform NIDRR’s performance monitoring and reporting, involved assessing the quality of outputs produced

Suggested Citation:"Summary." National Research Council. 2012. Review of Disability and Rehabilitation Research: NIDRR Grantmaking Processes and Products. Washington, DC: The National Academies Press. doi: 10.17226/13285.
×

by a sample of 30 NIDRR grantees.4 These grants were drawn from nine NIDRR program mechanisms: Burn Model System, Traumatic Brain Injury Model System, Spinal Cord Injury Model System, Rehabilitation Research and Training Center, Rehabilitation Engineering Research Center, Disability and Rehabilitation Research Project-General, Field Initiated Project, Small Business Innovation Research II, and Switzer Fellowship. The committee reviewed four different types of outputs, as defined by NIDRR: publications; tools, measures, and intervention protocols; technology products and devices; and informational products.

The committee developed and used four criteria to assess quality: (1) technical quality; (2) advancement of knowledge or the field; (3) likely or demonstrated impact (on science, persons with disabilities and their families, provider practice, health and social systems, social and health policy, and the private sector/commercialization); and (4) dissemination according to principles of appropriate knowledge translation.

A total of 148 outputs produced by the 30 grantees were rated on each criterion, using a 7-point scale, where 1 indicated poor quality, 4 indicated good quality, and 7 indicated excellent quality. Ratings on each of the four criteria were distributed fairly symmetrically along the scale, with the largest proportion of scores falling at the midpoint of 4 and with most being slightly skewed toward the higher end of the scale. Although close to 75 percent of the outputs rated fell in the “good to excellent” range of the quality scale (i.e., mean ratings of 4 or greater on the 7-point quality scale), 25 percent of the outputs fell in the lower quality range (1 or “poor” to 3 or “below good”) across all four criteria. The committee offers NIDRR two recommendations for assisting grantees in continuously improving the quality of their outputs.

First, the quality of outputs is the product of multiple complex factors that involve the priority-setting process, the funding level, the peer review process, the quality of the proposed science/research and the grantees, and ultimately the quality of the research findings. For grantees that are not performing optimally, NIDRR may conduct ongoing formative reviews with experts to identify strategies for improvement, increase its grant monitoring activities, and require additional grantee reporting. Grantees generally report that NIDRR’s oversight and reporting functions foster successful grants and high-quality outputs by assisting them in adhering to their budget and timeline, providing an external quality assurance mechanism for their project management, and prompting them to maintain their focus on project goals for high-quality products.

_______________

4 The committee performed a random sampling of grants at the level of program mechanism. Five mechanisms were excluded in consultation with NIDRR.

Suggested Citation:"Summary." National Research Council. 2012. Review of Disability and Rehabilitation Research: NIDRR Grantmaking Processes and Products. Washington, DC: The National Academies Press. doi: 10.17226/13285.
×

Recommendation 6-1: Although close to 75 percent of outputs were rated as “good to excellent” (i.e., 4 or higher on the 7-point quality scale), NIDRR should make it clear that it expects all grantees to produce the highest quality outputs.

The intent of this recommendation is for NIDRR to encourage all of its grantees to publish in peer-reviewed journals, present at national meetings, publish/disseminate materials, and bring technology solutions to market while producing these outputs at the highest levels of quality. To this end, NIDRR should push forward by establishing clear and consistent expectations for grantees to publish in higher-impact journals as one indicator of higher quality. For outputs other than publications, NIDRR should establish standards for quality to be achieved and adopt appropriate metrics to assess adherence to these standards. One way of setting the quality bar higher would be to encourage grantees to use standardized reporting forms and checklists5 for reviewing the technical quality of their own work before subjecting it to external review.

Second, despite limitations in the use of bibliometrics,6 they are a valuable and objective set of measures that can be used in combination with other assessment strategies. NIDRR has conducted bibliometric analyses in the past, but has not routinely incorporated use of these metrics into its performance measurement process.

Recommendation 6-2: NIDRR should consider undertaking bibliometric analyses of its grantees’ publications as a routine component of performance measurement.

Bibliometric analyses would take advantage of an existing data source for periodic measurement of the scientific impact of NIDRR grantee publications, and would provide an indicator of the extent to which these grant outputs are being disseminated and used. This type of metric is being recommended for use in combination with other measures, just as it was used in the committee’s evaluation along with expert review and supplemental evidence of the impact an article may have had on science, persons with disabilities and their families, provider practice, health and social systems, social and health policy, and the private sector/commercialization.

_______________

5 See http://www.equator-network.org/ for examples.

6 Common bibliometric measures include the impact factor of journals in which articles are published and the number of times an article is cited in other articles.

Suggested Citation:"Summary." National Research Council. 2012. Review of Disability and Rehabilitation Research: NIDRR Grantmaking Processes and Products. Washington, DC: The National Academies Press. doi: 10.17226/13285.
×

SELF-ASSESSMENT OF THE COMMITTEE’S METHODS FOR OUTPUT REVIEW AND RECOMMENDATIONS FOR FUTURE EVALUATIONS

The committee developed and implemented an evaluation process for assessing the outputs of NIDRR’s grantees and identifying the various levels of quality and characteristics of those outputs. The committee spent considerable time selecting and refining the criteria used to assess the quality of outputs. While some variation was evident in the independent scoring among the committee members, it was rarely extreme, particularly after group discussions. However, as summarized below, the committee encountered several challenges and obstacles during the course of its work that limited the generalization of its findings and restricted what could be said about the totality of outputs generated by all NIDRR grantees.

Defining Future Evaluation Objectives

The primary focus of the committee’s summative evaluation was on assessing the quality of outputs produced by grantees; the evaluation did not include in-depth examination or comparison of the larger contexts of the funding programs, grants, or projects within which the outputs were produced. However, the committee was asked to formulate an overall rating for each grant based on the outputs reviewed and the information available about the grant from the Annual Performance Report (APR). Results at the grant level were subject to limitations resulting from the general lack of information about how the outputs did or did not interrelate; whether, and if so how, grant objectives were accomplished; and the relative priority placed on the various outputs. In addition, for larger, more complex grants, such as center grants, a number of expectations for the grants, such as capacity building, dissemination, outreach, technical assistance, and training, are unlikely to be adequately reflected in the committee’s approach, which focused exclusively on specific outputs. The relationship of outputs to grants is more complex than this approach could address.

Recommendation 6-3: NIDRR should determine whether assessment of the quality of outputs should be the sole evaluation objective.

Strengthening the Output Assessment

The committee was able to develop and implement a quantifiable expert review process for evaluating the outputs of NIDRR grantees that was based on criteria used in assessing research programs in both the United States and other countries. With refinements, this method could be applied to the evaluation

Suggested Citation:"Summary." National Research Council. 2012. Review of Disability and Rehabilitation Research: NIDRR Grantmaking Processes and Products. Washington, DC: The National Academies Press. doi: 10.17226/13285.
×

of future outputs even more effectively. Nonetheless, in implementing this method, the committee encountered challenges and issues related to the diversity of outputs, the timing of evaluations, sources of information, and reviewer expertise.

Diversity of Outputs

There were acknowledged limitations in conducting the summative evaluation, such as the inability to generalize the results because of the small sample size, the need for more testing of the quality rating scale developed, and possible biases that could have arisen from sampling and measurement methods. In spite of these limitations, the quality rating system used for the committee’s summative evaluation worked well for publications in particular, which made up 70 percent of the outputs reviewed. Using the four criteria outlined above, the reviewers were able to identify and describe varying levels of quality and the characteristics associated with each. However, the committee’s quality criteria were not as easily applied to outputs such as websites, conferences, and interventions; these outputs require more individualized criteria for assessing specialized technical elements, and sometimes more in-depth evaluation methods. Applying one set of criteria, even though broad and flexible, could not guarantee sufficient or appropriate applicability to every type of output.

Timing of Evaluations

The question arises of when best to perform an evaluation of outputs. Evaluation of outputs during the final year of an award may not allow sufficient time for the outputs to have full impact. For example, some publications will be forthcoming at this point, and others will not have had sufficient time to have full impact. The trade-off of waiting a year or more after the end of a grant before performing the evaluation is the likelihood that staff involved with the original grant may not be available, recollection of grant activities may be compromised, and engagement or interest in demonstrating results may be reduced. However, publications can be tracked regardless of access to the grantee. Outputs other than publications, such as technology products, could undergo an interim evaluation to enable examination of the development and evolution of outputs.

Sources of Information

In addition to reviewing outputs directly, committee members considered information from two other sources in rating the quality of outputs: information submitted through the grantee’s APR and information provided

Suggested Citation:"Summary." National Research Council. 2012. Review of Disability and Rehabilitation Research: NIDRR Grantmaking Processes and Products. Washington, DC: The National Academies Press. doi: 10.17226/13285.
×

in a supplemental questionnaire developed by the committee. It is important to note that both of these sources involved grantee self-reports, which could be susceptible to social desirability bias. Moreover, the APR is designed as a grant monitoring tool rather than as a source of information for a program evaluation. Because the information supplied on the APR and the questionnaire was not always sufficient to inform the quality ratings, additional methods are needed to ensure complete information for such reviews.

Reviewer Expertise

The committee was directed to assess the quality of four types of prespecified outputs. While the most common output type was publications, NIDRR grants produce a variety of other outputs, including tools and measures, technology devices and standards, and informational products. These outputs vary widely in their complexity and the investment needed to produce them. The criteria used by the committee to assess the quality of outputs were based on the cumulative literature reviewed and the committee members’ own research expertise in diverse areas of disability and rehabilitation research, medicine, and engineering, as well as their expertise in evaluation, economics, knowledge translation, and policy. However, the committee’s combined expertise did not include every possible content area in the broad field of disability and rehabilitation research.

Recommendation 6-4: If future evaluations of output quality are conducted, the process developed by the committee should be implemented with refinements to strengthen the design related to the diversity of outputs, timing of evaluations, sources of information, and reviewer expertise.

Improving Use of the Annual Performance Report

The APR data set provided to the committee by NIDRR at the outset of the evaluation was helpful in profiling the grants for sampling and in listing all of the grantees’ projects and outputs. In addition, the narrative information provided in the reports was useful to the committee in compiling descriptions of the grants; however, they varied with respect to the quality of the information they contained.

Recommendation 6-5: NIDRR should consider revising its APR to better capture information needed to routinely evaluate the quality and impacts of outputs, grants, or program mechanisms. They might consider efforts such as consolidating existing data elements

Suggested Citation:"Summary." National Research Council. 2012. Review of Disability and Rehabilitation Research: NIDRR Grantmaking Processes and Products. Washington, DC: The National Academies Press. doi: 10.17226/13285.
×

or adding new elements to capture the quality criteria and dimensions used in the present summative evaluation.

Recommendation 6-6: NIDRR should investigate ways to work with grantees to ensure the completeness and consistency of information provided in the APR.

CONCLUSION

In summary, the committee concluded that NIDRR grants have produced valuable research, tools, and other outputs for advancing the field of disability and rehabilitation research in line with the agency’s mandate. Improvements to NIDRR’s priority-setting, peer review, and grant management processes, as well as consideration of alternative evaluation goals and strategies, would further enhance the quality of these processes, their results, and the agency’s efforts to improve the lives of individuals with disabilities.

Suggested Citation:"Summary." National Research Council. 2012. Review of Disability and Rehabilitation Research: NIDRR Grantmaking Processes and Products. Washington, DC: The National Academies Press. doi: 10.17226/13285.
×

This page intentionally left blank.

Suggested Citation:"Summary." National Research Council. 2012. Review of Disability and Rehabilitation Research: NIDRR Grantmaking Processes and Products. Washington, DC: The National Academies Press. doi: 10.17226/13285.
×
Page 1
Suggested Citation:"Summary." National Research Council. 2012. Review of Disability and Rehabilitation Research: NIDRR Grantmaking Processes and Products. Washington, DC: The National Academies Press. doi: 10.17226/13285.
×
Page 2
Suggested Citation:"Summary." National Research Council. 2012. Review of Disability and Rehabilitation Research: NIDRR Grantmaking Processes and Products. Washington, DC: The National Academies Press. doi: 10.17226/13285.
×
Page 3
Suggested Citation:"Summary." National Research Council. 2012. Review of Disability and Rehabilitation Research: NIDRR Grantmaking Processes and Products. Washington, DC: The National Academies Press. doi: 10.17226/13285.
×
Page 4
Suggested Citation:"Summary." National Research Council. 2012. Review of Disability and Rehabilitation Research: NIDRR Grantmaking Processes and Products. Washington, DC: The National Academies Press. doi: 10.17226/13285.
×
Page 5
Suggested Citation:"Summary." National Research Council. 2012. Review of Disability and Rehabilitation Research: NIDRR Grantmaking Processes and Products. Washington, DC: The National Academies Press. doi: 10.17226/13285.
×
Page 6
Suggested Citation:"Summary." National Research Council. 2012. Review of Disability and Rehabilitation Research: NIDRR Grantmaking Processes and Products. Washington, DC: The National Academies Press. doi: 10.17226/13285.
×
Page 7
Suggested Citation:"Summary." National Research Council. 2012. Review of Disability and Rehabilitation Research: NIDRR Grantmaking Processes and Products. Washington, DC: The National Academies Press. doi: 10.17226/13285.
×
Page 8
Suggested Citation:"Summary." National Research Council. 2012. Review of Disability and Rehabilitation Research: NIDRR Grantmaking Processes and Products. Washington, DC: The National Academies Press. doi: 10.17226/13285.
×
Page 9
Suggested Citation:"Summary." National Research Council. 2012. Review of Disability and Rehabilitation Research: NIDRR Grantmaking Processes and Products. Washington, DC: The National Academies Press. doi: 10.17226/13285.
×
Page 10
Suggested Citation:"Summary." National Research Council. 2012. Review of Disability and Rehabilitation Research: NIDRR Grantmaking Processes and Products. Washington, DC: The National Academies Press. doi: 10.17226/13285.
×
Page 11
Suggested Citation:"Summary." National Research Council. 2012. Review of Disability and Rehabilitation Research: NIDRR Grantmaking Processes and Products. Washington, DC: The National Academies Press. doi: 10.17226/13285.
×
Page 12
Suggested Citation:"Summary." National Research Council. 2012. Review of Disability and Rehabilitation Research: NIDRR Grantmaking Processes and Products. Washington, DC: The National Academies Press. doi: 10.17226/13285.
×
Page 13
Suggested Citation:"Summary." National Research Council. 2012. Review of Disability and Rehabilitation Research: NIDRR Grantmaking Processes and Products. Washington, DC: The National Academies Press. doi: 10.17226/13285.
×
Page 14
Suggested Citation:"Summary." National Research Council. 2012. Review of Disability and Rehabilitation Research: NIDRR Grantmaking Processes and Products. Washington, DC: The National Academies Press. doi: 10.17226/13285.
×
Page 15
Suggested Citation:"Summary." National Research Council. 2012. Review of Disability and Rehabilitation Research: NIDRR Grantmaking Processes and Products. Washington, DC: The National Academies Press. doi: 10.17226/13285.
×
Page 16
Next: 1 Introduction »
Review of Disability and Rehabilitation Research: NIDRR Grantmaking Processes and Products Get This Book
×
Buy Paperback | $64.00 Buy Ebook | $49.99
MyNAP members save 10% online.
Login or Register to save!
Download Free PDF

The National Institute on Disability and Rehabilitation Research (NIDRR) is the principal federal agency supporting applied research, training, and development to improve the lives of individuals with disabilities. NIDRR's mission is to generate new knowledge and promote its effective use in improving the ability of persons with disabilities to perform activities of their choice in the community, as well as to expand society's capacity to provide full opportunities and accommodations for its citizens with disabilities.

NIDRR prides itself on being proactive in establishing program performance measures and developing accountability data systems to track the progress of its grantees. An electronic annual reporting system is used to collect data from grantees on many aspects of grant operation and outputs. Various formative and summative evaluation approaches have been used to assess the quality of the performance and results of the agency's research portfolio and its grantees. Prompted by the need to provide more data on its program results, in 2009 NIDRR requested that the National Research Council (NRC) conduct an external evaluation of some of the agency's key processes and assess the quality of outputs produced by NIDRR grantees (National Institute on Disability and Rehabilitation Research, 2009a). Review of Disability and Rehabilitation Research presents the results of that evaluation.

  1. ×

    Welcome to OpenBook!

    You're looking at OpenBook, NAP.edu's online reading room since 1999. Based on feedback from you, our users, we've made some improvements that make it easier than ever to read thousands of publications on our website.

    Do you want to take a quick tour of the OpenBook's features?

    No Thanks Take a Tour »
  2. ×

    Show this book's table of contents, where you can jump to any chapter by name.

    « Back Next »
  3. ×

    ...or use these buttons to go back to the previous chapter or skip to the next one.

    « Back Next »
  4. ×

    Jump up to the previous page or down to the next one. Also, you can type in a page number and press Enter to go directly to that page in the book.

    « Back Next »
  5. ×

    Switch between the Original Pages, where you can read the report as it appeared in print, and Text Pages for the web version, where you can highlight and search the text.

    « Back Next »
  6. ×

    To search the entire text of this book, type in your search term here and press Enter.

    « Back Next »
  7. ×

    Share a link to this book page on your preferred social network or via email.

    « Back Next »
  8. ×

    View our suggested citation for this chapter.

    « Back Next »
  9. ×

    Ready to take your reading offline? Click here to buy this book in print or download it as a free PDF, if available.

    « Back Next »
Stay Connected!