Summary

The National Institute on Disability and Rehabilitation Research (NIDRR) is one of the principal federal agencies supporting applied research, training, and development to improve the lives of individuals with disabilities. NIDRR’s mission is to generate new knowledge and promote its effective use in improving the ability of persons with disabilities to perform activities of their choice in the community, as well as to expand society’s capacity to provide full opportunities and accommodations for its citizens with disabilities. Located within the Office of Special Education and Rehabilitative Services in the U.S. Department of Education (ED), NIDRR has an annual budget for grants and contracts of approximately $109 million, awarded through 14 separate program mechanisms that result in 1- to 5-year awards ranging in size from less than $100,000 to several million dollars. NIDRR has the largest budget of the three primary federal agencies with disability and rehabilitation as part of their mandate but also has the broadest mandate. NIDRR aims to reach all disability types and age groups, and its mission is tied to long-term outcomes such as independence, community participation, and employment.

Assessing the outcomes of research is a complex undertaking that can variably take into account the stated goals of the research, the contribution to the relevant field of research, the impact on the well-being of a particular population, or other related issues. The Government Performance and Results Act has led to a particular emphasis on establishing specific performance measures assessing the outputs of research programs.

NIDRR takes pride in proactively establishing program performance measures focused on the quality of its grantee outputs, developing accountability



The National Academies | 500 Fifth St. N.W. | Washington, D.C. 20001
Copyright © National Academy of Sciences. All rights reserved.
Terms of Use and Privacy Statement



Below are the first 10 and last 10 pages of uncorrected machine-read text (when available) of this chapter, followed by the top 30 algorithmically extracted key phrases from the chapter as a whole.
Intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text on the opening pages of each chapter. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.

Do not use for reproduction, copying, pasting, or reading; exclusively for search engines.

OCR for page 1
Summary The National Institute on Disability and Rehabilitation Research (NIDRR) is one of the principal federal agencies supporting applied re- search, training, and development to improve the lives of individuals with disabilities. NIDRR’s mission is to generate new knowledge and promote its effective use in improving the ability of persons with disabilities to perform activities of their choice in the community, as well as to expand society’s capacity to provide full opportunities and accommodations for its citizens with disabilities. Located within the Office of Special Education and Reha- bilitative Services in the U.S. Department of Education (ED), NIDRR has an annual budget for grants and contracts of approximately $109 million, awarded through 14 separate program mechanisms that result in 1- to 5-year awards ranging in size from less than $100,000 to several million dollars. NIDRR has the largest budget of the three primary federal agencies with disability and rehabilitation as part of their mandate but also has the broadest mandate. NIDRR aims to reach all disability types and age groups, and its mission is tied to long-term outcomes such as independence, com- munity participation, and employment. Assessing the outcomes of research is a complex undertaking that can variably take into account the stated goals of the research, the contribution to the relevant field of research, the impact on the well-being of a particu- lar population, or other related issues. The Government Performance and Results Act has led to a particular emphasis on establishing specific perfor- mance measures assessing the outputs of research programs. NIDRR takes pride in proactively establishing program performance measures focused on the quality of its grantee outputs, developing account- 1

OCR for page 1
2 REVIEW OF DISABILITY AND REHABILITATION RESEARCH ability data systems to track the results of those measures, and developing internal management systems to facilitate quality research. In 2009, NIDRR requested that the National Research Council form a committee to conduct a “process evaluation” of aspects of its grantmaking and a “summative evaluation” of the quality of grantee outputs. In addition, the committee was charged with assessing the methods it developed for conducting the summative evaluation and making recommendations for the conduct of future evaluations.1 The requested study was the most recent effort in a series of NIDRR-funded activities aimed at assessing and improving the agency’s performance. PROCESS EVALUATION NIDRR posed three questions specific to the process evaluation aimed at assessing the process used for priority writing, practices for peer review of grant applications, and the planning and budgetary processes used by grantees. The development of priorities determines the areas of emphasis for research and the specific topics to be targeted by potential applicants, while peer review is a fundamental component of the grant selection process. Although it is not possible to establish a clear causal link, these NIDRR processes, as well as planning and budgetary processes used by grantees, can influence the quality of the work produced by grantees. To address these questions, the committee reviewed existing docu- ments (e.g., legislation, Federal Register notices, NIDRR and ED policies and procedures) and interviewed NIDRR management to obtain a more thorough and cohesive understanding of these processes. The committee gained additional insight into NIDRR’s peer review process by listening to teleconferences held by three panels as they conducted their reviews of dif- ferent grant competitions. In addition, the committee collected original data through surveys of NIDRR staff, stakeholder organizations (other federal agencies, professional associations, and advocacy organizations), NIDRR peer reviewers, and principal investigators of NIDRR grants. Priority Setting To what extent is NIDRR’s priority-writing process conducted in such a way as to enhance the quality of the final results? As used in the study question, the term “priority-writing process” en- compasses many aspects of priority setting, including gathering input from 1 This aspect of the committee’s charge was summarized in a letter report provided to NIDRR in July 2011 and is also addressed in Chapter 6 of this report.

OCR for page 1
3 SUMMARY multiple sources (e.g., the field, stakeholder organizations, grantees, other agencies, and persons with disabilities and their families), identifying poten- tial topics and determining priorities for funding, writing the proposed pri- orities and having them cleared for release, and publishing notices inviting applications (NIAs) on these priorities. The term “priority setting” is used synonymously with “priority writing” in this report to clarify that the focus of the committee’s evaluation included this larger priority-setting process. The committee concluded that NIDRR’s long-range planning and priority-setting processes are successful in producing grants that are aligned with its mission and that stakeholders value as unique. Nonetheless, ar- eas for improvement were identified. First, the committee concluded that NIDRR needs to do more to inform and engage stakeholders with respect to its long-range planning and priority-setting processes. Although the re- sults of the stakeholder survey were generally positive, the transparency of the processes, responsiveness to stakeholder feedback, and use of NIDRR grantee products could be improved. Second, priority setting not only reflects the agency’s intent to influence the advancement of research in tar- geted areas but also offers specific funding opportunities for potential grant applications to the agency. The announced priorities should therefore be developed and communicated in a manner that attracts the best researchers to participate in disability and rehabilitation research. Attracting the larg- est pool of applications from which to select grantees increases the chances for the highest quality outputs. When establishing its priorities, the agency needs to consider continuity from one funding cycle to another, as well as identify future research challenges and societal needs. The committee offers recommendations in four areas to strengthen NIDRR’s long-range planning and priority-setting processes. Formation of an Advisory Council NIDRR has a broad and diverse mission that makes it challenging to set priorities that are responsive to the current state of the science and the needs of the stakeholder community. Currently, NIDRR relies on staff, the portfolio of existing projects, recent findings from completed grants, and the current research literature, as well as guidance from federal partners, for input to the priority-setting process. NIDRR’s statute directs it to establish a standing Rehabilitation Research Advisory Council to advise the director of the agency on research priorities and the development of the agency’s Long-Range Plans. While NIDRR has formed ad hoc advisory bodies to support the development of its Long-Range Plans, a standing body has never been formed. Given NIDRR’s mandate, the council should be tasked with providing advice on both disability and rehabilitation research.

OCR for page 1
4 REVIEW OF DISABILITY AND REHABILITATION RESEARCH Recommendation 3-12: NIDRR should fulfill the statutory mandate to form and utilize a standing disability and rehabilitation research advisory council to advise on the priority-setting process and pro- vide input for priority setting. Most federal funding agencies, including the National Institutes of Health (NIH), the National Science Foundation (NSF), and the National In- stitute for Occupational Safety and Health (NIOSH), use standing advisory bodies. A standing advisory body is likely to add stability and continuity to both NIDRR’s long-range planning and its priority setting. The commit- tee recognizes that NIDRR, like other federal research agencies, will face challenges in capturing the broad diversity of perspectives held by its many stakeholders. However, the committee feels strongly that, like other federal research agencies, NIDRR can meet these challenges. Strategic Planning NIDRR’s multiple stakeholders include persons with disabilities and their families, the scientific community, professional associations, and advo- cacy organizations representing a variety of disability groups. In the face of this diversity, it is important for the agency to have a consistent mechanism for gathering information and input to inform the strategic planning process beyond the input that will be possible through an advisory council. NIDRR utilizes input from multiple sources, such as its stakeholders, other federal agencies, the Interagency Committee on Disability Research (ICDR), the current literature, and state-of-the-science conferences. However, the pro- cesses for gathering input and developing proposed Long-Range Plans have varied from one plan to another. Negative comments from the field gener- ated by the last draft Long-Range Plan, coupled with the plan’s subsequent delay, which caused NIDRR to operate under the prior plan for several years beyond its intended time frame, suggest a breakdown in NIDRR’s priority- setting process. The lack of a permanent director also hampers and delays the agency’s priority-setting process.3 Recommendation 3-2: NIDRR should use a structured, consistent, and inclusive strategic planning process to develop its Long-Range Plans and priorities. NIDRR might consider the long-range planning and priority setting pro- cesses of other funding agencies, including NIH, NSF, and NIOSH, which 2 The committee’s recommendations are numbered according to the chapter of the report in which they appear. 3 At the time of this writing, a permanent NIDRR Director had been recently hired.

OCR for page 1
5 SUMMARY have sought to integrate long-range planning and priority-setting processes through specific initiatives such as the NIH Roadmap, the NSF Strategic Plan, and the National Occupational Research Agenda. Establishment of a Standard Calendar For many program mechanisms, NIDRR has not established a regular schedule for drafting and approving priorities and NIAs and disseminating them to the field. ED has a lengthy review and approval process for obtain- ing clearance for the release of priorities and NIAs. The variability in the length of the clearance process may be an important factor, among others, that impacts the timing of the release of NIAs. The irregular or delayed release of NIAs may affect NIDRR’s ability to provide individuals sufficient notice of grant opportunities or an optimal amount of time to complete ap- plications. An irregular schedule may discourage the best investigators from submitting applications. Additionally, certain program mechanisms (such as Model Systems) include collaboration between institutions. Irregular post- ing and shortened response times hamper the ability of applicants to identity and recruit appropriate collaborators. These factors are likely to limit the number of investigators who apply and adversely affect the quality of the applications they submit. Additionally, young investigators less familiar with NIDRR are more likely to pursue grants from other agencies. Recommendation 3-3: NIDRR should utilize a standard calendar for the setting of priorities, publication of notices inviting appli- cations, submission of applications, and peer review meetings to improve the efficiency of the process. NIDRR has made efforts to standardize the schedule for NIAs. The committee suggests that program mechanisms competed on a yearly basis have a consistent annual schedule for the submission and review of applica- tions. For multiyear grants, the committee recommends that NIDRR estab- lish a long-range operational plan listing projected future grant application submission dates, pending funding availability in that fiscal year. Soliciting Applications Like other federal agencies, NIDRR makes its NIAs available at Grants. gov in addition to publishing them in the Federal Register. The agency also uses a contractor to notify former grantees and others who, via their webpage, express an interest in receiving NIAs. NIDRR would benefit from more active efforts to solicit interest in its funding announcements.

OCR for page 1
6 REVIEW OF DISABILITY AND REHABILITATION RESEARCH Recommendation 3-4: NIDRR should expand its efforts to dis- seminate notices inviting applications to new potential applicants, including developing a communication strategy to ensure that the notices reach new audiences. To expand dissemination, notices should be sent to the disability and re- habilitation professional and research organizations that make up NIDRR’s stakeholder network and to university departments and offices of sponsored research. The latter could perhaps be accomplished through collaboration with other federal research programs that regularly send funding notices to universities. Peer Review To what extent are peer reviews of grant applications done in such a way as to enhance the quality of final results? NIDRR’s peer review process encompasses recruiting and training re- viewers, conducting the review, and approving the awards. As with priority setting, it is challenging to link the peer review process directly to specific re - sults because the quality of the portfolio, grants, and outputs emerging from the process is the product of multiple complex factors. It is clear, however, that the peer review process used by NIDRR contributes significantly to the success of the grant award program and the quality of the outputs produced. The responses to the committee’s peer review survey were largely posi- tive, including peer reviewers’ responses related to their experiences with NIDRR’s peer review process and how it compares with the processes used by other federal research agencies. While the committee concluded that NIDRR’s peer review process is generally good, there are opportunities for improvement that would likely enhance the process and the quality of final results. The committee offers three recommendations to this end. Enhancements to the Peer Review Process The committee concluded that NIDRR’s peer review process is ham- pered by a limited pool of potential reviewers. NIDRR’s competition managers take great care to assemble and facilitate qualified review panels and spend considerable time recruiting and screening potential reviewers. Competition managers regularly must manage potential conflicts of inter- est and rule out qualified reviewers. Despite these staff efforts, however, the committee found evidence that a number of panels are smaller than NIDRR’s recommended size, reviewers are added so close to the meeting

OCR for page 1
7 SUMMARY date that they have inadequate time to prepare, and reviewers lacking neces- sary scientific expertise may be participating in reviews. The formation of formal “cohorts” of reviewers with particular areas of expertise would reduce the recruiting burden on NIDRR staff in locat- ing reviewers needed for individual competitions with specific targeted expertise. In addition, reviewers surveyed by the committee reported that the quality of the training they received was inconsistent; enhancing this training would be a simple and effective way to improve the quality of the review process. Finally, considerable variation exists among competitions in the way NIDRR staff facilitate panel discussions. The result is variation in the quality of the discussions; such inconsistency also can result in confusion and negatively influence overall quality. Recommendation 4-1: NIDRR should further strengthen the peer review infrastructure by expanding the pool of high-quality review- ers; establishing standing panels, or formal cohorts of peer review- ers with specialized knowledge and expertise as appropriate for the program mechanisms; enhancing reviewer training; and improving the consistency of NIDRR staff facilitation of panel meetings and the quality of feedback provided to grantees. Reducing Reviewer Burden Participating in NIDRR’s peer review process is a significant burden for a large percentage of reviewers. Many reviewers spend more time than they would like preparing, and the review days are long and intense. This sig- nificant time commitment makes it less likely that qualified and experienced reviewers will participate. Reviewers surveyed also reported sometimes hav- ing insufficient time to review proposals, which could affect the quality of the review discussions. The committee concluded that the review process is so burdensome to peer reviewers as to threaten the quality of the process. Recommendation 4-2: NIDRR should streamline the review process in order to reduce the burden on peer reviewers. Use of Consumer Peer Reviewers To address its mission, NIDRR makes concerted efforts to include both scientists with disabilities and consumers without scientific expertise in the peer review process. Consumers can represent the experiences and views of their particular disability communities and can evaluate applications for relevance to their community’s needs and concerns.

OCR for page 1
8 REVIEW OF DISABILITY AND REHABILITATION RESEARCH All reviewers, including researchers and consumers, should have the appropriate expertise to review those elements of proposals to which they are assigned. If consumers are to review scientific aspects of proposals, they should have the relevant expertise, or NIDRR should consider providing them with relevant methodological training. NIDRR should review and monitor the role of consumers and researchers in peer review to ensure that quality is not compromised. Recommendation 4-3: NIDRR should continue to have consumer representation in the peer review process and establish procedures to guide the participation of those without scientific expertise. Many federal research programs involve consumers without scientific expertise in peer review. NIDRR may want to examine such practices at NIH, Congressionally Directed Medical Research Programs, Juvenile Dia- betes Research Foundation, and other agencies to inform its own approach to including nonresearchers in peer review. Grant Management What planning and budgetary processes does the grantee use to pro- mote high-quality outputs? The committee assessed grantee planning and budgetary processes in the larger context of NIDRR’s structure and processes supporting grant man- agement. To perform this assessment, the committee (1) reviewed existing documentation on the grant management and monitoring processes of ED and NIDRR, (2) gathered information from principal investigators about the processes they use for managing grants, and (3) interviewed NIDRR staff to obtain their perspectives on how grant monitoring facilitates grantees’ efforts to manage their grants for successful results. NIDRR appears to have developed a good plan for upgrading its routine monitoring of grants and for identifying and monitoring grants that are at risk of noncompliance with ED or NIDRR requirements and performance expectations. On the whole, grantees appear to appreciate aspects of NI- DRR’s grant management processes that facilitate their own grant manage- ment strategies. While grantees generally commented that NIDRR’s grant management processes were effective in facilitating their own grant manage- ment processes, they offered some suggestions for improvement that would help them further. NIDRR staff also offered suggestions for improvement, focused on strengthening their capacity to monitor grants and help grant- ees stay on course in implementing their grants and meeting performance expectations. Among other suggestions, they expressed the need for smaller

OCR for page 1
9 SUMMARY grant caseloads, additional travel funds for on-site monitoring of grants that require higher levels of technical assistance, more training for new project officers to promote consistency and quality in the monitoring process, and a freer flow of communication between project officers and NIDRR planning staff with respect to financial information. Recommendation 5-1: NIDRR should continue to focus efforts on improving its grant monitoring procedures and specific elements of its overall grant management system that impact grantee-level planning, budgets, and the quality of outputs. From its interviews with grantees, the committee also learned that some grants focused on developing technology innovations may not accord well with a management template that calls for strict up-front planning and adherence to original designs and timetables. Similarly, a grant funding a large multisite study may require more or different supervision, monitor- ing, and technical assistance than a more focused or limited study. Grantees expressed the need for greater flexibility in grant management so they can stay on the cutting edge of technology or adapt more easily to changing needs of multisite research projects. Recommendation 5-2: NIDRR should review the requirements placed upon technical innovation grants and large multisite studies to ensure that planning, reporting, supervisory, and technical as- sistance requirements fit their particular circumstances. To what extent are the results of the reviewed research and develop- ment outputs used to inform new projects by both the grantee and NIDRR? To assess how research and development outputs inform new projects, the committee (1) reviewed information from NIDRR management about how they use the results of their grantees’ research and (2) reviewed infor- mation from grantees about new projects that have been generated from their grants. The committee concluded that research and development outputs are used to generate new projects by grantees to a great extent and lead to substantial numbers of new collaborations with other researchers and organizations, as well as transfers of data, instruments, or models to other projects, and commercialization of technology products. SUMMATIVE EVALUATION The summative evaluation, designed to inform NIDRR’s performance monitoring and reporting, involved assessing the quality of outputs pro-

OCR for page 1
10 REVIEW OF DISABILITY AND REHABILITATION RESEARCH duced by a sample of 30 NIDRR grantees.4 These grants were drawn from nine NIDRR program mechanisms: Burn Model System, Traumatic Brain Injury Model System, Spinal Cord Injury Model System, Rehabilitation Research and Training Center, Rehabilitation Engineering Research Cen- ter, Disability and Rehabilitation Research Project-General, Field Initiated Project, Small Business Innovation Research II, and Switzer Fellowship. The committee reviewed four different types of outputs, as defined by NIDRR: publications; tools, measures, and intervention protocols; technology prod- ucts and devices; and informational products. The committee developed and used four criteria to assess quality: (1) technical quality; (2) advancement of knowledge or the field; (3) likely or demonstrated impact (on science, persons with disabilities and their families, provider practice, health and social systems, social and health policy, and the private sector/commercialization); and (4) dissemination according to principles of appropriate knowledge translation. A total of 148 outputs produced by the 30 grantees were rated on each criterion, using a 7-point scale, where 1 indicated poor quality, 4 indicated good quality, and 7 indicated excellent quality. Ratings on each of the four criteria were distributed fairly symmetrically along the scale, with the larg- est proportion of scores falling at the midpoint of 4 and with most being slightly skewed toward the higher end of the scale. Although close to 75 percent of the outputs rated fell in the “good to excellent” range of the quality scale (i.e., mean ratings of 4 or greater on the 7-point quality scale), 25 percent of the outputs fell in the lower quality range (1 or “poor” to 3 or “below good”) across all four criteria. The committee offers NIDRR two recommendations for assisting grantees in continuously improving the quality of their outputs. First, the quality of outputs is the product of multiple complex factors that involve the priority-setting process, the funding level, the peer review process, the quality of the proposed science/research and the grantees, and ultimately the quality of the research findings. For grantees that are not performing optimally, NIDRR may conduct ongoing formative reviews with experts to identify strategies for improvement, increase its grant monitor- ing activities, and require additional grantee reporting. Grantees generally report that NIDRR’s oversight and reporting functions foster successful grants and high-quality outputs by assisting them in adhering to their budget and timeline, providing an external quality assurance mechanism for their project management, and prompting them to maintain their focus on project goals for high-quality products. 4 The committee performed a random sampling of grants at the level of program mechanism. Five mechanisms were excluded in consultation with NIDRR.

OCR for page 1
11 SUMMARY Recommendation 6-1: Although close to 75 percent of outputs were rated as “good to excellent” (i.e., 4 or higher on the 7-point quality scale), NIDRR should make it clear that it expects all grantees to produce the highest quality outputs. The intent of this recommendation is for NIDRR to encourage all of its grantees to publish in peer-reviewed journals, present at national meetings, publish/disseminate materials, and bring technology solutions to market while producing these outputs at the highest levels of quality. To this end, NIDRR should push forward by establishing clear and consistent expecta- tions for grantees to publish in higher-impact journals as one indicator of higher quality. For outputs other than publications, NIDRR should establish standards for quality to be achieved and adopt appropriate metrics to as- sess adherence to these standards. One way of setting the quality bar higher would be to encourage grantees to use standardized reporting forms and checklists5 for reviewing the technical quality of their own work before subjecting it to external review. Second, despite limitations in the use of bibliometrics,6 they are a valu- able and objective set of measures that can be used in combination with other assessment strategies. NIDRR has conducted bibliometric analyses in the past, but has not routinely incorporated use of these metrics into its performance measurement process. Recommendation 6-2: NIDRR should consider undertaking biblio- metric analyses of its grantees’ publications as a routine component of performance measurement. Bibliometric analyses would take advantage of an existing data source for periodic measurement of the scientific impact of NIDRR grantee publi- cations, and would provide an indicator of the extent to which these grant outputs are being disseminated and used. This type of metric is being rec- ommended for use in combination with other measures, just as it was used in the committee’s evaluation along with expert review and supplemental evidence of the impact an article may have had on science, persons with disabilities and their families, provider practice, health and social systems, social and health policy, and the private sector/commercialization. 5 Seehttp://www.equator-network.org/ for examples. 6 Common bibliometric measures include the impact factor of journals in which articles are published and the number of times an article is cited in other articles.

OCR for page 1
12 REVIEW OF DISABILITY AND REHABILITATION RESEARCH SELF-ASSESSMENT OF THE COMMITTEE’S METHODS FOR OUTPUT REVIEW AND RECOMMENDATIONS FOR FUTURE EVALUATIONS The committee developed and implemented an evaluation process for assessing the outputs of NIDRR’s grantees and identifying the various levels of quality and characteristics of those outputs. The committee spent con- siderable time selecting and refining the criteria used to assess the quality of outputs. While some variation was evident in the independent scoring among the committee members, it was rarely extreme, particularly after group discussions. However, as summarized below, the committee encoun- tered several challenges and obstacles during the course of its work that limited the generalization of its findings and restricted what could be said about the totality of outputs generated by all NIDRR grantees. Defining Future Evaluation Objectives The primary focus of the committee’s summative evaluation was on assessing the quality of outputs produced by grantees; the evaluation did not include in-depth examination or comparison of the larger contexts of the funding programs, grants, or projects within which the outputs were produced. However, the committee was asked to formulate an overall rating for each grant based on the outputs reviewed and the information avail- able about the grant from the Annual Performance Report (APR). Results at the grant level were subject to limitations resulting from the general lack of information about how the outputs did or did not interrelate; whether, and if so how, grant objectives were accomplished; and the relative priority placed on the various outputs. In addition, for larger, more complex grants, such as center grants, a number of expectations for the grants, such as ca- pacity building, dissemination, outreach, technical assistance, and training, are unlikely to be adequately reflected in the committee’s approach, which focused exclusively on specific outputs. The relationship of outputs to grants is more complex than this approach could address. Recommendation 6-3: NIDRR should determine whether as- sessment of the quality of outputs should be the sole evaluation objective. Strengthening the Output Assessment The committee was able to develop and implement a quantifiable expert review process for evaluating the outputs of NIDRR grantees that was based on criteria used in assessing research programs in both the United States and other countries. With refinements, this method could be applied to the evalu-

OCR for page 1
13 SUMMARY ation of future outputs even more effectively. Nonetheless, in implementing this method, the committee encountered challenges and issues related to the diversity of outputs, the timing of evaluations, sources of information, and reviewer expertise. Diversity of Outputs There were acknowledged limitations in conducting the summative evaluation, such as the inability to generalize the results because of the small sample size, the need for more testing of the quality rating scale developed, and possible biases that could have arisen from sampling and measure- ment methods. In spite of these limitations, the quality rating system used for the committee’s summative evaluation worked well for publications in particular, which made up 70 percent of the outputs reviewed. Using the four criteria outlined above, the reviewers were able to identify and describe varying levels of quality and the characteristics associated with each. However, the committee’s quality criteria were not as easily applied to outputs such as websites, conferences, and interventions; these outputs require more individualized criteria for assessing specialized technical ele- ments, and sometimes more in-depth evaluation methods. Applying one set of criteria, even though broad and flexible, could not guarantee sufficient or appropriate applicability to every type of output. Timing of Evaluations The question arises of when best to perform an evaluation of outputs. Evaluation of outputs during the final year of an award may not allow sufficient time for the outputs to have full impact. For example, some pub- lications will be forthcoming at this point, and others will not have had sufficient time to have full impact. The trade-off of waiting a year or more after the end of a grant before performing the evaluation is the likelihood that staff involved with the original grant may not be available, recollec- tion of grant activities may be compromised, and engagement or interest in demonstrating results may be reduced. However, publications can be tracked regardless of access to the grantee. Outputs other than publications, such as technology products, could undergo an interim evaluation to enable examination of the development and evolution of outputs. Sources of Information In addition to reviewing outputs directly, committee members consid- ered information from two other sources in rating the quality of outputs: information submitted through the grantee’s APR and information provided

OCR for page 1
14 REVIEW OF DISABILITY AND REHABILITATION RESEARCH in a supplemental questionnaire developed by the committee. It is important to note that both of these sources involved grantee self-reports, which could be susceptible to social desirability bias. Moreover, the APR is designed as a grant monitoring tool rather than as a source of information for a program evaluation. Because the information supplied on the APR and the question- naire was not always sufficient to inform the quality ratings, additional methods are needed to ensure complete information for such reviews. Reviewer Expertise The committee was directed to assess the quality of four types of pre- specified outputs. While the most common output type was publications, NIDRR grants produce a variety of other outputs, including tools and measures, technology devices and standards, and informational products. These outputs vary widely in their complexity and the investment needed to produce them. The criteria used by the committee to assess the quality of outputs were based on the cumulative literature reviewed and the com- mittee members’ own research expertise in diverse areas of disability and rehabilitation research, medicine, and engineering, as well as their expertise in evaluation, economics, knowledge translation, and policy. However, the committee’s combined expertise did not include every possible content area in the broad field of disability and rehabilitation research. Recommendation 6-4: If future evaluations of output quality are conducted, the process developed by the committee should be implemented with refinements to strengthen the design related to the diversity of outputs, timing of evaluations, sources of informa- tion, and reviewer expertise. Improving Use of the Annual Performance Report The APR data set provided to the committee by NIDRR at the outset of the evaluation was helpful in profiling the grants for sampling and in listing all of the grantees’ projects and outputs. In addition, the narrative infor- mation provided in the reports was useful to the committee in compiling descriptions of the grants; however, they varied with respect to the quality of the information they contained. Recommendation 6-5: NIDRR should consider revising its APR to better capture information needed to routinely evaluate the qual- ity and impacts of outputs, grants, or program mechanisms. They might consider efforts such as consolidating existing data elements

OCR for page 1
15 SUMMARY or adding new elements to capture the quality criteria and dimen- sions used in the present summative evaluation. Recommendation 6-6: NIDRR should investigate ways to work with grantees to ensure the completeness and consistency of infor- mation provided in the APR. CONCLUSION In summary, the committee concluded that NIDRR grants have pro- duced valuable research, tools, and other outputs for advancing the field of disability and rehabilitation research in line with the agency’s mandate. Improvements to NIDRR’s priority-setting, peer review, and grant manage- ment processes, as well as consideration of alternative evaluation goals and strategies, would further enhance the quality of these processes, their results, and the agency’s efforts to improve the lives of individuals with disabilities.

OCR for page 1