Review of Disability and Rehabilitation Research: NIDRR Grantmaking Processes and Products (2012)

5

Grant Management

This chapter addresses the following two key study questions:

Key Question #3. What planning and budgetary processes does the grantee use to promote high-quality outputs?

Key Question #5. To what extent are the results of the reviewed research and development outputs used to inform new projects by both the grantee and NIDRR?

These questions are addressed together in this chapter within the larger framework of grant management because the information gathered by the committee indicated they are interrelated. A firm foundation of grant management processes at the agency and grant levels (in terms of planning, quality assurance, reporting, and resource management) sets the stage for successful grant implementation and production of outputs, which in turn can influence the likelihood of informing new projects.

To correspond with the Key Question #3, the first section of this chapter describes how the National Institute on Disability and Rehabilitation Research (NIDRR) manages grants through its agency structure and processes, the planning and budgetary processes used by grantees in managing their grants to promote high-quality outputs, and how NIDRR’s grant monitoring efforts facilitate grantees’ planning and budgetary processes. Corresponding to Key Question #5, the second section summarizes information from grantees concerning how their research and development outputs have been used to inform new projects and collaborations, as well as how NIDRR

uses the results of grantees’ research. Conclusions are presented at the end of each section; the first section also includes recommendations to improve the grant management process.

GRANT MANAGEMENT PROCESSES

To address the planning and budgetary processes used by grantees, it is necessary to examine these grantee-level processes and associated requirements in the larger context of the structure and processes that support grant management at NIDRR and to obtain the perspectives of both NIDRR grantees and NIDRR staff. To these ends, the committee reviewed existing documentation on NIDRR’s grant management and monitoring processes, interviewed NIDRR management to gather additional information about the processes,¹ collected information from principal investigators about the processes they use for managing grants, and interviewed NIDRR staff to obtain their perspectives on how grant monitoring facilitates grantees’ efforts to manage their grants for successful results.

NIDRR’s Grant Management Structure and Processes

NIDRR uses both the U.S. Department of Education (ED) and its own postaward grant management procedures and practices (U.S. Department of Education, 2009) to establish working partnerships with grantees and to monitor projects for performance and financial compliance. Grant management activities are supported by ED’s web-based grant management system, called G5; by the Office of Special Education and Rehabilitative Services’ (OSERS’) web-based records management system, called TRIM; and by NIDRR’s Annual Performance Report (APR) system. A postaward conference sets the stage for managing individual grants with regard to needs and expectations, and NIDRR uses various strategies to monitor grantee progress.

Setting the Stage for Individual Grant Management

The planning and budgetary processes used by grantees evolve directly from NIDRR procedures concerning grant selection, the peer review process, and negotiated postaward grant management activities (National Institute on Disability and Rehabilitation Research, 2010). The evaluation of grant proposals includes rating such elements as plans for operation and evaluation, as well as the adequacy and reasonableness of the budget

_______________

¹ The committee conducted interviews with NIDRR and ED management in four sessions during summer 2010 and one session in spring 2011.

and resources. Within 30 days of an award, NIDRR project staff conduct a postaward conference with the grantee (U.S. Department of Education, 2009). The purpose of this conference is to

•   establish a mutual understanding of the expected project outcomes;

•   establish a mutual understanding of the measures to be used for assessing the project’s progress and results;

•   clarify the frequency of and methods for project monitoring and ongoing communication between NIDRR and the grantee;

•   discuss other technical assistance to be provided by NIDRR or other service providers;

•   review and clarify relevant regulatory or statutory requirements; and

•   review and clarify project activity and/or budget issues and concerns.

NIDRR staff generally conduct the conference via telephone, but the conference may also take place in person or via e-mail or written communication. Staff use a standard checklist to conduct the conference that covers such items as the grant award notification; the content of the initial award letter; the content of the proposal; peer reviewers’ comments and concerns; progress toward Institutional Review Board (IRB) approval, where applicable; relevant performance measures; expectations regarding NIDRR’s online APR; and planning and reporting of outcomes. NIDRR staff document the content of the conference and all subsequent contacts with the grantee in the official grant file.

NIDRR’s Grant Monitoring

NIDRR’s written procedures call for establishing working partnerships with grantees to monitor projects for performance and financial compliance (U.S. Department of Education, 2009). Following the postaward conference, periodic monitoring takes place as appropriate in accordance with a basic set of monitoring procedures to ensure the achievement of results specific to the application and any revisions, as well as progress against established performance measures that were discussed in the conference. Monitoring tools include electronic quarterly fiscal reporting and annual monitoring reports on activities undertaken during the previous fiscal year. Recipients of multiyear discretionary awards must complete an APR and submit it to NIDRR.

Fiscal monitoring As part of the monitoring process, NIDRR project officers pay particular attention to grantees’ fiscal activities (U.S. Department of Education, 2009). The project officers review grantees’ cash drawn down

on an annual basis (or more frequently if a grantee experiences performance problems). They generally use the G5 system to determine whether actual cash draws are consistent with expected expenditures based on a project’s scope of work and milestones.

Annual Performance Report The APR that recipients of multiyear discretionary awards must submit to their project officer provides data that relate progress based on the scope and objectives of the approved application or any approved revisions (U.S. Department of Education, 2009). NIDRR has developed the APR as a web-based online reporting system (National Institute on Disability and Rehabilitation Research, 2009). The APR system contains sections on general information, award abstract, budget, funding, descriptions of research and development projects, output summaries, and descriptions of the most important outputs. Grantees also report on their progress in implementing their disability-focused research, development, training, technical assistance, and knowledge translation. Grantees use the APR as well to report on the results or accomplishments of this work. According to NIDRR documentation, the APR system exceeds the minimum ED requirements for reporting (National Institute on Disability and Rehabilitation Research, 2010). Project officers use their grantees’ APRs for monitoring and tracking progress and results. The APR is maintained by an external contractor who also performs analyses of the data for management and reporting purposes according to specifications provided by NIDRR.

Formative reviews Formative reviews with individual grantees are occasionally convened (two to three times a year) to give grantees who have been identified as needing additional assistance the opportunity to discuss their research methodology with experts in their field. A formative review is typically conducted after the project officer has tried other means of assisting the grantee. Experts from the original peer review panel are invited if available; if not, other experts in the grantee’s topic area are invited. The experts generally conduct the review by teleconference or webinar and provide suggestions to the grantee for improving the research plan.

Monitoring of at-risk grants The processes described above pertain to routine monitoring of grants. In response to a recent ED initiative, NIDRR is instituting a new process for identifying grants at risk of failure to comply with program requirements, reach performance goals, comply with grant administration and financial management requirements, and/or account for past performance (National Institute on Disability and Rehabilitation Research, 2010). At-risk grants may be identified at different points. One point may be during the peer review process. Although a grant application

may receive a fundable score in peer review, there may be concerns about certain elements of the application (e.g., feasibility of achieving the projected sample size). At-risk grants may also be identified at the postfunding conference, through routine teleconferences, or through a grantee’s APR. Criteria for identifying grants that may be at risk for not reaching performance goals include prior performance; peer reviewer concerns; concerns raised during the site visit; staff concerns regarding feasibility with respect to staffing, resources, or design; slow startup; failure to hire key staff; slow enrollment of subjects; loss of a key collaborator; or failure to report progress. Administrative and financial criteria for identifying at-risk grants include failure to draw down funds, excessive drawdown, or some other financial disclosure or deviation that demonstrates failure to adhere to ED guidelines. Strategies for monitoring at-risk grants, used as needed and in consultation with NIDRR management, include conducting formative reviews; establishing performance targets; scheduling regular, frequent written or oral updates; conducting site visits; changing the status of a grant to a cooperative agreement; and delaying continuation awards (National Institute on Disability and Rehabilitation Research, 2010).

Grantee Perspectives on Their Grant Management Processes

To obtain grantees’ perspectives on their grant management processes, the 30 grantees who participated in the summative evaluation were asked to complete a set of written questions asking them to describe what types of planning, project management, and budgetary processes were used to promote high-quality outputs. They were asked to consider the following questions when crafting their statements:

•   Which processes you used were helpful and how? How could they be improved?

•   Did you dedicate funds for quality assurance activities?

•   How did you track progress and spending against your original plans for the grant?

•   If grants or projects were jointly funded by NIDRR and other extramural or intramural sources, how did you ensure that NIDRR resources were used exclusively for NIDRR-funded activities?

•   How do NIDRR grant management processes influence the quality of outputs?

This set of questions on grant management was asked at the end of a longer questionnaire on which the grantees described the outputs that were to be reviewed by the committee in the summative evaluation. Principal investigators of 28 of the 30 grantees that participated in the summative evaluation responded to this set of questions. Because respondents did not

necessarily answer all the questions or answer the questions the same way, the number of respondents for each answer varies. Narrative data were analyzed using standard qualitative analysis techniques (see Chapter 2 for a description of these techniques). The number of grantees that responded to each set of questions is noted in the section headings below.

Consensus Views

Except for new product development projects and complex multicenter studies, grantees reported that the current NIDRR requirements for planning, project management, and budgeting generally helped ensure quality outputs. Several commented on the positive trend in changes to the requirements over the years. For example, one grantee cited the changes made during the grant under study:

In Years 1 and 2 of the grant they were required to report more of an “output count” methodology. In year 3 a different reporting method was introduced focused on the development of short-term and long-term goals. Then “accomplishment nuggets” were to be nominated, with rigorous reporting for each nomination, relating back to one of the outcome goals. Finally, in Years 4 and 5 NIDRR stabilized on a somewhat less rigorous, more easily understood reporting of “most important” outputs, tied to one of up to four outcome oriented goals…. NIDRR has been making good progress in recent years on stabilizing its evaluation protocols and procedures.

Planning (N = 14)

The bulk of planning for the full range of projects was done in the proposal-writing stage, for which NIDRR has detailed requirements. As one respondent explained:

The NIDRR requirements are quite detailed and extensive…. In the sense that the plans are well developed, the NIDRR grants make planning and management fairly routine since we are carrying out the commitments made in the proposal.

It was also noted that having an evaluation plan was particularly helpful, as was having a quality assurance plan.

In addition to up-front planning, continuous planning was carried out for more complex grants through regular planning meetings and consultation with other researchers as needed. Although the many requirements were off-putting for some in the beginning, they proved to be helpful (except for new product development). One respondent noted:

NIDRR built in the development of outputs into planning. This made us more thoughtful.

Respondents attributed success in implementing their plans to a number of factors—including not having overly ambitious goals, having a high degree of oversight by the project director, using feedback from multisite study investigators, and including the development of outputs in planning.

Respondents also noted difficulties in creating and maintaining an adequate plan, which may be instructive in designing and implementing changes to the process. Examples given for difficulties encountered included the following:

•   Projects deviating from more standard NIDRR research projects. For example, new product development was viewed as a fundamentally different type of project than other academic research efforts around which NIDRR processes were developed, so many requirements and metrics did not fit these projects.

•   A complex, multisite project for which there were no planning models or in which fidelity of implementation over large numbers of sites caused management and budget issues.

•   Projects may have been adequately planned under procedures in effect at the start of the grant, but plans had to be changed when NIDRR implemented process changes mid-project, such as imposing new reporting requirements.

•   Cases were cited in which unpredictable events impacted staffing. In one case, for example, project staff were jointly funded by funds for another project. When they lost the funds for the other project, the staff left the NIDRR project as well.

Project Management (N = 24)

The variety of the projects resulted in a range of management complexity, from a single action on small individual grants (e.g., Switzer)—writing one check to the university that oversaw the dispersal of salary funds—to highly complex protocols for larger center grants requiring complex management and fidelity in implementation across many sites for success. Key project management elements specified by one or more respondents included the following:

•   the APR was noted as particularly helpful in “keep[ing] to both the budget and the timeline,” although the APR may not reflect accomplishments, such as journal articles, not in the original plan and delayed until the end of the project;

•   quarterly meetings with the NIDRR project officer;

•   the quality of NIDRR project officers;

•   weekly, monthly, and annual meetings of project staff;

•   specification of the quality assurance responsibilities and tasks of project staff and committees;

•   annual meetings of project advisory boards;

•   online project planning tools, such as a commercial project management reporting system used by another research center or an online management tool provided by the institution;

•   task analytic project management;

•   sufficient training and supervision for complex, multisite grant implementers;

•   use of institutional grant management and budget management services where available;

•   frequent and consistent project monitoring; and

•   the quality of investigators and technical staff.

Dedicated Quality Assurance Funds (N = 14)

Of the 14 respondents who addressed this issue, only 8 reported separating out quality assurance activities from the rest of their budget. Quality assurance activities were generally an integral part of the planned project. To the extent that oversight was part of staff responsibilities, salary for those staff was obligated for quality assurance activities. A few respondents mentioned dedicating quality assurance funds for bringing in external experts on their oversight committee, for funding audio taping and ongoing data reports during the study, for monitoring implementation visits, for conducting conference calls, and for traveling to NIDRR and principal investigator meetings. One comment noted that the respondent’s institution would not allow budgeting for quality assurance activities; however, it was not clear how the institution defined quality assurance activities.

Budgetary Processes (N = 20)

Monitoring of expenditures and budgets often was done by institutional grant management or accounting divisions or through the use of project management software. Others tracked the budget as part of the project director’s monitoring. When asked specifically about what procedures grantees used to ensure that only NIDRR funds were used to fund NIDRR activities in jointly funded projects, only three grantees indicated the use of joint funding. One grantee reported working with the NIDRR project officer to ensure that there was no double billing of time. Another stated that the principal investigator and project director identified for the university financial office how NIDRR resources were used exclusively to support NIDRR-funded activities and to track in-kind and other funding.

How NIDRR Grant Management Processes Influence Results (N = 18)

When respondents were asked how NIDRR grant management processes influence grant results, topics cited most frequently with regard to fostering successful grants and high-quality outputs related to interactions with NIDRR project officers, reporting requirements, flexibility in management and budgeting, and timing of the grant application process. Following are some of the points made:

•   Interactions with NIDRR project officers

—The attitude (high standards and emphasis on quality) of the NIDRR project officer has the greatest impact on quality (based on experience with three project officers).

—One grantee stated that it would have been useful to have more regular contacts or reporting opportunities earlier in the final design and proceduralization stages of the study.

•   Reporting requirements

—The APR requires that budget information be provided by the accounting department, which has been very useful.

—The detailed tracking requirements of the online APR, although quite time-consuming and burdensome, provide motivation to keep focused on the overall project goals for high-quality products.

—Quarterly reports are helpful as a quality assurance mechanism.

—One grantee praised NIDRR for its recently implemented goaland objective-oriented reporting scheme, which was viewed as far superior to previous schemes. The grantee did note, however, that the new scheme limits reporting of accomplishments, meaning some good work is not being reported.

—One grantee suggested that reports should focus more on how the work of the grant is either succeeding or failing at bringing effective and practical new services and/or devices to the market.

—Another commented that methods should be developed for capturing outputs produced at the end, or shortly after the end, of a project.

•   Flexibility in management and budgeting

—One grantee commented that NIDRR needs more flexibility to extend its research and development grants and adapt budgets not only to fit the initial scope of a project but also to accommodate discoveries and opportunities encountered during the course of the project.

—Different management tools are needed for projects involving new product development rather than academic research.

—The level of funding of Model System grants is disproportionate to what is expected of grantees. For example, the insufficient funding is a major limiting factor for collaborative research efforts, which need to be well planned if scientifically rigorous treatment and intervention studies with adequate sample sizes are to be carried out.

•   Timing of the grant application process

—Standard deadlines are needed for grant applications. Having predetermined dates for application submissions would improve timely notification of grant availability and give principal investigators more time to prepare and consequently submit higherquality plans and applications.

—Applications submitted in January should receive notification of application status and award in the spring to avoid inadequate staffing at the start of the project and subsequent protracted timelines.

NIDRR Staff Perspectives on Grant Monitoring and Grantee Management

NIDRR project officers provide an important bridge between ED’s and NIDRR’s policies and procedures on grant management and the grantees’ management of their grants. They carry out the grant monitoring functions, described in the first section of this chapter, that are aimed at identifying issues related to performance and fiscal compliance with grant requirements.² NIDRR procedures emphasize that monitoring activities are conducted in working partnerships with grantees. Comments made by grantees cited above referred to some of the ways in which NIDRR’s procedures and processes assist grantees in managing their grants and influence the grant results (e.g., APRs help in keeping grantees focused on goals and products, quarterly reports serve as a quality assurance mechanism, and some project officers promote high standards and an emphasis on quality).

To learn more about this important interface between NIDRR’s grant monitoring processes and grantees’ management of their grants, 16 NIDRR project staff were interviewed in person and asked a series of open-ended questions about their activities and specific questions that related to grantee

_______________

² In addition to grant monitoring, project officers have duties related to peer review and priority setting (covered in other chapters of this report). Depending on their own areas of expertise, some also have responsibilities for coordinating activities for certain grant programs, such as the Field Initiated Project or Model System grants, and for developing special initiatives in such areas as knowledge translation. There are currently 14 project officer positions within NIDRR (National Institute on Disability and Rehabilitation Research, 2011), but 4 were vacant at the time of this writing. In fiscal year 2009, there were a total of 230 grants funded by NIDRR (U.S. Department of Education, 2011).

planning and budgeting processes designed to improve quality. Not all of the interviewees were project officers; some were in supervisory and other management roles. The discussion of these qualitative data is organized into issues and initiatives that could relate to the processes used by grantees to ensure quality.

Issues Related to Time

One issue that may be tangentially related to the amount and quality of grantee processes to promote high-quality outputs is the NIDRR staff time spent in monitoring activities. Staff reported a wide variation in the amount of time spent in monitoring grants—from 35 to 80 percent of their time. As stated above, NIDRR staff have other responsibilities to different degrees in addition to grant monitoring, which accounts for this wide variation. However, staff commented that the large workload of project officers (approximately 20 to 25 grants) often does not allow for as much attention to grantees as is needed. Also as reported in the NIDRR Fiscal Year 2011 Grant Monitoring Plan, NIDRR has seen a 25 percent reduction in staff over the last 6 years (National Institute on Disability and Rehabilitation Research, 2010), which would impact caseload size.

Issues Related to Type and Frequency of Communications with Grantees

Staff reported that the type and frequency of communications with grantees also vary, depending largely on grantee needs and issues related to performance and fiscal compliance. A variety of monitoring procedures are used, including periodic telephone conversations, e-mails that ask questions or provide consultation information, teleconferences, quarterly written reports, and the APR. It was commented that NIDRR’s forms and reporting requirements are not easy for grantees because of the complexity of federal rules, which staff often must spend time translating for new grantees. Staff reported that the structured postaward teleconference has been helpful in establishing expectations for grantee performance and preventing potential problems in complying with these federal rules. They commented that limited travel funds have not permitted sufficient on-site monitoring for grants that require higher levels of technical assistance. This observation is confirmed in the NIDRR Fiscal Year 2011 Grant Monitoring Plan (National Institute on Disability and Rehabilitation Research, 2010), which states that the number of on-site monitoring visits and formative reviews is affected by the budget.

Issues Related to Budget

One issue mentioned by several respondents was that grantees often underspend their budgets, especially during startup; this issue is not uncommon with research grants in general. This underspending extends their grants and changes the time frame for output development and project completion. One suggestion made was that staff might be able to identify potential problems and work more with grantees on this issue if they had project-level budget data in addition to grant-level data, especially for larger grants that have multiple projects starting and ending at different points. A related comment was that there is a need for improved ongoing communication between project officers and other NIDRR planning staff who have access to all of the relevant financial information. Another suggestion was that it would be helpful to have standard forms for requesting no-cost extensions.

Issues Related to Consistency

Although it was stated that the frequency of reporting depends on grantee needs, it was suggested that greater consistency is needed across project officers in the schedule for grantee reporting between APRs. It was also suggested that new project officers need more training to foster greater consistency in grant monitoring across grantees.

Initiatives

Staff mentioned that several initiatives are ongoing to improve and facilitate the grant monitoring process. A data group is working on developing new management reports on grant status to help project officers. Efforts are being made to help train new project officers by sharing experiences and providing advice informally. In addition, staff commented that parts of their regular staff meetings are being used to consult about grants that present special monitoring challenges.

Conclusions and Recommendations Related to Grant Management Processes

To address the key study question of what planning and budgetary processes are used by grantees to promote high-quality outputs, the committee considered these grantee-level processes in the larger context of NIDRR’s structure and processes that support grant management. With regard to the larger agency context, the NIDRR Fiscal Year 2011 Grant Monitoring Plan stemmed from initiatives within ED related to grant monitoring. In 2009, the U.S. Government Accountability Office (GAO) reported on a study of

grant monitoring within ED, which included OSERS. The report recommended that ED:

•   Develop department-wide guidance on risk assessment, continue efforts to develop new grantee risk assessment tools, … and work with the program offices to ensure these tools are implemented.

•   Implement a strategy to ensure each program office has staff with sufficient financial monitoring expertise to conduct or assist other program specialists in conducting financial compliance reviews …

•   Develop an easily accessible mechanism for sharing information across offices about grantees’ past and present performance, and an accessible forum for sharing promising practices in grant monitoring to ensure all program offices are able to effectively and efficiently perform all of their duties and responsibilities. (U.S. Government Accountability Office, 2009, p. 19)

In the context of this larger department initiative, NIDRR has developed new risk management assessment and monitoring procedures for grants at risk of failure to comply with program requirements, to reach performance goals, or to comply with grant administration and financial management requirements (National Institute on Disability and Rehabilitation Research, 2010). A follow-up interview with NIDRR executives in April 2011 showed that they are enthusiastically incorporating a risk management approach that will assist them in determining how much and what type of monitoring may be needed. To identify grants at risk, NIDRR is using an EDrecommended rubric with information tailored for NIDRR grantees and drawn from the peer review, the APR, and project monitors.

NIDRR appears to have a good plan and momentum in place for upgrading its routine monitoring and for identifying and monitoring grants that are at risk of noncompliance with the new monitoring plan. On the whole, it appears that grantees appreciate aspects of NIDRR’s grant management processes that facilitate their own grant management strategies. Grantees commented that NIDRR’s detailed proposal requirements facilitate their planning, that NIDRR builds output development into the planning by means of the postaward conference, that the APR helps grantees adhere to both the budget and the time frame for a project, that the quarterly reports facilitate quality assurance, and that NIDRR project officers with high standards and an emphasis on quality contribute to successful grant results. While grantees generally commented that NIDRR grant management processes were effective in helping them in their own grant management processes, they offered some suggestions for improvement.

Suggestions for improvement from NIDRR staff focused on strengthening their capacity to monitor grants and to help their grantees stay on course in implementing their grants and meeting performance expectations. Among other suggestions, they expressed the need for more workable grant case

loads, additional travel funds for on-site monitoring of grants that require higher levels of technical assistance, more training for new project officers to promote consistency and quality in their monitoring processes, and a freer flow of communication between project officers and NIDRR planning staff in relation to financial information.

Recommendation 5-1: NIDRR should continue to focus efforts on improving its grant monitoring procedures and specific elements of its overall grant management system that impact grantee-level planning, budgets, and the quality of outputs.

Particular emphasis should be placed on continuing to implement the new procedures for monitoring at-risk grants. Given the budgetary constraints on the number of on-site monitoring visits and formative reviews referred to earlier in the section on staff interviews, there will of course be challenges in this area. However, allocation of limited resources for effectively monitoring grantees’ planning, budgeting, and outcomes should be a critical consideration in overall resource allocation decisions.

Attention should also be given to grantee and staff suggestions for improving elements of NIDRR’s overall grant management system that could impact grantee planning and budgeting and the quality of outputs. In particular, NIDRR should consider the following:

•   how the timing of grant applications and notification impact the planning and quality of grant implementation and indirectly the quality of outputs subsequently produced;

•   methods NIDRR staff can use when monitoring large multiproject grant budgets to identify project-level variation that could be impacting the overall grant budget (e.g., personnel costs);

•   the need for improved communication between project officers and other NIDRR planning staff who have more access to financial information;

•   more focused reporting in the APR on new product development, stages of output development, and how the work of grants is bringing effective and practical new services and/or devices to the field or to the market;

•   methods for capturing information on outputs produced at the end, or shortly after the end, of grants so the quality and impact of these products can be assessed;

•   the need for more consistency across project officers in the schedule of grantee reporting between APRs; and

•   the extent to which new project officers are trained to promote consistency in expectations regarding standards of quality.

From the grantee questionnaires, the committee also learned that some grants that are developing innovations in new technologies may not fit well with a management template that calls for strict up-front planning and adherence to original designs and timetables. Similarly, grants funding large, multisite studies may require more or different supervision, monitoring, and technical assistance than more focused or limited studies. These grantees expressed the need for a greater degree of flexibility in management to allow them to stay on the cutting edge of technology or adapt more easily to changing needs of multisite research projects.

Recommendation 5-2: NIDRR should review the requirements placed upon technical innovation grants and large multisite studies to ensure that planning, reporting, supervisory, and technical assistance requirements fit their particular circumstances.

USE OF GRANT RESULTS/OUTPUTS TO INFORM FUTURE PROJECTS

In its logic model, NIDRR depicts its short-term outcomes as the array of outputs generated by grantees, which in turn are expected to inform and generate new projects (National Institute on Disability and Rehabilitation Research, 2006). The fifth main study question of this evaluation was: To what extent are the results of the reviewed research and development outputs used to inform new projects by both the grantee and NIDRR? To address this question, the committee gathered information from NIDRR management about how they use the results of their grantees’ research. The principal investigators of the grants reviewed for the summative evaluation provided supplemental information on this subject on the questionnaire they completed in submitting their outputs for review by the committee (see Appendix B).

The specific question asked of grantees was: Have the results of the research and development outputs from this grant, or your prior NIDRR grants, been used to inform the development of new grant applications or other kinds of projects? The objective was not to assess grantees’ individual or aggregate productivity, but to determine the extent to which NIDRR grants generate new projects and products. In this vein, the committee asked not only about respondents’ present grants but also about what was generated from prior NIDRR grants because of the length of time it takes, for example, to get technology products through various stages of development to commercialization or to develop collaborations that evolve into new projects. The information gathered from grantees is presented first.

Use of Research and Development Results/Outputs by Grantees

Twenty-eight of the 30 grantees that participated in the summative evaluation responded to this specific question. Of the 28 respondents, 24 indicated that the results of their current or prior NIDRR grants had been used to inform the development of new grant applications or other kinds of projects, funding opportunities, or collaborations. Table 5-1 summarizes the approximate number and type of new projects reported.

In some cases, grantees indicated that multiple grant applications or other project types were generated, without providing specific numbers. In these cases, a “1” was entered as the quantity so as not to overcount the number of projects generated.

As can be seen in the table, these 24 grants generated funding from more than 50 other sources for new projects stemming from the original grant. Nineteen of the new projects were NIDRR-funded, but 13 were funded by other federal agencies. In open-ended remarks, one principal investigator praised NIDRR for encouraging and funding innovative work that is often taken up and expanded by other federal research agencies. Annex 5-1 at the end of this chapter contains a table with more detail about the nature of the new projects generated from existing NIDRR grants. That table is organized by type of NIDRR program mechanism and separates the grants into rows within the program mechanisms. As an example, the table shows new projects that were generated from one Rehabilitation Research and Training Center grant and the various funding sources for those projects, which included NIDRR, the National Institute for Occupational Safety and Health (NIOSH), the Centers for Disease Control and Prevention (CDC), and a private foundation. Table 5-1 above also shows that these 24 grants generated more than 19 new collaborations. The table in Annex 5-1 shows examples of these, such as participants in a Burn Model System project being invited to participate in another university study, networking among NIDRR-funded centers, and collaboration with national-level organizations. Table 5-1 shows that another type of spin-off project involved applying specific outputs—data, instruments, or models—to other projects. Examples included transferring newly gained knowledge on stroke rehabilitation interventions to hip and knee replacement rehabilitation in different types of treatment settings, translating and testing surveys in other languages, and studying the efficacy and effectiveness of a telephone-based problem-solving treatment for a different population of service members after deployment. Commercialization of technology products is a concrete example of grant outputs that have generated new types of outputs and projects, illustrated by the evolution of assistive technology devices.

TABLE 5-1 Number^a and Type of New Projects Resulting from 24^b Grants

^aSome respondents indicated that “multiple” projects of a certain type were generated, rather than giving a specific listing or number. Therefore the totals given are approximate.

^bThe 4 other grantees (out of the 28 that responded to this question) indicated that their results were not used to inform the development of new grant applications and other kinds of projects. Two respondents stated that their grants had come to an end point because on one a long-standing question had been successfully addressed, and on the other that there was no opportunity to propose further work in his specific line of research as an independent researcher. A third respondent stated that the study was still in the grant cycle but was fully expected to inform the development of new grant applications and projects; the fourth offered no explanation.
SOURCE: Generated by the committee based on data from the grantee questionnaire.

Use of Research and Development Results by NIDRR

NIDRR management informed the committee during early interviews about how the agency uses the results of its grantees’ work. Results are used for performance monitoring, for priority setting, and for dissemination, all

of which in some way inform new projects, such as by shaping priorities and funding decisions. As stated in the first section of this chapter, NIDRR’s APR system collects routine information from grantees on their outputs and accomplishments. In addition to monitoring grantee performance, this information is used for NIDRR’s Government Performance and Results Act (GPRA) performance reporting and in annual reports to Congress through ED. These reports, among other considerations, may influence the funding of new projects.

In its annual priority-setting process, NIDRR considers the knowledge that has been gained from the products of its various program mechanisms as one key input. This information is obtained from NIDRR project officers, from the APR system, from state-of-the-science conferences organized by center grants, and from other grantee meetings during the year.

Knowledge dissemination and transfer is at the core of NIDRR’s logic model. Accordingly, the agency has incorporated requirements in grant applications for grantees to disseminate knowledge and information about outputs they have produced. Dissemination is accomplished in various ways, such as through publications, conferences, workshops, and websites.

Conclusions on the Use of Results and Outputs of NIDRR Grants

The key study questions for the external evaluation of NIDRR and its grantees were phrased in terms of the extent to which NIDRR’s key processes (i.e., priority setting, peer review, and grant management) are conducted in such a way as to enhance the quality of final results. Final results can be viewed in several different ways—as the quality of a research portfolio resulting from the priority-setting process; as the quality of grants resulting from peer review; or as the quality of outputs resulting from multiple influences, such as the scientific characteristics of the researchers and the key agency processes of priority setting, peer review, and grant management. This section examines final results in terms of NIDRR’s intermediate or short-term outcomes (i.e., the extent to which grants and their products generate new projects and/or are used to inform NIDRR priorities).

From the information gathered from grantees and presented at the beginning of this section, the committee found that the results of the reviewed research and development projects have been used to generate new projects by grantees to a great extent. That 24 grants generated twice as many new projects is evidence of “adoption and use of knowledge,” which is a core tenet of NIDRR’s mission.

Of course the projects that were generated vary in purpose, scope, and funding levels. For the present evaluation, the committee was able to take only a brief snapshot of this type of short-term outcome (i.e., generation of new projects). Grantees were asked to describe briefly in writing what

new projects or products were generated by their current grants after they had completed a long questionnaire asking for detailed information about the quality of their outputs. The volume and substance of the information they provided call for further exploration—for example, to examine more deeply the pathway and evolution of knowledge/product development for individual grants and for sets of grants in different NIDRR portfolios. Tracing the pathway of grants and outputs would be helpful in answering questions about NIDRR’s impact.

REFERENCES

National Institute on Disability and Rehabilitation Research. (2006). Department of Education:

National Institute on Disability and Rehabilitation Research—Notice of Final Long-Range Plan for Fiscal Years 2005–2009. Federal Register, 71(31), 8,166-8,200.

National Institute on Disability and Rehabilitation Research. (2009). Briefing book for The National Academies. Unpublished document. Washington, DC: National Institute on Disability and Rehabilitation Research.

National Institute on Disability and Rehabilitation Research. (2010). NIDRR fiscal year 2011 grant monitoring plan. Washington, DC: National Institute on Disability and Rehabilitation Research.

National Institute on Disability and Rehabilitation Research. (2011). NIDRR organizational chart. Unpublished document. Washington, DC: National Institute on Disability and Rehabilitation Research.

U.S. Department of Education. (2009). Handbook for the discretionary grant process. Washington, DC: U.S. Department of Education.

U.S. Department of Education, Office of Special Education and Rehabilitative Services, Rehabilitation Services Administration. (2011). Annual report on federal activities under the Rehabilitation Act, fiscal year 2011. Washington, DC: U.S. Department of Education.

U.S. Government Accountability Office. (2009). Department of Education could improve its processes with greater focus on assessing risks, acquiring financial skills, and sharing information. GAO-10-57. Washington, DC: U.S. Government Accountability Office.

ANNEX 5-1
SUMMARIES OF NEW GRANTS OR OTHER TYPES OF PROJECTS GENERATED BY NIDRR GRANTEES

Grantees were asked to describe briefly what new grant applications, other projects, funding opportunities, or collaborations have resulted from the grant that was reviewed by the committee or prior NIDRR grants. The following table corresponds to Table 5-1 in the main text of the chapter and provides additional detail about the new projects for each of the 24 grantees who provided this information. The table is organized by program mechanism and grantee. Under “New Grant,” “X” refers to funded grants; “X*” refers to grant applications/proposals that have not, or not as yet, been funded. An X or X* may refer to more than one grant.

TABLE A5-1

SOURCE: Generated by the committee based on data from the grantee questionnaire.