data systems to track the results of those measures, and developing internal management systems to facilitate quality research. In 2009, NIDRR requested that the National Research Council form a committee to conduct a “process evaluation” of aspects of its grantmaking and a “summative evaluation” of the quality of grantee outputs. In addition, the committee was charged with assessing the methods it developed for conducting the summative evaluation and making recommendations for the conduct of future evaluations.1 The requested study was the most recent effort in a series of NIDRR-funded activities aimed at assessing and improving the agency’s performance.

PROCESS EVALUATION

NIDRR posed three questions specific to the process evaluation aimed at assessing the process used for priority writing, practices for peer review of grant applications, and the planning and budgetary processes used by grantees. The development of priorities determines the areas of emphasis for research and the specific topics to be targeted by potential applicants, while peer review is a fundamental component of the grant selection process. Although it is not possible to establish a clear causal link, these NIDRR processes, as well as planning and budgetary processes used by grantees, can influence the quality of the work produced by grantees.

To address these questions, the committee reviewed existing documents (e.g., legislation, Federal Register notices, NIDRR and ED policies and procedures) and interviewed NIDRR management to obtain a more thorough and cohesive understanding of these processes. The committee gained additional insight into NIDRR’s peer review process by listening to teleconferences held by three panels as they conducted their reviews of different grant competitions. In addition, the committee collected original data through surveys of NIDRR staff, stakeholder organizations (other federal agencies, professional associations, and advocacy organizations), NIDRR peer reviewers, and principal investigators of NIDRR grants.

Priority Setting

To what extent is NIDRR’s priority-writing process conducted in such a way as to enhance the quality of the final results?

As used in the study question, the term “priority-writing process” encompasses many aspects of priority setting, including gathering input from

_______________

1 This aspect of the committee’s charge was summarized in a letter report provided to NIDRR in July 2011 and is also addressed in Chapter 6 of this report.



The National Academies | 500 Fifth St. N.W. | Washington, D.C. 20001
Copyright © National Academy of Sciences. All rights reserved.
Terms of Use and Privacy Statement