developed for selecting grantees who would be invited to participate in the evaluation. Fourth, the committee and staff worked with grantees who agreed to participate in the evaluation, and gathered and cataloged the outputs and supplemental information submitted for the committee’s review. Finally, the committee assessed the outputs through an expert review process.
Development of Quality Criteria
A key element of the summative evaluation was the response to NIDRR’s request to develop criteria for assessing the quality of its grantees’ outputs. In developing these criteria, the committee drew on its own research expertise, recommendations of the external advisory group convened by NIDRR in the course of planning this evaluation (National Institute on Disability and Rehabilitation Research, 2008), and methods used in other National Research Council (NRC) and international studies that have evaluated federal research programs (Bernstein et al., 2007; Chien et al., 2009; Ismail et al., 2010; National Research Council and Institute of Medicine, 2007; Panel on Return on Investment in Health Research, 2009; Wooding and Starkey, 2010; Wooding et al., 2009).
Four criteria were developed for the evaluation of grantee outputs: (1) technical quality, (2) advancement of knowledge or the field, (3) likely or demonstrated impact, and (4) dissemination.
Technical quality The technical quality of outputs was assessed using dimensions that included the application of standards of science and technology, appropriate methodology (quantitative or qualitative design and analysis), and the degree of accessibility and usability.
Advancement of knowledge or the field (e.g., research, practice, or policy as relevant) The dimensions used to assess this criterion included scientific advancement of methods, tools, and theory; the development of new information or technologies; closing of an identified gap; and use of methods and approaches that were innovative or novel.
Likely or demonstrated impact This criterion was used to assess the likely or demonstrated impact of outputs on science (impact, citations), consumers (health, quality of life, and participation for people with disabilities), provider practice, health and social systems, social and health policy, or the private sector/commercialization.