2

Evaluation Scope and Methods

This chapter begins by explaining the scope of the committee’s evaluation. It then describes the methods used for the evaluation. Both the scope of the evaluation and potential limitations of its findings are discussed to clarify the extent to which the findings can be generalized and used by the National Institute on Disability and Rehabilitation Research (NIDRR) to enhance its priority-setting, peer review, and grant management processes.

SCOPE OF THE EVALUATION

This section explains the parameters of the committee’s evaluation through a conceptual framework that guided the evaluation. It also defines “quality” as operationalized for this study.

Conceptual Framework

In designing the evaluation, the committee used NIDRR’s published logic model (National Institute on Disability and Rehabilitation Research, 2006) as a starting point. In this model, NIDRR’s investments in grants aimed at capacity building, research and development, and knowledge translation are intended to produce discoveries; theories, measures, and methods; and interventions, products, devices, and environmental adaptations (i.e., short-term outcomes). These outputs should promote the adoption and use of knowledge leading to changes in policy, practice, behavior, and system capacity (i.e., intermediate outcomes) for the ultimate benefit of persons with disabilities in the domains of employment, participation and



The National Academies | 500 Fifth St. N.W. | Washington, D.C. 20001
Copyright © National Academy of Sciences. All rights reserved.
Terms of Use and Privacy Statement



Below are the first 10 and last 10 pages of uncorrected machine-read text (when available) of this chapter, followed by the top 30 algorithmically extracted key phrases from the chapter as a whole.
Intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text on the opening pages of each chapter. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.

Do not use for reproduction, copying, pasting, or reading; exclusively for search engines.

OCR for page 34
2 Evaluation Scope and Methods This chapter begins by explaining the scope of the committee’s evalua- tion. It then describes the methods used for the evaluation. Both the scope of the evaluation and potential limitations of its findings are discussed to clarify the extent to which the findings can be generalized and used by the National Institute on Disability and Rehabilitation Research (NIDRR) to enhance its priority-setting, peer review, and grant management processes. SCOPE OF THE EVALUATION This section explains the parameters of the committee’s evaluation through a conceptual framework that guided the evaluation. It also defines “quality” as operationalized for this study. Conceptual Framework In designing the evaluation, the committee used NIDRR’s published logic model (National Institute on Disability and Rehabilitation Research, 2006) as a starting point. In this model, NIDRR’s investments in grants aimed at capacity building, research and development, and knowledge translation are intended to produce discoveries; theories, measures, and methods; and interventions, products, devices, and environmental adapta- tions (i.e., short-term outcomes). These outputs should promote the adop- tion and use of knowledge leading to changes in policy, practice, behavior, and system capacity (i.e., intermediate outcomes) for the ultimate benefit of persons with disabilities in the domains of employment, participation and 34

OCR for page 34
35 EVALUATION SCOPE AND METHODS community living, and health and function (i.e., long-term outcome arenas). NIDRR holds itself accountable primarily for the generation of knowledge in the short-term outcome arena, and it is this arena that was the focus of the committee’s external evaluation. The committee examined how NIDRR’s grant funding is prioritized for these investment areas, the processes used for reviewing and selecting grants, and the quality of the research and development outputs, as depicted in the conceptual framework in Figure 2-1. The committee developed this framework to guide the evaluation effort. The boxes labeled Q1 to Q5 (i.e., NIDRR’s process and summative evaluation questions 1 to 5; see Chapter 1), were the direct foci of the evaluation. The figure also includes other inputs, contextual factors, and implementation considerations as they are likely to influence the processes and short-term outcomes. The figure shows that the measurable elements of the short-term outcomes are what NIDRR considers to be the array of grant outputs (Q4) generated by grantees, which are expected to inform and generate new projects (Q5). Also shown are the expected long-term outcomes, which include an expanded knowledge base; improved programs and policy; and reduced disparities for people with disabilities in employment, participation and community living, and health and function. However, these long-term outcomes were beyond the scope of the committee’s evaluation. In summary, the scope of the evaluation encompassed key NIDRR processes of priority setting, peer review, and grant management (process evaluation) and the quality of grantee outputs (summative evaluation). It is important to point out that the scope of the summative evaluation did not include a larger explicit focus on assessing the overall performance of indi- vidual grants or NIDRR portfolios (e.g., Did grants achieve their proposed objectives? Did the various research and development portfolios operate as intended to produce the expected results?). Although capacity building is a major thrust of NIDRR’s center and training grants, the present evaluation did not include assessment of outputs related to capacity building (e.g., num- ber of trainees moving into research positions), which would have required methods different from those used for this study. Definition of “Quality” The evaluation focused on the quality of NIDRR’s priority-setting, peer review, and grant management processes and on the quality of the outputs generated by grants. A review of the literature on evaluation of federal research programs reveals that the term “quality” is operationalized in a variety of ways. For example, the National Research Council (NRC) and Institute of Medicine (IOM) (2007) developed a framework and suggested

OCR for page 34
Inputs Processes Outcomes 36 Q1: NIDRR PRIORITY SETTING Short-term Long-term NIDRR policies Long-range planning to identify priorities and procedures Stakeholder engagement New theory, Q4: GRANTEE OUTPUTS NIDRR methods, and Translation of priorities into funding Publications resources knowledge RFA writing and launching (dollars, staff, Tools, measures, and Improved management Publicizing competitions intervention protocols practice, contractor) programs, policy Technology products and Pool of potential devices Q2: NIDRR PEER REVIEW grantees Informational products Selection of peer reviewers Political support Competition process within Dept. of Peer review process and ranking Education Reduced Q5: INFORMING AND Legislative disparities GENERATING NEW framework Q3: PROJECT MANAGEMENT between people PROJECTS with disabilities NIDRR post award administration, project Public Evidence of movement of management, and reporting and general comments outputs along the R&D population in Grantee project management and reporting to pathway (e.g., theory to employment, Results of prior NIDRR measure, prototype testing to health and NIDRR market) functioning, assessments Evidence of outgrowths of GRANTEES participation first generation outputs (e.g., Standards and and community Implementation of R&D, capacity building, intervention found to be best practices living and knowledge translation grants effective with one population being adapted for another) Output development and testing Contextual Factors: Changing public policies, coordination, and cooperation with other government entities, etc. FIGURE 2-1 Conceptual framework for the committee’s external evaluation of NIDRR and its grantees.

OCR for page 34
37 EVALUATION SCOPE AND METHODS measures for assessing multiple research programs across the National Institute for Occupational Safety and Health. That approach refers to vari- ous quality-related criteria for assessing “outputs” and “intermediate out- comes.” In another example, within a return-on-investment framework (Panel on Return on Investment in Health Research, 2009), the Canadian Academy of Health Sciences developed a menu of indicators that can be used for evaluating research programs. “Quality criteria” were included as an indicator of “advancing knowledge,” along with “activity” (also called “outputs”) and “outreach” (also called “transfer”). A third example is a framework and web-based survey approach developed by RAND (Ismail et al., 2010; Wooding and Starkey, 2010; Wooding et al., 2009) for measur- ing outputs and impacts of research programs for the Arthritis Research Campaign (UK), which was also applied to the Medical Research Council (UK) and the Canadian Institute of Health Research-Institute of Muscu- loskeletal Health and Arthritis. The survey contains a series of questions for grantees that are focused on stages of research and development and possible impacts. A final example is an approach, developed in Taiwan for assessing a wide range of the country’s federal research programs, that refers to quality-related criteria as “performance attributes” in the areas of aca- demic achievement, technology output, knowledge services, and technology diffusion (Chien et al., 2009). Annex 2-1 at the end of this chapter shows the various quality criteria and dimensions used across these studies, as well as those compiled by an external advisory group convened by NIDRR in August 2008 to assist the agency in laying the groundwork for the current External Evaluation of NIDRR and its Grantees. Referring specifically to the four output categories used by NIDRR (i.e., publications, tools, technology products, and infor- mation products), the advisory group provided responses to the following questions: What criteria could be used by an external peer review panel to rate the quality of NIDRR grantee research outputs? What are some of the dimensions of quality? The first column in the table in Annex 2-1 summa- rizes the advisory group’s suggested criteria and dimensions for assessing the quality of NIDRR outputs and relates these to the criteria used in the other studies referred to above. The list of criteria is intended to be exhaustive to illustrate the types of criteria and dimensions that have been used in U.S. and international studies of federal research programs and that the committee drew on in developing the criteria used in this evaluation. Most of these criteria and dimensions were incorporated into the summative evaluation, as described in Chapter 6 of the report and as can be seen in the output quality rating form used for the evaluation (see Appendix B). In keeping with the literature and other approaches to evaluating fed- eral research programs, the committee used a broad concept of “quality”

OCR for page 34
38 REVIEW OF DISABILITY AND REHABILITATION RESEARCH encompassing attributes that lead to the selection of grants and eventual grant results that (1) meet technical and professional standards of science and technology; (2) will advance the knowledge base or the field of research, policy, or practice; (3) are likely to have or have had demonstrated impacts on science, consumers, provider practice, health and social systems, social and health policy, and the private sector/commercialization; and (4) are disseminated according to principles of appropriate knowledge translation. METHODS The committee used a cross-sectional design that incorporated both quantitative and qualitative methods to address the two sets of questions in the process and summative evaluation phases of the study. The process evaluation phase involved reviewing existing documentation and collect- ing testimonial data to examine how NIDRR, through its policies and procedures and in practice, develops its research and funding priorities, reviews and evaluates submitted proposals, makes decisions and awards grants based on these reviews, and manages grant-supported activities. The summative evaluation phase involved the use of expert panels to assess the quality of grant outputs. The following sections present the study methods that were used to address the two sets of questions for these two phases. Data collection took place between July 2010 and February 2011. Sources of Information, Data Collection, and Analysis Process Evaluation To address questions related to NIDRR’s priority-setting, peer re- view, and grant management processes, the committee reviewed existing documentation (e.g., legislation, Federal Register notices, NIDRR and U.S. Department of Education [ED] policies and procedures) and interviewed NIDRR and ED management to obtain a more thorough and cohesive understanding of these processes.1 The committee gained additional insight into NIDRR’s peer review process by listening to teleconferences held by three panels as they conducted their reviews of different competitions. The committee also collected original data from the following key informant groups. NIDRR staff and contractors NIDRR management provided the commit- tee with a list of the agency’s administrative and program management staff who had sufficient knowledge, experience, and responsibilities in the 1 The committee conducted interviews with NIDRR and ED management in four sessions during summer 2010 and one session in spring 2011.

OCR for page 34
39 EVALUATION SCOPE AND METHODS priority-setting, peer review, and grant management processes to respond to the committee’s interview questions. The committee interviewed 16 NIDRR staff from this list in person to gather information on their roles in and perspectives on these processes. Two-thirds of the interviewees were project officers or direct supervisors of project officers; the remaining held administrative positions. The interview questions were open-ended, and the interviews were recorded and transcribed. The committee also interviewed by telephone a manager from a NIDRR contractor that assists the agency with the logistics of convening peer review panels to obtain the contractor’s perspective on the peer review process. NIDRR stakeholder organizations The committee obtained a list of 130 organizations that operate in NIDRR’s arena from (1) NIDRR, which provided the names of professional and advocacy organizations the agency considers to be stakeholders, and (2) the Interagency Committee on Disabil- ity Research (ICDR), which provided the names of federal agencies that are statutory members of the ICDR or nonstatutory members that have partici- pated in ICDR special committees. The committee sent all 130 organizations invitations to participate in an online survey; the invitations were addressed specifically to the executive directors of the professional and advocacy orga- nizations and to the named representatives of the federal agencies. Invited respondents were asked to either complete the survey themselves or forward it to a member of their organization who would be knowledgeable about the organization’s relationships with NIDRR. The invitations were sent in an e-mail letter that provided a link to the online survey and a password for logging on to the secure website. If respondents were unable to access the survey online or preferred another method, the committee offered to send them the survey in hard-copy form or to conduct it by telephone. The survey contained a series of closed- and open-ended questions inquiring about the organizations’ role in the NIDRR planning and priority-setting process, respondents’ perspectives on the process, benefits their agencies derived from NIDRR grants or outputs, and suggestions for enhancements to the priority-setting process. Of the 130 organizations, 72 responded to the survey (a response rate of 55 percent). According to Baruch and Holtom (2008), who examined 175 organizational studies, 55 percent is an accept- able response rate for a survey targeting executive directors. NIDRR peer reviewers The committee sent invitations to complete a survey to all individuals (a total of 156) who served on NIDRR peer review panels during fiscal year 2008-2009. The invitations were sent in an e-mail letter that provided a link to the online survey and a password for logging on to the secure website. If respondents were unable to access the survey online or preferred another method, the committee offered to send it to them in hard-

OCR for page 34
40 REVIEW OF DISABILITY AND REHABILITATION RESEARCH copy form or to conduct the survey by telephone. The survey contained a series of closed- and open-ended questions inquiring about their experiences with and perspectives on the NIDRR peer review process, how it compares with the peer review processes of other federal research agencies, and sug- gestions for enhancements to the process. Four potential respondents were deleted from the list because their e-mail addresses were invalid even after a concentrated search. Of the 152 individuals successfully invited, 121 re- sponded to the questionnaire (a response rate of 80 percent). Principal investigators (PIs) of NIDRR grants The committee invited 30 PIs whose grants were randomly selected for review in the summative phase of the evaluation (see the section below on methods for the summative evalua- tion) to respond in writing to a special set of questions focused on NIDRR’s priority-setting, peer review, and grant management processes. One set of questions focused specifically on NIDRR’s third major study question, which related to planning and budgetary processes used by grantees to pro- mote high-quality outputs. Twenty-eight of the 30 grantees opted to respond to this set of process questions. Analysis of Process Data The committee analyzed quantitative data from the online surveys of stakeholders and peer reviewers descriptively to examine frequencies and measures of central distribution. For process data gathered from NIDRR staff, grantees, stakeholder organizations, and peer reviewers (i.e., responses to open-ended questions that were based on individuals’ opinions or perspectives), standard qualitative analysis techniques (Miles and Huberman, 1994; Patton, 2002) were used. These techniques involved a three-phase process of coding the data to identify common topics, categories, and themes. The first phase in the iterative process of coding each data source involved an initial examination or “read-through” of the complete data set. The initial examination resulted in a preliminary list of topic codes, which were then used to code the data. The second phase of the process involved reviewing the coded data in order to refine and final- ize the list of codes. During this effort, analysts generally combined two or more codes into one of the existing codes or into a new overarching code. Multiple variations were attempted before a final list of codes was determined and a final coding of the data was completed. The final phase of the process involved reviewing the finalized coded data and drawing out categories and themes. While researchers sought to perform each analytic phase similarly for each source of data, differences in the nature and volume of data from the different sources necessitated some variation:

OCR for page 34
41 EVALUATION SCOPE AND METHODS • G rantees, peer reviewers, and stakeholders—Open-ended data were collected from these sources in the form of their written answers to specific questions. The coding of these data was done initially by question, and codes were then compared and combined across the questions. • N IDRR staff—Qualitative data were collected from NIDRR staff through personal interviews that were audio recorded and tran- scribed. The questions concerning NIDRR processes of priority setting, peer review, and grant management were phrased in general terms, and respondents were then prompted to provide additional details or clarification. Preliminary coding was done on the first five transcribed interviews, then refined as additional interviews were analyzed. Code lists were significantly refined through the iterative process described above. It is important to recognize the limitations of qualitative analyses of responses elicited to open-ended survey questions. First, the data set does not generally represent the viewpoints of all respondents because it is com- mon for 15 to 35 percent not to respond to the open-ended questions (Ulin et al., 2005). Second, among those who do respond, a varying number of the responses either are not written in a coherent manner; do not represent complete thoughts; or are vague generalities, lacking detail or specificity. Third, it is common for respondents with critical comments or sugges- tions to respond more often than those with neutral or positive comments (Gendall et al., 1996). Finally, Miles and Huberman (1994) encourage using counts of the number of times that certain codes or topics are observed in the data because these counts come into play when describing results, such as the frequency or consistency of observations. In the NIDRR staff (N = 16 respondents) and grantee (N = 28 respondents) data sets collected for this evaluation, counts were used in a highly limited manner because the frequencies were very low in these small data sets. Where data sets were larger (stakeholders = 72 respondents and peer reviewers = 121), counts were used in reporting results of the qualitative analyses for greater transparency, but the commit- tee acknowledges that in most cases, the number of specific observations for certain topics also is quite low. Despite these limitations, the committee believes that the collected data and qualitative analyses add background, context, and insight to many issues raised by respondents. In addition, they can lend support to this report’s conclusions and recommendations when similar issues emerge across respondent groups or across both qualitative and quantitative data sources.

OCR for page 34
42 REVIEW OF DISABILITY AND REHABILITATION RESEARCH Summative Evaluation The summative study questions focused on the quality of outputs gener- ated by NIDRR grantees and the potential for the outputs to lead to further research and development. Chapter 6, which summarizes the results of the summative evaluation, also describes in detail the sampling, measurement, and data collection methods used to conduct the assessment of outputs. Therefore, these methods are described only briefly here. The committee and NRC staff sampled 30 grants from NIDRR’s port- folio, and the committee as a panel of experts reviewed outputs of these 30 grants. These grants were drawn from nine of NIDRR’s program mecha- nisms: Burn Model System, Traumatic Brain Injury Model System, Spinal Cord Injury Model System, Rehabilitation Research and Training Center, Rehabilitation Engineering Research Center, Disability and Rehabilitation Research Project, Field Initiated Project, Small Business Innovation Research II, and Switzer Fellowship. The primary focus of the committee’s summa- tive evaluation was on assessing the quality of research and development outputs produced by grantees. The review focused on four different types of outputs, as defined in NIDRR’s Annual Performance Report: (1) publi- cations (e.g., research reports and other publications in peer-reviewed and nonpeer-reviewed journals); (2) tools, measures, and intervention protocols (e.g., instruments or processes created to acquire quantitative or qualitative information, knowledge, or data on a specific disability or rehabilitation is- sue or to provide a rehabilitative intervention); (3) technology products and devices (e.g., industry standards/guidelines, software/netware, inventions, patents/licenses/patent disclosures, working prototypes, product(s) evalu- ated or field tested, product(s) transferred to industry for potential commer- cialization, product(s) in the marketplace); and (4) informational products (e.g., training manuals or curricula; fact sheets; newsletters; audiovisual materials; marketing tools; educational aids; websites or other Internet sites produced in conjunction with research and development, training, dissemi- nation, knowledge translation, and/or consumer involvement activities). In assessing the quality of outputs, the committee used the following four criteria, stemming from its definition of quality (as discussed earlier): (1) technical quality, (2) advancement of knowledge or the field, (3) likely or demonstrated impact (on science, consumers, provider practice, health and social systems, social and health policy, and the private sector/commer- cialization), and (4) dissemination according to principles of appropriate knowledge translation. The committee analyzed data from the summative evaluation using frequency distributions and reported ratings of the quality of outputs in the aggregate by quality criteria assessed. Table 2-1 summarizes the data collection and measurement methods described above.

OCR for page 34
TABLE 2-1 Summary of Data Collection and Measurement Methods Study Data Collection Question Measures/Instruments Data Sources Methods Process Evaluation (July to September 2010) 1. To what extent is NIDRR’s • xisting data: E NIDRR and Federal Extraction of priority-writing process – Rehabilitation Act of 1973, as amended, and other Register relevant detail from conducted in such a way as related regulations (Education Department General documentation to enhance the quality of the Administrative Regulations, Department of Education final results? Handbook for the Discretionary Grant Process) – NIDRR briefing documents (descriptive and procedural) – Current and proposed NIDRR Long-Range Plans – Notices inviting applications (NIAs) and application kits by program mechanism (2009 and 2010) (relevant to funded priorities) – NIDRR assessments: U.S. Office of Management and Budget Performance Assessment Rating Tool (2005); Annual Performance Assessment Expert Panel Reviews (pilot reports for 2006, 2007) – NIDRR External Advisory Group report • uestions for process evaluation interviews tailored for Q NIDRR personnel, Semistructured face-to- NIDRR management and staff stakeholder face interviews • eb-based survey (form tailored for NIDRR W organizations stakeholders) Web-based survey (self-administered) 43 continued

OCR for page 34
44 TABLE 2-1 Continued Study Data Collection Question Measures/Instruments Data Sources Methods 2. To what extent are peer • xisting data: As above for question #1, plus: E NIDRR and Federal Extraction of reviews of grant applications – NIDRR descriptive and procedural documents on Register relevant detail from done in such a way as to peer review: Application Technical Review Plan; peer documentation enhance the quality of the reviewer instructions final results? – NIAs and application kits by program mechanism (2009 and 2010) (relevant to criteria for rating grants) – Analysis of peer review tracking database (Synergy, 2008) • uestions for process evaluation interviews tailored for Q NIDRR personnel, Semistructured face-to- NIDRR management and staff peer reviewers face interviews • eb-based survey (form tailored for peer reviewers) W Web-based survey (self-administered) 3. What planning and budgetary • uestions for process evaluation interviews tailored for Q NIDRR personnel, Semistructured face-to- processes does the grantee NIDRR management and staff grantees (principal face interviews use to promote high-quality • rantee questionnaire G investigators [PIs]) outputs? Self-administered questionnaire

OCR for page 34
Summative Evaluation (October 2010 to January 2011) 4. To what extent are the final • xisting data: NIDRR Annual Performance Report E NIDRR Extraction of data from outputs from NIDRR grants database to describe and summarize the outputs APR of high quality? • rantee outputs and supplemental questionnaire with G Grantees (PIs) Self-administered items classifying and describing the types of outputs, questionnaire features of the outputs that reflect the quality criteria, and future research and development as an outgrowth • uality of outputs: committee member rating sheet, Q Committee’s peer Peer review of outputs based on criteria such as: review scores and other materials – Technical quality (APR, grantee – Advancement of knowledge or the field questionnaire) – Likely or demonstrated impact – Dissemination 5. To what extent are the results • uestions for process evaluation interviews Q NIDRR personnel Semistructured face-to- of the reviewed research and • rantee questionnaire: same questionnaire as in #4 G face interviews development outputs used to above for items related to the generation of new research inform new projects by both and outputs the grantee and NIDRR? Grantees (PIs) Self-administered questionnaire 45

OCR for page 34
46 REVIEW OF DISABILITY AND REHABILITATION RESEARCH Potential Limitations Although the committee used the most rigorous methods possible to conduct its process and summative evaluations, the evaluation results may have been affected by a number of potential limitations. First, based on the study scope as described above, the committee was limited to directly evaluating only grant outputs. Evaluation of grants was performed only as a second step through synthesis of the results of the output evaluation. Addi- tionally, several grant program mechanisms were not evaluated because the timing of their funding cycles did not accord with the timing of this study. Second, measurement validity is concerned with the degree to which the study indicators accurately portray the concept of interest (Newcomer, 2011). For the process evaluation, the committee gathered information from different sources (existing documentation, interviews, observation, surveys), but the interviews relied on the accuracy of the memories and perceptions of individuals, which could be susceptible to recall or social desirability biases. For example, the NIDRR staff who were interviewed may have felt pressed to provide positive input on the processes being reviewed. However, NRC staff who conducted the interviews believe all NIDRR staff members were candid in their comments. To assess the quality of outputs for the summative evaluation, the com- mittee used sound criteria that were based on the cumulative literature re- viewed and its members’ own research expertise in diverse areas of disability and rehabilitation research, medicine, engineering, and the social sciences, as well as their expertise in evaluation, economics, knowledge translation, and policy. However, the accuracy of the committee’s assessment of the quality of outputs could have been affected by a number of factors. For example, the committee’s combined expertise did not include every possible content area in such a broad field as disability and rehabilitation research. Because of the diversity of the field, the grants and outputs were extremely varied, so apply- ing one set of criteria, even though broad and flexible, could not guarantee accurate applicability to every output. For example, websites, conferences, training curricula, therapeutic interventions, and educational outreach ser- vices ideally would require additional evaluation methods tailored to those types of outputs. The limitations and challenges encountered in conducting the output assessment are discussed in more detail in Chapter 6. Third, measurement reliability is concerned with the extent to which a measurement can be expected to produce similar results on repeated obser- vations of the same condition or event (Newcomer, 2011). The expert review methods used to assess the quality of grantee outputs could pose a threat to measurement reliability in that they relied on subjective assessments of dif- ferent expert reviewers. To address this limitation, the committee members frequently discussed how they were applying the criteria and interpreting the anchors of the rating scale so they could calibrate their ratings. They rated

OCR for page 34
47 EVALUATION SCOPE AND METHODS the outputs independently and then discussed their results and determined overall ratings that reflected consensus scores. Results of an interrater reli- ability analysis are presented in Chapter 6. Fourth, with regard to the process evaluation, it is possible that the respondents choosing to respond to the online surveys of stakeholders and peer reviewers may have differed from those who declined to participate. However, the response rates were respectable on both surveys (80 percent on the peer reviewer survey and 55 percent on the stakeholder survey), and the results of those two surveys also appeared to be balanced and not biased. Finally, results of the summative evaluation cannot be generalized be- cause of the small sample size and the small number of outputs reviewed from each grant. A total of 30 grants were reviewed across nine program mechanisms from a pool of 111 grants. Another threat to the generalizabil- ity of the findings stems from the fact that most of the grants reviewed ended in 2009. Because of the length of time it takes to publish research articles, grantees may have been unable to share their most important work with the committee. Other potential biases in the summative evaluation methods are described in Chapter 6. Review of the Evaluation Plan Before the committee implemented its evaluation plan, the plan was reviewed by leading experts in the field who provided suggestions for strengthening the methods to be used. In addition, the evaluation plan was reviewed and approved by the Institutional Review Board of the National Academies, under the category of Expedited Review, as meeting all criteria related to data confidentiality, security, and final disposition; informed con- sent; and potential risks and benefits to human subjects. CONCLUSION In conclusion, the committee used a conceptual framework (see Figure 2-1) developed around NIDRR’s study questions and a definition of quality drawn from the literature as the foundation for its evaluation. The study was conducted with a cross-sectional design using quantitative and qualitative methods. The process evaluation included a review of existing documenta- tion, interviews, online surveys, and written questionnaires. The summa- tive evaluation included an expert panel review of outputs from randomly selected grantees. While the nature of the evaluation itself and the methods used suggest several potential limitations to the study findings, the commit- tee strove to address these limitations where possible and acknowledge cases in which doing so was not possible.

OCR for page 34
48 REVIEW OF DISABILITY AND REHABILITATION RESEARCH REFERENCES Baruch, Y., and Holtom, B.C. (2008). Survey levels and trends in organizational research. Human Relations, 61(8), 1139-1160. Chien, C.F., Chen, C.P., and Chen, C.H. (2009). Designing performance indices and a novel mechanism for evaluating government R&D projects. Journal of Quality, 19(2), 119-135. Gendall, P., Menelaou, H., and Brennan, M. (1996). Open-ended questions: Some implications for mail survey research. Marketing Bulletin, 7, 1-8. Ismail, S., Tiessen, J., and Wooding, S. (2010). Strengthening research portfolio evaluation at the Medical Research Council: Developing a survey for the collection of information about research outputs. Santa Monica, CA: RAND Corporation. Miles, M.B., and Huberman, A.M. (1994). Qualitative data analysis (2nd ed.). Thousand Oaks, CA: Sage. National Institute on Disability and Rehabilitation Research. (2006). Department of Educa- tion: National Institute on Disability and Rehabilitation Research—Notice of Final Long-Range Plan for Fiscal Years 2005–2009. Federal Register, 71(31), 8,166-8,200. National Institute on Disability and Rehabilitation Research. (2008, August). Meeting report of the NIDRR External Evaluation Advisory Group. Washington, DC: National Institute on Disability and Rehabilitation Research. National Research Council and Institute of Medicine. (2007). Mining safety and health re- search at NIOSH: Reviews of research programs of the National Institute for Occupa- tional Safety and Health. Committee to Review the NIOSH Mining Safety and Research Program. Committee on Earth Resources and Board on Earth Sciences and Resources. Washington, DC: The National Academies Press. Newcomer, K.E. (2011). Strategies to help strengthen validity and reliability of data. Unpub- lished presentation. George Washington University, Washington, DC Panel on Return on Investment in Health Research. (2009). Appendices—Making an impact: A preferred framework and indicators to measure returns on investment in health research. Ottawa, Canada: Canadian Academy of Health Sciences. Patton, M.Q. (2002). Qualitative research & evaluation methods. Thousand Oaks, CA: Sage. Synergy Enterprises, Inc. (2008). Draft task 4 analysis. Washington, DC: National Institute on Disability and Rehabilitation. Ulin, P., Robinson, E., and Tolley, E. (2005). Qualitative methods in public health: A field guide for applied research. San Francisco: Jossey-Bass. Wooding, S., and Starkey, T. (2010). Piloting the RAND/ARC Impact Scoring System (RAISS) tool in the Canadian context. Cambridge, England: RAND Europe. Wooding, S., Nason, E., Starkey, T., Haney, S., and Grant, J. (2009). Mapping the impact: Exploring the payback of arthritis research. Santa Monica, CA: RAND Corporation.

OCR for page 34
49 EVALUATION SCOPE AND METHODS ANNEX 2-1 SUMMARY OF QUALITY CRITERIA AND DIMENSIONS

OCR for page 34
TABLE A2-1 Summary of Quality Criteria and Dimensions 50 National Canadian Institute for Academy Occupational of Health NIDRR Safety and Sciences External Health (CAHS) Advisory (NIOSH) Indicators RAND (UK, Chien et al. Criteria/Dimensions Group (U.S.) (U.S.) (Canada) Canada) (Taiwan) Stage of Development of the Research and of the Output x (depending on type of research) • esearch (e.g., basic, clinical services, health services R delivery, output development) • utput: What stage of output development was carried O out during the grant? (e.g., prototype development, initial testing, regulatory approval, on the market, trialing in a new context) • ovement along research and/or development pathway M Peer Recognition of Output • umber of peer-reviewed articles (e.g., using such N x x x x x measures as “journal impact factor” for assessing relative quality, or counting publications in high-quality journals/publishers) —Number of original research articles x —Number of review articles x • umber of research reports N x x x x • umber of conference papers published N x x x • umber of citations in peer-reviewed articles (e.g., N x x x x using such measures as “crown indicator,” “highly cited publication counts,” “H-index”) • itations assimilated into clinical guidelines C x

OCR for page 34
• uantity of citations indicates “breakthrough” results Q x x • easures of esteem (number of awards and honors M x x x received) • umber of patents awarded N x x x x • umber of Web downloads or hits N x x x • umber of reprints or books sold N x Multiple and Interdisciplinary Audience Involvement in Output Development • ultiple consumer input/collaboration in all M x x stages (other researchers, clinicians, policy makers, manufacturers, intellectual property brokers) and evidence of their involvement • ppropriate targeting of audience A x Output Meets Acceptable Standards of Science and Technology • ppropriate methodology for answering research A x questions • utput is well defined and implemented with integrity O x • ell-documented conclusions, supported by the W x x literature • bility for the document to stand alone A x • eets human subjects protection requirements M x • eets ethical standards M x • pplies concepts of universal design and accessibility A x • esearch done on the need for and development of the R x instrument or device • easurement instruments tested and found to have M x acceptable type and level of validity • easurement instruments tested and found to have M x acceptable level of reliability 51 continued

OCR for page 34
TABLE A2-1 Continued 52 National Canadian Institute for Academy Occupational of Health NIDRR Safety and Sciences External Health (CAHS) Advisory (NIOSH) Indicators RAND (UK, Chien et al. Criteria/Dimensions Group (U.S.) (U.S.) (Canada) Canada) (Taiwan) • ilot testing conducted with consumers and adaptations P x made as indicated • heoretical underpinnings and evidence base (e.g., level T x of evidence per American Psychological Association, American Medical Association) used and well documented Output Has Potential to Improve Lives of People with Disabilities • alued by consumers V x • vidence of beneficial outcomes (i.e., How does the E x output improve abilities of people with disabilities to perform activities of their choice in the community and also expand society’s capacity to provide full opportunities and accommodations?) Output Usability • uilds upon person-environment interaction and human- B x system integration paradigm • bility to reach target population A x

OCR for page 34
• dheres to principles of knowledge translation A x • eadability; information translated into consumer R x x language that is easy to understand and implement • an be used effectively, repeatedly as intended by design C x • easibility F x • ccessibility A x • cceptability A x • eneralizability G x • ser-friendly U x x • ase of use (for measures: ease of scoring and E x administration) • lexibility F x • calability (for measures) S x • doptability A x • ffordability and repair cost (for devices) A x • ost/benefit C x • inancing available F x • ompatibility and durability C x • ustainability S x • afety issues considered S x Output Utility and Relevance • ddresses high-priority area A x x • ddresses NIDRR mission, International Classification A x of Function, Disability and Health (ICF) model, and Rehabilitation Act • ddresses a practical problem A x • mportance of contribution to the field (i.e., results I x x expand the knowledge base, advance understanding, define issues, fill a gap) 53 continued

OCR for page 34
TABLE A2-1 Continued 54 National Canadian Institute for Academy Occupational of Health NIDRR Safety and Sciences External Health (CAHS) Advisory (NIOSH) Indicators RAND (UK, Chien et al. Criteria/Dimensions Group (U.S.) (U.S.) (Canada) Canada) (Taiwan) • ignificance and magnitude of impact within and S x x x x x outside of field (i.e., Does output help build internal or extramural institutional knowledge? Does output transcend boundaries? Does output produce effective cross-agency, cross-institute, internal/external collaboration?) • ransferability (number of technology output transfers, T x x x x x number of companies transferred to, technology transfer sustained, source of citations in other fields) • ocuses on innovation or emerging technologies F x • bility to stimulate new research, new knowledge, or A x x new products • erceived as useful to practitioners; has changed practice P x x • erceived as useful to consumers P x • utputs are actually used in the field (e.g., adoption of O x x x x guidelines/protocols/interventions; acceptance of outputs via publications/conference proceedings; manufacture license; product available in marketplace; percentage of target audience that has adopted; example of implementation in the field) • utputs provide evidence to support state-level changes O x to policies and/or practices • utput recommendations have been implemented by O x x x policy makers

OCR for page 34
• equests for expert testimony, participation in national R x x advisory roles Dissemination of Outputs x • vidence of efforts to disseminate (e.g., dissemination E x plan and activities) • imely dissemination T x • epth and breadth (e.g., percentage of target group that D x x has used/adopted and duration of use) • issemination of research in practice literature and D x x conferences for transfer potential • se of multimedia and emerging technology (RAND U x x survey asks about specific sources for different audiences) • ode of dissemination matches target audience M x • ounts of products distributed (e.g., manuals, C x x newsletters, curricula) • valuation plan and consumer reviews E x • cope of training: topics covered, number of training S x events, hours of training, participant hours, participant characteristics • echnical assistance/consultation provided T x x • umber of sponsored conferences and workshops, with N x x x documented sponsorship, number and composition of attendees, products of the event, assessment of the event • issemination of research through databases D x x x x SOURCE: Generated by the committee based on cited sources. 55