Appendix B

Rating Sheets/Questionnaires

This appendix contains the following rating sheets and questionnaires used by the committee for its external evaluation of NIDRR and its grantees:

1.   Process Evaluation Measures

A.   Web-Based Questionnaire for NIDRR Stakeholders

B.   Web-Based Questionnaire for NIDRR Peer Reviewers

2.   Summative Evaluation Measures

A.   Letter to Grantees Inviting Participation

B.   Informed Consent Form for NIDRR Grantees

C.   Grantee Questionnaire

D.   Committee Member Review Procedures for Summative Evaluation

E.   Committee Member Rating Sheet of the Quality of Outputs



The National Academies | 500 Fifth St. N.W. | Washington, D.C. 20001
Copyright © National Academy of Sciences. All rights reserved.
Terms of Use and Privacy Statement



Below are the first 10 and last 10 pages of uncorrected machine-read text (when available) of this chapter, followed by the top 30 algorithmically extracted key phrases from the chapter as a whole.
Intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text on the opening pages of each chapter. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.

Do not use for reproduction, copying, pasting, or reading; exclusively for search engines.

OCR for page 305
Appendix B Rating Sheets/Questionnaires This appendix contains the following rating sheets and questionnaires used by the committee for its external evaluation of NIDRR and its grantees: 1. Process Evaluation Measures Page Number A. Web-Based Questionnaire for NIDRR Stakeholders 306 B. Web-Based Questionnaire for NIDRR Peer Reviewers 311 2. Summative Evaluation Measures A. Letter to Grantees Inviting Participation 316 B. Informed Consent Form for NIDRR Grantees 318 C. Grantee Questionnaire 320 D. Committee Member Review Procedures for Summative Evaluation 326 E. Committee Member Rating Sheet of the Quality of Outputs 328 305

OCR for page 305
306 REVIEW OF DISABILITY AND REHABILITATION RESEARCH 1A: WEB-BASED QUESTIONNAIRE FOR NIDRR STAKEHOLDERS Informed Consent Form What the study is about: An expert committee of the National Research Council, of The National Academies, in Washington, DC is conducting an evaluation of some of the activities of the National Institute on Disability and Rehabilitation Research (NIDRR). The committee will be (1) reviewing NIDRR’s priority setting and peer review processes; and (2) reviewing the quality of grantee “outputs” for a sample of NIDRR grants. [“Peer review” refers to a process in which experts review grant applications and make recommendations about whether they should be funded. “Outputs” are publications, measures, intervention protocols, devices, and information resources that are produced as part of a grant.] What we will ask you to do: We are inviting you to take part in the first part of the evaluation— specifically, the review of NIDRR’s priority setting processes. We will ask you a set of questions to help us understand these processes and how they may affect NIDRR’s work. The questionnaire will take about 15 to 20 minutes to complete. Your participation is completely voluntary. You can choose not to answer some of the questions or to stop at any point, and there will be no consequences. Benefits and risks: Benefits: By taking part in the evaluation you will provide information that may help NIDRR improve its research portfolio for the benefit of persons with disabilities. Risks and protections: You might be concerned that the information you share will not be kept private. However, we want to assure you that all of your comments will be kept confidential. Study results will be presented only in combined form, with no individual person or organization being identified. Your name will not be attached to your answers. A research identification number will be used instead. The Study Director will keep a list linking your name with your number. This list, along with the data collected, will be stored securely at the National Research Council. Only the study personnel will have access to the master list, and only for research purposes. Compensation: There is no compensation for taking part in the evaluation. If you have any questions: Please contact the Senior Program Officer, Dr. Jeanne Rivard, if you have any questions about this consent form or the study. She can be contacted by phone at: 202-334-2967, or by email at: jrivard@nas.edu. If you have any questions, comments, or concerns about taking part in this study, first contact Dr. Rivard. If for any reason you do not want to contact her, or you still have concerns after doing so, you may contact the Institutional Review Board (IRB) of the National Academy of Sciences (NAS), which reviewed and approved the study plans and this consent form. You can reach the chair of the IRB, Ronald D. Taylor, Human Protections Administrator, by telephone at 202-334-1659 or you may write to him at the National Academy of Sciences; Room 1026; 500 Fifth Street, NW; Washington, DC 20001. Statement of Consent: I have read the above information, understand it, and agree to take part in the study. I do not agree to take part in the study. B-3

OCR for page 305
307 APPENDIX B Web-Based Questionnaire for NIDRR Stakeholders Please respond as a representative of your organization so that your answers will reflect how your organization relates to NIDRR. 1. Although your organization has multiple functions, select from the list below the one type that most closely matches your own organization. Professional association Advocacy organization Service provider Funder of research and/or development grants Technical assistance and dissemination Commercial manufacturer/distributor Other (Please specify) 2. To what extent is your organization familiar with the National Institute on Disability and Rehabilitation Research (NIDRR)? 1 2 3 4 5 Not at all Somewhat Very much 3. Please check all of the ways in which your organization has interacted with NIDRR since January 1, 2005. Information Exchange Have used NIDRR website or related information sources to search for information, tools, or resources for working with consumers (non grant information) Have used NIDRR website or related resources to search for information about grants Have spoken with NIDRR staff in person or on the phone about specific professional issues Have attended NIDRR trainings, workshops, or conferences Other (Please specify in box below) Funding Have received funding from NIDRR for research & development grants Have received funding from NIDRR for training, knowledge transfer, or dissemination grants Have received funding from NIDRR for activities other than grants (e.g., contract or agreements to conduct an evaluation, provide expert consultation, or write a paper, etc.) Have applied for funding, but did not receive award Other (Please specify in box below) Collaboration Have collaborated with NIDRR as a member of a consortium or professional association Have participated in general planning and special purpose meetings convened by NIDRR Have coordinated activities to support joint priorities and/or to avoid duplication Other (Please specify in box below) Have not interacted with NIDRR on any of the above types of information, funding, or collaboration activities. 4. Please use the space below to comment on the above, or any other types of interactions, your organization has had with NIDRR. (Click in the upper left-hand corner to enter text.) B-4

OCR for page 305
308 REVIEW OF DISABILITY AND REHABILITATION RESEARCH 5. Has your organization had opportunities to review and comment on NIDRR's long range plan or funding priorities? Yes No Don’t know 6. How did your organization find out about opportunities to review and comment on NIDRR's long range plan or funding priorities? Federal Register Professional list serve University grants source Professional newspaper Personal invitation to our organization from NIDRR Other (please specify): Don’t know 7. Since January 1, 2005, has anyone from your organization submitted comments to NIDRR about its long-range plan? Yes No Don’t know 8. Since January 1, 2005, has anyone from your organization submitted comments to NIDRR about its specific funding priorities (e.g., grant announcements) published in the Federal Register? Yes No Don’t know B-5

OCR for page 305
309 APPENDIX B 9. In your opinion, to what extent are NIDRR’s long range planning and priority setting processes (Please check the boxes beneath the responses—select only one per row): Not at all Somewhat Very Don’t Much know Transparent 1 2 3 4 5 Publicized 1 2 3 4 5 Relevant to your 1 2 3 4 5 organization Responsive to emerging 1 2 3 4 5 issues in disability rehabilitation and research Welcoming of stakeholder 1 2 3 4 5 feedback Responsive to stakeholder 1 2 3 4 5 feedback 10. In your opinion, how do NIDRR’s long range planning and priority setting processes compare with those of other federal research agencies? NIDRR’s processes are weaker than other agencies’. NIDRR’s processes are about the same as other agencies’. NIDRR’s processes are stronger than other agencies’. Don’t know 11. How well are the grants that NIDRR funds advancing the field in a direction that is beneficial for your organization, and the members and consumers that you represent? (Please select only one response.) 1 2 3 4 5 Not at all Somewhat Very much Don’t know Not Applicable 12. What types of grants have served your organization, and the members and consumers that you represent? (Click in the upper left-hand corner to enter text.) B-6

OCR for page 305
310 REVIEW OF DISABILITY AND REHABILITATION RESEARCH 13. To what extent are the products of the grants that NIDRR funds (e.g., publications, websites, training or training materials, tools, devices, measures, interventions, etc.) used by your organization? (Please select one response.) 1 2 3 4 5 Not at all Somewhat Very much Don’t know Not Applicable 14. What are the three most important NIDRR-funded products used by your organization? (Click in the upper left-hand corner to enter text.) 15. What three things might NIDRR do to enhance its long range planning and priority setting processes? (Click in the upper left-hand corner to enter text.) Thank you very much for taking the time to complete this questionnaire. We may be contacting some respondents later to ask additional questions. If you would be willing to take part in brief follow-up interviews, please check here. Yes, it’s okay to contact me. No, thank you. I’m finished! Contact Information (Click on the leftmost portion of the line to enter text.) E-mail address: Phone number: B-7

OCR for page 305
311 APPENDIX B 1B: WEB-BASED QUESTIONNAIRE FOR NIDRR PEER REVIEWERS Informed Consent Form What the study is about: An expert committee of the National Research Council, of the National Academies, in Washington, DC is conducting an evaluation of some of the activities of the National Institute on Disability and Rehabilitation Research (NIDRR). The committee will be (1) reviewing NIDRR’s priority setting and peer review processes; and (2) reviewing the quality of grantee “outputs” for a sample of NIDRR grants. [“Peer review” refers to a process in which experts review grant applications and make recommendations about whether they should be funded. “Outputs” are publications, measures, intervention protocols, devices, and information resources that are produced as part of a grant.] What we will ask you to do: We are inviting you to take part in the first part of the evaluation— specifically, the review of NIDRR’s peer review processes. We will ask you a set of questions to help us understand these processes and how they may affect NIDRR’s work. The questionnaire will take about 15 to 20 minutes to complete. Your participation is completely voluntary. You can choose not to answer some of the questions or to stop at any point, and there will be no consequences. Benefits and risks: Benefits: By taking part in the evaluation you will provide information that may help NIDRR improve its research portfolio for the benefit of persons with disabilities. Risks and protections: You might be concerned that the information you share will not be kept private. However, we want to assure you that all of your comments will be kept confidential. Study results will be presented only in combined form, with no individual person or organization being identified. Your name will not be attached to your answers. A research identification number will be used instead. The Study Director will keep a list linking your name with your number. This list, along with the data collected, will be stored securely at the National Research Council. Only the study personnel will have access to the master list, and only for research purposes. Compensation: There is no compensation for taking part in the evaluation. If you have any questions: Please contact the Senior Program Officer, Dr. Jeanne Rivard, if you have any questions about this consent form or the study. She can be contacted by phone at: 202-334-2967, or by e-mail at: jrivard@nas.edu. If you have any questions, comments, or concerns about taking part in this study, first contact Dr. Rivard. If for any reason you do not want to contact her, or you still have concerns after doing so, you may contact the Institutional Review Board (IRB) of the National Academy of Sciences (NAS), which reviewed and approved the study plans and this consent form. You can reach the chair of the IRB, Ronald D. Taylor, Human Protections Administrator, by telephone at 202-334-1659 or you may write to him at the National Academy of Sciences; Room 1026; 500 Fifth Street, NW; Washington, DC 20001. Statement of Consent: I have read the above information, understand it, and agree to take part in the study. I do not agree to take part in the study. B-8

OCR for page 305
312 REVIEW OF DISABILITY AND REHABILITATION RESEARCH Web-Based Questionnaire for NIDRR Peer Reviewers 1. On how many NIDRR peer review panels have you served since January 1, 2005? (Please state the total number, such as 0, 1, 2, 3, 4, etc.): (By panel, we mean every time a group of peer reviewers is convened to conduct a review, not just membership on a given panel.) 2. Across the following program funding mechanisms, please check all of those for which you have reviewed applications since January 1, 2005 (See NIDRR website for a description of these mechanisms) Disability and Rehabilitation Research Projects (DRRP) – General Disability Business Technical Assistance Centers (DBTAC) Traumatic Brain Injury Model Systems Centers Burn Model Systems Centers Spinal Cord Injury Model Systems Centers Rehabilitation Research and Training Centers (RRTCs) Rehabilitation Engineering Research Centers (RERCs) Field Initiated Projects – Research Grants Field Initiated Projects – Development Grants Small Business Innovation Research (SBIR) – Phase I Small Business Innovation Research (SBIR) – Phase II DRRP - Knowledge Translation DRRP – Section 21 (Switzer) Research Fellowship Program Advanced Rehabilitation Research Training Projects Don’t Remember 3. In your experience, how would you rate the following aspects of the NIDRR peer review processes (Please check the boxes beneath the responses—select only one per row): Poor Adequate Excellent DON’T NOT KNOW APPLI- CABLE Quality of the training to prepare 1 2 3 4 5 you for the review Adequacy of time for review of 1 2 3 4 5 materials before the meeting Level of expertise of the peer 1 2 3 4 5 review panel members Appropriateness of the evaluation 1 2 3 4 5 criteria to applications under review Clarity of the criteria when 1 2 3 4 5 applying them to applications B-9

OCR for page 305
313 APPENDIX B Poor Adequate Excellent DON’T NOT KNOW APPLI- CABLE Appropriateness of scoring 1 2 3 4 5 system to applications under review Ease of applying scoring system 1 2 3 4 5 to applications Thoroughness of the deliberation 1 2 3 4 5 (i.e., grant scoring and discussion) during the meeting. Use of reviewers’ time during the 1 2 3 4 5 panel meeting Support and facilitation of the 1 2 3 4 5 review panel by NIDRR staff Guidance in writing your 1 2 3 4 5 reviewer comments Integrity of the peer review 1 2 3 4 5 process overall Consistency in the overall quality of the peer reviews across panels 2 3 4 5 1 (if you have served on three or more panels). 4. Any additional comments you may have on these aspects of NIDRR's peer review processes would be useful. Please use the space below. (Click in the upper left-hand corner to enter text.) 5. How has the overall quality of NIDRR’s peer review processes changed since January 1, 2005 (if you have served on different panels over time)? Quality has decreased. Quality is about the same. Quality has increased. Don’t know Not applicable 6. Typically, the number of applications you received for each NIDRR review panel was: More than you would like to review. About the right number. Fewer than you would like to review. 7. Typically, the amount of time you spent on each NIDRR review panel was: More time than you would like to spend. B-10

OCR for page 305
314 REVIEW OF DISABILITY AND REHABILITATION RESEARCH About the right amount. Less time than you would like to spend. 8. Have you participated in NIDRR peer reviews that were convened through: Yes No In-person meetings? Teleconference? 9. In your opinion, how did the quality of a NIDRR peer review compare when conducted in person versus teleconference? Quality of peer reviews is better in in-person meetings. Quality of peer reviews is better in teleconference meetings. Quality of peer reviews via in-person meetings or teleconference is about the same. Don’t know Not applicable 10. Have you served as a peer reviewer of proposals for federal agencies other than NIDRR since January 1, 2005? Yes No 11. How would you compare the following characteristics of NIDRR peer reviews with those of other federal agencies? (Please check the boxes beneath the responses—select only one per row.) NIDRR’s are About the NIDRR’s Don't Not much weaker same are much Know Appli- than other stronger cable agencies than other agencies Expertise of the panel 1 2 3 4 5 members Quality of the proposals 1 2 3 4 5 reviewed Quality of the review 1 2 3 4 5 process Transparency of the review 1 2 3 4 5 process Fairness of the review 1 2 3 4 5 process 1 2 3 4 5 Reliability of the ratings 12. What three things would you suggest to enhance NIDRR’s peer review processes? (Click in the upper left-hand corner to enter text.) B-11

OCR for page 305
315 APPENDIX B Thank you very much for taking the time to complete this questionnaire. We may be contacting some respondents later to ask additional questions. If you would be willing to take part in brief follow-up interviews, please check here. Yes, it’s okay to contact me. No, thank you. I’m finished! Contact Information (Click on the leftmost portion of the line to enter text.) E-mail address: Phone number: B-12

OCR for page 305
319 APPENDIX B such as output type (e.g., publications, tools, technology, information products); quality criteria assessed (e.g., technical quality, knowledge advancement, potential impact); or program funding type (e.g., center grant, field initiated grant, training grant, etc.). If your grant or your output represents one of a kind and there is a risk of identifying you because of this, your data will be aggregated with another larger group where identification will not be a risk. Every effort will be made to protect the confidentiality of the information that you provide. The Study Director will keep a list linking the grant and output research ID numbers with that output’s identifying information (institution, grant title). This list, along with the data collected, will be stored securely at the National Research Council, and will be accessible only by the Study personnel. If a telephone or videoconference interview is convened to gather additional follow-up information, the transcription of audio-recorded interviews will be combined in a dataset with the interviews of all of the other respondents, then analyzed for common themes across the interviews. The audiotapes, transcriptions, grantee questionnaires, committee ratings, and other raw data collected will be destroyed at the end of the study when the report is released. Compensation: There is no compensation for participating in the evaluation. If you have any questions: The Co-Study Director of the evaluation and contact for questions is Jeanne Rivard, Ph.D. If you have any questions about this consent form or the study, she can be contacted by phone at: 202-334-2697, or by e-mail at: jrivard@nas.edu. If you have any questions, comments, or concerns about taking part in this study, first talk to Dr. Rivard above. If for any reason you do not want to do this, or you still have concerns after doing so, you may contact the Institutional Review Board (IRB) of the National Academy of Sciences (NAS), which reviewed and approved the study plans and this consent form. You can reach the chair of the IRB by contacting Ronald D. Taylor, Human Protections Administrator, by telephone at 202-334-1659 or you may write to him at the National Academy of Sciences; Room 1026; 500 Fifth Street, NW; Washington, DC 20001. Statement of Consent: I have read the above information, and have received answers to any questions I asked. I consent to take parting in the study. Your Signature_______________________________________ Date__-__-__ Your Name (printed) _____________________________________________ In addition to agreeing to participate, if there is a follow-up interview, I also consent to having it tape- recorded. Your Signature_______________________________________ Date__-__-__ B-16

OCR for page 305
320 REVIEW OF DISABILITY AND REHABILITATION RESEARCH 2C: GRANTEE QUESTIONNAIRE Grant Award Number: Grant Title: Grantee: Program Mechanism: Grant End Date: INSTRUCTIONS TO GRANTEES This questionnaire has been designed to obtain information to assist the Committee in assessing the quality of your grant’s outputs. NIDRR has provided to the Committee and the National Research Council (NRC) staff copies of its Annual Performance Report (APR) database and your last APR and your final APR. As you will see in certain places on the questionnaire we have inserted information from your APR to facilitate your completion of the questionnaire (e.g., Table 1 lists your research and development projects, and Table 2 lists your outputs reported in the APR). Where this information is in error, we would appreciate your pointing the errors out to us and correcting it; and/or updating the information as needed. The questionnaire is divided into the following three parts: Part A. Nominating Outputs for Review. This section asks you to nominate, for the Committee’s review, the “top 2” outputs for each of your projects that best reflect your grant's achievements (Table 1 below). The Committee would prefer to review one publication and one other type of output for each project. However if you only have publications, please nominate these as your “top 2.” To make this process easier, the NRC staff has populated a list of the outputs (Table 2 below) that were reported for your grant in the APR. You could select the top 2 outputs from this table. However you are not constrained to select from this list if there are other outputs that you think better reflect your grant’s achievements. For Committee review we are requesting materials and information regarding the actual outputs selected as the top 2 for each project. For publications, the material for review would be pdf copies of each article.  For the other outputs, materials for review would include:  o Electronic or hard copies of the measures, tools, intervention protocols, manuals; or links to websites, pictures or other graphic representations of tools or devices that have been produced. o An abstract or summary of each output, which briefly describes: o what the output is, o its purpose, B-17

OCR for page 305
321 APPENDIX B o target audience, o methods, and o how the output fits into the overall goals and objectives of the project and grant Part B. Additional Questions about Outputs. For each of the outputs you nominated for review, the Committee has a series of questions related to their technical quality, how they may have advanced knowledge, their potential impact, and their dissemination. We ask that you complete the Part B section for each output. If the answers to certain questions would be the same across different outputs, you can note this and cut and paste responses from earlier output forms to other ones. Please make your responses brief, but as specific and quantitative as possible. Part C. Grant-level Questions. The questionnaire will also contain a few other items asking about how you managed your grants to produce the highest quality outputs, how your grant’s results may have generated new projects, and how key NIDRR processes influence results. Your complete package of materials will contain:  Your signed informed consent form  Copies of your publications and other outputs (e.g., measures, tools, intervention protocols, manuals, links to websites, pictures or other graphic representations of devices that have been produced)  Your completed Grantee Questionnaire Please send these materials by ___DATE___ to: Matt McDonough The National Academies National Research Council 500 Fifth Street, NW WS 1134 Washington, DC 20001 We are enclosing an addressed FedEx form that can be used when mailing your package of materials to us. We estimated a shipment cost that would cover a weight up to 10 lbs. (e.g., for large center grants or devices). If your package weighs more than this, FedEx will charge us the correct amount. If your package is light and you want to send it electronically, you could e-mail it to Matt at mmcdonough@nas.edu. However, you would need to scan your signed consent form, and send that in a pdf document. B-18

OCR for page 305
322 REVIEW OF DISABILITY AND REHABILITATION RESEARCH Part A. Nominating Outputs for Review When referring to “outputs,” we are using the four NIDRR categories of outputs as defined in NIDRR’s Annual Performance Report, which include: (a) Publications; (b) Tools, Measures, and Intervention Protocols; (c) Technology Products and Devices; and (d) Informational Products. Per the instructions for nominating outputs for review, please record your nominations for your “top 2” outputs for each of your projects in Table 1 below. (Reminder “top 2” refers to those that best reflect your grant's achievements). As you can see the NRC staff has already populated Table 1 with the names of your research and development projects from data in the APR. Table 2, which follows, contains a list of outputs from which you can cut and paste into Table 1 below. Please identify any errors in this information that we have provided from your APR and correct it as needed. Table 1. Projects and Nominated Outputs Names of R&D Projects in Grant Names of Top 2 Outputs for Each Project Outputs to be inserted by grantee from Table # 2 below or add others as needed Research Projects R1 1. 2. Development Projects D1 Dissemination 1. 2. INSERT GRANTEES' PROJECTS TABLE HERE The table below lists all of the publications and other outputs that were listed in the APR data provided by NIDRR. Please use this table below in selecting your top 2 outputs for each project. (You can cut and paste from Table 2 into Table 1.) However you are not constrained to select only from this list if there are other outputs that you think better reflect your grant’s achievements. Table 2. List of Outputs from APR Type of Output Title of Output publications (title) tool (title) B-19

OCR for page 305
323 APPENDIX B Part B. Additional Questions About Outputs Please use one copy of this form for each publication and each other output for the “top 2” outputs that you selected for each project in Part A above, and provide the following information. Please make your responses brief, but as specific and quantitative as possible. If you consider the criterion not to be applicable to your output, please explain. (Please note that an electronic copy of the questionnaire was included in the email version of this package.) Name of Output: B1. Technical Quality of Output In the space below, please describe examples of the technical quality of your output, such as: The particular approach or methodology used in developing your output  Relevant peer recognition such as peer reviews or evaluations, peer endorsements,  invitations to present at professional forums or conferences, invitations to present testimony, receipt of awards or honors, etc. Receipt of a patent, FDA approval, or use of your output in standards development  Evidence of the usability and accessibility of the output  B2. Advancement of Knowledge Please use the space below to describe how this output has advanced knowledge. To structure your response, include points such as: What the importance of your original question or issue was  How the output has advanced knowledge in arenas, such as:  o making discoveries o providing new information o establishing theories, measures, and methods o closing gaps in the knowledge base o developing new interventions, products, technology, and environmental adaptations B-20

OCR for page 305
324 REVIEW OF DISABILITY AND REHABILITATION RESEARCH B3. Potential Impact In the space below, please briefly describe evidence of your outputs’ potential (or actual) impact on the following audiences, as relevant to your output: Science (e.g., new areas of inquiry, methodology, etc.)  People with disabilities: health, quality of life, participation  Provider practice  Health and social systems  Social and health policy  Private sector/commercialization  Capacity building in the field of rehabilitation and disability research and development  (e.g., scientists, graduate students, etc.) Other  Include information about how this potential impact was tested, and what the results were. B4. Dissemination of Outputs In the space below please provide evidence of your dissemination efforts for this output. Describe this for publications if you have made any effort beyond those of the sponsor of the publication (journal, book, proceedings, etc.). Please include important aspects of dissemination such as: Stage and scope (e.g., local, regional, national) of dissemination  Dissemination activities  Identification and tailoring of materials for reaching different audience/user types  Collaboration with audience/users in identifying content and medium needs/preferences  Delivery of information through multiple media types and sources for optimal reach and  accessibility Evaluation of your dissemination efforts and impacts  B-21

OCR for page 305
325 APPENDIX B Part C. Grant-level Questions Please respond to these final questions for your overall grant, not by each output specifically as in Section B. C1. In the space below please describe what types of planning, project management, and budgetary processes were used to promote high quality outputs. In your statement consider the following types of questions: Which processes were useful and how? How could they be improved?  Did you dedicate funds for quality assurance activities?  How did you track progress and spending against your original plans for the grant?  If grants or projects were jointly funded by NIDRR and other extramural or intramural  sources, how did you ensure that NIDRR resources were used exclusively for NIDRR- funded activities? C2. Have the results of the research and development outputs from this grant, or prior NIDRR grants, been used to inform the development of new grant applications or other kinds of projects? No ____ Yes____ If yes, please use the space below to briefly describe what new grant applications, other projects, funding opportunities, or collaborations have emerged. C3. Please share any perspectives you may have about how NIDRR’s key processes (e.g., priority setting, peer review, and/or grants management) influence results, such as successful grants and high quality outputs. B-22

OCR for page 305
326 REVIEW OF DISABILITY AND REHABILITATION RESEARCH 2D: Committee Member Review Procedures for Summative Evaluation 2D: COMMITTEE MEMBER REVIEW PROCEDURES FOR SUMMATIVE EVALUATION A. Review Subgroups: Each subgroup that will be reviewing outputs will be composed of five Committee members. For each output one committee member will be assigned as the primary reviewer; the remaining four committee members will be secondary reviewers. B. Output Rating Procedures: 1. All reviewers will independently rate outputs using the following quality criteria (Dimensions of these criteria are shown on the attached rating sheet.):  Technical quality of output  Advancement of knowledge or the field (research, practice, or policy)  Likely impact  Dissemination The following scale will be used for rating the outputs: Poor Good Excellent Quality Quality Quality 1 2 3 4 5 6 7 2. The rating will be based on review of hard copy and electronic materials (i.e., articles/descriptive information about output and questionnaire responses) prior to the subcommittee meeting. 3. The grantee’s final summary APR, and a list of all outputs reported over the course of the project, is provided for contextual purposes. The APR also will be used to inform an overall, qualitative grant-level assessment. 4. Multiple outputs of one grant will generally be rated independently of each other. However, in some cases outputs may be rated as a pair with one score applied. This could occur when one output is a derivative or different expression of another output, and when the PI responses to criterion questions are basically the same. Examples of these include:  A manual describing a device (1) and a patent of the device (2)  A publication describing how a new technology for assessing a condition can be applied in disability rehabilitation (1) and a description of the technology itself (2)  A software application (e.g., map reader for persons with visual impairments) (1) and web-based method for individualizing the software for users (2) 5. The meeting will be structured as follows. B-23

OCR for page 305
327 APPENDIX B 5. The meeting will be structured as follows: The primary reviewer will open discussion of each output by presenting a brief  summary of the output and then his/her rationale for rating each relevant criterion (up to four) plus the overall score. Secondary reviewers will then present their ratings for each output and a brief  rationale. Using the same criteria, the subgroup will then develop consensus group ratings  for each output. Discussion will be facilitated by the subgroup chair. If there is a subgroup member with a significantly divergent view, his/her score and rationale will be captured separately. Staff will document discussion points that lead to the consensus group ratings and  will record the subgroup’s rationale for each criterion, the overall rating, and the grant performance rating in a brief narrative. At the end of the review of each output, the individual subgroup members’ rating  sheets will be gathered. C. Grant Assessment Once all outputs of an individual grant are reviewed, the subgroup will consider and rate the grant’s overall performance. The outputs reviewed had been identified by the grant’s Principal Investigator as the “top” two outputs per project, which best reflected the grant’s achievements. Taking into consideration this designation, the consensus group ratings of the entire set of outputs, and the grant’s overall purpose and objectives (using the grant’s APR), the subgroup will assign a grant performance rating using the same 7-point scale. These grant-level ratings and their rationale will also be documented by staff. B-24

OCR for page 305
328 REVIEW OF DISABILITY AND REHABILITATION RESEARCH 2E: COMMITTEE MEMBER RATING SHEET OF THE QUALITY OF OUTPUTS Committee Member: Output ID: Grantee ID: Date of Review: Output to Be Reviewed: To be completed by NRC staff Output Title: _______________________________________________________ Research Output: ______ Development Output: ________ Type and Subtype of Output (marked below): Type of Type of Tool, Type of Technology Type of Informational Publication Measure, or Product or Device Product Intervention Protocol 1. abstract 1. checklist 1. industry 1. training standards/guidelines manuals/curricula 2. book 2. survey or interview 2. software or netware 2. fact sheets schedule 3. book chapter 3. diagnostic or 3. invention 3. newsletters assessment instrument 4. journal article 4. outcome measure 4. patent, license, or 4. audiovisual materials disclosures 5. proceedings 5. intervention protocol 5. working prototype 5. marketing tools or program 6. technical 6. statistical technique 6. product evaluated or field 6. educational aids tested 7. web journal 7. database 7. product transferred to 7. websites or other industry for potential internet sites commercialization 8. other 8. other 8. product in marketplace 8. other B-25

OCR for page 305
329 APPENDIX B Quality Criteria and Dimensions For each criterion provide one rating using the scale below: Poor Good Excellent 1 2 3 4 5 6 7 Criteria and Dimensions Score Technical Quality of Output  Applying standards of science and technology  Appropriate methodology (quantitative or qualitative design and statistics)  Accessibility, usability, etc. Score Rationale: Advancement of Knowledge or the Field (research, practice, or policy as relevant)  Science: Establishment of methods, tools, theory  New information  Closing an identified gap  New technology  Innovative or novel Score Rationale: Likely or Demonstrated Impact On:  Science (impact factor, citations)  Consumers (people with disabilities: health, quality of life, participation)  Provider practice  Health and social system  Social and health policy  Private sector/commercialization  Other Score Rationale: Dissemination  Identification and tailoring of materials for reaching different audience/user types  Collaboration with audience/users in identifying content and medium needs/preferences  Delivery of information through multiple media types and sources for optimal reach and accessibility  Evaluation of dissemination efforts and impacts  Commercialization/patenting of devices, if applicable Score Rationale: Overall Score Score Rationale: B-26