National Academies Press: OpenBook
« Previous: Appendix A: Grant Summaries
Suggested Citation:"Appendix B: Rating Sheets/Questionnaires." National Research Council. 2012. Review of Disability and Rehabilitation Research: NIDRR Grantmaking Processes and Products. Washington, DC: The National Academies Press. doi: 10.17226/13285.
×

Appendix B

Rating Sheets/Questionnaires

This appendix contains the following rating sheets and questionnaires used by the committee for its external evaluation of NIDRR and its grantees:

1.   Process Evaluation Measures

A.   Web-Based Questionnaire for NIDRR Stakeholders

B.   Web-Based Questionnaire for NIDRR Peer Reviewers

2.   Summative Evaluation Measures

A.   Letter to Grantees Inviting Participation

B.   Informed Consent Form for NIDRR Grantees

C.   Grantee Questionnaire

D.   Committee Member Review Procedures for Summative Evaluation

E.   Committee Member Rating Sheet of the Quality of Outputs

Suggested Citation:"Appendix B: Rating Sheets/Questionnaires." National Research Council. 2012. Review of Disability and Rehabilitation Research: NIDRR Grantmaking Processes and Products. Washington, DC: The National Academies Press. doi: 10.17226/13285.
×

1A: WEB-BASED QUESTIONNAIRE FOR NIDRR STAKEHOLDERS

Informed Consent Form

What the study is about: An expert committee of the National Research Council, of The National Academies, in Washington, DC is conducting an evaluation of some of the activities of the National Institute on Disability and Rehabilitation Research (NIDRR). The committee will be (1) reviewing NIDRR’s priority setting and peer review processes; and (2) reviewing the quality of grantee “outputs” for a sample of NIDRR grants. [“Peer review” refers to a process in which experts review grant applications and make recommendations about whether they should be funded. “Outputs” are publications, measures, intervention protocols, devices, and information resources that are produced as part of a grant.]

What we will ask you to do: We are inviting you to take part in the first part of the evaluation— specifically, the review of NIDRR’s priority setting processes. We will ask you a set of questions to help us understand these processes and how they may affect NIDRR’s work. The questionnaire will take about 15 to 20 minutes to complete. Your participation is completely voluntary. You can choose not to answer some of the questions or to stop at any point, and there will be no consequences.

Benefits and risks:

Benefits: By taking part in the evaluation you will provide information that may help NIDRR improve its research portfolio for the benefit of persons with disabilities.

Risks and protections: You might be concerned that the information you share will not be kept private. However, we want to assure you that all of your comments will be kept confidential. Study results will be presented only in combined form, with no individual person or organization being identified. Your name will not be attached to your answers. A research identification number will be used instead. The Study Director will keep a list linking your name with your number. This list, along with the data collected, will be stored securely at the National Research Council. Only the study personnel will have access to the master list, and only for research purposes.

Compensation: There is no compensation for taking part in the evaluation.

If you have any questions: Please contact the Senior Program Officer, Dr. Jeanne Rivard, if you have any questions about this consent form or the study. She can be contacted by phone at: 202-334-2967, or by email at: jrivard@nas.edu.

If you have any questions, comments, or concerns about taking part in this study, first contact Dr. Rivard. If for any reason you do not want to contact her, or you still have concerns after doing so, you may contact the Institutional Review Board (IRB) of the National Academy of Sciences (NAS), which reviewed and approved the study plans and this consent form. You can reach the chair of the IRB, Ronald D. Taylor, Human Protections Administrator, by telephone at 202-334-1659 or you may write to him at the National Academy of Sciences; Room 1026; 500 Fifth Street, NW; Washington, DC 20001.

Statement of Consent:

image   I have read the above information, understand it, and agree to take part in the study.

image   I do not agree to take part in the study.

Suggested Citation:"Appendix B: Rating Sheets/Questionnaires." National Research Council. 2012. Review of Disability and Rehabilitation Research: NIDRR Grantmaking Processes and Products. Washington, DC: The National Academies Press. doi: 10.17226/13285.
×

Web-Based Questionnaire for NIDRR Stakeholders

Please respond as a representative of your organization so that your answers will reflect how your organization relates to NIDRR.

1. Although your organization has multiple functions, select from the list below the one type that most closely matches your own organization.

image   Professional association

image   Advocacy organization

image   Service provider

image   Funder of research and/or development grants

image   Technical assistance and dissemination

image   Commercial manufacturer/distributor

image   Other (Please specify)       ____________________________________________________

image

2. To what extent is your organization familiar with the National Institute on Disability and Rehabilitation Research (NIDRR)?

3. Please check all of the ways in which your organization has interacted with NIDRR since January 1, 2005.

Information Exchange

image   Have used NIDRR website or related information sources to search for information, tools, or resources for working with consumers (non grant information)

image   Have used NIDRR website or related resources to search for information about grants

image   Have spoken with NIDRR staff in person or on the phone about specific professional issues

image   Have attended NIDRR trainings, workshops, or conferences

image   Other (Please specify in box below)

Funding

image   Have received funding from NIDRR for research & development grants

image   Have received funding from NIDRR for training, knowledge transfer, or dissemination grants

image   Have received funding from NIDRR for activities other than grants (e.g., contract or agreements to conduct an evaluation, provide expert consultation, or write a paper, etc.)

image   Have applied for funding, but did not receive award

image   Other (Please specify in box below)

Collaboration

image   Have collaborated with NIDRR as a member of a consortium or professional association

image   Have participated in general planning and special purpose meetings convened by NIDRR

image   Have coordinated activities to support joint priorities and/or to avoid duplication

image   Other (Please specify in box below)

image   Have not interacted with NIDRR on any of the above types of information, funding, or collaboration activities.

4. Please use the space below to comment on the above, or any other types of interactions, your organization has had with NIDRR. (Click in the upper left-hand corner to enter text.)

Suggested Citation:"Appendix B: Rating Sheets/Questionnaires." National Research Council. 2012. Review of Disability and Rehabilitation Research: NIDRR Grantmaking Processes and Products. Washington, DC: The National Academies Press. doi: 10.17226/13285.
×

image

5. Has your organization had opportunities to review and comment on NIDRR's long range plan or funding priorities?

image   Yes

image   No

image   Don’t know

Suggested Citation:"Appendix B: Rating Sheets/Questionnaires." National Research Council. 2012. Review of Disability and Rehabilitation Research: NIDRR Grantmaking Processes and Products. Washington, DC: The National Academies Press. doi: 10.17226/13285.
×

6. How did your organization find out about opportunities to review and comment on NIDRR's long range plan or funding priorities?

image   Federal Register

image   Professional list serve

image   University grants source

image   Professional newspaper

image   Personal invitation to our organization from NIDRR

image   Other (please specify):_________________________________________________

image   Don’t know

7. Since January 1, 2005, has anyone from your organization submitted comments to NIDRR about its long-range plan?

image   Yes

image   No

image   Don’t know

8. Since January 1, 2005, has anyone from your organization submitted comments to NIDRR about its specific funding priorities (e.g., grant announcements) published in the Federal Register?

image   Yes

image   No

image   Don’t know

9. In your opinion, to what extent are NIDRR’s long range planning and priority setting processes (Please check the boxes beneath the responses—select only one per row):

image

10. In your opinion, how do NIDRR’s long range planning and priority setting processes compare with those of other federal research agencies?

image   NIDRR’s processes are weaker than other agencies’.

image   NIDRR’s processes are about the same as other agencies’.

image   NIDRR’s processes are stronger than other agencies’.

image   Don’t know

11. How well are the grants that NIDRR funds advancing the field in a direction that is beneficial for your organization, and the members and consumers that you represent? (Please select only one response.)

image

12. What types of grants have served your organization, and the members and consumers that you represent? (Click in the upper left-hand corner to enter text.)

image

Suggested Citation:"Appendix B: Rating Sheets/Questionnaires." National Research Council. 2012. Review of Disability and Rehabilitation Research: NIDRR Grantmaking Processes and Products. Washington, DC: The National Academies Press. doi: 10.17226/13285.
×

13. To what extent are the products of the grants that NIDRR funds (e.g., publications, websites, training or training materials, tools, devices, measures, interventions, etc.) used by your organization? (Please select one response.)

image

14. What are the three most important NIDRR-funded products used by your organization? (Click in the upper left-hand corner to enter text.)

image

15. What three things might NIDRR do to enhance its long range planning and priority setting processes? (Click in the upper left-hand corner to enter text.)

image

Thank you very much for taking the time to complete this questionnaire. We may be contacting some respondents later to ask additional questions. If you would be willing to take part in brief follow-up interviews, please check here.

image   Yes, it’s okay to contact me.

image   No, thank you. I’m finished!

Contact Information

(Click on the leftmost portion of the line to enter text.)

E-mail address:     ______________________________________________

Phone number:

Suggested Citation:"Appendix B: Rating Sheets/Questionnaires." National Research Council. 2012. Review of Disability and Rehabilitation Research: NIDRR Grantmaking Processes and Products. Washington, DC: The National Academies Press. doi: 10.17226/13285.
×

1B: WEB-BASED QUESTIONNAIRE FOR NIDRR PEER REVIEWERS

Informed Consent Form

What the study is about: An expert committee of the National Research Council, of the National Academies, in Washington, DC is conducting an evaluation of some of the activities of the National Institute on Disability and Rehabilitation Research (NIDRR). The committee will be (1) reviewing NIDRR’s priority setting and peer review processes; and (2) reviewing the quality of grantee “outputs” for a sample of NIDRR grants. [“Peer review” refers to a process in which experts review grant applications and make recommendations about whether they should be funded. “Outputs” are publications, measures, intervention protocols, devices, and information resources that are produced as part of a grant.]

What we will ask you to do: We are inviting you to take part in the first part of the evaluation— specifically, the review of NIDRR’s peer review processes. We will ask you a set of questions to help us understand these processes and how they may affect NIDRR’s work. The questionnaire will take about 15 to 20 minutes to complete. Your participation is completely voluntary. You can choose not to answer some of the questions or to stop at any point, and there will be no consequences.

Benefits and risks:

Benefits: By taking part in the evaluation you will provide information that may help NIDRR improve its research portfolio for the benefit of persons with disabilities.

Risks and protections: You might be concerned that the information you share will not be kept private. However, we want to assure you that all of your comments will be kept confidential. Study results will be presented only in combined form, with no individual person or organization being identified. Your name will not be attached to your answers. A research identification number will be used instead. The Study Director will keep a list linking your name with your number. This list, along with the data collected, will be stored securely at the National Research Council. Only the study personnel will have access to the master list, and only for research purposes.

Compensation: There is no compensation for taking part in the evaluation.

If you have any questions: Please contact the Senior Program Officer, Dr. Jeanne Rivard, if you have any questions about this consent form or the study. She can be contacted by phone at: 202-334-2967, or by e-mail at: jrivard@nas.edu.

If you have any questions, comments, or concerns about taking part in this study, first contact Dr. Rivard. If for any reason you do not want to contact her, or you still have concerns after doing so, you may contact the Institutional Review Board (IRB) of the National Academy of Sciences (NAS), which reviewed and approved the study plans and this consent form. You can reach the chair of the IRB, Ronald D. Taylor, Human Protections Administrator, by telephone at 202-334-1659 or you may write to him at the National Academy of Sciences; Room 1026; 500 Fifth Street, NW; Washington, DC 20001.

Statement of Consent:

image   I have read the above information, understand it, and agree to take part in the study.

image   I do not agree to take part in the study.

Suggested Citation:"Appendix B: Rating Sheets/Questionnaires." National Research Council. 2012. Review of Disability and Rehabilitation Research: NIDRR Grantmaking Processes and Products. Washington, DC: The National Academies Press. doi: 10.17226/13285.
×

Web-Based Questionnaire for NIDRR Peer Reviewers

1. On how many NIDRR peer review panels have you served since January 1, 2005? (Please state the total number, such as 0, 1, 2, 3, 4, etc.):_______

(By panel, we mean every time a group of peer reviewers is convened to conduct a review, not just membership on a given panel.)

2. Across the following program funding mechanisms, please check all of those for which you have reviewed applications since January 1, 2005 (See NIDRR website for a description of these mechanisms)

Disability and Rehabilitation Research Projects (DRRP) – General image
Disability Business Technical Assistance Centers (DBTAC) image
Traumatic Brain Injury Model Systems Centers image
Burn Model Systems Centers image
Spinal Cord Injury Model Systems Centers image
Rehabilitation Research and Training Centers (RRTCs) image
Rehabilitation Engineering Research Centers (RERCs) image
Field Initiated Projects – Research Grants image
Field Initiated Projects – Development Grants image
Small Business Innovation Research (SBIR) – Phase I image
Small Business Innovation Research (SBIR) – Phase II image
DRRP - Knowledge Translation image
DRRP – Section 21 image
(Switzer) Research Fellowship Program image
Advanced Rehabilitation Research Training Projects image
Don’t Remember image

3. In your experience, how would you rate the following aspects of the NIDRR peer review processes (Please check the boxes beneath the responses—select only one per row):

image

Suggested Citation:"Appendix B: Rating Sheets/Questionnaires." National Research Council. 2012. Review of Disability and Rehabilitation Research: NIDRR Grantmaking Processes and Products. Washington, DC: The National Academies Press. doi: 10.17226/13285.
×

image

4. Any additional comments you may have on these aspects of NIDRR's peer review processes would be useful. Please use the space below. (Click in the upper left-hand corner to enter text.)

image

5. How has the overall quality of NIDRR’s peer review processes changed since January 1, 2005 (if you have served on different panels over time)?

image   Quality has decreased.

image   Quality is about the same.

image   Quality has increased.

image   Don’t know Not applicable

6. Typically, the number of applications you received for each NIDRR review panel was:

image   More than you would like to review.

image   About the right number.

image   Fewer than you would like to review.

7. Typically, the amount of time you spent on each NIDRR review panel was:

image   More time than you would like to spend.

Suggested Citation:"Appendix B: Rating Sheets/Questionnaires." National Research Council. 2012. Review of Disability and Rehabilitation Research: NIDRR Grantmaking Processes and Products. Washington, DC: The National Academies Press. doi: 10.17226/13285.
×

image   About the right amount.

image   Less time than you would like to spend.

8. Have you participated in NIDRR peer reviews that were convened through:

  Yes No
In-person meetings? image image
Teleconference? image image

9. In your opinion, how did the quality of a NIDRR peer review compare when conducted in person versus teleconference?

image   Quality of peer reviews is better in in-person meetings.

image   Quality of peer reviews is better in teleconference meetings.

image   Quality of peer reviews via in-person meetings or teleconference is about the same.

image   Don’t know

image   Not applicable

10. Have you served as a peer reviewer of proposals for federal agencies other than NIDRR since January 1, 2005?

image   Yes

image   No

11. How would you compare the following characteristics of NIDRR peer reviews with those of other federal agencies? (Please check the boxes beneath the responses—select only one per row.)

image

12. What three things would you suggest to enhance NIDRR’s peer review processes? (Click in the upper left-hand corner to enter text.)

Suggested Citation:"Appendix B: Rating Sheets/Questionnaires." National Research Council. 2012. Review of Disability and Rehabilitation Research: NIDRR Grantmaking Processes and Products. Washington, DC: The National Academies Press. doi: 10.17226/13285.
×

image

Thank you very much for taking the time to complete this questionnaire. We may be contacting some respondents later to ask additional questions. If you would be willing to take part in brief follow-up interviews, please check here.

image   Yes, it’s okay to contact me.

image   No, thank you. I’m finished!

Contact Information

(Click on the leftmost portion of the line to enter text.)

E-mail address:   _______________________________________________

Phone number:

Suggested Citation:"Appendix B: Rating Sheets/Questionnaires." National Research Council. 2012. Review of Disability and Rehabilitation Research: NIDRR Grantmaking Processes and Products. Washington, DC: The National Academies Press. doi: 10.17226/13285.
×

2A: LETTER TO GRANTEES INVITING PARTICIPATION

image

October 6, 2010

Grantee Address

Dear Dr. _______:

Last week we sent you an email to inform you that you are being invited to participate in the External Evaluation of the National Institute on Disability and Rehabilitation Research (NIDRR) and its Grantees that is being conducted by an expert committee of the National Research Council of the National Academies. This independent evaluation is being sponsored by NIDRR for the purpose of: 1) assessing NIDRR's priority setting and peer review processes and 2) reviewing the quality of grantee outputs for a sample of grants that represent the NIDRR portfolio.

Your NIDRR-funded grant (Grant #__________, Grant Title:____________________________) was selected to be reviewed as part of the evaluation of grantee outputs. However, your participation is completely voluntary. If you do agree to be part of the evaluation, your participation will involve the following activities.

1.   We will ask you to nominate two outputs that were produced under each project that was funded by the grant. These will be outputs that best reflect your grant’s achievements. We are using the four NIDRR categories of outputs as defined in NIDRR's Annual Performance Report (APR), which include a) Publications; b) Tools, Measures, and Intervention Protocols; c) Technology Products and Devices; and d) Informational Products. The committee will assess the quality of the outputs which you identify using criteria of technical quality, the extent to which they advance knowledge, their potential impact, and their dissemination.

2.   To conduct the review, we would like to examine the actual outputs and to review any documentation that you may have about the outputs.

3.   We will ask you to complete a questionnaire about each output that asks you to briefly summarize evidence of their technical quality, how they advance knowledge, their potential impact, and how they were disseminated.

Suggested Citation:"Appendix B: Rating Sheets/Questionnaires." National Research Council. 2012. Review of Disability and Rehabilitation Research: NIDRR Grantmaking Processes and Products. Washington, DC: The National Academies Press. doi: 10.17226/13285.
×

4.   The questionnaire will also ask you to respond to a brief set of questions at the grant level about approaches you used in managing your grant, how the grant may have generated new research and projects, and about your perspectives of key NIDRR processes which may influence grant results.

5.   We may also ask you to participate in a follow-up telephone or videoconference interview. If so, we would like to audio-record the discussion between you and the committee.

The attachments that follow this letter include, first, an informed consent form that explains what we will do with the results of the evaluation and what steps we will take to protect the confidentiality of the evaluation results specific to your grant. The second attachment is the Grantee Questionnaire referred to above that provides instructions for:

•   identifying outputs to be reviewed,

•   completing the supplemental questions for each output, and

•   sending us your signed consent forms, your outputs, and your completed questionnaire.

Please note that we are sending this package to you in electronic and hard copy form.

We hope that you will decide to participate. If you have any questions please don’t hesitate to contact one of us using the information below.

Sincerely,

Jeanne Rivard, Ph.D., Co-Study Director

The National Academies

National Research Council

500 Fifth Street, NW

Washington, DC 20001

Phone: 202-334-2967

E-mail:jrivard@nas.edu

Mary Ellen O’Connell, Co-Study Director

The National Academies

National Research Council

500 Fifth Street, NW

Washington, DC 20001

Phone: 202-334-2607

E-mail: moconnell@nas.edu

Suggested Citation:"Appendix B: Rating Sheets/Questionnaires." National Research Council. 2012. Review of Disability and Rehabilitation Research: NIDRR Grantmaking Processes and Products. Washington, DC: The National Academies Press. doi: 10.17226/13285.
×

2B: INFORMED CONSENT FORM FOR NIDRR GRANTEES

What the study is about: An expert committee of the National Research Council of the National Academies, in Washington, DC is developing an evaluation framework that will be used to: (1) review NIDRR’s priority setting and peer review processes; and (2) review the quality of grantee “outputs” for a sample of grants that represent the NIDRR portfolio. [“Peer review” refers to a process in which experts review the merits of a grant application in considering whether it should be funded. “Outputs” are publications, measures, intervention protocols, devices, or information resources that are produced as part of a grant.] The committee will also assess the design and implementation of the evaluation process and make recommendations for additional evaluation cycles that may be performed subsequent to this effort.

What we will ask you to do: We would like to invite you to participate in the evaluation.

Your participation will involve:

1.   Having outputs produced under your grant peer reviewed through a quality assessment by an expert panel.

2.   To conduct the review, we would like to examine a copy of the actual outputs and to review any documentation that you may have about the outputs.

3.   We will ask you to complete a questionnaire about the outputs to assist in assessing their technical quality, the extent to which they advance knowledge, their potential impact, and if applicable, their dissemination.

4.   We may ask you to participate in a follow-up telephone or videoconference interview where we would audio-record the discussion between you and the committee.

5.   We will also ask you to respond to a brief set of questions at the grant level about your grant management, the generation of new research and projects, and about key NIDRR processes which may influence grant results.

Taking part is voluntary. Your participation is completely voluntary. You can choose not to answer some of the questions, and there will be no consequences.

Benefits and risks:

Benefits: By taking part in the evaluation you will provide information that may help NIDRR improve its research portfolio for the benefit of persons with disabilities.

Risks: Because NIDRR has funded some of your research and development activities, you may feel uncomfortable having your grant’s outputs formally rated in the study, or in sharing your opinions and perspectives on NIDRR’s key management processes. You might feel that this could be a risk to your future grant funding. We want you to know, however, that we will take every step necessary to protect your confidentiality and minimize this risk.

Your answers will be confidential. In the final evaluation report, we will briefly describe your grant and the outputs that were reviewed, by their titles and grantee institutions. Your name as the Principal Investigator, or other investigators on your team, will not be used. In most cases we expect that you have already placed information about the outputs in the public domain through publications, presentations at conferences, and through NIDRR’s National Rehabilitation Information Center (NARIC) website.

However, distinct from these descriptions will be your responses on the Grantee Questionnaire and the committee’s quality ratings of your outputs. For analysis and reporting, these narrative and quantitative data will be de-identified and aggregated across all outputs and all grants. A research identification number will be used to track grants and their specific outputs. Outputs may be analyzed by categories,

Suggested Citation:"Appendix B: Rating Sheets/Questionnaires." National Research Council. 2012. Review of Disability and Rehabilitation Research: NIDRR Grantmaking Processes and Products. Washington, DC: The National Academies Press. doi: 10.17226/13285.
×

such as output type (e.g., publications, tools, technology, information products); quality criteria assessed (e.g., technical quality, knowledge advancement, potential impact); or program funding type (e.g., center grant, field initiated grant, training grant, etc.). If your grant or your output represents one of a kind and there is a risk of identifying you because of this, your data will be aggregated with another larger group where identification will not be a risk.

Every effort will be made to protect the confidentiality of the information that you provide. The Study Director will keep a list linking the grant and output research ID numbers with that output’s identifying information (institution, grant title). This list, along with the data collected, will be stored securely at the National Research Council, and will be accessible only by the Study personnel. If a telephone or videoconference interview is convened to gather additional follow-up information, the transcription of audio-recorded interviews will be combined in a dataset with the interviews of all of the other respondents, then analyzed for common themes across the interviews. The audiotapes, transcriptions, grantee questionnaires, committee ratings, and other raw data collected will be destroyed at the end of the study when the report is released.

Compensation: There is no compensation for participating in the evaluation.

If you have any questions: The Co-Study Director of the evaluation and contact for questions is Jeanne Rivard, Ph.D. If you have any questions about this consent form or the study, she can be contacted by phone at: 202-334-2697, or by e-mail at: jrivard@nas.edu.

If you have any questions, comments, or concerns about taking part in this study, first talk to Dr. Rivard above. If for any reason you do not want to do this, or you still have concerns after doing so, you may contact the Institutional Review Board (IRB) of the National Academy of Sciences (NAS), which reviewed and approved the study plans and this consent form. You can reach the chair of the IRB by contacting Ronald D. Taylor, Human Protections Administrator, by telephone at 202-334-1659 or you may write to him at the National Academy of Sciences; Room 1026; 500 Fifth Street, NW; Washington, DC 20001.

Statement of Consent: I have read the above information, and have received answers to any questions I asked. I consent to take parting in the study.

Your Signature_______________________________________ Date__-__-__

Your Name (printed) _____________________________________________

In addition to agreeing to participate, if there is a follow-up interview, I also consent to having it taperecorded.

Your Signature_______________________________________ Date__-__-__

Suggested Citation:"Appendix B: Rating Sheets/Questionnaires." National Research Council. 2012. Review of Disability and Rehabilitation Research: NIDRR Grantmaking Processes and Products. Washington, DC: The National Academies Press. doi: 10.17226/13285.
×

2C: GRANTEE QUESTIONNAIRE

Grant Award Number:

Grant Title:

Grantee:

Program Mechanism:

Grant End Date:

INSTRUCTIONS TO GRANTEES

This questionnaire has been designed to obtain information to assist the Committee in assessing the quality of your grant’s outputs. NIDRR has provided to the Committee and the National Research Council (NRC) staff copies of its Annual Performance Report (APR) database and your last APR and your final APR. As you will see in certain places on the questionnaire we have inserted information from your APR to facilitate your completion of the questionnaire (e.g., Table 1 lists your research and development projects, and Table 2 lists your outputs reported in the APR). Where this information is in error, we would appreciate your pointing the errors out to us and correcting it; and/or updating the information as needed.

The questionnaire is divided into the following three parts:

Part A. Nominating Outputs for Review. This section asks you to nominate, for the Committee’s review, the “top 2” outputs for each of your projects that best reflect your grant's achievements (Table 1 below). The Committee would prefer to review one publication and one other type of output for each project. However if you only have publications, please nominate these as your “top 2.”

To make this process easier, the NRC staff has populated a list of the outputs (Table 2 below) that were reported for your grant in the APR. You could select the top 2 outputs from this table. However you are not constrained to select from this list if there are other outputs that you think better reflect your grant’s achievements.

For Committee review we are requesting materials and information regarding the actual outputs selected as the top 2 for each project.

•   For publications, the material for review would be pdf copies of each article.

•   For the other outputs, materials for review would include:

image   Electronic or hard copies of the measures, tools, intervention protocols, manuals; or links to websites, pictures or other graphic representations of tools or devices that have been produced.

image   An abstract or summary of each output, which briefly describes:

image   what the output is,

image   its purpose,

Suggested Citation:"Appendix B: Rating Sheets/Questionnaires." National Research Council. 2012. Review of Disability and Rehabilitation Research: NIDRR Grantmaking Processes and Products. Washington, DC: The National Academies Press. doi: 10.17226/13285.
×

image   target audience,

image   methods, and

image   how the output fits into the overall goals and objectives of the project and grant

Part B. Additional Questions about Outputs. For each of the outputs you nominated for review, the Committee has a series of questions related to their technical quality, how they may have advanced knowledge, their potential impact, and their dissemination. We ask that you complete the Part B section for each output. If the answers to certain questions would be the same across different outputs, you can note this and cut and paste responses from earlier output forms to other ones. Please make your responses brief, but as specific and quantitative as possible.

Part C. Grant-level Questions. The questionnaire will also contain a few other items asking about how you managed your grants to produce the highest quality outputs, how your grant’s results may have generated new projects, and how key NIDRR processes influence results.

Your complete package of materials will contain:

•   Your signed informed consent form

•   Copies of your publications and other outputs (e.g., measures, tools, intervention protocols, manuals, links to websites, pictures or other graphic representations of devices that have been produced)

•   Your completed Grantee Questionnaire

Please send these materials by ___DATE___ to:

Matt McDonough

The National Academies

National Research Council

500 Fifth Street, NW

WS 1134

Washington, DC 20001

We are enclosing an addressed FedEx form that can be used when mailing your package of materials to us. We estimated a shipment cost that would cover a weight up to 10 lbs. (e.g., for large center grants or devices). If your package weighs more than this, FedEx will charge us the correct amount.

If your package is light and you want to send it electronically, you could e-mail it to Matt at mmcdonough@nas.edu. However, you would need to scan your signed consent form, and send that in a pdf document.

Suggested Citation:"Appendix B: Rating Sheets/Questionnaires." National Research Council. 2012. Review of Disability and Rehabilitation Research: NIDRR Grantmaking Processes and Products. Washington, DC: The National Academies Press. doi: 10.17226/13285.
×

Part A. Nominating Outputs for Review

When referring to “outputs,” we are using the four NIDRR categories of outputs as defined in NIDRR’s Annual Performance Report, which include: (a) Publications; (b) Tools, Measures, and Intervention Protocols; (c) Technology Products and Devices; and (d) Informational Products.

Per the instructions for nominating outputs for review, please record your nominations for your “top 2” outputs for each of your projects in Table 1 below. (Reminder “top 2” refers to those that best reflect your grant's achievements). As you can see the NRC staff has already populated Table 1 with the names of your research and development projects from data in the APR. Table 2, which follows, contains a list of outputs from which you can cut and paste into Table 1 below. Please identify any errors in this information that we have provided from your APR and correct it as needed.

TABLE 1. Projects and Nominated Outputs

# Names of R&D Projects in Grant Names of Top 2 Outputs for Each ProjectOutputs to be inserted by grantee from Table 2 below or add others as needed
Research Projects
R1 1.

2.
Development Projects
D1 Dissemination 1.

2.

INSERT GRANTEES' PROJECTS TABLE HERE

The table below lists all of the publications and other outputs that were listed in the APR data provided by NIDRR. Please use this table below in selecting your top 2 outputs for each project. (You can cut and paste from Table 2 into Table 1.) However you are not constrained to select only from this list if there are other outputs that you think better reflect your grant’s achievements.

TABLE 2. List of Outputs from APR

Type of Output Title of Output
publications (title)
tool (title)
Suggested Citation:"Appendix B: Rating Sheets/Questionnaires." National Research Council. 2012. Review of Disability and Rehabilitation Research: NIDRR Grantmaking Processes and Products. Washington, DC: The National Academies Press. doi: 10.17226/13285.
×

Part B. Additional Questions About Outputs

Please use one copy of this form for each publication and each other output for the “top 2” outputs that you selected for each project in Part A above, and provide the following information. Please make your responses brief, but as specific and quantitative as possible. If you consider the criterion not to be applicable to your output, please explain. (Please note that an electronic copy of the questionnaire was included in the email version of this package.)

Name of Output:____________________________________________

B1. Technical Quality of Output

In the space below, please describe examples of the technical quality of your output, such as:

•   The particular approach or methodology used in developing your output

•   Relevant peer recognition such as peer reviews or evaluations, peer endorsements, invitations to present at professional forums or conferences, invitations to present testimony, receipt of awards or honors, etc.

•   Receipt of a patent, FDA approval, or use of your output in standards development

•   Evidence of the usability and accessibility of the output

image

B2. Advancement of Knowledge

Please use the space below to describe how this output has advanced knowledge. To structure your response, include points such as:

•   What the importance of your original question or issue was

•   How the output has advanced knowledge in arenas, such as:

image   making discoveries

image   providing new information

image   establishing theories, measures, and methods

image   closing gaps in the knowledge base

image   developing new interventions, products, technology, and environmental adaptations

image

Suggested Citation:"Appendix B: Rating Sheets/Questionnaires." National Research Council. 2012. Review of Disability and Rehabilitation Research: NIDRR Grantmaking Processes and Products. Washington, DC: The National Academies Press. doi: 10.17226/13285.
×

B3. Potential Impact

In the space below, please briefly describe evidence of your outputs’ potential (or actual) impact on the following audiences, as relevant to your output:

•   Science (e.g., new areas of inquiry, methodology, etc.)

•   People with disabilities: health, quality of life, participation

•   Provider practice

•   Health and social systems

•   Social and health policy

•   Private sector/commercialization

•   Capacity building in the field of rehabilitation and disability research and development (e.g., scientists, graduate students, etc.)

•   Other

Include information about how this potential impact was tested, and what the results were.

image

B4. Dissemination of Outputs

In the space below please provide evidence of your dissemination efforts for this output. Describe this for publications if you have made any effort beyond those of the sponsor of the publication (journal, book, proceedings, etc.). Please include important aspects of dissemination such as:

•   Stage and scope (e.g., local, regional, national) of dissemination

•   Dissemination activities

•   Identification and tailoring of materials for reaching different audience/user types

•   Collaboration with audience/users in identifying content and medium needs/preferences

•   Delivery of information through multiple media types and sources for optimal reach and accessibility

•   Evaluation of your dissemination efforts and impacts

image

Suggested Citation:"Appendix B: Rating Sheets/Questionnaires." National Research Council. 2012. Review of Disability and Rehabilitation Research: NIDRR Grantmaking Processes and Products. Washington, DC: The National Academies Press. doi: 10.17226/13285.
×

Part C. Grant-level Questions

Please respond to these final questions for your overall grant, not by each output specifically as in Section B.

C1. In the space below please describe what types of planning, project management, and budgetary processes were used to promote high quality outputs. In your statement consider the following types of questions:

•   Which processes were useful and how? How could they be improved?

•   Did you dedicate funds for quality assurance activities?

•   How did you track progress and spending against your original plans for the grant?

•   If grants or projects were jointly funded by NIDRR and other extramural or intramural sources, how did you ensure that NIDRR resources were used exclusively for NIDRRfunded activities?

image

C2. Have the results of the research and development outputs from this grant, or prior NIDRR grants, been used to inform the development of new grant applications or other kinds of projects?

No ____

Yes____

If yes, please use the space below to briefly describe what new grant applications, other projects, funding opportunities, or collaborations have emerged.

image

C3. Please share any perspectives you may have about how NIDRR’s key processes (e.g., priority setting, peer review, and/or grants management) influence results, such as successful grants and high quality outputs.

image

Suggested Citation:"Appendix B: Rating Sheets/Questionnaires." National Research Council. 2012. Review of Disability and Rehabilitation Research: NIDRR Grantmaking Processes and Products. Washington, DC: The National Academies Press. doi: 10.17226/13285.
×

2D: COMMITTEE MEMBER REVIEW PROCEDURES FOR SUMMATIVE EVALUATION

A. Review Subgroups: Each subgroup that will be reviewing outputs will be composed of five Committee members. For each output one committee member will be assigned as the primary reviewer; the remaining four committee members will be secondary reviewers.

B. Output Rating Procedures:

1. All reviewers will independently rate outputs using the following quality criteria (Dimensions of these criteria are shown on the attached rating sheet.):

•   Technical quality of output

•   Advancement of knowledge or the field (research, practice, or policy)

•   Likely impact

•   Dissemination

The following scale will be used for rating the outputs:

image

2. The rating will be based on review of hard copy and electronic materials (i.e., articles/descriptive information about output and questionnaire responses) prior to the subcommittee meeting.

3. The grantee’s final summary APR, and a list of all outputs reported over the course of the project, is provided for contextual purposes. The APR also will be used to inform an overall, qualitative grant-level assessment.

4. Multiple outputs of one grant will generally be rated independently of each other. However, in some cases outputs may be rated as a pair with one score applied. This could occur when one output is a derivative or different expression of another output, and when the PI responses to criterion questions are basically the same. Examples of these include:

•   A manual describing a device (1) and a patent of the device (2)

•   A publication describing how a new technology for assessing a condition can be applied in disability rehabilitation (1) and a description of the technology itself (2)

•   A software application (e.g., map reader for persons with visual impairments) (1) and web-based method for individualizing the software for users (2)

Suggested Citation:"Appendix B: Rating Sheets/Questionnaires." National Research Council. 2012. Review of Disability and Rehabilitation Research: NIDRR Grantmaking Processes and Products. Washington, DC: The National Academies Press. doi: 10.17226/13285.
×

5. The meeting will be structured as follows:

•   The primary reviewer will open discussion of each output by presenting a brief summary of the output and then his/her rationale for rating each relevant criterion (up to four) plus the overall score.

•   Secondary reviewers will then present their ratings for each output and a brief rationale.

•   Using the same criteria, the subgroup will then develop consensus group ratings for each output. Discussion will be facilitated by the subgroup chair. If there is a subgroup member with a significantly divergent view, his/her score and rationale will be captured separately.

•   Staff will document discussion points that lead to the consensus group ratings and will record the subgroup’s rationale for each criterion, the overall rating, and the grant performance rating in a brief narrative.

•   At the end of the review of each output, the individual subgroup members’ rating sheets will be gathered.

C. Grant Assessment

Once all outputs of an individual grant are reviewed, the subgroup will consider and rate the grant’s overall performance. The outputs reviewed had been identified by the grant’s Principal Investigator as the “top” two outputs per project, which best reflected the grant’s achievements. Taking into consideration this designation, the consensus group ratings of the entire set of outputs, and the grant’s overall purpose and objectives (using the grant’s APR), the subgroup will assign a grant performance rating using the same 7-point scale. These grant-level ratings and their rationale will also be documented by staff.

Suggested Citation:"Appendix B: Rating Sheets/Questionnaires." National Research Council. 2012. Review of Disability and Rehabilitation Research: NIDRR Grantmaking Processes and Products. Washington, DC: The National Academies Press. doi: 10.17226/13285.
×

2E: COMMITTEE MEMBER RATING SHEET OF THE QUALITY OF OUTPUTS

Committee Member:_______________

Output ID:______________________



Grantee ID:___________________________________________________

Date of Review:________________________________________________

Output to Be Reviewed:

                                                                                                                                      To be completed by NRC staff

Output Title:______________________________

Research Output: ______ Development Output: ________

Type and Subtype of Output (marked below):
Type of Publication   Type of Tool, Measure, or Intervention Protocol   Type of Technology Product or Device   Type of Informational Product  
1. abstract   1. checklist   1. industry standards/guidelines   1. training manuals/curriculaa  
2. book   2. survey or interview schedule   2. software or netware   2 fact sheets  
3. book chapter   3. diagnostic or assessment instrument   3. invention   3. newsletters  
4. journal article   4. outcome measure   4. patent, license, or disclosures   4. audiovisual materials  
5. proceedings   5. intervention protocol or program   5. working prototype   5. marketing tools  
6. technical   6. statistical technique   6. product evaluated or field tested   6. educational aids  
7. web journal   7. database   7. product transferred to industry for potential commercialization   7. websites or other internet sites  
8. other   8. other   8. product in marketplace   8. other  
Suggested Citation:"Appendix B: Rating Sheets/Questionnaires." National Research Council. 2012. Review of Disability and Rehabilitation Research: NIDRR Grantmaking Processes and Products. Washington, DC: The National Academies Press. doi: 10.17226/13285.
×

Quality Criteria and Dimensions

For each criterion provide one rating using the scale below:

poor   Good   Excellent

1   2   3   4   5   6   7

Criteria and Dimensions Score
Technical Quality of Output
• Applying standards of science and technology
•Appropriate methodology (quantitative or qualitative design and statistics)
• Accessibility, usability, etc.

Score Rationale:
 
Advancement of Knowledge or the Field (research, practice, or policy as relevant)
• Science. Establishment of methods, tools, theory
•Ken information
•Closing an identified gap
•New technology
•Innovative or novel

Score Rationale:
 
Likely or Demonstrated Impact On:
• Science (impact factor, citations)
• Consumers (people with disabilities: health, quality of life, participation)
•Provider practice
• Health and social system
• Social and health policy
• Private sector commercialization
• Other

Score Rationale:
 
Dissemination
•Identification and tailoring of materials for reaching different audience user types
•Collaboration with audience users in identifying content and medium needs-preferences
•Delivery of information through multiple media types and sources foi optimal reach and accessibility
•Evaluation of dissemination efforts and impacts
•Commercialization patenting of devices, if applicable

Score Rationale:
 
Overall Score


Score Rationale:
 
Suggested Citation:"Appendix B: Rating Sheets/Questionnaires." National Research Council. 2012. Review of Disability and Rehabilitation Research: NIDRR Grantmaking Processes and Products. Washington, DC: The National Academies Press. doi: 10.17226/13285.
×
Page 305
Suggested Citation:"Appendix B: Rating Sheets/Questionnaires." National Research Council. 2012. Review of Disability and Rehabilitation Research: NIDRR Grantmaking Processes and Products. Washington, DC: The National Academies Press. doi: 10.17226/13285.
×
Page 306
Suggested Citation:"Appendix B: Rating Sheets/Questionnaires." National Research Council. 2012. Review of Disability and Rehabilitation Research: NIDRR Grantmaking Processes and Products. Washington, DC: The National Academies Press. doi: 10.17226/13285.
×
Page 307
Suggested Citation:"Appendix B: Rating Sheets/Questionnaires." National Research Council. 2012. Review of Disability and Rehabilitation Research: NIDRR Grantmaking Processes and Products. Washington, DC: The National Academies Press. doi: 10.17226/13285.
×
Page 308
Suggested Citation:"Appendix B: Rating Sheets/Questionnaires." National Research Council. 2012. Review of Disability and Rehabilitation Research: NIDRR Grantmaking Processes and Products. Washington, DC: The National Academies Press. doi: 10.17226/13285.
×
Page 309
Suggested Citation:"Appendix B: Rating Sheets/Questionnaires." National Research Council. 2012. Review of Disability and Rehabilitation Research: NIDRR Grantmaking Processes and Products. Washington, DC: The National Academies Press. doi: 10.17226/13285.
×
Page 310
Suggested Citation:"Appendix B: Rating Sheets/Questionnaires." National Research Council. 2012. Review of Disability and Rehabilitation Research: NIDRR Grantmaking Processes and Products. Washington, DC: The National Academies Press. doi: 10.17226/13285.
×
Page 311
Suggested Citation:"Appendix B: Rating Sheets/Questionnaires." National Research Council. 2012. Review of Disability and Rehabilitation Research: NIDRR Grantmaking Processes and Products. Washington, DC: The National Academies Press. doi: 10.17226/13285.
×
Page 312
Suggested Citation:"Appendix B: Rating Sheets/Questionnaires." National Research Council. 2012. Review of Disability and Rehabilitation Research: NIDRR Grantmaking Processes and Products. Washington, DC: The National Academies Press. doi: 10.17226/13285.
×
Page 313
Suggested Citation:"Appendix B: Rating Sheets/Questionnaires." National Research Council. 2012. Review of Disability and Rehabilitation Research: NIDRR Grantmaking Processes and Products. Washington, DC: The National Academies Press. doi: 10.17226/13285.
×
Page 314
Suggested Citation:"Appendix B: Rating Sheets/Questionnaires." National Research Council. 2012. Review of Disability and Rehabilitation Research: NIDRR Grantmaking Processes and Products. Washington, DC: The National Academies Press. doi: 10.17226/13285.
×
Page 315
Suggested Citation:"Appendix B: Rating Sheets/Questionnaires." National Research Council. 2012. Review of Disability and Rehabilitation Research: NIDRR Grantmaking Processes and Products. Washington, DC: The National Academies Press. doi: 10.17226/13285.
×
Page 316
Suggested Citation:"Appendix B: Rating Sheets/Questionnaires." National Research Council. 2012. Review of Disability and Rehabilitation Research: NIDRR Grantmaking Processes and Products. Washington, DC: The National Academies Press. doi: 10.17226/13285.
×
Page 317
Suggested Citation:"Appendix B: Rating Sheets/Questionnaires." National Research Council. 2012. Review of Disability and Rehabilitation Research: NIDRR Grantmaking Processes and Products. Washington, DC: The National Academies Press. doi: 10.17226/13285.
×
Page 318
Suggested Citation:"Appendix B: Rating Sheets/Questionnaires." National Research Council. 2012. Review of Disability and Rehabilitation Research: NIDRR Grantmaking Processes and Products. Washington, DC: The National Academies Press. doi: 10.17226/13285.
×
Page 319
Suggested Citation:"Appendix B: Rating Sheets/Questionnaires." National Research Council. 2012. Review of Disability and Rehabilitation Research: NIDRR Grantmaking Processes and Products. Washington, DC: The National Academies Press. doi: 10.17226/13285.
×
Page 320
Suggested Citation:"Appendix B: Rating Sheets/Questionnaires." National Research Council. 2012. Review of Disability and Rehabilitation Research: NIDRR Grantmaking Processes and Products. Washington, DC: The National Academies Press. doi: 10.17226/13285.
×
Page 321
Suggested Citation:"Appendix B: Rating Sheets/Questionnaires." National Research Council. 2012. Review of Disability and Rehabilitation Research: NIDRR Grantmaking Processes and Products. Washington, DC: The National Academies Press. doi: 10.17226/13285.
×
Page 322
Suggested Citation:"Appendix B: Rating Sheets/Questionnaires." National Research Council. 2012. Review of Disability and Rehabilitation Research: NIDRR Grantmaking Processes and Products. Washington, DC: The National Academies Press. doi: 10.17226/13285.
×
Page 323
Suggested Citation:"Appendix B: Rating Sheets/Questionnaires." National Research Council. 2012. Review of Disability and Rehabilitation Research: NIDRR Grantmaking Processes and Products. Washington, DC: The National Academies Press. doi: 10.17226/13285.
×
Page 324
Suggested Citation:"Appendix B: Rating Sheets/Questionnaires." National Research Council. 2012. Review of Disability and Rehabilitation Research: NIDRR Grantmaking Processes and Products. Washington, DC: The National Academies Press. doi: 10.17226/13285.
×
Page 325
Suggested Citation:"Appendix B: Rating Sheets/Questionnaires." National Research Council. 2012. Review of Disability and Rehabilitation Research: NIDRR Grantmaking Processes and Products. Washington, DC: The National Academies Press. doi: 10.17226/13285.
×
Page 326
Suggested Citation:"Appendix B: Rating Sheets/Questionnaires." National Research Council. 2012. Review of Disability and Rehabilitation Research: NIDRR Grantmaking Processes and Products. Washington, DC: The National Academies Press. doi: 10.17226/13285.
×
Page 327
Suggested Citation:"Appendix B: Rating Sheets/Questionnaires." National Research Council. 2012. Review of Disability and Rehabilitation Research: NIDRR Grantmaking Processes and Products. Washington, DC: The National Academies Press. doi: 10.17226/13285.
×
Page 328
Suggested Citation:"Appendix B: Rating Sheets/Questionnaires." National Research Council. 2012. Review of Disability and Rehabilitation Research: NIDRR Grantmaking Processes and Products. Washington, DC: The National Academies Press. doi: 10.17226/13285.
×
Page 329
Next: Appendix C: Acronyms »
Review of Disability and Rehabilitation Research: NIDRR Grantmaking Processes and Products Get This Book
×
Buy Paperback | $64.00 Buy Ebook | $49.99
MyNAP members save 10% online.
Login or Register to save!
Download Free PDF

The National Institute on Disability and Rehabilitation Research (NIDRR) is the principal federal agency supporting applied research, training, and development to improve the lives of individuals with disabilities. NIDRR's mission is to generate new knowledge and promote its effective use in improving the ability of persons with disabilities to perform activities of their choice in the community, as well as to expand society's capacity to provide full opportunities and accommodations for its citizens with disabilities.

NIDRR prides itself on being proactive in establishing program performance measures and developing accountability data systems to track the progress of its grantees. An electronic annual reporting system is used to collect data from grantees on many aspects of grant operation and outputs. Various formative and summative evaluation approaches have been used to assess the quality of the performance and results of the agency's research portfolio and its grantees. Prompted by the need to provide more data on its program results, in 2009 NIDRR requested that the National Research Council (NRC) conduct an external evaluation of some of the agency's key processes and assess the quality of outputs produced by NIDRR grantees (National Institute on Disability and Rehabilitation Research, 2009a). Review of Disability and Rehabilitation Research presents the results of that evaluation.

  1. ×

    Welcome to OpenBook!

    You're looking at OpenBook, NAP.edu's online reading room since 1999. Based on feedback from you, our users, we've made some improvements that make it easier than ever to read thousands of publications on our website.

    Do you want to take a quick tour of the OpenBook's features?

    No Thanks Take a Tour »
  2. ×

    Show this book's table of contents, where you can jump to any chapter by name.

    « Back Next »
  3. ×

    ...or use these buttons to go back to the previous chapter or skip to the next one.

    « Back Next »
  4. ×

    Jump up to the previous page or down to the next one. Also, you can type in a page number and press Enter to go directly to that page in the book.

    « Back Next »
  5. ×

    Switch between the Original Pages, where you can read the report as it appeared in print, and Text Pages for the web version, where you can highlight and search the text.

    « Back Next »
  6. ×

    To search the entire text of this book, type in your search term here and press Enter.

    « Back Next »
  7. ×

    Share a link to this book page on your preferred social network or via email.

    « Back Next »
  8. ×

    View our suggested citation for this chapter.

    « Back Next »
  9. ×

    Ready to take your reading offline? Click here to buy this book in print or download it as a free PDF, if available.

    « Back Next »
Stay Connected!