Below are the first 10 and last 10 pages of uncorrected machine-read text (when available) of this chapter, followed by the top 30 algorithmically extracted key phrases from the chapter as a whole.
Intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text on the opening pages of each chapter.
Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.
Do not use for reproduction, copying, pasting, or reading; exclusively for search engines.
OCR for page 305
Appendix B
Rating Sheets/Questionnaires
This appendix contains the following rating sheets and questionnaires
used by the committee for its external evaluation of NIDRR and its grantees:
1. Process Evaluation Measures Page Number
A. Web-Based Questionnaire for NIDRR
Stakeholders 306
B. Web-Based Questionnaire for NIDRR
Peer Reviewers 311
2. Summative Evaluation Measures
A. Letter to Grantees Inviting Participation 316
B. Informed Consent Form for NIDRR Grantees 318
C. Grantee Questionnaire 320
D. Committee Member Review Procedures for
Summative Evaluation 326
E. Committee Member Rating Sheet of the Quality of
Outputs 328
305
OCR for page 306
306 REVIEW OF DISABILITY AND REHABILITATION RESEARCH
1A: WEB-BASED QUESTIONNAIRE FOR NIDRR STAKEHOLDERS
Informed Consent Form
What the study is about: An expert committee of the National Research Council, of The National
Academies, in Washington, DC is conducting an evaluation of some of the activities of the National
Institute on Disability and Rehabilitation Research (NIDRR). The committee will be (1) reviewing
NIDRR’s priority setting and peer review processes; and (2) reviewing the quality of grantee
“outputs” for a sample of NIDRR grants. [“Peer review” refers to a process in which experts review
grant applications and make recommendations about whether they should be funded. “Outputs” are
publications, measures, intervention protocols, devices, and information resources that are produced as
part of a grant.]
What we will ask you to do: We are inviting you to take part in the first part of the evaluation—
specifically, the review of NIDRR’s priority setting processes. We will ask you a set of questions to
help us understand these processes and how they may affect NIDRR’s work. The questionnaire will
take about 15 to 20 minutes to complete. Your participation is completely voluntary. You can choose
not to answer some of the questions or to stop at any point, and there will be no consequences.
Benefits and risks:
Benefits: By taking part in the evaluation you will provide information that may help NIDRR improve
its research portfolio for the benefit of persons with disabilities.
Risks and protections: You might be concerned that the information you share will not be kept private.
However, we want to assure you that all of your comments will be kept confidential. Study results will
be presented only in combined form, with no individual person or organization being identified. Your
name will not be attached to your answers. A research identification number will be used instead. The
Study Director will keep a list linking your name with your number. This list, along with the data
collected, will be stored securely at the National Research Council. Only the study personnel will have
access to the master list, and only for research purposes.
Compensation: There is no compensation for taking part in the evaluation.
If you have any questions: Please contact the Senior Program Officer, Dr. Jeanne Rivard, if you have
any questions about this consent form or the study. She can be contacted by phone at: 202-334-2967,
or by email at: jrivard@nas.edu.
If you have any questions, comments, or concerns about taking part in this study, first contact Dr.
Rivard. If for any reason you do not want to contact her, or you still have concerns after doing so, you
may contact the Institutional Review Board (IRB) of the National Academy of Sciences (NAS), which
reviewed and approved the study plans and this consent form. You can reach the chair of the IRB,
Ronald D. Taylor, Human Protections Administrator, by telephone at 202-334-1659 or you may write
to him at the National Academy of Sciences; Room 1026; 500 Fifth Street, NW; Washington, DC
20001.
Statement of Consent:
I have read the above information, understand it, and agree to take part in the study.
I do not agree to take part in the study.
B-3
OCR for page 307
307
APPENDIX B
Web-Based Questionnaire for NIDRR Stakeholders
Please respond as a representative of your organization so that your answers will reflect how your
organization relates to NIDRR.
1. Although your organization has multiple functions, select from the list below the one type
that most closely matches your own organization.
Professional association
Advocacy organization
Service provider
Funder of research and/or development grants
Technical assistance and dissemination
Commercial manufacturer/distributor
Other (Please specify)
2. To what extent is your organization familiar with the National Institute on Disability and
Rehabilitation Research (NIDRR)?
1 2 3 4 5
Not at all Somewhat Very much
3. Please check all of the ways in which your organization has interacted with NIDRR since
January 1, 2005.
Information Exchange
Have used NIDRR website or related information sources to search for information, tools, or
resources for working with consumers (non grant information)
Have used NIDRR website or related resources to search for information about grants
Have spoken with NIDRR staff in person or on the phone about specific professional issues
Have attended NIDRR trainings, workshops, or conferences
Other (Please specify in box below)
Funding
Have received funding from NIDRR for research & development grants
Have received funding from NIDRR for training, knowledge transfer, or dissemination grants
Have received funding from NIDRR for activities other than grants (e.g., contract or agreements to
conduct an evaluation, provide expert consultation, or write a paper, etc.)
Have applied for funding, but did not receive award
Other (Please specify in box below)
Collaboration
Have collaborated with NIDRR as a member of a consortium or professional association
Have participated in general planning and special purpose meetings convened by NIDRR
Have coordinated activities to support joint priorities and/or to avoid duplication
Other (Please specify in box below)
Have not interacted with NIDRR on any of the above types of information, funding, or
collaboration activities.
4. Please use the space below to comment on the above, or any other types of interactions, your
organization has had with NIDRR. (Click in the upper left-hand corner to enter text.)
B-4
OCR for page 308
308 REVIEW OF DISABILITY AND REHABILITATION RESEARCH
5. Has your organization had opportunities to review and comment on NIDRR's long range
plan or funding priorities?
Yes
No
Don’t know
6. How did your organization find out about opportunities to review and comment on NIDRR's
long range plan or funding priorities?
Federal Register
Professional list serve
University grants source
Professional newspaper
Personal invitation to our organization from NIDRR
Other (please specify):
Don’t know
7. Since January 1, 2005, has anyone from your organization submitted comments to NIDRR
about its long-range plan?
Yes
No
Don’t know
8. Since January 1, 2005, has anyone from your organization submitted comments to NIDRR
about its specific funding priorities (e.g., grant announcements) published in the Federal
Register?
Yes
No
Don’t know
B-5
OCR for page 309
309
APPENDIX B
9. In your opinion, to what extent are NIDRR’s long range planning and priority setting
processes (Please check the boxes beneath the responses—select only one per row):
Not at all Somewhat Very Don’t
Much know
Transparent 1 2 3 4 5
Publicized 1 2 3 4 5
Relevant to your 1 2 3 4 5
organization
Responsive to emerging
1 2 3 4 5
issues in disability
rehabilitation and research
Welcoming of stakeholder 1
2 3 4 5
feedback
Responsive to stakeholder 1 2 3 4 5
feedback
10. In your opinion, how do NIDRR’s long range planning and priority setting processes
compare with those of other federal research agencies?
NIDRR’s processes are weaker than other agencies’.
NIDRR’s processes are about the same as other agencies’.
NIDRR’s processes are stronger than other agencies’.
Don’t know
11. How well are the grants that NIDRR funds advancing the field in a direction that is
beneficial for your organization, and the members and consumers that you represent? (Please
select only one response.)
1 2 3 4 5
Not at all Somewhat Very much Don’t know Not
Applicable
12. What types of grants have served your organization, and the members and consumers that
you represent? (Click in the upper left-hand corner to enter text.)
B-6
OCR for page 310
310 REVIEW OF DISABILITY AND REHABILITATION RESEARCH
13. To what extent are the products of the grants that NIDRR funds (e.g., publications,
websites, training or training materials, tools, devices, measures, interventions, etc.) used by
your organization? (Please select one response.)
1 2 3 4 5
Not at all Somewhat Very much Don’t know Not
Applicable
14. What are the three most important NIDRR-funded products used by your organization?
(Click in the upper left-hand corner to enter text.)
15. What three things might NIDRR do to enhance its long range planning and priority setting
processes? (Click in the upper left-hand corner to enter text.)
Thank you very much for taking the time to complete this questionnaire. We may be contacting
some respondents later to ask additional questions. If you would be willing to take part in brief
follow-up interviews, please check here.
Yes, it’s okay to contact me.
No, thank you. I’m finished!
Contact Information
(Click on the leftmost portion of the line to enter text.)
E-mail address:
Phone number:
B-7
OCR for page 311
311
APPENDIX B
1B: WEB-BASED QUESTIONNAIRE FOR NIDRR PEER REVIEWERS
Informed Consent Form
What the study is about: An expert committee of the National Research Council, of the National
Academies, in Washington, DC is conducting an evaluation of some of the activities of the National
Institute on Disability and Rehabilitation Research (NIDRR). The committee will be (1) reviewing
NIDRR’s priority setting and peer review processes; and (2) reviewing the quality of grantee
“outputs” for a sample of NIDRR grants. [“Peer review” refers to a process in which experts review
grant applications and make recommendations about whether they should be funded. “Outputs” are
publications, measures, intervention protocols, devices, and information resources that are produced as
part of a grant.]
What we will ask you to do: We are inviting you to take part in the first part of the evaluation—
specifically, the review of NIDRR’s peer review processes. We will ask you a set of questions to help
us understand these processes and how they may affect NIDRR’s work. The questionnaire will take
about 15 to 20 minutes to complete. Your participation is completely voluntary. You can choose not to
answer some of the questions or to stop at any point, and there will be no consequences.
Benefits and risks:
Benefits: By taking part in the evaluation you will provide information that may help NIDRR improve
its research portfolio for the benefit of persons with disabilities.
Risks and protections: You might be concerned that the information you share will not be kept private.
However, we want to assure you that all of your comments will be kept confidential. Study results will
be presented only in combined form, with no individual person or organization being identified. Your
name will not be attached to your answers. A research identification number will be used instead. The
Study Director will keep a list linking your name with your number. This list, along with the data
collected, will be stored securely at the National Research Council. Only the study personnel will have
access to the master list, and only for research purposes.
Compensation: There is no compensation for taking part in the evaluation.
If you have any questions: Please contact the Senior Program Officer, Dr. Jeanne Rivard, if you have
any questions about this consent form or the study. She can be contacted by phone at: 202-334-2967,
or by e-mail at: jrivard@nas.edu.
If you have any questions, comments, or concerns about taking part in this study, first contact Dr.
Rivard. If for any reason you do not want to contact her, or you still have concerns after doing so, you
may contact the Institutional Review Board (IRB) of the National Academy of Sciences (NAS), which
reviewed and approved the study plans and this consent form. You can reach the chair of the IRB,
Ronald D. Taylor, Human Protections Administrator, by telephone at 202-334-1659 or you may write
to him at the National Academy of Sciences; Room 1026; 500 Fifth Street, NW; Washington, DC
20001.
Statement of Consent:
I have read the above information, understand it, and agree to take part in the study.
I do not agree to take part in the study.
B-8
OCR for page 312
312 REVIEW OF DISABILITY AND REHABILITATION RESEARCH
Web-Based Questionnaire for NIDRR Peer Reviewers
1. On how many NIDRR peer review panels have you served since January 1, 2005? (Please
state the total number, such as 0, 1, 2, 3, 4, etc.):
(By panel, we mean every time a group of peer reviewers is convened to conduct a review,
not just membership on a given panel.)
2. Across the following program funding mechanisms, please check all of those for which you
have reviewed applications since January 1, 2005 (See NIDRR website for a description of these
mechanisms)
Disability and Rehabilitation Research Projects (DRRP) – General
Disability Business Technical Assistance Centers (DBTAC)
Traumatic Brain Injury Model Systems Centers
Burn Model Systems Centers
Spinal Cord Injury Model Systems Centers
Rehabilitation Research and Training Centers (RRTCs)
Rehabilitation Engineering Research Centers (RERCs)
Field Initiated Projects – Research Grants
Field Initiated Projects – Development Grants
Small Business Innovation Research (SBIR) – Phase I
Small Business Innovation Research (SBIR) – Phase II
DRRP - Knowledge Translation
DRRP – Section 21
(Switzer) Research Fellowship Program
Advanced Rehabilitation Research Training Projects
Don’t Remember
3. In your experience, how would you rate the following aspects of the NIDRR peer review
processes (Please check the boxes beneath the responses—select only one per row):
Poor Adequate Excellent DON’T NOT
KNOW APPLI-
CABLE
Quality of the training to prepare 1 2 3 4 5
you for the review
Adequacy of time for review of 1 2 3 4 5
materials before the meeting
Level of expertise of the peer 1 2 3 4 5
review panel members
Appropriateness of the evaluation
1 2 3 4 5
criteria to applications under
review
Clarity of the criteria when 1 2 3 4 5
applying them to applications
B-9
OCR for page 313
313
APPENDIX B
Poor Adequate Excellent DON’T NOT
KNOW APPLI-
CABLE
Appropriateness of scoring
1 2 3 4 5
system to applications under
review
Ease of applying scoring system 1 2 3 4 5
to applications
Thoroughness of the deliberation
1 2 3 4 5
(i.e., grant scoring and discussion)
during the meeting.
Use of reviewers’ time during the 1 2 3 4 5
panel meeting
Support and facilitation of the 1 2 3 4 5
review panel by NIDRR staff
Guidance in writing your 1 2 3 4 5
reviewer comments
Integrity of the peer review 1 2 3 4 5
process overall
Consistency in the overall quality
of the peer reviews across panels 2 3 4 5
1
(if you have served on three or
more panels).
4. Any additional comments you may have on these aspects of NIDRR's peer review processes
would be useful. Please use the space below. (Click in the upper left-hand corner to enter text.)
5. How has the overall quality of NIDRR’s peer review processes changed since January 1, 2005
(if you have served on different panels over time)?
Quality has decreased.
Quality is about the same.
Quality has increased.
Don’t know
Not applicable
6. Typically, the number of applications you received for each NIDRR review panel was:
More than you would like to review.
About the right number.
Fewer than you would like to review.
7. Typically, the amount of time you spent on each NIDRR review panel was:
More time than you would like to spend.
B-10
OCR for page 314
314 REVIEW OF DISABILITY AND REHABILITATION RESEARCH
About the right amount.
Less time than you would like to spend.
8. Have you participated in NIDRR peer reviews that were convened through:
Yes No
In-person meetings?
Teleconference?
9. In your opinion, how did the quality of a NIDRR peer review compare when conducted in
person versus teleconference?
Quality of peer reviews is better in in-person meetings.
Quality of peer reviews is better in teleconference meetings.
Quality of peer reviews via in-person meetings or teleconference is about the same.
Don’t know
Not applicable
10. Have you served as a peer reviewer of proposals for federal agencies other than NIDRR
since January 1, 2005?
Yes
No
11. How would you compare the following characteristics of NIDRR peer reviews with those of
other federal agencies? (Please check the boxes beneath the responses—select only one per row.)
NIDRR’s are About the NIDRR’s Don't Not
much weaker same are much Know Appli-
than other stronger cable
agencies than other
agencies
Expertise of the panel 1 2 3 4 5
members
Quality of the proposals 1 2 3 4 5
reviewed
Quality of the review 1 2 3 4 5
process
Transparency of the review 1 2 3 4 5
process
Fairness of the review 1 2 3 4 5
process
1 2 3 4 5
Reliability of the ratings
12. What three things would you suggest to enhance NIDRR’s peer review processes? (Click in
the upper left-hand corner to enter text.)
B-11
OCR for page 315
315
APPENDIX B
Thank you very much for taking the time to complete this questionnaire. We may be contacting
some respondents later to ask additional questions. If you would be willing to take part in brief
follow-up interviews, please check here.
Yes, it’s okay to contact me.
No, thank you. I’m finished!
Contact Information
(Click on the leftmost portion of the line to enter text.)
E-mail address:
Phone number:
B-12
OCR for page 319
319
APPENDIX B
such as output type (e.g., publications, tools, technology, information products); quality criteria assessed
(e.g., technical quality, knowledge advancement, potential impact); or program funding type (e.g., center
grant, field initiated grant, training grant, etc.). If your grant or your output represents one of a kind and
there is a risk of identifying you because of this, your data will be aggregated with another larger group
where identification will not be a risk.
Every effort will be made to protect the confidentiality of the information that you provide. The Study
Director will keep a list linking the grant and output research ID numbers with that output’s identifying
information (institution, grant title). This list, along with the data collected, will be stored securely at the
National Research Council, and will be accessible only by the Study personnel. If a telephone or
videoconference interview is convened to gather additional follow-up information, the transcription of
audio-recorded interviews will be combined in a dataset with the interviews of all of the other
respondents, then analyzed for common themes across the interviews. The audiotapes, transcriptions,
grantee questionnaires, committee ratings, and other raw data collected will be destroyed at the end of the
study when the report is released.
Compensation: There is no compensation for participating in the evaluation.
If you have any questions: The Co-Study Director of the evaluation and contact for questions is Jeanne
Rivard, Ph.D. If you have any questions about this consent form or the study, she can be contacted by
phone at: 202-334-2697, or by e-mail at: jrivard@nas.edu.
If you have any questions, comments, or concerns about taking part in this study, first talk to Dr. Rivard
above. If for any reason you do not want to do this, or you still have concerns after doing so, you may
contact the Institutional Review Board (IRB) of the National Academy of Sciences (NAS), which
reviewed and approved the study plans and this consent form. You can reach the chair of the IRB by
contacting Ronald D. Taylor, Human Protections Administrator, by telephone at 202-334-1659 or you
may write to him at the National Academy of Sciences; Room 1026; 500 Fifth Street, NW; Washington,
DC 20001.
Statement of Consent: I have read the above information, and have received answers to any questions I
asked. I consent to take parting in the study.
Your Signature_______________________________________ Date__-__-__
Your Name (printed) _____________________________________________
In addition to agreeing to participate, if there is a follow-up interview, I also consent to having it tape-
recorded.
Your Signature_______________________________________ Date__-__-__
B-16
OCR for page 320
320 REVIEW OF DISABILITY AND REHABILITATION RESEARCH
2C: GRANTEE QUESTIONNAIRE
Grant Award Number:
Grant Title:
Grantee:
Program Mechanism:
Grant End Date:
INSTRUCTIONS TO GRANTEES
This questionnaire has been designed to obtain information to assist the Committee in assessing
the quality of your grant’s outputs. NIDRR has provided to the Committee and the National
Research Council (NRC) staff copies of its Annual Performance Report (APR) database and
your last APR and your final APR. As you will see in certain places on the questionnaire we
have inserted information from your APR to facilitate your completion of the questionnaire (e.g.,
Table 1 lists your research and development projects, and Table 2 lists your outputs reported in
the APR). Where this information is in error, we would appreciate your pointing the errors out to
us and correcting it; and/or updating the information as needed.
The questionnaire is divided into the following three parts:
Part A. Nominating Outputs for Review. This section asks you to nominate, for the
Committee’s review, the “top 2” outputs for each of your projects that best reflect your grant's
achievements (Table 1 below). The Committee would prefer to review one publication and one
other type of output for each project. However if you only have publications, please nominate
these as your “top 2.”
To make this process easier, the NRC staff has populated a list of the outputs (Table 2 below)
that were reported for your grant in the APR. You could select the top 2 outputs from this table.
However you are not constrained to select from this list if there are other outputs that you think
better reflect your grant’s achievements.
For Committee review we are requesting materials and information regarding the actual outputs
selected as the top 2 for each project.
For publications, the material for review would be pdf copies of each article.
For the other outputs, materials for review would include:
o Electronic or hard copies of the measures, tools, intervention protocols, manuals; or
links to websites, pictures or other graphic representations of tools or devices that have
been produced.
o An abstract or summary of each output, which briefly describes:
o what the output is,
o its purpose,
B-17
OCR for page 321
321
APPENDIX B
o target audience,
o methods, and
o how the output fits into the overall goals and objectives of the project and grant
Part B. Additional Questions about Outputs. For each of the outputs you nominated for
review, the Committee has a series of questions related to their technical quality, how they may
have advanced knowledge, their potential impact, and their dissemination. We ask that you
complete the Part B section for each output. If the answers to certain questions would be the
same across different outputs, you can note this and cut and paste responses from earlier output
forms to other ones. Please make your responses brief, but as specific and quantitative as
possible.
Part C. Grant-level Questions. The questionnaire will also contain a few other items asking
about how you managed your grants to produce the highest quality outputs, how your grant’s
results may have generated new projects, and how key NIDRR processes influence results.
Your complete package of materials will contain:
Your signed informed consent form
Copies of your publications and other outputs (e.g., measures, tools, intervention
protocols, manuals, links to websites, pictures or other graphic representations of
devices that have been produced)
Your completed Grantee Questionnaire
Please send these materials by ___DATE___ to:
Matt McDonough
The National Academies
National Research Council
500 Fifth Street, NW
WS 1134
Washington, DC 20001
We are enclosing an addressed FedEx form that can be used when mailing your package of
materials to us. We estimated a shipment cost that would cover a weight up to 10 lbs. (e.g., for
large center grants or devices). If your package weighs more than this, FedEx will charge us the
correct amount.
If your package is light and you want to send it electronically, you could e-mail it to Matt at
mmcdonough@nas.edu. However, you would need to scan your signed consent form, and send
that in a pdf document.
B-18
OCR for page 322
322 REVIEW OF DISABILITY AND REHABILITATION RESEARCH
Part A. Nominating Outputs for Review
When referring to “outputs,” we are using the four NIDRR categories of outputs as defined in
NIDRR’s Annual Performance Report, which include: (a) Publications; (b) Tools, Measures, and
Intervention Protocols; (c) Technology Products and Devices; and (d) Informational Products.
Per the instructions for nominating outputs for review, please record your nominations for your
“top 2” outputs for each of your projects in Table 1 below. (Reminder “top 2” refers to those that
best reflect your grant's achievements). As you can see the NRC staff has already populated
Table 1 with the names of your research and development projects from data in the APR. Table
2, which follows, contains a list of outputs from which you can cut and paste into Table 1 below.
Please identify any errors in this information that we have provided from your APR and correct it
as needed.
Table 1. Projects and Nominated Outputs
Names of R&D Projects in Grant Names of Top 2 Outputs for Each Project
Outputs to be inserted by grantee from Table
#
2 below or add others as needed
Research Projects
R1 1.
2.
Development Projects
D1 Dissemination 1.
2.
INSERT GRANTEES' PROJECTS TABLE HERE
The table below lists all of the publications and other outputs that were listed in the APR data
provided by NIDRR. Please use this table below in selecting your top 2 outputs for each project.
(You can cut and paste from Table 2 into Table 1.) However you are not constrained to select
only from this list if there are other outputs that you think better reflect your grant’s
achievements.
Table 2. List of Outputs from APR
Type of Output Title of Output
publications (title)
tool (title)
B-19
OCR for page 323
323
APPENDIX B
Part B. Additional Questions About Outputs
Please use one copy of this form for each publication and each other output
for the “top 2” outputs that you selected for each project in Part A above, and
provide the following information. Please make your responses brief, but as
specific and quantitative as possible. If you consider the criterion not to be
applicable to your output, please explain. (Please note that an electronic copy
of the questionnaire was included in the email version of this package.)
Name of Output:
B1. Technical Quality of Output
In the space below, please describe examples of the technical quality of your output, such as:
The particular approach or methodology used in developing your output
Relevant peer recognition such as peer reviews or evaluations, peer endorsements,
invitations to present at professional forums or conferences, invitations to present
testimony, receipt of awards or honors, etc.
Receipt of a patent, FDA approval, or use of your output in standards development
Evidence of the usability and accessibility of the output
B2. Advancement of Knowledge
Please use the space below to describe how this output has advanced knowledge. To structure
your response, include points such as:
What the importance of your original question or issue was
How the output has advanced knowledge in arenas, such as:
o making discoveries
o providing new information
o establishing theories, measures, and methods
o closing gaps in the knowledge base
o developing new interventions, products, technology, and environmental
adaptations
B-20
OCR for page 324
324 REVIEW OF DISABILITY AND REHABILITATION RESEARCH
B3. Potential Impact
In the space below, please briefly describe evidence of your outputs’ potential (or actual) impact
on the following audiences, as relevant to your output:
Science (e.g., new areas of inquiry, methodology, etc.)
People with disabilities: health, quality of life, participation
Provider practice
Health and social systems
Social and health policy
Private sector/commercialization
Capacity building in the field of rehabilitation and disability research and development
(e.g., scientists, graduate students, etc.)
Other
Include information about how this potential impact was tested, and what the results were.
B4. Dissemination of Outputs
In the space below please provide evidence of your dissemination efforts for this output.
Describe this for publications if you have made any effort beyond those of the sponsor of the
publication (journal, book, proceedings, etc.). Please include important aspects of dissemination
such as:
Stage and scope (e.g., local, regional, national) of dissemination
Dissemination activities
Identification and tailoring of materials for reaching different audience/user types
Collaboration with audience/users in identifying content and medium needs/preferences
Delivery of information through multiple media types and sources for optimal reach and
accessibility
Evaluation of your dissemination efforts and impacts
B-21
OCR for page 325
325
APPENDIX B
Part C. Grant-level Questions
Please respond to these final questions for your overall grant, not by each
output specifically as in Section B.
C1. In the space below please describe what types of planning, project management, and
budgetary processes were used to promote high quality outputs. In your statement consider
the following types of questions:
Which processes were useful and how? How could they be improved?
Did you dedicate funds for quality assurance activities?
How did you track progress and spending against your original plans for the grant?
If grants or projects were jointly funded by NIDRR and other extramural or intramural
sources, how did you ensure that NIDRR resources were used exclusively for NIDRR-
funded activities?
C2. Have the results of the research and development outputs from this grant, or prior
NIDRR grants, been used to inform the development of new grant applications or other
kinds of projects?
No ____
Yes____
If yes, please use the space below to briefly describe what new grant applications, other
projects, funding opportunities, or collaborations have emerged.
C3. Please share any perspectives you may have about how NIDRR’s key processes (e.g.,
priority setting, peer review, and/or grants management) influence results, such as
successful grants and high quality outputs.
B-22
OCR for page 326
326 REVIEW OF DISABILITY AND REHABILITATION RESEARCH
2D: Committee Member Review Procedures for Summative Evaluation
2D: COMMITTEE MEMBER REVIEW PROCEDURES FOR SUMMATIVE EVALUATION
A. Review Subgroups: Each subgroup that will be reviewing outputs will be composed of five
Committee members. For each output one committee member will be assigned as the primary
reviewer; the remaining four committee members will be secondary reviewers.
B. Output Rating Procedures:
1. All reviewers will independently rate outputs using the following quality criteria (Dimensions
of these criteria are shown on the attached rating sheet.):
Technical quality of output
Advancement of knowledge or the field (research, practice, or policy)
Likely impact
Dissemination
The following scale will be used for rating the outputs:
Poor Good Excellent
Quality Quality Quality
1 2 3 4 5 6 7
2. The rating will be based on review of hard copy and electronic materials (i.e.,
articles/descriptive information about output and questionnaire responses) prior to the
subcommittee meeting.
3. The grantee’s final summary APR, and a list of all outputs reported over the course of the
project, is provided for contextual purposes. The APR also will be used to inform an overall,
qualitative grant-level assessment.
4. Multiple outputs of one grant will generally be rated independently of each other. However,
in some cases outputs may be rated as a pair with one score applied. This could occur when one
output is a derivative or different expression of another output, and when the PI responses to
criterion questions are basically the same. Examples of these include:
A manual describing a device (1) and a patent of the device (2)
A publication describing how a new technology for assessing a condition can be
applied in disability rehabilitation (1) and a description of the technology itself (2)
A software application (e.g., map reader for persons with visual impairments) (1)
and web-based method for individualizing the software for users (2)
5. The meeting will be structured as follows.
B-23
OCR for page 327
327
APPENDIX B
5. The meeting will be structured as follows:
The primary reviewer will open discussion of each output by presenting a brief
summary of the output and then his/her rationale for rating each relevant criterion
(up to four) plus the overall score.
Secondary reviewers will then present their ratings for each output and a brief
rationale.
Using the same criteria, the subgroup will then develop consensus group ratings
for each output. Discussion will be facilitated by the subgroup chair. If there is a
subgroup member with a significantly divergent view, his/her score and rationale
will be captured separately.
Staff will document discussion points that lead to the consensus group ratings and
will record the subgroup’s rationale for each criterion, the overall rating, and the
grant performance rating in a brief narrative.
At the end of the review of each output, the individual subgroup members’ rating
sheets will be gathered.
C. Grant Assessment
Once all outputs of an individual grant are reviewed, the subgroup will consider and rate the
grant’s overall performance. The outputs reviewed had been identified by the grant’s Principal
Investigator as the “top” two outputs per project, which best reflected the grant’s achievements.
Taking into consideration this designation, the consensus group ratings of the entire set of
outputs, and the grant’s overall purpose and objectives (using the grant’s APR), the subgroup
will assign a grant performance rating using the same 7-point scale. These grant-level ratings and
their rationale will also be documented by staff.
B-24
OCR for page 328
328 REVIEW OF DISABILITY AND REHABILITATION RESEARCH
2E: COMMITTEE MEMBER RATING SHEET OF THE QUALITY OF OUTPUTS
Committee Member:
Output ID:
Grantee ID:
Date of Review:
Output to Be Reviewed:
To be completed by NRC staff
Output Title: _______________________________________________________
Research Output: ______ Development Output: ________
Type and Subtype of Output (marked below):
Type of Type of Tool, Type of Technology Type of Informational
Publication Measure, or Product or Device Product
Intervention Protocol
1. abstract 1. checklist 1. industry 1. training
standards/guidelines manuals/curricula
2. book 2. survey or interview 2. software or netware 2. fact sheets
schedule
3. book chapter 3. diagnostic or 3. invention 3. newsletters
assessment instrument
4. journal article 4. outcome measure 4. patent, license, or 4. audiovisual materials
disclosures
5. proceedings 5. intervention protocol 5. working prototype 5. marketing tools
or program
6. technical 6. statistical technique 6. product evaluated or field 6. educational aids
tested
7. web journal 7. database 7. product transferred to 7. websites or other
industry for potential internet sites
commercialization
8. other 8. other 8. product in marketplace 8. other
B-25
OCR for page 329
329
APPENDIX B
Quality Criteria and Dimensions
For each criterion provide one rating using the scale below:
Poor Good Excellent
1 2 3 4 5 6 7
Criteria and Dimensions Score
Technical Quality of Output
Applying standards of science and technology
Appropriate methodology (quantitative or qualitative design and statistics)
Accessibility, usability, etc.
Score Rationale:
Advancement of Knowledge or the Field (research, practice, or policy as relevant)
Science: Establishment of methods, tools, theory
New information
Closing an identified gap
New technology
Innovative or novel
Score Rationale:
Likely or Demonstrated Impact On:
Science (impact factor, citations)
Consumers (people with disabilities: health, quality of life, participation)
Provider practice
Health and social system
Social and health policy
Private sector/commercialization
Other
Score Rationale:
Dissemination
Identification and tailoring of materials for reaching different audience/user types
Collaboration with audience/users in identifying content and medium needs/preferences
Delivery of information through multiple media types and sources for optimal reach and
accessibility
Evaluation of dissemination efforts and impacts
Commercialization/patenting of devices, if applicable
Score Rationale:
Overall Score
Score Rationale:
B-26
