Attachment C
This attachment comprises five parts: C-1, the grantee invitation letter; C-2, an informed consent form; C-3, the grantee questionnaire for the summative evaluation; C-4, the committee review procedures for the summative evaluation; and C-5, the rating sheet for committee members for the quality of outputs.
C-1 GRANTEE INVITATION LETTER
THE NATIONAL ACADEMIES
Advisers to the Nation on Science, Engineering, and Medicine
Division of Behavioral and Social Sciences and Education Committee on Human-Systems Integration |
500 Fifth Street, NW Washington, DC 20001 Phone: 202 334 2678 Fax: 202 334 2210 Email: cohsi@nas.edu www.nationalacademies.org |
October 6, 2010
Grantee Address
Dear Dr. _______:
Last week we sent you an email to inform you that you are being invited to participate in the External Evaluation of the National Institute on Disability and Rehabilitation Research (NIDRR) and Its Grantees that is being conducted by an expert committee of the National Research Council of the National Academies. This independent evaluation is being sponsored by NIDRR for the purpose of: (1) assessing NIDRR's priority setting and peer review processes and (2) reviewing the quality of grantee outputs for a sample of grants that represent the NIDRR portfolio.
Your NIDRR-funded grant (Grant #__________, Grant Title:____________________________) was selected to be reviewed as part of the evaluation of grantee outputs. However, your participation is completely voluntary. If you do agree to be part of the evaluation, your participation will involve the following activities.
- We will ask you to nominate two outputs that were produced under each project that was funded by the grant. These will be outputs that best reflect your grant’s achievements. We are using the four NIDRR categories of outputs as defined in NIDRR's Annual Performance Report (APR), which include (a) Publications; (b) Tools, Measures, and Intervention Protocols; (c) Technology Products and Devices; and (d) Informational Products. The committee will assess the quality of the outputs which you identify using criteria of technical quality, the extent to which they advance knowledge, their potential impact, and their dissemination.
- To conduct the review, we would like to examine the actual outputs and to review any documentation that you may have about the outputs.
- We will ask you to complete a questionnaire about each output that asks you to briefly summarize evidence of their technical quality, how they advance knowledge, their potential impact, and how they were disseminated.
- The questionnaire will also ask you to respond to a brief set of questions at the grant level about approaches you used in managing your grant, how the grant may have generated new research and projects, and about your perspectives of key NIDRR processes which may influence grant results.
- We may also ask you to participate in a follow-up telephone or videoconference interview. If so, we would like to audio-record the discussion between you and the committee.
The attachments that follow this letter include, first, an informed consent form that explains what we will do with the results of the evaluation and what steps we will take to protect the confidentiality of the evaluation results specific to your grant. The second attachment is the Grantee Questionnaire referred to above that provides instructions for:
- identifying outputs to be reviewed,
- completing the supplemental questions for each output, and
- sending us your signed consent forms, your outputs, and your completed questionnaire.
Please note that we are sending this package to you in electronic and hard copy form.
We hope that you will decide to participate. If you have any questions please don’t hesitate to contact one of us using the information below.
Sincerely,
Jeanne Rivard, Ph.D., Co-Study Director | Mary Ellen O’Connell, Co-Study Director |
The National Academies | The National Academies |
National Research Council | National Research Council |
500 Fifth Street, NW | 500 Fifth Street, NW |
Washington, DC 20001 | Washington, DC 20001 |
Phone: 202-334-2967 | Phone: 202-334-2607 |
Email: jrivard@nas.edu | Email: moconnell@nas.edu |
C-2 INFORMED CONSENT FORM
(For NIDRR Grantees)
What the study is about: An expert committee of the National Research Council of the National Academies, in Washington, DC is developing an evaluation framework that will be used to: 1) review NIDRR’s priority setting and peer review processes; and 2) review the quality of grantee “outputs” for a sample of grants that represent the NIDRR portfolio. [“Peer review” refers to a process in which experts review the merits of a grant application in considering whether it should be funded. “Outputs” are publications, measures, intervention protocols, devices, or information resources that are produced as part of a grant.] The committee will also assess the design and implementation of the evaluation process and make recommendations for additional evaluation cycles that may be performed subsequent to this effort.
What we will ask you to do: We would like to invite you to participate in the evaluation.
Your participation will involve:
- Having outputs produced under your grant peer reviewed through a quality assessment by an expert panel.
- To conduct the review, we would like to examine a copy of the actual outputs and to review any documentation that you may have about the outputs.
- We will ask you to complete a questionnaire about the outputs to assist in assessing their technical quality, the extent to which they advance knowledge, their potential impact, and if applicable, their dissemination.
- We may ask you to participate in a follow-up telephone or videoconference interview where we would audio-record the discussion between you and the committee.
- We will also ask you to respond to a brief set of questions at the grant level about your grant management, the generation of new research and projects, and about key NIDRR processes which may influence grant results.
Taking part is voluntary. Your participation is completely voluntary. You can chose not to answer some of the questions, and there will be no consequences.
Benefits and risks:
Benefits: By taking part in the evaluation you will provide information that may help NIDRR improve its research portfolio for the benefit of persons with disabilities.
Risks: Because NIDRR has funded some of your research and development activities, you may feel uncomfortable having your grant’s outputs formally rated in the study, or in sharing your opinions and perspectives on NIDRR’s key management processes. You might feel that this could be a risk to your future grant funding. We want you to know, however, that we will take every step necessary to protect your confidentiality and minimize this risk.
Your answers will be confidential. In the final evaluation report, we will briefly describe your grant and the outputs that were reviewed, by their titles and grantee institutions. Your name as the Principal Investigator, or other investigators on your team, will not be used. In most cases we expect that you have already placed information about the outputs in the public domain through publications, presentations at conferences, and through NIDRR’s National Rehabilitation Information Center (NARIC) website.
However, distinct from these descriptions will be your responses on the Grantee Questionnaire and the committee’s quality ratings of your outputs. For analysis and reporting, these narrative and quantitative
data will be de-identified and aggregated across all outputs and all grants. A research identification number will be used to track grants and their specific outputs. Outputs may be analyzed by categories, such as output type (e.g., publications, tools, technology, information products); quality criteria assessed (e.g., technical quality, knowledge advancement, potential impact); or program funding type (e.g., center grant, field initiated grant, training grant, etc.). If your grant or your output represents one of a kind and there is a risk of identifying you because of this, your data will be aggregated with another larger group where identification will not be a risk.
Every effort will be made to protect the confidentiality of the information that you provide. The Study Director will keep a list linking the grant and output research ID numbers with that output’s identifying information (institution, grant title). This list, along with the data collected, will be stored securely at the National Research Council, and will be accessible only by the Study personnel. If a telephone or videoconference interview is convened to gather additional follow-up information, the transcription of audio-recorded interviews will be combined in a dataset with the interviews of all of the other respondents, then analyzed for common themes across the interviews. The audiotapes, transcriptions, grantee questionnaires, committee ratings, and other raw data collected will be destroyed at the end of the study when the report is released.
Compensation: There is no compensation for participating in the evaluation.
If you have any questions: The Co-Study Director of the evaluation and contact for questions is Jeanne Rivard, Ph.D. If you have any questions about this consent form or the study, she can be contacted by phone at: 202-334-2697, or by email at: jrivard@nas.edu.
If you have any questions, comments, or concerns about taking part in this study, first talk to Dr. Rivard above. If for any reason you do not want to do this, or you still have concerns after doing so, you may contact the Institutional Review Board (IRB) of the National Academy of Sciences (NAS), which reviewed and approved the study plans and this consent form. You can reach the chair of the IRB by contacting Ronald D. Taylor, Human Protections Administrator, by telephone at 202-334-1659 or you may write to him at the National Academy of Sciences; Room 1026; 500 Fifth Street, NW; Washington, DC 20001.
Statement of Consent: I have read the above information, and have received answers to any questions I asked. I consent to take parting in the study.
Your Signature_______________________________________ Date__-__-__
Your Name (printed) _____________________________________________
In addition to agreeing to participate, if there is a follow-up interview, I also consent to having it tape-recorded.
Your Signature_______________________________________ Date__-__-__
C-3 SUMMATIVE EVALUATION: GRANTEE QUESTIONNAIRE
Grant Award Number:
Grant Title:
Grantee:
Program Mechanism:
Grant End Date:
INSTRUCTIONS TO GRANTEES
This questionnaire has been designed to obtain information to assist the Committee in assessing the quality of your grant’s outputs. NIDRR has provided to the Committee and the National Research Council (NRC) staff copies of its Annual Performance Report (APR) database and your last APR and your final APR. As you will see in certain places on the questionnaire we have inserted information from your APR to facilitate your completion of the questionnaire (e.g., Table 1 lists your research and development projects, and Table 2 lists your outputs reported in the APR). Where this information is in error, we would appreciate your pointing the errors out to us and correcting it; and/or updating the information as needed.
The questionnaire is divided into the following three parts:
Part A. Nominating Outputs for Review. This section asks you to nominate, for the Committee’s review, the “top 2” outputs for each of your projects that best reflect your grant's achievements (Table 1 below). The Committee would prefer to review one publication and one other type of output for each project. However if you only have publications, please nominate these as your “top 2”.
To make this process easier, the NRC staff has populated a list of the outputs (Table 2 below) that were reported for your grant in the APR. You could select the top 2 outputs from this table. However you are not constrained to select from this list if there are other outputs that you think better reflect your grant’s achievements.
For Committee review we are requesting materials and information regarding the actual outputs selected as the top 2 for each project.
- For publications, the material for review would be pdf copies of each article.
- For the other outputs, materials for review would include:
- Electronic or hard copies of the measures, tools, intervention protocols, manuals; or links to websites, pictures or other graphic representations of tools or devices that have been produced.
- An abstract or summary of each output, which briefly describes:
- o what the output is,
- o its purpose,
- o target audience,
- o methods, and
- o how the output fits into the overall goals and objectives of the project and grant
Part B. Additional Questions about Outputs. For each of the outputs you nominated for review, the Committee has a series of questions related to their technical quality, how they may have advanced knowledge, their potential impact, and their dissemination. We ask that you complete the Part B section for each output. If the answers to certain questions would be the same across different outputs, you can note this and cut and paste responses from earlier output forms to other ones. Please make your responses brief, but as specific and quantitative as possible.
Part C. Grant-level Questions. The questionnaire will also contain a few other items asking about how you managed your grants to produce the highest quality outputs, how your grant’s results may have generated new projects, and how key NIDRR processes influence results.
Your complete package of materials will contain:
- Your signed informed consent form
- Copies of your publications and other outputs (e.g., measures, tools, intervention protocols, manuals, links to websites, pictures or other graphic representations of devices that have been produced)
- Your completed Grantee Questionnaire
Please send these materials by ___DATE___ to:
Matt McDonough
The National Academies
National Research Council
500 Fifth Street, NW
WS 1134
Washington, DC 20001
We are enclosing an addressed Fedex form that can be used when mailing your package of materials to us. We estimated a shipment cost that would cover a weight up to 10 lbs. (e.g., for large center grants or devices). If your package weighs more than this, Fedex will charge us the correct amount.
If your package is light and you want to send it electronically, you could email it to Matt at mmcdonough@nas.edu. However, you would need to scan your signed consent form, and send that in a pdf document.
Part A. Nominating Outputs for Review
When referring to “outputs”, we are using the four NIDRR categories of outputs as defined in NIDRR's Annual Performance Report, which include: a) Publications; b) Tools, Measures, and Intervention Protocols; c) Technology Products and Devices; and d) Informational Products.
Per the instructions for nominating outputs for review, please record your nominations for your “top 2” outputs for each of your projects in Table 1 below. (Reminder “top 2” refers to those that best reflect your grant's achievements). As you can see the NRC staff has already populated Table 1 with the names of your research and development projects from data in the APR. Table 2, which follows, contains a list of outputs from which you can cut and paste into Table 1 below. Please identify any errors in this information that we have provided from your APR and correct it as needed.
Table 1. Projects and Nominated Outputs
# | Names of R & D Projects in Grant | Names of Top 2 Outputs for each Project Outputs to be inserted by grantee from Table 2 below or add others as needed |
Research Projects | ||
R1 | 1. | |
2. | ||
Development Projects | ||
D1 | Dissemination | 1. |
2. |
INSERT GRANTEES' PROJECTS TABLE HERE
The table below lists all of the publications and other outputs that were listed in the APR data provided by NIDRR. Please use this table below in selecting your top 2 outputs for each project. (You can cut and paste from Table 2 into Table 1.) However you are not constrained to select only from this list if there are other outputs that you think better reflect your grant’s achievements.
Table 2. List of Outputs from APR
Type of Output | Title of Output |
publications | (title) |
tool | (title) |
Part B. Additional Questions about Outputs
Please use one copy of this form for each publication and each other output for the “top 2” outputs that you selected for each project in Part A above, and provide the following information. Please make your responses brief, but as specific and quantitative as possible. If you consider the criterion not to be applicable to your output, please explain. (Please note that an electronic copy of the questionnaire was included in the email version of this package.)
Name of Output:____________________________________________
B1. Technical Quality of Output
In the space below, please describe examples of the technical quality of your output, such as:
- The particular approach or methodology used in developing your output
- Relevant peer recognition such as peer reviews or evaluations, peer endorsements, invitations to present at professional forums or conferences, invitations to present testimony, receipt of awards or honors, etc.
- Receipt of a patent, FDA approval, or use of your output in standards development
- Evidence of the usability and accessibility of the output
B2. Advancement of Knowledge
Please use the space below to describe how this output has advanced knowledge. To structure your response, include points such as:
- What the importance of your original question or issue was
- How the output has advanced knowledge in arenas, such as:
o making discoveries
o providing new information
o establishing theories, measures, and methods
o closing gaps in the knowledge base
o developing new interventions, products, technology, and environmental adaptations
B3. Potential Impact
In the space below, please briefly describe evidence of your outputs’ potential (or actual) impact on the following audiences, as relevant to your output:
- Science (e.g., new areas of inquiry, methodology, etc.)
- People with disabilities: health, quality of life, participation
- Provider practice
- Health and social systems
- Social and health policy
- Private sector/commercialization
- Capacity building in the field of rehabilitation and disability research and development (e.g., scientists, graduate students, etc.)
- Other
Include information about how this potential impact was tested, and what the results were.
B4. Dissemination of Outputs
In the space below please provide evidence of your dissemination efforts for this output. Describe this for publications if you have made any effort beyond those of the sponsor of the publication (journal, book, proceedings, etc.). Please include important aspects of dissemination such as:
- Stage and scope (e.g., local, regional, national) of dissemination
- Dissemination activities
- Identification and tailoring of materials for reaching different audience/user types
- Collaboration with audience/users in identifying content and medium needs/preferences
- Delivery of information through multiple media types and sources for optimal reach and accessibility
- Evaluation of your dissemination efforts and impacts
Part C. Grant-level Questions
Please respond to these final questions for your overall grant, not by each output specifically as in Section B.
C1. In the space below please describe what types of planning, project management, and budgetary processes were used to promote high quality outputs. In your statement consider the following types of questions:
- Which processes were useful and how? How could they be improved?
- Did you dedicate funds for quality assurance activities?
- How did you track progress and spending against your original plans for the grant?
- If grants or projects were jointly funded by NIDRR and other extramural or intramural sources, how did you ensure that NIDRR resources were used exclusively for NIDRR-funded activities?
C2. Have the results of the research and development outputs from this grant, or prior NIDRR grants, been used to inform the development of new grant applications or other kinds of projects?
No ____
Yes____
If yes, please use the space below to briefly describe what new grant applications, other projects, funding opportunities, or collaborations have emerged.
C3. Please share any perspectives you may have about how NIDRR’s key processes (e.g., priority setting, peer review, and/or grants management) influence results, such as successful grants and high quality outputs.
C-4 COMMITTEE MEMBER REVIEW PROCEDURES FOR SUMMATIVE EVALUATION
(Revised 11-08-10)
A. Review Subgroups: Each subgroup that will be reviewing outputs will be composed of five Committee members. For each output one committee member will be assigned as the primary reviewer; the remaining four committee members will be secondary reviewers.
B. Output Rating Procedures:
1. All reviewers will independently rate outputs using the following quality criteria (Dimensions of these criteria are shown on the attached rating sheet.):
- Technical quality of output
- Advancement of knowledge or the field (research, practice, or policy)
- Likely impact
- Dissemination
The following scale will be used for rating the outputs:
2. The rating will be based on review of hard copy and electronic materials (i.e., articles/descriptive information about output and questionnaire responses) prior to the subcommittee meeting.
3. The grantee’s final summary APR, and a list of all outputs reported over the course of the project, is provided for contextual purposes. The APR also will be used to inform an overall, qualitative grant-level assessment.
4. Multiple outputs of one grant will generally be rated independently of each other. However, in some cases outputs may be rated as a pair with one score applied. This could occur when one output is a derivative or different expression of another output, and when the PI responses to criterion questions are basically the same. Examples of these include:
- A manual describing a device (1) and a patent of the device (2)
- A publication describing how a new technology for assessing a condition can be applied in disability rehabilitation (1) and a description of the technology itself (2)
- A software application (e.g., map reader for persons with visual impairments) (1) and web-based method for individualizing the software for users (2)
5. The meeting will be structured as follows.
- The primary reviewer will open discussion of each output by presenting a brief summary of the output and then his/her rationale for rating each relevant criterion (up to four) plus the overall score,
- Secondary reviewers will then present their ratings for each output and a brief rationale.
- Using the same criteria, the subgroup will then develop consensus group ratings for each output. Discussion will be facilitated by the subgroup chair. If there is a subgroup member with a significantly divergent view, his/her score and rationale will be captured separately.
- Staff will document discussion points that lead to the consensus group ratings and will record the subgroup’s rationale for each criterion, the overall rating, and the grant performance rating in a brief narrative.
- At the end of the review of each output, the individual subgroup members’ rating sheets will be gathered.
C. Grant Assessment
Once all outputs of an individual grant are reviewed, the subgroup will consider and rate the grant’s overall performance. The outputs reviewed had been identified by the grant’s Principal Investigator as the “top” two outputs per project, which best reflected the grant’s achievements. Taking into consideration this designation, the consensus group ratings of the entire set of outputs, and the grant’s overall purpose and objectives (using the grant’s APR), the subgroup will assign a grant performance rating using the same 7-point scale. These grant-level ratings and their rationale will also be documented by staff.
C-5 COMMITTEE MEMBER RATING SHEET OF THE QUALITY OF OUTPUTS
Grantee ID:
Date of Review:
Output to be Reviewed:
To be completed by NRC staff Output Title: _______________________________________________________ Research Output: ______ Development Output: ________ Type and Subtype of Output (marked below): |
|||||||
Type of Publication | Type of Tool, Measure, or Intervention Protocol | Type of Technology Product or Device | Type of Informational Product | ||||
1. abstract | 1. checklist | 1. industry standards/guidelines | 1. training manuals/curricula | ||||
2. book | 2. survey or interview schedule | 2. software or netware | 2. fact sheets | ||||
3. book chapter | 3. diagnostic or assessment instrument | 3. invention | 3. newsletters | ||||
4. journal article | 4. outcome measure | 4. patent, license, or disclosures | 4. audiovisual materials | ||||
5. proceedings | 5. intervention protocol or program | 5. working prototype | 5. marketing tools | ||||
6. technical | 6. statistical technique | 6. product evaluated or field tested | 6. educational aids | ||||
7. web journal | 7. database | 7. product transferred to industry for potential commercialization | 7. websites or other internet sites | ||||
8. other | 8. other | 8. product in marketplace | 8. other |
Quality Criteria and Dimensions
For each criterion provide one rating using the scale below:
Criteria and Dimensions | Score |
Technical Quality of Output
Score Rationale: |
|
Advancement of Knowledge or the Field (research, practice, or policy as relevant)
Score Rationale: |
|
Likely or Demonstrated Impact On:
Score Rationale: |
|
Dissemination
Score Rationale: |
|
Overall Score Score Rationale: |