Appendix D
AAAS Science & Technology Policy Fellowships: Candidate Evaluation Summary, Guidelines, and Scoring Outline12

MISSION AND OBJECTIVES

The AAAS Science & Technology Policy Fellowships are provided as a professional development opportunity with the aim of fostering more policy-savvy scientists and engineers. Fellowships are awarded to highly qualified individuals interested in learning about the science-policy interface while applying their scientific and technical knowledge and analytical skills in the federal policy realm.

The overarching goals of the Fellowships are to increase the capacity of scientists to inform the discussions and decisions of individuals and institutions that influence or determine policies, and to provide scientific knowledge and analysis to support decision-makers confronting increasingly complex scientific and technological issues. The Fellowships aim to:

  • educate scientists and engineers on the intricacies of federal policymaking;

  • foster positive exchange between scientists and policymakers;

  • empower scientists and engineers to conduct policy-relevant research and other activities that address challenges facing society; and

  • increase the involvement and visibility of scientists and engineers in the public policy realm.

REVIEW AND SELECTION SUMMARY

The AAAS Science & Technology Policy Fellowships are extremely competitive, involving an intensive process to review applications and select fellowship finalists, which involves the following steps:

  1. AAAS Fellowships staff review all applications to ensure applicants meet the basic eligibility criteria and that all required materials are submitted.

  2. Each application is read and scored by three Readers (former Fellows), who rate it according to the guidelines provided in this document. Readers submit scores via the online review system. Raw and Z-scores are then calculated and utilized to designate the top-ranking candidates that will be forwarded to the Selection Committees.

  3. These candidates are then evaluated and rated by a minimum of three Selection Committee members, using the guidelines provided in this document. Scores are submitted via the

12

American Association for the Advancement of Science. 2008. AAAS Science & Technology Policy Fellowships: Candidate Evaluation Summary, Guidelines, and Scoring Outline. AAAS, Washington, DC.



The National Academies | 500 Fifth St. N.W. | Washington, D.C. 20001
Copyright © National Academy of Sciences. All rights reserved.
Terms of Use and Privacy Statement



Below are the first 10 and last 10 pages of uncorrected machine-read text (when available) of this chapter, followed by the top 30 algorithmically extracted key phrases from the chapter as a whole.
Intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text on the opening pages of each chapter. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.

Do not use for reproduction, copying, pasting, or reading; exclusively for search engines.

OCR for page 25
Appendix D AAAS Science & Technology Policy Fellowships: Candidate Evaluation Summary, Guidelines, and Scoring Outline12 MISSION AND OBJECTIVES The AAAS Science & Technology Policy Fellowships are provided as a professional development opportunity with the aim of fostering more policy-savvy scientists and engineers. Fellowships are awarded to highly qualified individuals interested in learning about the science- policy interface while applying their scientific and technical knowledge and analytical skills in the federal policy realm. The overarching goals of the Fellowships are to increase the capacity of scientists to inform the discussions and decisions of individuals and institutions that influence or determine policies, and to provide scientific knowledge and analysis to support decision-makers confronting increasingly complex scientific and technological issues. The Fellowships aim to: educate scientists and engineers on the intricacies of federal policymaking; foster positive exchange between scientists and policymakers; empower scientists and engineers to conduct policy-relevant research and other activities that address challenges facing society; and increase the involvement and visibility of scientists and engineers in the public policy realm. REVIEW AND SELECTION SUMMARY The AAAS Science & Technology Policy Fellowships are extremely competitive, involving an intensive process to review applications and select fellowship finalists, which involves the following steps: 1) AAAS Fellowships staff review all applications to ensure applicants meet the basic eligibility criteria and that all required materials are submitted. 2) Each application is read and scored by three Readers (former Fellows), who rate it according to the guidelines provided in this document. Readers submit scores via the online review system. Raw and Z-scores are then calculated and utilized to designate the top-ranking candidates that will be forwarded to the Selection Committees. 3) These candidates are then evaluated and rated by a minimum of three Selection Committee members, using the guidelines provided in this document. Scores are submitted via the 12 American Association for the Advancement of Science. 2008. AAAS Science & Technology Policy Fellowships: Candidate Evaluation Summary, Guidelines, and Scoring Outline. AAAS, Washington, DC. 25

OCR for page 25
online review system. Raw and Z-scores are then calculated and provided to the Selection Committee members, along with a summary of reviewers’ comments, before the first Selection Committee meeting. 4) Selection Committees convene for a breakfast or luncheon meeting to discuss the candidates and determine the semi-finalists to interview. The raw scores, Z-scores and comments are used to enrich the discussions. At this meeting the Selection Committee members also determine the briefing memo topic(s). 5) Semi-finalists submit a briefing memo either on a designated topic or on their choice of topic from among a range of issues (to be determined by the Selection Committees at their first meeting) that is read by all Selection Committee members before the interviews. 6) Semi-finalists are interviewed in person by Selection Committees in Washington, DC. At the end of each day of interviews, the semi-finalists are scored, ranked, discussed, and voted on to determine the fellowship finalists. Once candidates are designated as finalists, they are invited to participate in Placement Week in Washington, DC, in April (except Congressional Fellows, who select their assignments following the September orientation). Upon acceptance of a suitable placement offer, finalists are officially offered a AAAS Fellowship. ONLINE REVIEW PROCESS Selection Committee members review assigned applications and input comments and scores via the online review system. Each reviewer is sent an e-mail with a web link that connects directly to the applications assigned to him/her. Ratings of the candidates provide a means by which to compare applications for purposes of initial sorting and for the initial Selection Committee decisions regarding which candidates to interview. This framework is only for initial sorting. Final fellowship award decisions are based on the Selection Committee discussions and rankings following the individual interviews. The scores and comments are critical to the review process. In addition to the numerical indices that help to sort the applicant pool, the written comments from Selection Committee members are critical to gain a substantive sense of opinions and rationale for scores. Summaries of the scores and comments are distributed to the designated Selection Committee prior to its first meeting so that all members of that committee may see how others have reacted to candidates. Selection Committee members should use the comments section to provide information on the following: general assessment of the applicant; the reason for an exceptionally high or low score in any rating area; special considerations for further review—why the candidate is particularly meritorious or not; and input to consider if the applicant becomes an “on the fence” candidate. 26

OCR for page 25
Scores in the five rating areas (detailed sections below) will be totaled automatically by the online system. Reviewers may stop and save score sheets in progress and return to them later. Scores may be changed at any time prior to clicking the submit button. Before hitting the submit button, Selection Committee members should assess the overall scores to ensure that rankings across candidates are consistent and appropriate, using the guidelines outlined in the sections below. When the Selection Committee member is satisfied with all scores and comments, he/she must click the submit button. Once scores and comments are submitted they cannot be changed. The sections below outline AAAS Fellowship eligibility, review considerations, and the criteria and scoring mechanism to use when reviewing applicants. CANDIDATE ELIGIBILITY AAAS accepts candidates from a broad array of backgrounds and a diversity of geographic, disciplinary, gender, and ethnic perspectives. Fellows come from a range of sectors, including academia, industry, non-governmental organizations, and government (postdocs and contract only; permanent federal employees are not eligible). Fellows also represent a spectrum of career stages, from recent Ph.D. graduates to faculty on sabbatical to retired scientists and engineers. They have ranged in age from late twenties to early seventies. All applications forwarded to the Readers and Selection Committee members have been reviewed by AAAS Fellowships staff, and the applicants are deemed eligible for a fellowship based on meeting the following requirements: Doctoral-level scientist (Ph.D., MD, DVM, D.Sc. and other terminal degrees), in any physical, biological, medical/health, or social/behavioral science, any field of engineering, or any relevant interdisciplinary field. Or an individual with a master’s degree in engineering and at least three years of post-master’s degree professional experience. All degree requirements completed by 12/20. Not a permanent federal employee. Hold U.S. citizenship. REVIEW CONSIDERATIONS There are a number of issues to be addressed individually and collectively when reviewing fellowship applications. This section presents important considerations for Readers and Selection Committee members to keep in mind throughout the candidate assessment and selection process. Career Stage A range of career stages is represented in the pool of applicants, which can make comparisons among candidates challenging. Candidates at an earlier stage in their career are not expected to have accomplished as much as more senior individuals. Applicants earlier in their 27

OCR for page 25
career may have fewer publications, grants, collaborative activities, or a shorter history of leadership roles. This should be taken into consideration when assessing and scoring the candidates. Disciplinary Diversity Often it is difficult to make comparisons among candidates from diverse disciplinary backgrounds and professional fields. This may be reflected in the number and type of publications and professional activities (typically driven by the priorities of the institutional settings in which the candidates have worked). Individuals in some fields, such as engineering or medicine, may not publish as prolifically in peer-reviewed journals or be as involved in research activities. These contextual factors should be taken into account when assessing and scoring the candidates. Gender, Race, and Cultural Diversity In no sense are there quotas for gender, racial, or cultural diversity. The caliber of a candidate supersedes these considerations. However, a thoughtful review will include consideration of the candidate’s cultural background, his/her challenges for success, and how potential disadvantages might be reflected in his/her application in terms of barriers not faced by other candidates, or ways in which the candidate, via superior ability, may have overcome them. Institutional Affiliation AAAS welcomes applicants from academic institutions of all sizes, structures and foci (public, private, large, small, various disciplinary emphases), as well as from the nonprofit sector, from industry, and from government (if federal, only postdocs or contractors; permanent federal employees are not eligible). Candidates should be considered equal on the basis of their current institutional affiliation alone (we do not give greater weight to applicants from academia versus the other sectors, or to select institutions from within the sectors). Previous Applications There is no limit to the number of times an individual may apply for a AAAS Fellowship if he/she has not previously been selected for a fellowship, or has not previously accepted a fellowship (e.g., a candidate may have had to withdraw or decline due to health issues, an unexpected change in family circumstances, etc.). A candidate’s application history should not be a factor in assessing his/her current application. 28

OCR for page 25
Applications from Current/Former Fellows AAAS accepts applications from current or former Fellows who present a strong rationale for the value and benefit of a second fellowship opportunity, and who clearly articulate how they will apply the experience in the future. An individual may have no more than two AAAS Fellowships. Fellows may not apply for a fellowship in the same area in which they currently serve, or have previously served (AAAS staff have already screened for these factors). CRITERIA AND SCORING GUIDELINES Candidates are evaluated and scored based on the five categories outlined below. Scientific/Technical Background and Professional Accomplishment are weighted more heavily than the other four categories, as first and foremost the AAAS Science & Technology Policy Fellowships are professional development opportunities for highly-qualified scientists and engineers. However, that alone is not sufficient for success as a Fellow. We also seek individuals who have a combination of leadership attributes, initiative, analytical and problem-solving abilities, and communication skills, who demonstrate commitment to this professional development opportunity and the fellowship objectives, and whose experiences and interests fit with the focus of the specific fellowship area. The five criteria categories are presented with descriptors of what we seek in successful fellowship finalists, and the point ranges to use when evaluating applicants in each category. A perfect single score for a candidate would be 100 points. Scientific/Technical Background and Professional Accomplishment (1–40 points) Solid scientific/technical education and experience in area of expertise, appropriate to career stage Employment in relevant academic, applied scientific/technical, research, administration, outreach or policy positions appropriate to career stage and field Record of publications and/or presentations appropriate to career stage, field, and institutional setting Record of grants and/or participation in research projects or other scientific/technical initiatives appropriate to career stage, field, and institutional setting Leadership and Potential (1–15 points) Prior leadership roles relevant to career stage (graduate student governance or faculty committees; advisory or editorial committees; active in professional societies, non-profit, or community initiatives; other) Skill/potential to organize, build consensus, lead projects and people toward positive outcomes Confidence, maturity, and self-direction with the capacity, initiative and flexibility to work well independently as well as in groups Ability to identify personal strengths and areas for growth and development 29

OCR for page 25
Potential to take initiative to make the fellowship a rich and positive experience, to disseminate the skills learned through the fellowship, and take advantage of networks developed Analytical and Problem-Solving Abilities (1–15 points) Evidence of creative thinking and analytical skill Ability to translate and apply theoretical concepts into practice to solve problems Capacity to make connections between science/technology and broader economic, social, political issues Communication, Interpersonal, and Outreach Skills (1–15 points) Excellent communication skills: articulate, cohesive, concise, rational flow of information, and clear in both context and detail Ability to convey scientific knowledge in broader, non-scientific contexts Capacity to work effectively with diverse stakeholders outside scientific/engineering communities Commitment to AAAS Fellowship Mission and Opportunities (1–15 points) Strong interest in applying his/her knowledge toward the solution of problems in areas in which the fellowship would be served Clarity of objectives for applying to the fellowship, and how he/she imagines using the fellowship experience in the future Willingness and flexibility to tackle issues beyond area of expertise, openness and capacity to expand experience in the policy realm, and to interact with policymakers and regulators Realistic expectations, open-minded and adaptable to fellowship opportunities as well as working through challenges Demonstrates/communicates commitment to apply scientific/technical expertise to serve society When reviewing and scoring applications, consider all the materials, including the candidate statement, CV and educational record, summary of extracurricular activities, and the letters of recommendation. TOTAL SCORES AND COMPARISONS Once you complete all of your reviews and enter your scores and comments in the grids at the bottom of each application, you will see a summary of your ratings for the candidates. Your scores should be distributed among the applicants you consider the most qualified and the least qualified. The following guide is provided to reduce subjectivity and promote consistency in scoring. If an applicant’s score does not correspond with the descriptions below, please reassess your rating. 30

OCR for page 25
90–100: The applicant has an excellent essay, CV and qualifications, record of extracurricular activities, and recommendations. Overall, she/he excels in the outlined criteria, and therefore is highly qualified for a AAAS Fellowship. 79–89: The applicant has submitted a good essay, has a solid CV and qualifications and summary of extracurricular activities, and the references attest to the quality of the candidate. Overall, the candidate meets or exceeds the outlined criteria and is well-qualified for a AAAS Fellowship. 61–78: The applicant appears average for his/her career stage and does not stand out, therefore he/she is not well qualified for a AAAS Fellowship. 31–60: The applicant demonstrates merit only in some portions of his/her application, and the credentials are not clearly communicated through the essay, and/or the CV, record of extracurricular activities, recommendations. The applicant meets only some of the outlined criteria and therefore is not qualified. 0–30: The applicant does not meet the majority of criteria for a AAAS Fellowship and therefore is not qualified. SCORING SUMMARY The online review system tallies raw scores and calculates Z-scores for use in evaluating the applicants. No set of scores can fully describe the qualities sought, and therefore cannot serve as the ultimate arbiter of final decisions. That task will fall to the Selection Committee members following the completion of interviews. Raw Scores These scores are the total of the ratings each Selection Committee member provides on the candidates in each of the five criteria areas. 1. Scientific/Technical Background & Professional Accomplishment 40 points 2. Leadership & Potential 15 points 3. Analytical and Problem-Solving Abilities 15 points 4. Communication, Interpersonal & Outreach Skills 15 points 5. Commitment to AAAS Fellowship Mission & Opportunities 15 points TOTAL: 100 points Z-Scores These scores are determined by a statistical treatment to account for Selection Committee member variability—that is, the differences between the average of each Selection Committee member’s scores (i.e., is a particular reviewer an “easy” or “hard” grader) and the distribution of their scores for all the applicants they evaluated. 31

OCR for page 25
The Z-score expresses each individual score as a standardized function of the mean and distribution for all the applicants scored by a particular Selection Committee member. For example, a zero Z-score for a candidate would indicate that he or she is an average among those reviewed by a particular individual. A positive Z-score indicates an above average candidate, and a negative Z-score denotes a below average candidate. Thus, the Z-score allows better comparability among reviewers. It also reduces the possibility that a candidate receives an unusually low or high grand total raw score (the sum of the three assigned Selection Committee members’ scores) because they happened to be assigned to “hard” or “easy” reviewers. The Z- scores for each candidate are then averaged. While Z-scores highlight many of the most talented candidates, Selection Committee discussions shed new light on other candidates not as highly ranked in the initial reviews. 32