Below are the first 10 and last 10 pages of uncorrected machine-read text (when available) of this chapter, followed by the top 30 algorithmically extracted key phrases from the chapter as a whole.
Intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text on the opening pages of each chapter. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.
Do not use for reproduction, copying, pasting, or reading; exclusively for search engines.
OCR for page 23
Approaches for Evaluating the NRC Resident Research Associateship Program at NIST 2 Recruitment and Selection The NRC Research Associateship Program at NIST (hereafter “NIST/NRC RAP”) is thought to be a value to both the postdoctoral recipients and to NIST itself. This chapter is divided into three sections. First, the recruitment of RAs is put into context by examining trends in Ph.D. production and trends in postdoctoral appointments. Second, the application process is examined. Finally, characteristics of applicants and awardees are described. TRENDS IN DOCTORATES AND POSTDOCTORATES Trends in Doctorates As noted in the previous chapter, information on doctorates comes from the National Science Foundation’s Survey of Earned Doctorates (see Appendix B for a recent questionnaire). The number of doctorates granted in the United States has generally grown over the past 100 years, peaking in 1973 and 1998. Prior to 1953, more doctorates were produced at private institutions (NSF, 2006c). Examining trends from 1920-1999, for all doctorates—not just U.S. citizens: About two-thirds of doctorates were awarded in science and engineering (S&E). Between 1920 and 1974, between 87.2 and 93.6 percent of doctorates in S&E were awarded to men. This figure dropped steadily from 1975 to 1999. In the period 1995-1999, it stood at 66.8 percent.30 A growing percentage of S&E doctorates were awarded to foreign nationals: 38.6 percent by the 1990s.31 Underrepresented minorities receive few Ph.D.s relative to whites and Asians: 7.4 percent of S&E doctorates awarded in the second half of the 1990s went to underrepresented minorities.32 But the number and percentage of underrepresented minorities receiving S&E Ph.D.s has grown from the 1970s to the 1990s. The median age of doctorate recipients has been increasing over time; although the median age for recipients of Ph.D.s in S&E is much lower than the age for those receiving non-S&E Ph.D. degrees. A majority of doctoral recipients were married at the time of graduation, though the percentage of married graduates has been declining since the 1960s (NSF, 2006c). For the years 2000 to 2005, some trends have continued (see appendix C for underlying data). In 2005, S&E doctorates accounted for 64 percent of all doctorates awarded, which is similar to the ratio in the 1990s. However, more and more women are receiving doctorates in S&E. In 2005, about 38 percent of S&E Ph.D.s went to women. The number of U.S. citizens 30 These data are available at: http://www.nsf.gov/statistics/nsf06319/pdf/fig03-03.pdf. 31 These data are available at: http://www.nsf.gov/statistics/nsf06319/pdf/fig03-06.pdf. 32 Underrepresented minorities includes American Indian/Alaskan Native, Black, and Hispanic; and excludes Asian/Pacific Islander.
OCR for page 24
Approaches for Evaluating the NRC Resident Research Associateship Program at NIST receiving doctorates in science and engineering has declined somewhat from 2000 to 2005. Among doctorates where citizenship was known, in 2005, only about 56 percent of S&E Ph.D.s were awarded to U.S. citizens. Finally, the number of American Indian/Alaska Natives receiving S&E Ph.D.s has declined from 2000 to 2005; the number of Black/African Americans receiving S&E Ph.D.s has stagnated; and the number of Hispanics receiving S&E Ph.D.s has increased somewhat. As a result the same percent—about 10—of S&E Ph.D.s went to underrepresented minorities. Trends in Postdoctoral Appointments Postdoctoral appointments date back over 100 years; however the hiring of postdocs did not grow significantly until the second half of the twentieth century. An initial period of rapid growth occurred in the 1950s, stimulated by the Cold War demand for scientists and engineers. In the 1970s, and again during the recession of the 1990s, the number of postdoctoral positions increased due to a weaker economic market for Ph.D.s. (NAS/NAE/IOM, 2000; Davis, 2005). Postdoctoral appointments can provide benefits both to the recipients and the employers. For postdocs, the position is a way to obtain further training. Postdoctoral appointments in federal labs or industry can be an entrée into non-academic careers. Concerning the impact on the employer, one report notes that “As a whole, the postdoctoral population has become indispensable to the science and engineering enterprise, performing a substantial portion of the nation’s research in every setting. For example, a survey of research articles in two recent issues of Science found that 43 percent of the first authors were postdocs.33 In many labs, postdocs also educate, train, and supervise junior members, help write grant proposals and papers, and present the laboratory’s research results at professional society meetings” (NAS/NAE/IOM, 2000:10). However, it is important to note that there have been some complaints about the situation for postdocs. According to the NSF, in 2005 there were approximately 35,000 postdocs in academia, across all science and engineering fields broadly defined (NSF, 2007). However, there are differences by field. “In some fields, such as computer science and engineering, there is relatively little incentive to pursue a postdoc—or even a Ph.D.—because rewarding jobs are available at the bachelor’s and master’s levels. In other fields, such as biology and physics, a postdoc is virtually mandatory, especially for academic employment” (NAS/NAE/IOM, 2000:14). Table 2-1 gives a field breakdown for number of postdoctoral appointees, while Table 2-2 lists the percentage of doctoral recipients with definite plans to pursue postdoctoral study or research by field. 33 Vogel, G. Science, 1999, Vol. 285, p. 1531.
OCR for page 25
Approaches for Evaluating the NRC Resident Research Associateship Program at NIST TABLE 2-1 Science and Engineering Postdoctoral Appointees in Doctorate-Granting Institutions, by Field, 1998-2005 Field 1998 1999 2000 2001 2002 2003 2004 2005 Science and engineering 27,826 28,943 30,197 30,163 31,871 33,516 33,898 r 34,535 Science 24,973 25,747 26,884 26,997 28,303 29,696 29,935 r 30,374 Agricultural sciences 695 749 822 835 945 1,052 941 988 Biological sciences 15,755 16,091 16,729 17,022 17,640 18,605 18,675 r 18,995 Computer sciences 371 332 341 335 359 358 384 406 Earth, atmospheric, and ocean sciences 898 923 1,155 1,036 1,113 1,166 1,253 1,364 Mathematical sciences 279 351 385 353 391 447 466 496 Physical sciences 5,973 6,136 6,252 6,198 6,587 6,707 6,945 6,865 Engineering 2,853 3,196 3,313 3,166 3,568 3,820 3,963 4,161 Note: r = data significantly revised; replaces previously published data. Source: National Science Foundation/Division of Science Resources Statistics, Survey of Graduate Students and Postdoctorates in Science and Engineering in NSF (2007). Adapted from Table 49. TABLE 2-2 Percent of Doctoral Recipients with Definite Commitments Who Plan Postdoctoral Study or Research, by Broad Field of Study, 1982 and 2002 Field of Study 1982 2002 Biological sciences 72.1 74.4 Physics/astronomy 48.8 66.7 Chemistry 39.8 52.9 Earth, atmospheric, ocean sciences 25.9 51.6 Mathematics 15.8 42.5 Agricultural sciences 15.9 38.3 Engineering 11.4 24.8 Health sciences 15.4 21.1 Computer sciences 9.1 19.7 Source: NSF/NIH/USED/NEH/USDA/NASA, Survey of Earned Doctorates, in Hill et al., 2004: Figure 1. As there are differences by field, so too do postdocs vary by demographic characteristics. Based on the Sigma Xi survey of postdocs, the following conclusions are noted: The majority of postdocs in the life and health sciences, in the physical sciences, and in engineering are men. Men also comprise the majority of postdocs who are temporary-visa holders. About 75 percent of citizen and permanent resident postdocs identified themselves as white. The majority of postdocs responding to the survey held temporary visas. 40 percent were U.S. citizens and 6 percent were permanent residents. The majority of postdocs were between 30 and 35 years old; 69 percent were married or otherwise partnered; and about a third had children (Davis, 2005).34 34 It is difficult to know how generalizable the results of this survey are: the percentages are based on a 34 percent response rate and from postdocs at select institutions. Nonresponse bias may have affected the survey estimates.
OCR for page 26
Approaches for Evaluating the NRC Resident Research Associateship Program at NIST THE SELECTION PROCESS The process by which applicants apply and are selected to become NIST/NRC RAs can be summarized in a few basic steps: Potential applicants hear about the NIST/NRC RAPs Applicants apply to the Program Applications are reviewed by expert panels overseen by the NRC’s Fellowships Office. Each applicant receives a rating based on the average scores of three reviewers (possibly two more if the scores are disparate (differ by 1.5 points between highest and lowest score) The Fellowships Office forwards ranked (highest to lowest rated) applicants on to NIST Partly on the basis of those rankings, NIST offers selected applicants postdoctoral positions Most of those who receive offers accept and become research associates There are a number of ways to publicize postdoctoral positions. Both the National Academies and NIST have links to the program on their respective websites.35 Staff from the National Academies attend conferences where they make information on the program available. Advertisements are also placed in relevant publications, such as Physics Today, Science, and the Chronicle of Higher Education. Once prospective applicants hear about the program, the next step is to complete an application (see Appendix D). Noteworthy information collected by the application includes: Educational background Demographic data (e.g., citizenship, gender, date of birth, race/ethnicity, marital status) How the applicant heard about the program Previous research and publications A research proposal intended to be carried out during the postdoctoral tenure Applicants also have recommendations submitted on their behalf. In addition, the proposed research advisor at the laboratory/center also reviews the applicant’s proposed research project. Completed applications are collected by the National Academies. This is followed by a review process, as described on the National Academies’ Web site “Review Criteria” (see Box 2-1).36 35 The National Academies, “RAP Home,” available at: http://www7.nationalacademies.org/rap/ and NIST, “NIST Postdoctoral Research Associateships Program,” available at: http://www.nist.gov/oiaa/postdoc.htm. 36 The National Academies, “Review Criteria,” available at: http://www7.nationalacademies.org/rap/Review_Criteria.html.
OCR for page 27
Approaches for Evaluating the NRC Resident Research Associateship Program at NIST Box 2-1 Review Criteria Applications for awards from the NRC Research Associateship Programs are reviewed by panels of experts in 6 broad discipline areas: Chemistry; Earth and Atmospheric Sciences; Engineering, Applied Sciences and Mathematics; Life Sciences; Physics; and Space Sciences. Each application is read by a minimum of 3 panelists. Panelists assess the quality of an application, the likelihood for success and the contribution of the research to the mission of the sponsoring federal laboratory. Postdoctoral applicants are evaluated on the basis of demonstrated ability as a student and on their potential for making contributions as an independent scientist. Senior applicants, including applicants to Summer Faculty programs, are evaluated on the basis of proven ability and demonstrated research accomplishments. Evaluations are made without regard to age, sex, marital status, national origin, creed, racial group, or ethnic group. Each application is assigned a numerical score and the applicant’s final score is an average of all reviews. Scoring is on a 10-point scale and only applicants scoring 7.5 or above are considered for awards. Sponsoring laboratories offer awards to the highest scoring applicants first and continue to make awards until available slots are filled. In the review process each applicant is evaluated on four major elements with the approximate weighting as indicated: Scientific merit of the proposed research (40%) The research proposal is the most important element of the application and as such is weighted most heavily in the review. The proposal is evaluated for: importance of the proposed research area, clearly stated objectives, technical soundness of the work plan, innovative aspects of the proposal, feasibility of success, timeliness (can the proposal be completed in the allotted time), likelihood that the research will result in publication, and contribution of the research to the mission of the sponsoring laboratory. Reference reports or letters of recommendation (20%) Reference Reports or letters of reference contain opinions of persons who should have had a close professional relationship with an applicant; references provide reviewers with important information regarding the applicant’s scholarly abilities. Reference reports are given greater weight for Postdoctoral applicants, where a publication record may not be as extensive as that of a Senior applicant. Academic and research record (20%) Panelists review the appropriateness of the applicant’s training for the proposed research, previous research experience and record of publication. For Postdoctoral applicants only, a transcript of the academic record is required. Laboratory technical evaluation (20%) The Laboratory/Center Review form includes comments of the prospective Advisor and the disposition of the Laboratory/Center’s program committee or representative concerning the suitability of the applicant’s proposed research. This information aids reviewers in determining the value of the proposed research to the sponsoring agency.
OCR for page 28
Approaches for Evaluating the NRC Resident Research Associateship Program at NIST During the expert panel meetings, applicants are rated from 10 to 0, with 10 being the highest possible score. In practice, scores have ranged from 9.90 to 0 and applicants’ scores may differ by as little as 0.02 (e.g., 9.65 to 9.63). The review process is seen as something of a mystery to NIST staff. During the expert panel with former research associates, they commented that they did not understand how candidates were ranked. Former research associates had their own ideas about how candidates should be ranked. As a consequence of the personal connections between NIST employees and potential applicants, NIST employees formed opinions about who they thought top candidates were, and then registered their surprise when those people did not end up ranked at the top. In a related comment, former research associates commented that they would like to have a much better idea of how the research project that the applicant intends to work on (as described in the application) is graded. They felt that the current grading system is not transparent and is too general. NIST staff submit a recommendation on behalf of applicants and the staff do not seem sure what they ought to stress in their recommendation. Advisors and division chiefs focused on another issue in their expert panel: the review panels. They questioned whether the NRC review panels were organized well. They were concerned the panels were skewed to reviewers from academia. They did not know who serves on the panels. They would like to have more input into panel makeup, for example by suggesting names of potential reviewers (and then knowing if their suggestions were used). Another concern was whether rankings were normalized across panels. The applicants’ reviews, with scores, are sent to NIST, which selects the applicants to be offered postdoctoral positions. In practice, most of those offered, will accept a Research Associate. RECRUITMENT A concern for NIST and The National Academies is how well the program is reaching out to potential applicants. During the expert panels, current research associates noted that NIST was often their first choice. Reasons why included: family reasons, quality of advisors, ability to collaborate, and location. The research associates had applied to multiple positions, for example in academia or other government labs. Concerning how research associates heard about the position, the most frequent answer involved personal communication—either they met their future advisor at a conference, job fair, or when the advisor gave a lecture at their school; or a graduate advisor suggested they apply. (Personal communication was also repeatedly mentioned by the advisors and chiefs—that personal relationships had the best return on investment.) In several cases, their advisors were former research associates. Word of mouth was very important. Former research associates echoed these comments. Answers given by participants included: had a personal relationship—in one case the former research associate’s spouse worked at NIST; were recruited by advisors (at conferences or presentations made by NIST employees at universities); met researchers at NIST (this was suggested by former RA from local universities); and came across the Web site describing the RAPs. One RA had applied to a regular position at NIST and then saw the listing for the RAP. The former RAs noted that sometimes staff invite graduate students to give research talks at NIST as a way to bring potential applicants out to NIST. It did seem to the participants that the personal connections were much more effective. One former research associate noted that there is less outreach by NIST in the biological areas. This is important, as NIST may be moving in the future to more interdisciplinary research that
OCR for page 29
Approaches for Evaluating the NRC Resident Research Associateship Program at NIST has a biological component and there will be a need for more people with training in biological sciences. A physicist noted that at one point in the past, there was an advertisement in Physics Today, but he had not seen it recently. He felt that a better job of advertising could be done. Finally, former research associates noted that some had applied to multiple postdoctoral positions and had chosen NIST as the better alternative. Advisors and lab chiefs did note that in some areas they felt that the number of applications was low. Of particular concern were some areas within engineering and computer science. They noted that doctorates had many options in these areas and many forego postdocs, as well as the view that there are more foreign nationals and fewer U.S. citizens getting degrees in these areas. They did agree with the current and former research associates that personal communication seemed to be the best way to recruit. Participants in this panel did note that different labs differed in how they attempted to recruit applicants and to what degree they tried. Overall, participants reported that: The program is very prestigious (although some current RAs felt that the program was less so); The program is well-known (again, with some minority comment that it is not that well known); and The most common way people heard about the program was through personal communication. To examine these findings more broadly, the committee turned to data collected on the NRC’s Fellowships Office RAP application form. The application for postdoctoral programs includes a question on how the applicant heard about the position.37 Applicants were requested to select one of the following: colleague or fellow graduate student; Ph.D. thesis advisor or other professor; university placement office; former or current NRC Research Associate; research advisor or other scientific staff at the federal Laboratory; RAP’s staff member at professional scientific meeting; Advertisement in professional publication; or other. Data are available for 1989 to 2007. The dataset contained 24,849 applications, of which 2,743 were applications to the NIST/NRC RAP. The number of applicants is much less than this since applicants can apply for multiple positions in the same year or across years. (Because the NIST/NRC RAP was until recently reviewed once per year, there are only a handful of cases where an applicant applied more than once per year to this particular program.)38 In assessing how applicants heard about the program, we combined information from individuals who applied for multiple positions, if they selected different information sources, into a single record. This was done because in most cases, an applicant applying to multiple positions identified the same source in each case. Thus, the dataset was updated so that there was one record for each individual, regardless of how many applications they submitted. An exception occurred for applicants who applied to both the NIST/NRC RAP and any other RAP. Since a goal is to compare the sources identified by applicants to NIST and all other federal agency RAP postdoctoral positions, any individual who applied to both programs remained in the database twice. Two hundred and thirty-six individuals who applied to both NIST and other federal positions at the same time fit this 37 The question is: “To assist us in making information available to a greater number of potential applicants, it is important for us to learn how you initially heard about the National Academies RAPs.” 38 It is not clear if this is a data entry error.
OCR for page 30
Approaches for Evaluating the NRC Resident Research Associateship Program at NIST exception. After reducing duplicate entries, the database consisted of 12,737 records: 2,717 applicants to NIST and 10,020 applicants to other federal RAPs (with 236 applicants appearing on both lists). The NIST applicants identified a total of 2,890 sources and the non-NIST applicants identified a total of 10,497 sources. As Table 2-3 shows, the applicants to the NIST/NRC RAP were twice as likely as applicants to the other RAPs to hear about the position initially from their Ph.D. advisor or other professor and somewhat more likely to hear about the program from colleagues or fellow graduate students. Surprisingly, they were less likely to hear about the program from a research advisor or other scientific staff at the federal laboratory, compared with applicants to other federal RAPs. TABLE 2-3 How Applicants First Heard About the Research Associateship Program, 1989-2007 Source of Information NIST/NRC RAP (%) Other RAPs (%) All Programs (%) Colleague 23.3 20.9 21.4 Professor 33.7 16.6 20.3 Placement office 1.0 1.7 1.5 NRC associate 8.2 9.3 9.1 Laboratory staff 18.0 26.7 24.8 Journal 3.0 8.7 7.4 NRC staff 0.2 0.3 0.3 Professional meeting 4.8 5.3 5.2 Other 7.8 10.6 10.0 Source: National Academies, DataRAP Database, tabulations by staff. Regarding advertisements in professional publications, applicants to NIST/NRC RAP identified about 30 publications, while applicants to non-NIST RAPs identified approximately 190 publications. (However, since the other RAPs include a broader set of fields, this larger number of publications should be expected and the smaller number of publications for the NIST/NRC RAP is not an indication, by itself, of less effort to reach potential applicants via publications. Top publications identified by applicants to NIST/NRC RAP: Physics Today, Fellowships Office mailing, Chemical & Engineering News, Mechanical Engineering, Spectrum of the IEEE, and Science. Examining trends in the sources cited over time, as is done in Table 2-4, shows that applicants for NIST/NRC RAs did not usually first find out about the program via an advertisement in a publication.
OCR for page 31
Approaches for Evaluating the NRC Resident Research Associateship Program at NIST TABLE 2-4 How Applicants to the NIST/NRC Research Associateship Program First Heard About the Program, 1989-2007 Year Colleague (%) Professor (%) Placement Office (%) NRC Associate (%) Lab Staff (%) Journal (%) NRC Staff (%) Professional Meeting (%) Other (%) N 1989 100.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1 1990 10.3 38.3 0.9 5.6 27.1 6.5 0.0 1.9 9.3 107 1991 33.3 63.0 3.7 13.9 27.8 12.0 0.0 9.3 18.5 108 1992 22.5 34.7 2.3 10.8 20.3 5.0 0.0 3.2 1.4 222 1993 22.6 35.5 0.9 9.0 15.0 6.8 0.9 5.1 4.3 234 1994 15.6 18.9 0.9 5.3 5.0 2.9 0.2 2.6 2.0 456 1995 34.3 37.7 1.0 0.5 17.4 1.9 0.0 4.8 2.4 207 1996 31.1 28.8 0.8 0.8 24.2 2.3 0.0 6.1 6.1 132 1997 15.6 14.7 0.6 3.2 6.5 0.9 0.3 2.4 6.8 339 1998 38.2 27.3 0.0 4.5 14.5 0.0 0.9 5.5 9.1 110 1999 23.3 22.5 0.8 14.2 23.3 0.8 0.0 2.5 12.5 120 2000 6.1 15.7 0.9 6.1 12.2 0.9 0.4 1.7 5.7 230 2001 21.7 34.8 0.0 13.0 19.6 0.0 0.0 4.3 6.5 46 2002 18.1 23.3 0.0 14.4 23.3 1.9 0.0 8.4 10.7 215 2003 11.5 24.5 0.0 4.2 10.3 0.8 0.0 6.1 6.9 261 2004 20.3 38.0 0.6 7.6 17.7 1.3 0.0 1.3 13.3 158 2005 23.1 34.0 1.3 12.2 16.7 1.3 0.0 2.6 9.0 156 2006 10.2 23.6 0.6 3.8 12.7 0.6 0.0 3.2 2.9 314 2007 22.4 40.2 0.0 6.5 15.9 0.9 0.0 4.7 9.3 107 Source: National Academies, DataRAP Database, tabulations by staff. Journals seemed to be more important as a source in the first half of the 1990s than in the current decade. Another finding of note is that placement offices at universities tend to be an infrequently cited source of information. Although the committee did not have prior expectations, it was still a bit surprising that the proportion of applicants hearing about the program from former or current RAs was not that great. We next examined some characteristics of applicants to the NIST/NRC RAP to see if different types of applicants differed on how they first heard about the program. Comparing men and women, we found no significant differences, except for presentations at professional meetings, where women were twice as likely as men to first hear about the program.
OCR for page 32
Approaches for Evaluating the NRC Resident Research Associateship Program at NIST TABLE 2-5 How Applicants to the NIST/NRC Research Associateship Program First Heard About the Program, by Gender, 1989-2007 Source of Information Women (%) Men (%) Colleague 22.2 23.6 Professor 32.1 34.1 Placement office 0.6 1.1 NRC associate 8.1 8.2 Laboratory staff 17.4 18.2 Journal 2.4 3.1 NRC staff 0.2 0.2 Professional meeting 8.3 4.1 Other 8.7 7.6 N 505 2385 Source: National Academies, DataRAP Database, tabulations by staff. Turning to race/ethnicity, we compared how whites and all other ethnic/racial groups first heard about the program. Similar to women, the results were fairly consistent across these two groups and minorities were more likely than whites to hear about the program via a presentation at a professional meeting—but not significantly so. TABLE 2-6 How Applicants to the NIST/NRC Research Associateship Program First Heard about the Program, by Race/Ethnicity, 1989-2007 Source of Information White (%) All Other (%) Colleague 23.2 20.7 Professor 34.6 33.0 Placement office 0.9 1.1 NRC associate 8.0 8.6 Laboratory staff 18.0 18.7 Journal 2.5 2.9 NRC staff 0.1 0.6 Professional meeting 4.9 6.6 Other 7.9 7.8 N 2298 348 Source: National Academies, DataRAP Database, tabulations by staff. As a next step, future analysis could examine the relationship between different sources of information among applicants and outcomes of applications. For example, Table 2-7 examines this association in general for applicants to NIST/NRC RAPs and other RAPs.
OCR for page 33
Approaches for Evaluating the NRC Resident Research Associateship Program at NIST TABLE 2-7 Percent of Awardees Among Applicants by Source of Information About the Program, 1965-2007 Awardees among Applicants (%) Source of Information NIST/NRC RAP Other RAP Colleague 28.2 36.1 Professor 26.1 40.3 Placement office 15.6 30.5 NRC associate 31.4 43.2 Laboratory staff 28.3 47.3 Journal 17.2 26.4 NRC staff 100.0 42.3 Professional meeting 28.4 37.0 Other 29.2 28.8 Source: National Academies, DataRAP Database, tabulations by staff. As Table 2-7 illustrates, successful applicants to the NIST/NRC RAP were more likely to hear about the Program from NRC staff or an NRC associate; although in these cases, few applicants had heard about the Program from these sources. There seems to be much more variability in how successful applicants hear about the NIST/NRC RAP, as compared with successful applicants to the other RAPs. Table 2-8 focuses on just those applicants who received RAs. TABLE 2-8 Percent of Awardees by Source of Information About the Program, 1965-2007 Awardees (%) Source of Information NIST/NRC RAP Other RAP Colleague 23.9 19.4 Professor 32.2 17.0 Placement office 0.6 1.3 NRC associate 9.3 10.4 Laboratory staff 18.3 32.7 Journal 1.9 5.9 NRC staff 0.2 0.3 Professional meeting 5.1 5.0 Other 8.4 7.9 N 825 4022 Source: National Academies, DataRAP Database, tabulations by staff. Table 2-8 shows that about one-third of awardees to the NIST/NRC RAP first heard about the program via a professor; whereas about one-third of awardees to the other RAPs first heard about the program from lab staff. Taken together, Tables 2-6 to 2-8 suggest that personnel communication was the most important mechanism for transmitting information about the pograms to prospective applicants. This sort of analysis could be extended by focusing on subsets of applicants, that is, by gender, race/ethnicity, or discipline, to see how well outreach succeeds.
OCR for page 50
Approaches for Evaluating the NRC Resident Research Associateship Program at NIST Pennsylvania State University Park 40 2.9 Johns Hopkins University 37 2.7 Princeton University 34 2.4 University of Maryland College Park 33 2.4 Carnegie Mellon University 30 2.2 University of Florida 28 2.0 Cornell University 27 1.9 University of Virginia 25 1.8 University of Minnesota-Twin Cities 24 1.7 University of Massachusetts-Amherst 24 1.7 University of Texas-Austin 23 1.7 Total 718 51.7 Mathematics/CS Number of Applications from Institution % University of Wisconsin-Madison 15 5.8 University of Maryland College Park 13 5.0 Northwestern University 11 4.3 Cornell University 11 4.3 Johns Hopkins University 9 3.5 Purdue University 8 3.1 University of California-Berkeley 7 2.7 New York University 7 2.7 University of California-Santa Barbara 6 2.3 State University of New York-Stony Brook 6 2.3 Brown University/RI 6 2.3 Massachusetts Institute of Technology 5 1.9 University of Michigan 5 1.9 University of Colorado 5 1.9 Ohio State University 5 1.9 University of Southern California 5 1.9 Total 124 48.1 Physical sciences Number of Applications from Institution % University of California-Berkeley 172 3.9 Cornell University 155 3.5 University of Colorado 155 3.5 University of Maryland College Park 150 3.4 Harvard University 129 2.9 Massachusetts Institute of Technology 122 2.8 University of Illinois-Urbana-Champaign 109 2.5 University of Wisconsin-Madison 106 2.4 University of Chicago 95 2.2 Stanford University 88 2.0 Pennsylvania State University Park 84 1.9 University of Michigan 81 1.8 University of Texas-Austin 81 1.8 Yale University 71 1.6 University of Virginia 69 1.6 University of California-Santa Barbara 66 1.5
OCR for page 51
Approaches for Evaluating the NRC Resident Research Associateship Program at NIST Iowa State University 66 1.5 State University of New York-Stony Brook 56 1.3 California Institute of Technology 55 1.2 University of Florida 55 1.2 Total 1965 44.5 Note: In 2007, not all application cycles have been completed and recorded in the database. Source: National Academies, DataRAP Database, tabulations by staff. TABLE 2-16 Most Common Doctoral-Granting Institutions of NIST/NRC Research Associates, by Major Field, 1965-2007 Bio/Biomed/Health No. % Johns Hopkins University 3 10.0 Georgetown University 2 7.0 Johns Hopkins University-Medical Insts. 2 7.0 State University of New York-Stony Brook 2 7.0 Total 9 31.0 Engineering No. % Massachusetts Institute of Technology 21 6.0 University of Michigan 21 6.0 Princeton University 16 4.6 Northwestern University 14 4.0 Pennsylvania State University Park 12 3.4 University of Illinois-Urbana-Champaign 12 3.4 Carnegie Mellon University 11 3.2 Stanford University 11 3.2 University of Massachusetts-Amherst 11 3.2 University of California-Berkeley 10 2.9 University of Colorado 8 2.3 University of Minnesota-Twin Cities 8 2.3 Virginia Polytech Institute and State U 8 2.3 University of Florida 7 2.0 University of Texas-Austin 7 2.0 University of Washington 7 2.0 Iowa State University 7 2.0 Lehigh University 7 2.0 University of Maryland College Park 7 2.0 Johns Hopkins University 6 1.7 Total 211 60.5 Mathematics/CS No. % University of Maryland College Park 4 6.0 University of Wisconsin-Madison 4 6.0 Northwestern University 3 5.0 Pennsylvania State University Park 3 5.0 Cornell University 3 5.0 University of California-Santa Barbara 3 5.0 University of Colorado 2 3.0
OCR for page 52
Approaches for Evaluating the NRC Resident Research Associateship Program at NIST University of Washington 2 3.0 Johns Hopkins University 2 3.0 Ohio State University 2 3.0 Syracuse University 2 3.0 Total 30 47.0 Physical sciences No. % University of Colorado 47 5.0 Harvard University 45 4.8 University of Maryland College Park 41 4.4 University of California-Berkeley 40 4.3 Cornell University 35 3.7 University of Illinois-Urbana-Champaign 34 3.6 University of Wisconsin-Madison 31 3.3 Massachusetts Institute of Technology 29 3.1 Stanford University 28 3.0 Yale University 21 2.2 University of Texas-Austin 21 2.2 California Institute of Technology 17 1.8 University of Chicago 17 1.8 University of California-Santa Barbara 16 1.7 University of Virginia 16 1.7 University of Michigan 16 1.7 Iowa State University 16 1.7 Pennsylvania State University Park 15 1.6 Indiana University-Bloomington 13 1.4 Total 498 53.0 Note: In 2007, not all application cycles have been completed and recorded in the database. Source: National Academies, DataRAP Database, tabulations by staff. Age As noted in the start of the chapter, the average age of postdocs has been creeping upwards, in part due to longer time to complete doctorates. Figure 2-12 compares the average age of applicants to and awardees of NIST/NRC RAP and the other RAPs.
OCR for page 53
Approaches for Evaluating the NRC Resident Research Associateship Program at NIST FIGURE 2-12 Average age of applicants and awardees, by Research Associateship Program, 1965-2007. Note: In 2007, not all application cycles have been completed and recorded in the database. Source: National Academies, DataRAP Database, tabulations by staff. As the figure shows, awardees tend to be younger than applicants and applicants to the NIST/NRC RAP and awardees of NIST/NRC Research Associates are younger on average than those who apply for and are awarded other RAPs. It is likely that part of the explanation for this is the group of postdocs coming out of the biological sciences, who are largely absent from the NIST cohort. A second explanation may involve the role of foreign students: NIST RAs are U.S. citizens, who also tend to get to postdoctoral status quicker than international students. Marital Status Marital status can be an important demographic characteristic for postdoctoral programs. When many doctorates pursue postdoctoral appointments, they are also at an age when many are married and thinking about starting families. Many scientists are married to other scientists. Knowing this demographic can be helpful in dealing with related issues of: dual-career couples; salary, benefits and cost of living; child care and parental leave. The application form includes a question on marital status. Two categories are available: married and single; although many applicants leave this answer blank. (Additionally, 4 applicants chose “F”—possibly a data coding entry with gender.) Twenty-seven applicants to NIST left this question blank and 368 applicants to non-NIST left it blank. For awards, 2 awardees chose “F,” 4 awardees at NIST left it blank, and 121 awardees at other RAPs did not answer the question. The percentages of applicants and awardees that were married or single among those who noted marital status, are examined in Figures 2-13 and 2-14.
OCR for page 54
Approaches for Evaluating the NRC Resident Research Associateship Program at NIST FIGURE 2-13 Percent of applicants who are married, by Research Associateship Program, 1965-2007. Note: In 2007, not all application cycles have been completed and recorded in the database. Source: National Academies, DataRAP Database, tabulations by staff.
OCR for page 55
Approaches for Evaluating the NRC Resident Research Associateship Program at NIST FIGURE 2-14 Percent of awardees who are married, by Research Associateship Program, 1965-2007. Note: In 2007, not all application cycles have been completed and recorded in the database. Source: National Academies, DataRAP Database, tabulations by staff. Applicants and RAs in other RAPs tend to be older and more likely to be married than NIST/NRC RAP applicants and awardees. (See Appendix Table B-10 for the underlying data.) As Figure 2-13 shows, other RAP applicants are more likely to be married. Figure 2-14 shows that other RAP awardees are also more likely to be married. These figures raise a question of whether the NIST/NRC RAP is less attractive to married scientists and engineers or whether some other characteristic of applicants to the NIST/NRC RAP explains the trend that NIST/NRC Research Associates are more likely to be single. Laboratories Applicants to the NIST/NRC RAP select a lab on their application form. Over the years NIST has reorganized, which means that some older organizational names are no longer valid, while some recent laboratories may not yet have any applicants. Since 1965, applicants have applied to 18 different parts of NIST. We used the current organizational chart to map older institutional names onto current names (see Appendix E). This was problematic in a number of ways. First, 268 applicants simply put “National Institute of Standards and Technology.” Second, several older divisions—e.g., National Engineering Laboratory and the National Measurement Laboratory—map onto multiple contemporary divisions. We combined these situations into a new category: “Multiple,” but it can also be thought of as an unknown category. Finally, in spite of the efforts to map the laboratory names, viewing the data over time shows that this is not fully successful. Applicants to Technology Services covered the years 1965 to 1978,
OCR for page 56
Approaches for Evaluating the NRC Resident Research Associateship Program at NIST but then stopped, although the name still exists, perhaps what the applicants were researching fit better elsewhere. Other labs apparently had no applicants until 1992, again which might reflect an organizational change. Thus, analysis over time, and analysis by race/ethnicity—for which data do not exist prior to 1980—are unwarranted. With available data, two tables can be presented, which focus on applications and acceptances by lab in total, and by gender. TABLE 2-17 Applications and Awards for the NIST/NRC Research Associateship Program, by Laboratory, 1965-2007 Laboratory Applications % Acceptances % Accept. As % of Apps. Building and Fire Research Laboratory 103 1.7 31 2.3 30.1 Chemical Science and Technology Laboratory 755 12.5 173 12.8 22.9 Electronics and Electrical Engineering Laboratory 355 5.9 101 7.5 28.5 Information Technology Laboratory 250 4.1 43 3.2 17.2 Manufacturing Engineering Laboratory 92 1.5 34 2.5 37.0 Materials Science and Engineering Laboratory 1448 23.9 304 22.5 21.0 Physics Laboratory 674 11.1 203 15.0 30.1 Multiple 2370 39.2 465 34.3 19.6 Total 6047 100.0 1354 100.0 22.4 Note: In 2007, not all application cycles have been completed and recorded in the database. Source: National Academies, DataRAP Database, tabulations by staff. The principal finding here is that applications are not evenly distributed among labs. Some labs attract more applicants than others. Likewise, some labs see more research associateships awarded than others. A second finding is that the acceptance rate varies by more than a factor of two—a large range. It may be instructive to determine why this might be the case. Possible answers could focus on the field, other opportunities for recent doctorates in those fields, and outreach by the different labs. Another possible explanation is an intentional “share the wealth” effort. Table 2-18 continues this examination for gender. Half of the applications from women were to the Chemical Science and Technology Laboratory and the Materials Science and Engineering Laboratory. These are also the two places where a greater proportion of women receive awards. Other labs receive very few applications from women—the Building and Fire Research Laboratory, for example. A second finding is that, in general, the percentages for female applications are similar to the percentage of female awardees. For example, 7 percent of applications to the Electronics and Electrical Engineering Laboratory came from women, while 9 percent of awardees to this lab were women. Additionally, 14 percent of women who applied for the NIST/NRC RAP applied to this lab and 18 percent of women who received awards were in this lab. Again, it may be instructive to seek out explanations for differences across the labs in terms of the gender (or for that matter, the race/ethnicity) of postdocs. Possible explanations might focus on the role of lab staff in recruiting women candidates; how female-friendly the lab is perceived to be; or the underlying number of female doctorates in fields appropriate to the research of each lab.
OCR for page 57
Approaches for Evaluating the NRC Resident Research Associateship Program at NIST TABLE 2-18 Applications and Awards for the NIST/NRC Research Associateship Program, by Laboratory and Gender, 1965-2007 Laboratory Female Applications (N) Female Applications (%) Female of Total Applications (%) Female Awards (N) Female Awards (%) Female of Total Awards (%) Building and fire research laboratory 5 0.7 4.9 1 0.5 3.2 Chemical science and technology laboratory 168 23.6 22.3 54 26.7 31.2 Electronics and electrical engineering laboratory 51 7.2 14.4 18 8.9 17.8 Information technology laboratory 28 3.9 11.2 3 1.5 7.0 Manufacturing engineering laboratory 10 1.4 10.9 1 0.5 2.9 Materials science and engineering laboratory 186 26.1 12.8 56 27.7 18.4 Multiple 176 24.7 7.4 31 15.3 6.7 Physics laboratory 89 12.5 13.2 38 18.8 18.7 Total 713 100.0 11.8 202 100.0 14.9 Note: In 2007, not all application cycles have been completed and recorded in the database. Source: National Academies, DataRAP Database, tabulations by staff.
OCR for page 58
Approaches for Evaluating the NRC Resident Research Associateship Program at NIST Declined Offers One concern heard at focus groups was that NIST was not quick enough at offering applicants awards. Individuals were accepting other positions instead of coming to NIST. This section looks at offers that were accepted and those that were declined. Table 2-19 shows the number and percentage of offers declined among those who accepted or declined. On average, 7 people per year decline to accept an award from NIST, compared with an average of 68 people per year for other RAPs. TABLE 2-19 Number of Individuals Offered a Research Associateship Who Decline, by Research Associateship Program, 1965-2007 NIST/NRC RAP Other RAPs Year Number % Number % 1965 0 0.0 4 3.4 1966 1 2.7 10 6.9 1967 0 0.0 8 5.2 1968 9 37.5 36 16.7 1969 8 33.3 75 35.4 1970 14 48.3 76 30.6 1971 7 30.4 89 29.2 1972 19 50.0 107 29.7 1973 9 31.0 103 33.8 1974 13 43.3 101 33.0 1975 6 26.1 88 27.8 1976 6 20.7 92 32.4 1977 12 35.3 68 32.7 1978 16 41.0 103 36.9 1979 20 45.5 85 32.1 1980 11 30.6 114 36.9 1981 23 51.1 87 28.2 1982 3 15.8 70 26.4 1983 11 31.4 79 28.7 1984 14 35.0 104 32.1 1985 14 33.3 138 36.6 1986 N/A N/A 71 20.0 1987 8 26.7 65 20.0 1988 14 38.9 71 18.6 1989 5 20.0 54 14.6 1990 5 16.7 68 19.2 1991 2 7.1 66 17.2 1992 8 20.5 64 18.0 1993 5 12.8 82 18.8 1994 11 21.6 66 15.2 1995 2 5.0 66 19.4 1996 1 1.7 55 15.8 1997 6 13.0 53 15.8 1998 1 1.7 40 15.4 1999 2 4.9 31 13.0
OCR for page 59
Approaches for Evaluating the NRC Resident Research Associateship Program at NIST 2000 6 9.5 46 17.2 2001 0 0.0 41 12.3 2002 1 1.9 59 12.9 2003 0 0.0 46 13.0 2004 6 11.1 44 16.9 2005 15 20.3 33 16.3 2006 9 13.2 30 17.0 2007 1 2.1 11 18.0 Total 324 19.0 2799 22.2 Note: No awards were made in the NIST program in 1986. In 2007, not all application cycles have been completed and recorded in the database. Source: National Academies, DataRAP Database, tabulations by staff. Since the number of research associates is relatively small for NIST, these declinations may be more noticeable. Interestingly, the rate of declined offers is itself declining. Perhaps the program is becoming more prestigious, the job market has changed, or the offer process has changed. PRELIMINARY RESULTS Outreach efforts produce more qualified applicants than NIST has slots to fill for research associates; and the pool of applicants includes many from top research institutions and is increasingly diverse. Overall, 22 percent of applicants were awarded an appointment—a lower ratio than for RAPs elsewhere. Women are increasingly applying to the NIST/NRC RAP and being awarded research associateships. The NIST/NRC RAP seems to be as popular as the other RAPs for women. Underrepresented minorities are also increasingly applying to the NIST/NRC RAP and being awarded research associateships. For applicants to the NIST/NRC RAP and awardees, at least half came from 20 of the top doctoral-granting institutions in the United States Applicants and awardees to the NIST/NRC RAP differ from their counterparts in the other RAPs. Since 1990, underrepresented minorities are proportionately more likely to be awarded a NIST/NRC Research Associateship than a research associateship in another program. Applicants to, and awardees of, NIST research associateships are younger on average than those who apply for other research associateships. NIST/NRC RAP applicants and awardees are more likely to be single. They are more likely to have Ph.D.s in the physical sciences than biological. The majority of awards go to doctorates from the physical sciences. But, because there are so many applications from this discipline, only about one in five applicants with this background receive awards. Preliminary analysis suggests that labs receive different amounts of applications and awards are not made uniformly across different labs. Some awardees do decline NIST/NRC Research Associateships, though the percentage of declined offers is often lower than that for the other RAPs and has declined over time. RECOMMENDATIONS NIST should conduct an evaluation of outreach efforts. To conduct such an evaluation, data need to be collected. In this regard, the question on the application about how applicants hear about the program is
OCR for page 60
Approaches for Evaluating the NRC Resident Research Associateship Program at NIST helpful and should be retained. However, the “Other” category should be further analyzed and a choice of “Website” should be added as a category. Additional data could be collected from NIST personnel and former or current NIST RAs. Such data could be used to answer such questions as: What mechanisms do NIST personnel and RAs use to interact with potential applicants and Which mechanisms seem to work best? Has there been any effort to focus specifically on diversity? How? Such research could be undertaken via a combination of expert panels or surveys of NIST staff and current or former RAs to answer the first and third questions and to provide information for an assessment of the second question. Information should also be collected on the costs for individual outreach efforts (e.g., money spent on advertisements, time spent meeting with graduates) to compare to the benefits (how many applicants come from each individual outreach type). A second step to facilitate an evaluation of outreach efforts is to identify metrics for quantifying value obtained from different outreach strategies, such as hits to the website or number of graduate students met with at professional meetings. Examine individual outreach strategies for return on investment. This could include such strategies as assessing the NIST website for usability and informational content or assessing the return on advertising in publications. As part of the assessment of the NIST Web site, NIST could consider adding contact information for research advisors to facilitate a dialogue between potential applicants and relevant NIST staff. Finally, consider whether there might be other outreach strategies that are being underused currently, and which might have potential value, such as direct mail to deans, department heads and other university administrators. In addition, it is important to determine if any groups of graduate students—and potential applicants—who would make good candidates for the NIST/NRC RAP are unaware of the Program and how one applies. It would be difficult to craft a random sample of graduate students, but a limited survey might be possible. NIST should conduct an evaluation of individuals who decline offers of Research Associateships. This could be done as a telephone interview or via a survey. As there are only a few people who decline each cycle, the burden would be relatively small. Two basic questions should be asked of those who are awarded but decline: (1) why are you declining, and (2) what are you planning to do instead? The NRC should amend the application form. The number of fields should be reduced, in particular by collapsing very similar labels and by removing labels that are for multiple fields (e.g., “Biophysics Physics Biochemistry”). At least with regard to Ph.D. fields, an example of a smaller field list is found in the NSF’s Survey of Earned Doctorates (see Appendix B). The NRC should update the DataRAP database to replace organizational names (e.g., institutes or labs) that no longer exist at NIST with current equivalents.