. "2 Recruitment and Selection." Approaches for Evaluating the NRC Resident Research Associateship Program at NIST. Washington, DC: The National Academies Press, 2007.
The following HTML text is provided to enhance online
readability. Many aspects of typography translate only awkwardly to HTML.
Please use the page image
as the authoritative form to ensure accuracy.
Approaches for Evaluating the NRC Resident Research Associateship Program at NIST
2 Recruitment and Selection
The NRC Research Associateship Program at NIST (hereafter “NIST/NRC RAP”) is thought to be a value to both the postdoctoral recipients and to NIST itself. This chapter is divided into three sections. First, the recruitment of RAs is put into context by examining trends in Ph.D. production and trends in postdoctoral appointments. Second, the application process is examined. Finally, characteristics of applicants and awardees are described.
TRENDS IN DOCTORATES AND POSTDOCTORATES
Trends in Doctorates
As noted in the previous chapter, information on doctorates comes from the National Science Foundation’s Survey of Earned Doctorates (see Appendix B for a recent questionnaire). The number of doctorates granted in the United States has generally grown over the past 100 years, peaking in 1973 and 1998. Prior to 1953, more doctorates were produced at private institutions (NSF, 2006c). Examining trends from 1920-1999, for all doctorates—not just U.S. citizens:
About two-thirds of doctorates were awarded in science and engineering (S&E).
Between 1920 and 1974, between 87.2 and 93.6 percent of doctorates in S&E were awarded to men. This figure dropped steadily from 1975 to 1999. In the period 1995-1999, it stood at 66.8 percent.30
A growing percentage of S&E doctorates were awarded to foreign nationals: 38.6 percent by the 1990s.31
Underrepresented minorities receive few Ph.D.s relative to whites and Asians: 7.4 percent of S&E doctorates awarded in the second half of the 1990s went to underrepresented minorities.32 But the number and percentage of underrepresented minorities receiving S&E Ph.D.s has grown from the 1970s to the 1990s.
The median age of doctorate recipients has been increasing over time; although the median age for recipients of Ph.D.s in S&E is much lower than the age for those receiving non-S&E Ph.D. degrees.
A majority of doctoral recipients were married at the time of graduation, though the percentage of married graduates has been declining since the 1960s (NSF, 2006c).
For the years 2000 to 2005, some trends have continued (see appendix C for underlying data). In 2005, S&E doctorates accounted for 64 percent of all doctorates awarded, which is similar to the ratio in the 1990s. However, more and more women are receiving doctorates in S&E. In 2005, about 38 percent of S&E Ph.D.s went to women. The number of U.S. citizens
Below are the first 10 and last 10 pages of uncorrected machine-read text (when available) of this chapter, followed by the top 30 algorithmically extracted key phrases from the chapter as a whole. Intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text on the opening pages of each chapter.
Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.
Do not use for reproduction, copying, pasting, or reading; exclusively for search engines.
OCR for page 23
Approaches for Evaluating the NRC Resident Research Associateship Program at NIST
2
Recruitment and Selection
The NRC Research Associateship Program at NIST (hereafter “NIST/NRC RAP”) is thought to be a value to both the postdoctoral recipients and to NIST itself. This chapter is divided into three sections. First, the recruitment of RAs is put into context by examining trends in Ph.D. production and trends in postdoctoral appointments. Second, the application process is examined. Finally, characteristics of applicants and awardees are described.
TRENDS IN DOCTORATES AND POSTDOCTORATES
Trends in Doctorates
As noted in the previous chapter, information on doctorates comes from the National Science Foundation’s Survey of Earned Doctorates (see Appendix B for a recent questionnaire). The number of doctorates granted in the United States has generally grown over the past 100 years, peaking in 1973 and 1998. Prior to 1953, more doctorates were produced at private institutions (NSF, 2006c). Examining trends from 1920-1999, for all doctorates—not just U.S. citizens:
About two-thirds of doctorates were awarded in science and engineering (S&E).
Between 1920 and 1974, between 87.2 and 93.6 percent of doctorates in S&E were awarded to men. This figure dropped steadily from 1975 to 1999. In the period 1995-1999, it stood at 66.8 percent.30
A growing percentage of S&E doctorates were awarded to foreign nationals: 38.6 percent by the 1990s.31
Underrepresented minorities receive few Ph.D.s relative to whites and Asians: 7.4 percent of S&E doctorates awarded in the second half of the 1990s went to underrepresented minorities.32 But the number and percentage of underrepresented minorities receiving S&E Ph.D.s has grown from the 1970s to the 1990s.
The median age of doctorate recipients has been increasing over time; although the median age for recipients of Ph.D.s in S&E is much lower than the age for those receiving non-S&E Ph.D. degrees.
A majority of doctoral recipients were married at the time of graduation, though the percentage of married graduates has been declining since the 1960s (NSF, 2006c).
For the years 2000 to 2005, some trends have continued (see appendix C for underlying data). In 2005, S&E doctorates accounted for 64 percent of all doctorates awarded, which is similar to the ratio in the 1990s. However, more and more women are receiving doctorates in S&E. In 2005, about 38 percent of S&E Ph.D.s went to women. The number of U.S. citizens
30
These data are available at: http://www.nsf.gov/statistics/nsf06319/pdf/fig03-03.pdf.
31
These data are available at: http://www.nsf.gov/statistics/nsf06319/pdf/fig03-06.pdf.
32
Underrepresented minorities includes American Indian/Alaskan Native, Black, and Hispanic; and excludes Asian/Pacific Islander.
OCR for page 24
Approaches for Evaluating the NRC Resident Research Associateship Program at NIST
receiving doctorates in science and engineering has declined somewhat from 2000 to 2005. Among doctorates where citizenship was known, in 2005, only about 56 percent of S&E Ph.D.s were awarded to U.S. citizens. Finally, the number of American Indian/Alaska Natives receiving S&E Ph.D.s has declined from 2000 to 2005; the number of Black/African Americans receiving S&E Ph.D.s has stagnated; and the number of Hispanics receiving S&E Ph.D.s has increased somewhat. As a result the same percent—about 10—of S&E Ph.D.s went to underrepresented minorities.
Trends in Postdoctoral Appointments
Postdoctoral appointments date back over 100 years; however the hiring of postdocs did not grow significantly until the second half of the twentieth century. An initial period of rapid growth occurred in the 1950s, stimulated by the Cold War demand for scientists and engineers. In the 1970s, and again during the recession of the 1990s, the number of postdoctoral positions increased due to a weaker economic market for Ph.D.s. (NAS/NAE/IOM, 2000; Davis, 2005).
Postdoctoral appointments can provide benefits both to the recipients and the employers. For postdocs, the position is a way to obtain further training. Postdoctoral appointments in federal labs or industry can be an entrée into non-academic careers. Concerning the impact on the employer, one report notes that “As a whole, the postdoctoral population has become indispensable to the science and engineering enterprise, performing a substantial portion of the nation’s research in every setting. For example, a survey of research articles in two recent issues of Science found that 43 percent of the first authors were postdocs.33 In many labs, postdocs also educate, train, and supervise junior members, help write grant proposals and papers, and present the laboratory’s research results at professional society meetings” (NAS/NAE/IOM, 2000:10). However, it is important to note that there have been some complaints about the situation for postdocs.
According to the NSF, in 2005 there were approximately 35,000 postdocs in academia, across all science and engineering fields broadly defined (NSF, 2007). However, there are differences by field. “In some fields, such as computer science and engineering, there is relatively little incentive to pursue a postdoc—or even a Ph.D.—because rewarding jobs are available at the bachelor’s and master’s levels. In other fields, such as biology and physics, a postdoc is virtually mandatory, especially for academic employment” (NAS/NAE/IOM, 2000:14). Table 2-1 gives a field breakdown for number of postdoctoral appointees, while Table 2-2 lists the percentage of doctoral recipients with definite plans to pursue postdoctoral study or research by field.
33
Vogel, G. Science, 1999, Vol. 285, p. 1531.
OCR for page 25
Approaches for Evaluating the NRC Resident Research Associateship Program at NIST
TABLE 2-1 Science and Engineering Postdoctoral Appointees in Doctorate-Granting Institutions, by Field, 1998-2005
Field
1998
1999
2000
2001
2002
2003
2004
2005
Science and engineering
27,826
28,943
30,197
30,163
31,871
33,516
33,898
r
34,535
Science
24,973
25,747
26,884
26,997
28,303
29,696
29,935
r
30,374
Agricultural sciences
695
749
822
835
945
1,052
941
988
Biological sciences
15,755
16,091
16,729
17,022
17,640
18,605
18,675
r
18,995
Computer sciences
371
332
341
335
359
358
384
406
Earth, atmospheric, and ocean sciences
898
923
1,155
1,036
1,113
1,166
1,253
1,364
Mathematical sciences
279
351
385
353
391
447
466
496
Physical sciences
5,973
6,136
6,252
6,198
6,587
6,707
6,945
6,865
Engineering
2,853
3,196
3,313
3,166
3,568
3,820
3,963
4,161
Note: r = data significantly revised; replaces previously published data.
Source: National Science Foundation/Division of Science Resources Statistics, Survey of Graduate Students and Postdoctorates in Science and Engineering in NSF (2007). Adapted from Table 49.
TABLE 2-2 Percent of Doctoral Recipients with Definite Commitments Who Plan Postdoctoral Study or Research, by Broad Field of Study, 1982 and 2002
Field of Study
1982
2002
Biological sciences
72.1
74.4
Physics/astronomy
48.8
66.7
Chemistry
39.8
52.9
Earth, atmospheric, ocean sciences
25.9
51.6
Mathematics
15.8
42.5
Agricultural sciences
15.9
38.3
Engineering
11.4
24.8
Health sciences
15.4
21.1
Computer sciences
9.1
19.7
Source: NSF/NIH/USED/NEH/USDA/NASA, Survey of Earned Doctorates, in Hill et al., 2004: Figure 1.
As there are differences by field, so too do postdocs vary by demographic characteristics. Based on the Sigma Xi survey of postdocs, the following conclusions are noted:
The majority of postdocs in the life and health sciences, in the physical sciences, and in engineering are men. Men also comprise the majority of postdocs who are temporary-visa holders.
About 75 percent of citizen and permanent resident postdocs identified themselves as white.
The majority of postdocs responding to the survey held temporary visas. 40 percent were U.S. citizens and 6 percent were permanent residents.
The majority of postdocs were between 30 and 35 years old; 69 percent were married or otherwise partnered; and about a third had children (Davis, 2005).34
34
It is difficult to know how generalizable the results of this survey are: the percentages are based on a 34 percent response rate and from postdocs at select institutions. Nonresponse bias may have affected the survey estimates.
OCR for page 26
Approaches for Evaluating the NRC Resident Research Associateship Program at NIST
THE SELECTION PROCESS
The process by which applicants apply and are selected to become NIST/NRC RAs can be summarized in a few basic steps:
Potential applicants hear about the NIST/NRC RAPs
Applicants apply to the Program
Applications are reviewed by expert panels overseen by the NRC’s Fellowships Office. Each applicant receives a rating based on the average scores of three reviewers (possibly two more if the scores are disparate (differ by 1.5 points between highest and lowest score)
The Fellowships Office forwards ranked (highest to lowest rated) applicants on to NIST
Partly on the basis of those rankings, NIST offers selected applicants postdoctoral positions
Most of those who receive offers accept and become research associates
There are a number of ways to publicize postdoctoral positions. Both the National Academies and NIST have links to the program on their respective websites.35 Staff from the National Academies attend conferences where they make information on the program available. Advertisements are also placed in relevant publications, such as Physics Today, Science, and the Chronicle of Higher Education.
Once prospective applicants hear about the program, the next step is to complete an application (see Appendix D). Noteworthy information collected by the application includes:
Educational background
Demographic data (e.g., citizenship, gender, date of birth, race/ethnicity, marital status)
How the applicant heard about the program
Previous research and publications
A research proposal intended to be carried out during the postdoctoral tenure
Applicants also have recommendations submitted on their behalf. In addition, the proposed research advisor at the laboratory/center also reviews the applicant’s proposed research project.
Completed applications are collected by the National Academies. This is followed by a review process, as described on the National Academies’ Web site “Review Criteria” (see Box 2-1).36
35
The National Academies, “RAP Home,” available at: http://www7.nationalacademies.org/rap/ and NIST, “NIST Postdoctoral Research Associateships Program,” available at: http://www.nist.gov/oiaa/postdoc.htm.
36
The National Academies, “Review Criteria,” available at: http://www7.nationalacademies.org/rap/Review_Criteria.html.
OCR for page 27
Approaches for Evaluating the NRC Resident Research Associateship Program at NIST
Box 2-1
Review Criteria
Applications for awards from the NRC Research Associateship Programs are reviewed by panels of experts in 6 broad discipline areas: Chemistry; Earth and Atmospheric Sciences; Engineering, Applied Sciences and Mathematics; Life Sciences; Physics; and Space Sciences. Each application is read by a minimum of 3 panelists. Panelists assess the quality of an application, the likelihood for success and the contribution of the research to the mission of the sponsoring federal laboratory. Postdoctoral applicants are evaluated on the basis of demonstrated ability as a student and on their potential for making contributions as an independent scientist. Senior applicants, including applicants to Summer Faculty programs, are evaluated on the basis of proven ability and demonstrated research accomplishments. Evaluations are made without regard to age, sex, marital status, national origin, creed, racial group, or ethnic group.
Each application is assigned a numerical score and the applicant’s final score is an average of all reviews. Scoring is on a 10-point scale and only applicants scoring 7.5 or above are considered for awards. Sponsoring laboratories offer awards to the highest scoring applicants first and continue to make awards until available slots are filled. In the review process each applicant is evaluated on four major elements with the approximate weighting as indicated:
Scientific merit of the proposed research (40%)
The research proposal is the most important element of the application and as such is weighted most heavily in the review. The proposal is evaluated for: importance of the proposed research area, clearly stated objectives, technical soundness of the work plan, innovative aspects of the proposal, feasibility of success, timeliness (can the proposal be completed in the allotted time), likelihood that the research will result in publication, and contribution of the research to the mission of the sponsoring laboratory.
Reference reports or letters of recommendation (20%)
Reference Reports or letters of reference contain opinions of persons who should have had a close professional relationship with an applicant; references provide reviewers with important information regarding the applicant’s scholarly abilities. Reference reports are given greater weight for Postdoctoral applicants, where a publication record may not be as extensive as that of a Senior applicant.
Academic and research record (20%)
Panelists review the appropriateness of the applicant’s training for the proposed research, previous research experience and record of publication. For Postdoctoral applicants only, a transcript of the academic record is required.
Laboratory technical evaluation (20%)
The Laboratory/Center Review form includes comments of the prospective Advisor and the disposition of the Laboratory/Center’s program committee or representative concerning the suitability of the applicant’s proposed research. This information aids reviewers in determining the value of the proposed research to the sponsoring agency.
OCR for page 28
Approaches for Evaluating the NRC Resident Research Associateship Program at NIST
During the expert panel meetings, applicants are rated from 10 to 0, with 10 being the highest possible score. In practice, scores have ranged from 9.90 to 0 and applicants’ scores may differ by as little as 0.02 (e.g., 9.65 to 9.63). The review process is seen as something of a mystery to NIST staff. During the expert panel with former research associates, they commented that they did not understand how candidates were ranked. Former research associates had their own ideas about how candidates should be ranked. As a consequence of the personal connections between NIST employees and potential applicants, NIST employees formed opinions about who they thought top candidates were, and then registered their surprise when those people did not end up ranked at the top. In a related comment, former research associates commented that they would like to have a much better idea of how the research project that the applicant intends to work on (as described in the application) is graded. They felt that the current grading system is not transparent and is too general. NIST staff submit a recommendation on behalf of applicants and the staff do not seem sure what they ought to stress in their recommendation.
Advisors and division chiefs focused on another issue in their expert panel: the review panels. They questioned whether the NRC review panels were organized well. They were concerned the panels were skewed to reviewers from academia. They did not know who serves on the panels. They would like to have more input into panel makeup, for example by suggesting names of potential reviewers (and then knowing if their suggestions were used). Another concern was whether rankings were normalized across panels.
The applicants’ reviews, with scores, are sent to NIST, which selects the applicants to be offered postdoctoral positions. In practice, most of those offered, will accept a Research Associate.
RECRUITMENT
A concern for NIST and The National Academies is how well the program is reaching out to potential applicants. During the expert panels, current research associates noted that NIST was often their first choice. Reasons why included: family reasons, quality of advisors, ability to collaborate, and location. The research associates had applied to multiple positions, for example in academia or other government labs. Concerning how research associates heard about the position, the most frequent answer involved personal communication—either they met their future advisor at a conference, job fair, or when the advisor gave a lecture at their school; or a graduate advisor suggested they apply. (Personal communication was also repeatedly mentioned by the advisors and chiefs—that personal relationships had the best return on investment.) In several cases, their advisors were former research associates. Word of mouth was very important.
Former research associates echoed these comments. Answers given by participants included: had a personal relationship—in one case the former research associate’s spouse worked at NIST; were recruited by advisors (at conferences or presentations made by NIST employees at universities); met researchers at NIST (this was suggested by former RA from local universities); and came across the Web site describing the RAPs. One RA had applied to a regular position at NIST and then saw the listing for the RAP. The former RAs noted that sometimes staff invite graduate students to give research talks at NIST as a way to bring potential applicants out to NIST. It did seem to the participants that the personal connections were much more effective. One former research associate noted that there is less outreach by NIST in the biological areas. This is important, as NIST may be moving in the future to more interdisciplinary research that
OCR for page 29
Approaches for Evaluating the NRC Resident Research Associateship Program at NIST
has a biological component and there will be a need for more people with training in biological sciences. A physicist noted that at one point in the past, there was an advertisement in Physics Today, but he had not seen it recently. He felt that a better job of advertising could be done. Finally, former research associates noted that some had applied to multiple postdoctoral positions and had chosen NIST as the better alternative.
Advisors and lab chiefs did note that in some areas they felt that the number of applications was low. Of particular concern were some areas within engineering and computer science. They noted that doctorates had many options in these areas and many forego postdocs, as well as the view that there are more foreign nationals and fewer U.S. citizens getting degrees in these areas. They did agree with the current and former research associates that personal communication seemed to be the best way to recruit. Participants in this panel did note that different labs differed in how they attempted to recruit applicants and to what degree they tried.
Overall, participants reported that:
The program is very prestigious (although some current RAs felt that the program was less so);
The program is well-known (again, with some minority comment that it is not that well known); and
The most common way people heard about the program was through personal communication.
To examine these findings more broadly, the committee turned to data collected on the NRC’s Fellowships Office RAP application form. The application for postdoctoral programs includes a question on how the applicant heard about the position.37 Applicants were requested to select one of the following: colleague or fellow graduate student; Ph.D. thesis advisor or other professor; university placement office; former or current NRC Research Associate; research advisor or other scientific staff at the federal Laboratory; RAP’s staff member at professional scientific meeting; Advertisement in professional publication; or other. Data are available for 1989 to 2007.
The dataset contained 24,849 applications, of which 2,743 were applications to the NIST/NRC RAP. The number of applicants is much less than this since applicants can apply for multiple positions in the same year or across years. (Because the NIST/NRC RAP was until recently reviewed once per year, there are only a handful of cases where an applicant applied more than once per year to this particular program.)38 In assessing how applicants heard about the program, we combined information from individuals who applied for multiple positions, if they selected different information sources, into a single record. This was done because in most cases, an applicant applying to multiple positions identified the same source in each case. Thus, the dataset was updated so that there was one record for each individual, regardless of how many applications they submitted. An exception occurred for applicants who applied to both the NIST/NRC RAP and any other RAP. Since a goal is to compare the sources identified by applicants to NIST and all other federal agency RAP postdoctoral positions, any individual who applied to both programs remained in the database twice. Two hundred and thirty-six individuals who applied to both NIST and other federal positions at the same time fit this
37
The question is: “To assist us in making information available to a greater number of potential applicants, it is important for us to learn how you initially heard about the National Academies RAPs.”
38
It is not clear if this is a data entry error.
OCR for page 30
Approaches for Evaluating the NRC Resident Research Associateship Program at NIST
exception. After reducing duplicate entries, the database consisted of 12,737 records: 2,717 applicants to NIST and 10,020 applicants to other federal RAPs (with 236 applicants appearing on both lists). The NIST applicants identified a total of 2,890 sources and the non-NIST applicants identified a total of 10,497 sources.
As Table 2-3 shows, the applicants to the NIST/NRC RAP were twice as likely as applicants to the other RAPs to hear about the position initially from their Ph.D. advisor or other professor and somewhat more likely to hear about the program from colleagues or fellow graduate students. Surprisingly, they were less likely to hear about the program from a research advisor or other scientific staff at the federal laboratory, compared with applicants to other federal RAPs.
TABLE 2-3 How Applicants First Heard About the Research Associateship Program, 1989-2007
Source of Information
NIST/NRC RAP (%)
Other RAPs (%)
All Programs (%)
Colleague
23.3
20.9
21.4
Professor
33.7
16.6
20.3
Placement office
1.0
1.7
1.5
NRC associate
8.2
9.3
9.1
Laboratory staff
18.0
26.7
24.8
Journal
3.0
8.7
7.4
NRC staff
0.2
0.3
0.3
Professional meeting
4.8
5.3
5.2
Other
7.8
10.6
10.0
Source: National Academies, DataRAP Database, tabulations by staff.
Regarding advertisements in professional publications, applicants to NIST/NRC RAP identified about 30 publications, while applicants to non-NIST RAPs identified approximately 190 publications. (However, since the other RAPs include a broader set of fields, this larger number of publications should be expected and the smaller number of publications for the NIST/NRC RAP is not an indication, by itself, of less effort to reach potential applicants via publications. Top publications identified by applicants to NIST/NRC RAP: Physics Today, Fellowships Office mailing, Chemical & Engineering News, Mechanical Engineering, Spectrum of the IEEE, and Science. Examining trends in the sources cited over time, as is done in Table 2-4, shows that applicants for NIST/NRC RAs did not usually first find out about the program via an advertisement in a publication.
OCR for page 31
Approaches for Evaluating the NRC Resident Research Associateship Program at NIST
TABLE 2-4 How Applicants to the NIST/NRC Research Associateship Program First Heard About the Program, 1989-2007
Year
Colleague (%)
Professor (%)
Placement Office (%)
NRC Associate (%)
Lab Staff (%)
Journal (%)
NRC Staff (%)
Professional Meeting (%)
Other (%)
N
1989
100.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
1
1990
10.3
38.3
0.9
5.6
27.1
6.5
0.0
1.9
9.3
107
1991
33.3
63.0
3.7
13.9
27.8
12.0
0.0
9.3
18.5
108
1992
22.5
34.7
2.3
10.8
20.3
5.0
0.0
3.2
1.4
222
1993
22.6
35.5
0.9
9.0
15.0
6.8
0.9
5.1
4.3
234
1994
15.6
18.9
0.9
5.3
5.0
2.9
0.2
2.6
2.0
456
1995
34.3
37.7
1.0
0.5
17.4
1.9
0.0
4.8
2.4
207
1996
31.1
28.8
0.8
0.8
24.2
2.3
0.0
6.1
6.1
132
1997
15.6
14.7
0.6
3.2
6.5
0.9
0.3
2.4
6.8
339
1998
38.2
27.3
0.0
4.5
14.5
0.0
0.9
5.5
9.1
110
1999
23.3
22.5
0.8
14.2
23.3
0.8
0.0
2.5
12.5
120
2000
6.1
15.7
0.9
6.1
12.2
0.9
0.4
1.7
5.7
230
2001
21.7
34.8
0.0
13.0
19.6
0.0
0.0
4.3
6.5
46
2002
18.1
23.3
0.0
14.4
23.3
1.9
0.0
8.4
10.7
215
2003
11.5
24.5
0.0
4.2
10.3
0.8
0.0
6.1
6.9
261
2004
20.3
38.0
0.6
7.6
17.7
1.3
0.0
1.3
13.3
158
2005
23.1
34.0
1.3
12.2
16.7
1.3
0.0
2.6
9.0
156
2006
10.2
23.6
0.6
3.8
12.7
0.6
0.0
3.2
2.9
314
2007
22.4
40.2
0.0
6.5
15.9
0.9
0.0
4.7
9.3
107
Source: National Academies, DataRAP Database, tabulations by staff.
Journals seemed to be more important as a source in the first half of the 1990s than in the current decade. Another finding of note is that placement offices at universities tend to be an infrequently cited source of information. Although the committee did not have prior expectations, it was still a bit surprising that the proportion of applicants hearing about the program from former or current RAs was not that great.
We next examined some characteristics of applicants to the NIST/NRC RAP to see if different types of applicants differed on how they first heard about the program. Comparing men and women, we found no significant differences, except for presentations at professional meetings, where women were twice as likely as men to first hear about the program.
OCR for page 32
Approaches for Evaluating the NRC Resident Research Associateship Program at NIST
TABLE 2-5 How Applicants to the NIST/NRC Research Associateship Program First Heard About the Program, by Gender, 1989-2007
Source of Information
Women (%)
Men (%)
Colleague
22.2
23.6
Professor
32.1
34.1
Placement office
0.6
1.1
NRC associate
8.1
8.2
Laboratory staff
17.4
18.2
Journal
2.4
3.1
NRC staff
0.2
0.2
Professional meeting
8.3
4.1
Other
8.7
7.6
N
505
2385
Source: National Academies, DataRAP Database, tabulations by staff.
Turning to race/ethnicity, we compared how whites and all other ethnic/racial groups first heard about the program. Similar to women, the results were fairly consistent across these two groups and minorities were more likely than whites to hear about the program via a presentation at a professional meeting—but not significantly so.
TABLE 2-6 How Applicants to the NIST/NRC Research Associateship Program First Heard about the Program, by Race/Ethnicity, 1989-2007
Source of Information
White (%)
All Other (%)
Colleague
23.2
20.7
Professor
34.6
33.0
Placement office
0.9
1.1
NRC associate
8.0
8.6
Laboratory staff
18.0
18.7
Journal
2.5
2.9
NRC staff
0.1
0.6
Professional meeting
4.9
6.6
Other
7.9
7.8
N
2298
348
Source: National Academies, DataRAP Database, tabulations by staff.
As a next step, future analysis could examine the relationship between different sources of information among applicants and outcomes of applications. For example, Table 2-7 examines this association in general for applicants to NIST/NRC RAPs and other RAPs.
OCR for page 33
Approaches for Evaluating the NRC Resident Research Associateship Program at NIST
TABLE 2-7 Percent of Awardees Among Applicants by Source of Information About the Program, 1965-2007
Awardees among Applicants (%)
Source of Information
NIST/NRC RAP
Other RAP
Colleague
28.2
36.1
Professor
26.1
40.3
Placement office
15.6
30.5
NRC associate
31.4
43.2
Laboratory staff
28.3
47.3
Journal
17.2
26.4
NRC staff
100.0
42.3
Professional meeting
28.4
37.0
Other
29.2
28.8
Source: National Academies, DataRAP Database, tabulations by staff.
As Table 2-7 illustrates, successful applicants to the NIST/NRC RAP were more likely to hear about the Program from NRC staff or an NRC associate; although in these cases, few applicants had heard about the Program from these sources. There seems to be much more variability in how successful applicants hear about the NIST/NRC RAP, as compared with successful applicants to the other RAPs. Table 2-8 focuses on just those applicants who received RAs.
TABLE 2-8 Percent of Awardees by Source of Information About the Program, 1965-2007
Awardees (%)
Source of Information
NIST/NRC RAP
Other RAP
Colleague
23.9
19.4
Professor
32.2
17.0
Placement office
0.6
1.3
NRC associate
9.3
10.4
Laboratory staff
18.3
32.7
Journal
1.9
5.9
NRC staff
0.2
0.3
Professional meeting
5.1
5.0
Other
8.4
7.9
N
825
4022
Source: National Academies, DataRAP Database, tabulations by staff.
Table 2-8 shows that about one-third of awardees to the NIST/NRC RAP first heard about the program via a professor; whereas about one-third of awardees to the other RAPs first heard about the program from lab staff. Taken together, Tables 2-6 to 2-8 suggest that personnel communication was the most important mechanism for transmitting information about the pograms to prospective applicants. This sort of analysis could be extended by focusing on subsets of applicants, that is, by gender, race/ethnicity, or discipline, to see how well outreach succeeds.
OCR for page 50
Approaches for Evaluating the NRC Resident Research Associateship Program at NIST
Pennsylvania State University Park
40
2.9
Johns Hopkins University
37
2.7
Princeton University
34
2.4
University of Maryland College Park
33
2.4
Carnegie Mellon University
30
2.2
University of Florida
28
2.0
Cornell University
27
1.9
University of Virginia
25
1.8
University of Minnesota-Twin Cities
24
1.7
University of Massachusetts-Amherst
24
1.7
University of Texas-Austin
23
1.7
Total
718
51.7
Mathematics/CS
Number of Applications from Institution
%
University of Wisconsin-Madison
15
5.8
University of Maryland College Park
13
5.0
Northwestern University
11
4.3
Cornell University
11
4.3
Johns Hopkins University
9
3.5
Purdue University
8
3.1
University of California-Berkeley
7
2.7
New York University
7
2.7
University of California-Santa Barbara
6
2.3
State University of New York-Stony Brook
6
2.3
Brown University/RI
6
2.3
Massachusetts Institute of Technology
5
1.9
University of Michigan
5
1.9
University of Colorado
5
1.9
Ohio State University
5
1.9
University of Southern California
5
1.9
Total
124
48.1
Physical sciences
Number of Applications from Institution
%
University of California-Berkeley
172
3.9
Cornell University
155
3.5
University of Colorado
155
3.5
University of Maryland College Park
150
3.4
Harvard University
129
2.9
Massachusetts Institute of Technology
122
2.8
University of Illinois-Urbana-Champaign
109
2.5
University of Wisconsin-Madison
106
2.4
University of Chicago
95
2.2
Stanford University
88
2.0
Pennsylvania State University Park
84
1.9
University of Michigan
81
1.8
University of Texas-Austin
81
1.8
Yale University
71
1.6
University of Virginia
69
1.6
University of California-Santa Barbara
66
1.5
OCR for page 51
Approaches for Evaluating the NRC Resident Research Associateship Program at NIST
Iowa State University
66
1.5
State University of New York-Stony Brook
56
1.3
California Institute of Technology
55
1.2
University of Florida
55
1.2
Total
1965
44.5
Note: In 2007, not all application cycles have been completed and recorded in the database.
Source: National Academies, DataRAP Database, tabulations by staff.
TABLE 2-16 Most Common Doctoral-Granting Institutions of NIST/NRC Research Associates, by Major Field, 1965-2007
Bio/Biomed/Health
No.
%
Johns Hopkins University
3
10.0
Georgetown University
2
7.0
Johns Hopkins University-Medical Insts.
2
7.0
State University of New York-Stony Brook
2
7.0
Total
9
31.0
Engineering
No.
%
Massachusetts Institute of Technology
21
6.0
University of Michigan
21
6.0
Princeton University
16
4.6
Northwestern University
14
4.0
Pennsylvania State University Park
12
3.4
University of Illinois-Urbana-Champaign
12
3.4
Carnegie Mellon University
11
3.2
Stanford University
11
3.2
University of Massachusetts-Amherst
11
3.2
University of California-Berkeley
10
2.9
University of Colorado
8
2.3
University of Minnesota-Twin Cities
8
2.3
Virginia Polytech Institute and State U
8
2.3
University of Florida
7
2.0
University of Texas-Austin
7
2.0
University of Washington
7
2.0
Iowa State University
7
2.0
Lehigh University
7
2.0
University of Maryland College Park
7
2.0
Johns Hopkins University
6
1.7
Total
211
60.5
Mathematics/CS
No.
%
University of Maryland College Park
4
6.0
University of Wisconsin-Madison
4
6.0
Northwestern University
3
5.0
Pennsylvania State University Park
3
5.0
Cornell University
3
5.0
University of California-Santa Barbara
3
5.0
University of Colorado
2
3.0
OCR for page 52
Approaches for Evaluating the NRC Resident Research Associateship Program at NIST
University of Washington
2
3.0
Johns Hopkins University
2
3.0
Ohio State University
2
3.0
Syracuse University
2
3.0
Total
30
47.0
Physical sciences
No.
%
University of Colorado
47
5.0
Harvard University
45
4.8
University of Maryland College Park
41
4.4
University of California-Berkeley
40
4.3
Cornell University
35
3.7
University of Illinois-Urbana-Champaign
34
3.6
University of Wisconsin-Madison
31
3.3
Massachusetts Institute of Technology
29
3.1
Stanford University
28
3.0
Yale University
21
2.2
University of Texas-Austin
21
2.2
California Institute of Technology
17
1.8
University of Chicago
17
1.8
University of California-Santa Barbara
16
1.7
University of Virginia
16
1.7
University of Michigan
16
1.7
Iowa State University
16
1.7
Pennsylvania State University Park
15
1.6
Indiana University-Bloomington
13
1.4
Total
498
53.0
Note: In 2007, not all application cycles have been completed and recorded in the database.
Source: National Academies, DataRAP Database, tabulations by staff.
Age
As noted in the start of the chapter, the average age of postdocs has been creeping upwards, in part due to longer time to complete doctorates. Figure 2-12 compares the average age of applicants to and awardees of NIST/NRC RAP and the other RAPs.
OCR for page 53
Approaches for Evaluating the NRC Resident Research Associateship Program at NIST
FIGURE 2-12 Average age of applicants and awardees, by Research Associateship Program, 1965-2007.
Note: In 2007, not all application cycles have been completed and recorded in the database.
Source: National Academies, DataRAP Database, tabulations by staff.
As the figure shows, awardees tend to be younger than applicants and applicants to the NIST/NRC RAP and awardees of NIST/NRC Research Associates are younger on average than those who apply for and are awarded other RAPs. It is likely that part of the explanation for this is the group of postdocs coming out of the biological sciences, who are largely absent from the NIST cohort. A second explanation may involve the role of foreign students: NIST RAs are U.S. citizens, who also tend to get to postdoctoral status quicker than international students.
Marital Status
Marital status can be an important demographic characteristic for postdoctoral programs. When many doctorates pursue postdoctoral appointments, they are also at an age when many are married and thinking about starting families. Many scientists are married to other scientists. Knowing this demographic can be helpful in dealing with related issues of: dual-career couples; salary, benefits and cost of living; child care and parental leave. The application form includes a question on marital status. Two categories are available: married and single; although many applicants leave this answer blank. (Additionally, 4 applicants chose “F”—possibly a data coding entry with gender.) Twenty-seven applicants to NIST left this question blank and 368 applicants to non-NIST left it blank. For awards, 2 awardees chose “F,” 4 awardees at NIST left it blank, and 121 awardees at other RAPs did not answer the question. The percentages of applicants and awardees that were married or single among those who noted marital status, are examined in Figures 2-13 and 2-14.
OCR for page 54
Approaches for Evaluating the NRC Resident Research Associateship Program at NIST
FIGURE 2-13 Percent of applicants who are married, by Research Associateship Program, 1965-2007.
Note: In 2007, not all application cycles have been completed and recorded in the database.
Source: National Academies, DataRAP Database, tabulations by staff.
OCR for page 55
Approaches for Evaluating the NRC Resident Research Associateship Program at NIST
FIGURE 2-14 Percent of awardees who are married, by Research Associateship Program, 1965-2007.
Note: In 2007, not all application cycles have been completed and recorded in the database.
Source: National Academies, DataRAP Database, tabulations by staff.
Applicants and RAs in other RAPs tend to be older and more likely to be married than NIST/NRC RAP applicants and awardees. (See Appendix Table B-10 for the underlying data.) As Figure 2-13 shows, other RAP applicants are more likely to be married. Figure 2-14 shows that other RAP awardees are also more likely to be married. These figures raise a question of whether the NIST/NRC RAP is less attractive to married scientists and engineers or whether some other characteristic of applicants to the NIST/NRC RAP explains the trend that NIST/NRC Research Associates are more likely to be single.
Laboratories
Applicants to the NIST/NRC RAP select a lab on their application form. Over the years NIST has reorganized, which means that some older organizational names are no longer valid, while some recent laboratories may not yet have any applicants. Since 1965, applicants have applied to 18 different parts of NIST. We used the current organizational chart to map older institutional names onto current names (see Appendix E). This was problematic in a number of ways. First, 268 applicants simply put “National Institute of Standards and Technology.” Second, several older divisions—e.g., National Engineering Laboratory and the National Measurement Laboratory—map onto multiple contemporary divisions. We combined these situations into a new category: “Multiple,” but it can also be thought of as an unknown category. Finally, in spite of the efforts to map the laboratory names, viewing the data over time shows that this is not fully successful. Applicants to Technology Services covered the years 1965 to 1978,
OCR for page 56
Approaches for Evaluating the NRC Resident Research Associateship Program at NIST
but then stopped, although the name still exists, perhaps what the applicants were researching fit better elsewhere. Other labs apparently had no applicants until 1992, again which might reflect an organizational change. Thus, analysis over time, and analysis by race/ethnicity—for which data do not exist prior to 1980—are unwarranted. With available data, two tables can be presented, which focus on applications and acceptances by lab in total, and by gender.
TABLE 2-17 Applications and Awards for the NIST/NRC Research Associateship Program, by Laboratory, 1965-2007
Laboratory
Applications
%
Acceptances
%
Accept. As % of Apps.
Building and Fire Research Laboratory
103
1.7
31
2.3
30.1
Chemical Science and Technology Laboratory
755
12.5
173
12.8
22.9
Electronics and Electrical Engineering Laboratory
355
5.9
101
7.5
28.5
Information Technology Laboratory
250
4.1
43
3.2
17.2
Manufacturing Engineering Laboratory
92
1.5
34
2.5
37.0
Materials Science and Engineering Laboratory
1448
23.9
304
22.5
21.0
Physics Laboratory
674
11.1
203
15.0
30.1
Multiple
2370
39.2
465
34.3
19.6
Total
6047
100.0
1354
100.0
22.4
Note: In 2007, not all application cycles have been completed and recorded in the database.
Source: National Academies, DataRAP Database, tabulations by staff.
The principal finding here is that applications are not evenly distributed among labs. Some labs attract more applicants than others. Likewise, some labs see more research associateships awarded than others. A second finding is that the acceptance rate varies by more than a factor of two—a large range. It may be instructive to determine why this might be the case. Possible answers could focus on the field, other opportunities for recent doctorates in those fields, and outreach by the different labs. Another possible explanation is an intentional “share the wealth” effort.
Table 2-18 continues this examination for gender. Half of the applications from women were to the Chemical Science and Technology Laboratory and the Materials Science and Engineering Laboratory. These are also the two places where a greater proportion of women receive awards. Other labs receive very few applications from women—the Building and Fire Research Laboratory, for example. A second finding is that, in general, the percentages for female applications are similar to the percentage of female awardees. For example, 7 percent of applications to the Electronics and Electrical Engineering Laboratory came from women, while 9 percent of awardees to this lab were women. Additionally, 14 percent of women who applied for the NIST/NRC RAP applied to this lab and 18 percent of women who received awards were in this lab.
Again, it may be instructive to seek out explanations for differences across the labs in terms of the gender (or for that matter, the race/ethnicity) of postdocs. Possible explanations might focus on the role of lab staff in recruiting women candidates; how female-friendly the lab is perceived to be; or the underlying number of female doctorates in fields appropriate to the research of each lab.
OCR for page 57
Approaches for Evaluating the NRC Resident Research Associateship Program at NIST
TABLE 2-18 Applications and Awards for the NIST/NRC Research Associateship Program, by Laboratory and Gender, 1965-2007
Laboratory
Female Applications (N)
Female Applications (%)
Female of Total Applications (%)
Female Awards (N)
Female Awards (%)
Female of Total Awards (%)
Building and fire research laboratory
5
0.7
4.9
1
0.5
3.2
Chemical science and technology laboratory
168
23.6
22.3
54
26.7
31.2
Electronics and electrical engineering laboratory
51
7.2
14.4
18
8.9
17.8
Information technology laboratory
28
3.9
11.2
3
1.5
7.0
Manufacturing engineering laboratory
10
1.4
10.9
1
0.5
2.9
Materials science and engineering laboratory
186
26.1
12.8
56
27.7
18.4
Multiple
176
24.7
7.4
31
15.3
6.7
Physics laboratory
89
12.5
13.2
38
18.8
18.7
Total
713
100.0
11.8
202
100.0
14.9
Note: In 2007, not all application cycles have been completed and recorded in the database.
Source: National Academies, DataRAP Database, tabulations by staff.
OCR for page 58
Approaches for Evaluating the NRC Resident Research Associateship Program at NIST
Declined Offers
One concern heard at focus groups was that NIST was not quick enough at offering applicants awards. Individuals were accepting other positions instead of coming to NIST. This section looks at offers that were accepted and those that were declined. Table 2-19 shows the number and percentage of offers declined among those who accepted or declined. On average, 7 people per year decline to accept an award from NIST, compared with an average of 68 people per year for other RAPs.
TABLE 2-19 Number of Individuals Offered a Research Associateship Who Decline, by Research Associateship Program, 1965-2007
NIST/NRC RAP
Other RAPs
Year
Number
%
Number
%
1965
0
0.0
4
3.4
1966
1
2.7
10
6.9
1967
0
0.0
8
5.2
1968
9
37.5
36
16.7
1969
8
33.3
75
35.4
1970
14
48.3
76
30.6
1971
7
30.4
89
29.2
1972
19
50.0
107
29.7
1973
9
31.0
103
33.8
1974
13
43.3
101
33.0
1975
6
26.1
88
27.8
1976
6
20.7
92
32.4
1977
12
35.3
68
32.7
1978
16
41.0
103
36.9
1979
20
45.5
85
32.1
1980
11
30.6
114
36.9
1981
23
51.1
87
28.2
1982
3
15.8
70
26.4
1983
11
31.4
79
28.7
1984
14
35.0
104
32.1
1985
14
33.3
138
36.6
1986
N/A
N/A
71
20.0
1987
8
26.7
65
20.0
1988
14
38.9
71
18.6
1989
5
20.0
54
14.6
1990
5
16.7
68
19.2
1991
2
7.1
66
17.2
1992
8
20.5
64
18.0
1993
5
12.8
82
18.8
1994
11
21.6
66
15.2
1995
2
5.0
66
19.4
1996
1
1.7
55
15.8
1997
6
13.0
53
15.8
1998
1
1.7
40
15.4
1999
2
4.9
31
13.0
OCR for page 59
Approaches for Evaluating the NRC Resident Research Associateship Program at NIST
2000
6
9.5
46
17.2
2001
0
0.0
41
12.3
2002
1
1.9
59
12.9
2003
0
0.0
46
13.0
2004
6
11.1
44
16.9
2005
15
20.3
33
16.3
2006
9
13.2
30
17.0
2007
1
2.1
11
18.0
Total
324
19.0
2799
22.2
Note: No awards were made in the NIST program in 1986. In 2007, not all application cycles have been completed and recorded in the database.
Source: National Academies, DataRAP Database, tabulations by staff.
Since the number of research associates is relatively small for NIST, these declinations may be more noticeable. Interestingly, the rate of declined offers is itself declining. Perhaps the program is becoming more prestigious, the job market has changed, or the offer process has changed.
PRELIMINARY RESULTS
Outreach efforts produce more qualified applicants than NIST has slots to fill for research associates; and the pool of applicants includes many from top research institutions and is increasingly diverse. Overall, 22 percent of applicants were awarded an appointment—a lower ratio than for RAPs elsewhere. Women are increasingly applying to the NIST/NRC RAP and being awarded research associateships. The NIST/NRC RAP seems to be as popular as the other RAPs for women. Underrepresented minorities are also increasingly applying to the NIST/NRC RAP and being awarded research associateships. For applicants to the NIST/NRC RAP and awardees, at least half came from 20 of the top doctoral-granting institutions in the United States
Applicants and awardees to the NIST/NRC RAP differ from their counterparts in the other RAPs. Since 1990, underrepresented minorities are proportionately more likely to be awarded a NIST/NRC Research Associateship than a research associateship in another program. Applicants to, and awardees of, NIST research associateships are younger on average than those who apply for other research associateships. NIST/NRC RAP applicants and awardees are more likely to be single. They are more likely to have Ph.D.s in the physical sciences than biological. The majority of awards go to doctorates from the physical sciences. But, because there are so many applications from this discipline, only about one in five applicants with this background receive awards.
Preliminary analysis suggests that labs receive different amounts of applications and awards are not made uniformly across different labs. Some awardees do decline NIST/NRC Research Associateships, though the percentage of declined offers is often lower than that for the other RAPs and has declined over time.
RECOMMENDATIONS
NIST should conduct an evaluation of outreach efforts.
To conduct such an evaluation, data need to be collected. In this regard, the question on the application about how applicants hear about the program is
OCR for page 60
Approaches for Evaluating the NRC Resident Research Associateship Program at NIST
helpful and should be retained. However, the “Other” category should be further analyzed and a choice of “Website” should be added as a category.
Additional data could be collected from NIST personnel and former or current NIST RAs. Such data could be used to answer such questions as:
What mechanisms do NIST personnel and RAs use to interact with potential applicants and
Which mechanisms seem to work best?
Has there been any effort to focus specifically on diversity? How?
Such research could be undertaken via a combination of expert panels or surveys of NIST staff and current or former RAs to answer the first and third questions and to provide information for an assessment of the second question. Information should also be collected on the costs for individual outreach efforts (e.g., money spent on advertisements, time spent meeting with graduates) to compare to the benefits (how many applicants come from each individual outreach type).
A second step to facilitate an evaluation of outreach efforts is to identify metrics for quantifying value obtained from different outreach strategies, such as hits to the website or number of graduate students met with at professional meetings.
Examine individual outreach strategies for return on investment. This could include such strategies as assessing the NIST website for usability and informational content or assessing the return on advertising in publications. As part of the assessment of the NIST Web site, NIST could consider adding contact information for research advisors to facilitate a dialogue between potential applicants and relevant NIST staff.
Finally, consider whether there might be other outreach strategies that are being underused currently, and which might have potential value, such as direct mail to deans, department heads and other university administrators.
In addition, it is important to determine if any groups of graduate students—and potential applicants—who would make good candidates for the NIST/NRC RAP are unaware of the Program and how one applies. It would be difficult to craft a random sample of graduate students, but a limited survey might be possible.
NIST should conduct an evaluation of individuals who decline offers of Research Associateships. This could be done as a telephone interview or via a survey. As there are only a few people who decline each cycle, the burden would be relatively small. Two basic questions should be asked of those who are awarded but decline: (1) why are you declining, and (2) what are you planning to do instead?
The NRC should amend the application form. The number of fields should be reduced, in particular by collapsing very similar labels and by removing labels that are for multiple fields (e.g., “Biophysics Physics Biochemistry”). At least with regard to Ph.D. fields, an example of a smaller field list is found in the NSF’s Survey of Earned Doctorates (see Appendix B).
The NRC should update the DataRAP database to replace organizational names (e.g., institutes or labs) that no longer exist at NIST with current equivalents.
