Approaches for Evaluating the NRC Resident Research Associateship Program at NIST (2007)

Below is the uncorrected machine-read text of this chapter, intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text of each book. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.

3 Research Associates’ Experiences Research Associates (RAs) spend up to two years in residence at NIST conducting research. During the expert panels we asked current and former research associates several questions about their experiences at NIST. Among the current RAs, they were quick to point out that this appointment was their first professional activity after graduate school and their first experience outside academia. Current research associates reported that one could have a fair amount of research freedom at NIST, but that it varied across labs and groups. Variation across different parts of NIST was perhaps one of the most important themes that emerged from the expert panels. The current research associates felt that the position gave them a broader view of science. They were overwhelmingly satisfied with the experience, noting that NIST was a great place to work. They also noted that being in the NIST/NRC RAP gave them better benefits and more access on the NIST campus. Finally, some participants felt that access to resources at places such as NIST were likely to be better than in other settings, such as academia. On the negative side, they felt that they faced additional bureaucracy by being in the program. Other concerns of current research associates included that there were no teaching opportunities and that there were few graduate students in their labs. Current RAs seemed to refer to the NIST/NRC RAP in terms of their recent academic experience and while some of them reported that their colleagues treated them as equals, others felt like they were the “lowest on the totem pole.” Internal review of manuscripts was also seen as something of a burden, again perhaps in reference to graduate school, although participants noted that in practice the delay caused by internal review was not long and that having additional reviewers helped improve their work. Current research associates’ suggestions for improving the program were to increase the salary, possibly increase the duration of the appointment, and create a better family leave policy. In referencing the salary, research associates believed the salary to be higher than many other— especially academic—postdoctoral positions. The annual base salary for a NIST RA is $60,000. The Sigma Xi postdoctoral survey noted that the average salary for postdocs was $38,000 (Davis, 2005). A recent survey of postdocs in the life sciences found that in 2006 the average salary for postdocs was $52,750 in industry; $50,000 in government; $40,000 in medical schools; and $38,000 in academia (Austin, 2006). They did, however, identify at least one postdoctoral position that they thought had a higher salary (Sandia), but primarily their motivation for increasing the salary tended to do with the high cost of living in the Washington metro area, where the majority of NIST RAs work. Participants mentioned that one reason for considering longer postdoctoral terms had to do with research associates’ shifting fields of research. They noted that moving into a new research area required significant time to get up to speed in that area and then at the end, there was a rush to conduct research and publish findings. Participants also believed that family leave was insufficient. They were concerned that if they took time off because of a new baby, they would simply lose time off their appointment, which was not replaced. Former research associates (and as noted in Chapter 1, these were limited to those who stayed at NIST) also held the program in high esteem. Perhaps more so than the current research 61 

associates, they viewed the program as very prestigious. The postdoc was seen as a good stepping stone, particularly for preparing for a government career. Former RAs saw several benefits to the program, including: collaboration (including the range of scientists or engineers worked with), a good stepping stone on the career pathway, and a good way to get a job at NIST. Former RAs still working at NIST were more likely to view their colleagues as treating them as professional colleagues, although this of course might be expected given that the former RAs were currently on temporary or career conditional appointments. On the negative side, some participants noted that the postdoc was not as helpful for RAs seeking academic positions in liberal arts colleges. Former RAs did note that NIST was not, in their view, set up to provide a lot of mentoring and that advising and mentoring were less common at NIST. They felt that RAs who needed a lot of mentoring or hand-holding did not do well at NIST, but that self-sufficiency and independence were characteristics of more successful (perhaps more satisfied) RAs. Neither described as a negative nor a positive, it was noted by former RAs that some RAs do not end up researching what they originally proposed in their applications. Former RAs made a few suggestions regarding possible ways to improve the program. Like the current RAs, they saw the salary as a bit low, again in reference to the high cost of living they felt in the Washington metro area. An interesting discussion took place over whether RAs could get raises. They found information on raises to be less transparent then they preferred and some former RAs believed that because of bureaucratic maneuvering they could not get raises. Again, there were different opinions based largely on which lab the participants were familiar with. Former RAs felt that the two-year time period was appropriate for the appointment. Finally, former RAs were unsure of who in NIST was the “champion” for RAs (in reference to who could help them if they experienced any problems). Advisors and division leaders offered a different view of the benefits of the program, focusing more on the benefits to NIST. They noted the program was a good way to recruit potential employees, try people out for two years, and retain good people. Some participants noted that the program was the “primary” way NIST recruited. They also see many benefits in having the RAs at NIST, including: covering a wider range of expertise, getting research done, bringing in new ideas, doing innovative research, and helping NIST connect to universities. Advisors and division leaders also offered different suggestions for improving the program. They felt that better recruiting was needed. Participants believed that personal relationships were key to recruiting. Second, they felt that there should be a NIST-wide support mechanism for RAs. They wanted to see more activities for RAs to interact and network. Third, like the former RAs, they felt that two years was an appropriate duration, although they were willing to explore a third year option for selected RAs. Advisors and division leaders felt that in some areas there were not enough applicants. One way to get more applications was to consider opening up the award to non-citizens, although they note that non-citizens face more restrictions in working at NIST. An alternative idea mentioned was to open the award up to green card holders. Overall, participants in the three expert panels felt that the program had myriad benefits to the individual RAs and to NIST. In terms of quantifying those benefits, there is less to work with. The two principal sources are an evaluation form filled out by the RAs when they complete their tenure, or term, at NIST; and an evaluation of the RAs by their research advisor. (See Appendixes 8 and 9 for the forms.) Neither form was viewed by many as required and was not filled out by most RAs or advisors. Only 253 NIST/NRC RAs partially or completely filled out the final report for a response rate of 62 

about 20 percent.41 (Additionally, in the beginning of the program very few RAs filled out the form compared with the last t10 years or so.) For RAs in the other RAPs, 6,936 partially or completely filled out the form, for a response rate of about 69 percent.42 The research advisor’s evaluation form was not filled out by a sufficient number of advisors at NIST (less than 20 did so) to include information from the form in an assessment of the RAs. Therefore, results drawing on these data should be taken cautiously. There may be nonresponse bias, particularly if RAs who had better experiences or more positive outcomes also tended to be more likely to fill out the form. Additionally, in some cases, the forms ask for historical information, such as the number of presentations RAs gave. It is possible that some respondents answered inaccurately, either unintentionally or intentionally. But this seems unlikely, as respondents are asked to provide details (e.g., title, publication) rather than counts (i.e., the number of journal articles published). In any case, there was no way in this study to independently confirm these data, as respondents were not identified by name. However, future analysis, if confidentiality concerns could be met, could undertake a CV analysis of a sample of respondents to compare with responses on the final reports. Hopefully, as the forms are completed more in the future, they will provide more data for NIST. There is no evaluation by NIST employees of the value of the program overall, save for a question on the research advisor’s evaluation form. As an open-ended question, it is difficult to quantify answers. The final report completed by RAs provides information in two areas: productivity of the research associate during the tenure and their views of the Program. Each of these areas is addressed in turn. PRODUCTIVITY DURING THE POSTDOCTORAL APPOINTMENT Productivity measures focus on both quantity and quality metrics, although only the quantitative ones are entered into the NRC’s DataRAP database. Measures focus on publications (peer-reviewed journals); books, book chapters, other publications; patents, international or domestic presentations, seminars or lectures delivered, manuscripts in preparation and manuscripts submitted; and professional awards received. Selected outcome measures are examined. Journals In the final report, RAs are asked to provide complete citations for their publications, including journal articles in peer-reviewed journals. The data which is entered into the database consists of counts of journal articles. Fifty-five percent (of 253) NIST/NRC RAs and 34 percent (of 6,936) of RAs from other RAPs provided information on journals. As Table 3-1 shows, for those who responded, NIST/NRC RAs published between 0 and 13 articles; while RAs from other RAPs published between 0 and 36 articles during their appointments. On average, NIST/NRC RAs published slightly fewer articles in peer-reviewed journals than RAs in other RAPs (2.3 to 2.7). This difference is statistically significant at the 0.05 level.43 However, a greater percentage of responding NIST/NRC RAs published at least one article. 41 Counting partial interviews as respondents. 42 Counting partial interviews as respondents. 43 Based on an unpaired t-test with unequal variances. Note the different sample sizes and that the samples are not normally distributed. 63 

TABLE 3-1 Number of Articles Published in Peer-Reviewed Journals by Research Associates, by Program No. of NIST/NRC Research Other Research Articles Associates Associates Published No. % No. % 0 22 15.9 553 23.5 1 40 29.0 506 21.5 2 23 16.7 369 15.7 3 21 15.2 296 12.6 4 11 8.0 174 7.4 5 10 7.2 125 5.3 6 4 2.9 89 3.8 7 4 2.9 72 3.1 8 2 1.4 64 2.7 9 0.0 31 1.3 10 0.0 25 1.1 11 0.0 12 0.5 12 0.0 8 0.3 13 1 0.7 7 0.3 14 0.0 3 0.1 15 0.0 3 0.1 16 0.0 2 0.1 17 0.0 3 0.1 18 0.0 1 0.0 19 0.0 2 0.1 21 0.0 2 0.1 23 0.0 2 0.1 25 0.0 1 0.0 30 0.0 1 0.0 36 0.0 1 0.0 N 138 100.0 2352 100.0 Source: National Academies, DataRAP Database, tabulations by staff. Presentations Research associates were also asked to provide details on each of their presentations (both domestic and international) made during their appointment. Fifty-six percent (of 253) NIST/NRC RAs and 35 percent (of 6,936) of RAs from other RAPs provided information on domestic presentations. NIST/NRC RAs gave between 0 and 14 domestic presentations, while RAs from other RAPs gave between 0 and 27 presentations, although more than 90 percent of all RAs gave at least one presentation. As Table 3-2 shows, for those who responded, on average, NIST/NRC RAs gave slightly more domestic presentations at scientific meetings or conferences than RAs in other RAPs did (3.5 to 3.0). This difference is statistically significant at the 0.05 level.44 And NIST/NRC RAs were more likely to give at least one domestic presentation than their counterparts at other RAPs. 44 Based on an unpaired t-test with unequal variances. Note the different sample sizes. 64 

TABLE 3-2 Number of Domestic Presentations, by Program No. of Domestic NIST/NRC Research Associates Other Research Associates Presentations Made No. % No. % 0 9 6.4 398 16.5 1 23 16.3 455 18.9 2 32 22.7 389 16.1 3 12 8.5 373 15.5 4 24 17.0 238 9.9 5 16 11.3 179 7.4 6 10 7.1 118 4.9 7 8 5.7 88 3.7 8 3 2.1 42 1.7 9 0.0 38 1.6 10 1 0.7 31 1.3 11 1 0.7 12 0.5 12 0.0 18 0.7 13 1 0.7 10 0.4 14 1 0.7 4 0.2 15 0.0 4 0.2 16 0.0 5 0.2 17 0.0 4 0.2 18 0.0 2 0.1 19 0.0 1 0.0 27 0.0 1 0.0 N 141 100.0 2410 100.0 Source: National Academies, DataRAP Database, tabulations by staff. For international presentations, 49 percent (of 253) NIST/NRC RAs and 31 percent (of 6,936) of RAs from other RAPs provided information. NIST/NRC RAs gave between 0 and 7 international presentations, while RAs from other RAPs gave between 0 and 25 presentations. Among those who responded, a greater percentage of NIST/NRC RAs gave at least one international presentation. As Table 3-2 shows, neither NIST/NRC RAs nor RAs in other RAPs give many international presentations, however, with an average of one presentation. While the NIST/NRC RAs have a slightly lower average than RAs in other RAPs (0.9 to 1), the difference is not statistically significant. 65 

TABLE 3-3 Number of International Presentations, by Program No. of International NIST/NRC Research Associates Other Research Associates Presentations No. % No. % 0 64 51.2 1190 54.9 1 35 28.0 480 22.1 2 14 11.2 247 11.4 3 6 4.8 107 4.9 4 3 2.4 52 2.4 5 1 0.8 40 1.8 6 1 0.8 17 0.8 7 1 0.8 8 0.4 8 0.0 4 0.2 9 0.0 7 0.3 10 0.0 4 0.2 11 0.0 1 0.0 12 0.0 7 0.3 13 0.0 4 0.2 25 0.0 1 0.0 N 125 100.0 2169 100.0 Source: National Academies, DataRAP Database, tabulations by staff. Patents Patents during a postdoctoral appointment were quite rare for Research Associates. Fifty- seven percent (of 253) NIST/NRC Research Associates and 28 percent (of 6,936) of research associates from other RAPs provided information on patents received. Among those who responded, as presented in Table 3-4, similar percentages of NIST/NRC Research Associates and Research Associates in other RAPs indicated that they had one or more. Numerically, both NIST/NRC Research Associates and Research Associates of other RAPs reported an average of 0.1 patents, with no statistically significant difference. TABLE 3-4 Number of Patents, by Program NIST/NRC Research Associates Other Research Associates No. of Patents No. % No. % 0 112 93.3 1733 90.6 1 5 4.2 127 6.6 2 2 1.7 31 1.6 3 1 0.8 16 0.8 4 0.0 3 0.2 5 0.0 2 0.1 N 120 100.0 1912 100.0 Source: National Academies, DataRAP Database, tabulations by staff. 66 

Awards Awards were also rare for Research Associates in any Program. Twenty-nine percent (of 253) NIST/NRC Research Associates and 10 percent (of 6,936) of research associates from other RAPs provided information on awards received. As with patents the results were quite similar, both in percentage terms and numerically. Among those who responded, as Table 3-5 illustrates, 12 percent of NIST/NRC Research Associates had received an award, compared with 16 percent of other Research Associates who had received one or more awards. Numerically, on average, NIST/NRC Research Associates received fewer of these rare awards than did Research Associates of other RAPs (0.1 to 0.2). This difference is statistically significant at the 0.05 level.45 However, quantitative measures of awards are not very helpful, as some awards are clearly more important than others. TABLE 3-5 Number of Awards, by Program NIST/NRC RAP Other RAPs No. of Awards No. % No. % 0 64 87.7 557 84.0 1 9 12.0 94 14.0 2 0.0 10 2.0 3 0.0 2 0.0 4 0.0 2 0.0 8 0.0 1 0.0 N 73 100.0 666 100.0 Source: National Academies, DataRAP Database, tabulations by staff. RESEARCH ASSOCIATES’ VIEWS OF THE PROGRAM The evaluation form asks Research Associates to rate the RAP on six dimensions on a scale of 1 to 10, with 1 representing poor and 10 meaning excellent. Not surprisingly, given the positive feedback heard in the expert panels, most respondents had a very positive view of the program. Note that the number of responses to this part of the questionnaire was much lower than other parts, possibly because it comes at the end of the questionnaire. Here the response rate is around 15 to 20 percent. As a result all of these findings need to be taken cautiously. Short Term This dimension asks Research Associates to evaluate the program in terms of the development of knowledge, skills, and research productivity. Thirty-seven percent (of 253) NIST/NRC Research Associates and 14 percent (of 6,936) of research associates from other RAPs provided information on this category. Among those who responded, as seen in Table 3-6, the average response was an 8.7, with NIST/NRC Research Associates answering 8.5 on average 45 Based on an unpaired t-test with unequal variances. Note the different sample sizes and that the samples are not normally distributed. 67 

and Research Associates of other RAPs answering 8.7. The difference between the two groups was not statistically significant. TABLE 3-6 Research Associates’ Appraisal of the Short-Term Value of the Research Associateship Program, by Program On A Scale of 1-10 (Poor-Excellent), Please Rate the Following: Short Term Value NIST/NRC RAP Other RAPs 1 1 9 2 2 3 1 4 4 6 5 1 31 6 4 21 7 8 81 8 25 217 9 28 202 10 26 391 N 94 964 Source: National Academies, DataRAP Database, tabulations by staff. Long Term This question focuses on how the award has affected the Research Associate’s career to date. This question is a bit problematic, given that it is not clear what an excellent or poor affect is, or whether all Research Associates are thinking in the same terms in answering this question. Forty-one percent (of 253) NIST/NRC Research Associates and 21 percent (of 6,936) of research associates from other RAPs provided information on this category. Among those who responded, as seen in Table 3-7, the scores were similar to those for short term value above, with an average of 8.7. NIST/NRC Research Associates again were slightly lower at 8.5, as compared with the average of 8.8 for Research Associates of other RAPs. This difference was statistically significant at the 0.05 level.46 46 Based on a t-test. Note the unequal sample sizes and that the samples are not normally distributed. 68 

TABLE 3-7 Research Associates’ Appraisal of the Long-Term Value of the Research Associateship Program, by Program On A Scale of 1-10 (Poor-Excellent), Please Rate the Following: Long Term Value NIST/NRC RAP Other RAPs 1 1 13 3 11 4 2 7 5 5 52 6 2 29 7 8 109 8 25 278 9 33 330 10 28 650 N 104 1479 Source: National Academies, DataRAP Database, tabulations by staff. Laboratory Support Research Associates’ appraisal of lab support encompasses a number of issues: equipment, funding, orientation, safety and health guidelines, etc. Forty-two percent (of 253) NIST/NRC Research Associates and 22 percent (of 6,936) of research associates from other RAPs provided information on this category. Among those who responded, as seen in Table 3-8, as before, the scores are quite high, on average at 8.5. NIST/NRC Research Associates averaged 8.4; while Research Associates of other RAPs averaged 8.5. These differences were not statistically significant. TABLE 3-8 Research Associates’ Appraisal of Laboratory Support Research Associateship Program, by Program On A Scale of 1-10 (Poor-Excellent), Please Rate the Following: Lab Support NIST/NRC RAP Other RAPs 1 1 12 2 10 3 1 15 4 1 21 5 7 63 6 3 53 7 12 151 8 19 279 9 24 277 10 38 615 N 106 1496 Source: National Academies, DataRAP Database, tabulations by staff. Advisor Support This number of responses declined significantly at this point in the questionnaire. Very few respondents answered this question, which focuses on the quality of mentoring received from the 69 

research advisor. This question ignores the possibility that the Research Associate received mentoring from someone else either at the host agency or external to it. Sixteen percent (of 253) NIST/NRC Research Associates and three percent (of 6,936) of research associates from other RAPs provided information on this category. Among those who responded, as seen in Table 3-9, and somewhat surprisingly given the comments of the former Research Associates, the average score was quite high at 8.6, and again NIST/NRC Research Associates were slightly lower than Research Associates of other RAPs (8.2 to 8.7). This difference is statistically significant at the 0.05 level, but should be taken very cautiously. However, given the very low response rate, it is likely that people who received a lot of support might be more likely to answer the question. TABLE 3-9 Research Associates’ Appraisal of the Quality of Mentoring by Their Advisor, by Program On A Scale of 1-10 (Poor-Excellent), Please Rate the Following: Advisor/Mentor Support NIST/NRC RAP Other RAPs 1 1 2 1 1 3 1 1 4 1 2 5 1 7 6 1 8 7 5 14 8 10 43 9 6 42 10 14 93 N 40 212 Sources: National Academies, DataRAP Database, tabulations by staff. LPR (administrative) Support “LPR” is an acronym for Laboratory (e.g., NIST) NRC Program Representative. This question is designed to tap the administrative support Research Associates get on-site. Even less respondents answered this question. Fourteen percent (of 253) NIST/NRC Research Associates and three percent (of 6,936) of research associates from other RAPs provided information on this category. Among those who responded, as seen in Table 3-10, and although little should be made of this finding, it was the factor with the lowest average score (7.8). The NIST/NRC Research Associates gave it a 7.7, while the Research Associates of other RAPs gave it an average score of 7.9. This difference is not statistically significant. 70 

TABLE 3-10 Research Associates’ Appraisal of Support at Their Host Agency, by Program On A Scale of 1-10 (Poor-Excellent), Please Rate the Following: LPR Support NIST/NRC RAP Other RAPs 0 1 1 4 2 4 3 2 6 4 1 5 4 18 6 6 7 8 27 8 7 51 9 8 26 10 6 67 N 35 211 Source: National Academies, DataRAP Database, tabulations by staff. NRC Support Finally, Research Associates are asked to appraise administrative support provided by the NRC. It is the case that all Research Associates have interaction with the NRC, as it administers the RAPs. Interestingly, more respondents answered this question than the questions on support, even though this question appears last on the list. Twenty-six percent (of 253) NIST/NRC Research Associates and 21 percent (of 6,936) of research associates from other RAPs provided information on this category. Among those who responded, as seen in Table 3-11, in general, there was positive feedback from Research Associates with an average score of 8.8. However, in this one instance, there was a larger difference between the NIST/NRC Research Associates and the Research Associates of other RAPs (8.0 to 8.8). This difference is statistically significant at the 0.05 level. There is no evidence as to why the NIST/NRC Research Associates would have a less favorable view of the NRC’s administration. This might be a fruitful line of future inquiry by the NRC’s Fellowships Office. 71 

TABLE 3-11 Research Associates’ Appraisal of the Support of the NRC, by Program On A Scale of 1-10 (Poor-Excellent), Please Rate the Following: NRC Support NIST/NRC RAP Other RAPs 1 2 2 1 6 3 1 6 4 1 10 5 7 45 6 2 31 7 8 118 8 19 267 9 11 326 10 17 676 N 67 1487 Source: National Academies, DataRAP Database, tabulations by staff. RESEARCH ADVISORS’ EVALUATION OF RESEARCH ASSOCIATES The current evaluation of the associate by the research advisor (see Appendix I) is limited as it is currently not filled out by many advisors, does not go into much depth concerning the value of the program, and is currently designed to tackle to separate objectives: whether the evaluation is for the renewal of their term (where the data are collected by the host agency) or for the end of the term (where the data are collected by the NRC). PRELIMINARY RESULTS Currently available data do not allow for a program evaluation of immediate outcomes of the Program. Little data are collected on Research Associates’ experiences or on research advisors’ evaluation of Research Associates. Data are also not collected on the value of the program to NIST or to the broader scientific and engineering community. Second, with the caveat that this conclusion is based on very limited data that may be biased by nonresponse, NIST/NRC RAs are as productive as Research Associates in other programs. NIST/NRC RAs, statistically were more likely to receive an award or give domestic presentations than Research Associates in other Programs. Conversely, they published fewer journal articles. However, while these differences were statistically significant, they were not substantively large. NIST/NRC RAs patent or give international presentations comparably with Research Associates in other Programs. Finally, and subject to the same caveat, RAs are quite satisfied with the program. On a scale of 1 to 10, with 10 being excellent, NIST/NRC RAs rated short-term and long-term value of the program; lab, advisor, administrative (NIST and NRC) support between 7.7 and 8.5. In half the categories NIST/NRC RAs and RAs in other programs reported statistically similar levels of satisfaction. In the other half, other RAs reported higher levels of satisfaction. 72 

RECOMMENDATIONS 1. NIST should conduct a more thorough assessment of RAs’ experiences during the postdoctoral appointment, their satisfaction with and views on the benefits of the Program, and NIST staff’s satisfaction with and views on the benefits of the Program. a. To assist in this, the NRC should redesign the final report and the Research Advisor’s evaluation form to maximize the collection of data from these instruments (see Box 3-1 and Box 3-2 for suggested questions). b. The final report and the Research Advisor’s evaluation should be made mandatory. c. Some elements of the current data collected could be subjected to further analysis. i. For example, NIST may wish to conduct further analysis on peer-reviewed journals, for example by: 1. asking whether the RA was sole or lead author, 2. examining whether RAs publish with NIST staff, and 3. examining the quality of the journals in which RAs publish, although this requires some ranking of journals. ii. NIST may wish to conduct an impact analysis of RAs’ productivity, for example by: 1. conducting a citation analysis to see how often RAs’ publications are referenced by others (note this can be accomplished using citation indexes), or 2. assessing the type or size of grants postdocs receive. iii. NIST may wish to conduct a more thorough review of their support of RAs, asking how familiar they are with NIST administrative offices, how often they turn to those offices for help, and for what reasons. d. NIST could also conduct a social network analysis of the collaboration of the RAs (or of NIST employees) to see how the RAPs facilitates new or wider collaboration among scientists and engineers. e. When data allow, NIST could consider disaggregating productivity and satisfaction measures for RAs by lab, gender, and race/ethnicity. 73 

Box 3-1 Suggested Final Report for Research Associates 1. Name 2. Contact Information a. Address b. Phone c. Email 3. Information about the postdoctoral appointment a. Agency name b. Laboratory or center name c. Division/Directorate/Department d. Postdoctoral start date e. Postdoctoral end dates f. Name of advisor g. Title of research proposal h. Summary of research i. Relationship of research conducted to research proposal i. I did what I proposed ii. I did what I proposed, and also did other research projects iii. I did some of what I proposed and also did other research iv. I did not do what I proposed 4. Was the agency where you undertook the RAP your first choice? a. If no, why not: _____________ 5. What was your primary reason for taking this postdoc? a. Additional training in Ph.D. field b. Training in an area outside of Ph.D. field c. Work with a specific person or place d. Other employment not available e. Postdoc generally expected for a career in this field f. Salary/benefits g. Location h. Only offer received i. Some other reason: ______________ 6. When you applied to the RAP, did you apply to multiple agencies? Which other ones did you apply to? 7. Around the time you applied to the RAP, did you apply to other postdoctoral positions? 8. Were you offered more than one position? 74 

Box 3-1 (continued) Suggested Final Report for Research Associates 9. If yes, why did you choose the NIST postdoc? a. Stipend was better b. Prestige of agency c. Stepping stone to career d. Salary/benefits e. Location f. Other: ____________ 10. In addition to conducting research, which of the following professional or career development activities did you engage in during your postdoctoral appointment? a. Guest lecturing at host institution b. Advising/mentoring others at host institution c. Organizing seminars or workshops d. Attending workshops, lectures, seminars in your research area e. Attending seminars on proposal writing/grant making f. None of the above g. Other: ______________ 11. To what extent did the postdoctoral appointment … (all should be on 1-5 scale) a. Increase you subject matter knowledge or expertise (great extent, somewhat, not at all) b. Improve specific research skills or techniques c. Increase contacts with colleagues in your field d. Provide opportunities to use specialized equipment e. Improve your problem-solving skills f. Enhance your career opportunities g. Help in other areas: _______________ 12. Was the postdoc experience what you expected in terms of… a. Ability to conduct your own research b. Access to research equipment, facilities, resources c. Ability to work independently d. Ability to collaborate/network with others at the agency e. Ability to collaborate/network with others outside the agency f. Ability to network with other postdocs g. Mentoring or advising h. Ability to publish i. Ability to apply for grants j. Ability to travel to conferences, professional meetings, etc. k. Administrative support from agency l. Administrative support from NRC 75 

Box 3-1 (continued) Suggested Final Report for Research Associates 13. In your opinion, what would have been the optimal duration of your postdoctoral appointment? ______ months 14. Outcomes. Please list your… a. Publications i. Books, book chapters ii. Publications in peer-reviewed journals iii. Other publications b. Patents awarded c. Presentations i. Domestic ii. International d. Seminars or lectures delivered e. Awards received f. Grants 15. Thinking about your career plans when you began the postdoc and now, has your postdoctoral experience had an effect on your career preferences? Would you say that today you are more likely, about as likely, or less likely to work in: a. Government b. Industry c. Academia d. Nonprofit e. Self employed 16. Where are you planning to go next? 76 

Box 3-1 (continued) Suggested Final Report for Research Associates 17. Post-postdoc career plans a. What are your current plans? i. Looking for another postdoc ii. Looking for employment iii. Employed iv. Not looking for employment b. If employed… i. Position title ii. Employer name iii. Employer type 1. Remain at host agency as permanent employee 2. Remain at host agency as contract/temporary employee 3. Other government position 4. Academic position 5. Industry 6. Nonprofit 7. Self employed 8. Other: ____________ 18. What were the best features of the postdoc? 19. What were the worst features of the postdoc? 20. Have you recommended the postdoc to others? 21. If you could make improvements to the program what would they be? 77 

Box 3-2 Suggested Research Advisor Evaluation 1. Name 2. Contact Information a. Address b. Phone c. Email 3. Have you ever been an advisor to a postdoc? (If yes, skip to 5) 4. If no, why not? (Continue to 10) 5. If yes, how many in the past five years? 6. Do you keep in touch with former postdocs you advised? 7. Are you currently an advisor to a postdoc? 8. If yes, how would you rate the postdoc associate to other comparable employees at your agency? a. Knowledge of field (below ave, ave, above ave, good, exceptional) b. Research technique c. Motivation/initiative d. Independent research e. Innovative thinking f. Overall scientific ability 9. Would you like the postdoc as a professional colleague at your agency? 10. Should the postdoc program be limited to U.S. citizens (if no, who else should be allowed to apply?) 11. How do you think most doctorates hear about the postdoctoral program at your agency? a. Word of mouth from fellow graduate students/doctorates b. Ph.D. thesis advisor or other professor c. University placement office d. Former or current postdoc with your agency e. Agency employee f. NRC presentation at professional meeting g. Advertisement in professional publication h. Internet i. Other: _______________ 12. What type of outreach do you think is most effective? 13. In your opinion, is 2 years the optimal duration for the postdoctoral appointment? 78 

Box 3-2 (continued) Suggested Research Advisor Evaluation 22. To what extent do you agree or disagree with the following statements? (Have them use 1-5 scale) a. Program increases postdocs’ knowledge of field b. Program allows postdocs to learn new fields c. Program allows postdocs to Become more interdisciplinary d. Program allows postdocs to Try out working at agency e. Program allows postdocs to Improve research techniques f. Program allows postdocs to practice prepare grants g. Program allows postdocs to Practice giving presentations h. Program allows postdocs to Publishing i. Program allows postdocs to Collaboration with agency employees j. Program allows postdocs to Collaboration with others outside agency k. Program makes postdocs more independent as researchers l. Program makes postdocs more innovative thinkers 23. To what extent do you agree or disagree with the following statements? a. My agency has increased collaboration due to the postdoc program b. My agency is able to cover a wider range of research topics because of the postdocs c. My agency is able to get more research done because of the postdocs d. The postdoc program increases the applicant pool for regular appointments at my agency 24. What benefits did you get from being an advisor? 25. Have you made any efforts to recruit postdocs? If yes, did you a. Give presentations at universities b. Meet with graduate students/doctorates at universities c. Meet with graduate students/doctorates at professional meetings d. Invite graduate students/doctorates to visit or give presentations at your agency e. Other: ________________ 26. Does your group make any efforts to recruit postdocs a. If yes, what? 27. Does your lab or center make any efforts to recruit postdocs? a. If yes, what? 28. What were the best features of the postdoc? 29. What were the worst features of the postdoc? 30. If you could make improvements to the program what would they be? 79