Approaches for Evaluating the NRC Resident Research Associateship Program at NIST (2007)

Below is the uncorrected machine-read text of this chapter, intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text of each book. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.

Executive Summary The NRC Resident Research Associateship Program (RAP) of the National Institute of Standards and Technology (NIST), hereafter “NIST/NRC RAP,” was started in 1954. The program provides two-year temporary appointments at NIST for outstanding scientists and engineers chosen through a national competition. These appointments are designed to provide an opportunity for some of the nation’s best scientists, mathematicians, and engineers to engage in state-of-the-art research in association with senior research specialists of NIST’s staff and to make use of the well-regarded and often unique research facilities at NIST. The RAP is perceived to provide multiple benefits to the postdoctoral research associates, to NIST, and to the scientific and engineering community at large. NIST approached The National Academies with a desire to see what sort of evaluation could be undertaken given available data. In addition, NIST was interested in recommendations for future data collection where data were found to be currently lacking, and for more in-depth evaluation strategies that could be done in the future. Based on this request from NIST, an ad hoc study committee—the Committee on Approaches for the Evaluation of the NIST/NRC Postdoctoral Research Associateships Program—was appointed by the National Research Council to conduct a study. The committee’s specific charge is presented below: “The Academic Competitiveness Council, in furtherance of the Administration’s American Competitiveness Initiative, seeks to ensure that the nation invests wisely and effectively in educational programs to meet its science and technology goals. The ACC, therefore, requires evaluations of important STEM1 education programs, including the NIST/NRC Postdoctoral Research Associateship Program. An ad hoc committee under the auspices of the Board on Higher Education and Workforce will describe the pool of applicants for and recipients of the NIST/NRC Postdoctoral Research Associateship Program and carry out a descriptive analysis of career outcomes of NIST postdoctoral scholars based on available information. As possible given available data, the committee will also describe how the program addresses agency goals. The committee will also outline an approach to evaluating the program relative to national S&E goals, NIST agency goals, and the value of the program to participants, which could be undertaken in a future study.” To meet its charge, the committee focused on three objectives: (1) to describe characteristics of NIST applicants compared to the general pool of new science and engineering doctorates; (2) to describe the experiences of Research Associates at NIST compared to other Research Associates in other programs; and (3) to offer suggestions for conducting a more in-depth assessment of the careers of Research Associates at NIST, with a particular focus on quantifying the benefits of the appointment to the recipients as well as to NIST—during and after the appointment period. The committee was guided by two principal questions: (1) Is NIST attracting the “best and the brightest” to the Research Associateship Program? 1 STEM is science, technology, engineering, and mathematics. 1 

(2) What is the impact of the Program on the Research Associates, NIST, and relevant scientific fields in general?2 In addition, the committee determined to offer recommendations where appropriate regarding data collection on applicants to the Program; on the experiences of current and former NIST Research Associates; and on the views of Research Associates and NIST employees towards the value of the Program to Research Associates, to NIST, and to science and engineering broadly. Multiple sources of information were identified and used in preparing this report. The primary source of information was data collected by the NRC’s Fellowships Office, including: application data, data from final reports prepared by the Research Associates, and a directory of past Research Associates. In addition, the committee collected original data from three expert panels. In general, however, the data were inadequate to draw definitive conclusions. This report is divided into five chapters. Chapter One describes the program and the approach and scope of the study. Chapter Two examines applicants to the NIST/NRC RAP and compares them to applicants to other Research Associateship Programs. It also examines applications and awards disaggregated across several dimensions, such as gender or doctoral- granting institution. Chapter Three examines the experiences of NIST/NRC Research Associates and Research Associates at other federal agencies, as well as Research Associates’ views on the value of the program they participated in. Chapter Four examines the careers of former Research Associates. Chapter Five presents an overall summary of preliminary results and recommendations. PRELIMINARY RESULTS BASED ON AVAILABLE DATA 1. Outreach efforts produce more qualified applicants than NIST has slots to fill for Research Associates; and the pool of applicants includes many from top research institutions and is increasingly diverse. Overall, 22 percent of applicants to NIST were awarded an appointment—a lower awards ratio than for other RAPs overall. The award ratios for NIST and other RAP applicants vary by gender, race, and field. Across all fields, the proportion of women and underrepresented minorities in the applicant pool and as awardees has grown over time, however less so than the proportion among Ph.D.s and those intending to be postdoctorates. Personal communication is the primary way that NIST/NRC Research Associates heard about the program. 2. NIST/NRC Research Associates appear to be about as productive as Research Associates in other Programs. On average, NIST/NRC Research Associates publish about two articles, give about four presentations, but rarely receive a patent or award during their appointments. They are more likely to give a domestic presentation or win an award, less likely to publish journal articles, and as likely to receive a patent or give an international presentation. Productivity data, though, are derived from a survey with a low response rate and possible nonresponse bias. 3. Research Associates are quite satisfied with the program. For those Research Associates who provided information on their final reports, on a scale of 1 to 10, with 10 being excellent, NIST/NRC Research Associates rated short-term and long-term value of the program; lab, advisor, administrative (NIST and NRC) support between 7.7 and 8.5. 2 Given the limited data available and the charge to the committee, the committee could not provide a full assessment of the impact of the program in this report. 2 

In half the categories NIST/NRC Research Associates and Research Associates in other programs reported statistically similar levels of satisfaction. In the other half, other Research Associates reported higher levels of satisfaction. Satisfaction data, though, are derived from a survey with a low response rate and possible nonresponse bias. 4. Research Associates contribute to the pool of qualified applicants to permanent positions at NIST. Among Research Associates completing their awards, 45 percent of those at NIST reported that their immediate post-tenure position was as a permanent, temporary, or contract employee—a higher percentage than Research Associates at other federal agencies. The employment of nonrespondents remains unknown. A survey of former Research Associates found that a higher percentage of former NIST/NRC Research Associates stayed at NIST than Research Associates in other programs stayed at their host agency (37.6 to 28.1 percent), among those former Research Associates who could be located. CONCLUSION 1. Currently available data do not allow for a full program evaluation. Currently, the most thorough data are collected on applicants. Little data are collected on Research Associates’ experiences; research advisors’ evaluation of Research Associates; career outcomes of former Research Associates; and the value of the program to NIST or to the broader scientific and engineering community. RECOMMENDATIONS 1. NIST should conduct a more thorough evaluation of the NIST/NRC Research Associateship Program. a. As a first step, NIST and the NRC should review specific goals of the program. b. The evaluation should include the following components: an assessment of outreach to potential applicants; an assessment of individuals who decline to accept a Research Associate position; an assessment of the benefits of the program on the Research Associates after they complete their appointments; an assessment of the benefits to NIST of hosting Research Associates; and an assessment of benefits of the program to the broader scientific and engineering community. 2. NIST should conduct an evaluation of outreach efforts. a. Additional analysis could be undertaken on how applicants hear about the program (e.g., focusing on the “Other” category). b. Additional data could be collected from NIST personnel and former or current NIST Research Associates. Such data could be used to answer such questions as how NIST personnel and Research Associates interact with potential applicants and which mechanisms seem to work best. c. A second step to facilitate an evaluation of outreach efforts is to identify metrics for quantifying value obtained from different outreach strategies. d. Examine individual outreach strategies for return on investment. 3 

e. Finally, consider whether there might be other outreach strategies that are being underused currently, and which might have potential value, such as direct mail to deans, department heads and other university administrators. 3. NIST should conduct an evaluation of individuals who decline offers of research associateships. This could be done as a telephone interview or via a survey. Two basic questions should be asked of those who are awarded but decline: (1) why are you declining, and (2) what are you planning to do instead? 4. The NRC should amend the application form. The list of fields should be reduced, in particular by collapsing very similar labels and by removing labels that are for multiple fields. 5. The NRC should update the DataRAP database to replace organizational names (e.g., institutes or labs) that no longer exist at NIST with current equivalents. 6. NIST should conduct a more thorough assessment of Research Associates’ experiences during the postdoctoral appointment, their satisfaction with and views on the benefits of the program, and NIST staff’s satisfaction with and views on the benefits of the program. a. To assist in this, the NRC should redesign the final report and the Research Advisor’s evaluation form to maximize the collection of data from these instruments. b. The final report and the Research Advisor’s evaluation should be made mandatory. c. Some elements of the current data collected could be subjected to further analysis. i. NIST may wish to conduct further analysis on peer-reviewed journals. ii. NIST may wish to conduct an impact analysis of Research Associates’ productivity. iii. NIST may wish to conduct a more thorough review of their support of Research Associates, asking how familiar they are with NIST administrative offices, how often they turn to those offices for help, and for what reasons. d. NIST could also conduct a social network analysis of the collaboration of the Research Associates (or of NIST employees) to see how the Research Associateship Program facilitates new or wider collaboration among scientists and engineers. e. When data allow, NIST could consider disaggregating productivity and satisfaction measures for Research Associates by lab, gender, and race/ethnicity. 7. NIST should conduct a broad evaluation of the careers of former Research Associates to evaluate the impact of the Program on Research Associates’ careers, NIST, and the broader science and engineering community. The best approach for doing this is a survey, which would compare the career outcomes of NIST/NRC Research Associates to similar postdocs. The survey would be directed towards these former Research Associates and a suitable control group. Ideally, two possible comparisons could be made. First, one could construct a peer group. This would consist of a matched or stratified sample of individuals who had postdocs similar to the one at NIST for the comparison group. Although not ideal, one solution would be to take a stratified sample of former Research Associates from the Fellowships Office’s Directory. This is a census of former Research Associates; but as noted earlier in the report, many of these 4 

individuals could not be found or failed to respond to an earlier survey designed to collect information on their current employment. A second comparison group would consist of similar doctorates. A roster could be assembled by tapping the group of applicants to RAPs, who did not receive an award. These individuals will likely exhibit a diversity of career paths, including some who took postdocs (in academia or industry) and others who went straight into employment. 5