Below are the first 10 and last 10 pages of uncorrected machine-read text (when available) of this chapter, followed by the top 30 algorithmically extracted key phrases from the chapter as a whole.
Intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text on the opening pages of each chapter. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.
Do not use for reproduction, copying, pasting, or reading; exclusively for search engines.
OCR for page 106
Enhancing Philanthropy’s Support of Biomedical Scientists: Proceedings of a Workshop on Evaluation Outcomes and Impacts of the National Science Foundation’s Minority Postdoctoral Research Fellowships Program Carter Kimsey The Directorate for Biological Sciences of the National Science Foundation (NSF) supports training for graduate students and postdoctorals in basic biology. A distinction is made in the areas of research for awards funded by NSF, and National Institutes of Health (NIH) awards and duplicate submissions of proposals are not allowed unless the applicants are young investigators. Awards made by NSF must be framed in such a way that they can be categorized as basic biology. NSF targets its awards to young investigators. For many young investigators, the research pipeline begins with an NSF award, followed by grants from NIH. NSF supports the research of individual scientists, but it also supports things like observatories and oceangoing vessels. NSF is in the business of looking at the health of science across all fields and conducts evaluations in many, many different ways. Frequently, these evaluations are built into the programs or are congressionally mandated. There are four aspects of NSF that constrain the agency’s ability to conduct evaluations that are important to understanding evaluation activities of the Directorate for Biological Sciences. The first is that NSF is not a biomedical research agency. The Directorate for Biological Sciences supports a broad range of biology, ecology, physiology, and some molecular and cellular biology but does not support biomedical research. Biomedical research is the domain of the NIH. Data on its evaluation efforts are reported elsewhere in this volume. The second constraint to NSF’s ability to conduct evaluations is the
OCR for page 107
Enhancing Philanthropy’s Support of Biomedical Scientists: Proceedings of a Workshop on Evaluation necessity to obtain clearance from the Office of Management and Budget (OMB). Part of the Paperwork Reduction Act of 1995 is designed to ensure that government agencies do not burden the public with needless paperwork. Consequently, with a few exceptions, any survey sent to the public must go through a review by OMB that demonstrates that the data are not already collected and the collection effort is not burdensome. The bad part of OMB clearance is that it takes a long time, and this time must be built into any evaluation plans. The good part is that NSF has staff that is very talented at writing OMB clearance packages. Consequently, surveys can be conducted as part of evaluation activities, but they can be expensive and time-consuming. The third constraint to NSF’s ability to conduct evaluations derives from the Government Performance Results Act. GPRA requires a great deal of input from program officers who review all the annual reports they receive from grantees and synthesize highlights into nuggets. These are written up as short paragraphs and submitted into the NSF database. Each January this large number of nuggets is transmitted to the Congress as part of a large package of materials to meet GPRA reporting requirements. At the same time, GPRA reporting is tied into NSF’s strategic planning exercises. The NSF budget cycle is always three years in advance. Consequently, planning always looks three years in advance, but the GRPR reports focus on how funds were allocated during the current fiscal year. This disconnect between planning and reporting cycles hampers NSF’s ability to conduct evaluations because there is not enough time for outcomes to occur. The final constraint to NSF’s ability to conduct evaluations is the privacy act. The privacy act limits the collection of data, limits the use of Social Security numbers as an identifier, and limits access to data collected. While the privacy act does permit disclosure of personal information for program evaluation, interpretation of the act is not consistent across government agencies. Many agencies restrict the disclosure of personal data for nearly all reasons. This paper describes one of the evaluations conducted by the Directorate for Biological Sciences through a contract to SRI International to examine outcomes and impacts of the Minority Postdoctoral Research Fellowship (MPRF) program. This program was started in 1990 and is still in operation. It is a postdoctoral fellowship program in (1) biological sciences and (2) social and behavioral sciences and economics. The goal of the program is to increase the number of underrepresented minorities in leadership positions in the United States in academia, industry, and government. The number of applicants and awards in this program has been relatively small, averaging 26 applications and 13 awards each year. By 2002
OCR for page 108
Enhancing Philanthropy’s Support of Biomedical Scientists: Proceedings of a Workshop on Evaluation NSF had vital information on 155 fellows, 96 percent of the total number of all the fellowships awarded. NSF conducted a survey of the 155 fellows and received responses from 131, for an 84.5 percent response rate. While this response rate is exemplary, NSF maintains close contact with the MPRF fellows, so a high response rate was expected. The objectives of the evaluation were to (1) document the fellows’ career paths, (2) assess MPRF program’s contribution, and (3) estimate the potential pool of persons eligible for the MPRF program. NSF wanted to quantify as well as possible all that it knew about the fellows and elected to utilize a Web-based survey. Outcomes that were examined included (1) a proposal history at both NIH and NSF, (2) a publications record study, and (3) a history of education and employment. NSF wanted to know what the national pool of minority scientists looked like and, because the numbers were small, was worried about the ability to generalize. Because the survey was divided into different components, everyone who either formerly had had a fellowship or currently had one was surveyed. In addition to the career development process, NSF wanted to hear from fellows about the application process and their perceptions of dealing with NSF so that the agency could make any improvements in the application process. At NSF about 6,000 postdocs are funded each year on research grants. Fewer than 200 are supported on fellowships. So, these fellows are a very select group. One of the survey’s goals was to provide insight into what NSF should be doing for the postdoctorals who are on fellowships. As a first step, the agency wanted to better understand the fellowship process to learn whether the things designed to impact positively on the fellows are really working. The survey was well designed, and a lot of good comments were received. This is not unexpected when you have given someone funding for postgraduate training. Fellows are probably not going to say it wasn’t a good experience. But when NSF analyzed the individual comments, there was a lot to learn. The agency was looking for constructive criticism and, while open to some negativity, most comments were pretty positive. NSF found that the program was meeting its goals. Fellows’ current institutional affiliations and their current positions were analyzed. In addition to survey responses, fellows’ written comments in the survey were examined. Also, fellows’ subsequent success with NIH and NSF awards was documented. It is important to remember that the MPRF program has two parts: a biological sciences part and a social and behavioral sciences and economics part. Most of the fellows are from the biological sciences (BIO fellows). There are fewer social and behavioral science and economics fellows (SBE fellows). NSF received surveys from 98 former BIO fellows and 18 former
OCR for page 109
Enhancing Philanthropy’s Support of Biomedical Scientists: Proceedings of a Workshop on Evaluation SBE fellows. The number of responses was very small for the SBE fellows, but they were all employed in good positions (see Table 1). Moreover, the fellows stated that the quality and the direction of their research were aided by the fact that they had their own research grants. This meant that instead of working on somebody else’s grant, they were somewhat in control of their research agenda. In the long term, NSF believes that this independence may prove to be the most important aspect of the award. The fellows mentioned that they increased their skills, confidence, knowledge, and contacts as a result of their fellowships. Most fellows said that the MPRF (1) helped their career, (2) helped develop their professional expertise, and (3) helped improve the quality and direction of their research. They were proud to have been MPRF fellows and would recommend the program to their colleagues and students. In analyzing the written comments from fellows, five themes emerged: Fellows could pursue their own research interests. They became much more qualified for research positions. The worked with highly ranked researchers. The program opened doors to professional networks. It allowed them to lever fellowship prestige into starter grants. The NIH and NSF databases were examined to document fellows’ success at obtaining grants. The fellows were very successful in their efforts to obtain NSF grants. Many of the eligible BIO fellows submitted proposals, and 37 were successful in getting one or more NSF awards. These BIO fellows submitted 150 proposals and had a funding success rate of 48 percent. An additional four (out of nine) fellows received CAREER awards. Nearly all eligible SBE fellows submitted proposals to NSF, and they had a 55 percent success rate. These included one CAREER award and one ADVANCE award. TABLE 1 Current Employment of Former BIO and SBE Fellows Current Employment BIO Fellows SBE Fellows Major Research University 35 12 Other University or Medical School 37 4 Private Sector 19 2 Federal Agency 7 0 Total 98 18
OCR for page 110
Enhancing Philanthropy’s Support of Biomedical Scientists: Proceedings of a Workshop on Evaluation NSF was concerned that the number of applications has not been increasing. It was good to see that fellows are recommending the MPRF program to their students and colleagues. However, the national pool of minority scientists is so low that not much of an increase in applications is expected. The number of minority doctorates in the biological sciences grew slowly from 139 in 1989 to 320 in 2000. During this interval 2,822 biological science doctorates were awarded to minorities (an average of 235 per year); 1,898 of them sought postdoctoral support; and the MPRF program supported 10.6 percent of these postdoctorals. The numbers are similar for SBE doctorates. Between 1989 and 2000, the number of doctorates in social and behavioral sciences and economics awarded to minorities increased from 264 to 514. During this interval 4,703 doctorates were awarded to minorities in SBE fields (an average of 392 per year); 921 sought postdoctoral support, and 4.7 percent of them were supported by the MPRF program. The success rates for fellows in the MPRF program were high. The problem, of course, is that there was no comparison group. It was difficult enough getting data on the study group, since the numbers were so small. But the success rates were good, and NSF considers the MPRF to be an exemplary program.
Representative terms from entire chapter: