National Academies Press: OpenBook

Enhancing Philanthropy's Support of Biomedical Scientists: Proceedings of a Workshop on Evaluation (2006)

Chapter: Outcomes and Impacts of the National Science Foundation’s Minority Postdoctoral Research Fellowships Program--Carter Kimsey

« Previous: Program Assessment in HHMI-Sponsored Medical Student Research Training Programs--Min K. Lee, Barbara Ziff, and William R. Galey
Suggested Citation:"Outcomes and Impacts of the National Science Foundation’s Minority Postdoctoral Research Fellowships Program--Carter Kimsey." National Research Council. 2006. Enhancing Philanthropy's Support of Biomedical Scientists: Proceedings of a Workshop on Evaluation. Washington, DC: The National Academies Press. doi: 10.17226/11646.
×

Outcomes and Impacts of the National Science Foundation’s Minority Postdoctoral Research Fellowships Program

Carter Kimsey


The Directorate for Biological Sciences of the National Science Foundation (NSF) supports training for graduate students and postdoctorals in basic biology. A distinction is made in the areas of research for awards funded by NSF, and National Institutes of Health (NIH) awards and duplicate submissions of proposals are not allowed unless the applicants are young investigators. Awards made by NSF must be framed in such a way that they can be categorized as basic biology. NSF targets its awards to young investigators. For many young investigators, the research pipeline begins with an NSF award, followed by grants from NIH.

NSF supports the research of individual scientists, but it also supports things like observatories and oceangoing vessels. NSF is in the business of looking at the health of science across all fields and conducts evaluations in many, many different ways. Frequently, these evaluations are built into the programs or are congressionally mandated.

There are four aspects of NSF that constrain the agency’s ability to conduct evaluations that are important to understanding evaluation activities of the Directorate for Biological Sciences. The first is that NSF is not a biomedical research agency. The Directorate for Biological Sciences supports a broad range of biology, ecology, physiology, and some molecular and cellular biology but does not support biomedical research. Biomedical research is the domain of the NIH. Data on its evaluation efforts are reported elsewhere in this volume.

The second constraint to NSF’s ability to conduct evaluations is the

Suggested Citation:"Outcomes and Impacts of the National Science Foundation’s Minority Postdoctoral Research Fellowships Program--Carter Kimsey." National Research Council. 2006. Enhancing Philanthropy's Support of Biomedical Scientists: Proceedings of a Workshop on Evaluation. Washington, DC: The National Academies Press. doi: 10.17226/11646.
×

necessity to obtain clearance from the Office of Management and Budget (OMB). Part of the Paperwork Reduction Act of 1995 is designed to ensure that government agencies do not burden the public with needless paperwork. Consequently, with a few exceptions, any survey sent to the public must go through a review by OMB that demonstrates that the data are not already collected and the collection effort is not burdensome. The bad part of OMB clearance is that it takes a long time, and this time must be built into any evaluation plans. The good part is that NSF has staff that is very talented at writing OMB clearance packages. Consequently, surveys can be conducted as part of evaluation activities, but they can be expensive and time-consuming.

The third constraint to NSF’s ability to conduct evaluations derives from the Government Performance Results Act. GPRA requires a great deal of input from program officers who review all the annual reports they receive from grantees and synthesize highlights into nuggets. These are written up as short paragraphs and submitted into the NSF database. Each January this large number of nuggets is transmitted to the Congress as part of a large package of materials to meet GPRA reporting requirements. At the same time, GPRA reporting is tied into NSF’s strategic planning exercises. The NSF budget cycle is always three years in advance. Consequently, planning always looks three years in advance, but the GRPR reports focus on how funds were allocated during the current fiscal year. This disconnect between planning and reporting cycles hampers NSF’s ability to conduct evaluations because there is not enough time for outcomes to occur.

The final constraint to NSF’s ability to conduct evaluations is the privacy act. The privacy act limits the collection of data, limits the use of Social Security numbers as an identifier, and limits access to data collected. While the privacy act does permit disclosure of personal information for program evaluation, interpretation of the act is not consistent across government agencies. Many agencies restrict the disclosure of personal data for nearly all reasons.

This paper describes one of the evaluations conducted by the Directorate for Biological Sciences through a contract to SRI International to examine outcomes and impacts of the Minority Postdoctoral Research Fellowship (MPRF) program. This program was started in 1990 and is still in operation. It is a postdoctoral fellowship program in (1) biological sciences and (2) social and behavioral sciences and economics. The goal of the program is to increase the number of underrepresented minorities in leadership positions in the United States in academia, industry, and government.

The number of applicants and awards in this program has been relatively small, averaging 26 applications and 13 awards each year. By 2002

Suggested Citation:"Outcomes and Impacts of the National Science Foundation’s Minority Postdoctoral Research Fellowships Program--Carter Kimsey." National Research Council. 2006. Enhancing Philanthropy's Support of Biomedical Scientists: Proceedings of a Workshop on Evaluation. Washington, DC: The National Academies Press. doi: 10.17226/11646.
×

NSF had vital information on 155 fellows, 96 percent of the total number of all the fellowships awarded. NSF conducted a survey of the 155 fellows and received responses from 131, for an 84.5 percent response rate. While this response rate is exemplary, NSF maintains close contact with the MPRF fellows, so a high response rate was expected. The objectives of the evaluation were to (1) document the fellows’ career paths, (2) assess MPRF program’s contribution, and (3) estimate the potential pool of persons eligible for the MPRF program.

NSF wanted to quantify as well as possible all that it knew about the fellows and elected to utilize a Web-based survey. Outcomes that were examined included (1) a proposal history at both NIH and NSF, (2) a publications record study, and (3) a history of education and employment. NSF wanted to know what the national pool of minority scientists looked like and, because the numbers were small, was worried about the ability to generalize.

Because the survey was divided into different components, everyone who either formerly had had a fellowship or currently had one was surveyed. In addition to the career development process, NSF wanted to hear from fellows about the application process and their perceptions of dealing with NSF so that the agency could make any improvements in the application process. At NSF about 6,000 postdocs are funded each year on research grants. Fewer than 200 are supported on fellowships. So, these fellows are a very select group. One of the survey’s goals was to provide insight into what NSF should be doing for the postdoctorals who are on fellowships. As a first step, the agency wanted to better understand the fellowship process to learn whether the things designed to impact positively on the fellows are really working.

The survey was well designed, and a lot of good comments were received. This is not unexpected when you have given someone funding for postgraduate training. Fellows are probably not going to say it wasn’t a good experience. But when NSF analyzed the individual comments, there was a lot to learn. The agency was looking for constructive criticism and, while open to some negativity, most comments were pretty positive.

NSF found that the program was meeting its goals. Fellows’ current institutional affiliations and their current positions were analyzed. In addition to survey responses, fellows’ written comments in the survey were examined. Also, fellows’ subsequent success with NIH and NSF awards was documented.

It is important to remember that the MPRF program has two parts: a biological sciences part and a social and behavioral sciences and economics part. Most of the fellows are from the biological sciences (BIO fellows). There are fewer social and behavioral science and economics fellows (SBE fellows). NSF received surveys from 98 former BIO fellows and 18 former

Suggested Citation:"Outcomes and Impacts of the National Science Foundation’s Minority Postdoctoral Research Fellowships Program--Carter Kimsey." National Research Council. 2006. Enhancing Philanthropy's Support of Biomedical Scientists: Proceedings of a Workshop on Evaluation. Washington, DC: The National Academies Press. doi: 10.17226/11646.
×

SBE fellows. The number of responses was very small for the SBE fellows, but they were all employed in good positions (see Table 1).

Moreover, the fellows stated that the quality and the direction of their research were aided by the fact that they had their own research grants. This meant that instead of working on somebody else’s grant, they were somewhat in control of their research agenda. In the long term, NSF believes that this independence may prove to be the most important aspect of the award. The fellows mentioned that they increased their skills, confidence, knowledge, and contacts as a result of their fellowships. Most fellows said that the MPRF (1) helped their career, (2) helped develop their professional expertise, and (3) helped improve the quality and direction of their research. They were proud to have been MPRF fellows and would recommend the program to their colleagues and students.

In analyzing the written comments from fellows, five themes emerged:

  • Fellows could pursue their own research interests.

  • They became much more qualified for research positions.

  • The worked with highly ranked researchers.

  • The program opened doors to professional networks.

  • It allowed them to lever fellowship prestige into starter grants.

The NIH and NSF databases were examined to document fellows’ success at obtaining grants. The fellows were very successful in their efforts to obtain NSF grants. Many of the eligible BIO fellows submitted proposals, and 37 were successful in getting one or more NSF awards. These BIO fellows submitted 150 proposals and had a funding success rate of 48 percent. An additional four (out of nine) fellows received CAREER awards. Nearly all eligible SBE fellows submitted proposals to NSF, and they had a 55 percent success rate. These included one CAREER award and one ADVANCE award.

TABLE 1 Current Employment of Former BIO and SBE Fellows

Current Employment

BIO Fellows

SBE Fellows

Major Research University

35

12

Other University or Medical School

37

4

Private Sector

19

2

Federal Agency

7

0

Total

98

18

Suggested Citation:"Outcomes and Impacts of the National Science Foundation’s Minority Postdoctoral Research Fellowships Program--Carter Kimsey." National Research Council. 2006. Enhancing Philanthropy's Support of Biomedical Scientists: Proceedings of a Workshop on Evaluation. Washington, DC: The National Academies Press. doi: 10.17226/11646.
×

NSF was concerned that the number of applications has not been increasing. It was good to see that fellows are recommending the MPRF program to their students and colleagues. However, the national pool of minority scientists is so low that not much of an increase in applications is expected. The number of minority doctorates in the biological sciences grew slowly from 139 in 1989 to 320 in 2000. During this interval 2,822 biological science doctorates were awarded to minorities (an average of 235 per year); 1,898 of them sought postdoctoral support; and the MPRF program supported 10.6 percent of these postdoctorals. The numbers are similar for SBE doctorates. Between 1989 and 2000, the number of doctorates in social and behavioral sciences and economics awarded to minorities increased from 264 to 514. During this interval 4,703 doctorates were awarded to minorities in SBE fields (an average of 392 per year); 921 sought postdoctoral support, and 4.7 percent of them were supported by the MPRF program.

The success rates for fellows in the MPRF program were high. The problem, of course, is that there was no comparison group. It was difficult enough getting data on the study group, since the numbers were so small. But the success rates were good, and NSF considers the MPRF to be an exemplary program.

Suggested Citation:"Outcomes and Impacts of the National Science Foundation’s Minority Postdoctoral Research Fellowships Program--Carter Kimsey." National Research Council. 2006. Enhancing Philanthropy's Support of Biomedical Scientists: Proceedings of a Workshop on Evaluation. Washington, DC: The National Academies Press. doi: 10.17226/11646.
×
Page 106
Suggested Citation:"Outcomes and Impacts of the National Science Foundation’s Minority Postdoctoral Research Fellowships Program--Carter Kimsey." National Research Council. 2006. Enhancing Philanthropy's Support of Biomedical Scientists: Proceedings of a Workshop on Evaluation. Washington, DC: The National Academies Press. doi: 10.17226/11646.
×
Page 107
Suggested Citation:"Outcomes and Impacts of the National Science Foundation’s Minority Postdoctoral Research Fellowships Program--Carter Kimsey." National Research Council. 2006. Enhancing Philanthropy's Support of Biomedical Scientists: Proceedings of a Workshop on Evaluation. Washington, DC: The National Academies Press. doi: 10.17226/11646.
×
Page 108
Suggested Citation:"Outcomes and Impacts of the National Science Foundation’s Minority Postdoctoral Research Fellowships Program--Carter Kimsey." National Research Council. 2006. Enhancing Philanthropy's Support of Biomedical Scientists: Proceedings of a Workshop on Evaluation. Washington, DC: The National Academies Press. doi: 10.17226/11646.
×
Page 109
Suggested Citation:"Outcomes and Impacts of the National Science Foundation’s Minority Postdoctoral Research Fellowships Program--Carter Kimsey." National Research Council. 2006. Enhancing Philanthropy's Support of Biomedical Scientists: Proceedings of a Workshop on Evaluation. Washington, DC: The National Academies Press. doi: 10.17226/11646.
×
Page 110
Next: Evaluation of Research Training and Career Development Programs at NIH: Current Capabilities and Continuing Needs--Charles R. Sherman »
Enhancing Philanthropy's Support of Biomedical Scientists: Proceedings of a Workshop on Evaluation Get This Book
×
Buy Paperback | $48.00 Buy Ebook | $38.99
MyNAP members save 10% online.
Login or Register to save!
Download Free PDF

During an interval of 15 years, the Lucille P. Markey Charitable Trust spent over $500 million on four programs in the basic biomedical sciences that support the education and research of graduate students, postdoctoral fellows, junior faculty, and senior researchers. The Markey Trust asked the NRC to evaluate these programs with two questions in mind: “Were these funds well spent?” and “What can others in the biomedical and philanthropic communities learn from the programs of the Markey Trust, both as an approach to funding biomedical research and as a model of philanthropy?” One of five resulting reports, this volume contains the proceedings of a workshop held in June 2005 to investigate methods used to evaluate funding of the biomedical scientists by philanthropic and public funders. In addition to the Markey Trust, representatives from the Howard Hughes Medical Institute, the American Heart Association, the Doris Duke Charitable Foundation, the National Institutes of Health, and six other funders of biomedical scientists presented information on evaluation methodologies and outcomes.

  1. ×

    Welcome to OpenBook!

    You're looking at OpenBook, NAP.edu's online reading room since 1999. Based on feedback from you, our users, we've made some improvements that make it easier than ever to read thousands of publications on our website.

    Do you want to take a quick tour of the OpenBook's features?

    No Thanks Take a Tour »
  2. ×

    Show this book's table of contents, where you can jump to any chapter by name.

    « Back Next »
  3. ×

    ...or use these buttons to go back to the previous chapter or skip to the next one.

    « Back Next »
  4. ×

    Jump up to the previous page or down to the next one. Also, you can type in a page number and press Enter to go directly to that page in the book.

    « Back Next »
  5. ×

    Switch between the Original Pages, where you can read the report as it appeared in print, and Text Pages for the web version, where you can highlight and search the text.

    « Back Next »
  6. ×

    To search the entire text of this book, type in your search term here and press Enter.

    « Back Next »
  7. ×

    Share a link to this book page on your preferred social network or via email.

    « Back Next »
  8. ×

    View our suggested citation for this chapter.

    « Back Next »
  9. ×

    Ready to take your reading offline? Click here to buy this book in print or download it as a free PDF, if available.

    « Back Next »
Stay Connected!