National Academies Press: OpenBook

Enhancing Philanthropy's Support of Biomedical Scientists: Proceedings of a Workshop on Evaluation (2006)

Chapter: Program Assessment in HHMI-Sponsored Medical Student Research Training Programs--Min K. Lee, Barbara Ziff, and William R. Galey

« Previous: Exploring Program Effects on Life Sciences Faculty Diversity: Assessing the Ford Foundation Postdoctoral Fellowships for Minorities--Connie L. McNeely and Christine O’Brien
Suggested Citation:"Program Assessment in HHMI-Sponsored Medical Student Research Training Programs--Min K. Lee, Barbara Ziff, and William R. Galey." National Research Council. 2006. Enhancing Philanthropy's Support of Biomedical Scientists: Proceedings of a Workshop on Evaluation. Washington, DC: The National Academies Press. doi: 10.17226/11646.
×

Program Assessment in HHMI-Sponsored Medical Student Research Training Programs

Min K. Lee, Barbara Ziff, and William R. Galey


For over 20 years, the Howard Hughes Medical Institute has been involved in research training for medical students. The HHMI-NIH Medical Research Scholars Program was established in 1985 as a partnership with the National Institutes of Health (NIH) to provide an opportunity for students from U.S. medical and dental schools to conduct a full year of research at the NIH laboratories in Bethesda, Maryland. Currently, 42 students are provided salary, travel and educational support, and research facilities. The students are housed in a facility on the NIH campus that once was a convent for cloistered nuns; therefore the program has become known as the “Cloister Program.” Its sister program, the Research Training Fellowships for Medical Students program, provides support for U.S. medical and dental students to conduct a year of research at their home institution or nearly any other academic institution in the United States, except the NIH. The primary goal of both programs is to enhance the number and quality of physician-scientists in this country, but HHMI is confident that the year of research experience will also enhance the careers of the alumni who choose purely clinical careers.

As part of HHMI’s ongoing program evaluation, both anonymous and nonanonymous feedback is gathered from alumni regarding their perceptions of the program at the beginning, during, and end of their research training experiences. To determine the effectiveness of this program, the career development of the former program participants is also followed. Two major challenges hampering this effort are (1) the long period between program participation and career initiation and (2) the

Suggested Citation:"Program Assessment in HHMI-Sponsored Medical Student Research Training Programs--Min K. Lee, Barbara Ziff, and William R. Galey." National Research Council. 2006. Enhancing Philanthropy's Support of Biomedical Scientists: Proceedings of a Workshop on Evaluation. Washington, DC: The National Academies Press. doi: 10.17226/11646.
×

natural “scattering” of participants after their involvement in our programs as they pursue further training. In addition, HHMI depends on its alumni to voluntarily provide career information.

This paper describes the information collected about the alumni, the method and verification of the collected information, and two approaches used to assess HHMI programs.

DATA COLLECTED ON HHMI ALUMNI

The primary information collected from HHMI awardees, either at the time of their award and in subsequent years, is:

  • name (and any name changes thereafter) and social security number;

  • personal and professional addresses, e-mail, and telephone numbers;

  • permanent or parental addresses, telephone, and e-mail contact information;

  • medical school information and dates of matriculation and graduation;

  • location and dates of residency and other training information;

  • professional appointments;

  • publications and awards;

  • research funding (NIH and other sources); and

  • curricula vitae.

MEANS OF DATA COLLECTION

This information is collected from a number of sources. The first is the application materials, where permanent and current name and address information as well as medical school entrance information is obtained. A significant amount of immediately useful information about the program is gained from end-of-year interviews and an anonymous online exit survey. While most of this information helps in evaluating and modifying program elements, it provides limited information on awardee career outcomes.

Subsequent to the end of the participants’ involvement in the program, HHMI begins to follow the career development and successes of each individual. HHMI tried to follow each person through an annual Web-based update survey that asks the individual to add or modify only information that is new or changed. Unfortunately, this approach has had only limited success, as a significant number of alumni are either unmotivated or cannot find the time to complete the survey on an annual basis.

Suggested Citation:"Program Assessment in HHMI-Sponsored Medical Student Research Training Programs--Min K. Lee, Barbara Ziff, and William R. Galey." National Research Council. 2006. Enhancing Philanthropy's Support of Biomedical Scientists: Proceedings of a Workshop on Evaluation. Washington, DC: The National Academies Press. doi: 10.17226/11646.
×

To provide alumni with an incentive to supply updated information, HHMI has recently begun to offer them its popular program alumni directory upon completion of the online survey.

A surprising amount of information on our alumni is collected from informal interactions with them. These include national scientific meetings, when they return to visit the program facilities, and during staff recruitment trips. HHMI often initiates the contact, but occasionally alumni will spontaneously contact program staff. Most recently, a Regional HHMI Trainee Alumni Network was established in which alumni from all HHMI graduate and medical research trainee programs are invited to attend and meet each other at regional gatherings. HHMI attempts to have program staff attend these events, which often include a combination of scientific presentations and career development and networking opportunities for alumni. These meetings have been very informative and HHMI believes they will also increase participation in the annual online updates. Finally, it has been found useful to congregate alumni members into focus groups to solicit their input and evaluation on specific issues. Similar to the involvement of individuals in the alumni regional programs, the focus groups also seem to renew interest in updating individual information.

Occasionally, contact is lost with some alumni. When this happens, HHMI tries to relocate them using online search tools such as Google, Web sites of state medical boards of licensure, medical school faculty directories, hospital staff directories, and personal and practice Web pages. On a few occasions commercial locator services have been used. In short, numerous formal and informal tools have been used to determine the career outcomes of HHMI alumni.

A TALE OF TWO ASSESSMENT APPROACHES

In recent years, HHMI has found it appropriate to evaluate (1) whether the two programs have been successful in increasing the likelihood that alumni will become physician-scientists, (2) the level of success attained by participants of each of the two programs, and (3) if the outcomes of the two programs are different. This was done to help the institute decide what, if any, changes should be made to either or both programs to increase their effectiveness and to help institutional leadership decide whether the programs were achieving their purpose and should continue to be supported. Further, the institute wanted to inform the research, medical, and medical education communities as to the success or failure of the general paradigm these programs represent of “year-out research training programs” as a model for training much-needed physician-scientists.

Suggested Citation:"Program Assessment in HHMI-Sponsored Medical Student Research Training Programs--Min K. Lee, Barbara Ziff, and William R. Galey." National Research Council. 2006. Enhancing Philanthropy's Support of Biomedical Scientists: Proceedings of a Workshop on Evaluation. Washington, DC: The National Academies Press. doi: 10.17226/11646.
×

Approach #1: One approach was to enlist the aid of an independent organization to evaluate the program through the level of success enjoyed by alumni representing a segment of graduation years (1987–1995 and 1991–1995) of each HHMI program (1). These individuals were studied because it was thought that enough time had passed to be able to determine if their early professional careers had been influenced by their program. The approach was to compare the awardees of the two HHMI-sponsored programs with Medical Science Training Program (MSTP) and non-MSTP M.D.-Ph.D. graduates and general M.D. graduates. Assessments were made employing logistic regression analyses to control for academic and demographic variables (such as MCAT (Medical College Admission Test) scores and medical school rank) that could influence the selection to the various programs.

Results from the above study showed that, while women and underrepresented minorities were represented proportionately to their percentage in the overall pool of medical students in the two HHMI-sponsored programs, they were not represented to the same degree in M.D.-Ph.D. programs. The data also showed that alumni from the HHMI programs were more likely or equally likely to receive a faculty appointment in a medical school that had significant research responsibility. In addition, they were as likely as non-MSTP M.D.-Ph.D. students to receive NIH postdoctoral grant support (an indicator of future success at obtaining NIH research support). This study concluded that the HHMI training programs are “an attractive strategy for training physician-scientists” (Fang and Meyer, 2003). Data for the analysis was obtained from the Association of American Medical Colleges (AAMC) Medical Student databases, the AAMC Faculty Roster Database, and the Medline and CRISP databases.

Approach #2: The second approach to evaluate the success of HHMI program alumni was to utilize the data collected by Cloister staff from the interactions described above. The results of the simple studies documenting the early careers of the former awardees are presented in Table 1.

The data in Table 1 show that alumni of the two HHMI programs are essentially equivalent. Roughly 60 percent were engaged in research 10 years after participation in the programs and over 50 percent went on to early careers in academic medicine. The data also show that 7 to 8 percent went on to study for a Ph.D. degree and that alumni of both programs published an average of over one paper a year even through their clinical training years. In addition, anecdotal testimonies were available from the accumulated information. The following quote is from an alumnus who is an associate professor in a major eastern medical school:

Suggested Citation:"Program Assessment in HHMI-Sponsored Medical Student Research Training Programs--Min K. Lee, Barbara Ziff, and William R. Galey." National Research Council. 2006. Enhancing Philanthropy's Support of Biomedical Scientists: Proceedings of a Workshop on Evaluation. Washington, DC: The National Academies Press. doi: 10.17226/11646.
×

TABLE 1 Intramural Study of HHMI-Supported Medical Student Research Training Programs, 1985–1992 Classes

Classes 1985a–1992

99% Reporting

Med Fellowsa

Cloister

Currently doing research (25% or more of time)

59%

58%

Hold an academic appointment

52%

51%

Went on for Ph.D.

8%

7%

Work in industry

3%

5%

Average number of publications per year

1.2

1.1

aMed fellows program began in 1989.

It is not an overstatement to say that the Research Scholars Program changed my career. Before entering the program, after my third year of medical school, I was going to be a head and neck surgeon. Surgery was exciting, interesting, and challenging. During my tenure in the laboratory, I realized that science was the same. There is no doubt that my year as a Research Scholar opened doors in both the medical and scientific communities.

EVALUATION OF THE TWO APPROACHES TO PROGRAM ASSESSMENT

Although both the internal and external program evaluations showed both HHMI training efforts to be equivalent and effective, they present differing views of the two programs and their relationship to other training opportunities. Because HHMI’s internal program evaluation has such detailed and up-to-date information on the career activities of alumni, it provides quantitatively different information from the external evaluation process that utilizes more publicly accessible information.

For instance, although the data were collected by both evaluations at essentially the same time, the internal analysis identified over 306 publications by a cadre of alumni, whereas the external assessment identified only 109 publications for the same group of individuals. A close look showed a number of differences, including:

  • Common names made it necessary to use author search criteria that selected not only on the name of the individual but also the institution of affiliation used in the publication. This resulted in some publica-

Suggested Citation:"Program Assessment in HHMI-Sponsored Medical Student Research Training Programs--Min K. Lee, Barbara Ziff, and William R. Galey." National Research Council. 2006. Enhancing Philanthropy's Support of Biomedical Scientists: Proceedings of a Workshop on Evaluation. Washington, DC: The National Academies Press. doi: 10.17226/11646.
×

tions not being attributed to a particular alumnus if he or she published using a slightly different institutional name or listed an institutional affiliation (such as a hospital or an institute) that differed from the institutional affiliation of the individual’s “academic appointment” used in the electronic database search.

  • Publications of some individuals were not found because some alumni had changed their names due to marriage.

  • Occasionally, individual publications were not found because they were attributed to an institution where the individual had a former affiliation rather than the current “academic appointment.”

  • Apparently some publications were not attributed to the alumni because the publication was identified with a collaborator rather than the alumnus.

Discrepancies were also identified between the database-oriented external evaluation and the internally derived data in determining “faculty positions” of alumni. The in-house-derived data obtained from program files showed that for a group of program alumni, 85 had obtained a faculty position, while the data derived from the AAMC Faculty Roster Schedule (FRS) found 75. The externally derived data were further compromised when it was found that of the 75 academic positions identified, 18 individuals had already left their positions and 28 attained faculty positions had not yet been listed in the FRS database by the academic institutions.

SUMMARY

Both the internal and external evaluations of the two HHMI medical student research training programs indicate that the two programs are successful and roughly equivalent. There are a number of advantages and disadvantages for each approach to the evaluation of alumni success.

The external review process is inherently respected for its presumed lack of bias and hence may have more external acceptance. Consequently, such studies may have greater public impact. Furthermore, the use of large databases such as those used in the study by Fang and Meyer allow much more adjustment for confounding factors, comparisons with other trainee populations, and powerful statistical analyses. Finally, the cost is significantly less, is nonreoccurring, and requires relatively little institutional staff effort.

Alternatively, the internal evaluation approach makes use of already-existing alumni data, is likely to be more accurate and complete, and is able to evaluate grant funding from non-NIH sources. However, for this to be true, a major effort must be made to assure that the data collected are

Suggested Citation:"Program Assessment in HHMI-Sponsored Medical Student Research Training Programs--Min K. Lee, Barbara Ziff, and William R. Galey." National Research Council. 2006. Enhancing Philanthropy's Support of Biomedical Scientists: Proceedings of a Workshop on Evaluation. Washington, DC: The National Academies Press. doi: 10.17226/11646.
×

comprehensive and up to date. It was perceived that although the external review was intellectually more rigorous, it appeared that decision-makers evaluating the program saw the sophisticated statistical analyses as too complicated and perhaps obfuscatory. The simple results and personal testaments provided by the less involved internal analyses seemed to have more impact.

ACKNOWLEDGMENT

The authors thank Dr. Anh-Chi Le for her help in preparing the presentation and manuscript.

REFERENCE

Fang, D., and R. E. Meyer. 2003. Effect of two Howard Hughes Medical Institute research training programs for medical students on the likelihood of pursuing research careers. Academic Medicine 78:1271–1280.

Suggested Citation:"Program Assessment in HHMI-Sponsored Medical Student Research Training Programs--Min K. Lee, Barbara Ziff, and William R. Galey." National Research Council. 2006. Enhancing Philanthropy's Support of Biomedical Scientists: Proceedings of a Workshop on Evaluation. Washington, DC: The National Academies Press. doi: 10.17226/11646.
×
Page 99
Suggested Citation:"Program Assessment in HHMI-Sponsored Medical Student Research Training Programs--Min K. Lee, Barbara Ziff, and William R. Galey." National Research Council. 2006. Enhancing Philanthropy's Support of Biomedical Scientists: Proceedings of a Workshop on Evaluation. Washington, DC: The National Academies Press. doi: 10.17226/11646.
×
Page 100
Suggested Citation:"Program Assessment in HHMI-Sponsored Medical Student Research Training Programs--Min K. Lee, Barbara Ziff, and William R. Galey." National Research Council. 2006. Enhancing Philanthropy's Support of Biomedical Scientists: Proceedings of a Workshop on Evaluation. Washington, DC: The National Academies Press. doi: 10.17226/11646.
×
Page 101
Suggested Citation:"Program Assessment in HHMI-Sponsored Medical Student Research Training Programs--Min K. Lee, Barbara Ziff, and William R. Galey." National Research Council. 2006. Enhancing Philanthropy's Support of Biomedical Scientists: Proceedings of a Workshop on Evaluation. Washington, DC: The National Academies Press. doi: 10.17226/11646.
×
Page 102
Suggested Citation:"Program Assessment in HHMI-Sponsored Medical Student Research Training Programs--Min K. Lee, Barbara Ziff, and William R. Galey." National Research Council. 2006. Enhancing Philanthropy's Support of Biomedical Scientists: Proceedings of a Workshop on Evaluation. Washington, DC: The National Academies Press. doi: 10.17226/11646.
×
Page 103
Suggested Citation:"Program Assessment in HHMI-Sponsored Medical Student Research Training Programs--Min K. Lee, Barbara Ziff, and William R. Galey." National Research Council. 2006. Enhancing Philanthropy's Support of Biomedical Scientists: Proceedings of a Workshop on Evaluation. Washington, DC: The National Academies Press. doi: 10.17226/11646.
×
Page 104
Suggested Citation:"Program Assessment in HHMI-Sponsored Medical Student Research Training Programs--Min K. Lee, Barbara Ziff, and William R. Galey." National Research Council. 2006. Enhancing Philanthropy's Support of Biomedical Scientists: Proceedings of a Workshop on Evaluation. Washington, DC: The National Academies Press. doi: 10.17226/11646.
×
Page 105
Next: Outcomes and Impacts of the National Science Foundation’s Minority Postdoctoral Research Fellowships Program--Carter Kimsey »
Enhancing Philanthropy's Support of Biomedical Scientists: Proceedings of a Workshop on Evaluation Get This Book
×
Buy Paperback | $48.00 Buy Ebook | $38.99
MyNAP members save 10% online.
Login or Register to save!
Download Free PDF

During an interval of 15 years, the Lucille P. Markey Charitable Trust spent over $500 million on four programs in the basic biomedical sciences that support the education and research of graduate students, postdoctoral fellows, junior faculty, and senior researchers. The Markey Trust asked the NRC to evaluate these programs with two questions in mind: “Were these funds well spent?” and “What can others in the biomedical and philanthropic communities learn from the programs of the Markey Trust, both as an approach to funding biomedical research and as a model of philanthropy?” One of five resulting reports, this volume contains the proceedings of a workshop held in June 2005 to investigate methods used to evaluate funding of the biomedical scientists by philanthropic and public funders. In addition to the Markey Trust, representatives from the Howard Hughes Medical Institute, the American Heart Association, the Doris Duke Charitable Foundation, the National Institutes of Health, and six other funders of biomedical scientists presented information on evaluation methodologies and outcomes.

  1. ×

    Welcome to OpenBook!

    You're looking at OpenBook, NAP.edu's online reading room since 1999. Based on feedback from you, our users, we've made some improvements that make it easier than ever to read thousands of publications on our website.

    Do you want to take a quick tour of the OpenBook's features?

    No Thanks Take a Tour »
  2. ×

    Show this book's table of contents, where you can jump to any chapter by name.

    « Back Next »
  3. ×

    ...or use these buttons to go back to the previous chapter or skip to the next one.

    « Back Next »
  4. ×

    Jump up to the previous page or down to the next one. Also, you can type in a page number and press Enter to go directly to that page in the book.

    « Back Next »
  5. ×

    Switch between the Original Pages, where you can read the report as it appeared in print, and Text Pages for the web version, where you can highlight and search the text.

    « Back Next »
  6. ×

    To search the entire text of this book, type in your search term here and press Enter.

    « Back Next »
  7. ×

    Share a link to this book page on your preferred social network or via email.

    « Back Next »
  8. ×

    View our suggested citation for this chapter.

    « Back Next »
  9. ×

    Ready to take your reading offline? Click here to buy this book in print or download it as a free PDF, if available.

    « Back Next »
Stay Connected!