Assessing the Markey Research Program Grants
Three different approaches were used to assess the Research Programs Grants. First, all grantees were required to submit annual progress reports to the Trust. No specific format for the annual reports was imposed with the consequence that progress reports varied greatly in what and how they reported. The progress reports of some grantees provided a detailed insight into the outcomes of the research conducted, as well as a diary of the process used to reach these outcomes. The progress reports of some grantees were less detailed and provided only thumbnail descriptions of activities conducted by the recipient organization. Despite the unevenness of the progress reports, the committee was able to use them to document some milestones for the grantees, including data on staffing changes, construction and renovation, and purchase of major equipment.
In addition, the committee and NRC staff made 19 site visits, conducted 12 telephone interviews with principal investigators, and received two letter reports. The selection of institutions for site visits was based on an intersection of several constructs. First, the committee recognized that there was neither the time nor the resources to visit all awardees. Second, the committee wanted to visit sites that received both large and small awards and sites that were infrastructure development and investigator initiated awardees. Third, the committee wanted to restrict site visits to those programs for which the principal investigator was still actively engaged with the program. One site was unable to participate as all staff with any institutional knowledge of the grant had left the institution.
Finally, in order to preserve resources, the committee concluded that, whenever possible, site visits should be made in clusters, minimizing travel time and expenses. Telephone interviews were used to augment the site visits. In two cases, NRC staff were unable to schedule a time for the telephone interview with the principal investigator, who submitted a letter report in place of the telephone interview. From some grantees telephone interviews and site visits and/or letter reports were obtained. Ultimately, data were obtained from 25 recipients.
These data provided the committee with valuable insights into how funds were used in a particular institution. The committee found, however, that it was difficult to generalize the insights garnered from these sources because of the diverse nature of the problems studied by grantees and the variety of awards made. The committee came to the conclusion that Research Program Grants were awarded to a heterogeneous group of investigators at a number of different universities within differently configured research centers.
The committee recognized that, at a minimum, Markey Research Program Grants awards could be classified into two categories: infrastructure development and investigator-initiated awards. The infrastructure development awards were used to create, expand, or enhance an existing department, center, or program or to develop new centers that focused on a particular aspect of the biological sciences. For several recipients, the awards resulted in the development of multidisciplinary departments within the biological sciences. Investigator-initiated awards focused on one or more particular research projects tied to a particular investigator or team of investigators. In addition, the committee classified Research Program Grants awards on a second dimension—the size of the award. The award amounts varied from less than $1 million to more than $13 million. The committee, somewhat arbitrarily, designated $4 million in total funding as the boundary between large and small awards. These awards are shown in Table A.
Infrastructure development awards tended to be made during the initial years of the Trust’s philanthropy and were, in general, large awards—that is, in excess of $4 million. Investigator-initiated awards tended to be made during the concluding years of the Trust’s philanthropy and were, in general, smaller awards—less than $4 million. However, there was sufficient variability in these awards that they overlapped in size with those for infrastructure development. Site visits were made to evaluate both infrastructure development and investigator-initiated awards and to recipients of both large and small awards.
A good example of such an intersection of dimensions is one of the earliest awards made by the Markey Trust to the University of California, San Francisco. This award of nearly $14 million enabled the university to
TABLE A
Grant Recipient |
Award Amount |
Years of Funding |
|
Beginning |
Ending |
||
Large - Infrastructure Development |
|||
California Institute of Technology |
$13,000,000 |
1986 |
1991 |
Case Western Reserve University |
$5,500,000 |
1988 |
1997 |
Cold Spring Harbor Laboratory |
$4,500,000 |
1991 |
1996 |
Columbia University |
$6,500,000 |
1988 |
1996 |
Cornell University Medical College |
$4,000,000 |
1992 |
1997 |
Duke University |
$8,000,000 |
1990 |
1994 |
Florida State University |
$4,500,000 |
1991 |
2000 |
Fox Chase Cancer Center |
$4,000,000 |
1991 |
1996 |
Harvard Medical School |
$11,000,000 |
1988 |
1993 |
Johns Hopkins University |
$7,150,000 |
1988 |
1996 |
Northwestern University |
$5,890,000 |
1989 |
1993 |
Purdue University |
$6,990,000 |
1988 |
1997 |
Stanford University |
$12,613,550 |
1986 |
1997 |
The Whitehead Institute for Biomedical Research |
$7,650,000 |
1988 |
1993 |
University of California, Los Angeles |
$4,350,000 |
1988 |
1997 |
University of California, San Diego |
$4,320,000 |
1988 |
1998 |
University of Colorado Health Sciences Center |
$5,000,000 |
1991 |
1996 |
University of Miami |
$6,270,000 |
1988 |
1999 |
University of Virginia |
$6,100,000 |
1990 |
1996 |
Washington University in St. Louis |
$12,100,000 |
1988 |
1994 |
Yale University |
$12,100,000 |
1988 |
1997 |
Large - Investigator Initiated |
|||
Princeton University |
$4,500,000 |
1992 |
1997 |
The Scripps Research Institute |
$5,000,000 |
1992 |
1996 |
The University of Michigan |
$8,250,000 |
1989 |
1997 |
University of California, Berkeley |
$8,500,000 |
1989 |
1994 |
University of Chicago |
$9,219,223 |
1986 |
1992 |
University of Pennsylvania |
$4,720,402 |
1988 |
1996 |
University of Rochester School of Medicine/Dentistry |
$4,000,000 |
1991 |
1997 |
University of Washington |
$7,500,000 |
1990 |
1997 |
Vanderbilt University |
$5,500,000 |
1991 |
1996 |
Small - Infrastructure Development |
|||
Carnegie Institute of Washington |
$2,700,000 |
1988 |
1997 |
Carnegie-Mellon University |
$1,925,000 |
1986 |
1992 |
Children’s Memorial Medical Center |
$1,000,000 |
1995 |
1997 |
Cincinnati Children’s Hospital Medical Center |
$3,500,000 |
1992 |
1996 |
Grant Recipient |
Award Amount |
Years of Funding |
|
Beginning |
Ending |
||
Harvard University |
$1,600,000 |
1995 |
1998 |
Harvard University, School of Public Health |
$3,500,000 |
1991 |
1996 |
Massachusetts General Hospital |
$3,000,000 |
1993 |
1997 |
New York University |
$2,600,000 |
1991 |
1997 |
Public Health Research Institute |
$2,500,000 |
1992 |
1996 |
Stanford University |
$1,200,000 |
1995 |
1997 |
The Burnham Institute |
$1,500,000 |
1992 |
1996 |
The Children’s Hospital, Boston |
$2,475,000 |
1988 |
1993 |
The Salk Institute for Biological Studies |
$2,600,000 |
1994 |
1996 |
The University of Utah |
$2,500,000 |
1993 |
1997 |
Thomas Jefferson University |
$3,500,000 |
1990 |
1994 |
University of California, Santa Cruz |
$2,500,000 |
1992 |
1999 |
University of Colorado, Boulder |
$1,500,000 |
1995 |
1997 |
University of Massachusetts Medical Center |
$1,500,000 |
1995 |
1997 |
University of North Carolina at Chapel Hill |
$1,500,000 |
1995 |
1997 |
University of Oregon |
$3,300,000 |
1988 |
1995 |
University of Texas-Houston Health Sciences Center |
$1,000,000 |
1995 |
1997 |
University of Texas-Southwestern Medical Center |
$2,280,000 |
1986 |
1992 |
University of Texas-Southwestern Medical Center |
$1,045,000 |
1988 |
1994 |
University of Texas Medical Branch at Galveston |
$1,000,000 |
1995 |
1996 |
University of Vermont |
$2,300,000 |
1991 |
1999 |
Wisconsin University-Madison |
$990,000 |
1988 |
1992 |
Small - Investigator Initiated |
|||
Albert Einstein College of Medicine of Yeshiva Univ. |
$2,310,000 |
1988 |
1995 |
Baylor College of Medicine |
$1,400,000 |
1994 |
1999 |
Brandeis University |
$3,200,000 |
1988 |
1996 |
Brown University |
$1,300,000 |
1994 |
1998 |
Cornell University |
$1,200,000 |
1995 |
1999 |
Dana Farber Cancer Institute |
$1,500,000 |
1995 |
1997 |
Dartmouth Hitchcock Medical School |
$1,500,000 |
1994 |
1997 |
Eleanor Roosevelt Institute for Cancer Research |
$1,475,000 |
1988 |
1993 |
Georgetown University |
$1,000,000 |
1995 |
1997 |
Johns Hopkins University |
$1,300,000 |
1995 |
1997 |
Joslin Diabetes Center |
$3,500,000 |
1993 |
1999 |
Kennedy Krieger Institute |
$500,000 |
1995 |
1997 |
Grant Recipient |
Award Amount |
Years of Funding |
|
Beginning |
Ending |
||
Massachusetts Institute of Technology |
$3,850,000 |
1991 |
1999 |
Memorial Sloan-Kettering |
$2,700,000 |
1991 |
1994 |
Mount Sinai Medical Center |
$3,000,000 |
1993 |
1997 |
Neurosciences Institute |
$1,375,000 |
1988 |
1995 |
Oregan Health Sciences University |
$1,300,000 |
1995 |
1997 |
Rice University |
$1,200,000 |
1995 |
1997 |
Rush Presbyterian St. Lukes Medical Center |
$1,000,000 |
1995 |
1998 |
Schepens Eye Research Institute |
$1,000,000 |
1995 |
1997 |
SUNY-Buffalo |
$1,000,000 |
1995 |
1997 |
Temple University |
$2,500,000 |
1990 |
1996 |
Texas A&M University |
$1,000,000 |
1995 |
1998 |
Tufts University |
$2,000,000 |
1993 |
1996 |
University of Alabama at Birmingham |
$1,500,000 |
1991 |
1995 |
University of California, Davis |
$1,600,000 |
1995 |
2002 |
University of California, Irvine |
$1,000,000 |
1995 |
1999 |
University of Florida |
$1,600,000 |
1995 |
2000 |
University of Illinois Urbana Champagne |
$3,000,000 |
1992 |
1998 |
University of Kentucky |
$1,900,000 |
1995 |
1998 |
University of Maryland Biotechnology Institute |
$1,000,000 |
1995 |
1997 |
University of Miami |
$1,000,000 |
1995 |
1998 |
University of Pittsburgh |
$1,000,000 |
1995 |
1997 |
University of Southern California |
$1,800,000 |
1994 |
1998 |
University of Wisconsin-Madison |
$3,000,000 |
1992 |
2003 |
Worchester Foundation |
$1,000,000 |
1994 |
1997 |
establish the Program in Biomedical Science, which restructured research and graduate education in the School of Medicine. Although this award was originally classified as a General Organizational Grant by Markey Trustees, the committee saw in this award the genesis of the infrastructure development and concluded that for analytical purposes it should be considered a Research Program Grants award. This program subsequently received a site visit by the committee.
Although the PIBS program at UCSF was site visited by the committee and the committee considered it an exemplary program, technically it was not a Research Program Grants award. Consequently, data on the UCSF awards are not included in Appendix D. The Markey Trust awarded the first General Organizational Grant award to UCSF in 1988. In subsequent years, the Trustees changed the focus of General Organizational
Grants to training in translational research. For a more thorough assessment of General Organizational Grants in general and the PIBS program in particular, see Bridging the Bed-Bench Gap: Contributions of the Markey Trust, published by National Academies Press in 2004.
A third source of information came from analysis of the Lucille P. Markey Charitable Trust Records. As noted earlier, as the Trust was entering its final years, it arranged for all Trust documents to be archived at the Rockefeller Archive Center in Sleepy Hollow, New York. Following the conclusion of the Trust in 1997, all documents were transferred to the center, classified, and microfilmed. The archived Lucille P. Markey Charitable Trust Records currently consist of 153 reels of microfilm with approximately 800 frames on each reel. They are a rich source of information on all aspects of the Trust and will be made available to the public in 2007. The NRC staff searched the archive for information on the process used by the Trust to (1) define the rationale and focus of the Research Program Grants awards, (2) develop the solicitation process, (3) develop the mechanism and protocols for funding these awards, (4) establish the selection process for the awards, and (5) gain understanding of the decisions that led to the 18 supplemental grants.
LIMITATIONS OF THIS ASSESSMENT
Examination of these three data sources provided insights into the purpose of the awards, the processes of selecting grantees, and the impact of grant funds on researchers and recipient institutions. The committee realized that it was impossible to systematically collect data that would enable an assessment of individual programs. The committee had considered reviewing all publications that emerged from Markey funded projects; examining patents and licenses produced with Markey funding; and tabulating subsequent extramural funding that was produced by the Research Program Grants as one way to assess individual programs. However, the committee came to the conclusion that the data needed for such an assessment were inconsistent or missing or both—not only between programs, but also within them. For several additional reasons the committee’s ability to conduct an evaluation was limited:
-
The grantees, by design, were not homogeneous; rather they represented a broad spectrum of large and small universities, academic medical centers, and research institutes.
-
There was no single, overriding principle that directed funding for Research Program Grants. Rather, the Trustees were directed by the guideline “for the purposes of supporting and encouraging basic medical research,” and this guideline was broadly and variously interpreted.
-
The focus of Research Program Grants changed over time from a few large awards to many smaller awards. Another part of the reason for this change was a conscious decision by Trustees to make a larger number of smaller awards to newly expanding programs in biomedical research. Another part of the reason for this change was pragmatic, based on the decreased availability of funding as the Trust neared completion.
-
Grant applications were relatively short and were more conceptual than comprehensive. Consequently, goals, objectives, milestones, and activities were not stated in sufficient detail to be evaluated….
-
Systematic data were not collected on grantees’ progress toward completing the goals and milestones specified in a grant. Although grantees were required to submit annual progress reports in order to receive subsequent-year funding, there was no specific format for these progress reports and they varied considerably in length and detail both longitudinally and across grantees.
-
No comparison group of similar recipients of grants from other sources could be identified.
The committee concluded that neither the data nor the existing resources would permit a rigorous evaluation of the program. Unlike grant programs funded by NIH and NSF, the Markey Research Program Grants were not guided by a systematic, uniform solicitation on which proposals were based. In addition, the focus of the Research Program Grants was fluid—some grants were made to create or build programs, some were for research, and some were for infrastructure development—reflecting both the availability of funds and changes in the goals of the Trustees. Finally, the Trustees did not request data from the grantees that would permit the evaluation of Research Program Grants. Although the Trustees recognized the need for an assessment of the Trust’s funding activities, the decision to conduct an assessment was made toward the end of the Trust’s tenure. Ideally, program evaluation would have been built in from the very beginning of the program, with variables for measuring program outcomes identified and collected from its onset. In any case, the outcome of basic research is a very long term prospect and may not lend itself to easy assessment. Thus, intermediate or other variables, such as publication rates, new faculty and postdocs, or funding rates, might be used as proxies. Foundations or government agencies that wish to assess their funding programs will have to determine what kinds of information would be useful to guide future decisions about such funding schemes and build their evaluation around such measures.
The committee recognized that systematic collection of key variables would be essential to monitor grantees and assess the outcomes of grants
such as those awarded through the Research Program Grants. The committee considered a number of issues, including:
-
Should there be a minimum data set (MDS) of variables collected from all grantees and, if so, what variables should be included in the MDS;
-
How frequently should data be collected from grantees; and
-
What mechanism should be used to collect these MDS data?
The committee found guidelines for these issues from the practices used by NIH and NSF for renewal or awards. Both agencies have established systematic reporting mechanisms for the renewal of grants. NSF uses the Annual Progress Report (through Fastlane) and NIH uses the Grant Progress Report (PHS 2590) to collect these interim data.
While the content of the two mechanisms differ in details, both agencies collect monitoring data in four areas:
-
Detailed budget information, including costs for personnel (itemized for all staff employed on the project), consultants, equipment (itemized), supplies (itemized by category), travel, alterations/renovations, construction, other costs not elsewhere classified, and overhead.
-
A listing of key personnel, their role on the project, and their annual effort;
-
A narrative describing the project’s progress including specific aims, studies and results, significance of findings, and plans for the coming year; and
-
A listing of publications generated by the project.
The committee believes that these data elements could serve as the basis for establishing an MDS for annual progress reports grantees submit to funders. Both NIH and NSF utilize electronic transmittal of progress reports (NSF uses Fastlane and NIH uses SNAP, the Streamlined Noncompetitive Award Process). The committee recognizes the advantage of electronic transmittal, but believes that submission in a standardized protocol via the Internet would meet the needs of most philanthropic funders.
In addition, the committee recognizes that establishing an MDS in itself is not sufficient to assess the outcomes of programs like the Research Program Grants. This requires identification of the possible products of such programs and the more careful, in-depth, and systematic measurement of these outcomes by the evaluation team. Moreover, the committee recognizes that if the Markey Trustees had adopted an MDS and established goals and outcomes for grantees, its evaluation of the Research Program Grants would have been much more straightforward; although
the committee also recognizes that the implementation of the MDS and the identification, measurement, and collection of specific program products and outcomes will inevitably lead to greater bureaucratic requirements and less flexibility in program administration, characteristics identified as strengths of the Markey Trust’s framework.