Click for next page ( 2


The National Academies | 500 Fifth St. N.W. | Washington, D.C. 20001
Copyright © National Academy of Sciences. All rights reserved.
Terms of Use and Privacy Statement



Below are the first 10 and last 10 pages of uncorrected machine-read text (when available) of this chapter, followed by the top 30 algorithmically extracted key phrases from the chapter as a whole.
Intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text on the opening pages of each chapter. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.

Do not use for reproduction, copying, pasting, or reading; exclusively for search engines.

OCR for page 1
EVALUATING THE NATIONAL RESEARCH SERVICE AWARD PROGRAM: A REVIEW AND RECOMMENDATIONS FOR THE FUTURE Georgine M. Pioni The focus of this paper is (1) to review previous evaluation activities of the National Research Service Award (NRSA) program and (2) to propose an agenda describing the types of evaluation activities that should be carried out over the next 5 years. In line with this emphasis, a description of the major evaluation questions of interest to key program constituencies will be presented. Then, previous evaluation efforts will be discussed in terms of whether they addressed these questions and provided answers that could be viewed with a reasonable degree of confidence. The ''match"' between the questions of interest and the avaiJability.of sound evidence for answering these questions as gleaned from previous evaluations to date will serve as the basis for recommending future evaluation priorities. Throughout this paper, two considerations should be kept in mind. One concerns the diversity of the NRSA program itself. At first glance the overarching goal of this program is relatively straightforward: to train individuals for health-related research and teaching careers. However, in achieving this mandate, several different components and activities are encompassed by the program. For example, NRSA awards support training in a heterogeneous group of disciplines, ranging from genetics to health services research, and activities are administered by a variety of federal agencies and institutes, each with varying levels of experience in supporting research training. The training sponsored by these agencies also differs in terms of academic level (undergraduate, predoctoral, or postdoctoral), target populations, e.g., M.D.s, Ph.D.s, or ethnic minorities), and strategies (e.g., short-term training versus for-mar degree programs or disciplinary versus multidisciplinary approaches). Further, distinct funding mechanisms (individual) fellowships versus institutional training grants) are used to PI would like to thank David Cordray, Peter Rossi, Robert McGinnis, Grace Carter, and Robert Boruch for their critical and insightful comments on previous versions of this paper. Also, all the individuals interviewed, particularly Charles Sherman and Walter Schaffer, deserve special thanks for their willingness to answer questions, identify and locate materials, and discuss issues. The opinions expressed in this paper are the author's and do not reflect those of either the author's affiliation-- Vanderbilt (University) Institute for Public Policy Studies, the Committee on Biomedical and Behavioral Research Personnel, or the National Research Council.

OCR for page 1
support activities, incorporating different selection procedures and educational strategies (e.g., ''one-to-one" individually negotiated, student/mentor apprenticeships versus more formally structured degree programs within an institution). Consequently, previous evaluations in the NRSA program differ in terms of the specific program of interest, the target populations examined, the training activities invest ved, and the outcomes studied. It also must be remembered that the NRSA program is but one benefactor of research training. Both the National Institutes or Health (NIH) and the Alcohol, Drug Abuse, and Mental Health Administration (ADAMHA) sponsor other programs with quite similar aims (i.e., increasing the supply of productive researchers in health-related areas). These programs either directly or indirectly sponsor research training and/or research career development (e.g., Research Career Development Awards, Clinical Investigator Awards, the Minority Biomedical Research Support program, and individual investigator RO-1 grants). In addition, other federal agencies and nonfederal organizations support research training in biomedical and behavioral research at some levee (e.g., the National Science Foundation's Graduate Fellowship Program and the Robert Wood Johnson Clinical Scholars Program). Thus, the goals, functioning, outcomes and effects of NRSA programs must be viewed within the larger context of research training occurring in university departments, medical schools, faculty laboratories, and independent research canters. WHAT TYPES OF EVALUATION QUESTIONS ARE OF CURRENT INTEREST? In reviewing the quality of NRSA evaluation efforts, a major issue concerns the extent to which previous evaluations have addressed questions posed by major constituencies. Given that evaluations are intended to provide useful results, studies should speak to the key concerns expressed by the various stakeholders involved with the program. Four major constituencies for the NRSA program can be identified. They include (1) Congress, which is responsible for authorizing the program and appropriating funds; (2) NIH and ADAMHA, along with their individual institutes, which are in charge of administering the programs; (3) the individual fellows, trainees, and faculty involved in NRSA-supported training activities; and (4) other audiences with vested interests in training researchers (e.g., professional associations, scientific societies, and national ''blue ribbon" committees concerned with research and science policy). In order to identify the issues of primary interest to these constituencies, relevant legislation and evaluation reports were reviewed. Individual interviews with congressional staff, federal agency personnel responsible for NRSA policies and evaluation activities, and individuals in 2

OCR for page 1
charge of specific NRSA programs (N = 16) also yielded insight into the questions for which evaluative data are sought. 2 It should be noted that neither is interest in these questions always generated independently by each constituency nor is each constituency equal in terms of the urgency with which its demands are accommodated. For example' questions dictated by reauthorizing legislation and formal requests from congressional oversight committees to federal agencies mandate a response; Congress indeed is the holder of the purse strings, and given limited time and resources, its requests often rank higher on the list of agency priorities for evaluation. In addition, the evaluation questions of most interest to a particular group often depend on the extent of its nature with the NRSA program. For example, agency staff whose major responsibility lies in administering institutional training grants may be most enthusiastic about collecting data that could improve their ability to monitor and guide programs; in contrast, scientific societies' demands may stem primarily from their desire to develop stronger arguments for increased NRSA funding in their respective discipliners). The major evaluation questions that have been and/or currently are of interest to key NRSA constituencies can be categorized into seven generic types. These include questions about3 o the demand for the NRSA program (e.g., the adequacy of the current supply of for biomedical researchers); o o o levels of program participation, including numbers and characteristics of awardees; characteristics of program operation and functioning, such as whether payback requirements affect the attractiveness of the NRSA program to qualified applicants; program outcomes (e.g., the research career accomplishments of awardees) and/or program effectiveness (e.g., whether the subsequent success of awardees in obtaining federal grants is directly attributable to the program); 2A list of individuals interviewed is available from the author upon request. 3Because of time constraints, attention was focused on those constituencies most involved in setting priorities and administering policies for NRSA programs (i.e., Congress and federal agencies). 3

OCR for page 1
o outcomes and/or effectiveness of individual NRSA components (e.g., whether the Medical Scientist Training Program is more successful in training physician/investigators than extramural postdoctoral traineeships, intramural fellowships, physician/scientist awards, and/or a combination of training support mechanisms); o cost-effectiveness of the NRSA program; and o the development and maturation of scientific careers in General and the role of research training in this process (e.g., the components and determinants of scientific productivity). Appendix A provides detailed examples of the questions that emerged during interviews with congressional staff, federal agency personnel, and others involved in research training activities and policy. DEMAND FOR THE PROGRAM AND LEVELS OF PARTICIPATION Having definitive data on the need for research training support and levels of participation in NRSA programs are "bottom line" demands of all major stakeholders. For example, both the authorizing and reauthorizing legislation for the NRSA (e.g., P.L. 93-348 and P.L. 100-607) specify that awards are to be made only in areas/fields that have demonstrated a need for researchers. As such, Congressional appropriation committees traditionally have sought to base their fiscal decisions on this information, and actual "numerical recommendations' that indicate the number of training slots necessary to address shortages of researchers in specific areas have been frequently requested. 4 Agency staff share this predilection for reasonably precise estimates of researchers needed in specific fields, disease categories, or problem areas. Other groups, including both those who lobby Congress regularly for NRSA funding in individual disciplines and those concerned with the overall health of the scientific enterprise, also clamor for better projections of supply and demand. Occasionally, these stakeholders have even launched their own data collection efforts in an attempt to obtain this information (e.g., Barries, 1986; Porter, 1979~. 4In addition to specifying the number of training slots needed in a field or research area, there was a consensus among congressional staff that better explanations about the ways in which recommendations were derived (e.g., the assumptions underlying supply and demand models) were needed. 4

OCR for page 1
Related to supply and demand issues are questions about the "niche" occupied by NRSA programs in the overall landscape of research training support. All constituencies interviewed want to know the types of sponsors, the levels of their investment, and major priority areas for funding. Congress, in particular, wants such descriptive information so as to ascertain what the appropriate role of the federal government should be in the research training enterprise. A third question of perennial interest, frequently arising at congressional hearings, centers around the distribution of NRSA programs and funds. All constituencies want an accounting of awarded fellowships and traineeships, the research fields supported (e.g., nursing or primary care research), and changes over time. Such data are perceived as crucial to determining whether NRSA expenditures are targeted at "shortage" areas, to ascertaining whether agencies have responded to specific congressional directives, and to identifying where changes in NRSA program priorities or policies may be warranted. Also viewed as important is information on the characteristics of awardees, typically in terms of their sex, race/ethnicity, and institutional affiliation. Many of these questions have been spurred by disappointment in the low rates of participation by women and ethnic minorities in science, coupled with a concern that the nation's pool of scientists and engineers may prove inadequate to meet future challenges (e.g., Office of Technology Assessment 1985; Vetter, 1989~. PROGRAM CHARACTERISTICS AND OPERATION Of primary interest to federal agency staff who administer NRSA programs and policies are questions related to program functioning. These questions are quite diverse in their scope and content. They include requests for information on how institutional review groups (IRGs) make decisions about training grant awards, the amount and types of research training received by predoctoral and postdoctoral trainees, and whether faculty mentors indeed have active research programs in areas most relevant to an institute's goals and objectives. Program officers, however, are not the only source of these questions. The legislation for the NRSA program itself (e.g., P.L. 100-607) speaks to the general need for program monitoring so as to "determine what modifications in the [NRSA] programs are required to meet the needs E for research personnel]." More explicitly framed "operational" questions also have been posed by Congress, including how the payback requirement and current stipends for NRSA awards affect participation in the program. 5

OCR for page 1
PROGRAM OUTCOMES AND PROGRAM EFFECTIVENESS Questions related to both absolute outcomes (the accomplishments of NRSA trainees and fellows) and comparative outcomes (e.g., the performance of NRSA-funded predoctoral students in the life sciences as contrasted with those supported by the National Science Foundation) are specified clearly in the legislation. All authorizing and reauthorizing language states that National Academy of Sciences shall "identify the kinds of research positions available to and held by individuals completing NRSA and other current training programs]" (e.g., P.L. 100-607, Part F. Section 489~. Another example appears in the Health Research Extension Act of 1985 (P.L. 99-1258), which requested data on the "number of persons who receive NRSA awards and who engage in health research or training as a career." On the other hand, questions related to program effectiveness (i.e., whether outcomes are directly attributable to NRSA-supported training) are much less frequent and clearly enunciated. A broad and relatively vague mandate for effectiveness data appears in the Jaw (P.L. 100-607~; NAS is directed to "assess Current NRSA programs] and other current training programs available for . . . such personnel." Aside from this fairly global injunction, however, being able to confidently link NRSA training with specific achievements ranks lower on Congress's list of evaluation priorities. Agency staff members also express less interest in effectiveness questions, particularly if the attention paid to them is at the expense of other data collection efforts. What does generate enthusiasm among this group, is obtaining better information on program outcomes--both in absolute and comparative terms. Training officers at the various institutes want to know what happens to their awardees--for example, whether they remain active in research, whether their research is in the area of their NRSA training, and whether they have been instrumental in training other researchers. If these outcomes ultimately can be contrasted with the performance of individuals who received other types of research training that is supported by either their own agencies or by other sponsors, this would be an additional asset. It is likely that the lower priority assigned to addressing effectiveness issues stems from an array of factors. For example, there is an awareness of the enormous difficulty and cost involved in obtaining unequivocal data on the effects of research training, given the current structure of these programs (e.g., the heterogeneity of training experiences and the lack of uniformly applied selection criteria). Related to this is the strong sentiment, based on the substantial erosion in NRSA training monies over the last decade, that the first priority for spending any additional funds appropriated by Congress must be to 6

OCR for page 1
increase the number of training slots rather than to initiate rigorous impact evaluations. Further, in many programs staff members maintain that the necessary data for answering more basic questions about program demand and operation are not available and that this situation must be corrected before such "second- order" questions as program effectiveness are considered. When questions as to the effectiveness of NRSA programs do surface, they typically center around issues of relative effectiveness. For example, data that can "tease out" the effects of NRSA programs in producing biomedical researchers relative to the performance of other research training programs with similar goals are deemed more salient than evaluations aimed at understanding whether NRSA training is more effective than no research training at all or research training that is entirely financed by the individual through loans or other personal sources. OUTCOMES AND/OR EFFECTIVENESS OF INDIVIDUAL PROGRAM COMPONENTS As previously mentioned, the NRSA comprises a heterogeneous group of programs, many of which also have distinct program components. These include different funding mechanisms, different target populations, and different training philosophies and strategies. Outcomes associated with these individual components and their relative effectiveness have comprised the focus of congressional and administrative inquiries. For example, the Health Research Extension Act of 1985 requested a study on "the effectiveness of Ache training grant] mechanism in encouraging individuals to engage in health research and training as a career." Of constant concern to agency staff is "what works best" among or within NRSA components. Illustrative of this interest are such questions as "Are M.D./Ph.D. programs or postdoctoraI traineeships more efficacious in producing physician/investigators?" and "Is predoctoral training that is grounded in a particular discipline more successful for increasing the number of researchers attacking alcohol-related health problems than predoctoral training that incorporates several disciplinary perspectives and methodologies?" Cost-Effectiveness For the most part, cost-effectiveness questions do not constitute a high priority among major constituencies. The few questions that emerged in the interviews pertained to identifying ways to "best use the training buck," particularly if research training funds continue to erode. Somewhat redated to this concern are more global questions associated with the personal, disciplinary, and social costs incurred from having an insufficient amount of research monies available to support the 7

OCR for page 1
number of high-quality applications for individual investigator awards from researchers who have been trained in NRSA programs. Development and Maturation of Scientific Careers More frequently, the questions of interest to key NRSA constituencies are those that address research training, scientific productivity, and scientific career development in general rather than with regard to NRSA programs in particular. These questions span a variety of topics, including the relationship between research training and the quality of research, the factors governing an individual's choice to pursue a scientific career, and the resources required to most successfully maintain a productive research career. Also included in this category are those questions posed by agency staff about how to best measure relevant outcomes of research training (e.g., "active involvement in research" and "quality of researched. Although such questions are important for guiding and improving future evaluations of the NRSA program and can indeed be addressed by wel1-designed studies, it must be kept in mind that providing answers is neither the sole responsibility of NRSA nor the evaluation efforts connected with this program. Where Improvements in Evaluation Activities Are Needed? In the previous section the major evaluation questions of current interest to key constituencies were identified. Although these questions covered all aspects of the program, the priority areas centered around those issues associated with demand for the program, levels of participation, characteristics of training and recipients, and program outcomes. To date, past evaluation efforts have provided a wealth of data about National Research Service Awards, but many questions basic to understanding how these programs operate and what happens to awardees remain. Most individuals in charge of NRSA programs often continue to find themselves operating in almost a vacuum with regard to having sound, empirical data about how awardees are selected, the characteristics of participants, and the training environments and activities supported. Further, knowledge about the subsequent performance of awardees currently is confined to a limited set of indicators that vary considerably across individual NRSA programs and that incompletely characterize the intended outcomes. Improving this situation (i.e., "filling the gaps") is what must drive the individual items included in any portfolio of future evaluation activities. Four major gaps exist in terms of having an adequate knowledge base about NRSA programs: 8

OCR for page 1
Basic questions about program participation and functioning, although of great interest to key constituencies, have remained inadequately addressed. 2. Our understanding of program outcomes, let alone program effects, is still limited. 3. Insufficient attention has been given to determining what works best across and within program components. Evaluation efforts have been sparse in many fields and research problem areas. The first three gaps focus on "points of slippage" between the types of evaluation questions currently of interest to major constituencies and those that have comprised the thrust of prior evaluation efforts. An examination of the generic questions addressed by the 16 evaluation studies/reports reviewed indicated that 56 percent (N = 9) addressed program participation issues, and 50 percent (N = 8) collected data on program characteristics and operation. The overwhelming majority (94 percent, N = 15) presented information on one or more outcomes for programs or program elements, and 38 percent (N = 6) attempted to address in some way the effectiveness of NRSA programs or distinct components. None of the studies reviewed dealt with issues of r cost-effectiveness. ~ At first glance these percentages suggest that many of the questions of interest to NRSA stakeholders (e.g.' program outcomes) indeed have been addressed. However, it must be remembered that within each of these generic evaluation issues lie a variety of subquestions. For example, questions about program operation encompass the nature of the trainee selection process, characteristics of training, and the relationship of the payback requirement to participation levels. As shall be seen, the match between constituency priorities and evaluation efforts 5Given that studies could focus on more than one type of evaluation question, these percentages do not sum to 100 percent. Information on how the evaluations were chosen for review and on the broad categories of questions addressed by each is presented in Appendixes B and C, respectively. In classifying these evaluations, distinguishing between "outcome" studies and "effectiveness" studies has been in many cases a matter of judgment. For the purposes of this paper, an effectiveness study is one that incorporated either methodological (e.g., matching) and/or statistical procedures so as to control for selectivity bias. Those studies float attempted to compare outcomes but did not include any real consideration of selectivity bias were designated, rightly or wrongly, outcome studies. 9

OCR for page 1
on these more specific questions is where the discrepancies surface. (See Appendix A for a detailed enumeration of the questions posed by constituencies and Appendix D for a listing of those addressed by evaluation activities.) GAPS IN UNDERSTANDING PROGRAM PARTICIPATION AND OPERATION Although past evaluation efforts have addressed aspects of program participation and operation, several issues have escaped careful examination, including some that are basic to understanding any discrete program or intervention (e.g., program implementation). This situation is partly an outgrowth of the limited amount of resources that have been allotted for evaluation activities. Consequently, some programs (e.g., those sponsored by ADAMHA) have received little scrutiny. Another problem has concerned the fact that when evaluations were initiated, the short timelines imposed often dictated that the focus be on collecting outcome data (no mean feat by itself), with only secondary attention given to examining participation levels or program characteristics. As a result, summary profiles describing NRSA applicants, awardees, and program activities are either nonexistent, sketchy, or idiosyncratic in terms of the populations covered, the variables of interest, and the time periods examined. Needed Information on the Demand for the Program Although development of better supply and demand indicators for biomedical and behavioral science research personnel is covered more thoroughly in the full committee report, one related component deserves special mention in this paper. This concerns the extent of our knowledge about the research training enterprise as a whole (e.g., the total amount of funds, training opportunities, and types of training provided by all sponsors). Congress' motivation for having such information stems from its desire to ascertain what its role should be in financing research training and then to apply this understanding when making decisions about the NRSA program. Similarly, the interest of NIH and ADAMHA staff arises from their wish to better understand their own agency's total involvement in research training, particularly by mechanisms other than those covered under the NRSA umbrella (e.g., research assistantships paid by grants to individual investigators). Answers to these questions also are requested in the charge for evaluation specified in the authorizing and reauthorizing legislation for the NRSA program: to "assess current MESA programs] and other current training programs available for the training of such personnel" (P.L. 100-607, Part F. Section 489~. To carry out this charge, a map of the geography and topography 10

OCR for page 1
of non-NRSA funding sources and mechanisms for research training must be constructed. Previous committee reports (e.g., National Research Council, 1977, 1981) have attempted to survey this terrain, but this is no easy task. Currently, the best sources of data are the annual Survey of Earned Doctorates (SED) conducted by the National Research Council and the National Science Foundation's (NSF) Survey of Graduate Science Students and Postdoctorals. However, each has certain limitations. For example, the NSF survey requires institutions to indicate only one source of support for a graduate student. Although respondents to the SED are instructed to identify all sources of support and estimate the percentage of support received from each source, their ability to reconstruct these data accurately is unclear. Understanding the variety of research training activities sponsored by NIH and ADAMHA via non-NRSA mechanisms represents one step toward mapping the terrain, however. Of particular interest here is predoctoral and postdoctoral research training paid by research grants to individual investigators. Available data suggest that the use of this mechanism in supporting research training is not infrequent; research assistantships paid by federal and other grants were a source of predoctoral support for 16 percent of the 1987 Ph.D. recipients in the life sciences- -an almost equal percentage to that reported for NIH traineeships (Coyle and Thurgood, 1989~. Developing this capacity to obtain detailed training support data for all Public Health Service agencies may be more feasible than one might expect, given earlier and more recent efforts by NIH and ADAMHA. Information on all paid and unpaid personnel working on research grants was collected on a sample basis for all PHS grants in 1963 and NIH grants in 1969; beginning in 1973, these data were again requested of all NIH grantees in the NIH Research Grants Manpower Survey. Unfortunately, this effort was abolished in 1980 despite a reasonably favorable evaluation (Williams, 1979). ADAMHA staff members currently are investigating the feasibility of implementing a similar system for their own research grants and have already developed the system specifications, along with conducting some preliminary pilot tests (see Tjioe, 1989, for a description of this system). Variables in this system include name, social security number, role on the grant, type of position, sex, highest degrees), year of degree, birth date, field (e.g., surgery), and research discipline (e.g., brain damage) for all personnel connected with awards made by ADAMHA institutes. If resources were available to establish and maintain this data base for all PHS awards, questions relating to the various ways in which research training 11

OCR for page 1
greater confidence which types of training strategies may work better than others (e.g., multidisciplinary programs versus those that focus on one discipline or specialty). Efforts need to be initiated to rectify this situation. A1 though retrospective studies of former trainees and fellows have provided useful insight into training gaps and deficiencies (e.g., Gentile et al., 1987), they are limited. Surveys of recent Ph.D.s and M.D.s/Ph.D.s upon receipt of their doctorate could provide some preliminary insight into the types of variables that should be examined and how they can be measured best (e.g., the phrasing of survey questions, if appropriate). The same can be said for surveys of NRSA-supported postdoctoral trainees and fellows who have just completed their postdoctoral appointment. The most preferred strategy is to collect data from individuals at key intervals during the training process so as to avoid the problems of selective memory and to obtain a more complete picture of how and when training actually occurs. Such efforts also could provide data on many other issues. Program implementation could be examined (i.e., did individuals actually receive the training set forth in the funded applications. If information on trainee satisfaction were collected, the need for modifications in training policies or regulations also might be identified. Finally, if such efforts were extended to a sample of predoctoral students or postdoctoral associates without traineeships in the same academic program and possibly to students in similar programs in departments without training grants, we would begin to develop a better sense of the strength and integrity of various NRSA "treatments" (e.g., whether NRSA programs provide training experiences sufficiently different from those received by students in the same program or in other programs). The manner in which trainees are selected is another important area worthy of close scrutiny. We know, at least for the biomedical sciences, that predoctoral trainees tend to receive their training in top-ranked institutions/departments; presumably, the training grant application review process is selecting the best programs, and the departments in which these programs are based are highly selective in terms of graduate student admitting policies. However, we do not know whether 7These programs have been described as "contests" (Ross), personal communication, May, 1989) in that the most promising applicants are chosen to receive awards. It appears that the main contestants are institutions; the top-ranked research universities clearly are given preference. However, within the departments that house the NRSA programs, it is not clear whether the contest rules are the same, given the problems described in 30

OCR for page 1
those who actually receive traineeships are awarded them because they are the best "match" in terms of the requirements of the NRSA-supported program, because it is a convenient source of money at a particular period of time; or for other reasons. The same problem plagues applicants for postdoctoral appointments, particularly in medical schools. Needless to say, the ability to ultimately provide answers to questions seeking evidence on effectiveness is hampered by our lack of knowledge about how individuals are selected to receive NRSA support. INCREASED ATTENTION TO A8SE88ING PROGRAM OUTCOMES Those associated with NRSA programs, particularly at the program administrative level, want to know what works best in research training, either in terms of specific training models (e.g., broad versus specialized training) or in terms of specific populations (e.g., predoctoral support versus postdoctoral support versus both types of support in producing physician/ scientists). Previous large-scale evaluation efforts that have had to rely on retrospective assessment strategies have been unable to confidently provide unequivocal answers to these questions because of the probe ems associated with drawing solid comparison groups. Given this substantial interest in program outcomes, particularly comparative outcome data, concerted attention should be devoted to examining the full range of outcomes implied by the goals of the NRSA program (i.e., both research and teaching activities as specified in program announcements and payback requirements for NRSA programs) and to pilot testing new data collection strategies. The issues and variables warranting consideration are discussed by Fox (1983) and Gee (1989) and should be guided by previous research on scientific careers (e.g., Long et al., 1979; McGinnis et al., 1982; McGinnis and Long, 1988; Stephan, 1986~. Not only would efforts to accurately measure outcomes and test various measurement strategies help to better examine scientific productivity and career development, but they also would allow an understanding of the marginal gain associated with using these measures as contrasted to those previous approaches that have had to rely on existing archival data bases. Outcome assessments also could benefit from efforts geared at improving the use of existing archival data bases. For example, rather than using archival data bases to obtain a "snapshot" at one point in time of research grant success, the feasibility of using such data bases to track individuals over time, similar to that performed by Biddle et al. (1989) in this paper. This area requires further investigation, modeled on the work by Carter et al. (1987) for Research Career Development Awards. 31

OCR for page 1
examining the career accomplishments of Research Scientist Development awardees, could be explored. Another possibility would be to assess the feasibility of using relevant data from other archival sources that could augment the information on traditionally used measures (e.g., research grant success) or provide data on outcomes that have received only minimal attention (e.g., teaching others to be researchers). For example, Yasumura (1986) found that recipients of the KO-6 awards were more likely to receive PHS training grants--a relevant outcome, based on the request made by Congress for obtaining information on the number of awardees who engage in health research or teaching as a career (P.L. 99-1258~. In addition, exploring relationships between program characteristics and outcomes also is needed. For example, recent data on NHLBI awardees suggest that those M.D.s who were postdoctoral trainees in programs designed to train both M.D.s and Ph.D.s were more likely to be receive a NIH grant than those who were postdoctoral trainees in programs only aimed at M.D.s (C. Roth) personal communication, June, 1989~. More efforts of this kind can guide subsequent outcome studies in terms of identifying key programmatic variables that should be considered. Another focus should be on designing outcome studies where the entire "dosage" of training is measured. At the very least, for outcome studies of ADAMHA, NIH, or other PHS support, this information should be available from the TFF currently in existence. However, given that NRSA research training is just one part of funding for scientific personnel development, measuring the length of training must take into account all types of research training and development (e.g., short-term research training, predoctoral and postdoctoral traineeships and fellowships, and career development awards). If the previously described efforts for developing and/or reinstituting data bases on personnel working on PHS grants also are initiated, this would provide additional information on dosage by getting data on predoctoral or postdoctoral support financed by faculty research grants from these agencies. The completeness and accuracy of this information then could be checked by actually contacting the individuals chosen for examining outcomes to determine what types of training support they actually received--both from PHS and other sponsors. The final point is a call for evaluation efforts that incorporate a more longitudinal perspective. To date, the majority of the studies have been of the snapshot variety-- career accomplishments at one point in time. Future evaluation activities ideally must track individuals from the time they begin training (or apply for training opportunities) through the completion of their training and during their scientific careers. Even more limited efforts would benefit from following the Carter et al. (1987) and Biddle et al. (1988) approach to measuring 32

OCR for page 1
outcomes (e.g., PHS research grant activity) at several points after the individual has completed training. Only within a longitudinal perspective can a specific achievement or pattern of achievements be interpreted and yield information that can illuminate our understanding of these programs and research training in general. INCREASING THE AVAILABILITY OF RE80URCES FOR RESEARCH ON RESEARCH TRAINING Developing better outcome strategies is at least partially dependent on better understanding the nature of scientific careers, what influences productivity, and other similar relationships. Currently, funding for research on scientific careers, resources, and so forth is not abundant. The NSF's Division of Science Resources devotes some funds to supporting studies in these areas (e.g., National Science Foundation, 1986~. However, monies for this program are limited, and proposals often must include all scientific and engineering fields rather than those most relevant to NRSA programs or must focus on using preexisting survey data collected by NSF (data that typically includes, for example, only small numbers of NRSA-supported individuals in various categories). Research on scientific career development and enhancement can go a long way toward addressing basic questions about productivity, motivations for choosing a scientific career, the factors that facilitate or inhibit research productivity, and so on. In conjunction with these research programs, efforts need to be targeted at developing new measures of scientific productivity and quality of research and testing their feasibility. Work on assessing the quality of training programs, apart from the accomplishments of their graduates, is another component that requires attention, particularly if future evaluation efforts aimed at judging the quality of training grant recipients (i.e., programs/departments within institutions) are considered worthy of exploration. The list of research questions is endless, and reasonably strong arguments could be made for choosing any particular question or methodological strategy as the initial starting point. The major point is that resources for these efforts should be increased so that our ability to carry out evaluation activities that can both meet the needs of major NRSA constituencies and assist in the continual improvement of training activities in all fields of health-related research can be enhanced. 33

OCR for page 1
EXPLORING TRE FEA8IBILITY OF EVALUATTON DESIGN8 FOR ASSE8SING EFFECTIVENESS Given our lack of knowledge about the nature of the selection and training processes, improving the quality of future designs certainly represents a technical challenge. Perhaps the first step toward obtaining better estimates of the relative effectiveness of different types of research training programs and determining the training strategies or mechanisms that work best is to identify where the use of high quality designs can be implemented. One way to begin this process is to focus initially on exploring opportunities that may exist for assessing specific components. For example, the MSTP appears to be a promising candidate for consideration, given that training program directors often vigorously recruit students. Thus, a pool of applicants may be available to use as a comparison group. The feasibility of "beating the bushes" to augment the size of the applicant pool, of convincing directors to collect necessary data on applicant characteristics, and of persuading them to adhere to a standard selection process and criteria should be explored. Even if random assignment is not possible (a likely event), the use of such designs as regression-discontinuity is worth examining. This latter type of design was used successfully in Carter et al.'s (1987) evaluation of the Research Career Development Award. In addition, exploring the feasibility of implementing high quality evaluation designs for the MST program is attractive, given the level of interest in this component. For example, some concern has been expressed over the available supply of physician-researchers, and MD-Ph.D. training programs are being established by other agencies to resolve this problem. At the same time, however, the best way to produce physician-scientists is a point of controversy. Further, questions have been raised about the fact that the total expenditures per graduate for MST programs have been increasing over the last decade and are now significantly higher than for graduates supported by other predoctoral training grants (National Research Council, 1983~. Well-designed evaluations of the MST program, focusing on the outcomes of its graduates and its effectiveness on certain outcomes and as compared to other research training alternatives for physician-scientists, would do much towards addressing these issues. The lower level of interest in effectiveness questions expressed by constituencies, coupled with the difficulty in carrying out high quality studies to address these questions, does not, however, imply that exploring the feasibility of implementing rigorous designs for estimating program effects not important. The long-term objective for NRSA evaluation 34 IS

OCR for page 1
efforts should be to gain an understanding of what works in training, which programs and program elements work better, and how training should be assessed. Small-scale, pilot tests of more rigorous approaches can be quite instructive in terms of identifying where more better designs can be implemented and ultimately yield an understanding of research training itself. Many of the concerns being expressed today (e.g., future shortages of trained scientists, issues of scientific misconduct, and the lack of interdisciplinary research efforts in many major problem areas) have their core issues surrounding human resources and training. If we are successful in providing a better understanding of NRSA research training and how it operates and contributes to the development of outstanding researchers and research mentors, one step toward addressing these issues will have been taken. REFERENCES Alcohol, Drug Abuse, and Mental Health Administration (ADAMHA). 1989a. ADAMHA NRSA Research Training Tables FY 1988. Rockvil~e, MD. Alcohol, Drug Abuse, and Mental Health Administration. 1989b. Addressing the Shortage of Young ADAMHA Principal Investigators. (Available from Walter Schaffer, ADAMHA, Parklawn Building, 5600 Fishers Lane, Rockvil~e, MD 20857.) Barries, J. 1986. Projected supply and demand for Ph.D. Biologists. Paper presented at the meeting of the American Zoological Society, December 1986. Bicke1, J. W., C. R. Sherman, J. Ferguson, L. Baker, and T. E. Morgan. 1981. The Role of M.D.-Pl~.D. Training in Increasing the Supply of Physician-Scientists. New England Journal of Medicine 304:1265-1268. Biddle, A. R., G. M. Carter, J. S. Uebersax, and J. D. Winkler. 1988. Research Careers of Recipients of the Research Scientist and Research Scientist Development Award. Santa Monica, CA: The RAND Corporation. Carter, G. M., A. Robyn, and A. M. Singer. (1983~. The Supply of Physician Researchers and Support for Research Training: Part I of an Evaluation of the Hartford Fellowship Program (N-2003-HF). Santa Monica, CA: The RAND Corporation. Carter, G. M., J. D. Winkler, and A. K. Biddle. 1987. An Evaluation of the NIH Research Career Development Award. Santa Monica, CA: The RAND Corporation. 35

OCR for page 1
Clouet, D. H. 1986. The Career Achievements of Trainees and Fellows Supported by the National Institute of Drug Abuse. Rockville, MD: NIDA. Coggeshall, P. E., and P. W. Brown. 1984. The Career Achievements of NIH Predoctora~ Trainees and Fellows. Washington, D.C.: National Academy Press. Coyle, S. L., and Thurgood, D. H. 1989. Summary Report 1987: Doctorate Recipients from United States Universities. Washington, D.C.: National Academy Press. Fox, M. F. 1983. Publication Productivity Among Scientists: A Critical Review. Social Studies of Science 13:285-305. Garrison, H. H., and P. W. Brown, 1986. Minority Access to Research Careers: An Evaluation of the Honors Undergraduate Research Training Program. Washington, D.C.: National Academy Press. Gee, H. 1989. Productivity. Paper commissioned by the Biomedical and Behavioral Research Personnel Committee, National Academy of Sciences, Washington, D.C. Gentile, N. 0., G. S. Levey, P. Jolly, and T. H. Dial. 1987. Post-doctoral Research Training of Full-Time Faculty in Departments of Medicine. Washington, D.C.: Association of Professors of Medicine and Association of American Medical Colleges. Gerstein, D. R., Luce, R. D., N. J. Smelser, and S. Sperlich (eds.~. 1988. The Behavioral and Social Sciences: Achievements and Opportunities. Washington, D.C.: National Academy Press. Levey, G. S., C. R. Sherman, N. O. Gentile, L. J. Hough, T. H. Dial, and P. Jolly. 1988. Postdoctoral research training of full-time faculty in academic departments of medicine. Annals of Internal Medicine 109:414-418. Long, J. S., P. D. Allison, and R. McGinnis. 1979. Entrance into the academic career. American Sociological Review, 44:816-830. Look, M. 1989. Final Report on the Proposed Data Matrix for the Biomedical and Behavioral Research Personnel Committee. Bethesda, MD: Quantum Research Corporation. 36

OCR for page 1
Martin, M. R. 1986. The Results of a Manual Look-up of 300 References from Non-foreign Medical School Faculty Clinicians from the B/I/D Journal Set and an Analysis of Report by the AAMC on the Number and Character of Physical Researchers. (Available from Division of Program Analysis, National Institutes of Health, 9000 Rockville Pike, Bethesda, MD 20092.) McGinnis, R., and J. S. Long. 1988. Entry into academia: Effects of stratification, geography, and ecology. In D. W. Brenneman and T. I. K. Youn teds.) Academic Labor Markets and Careers, New York: Falmer Press. McGinnis, R., P. D. Allison, and J. S. Long. 1982. Postdoctoral training in bioscience: Allocation and outcomes. Social Forces 60:701-723. National Institute of Dental Research (NIDR) Ad Hoc Consultant Panel. 1988. The National Research Service Award Program: Report of the NIDR Ad Hoc Consultant Panel. Bethesda, MD: NIDR. National Institutes of Health (NIH). 1986. Effects of the National Research Service Award Program on Biomedical Research and Teaching Careers. Washington, D.C.: NIH. National Institutes of Health. 1987. NIH Data Book 1987. Washington, D.C. National Institutes of Health. 1988. NHLBI Fact Book Fiscal Year 1988. Washington, D.C.: NTH. National Research Council (NRC). 1976. Research Training and Career Patterns of Bioscientists: The Training Programs of the National Institutes of Health. Washington, D.C.: National Academy of Sciences. National Research Counci1. 1977. Personnel Needs and Training for Biomedical and Behavioral Research. Washington, D.C.: National Academy of Sciences. National Research Council. 1978. for Biomedical and Behavioral National Academy of Sciences. National Research Council. 1979. for Biomedical and Behavioral National Academy of Sciences. National Research Council. 19Bla. for Biomedical and Behavioral National Academy of Sciences. 37 Personnel Needs and Training Research. Washington, D.C.: Personnel Needs and Training Research. Washington, D.C.: Personnel Needs and Training Research. Washington, D.C.:

OCR for page 1
National Research Council. 198Ib. Postdoctoral Appointments and Disappointments. Washington, D.C.: NRC. National Research Council. 1983. Personnel Needs and Training for Biomedical and Behavioral Research. Washington, D.C.: National Academy Press. National Research Council. 1985. Personnel Needs and Training for Biomedical and Behavioral Research. Washington, D.C.: National Academy Press. National Research Council. 1989. Biomedical and Behavioral Research Scientists: Their Training and Supply. Washington, D.C.: NRC. National Science Foundation. 1988. (NSF 88-336~. Washington, D.C. Office. Project Summaries: FY 1988 : U.S. Government Printing Office of Technology Assessment. 1985. Demographic Trends and the Scientific and Engineering Work Force (OTA-TM-SET-35~. Washington, D.C.: U.S. Government Printing Office. Porter, B. F. 1979. Transition: A follow-up study of 1973 postdoctorals. In the American Physical Society, The Transition in Physics Doctoral Employment 1960-1990. New York, NY: APS. Primos, M. E. 1989a. Minority Access to Research Careers Program: Briefing on Program Activities, FY 1980 to 1986. (Available from M. Primos, Minority Research Resources Branch, NIMH, Parklawn Building, 5600 Fishers Lane, Rockville, MD 20857.) Primos, M. E. 1989b. Minority Fellowship Program: Summary of Program Activities, FY 1973 to FY 1987. (Available from M. Primos, Minority Research Resources Branch, NIMH, Parklawn Building, 5600 Fishers Lane, Rockville, MD 20857.) Schneider, S. 1980. Positions of Psychologists Trained for Research. American Psychologist 35:861-866. Sherman, C. R. 1983a. Notes on the NIH Role in Support of Postdoctoral Research Training of Two Groups of Physicians. (Available from Charles Sherman, NIH, 9000 Rockville Pike, Bethesda, MD 20092.) - Sherman, C. R. 1983b. Training and Manpower Development. Presentation at the Meeting of the Advisory Committee to the Director, NIH, Bethesda, MD. 38

OCR for page 1
Sherman, C. R. 1984. Perspectives on Short-Term Training in Schools of the Health Professions. (Available from Charles Sherman, NTH, 9000 Rockville Pike, Bethesda, MD 20092.) Sherman, C. R. 1989. The NIH Role in the Training of Individual Physician Faculty: A Supplementary Analysis. (Available from Charles Sherman, Office of Science Policy and Legislation, NIH, 9000 Rockvi1le Pike, Bethesda, MD 20092.) Sherman, C. R., H. P. Jolly, T. E. Morgan, E. J. Higgins, D. Hollander, T. Bry1l, and E. R. Sevilla III. 1981. On the Status of Medical School Faculty and Clinical Research Manpower 1968-1990. Washington, D.C.: National Institutes of Health. Snyder, J. 1988. Early Career Achievements of National Science Foundation Graduate Fellows, 1967-1976. Washington, D.C.: National Research Counci1. Stephan, P. 1986. Age publishing patterns in science: Estimation and measurement issues. In Anthony F. J. van Raan fed.), A Handbook of Quantitative Studies of Science and Technology. Amsterdam, Holland: Elsevier Science Publishing Division of North Holland. Task Force on Minority Research Training. 1986. Minority Research Training Programs at the National Institute of Mental Health: Descriptions and recommendations. Rockville, MD: National Institute of Mental Health. Tjioe, P. 1989. Researchers' Tracking System (RTS) User's Guide. Laurel, MD: General Sciences Corporation. Velletri, P. A., C. R. Sherman, and G. Bowden. 1985. A Comparison of the Career Achievements of NIH Predoctoral Trainees and Fellows. Washington, D.C.: National Institutes of Health. Vetter, B. M. 1989. Recruiting Doctoral Scientists and Engineers Today and Tomorrow (Occasional Paper 89-2~. Washington, D.C.: Commission on Professionals in Science and Technology. Williams, P. C. 1979. Evaluation of the Annual Manpower Survey of NIH Research Grants. Sumner, MD: Cooper-Wi1liams Associates. Yasumura, S. 1984. The Research Career Award (R06~: A 20-Year Perspective and an Analysis of Research Productivity. Washington, D.C.: National Institutes of Health. 39

OCR for page 1
Zumeta, W. 1985. Extending the Educational Ladder: The Changing Quality and Value of Postdoctoral Study. Lexington, Mass.: D.C. Heath. 40