Below are the first 10 and last 10 pages of uncorrected machine-read text (when available) of this chapter, followed by the top 30 algorithmically extracted key phrases from the chapter as a whole.
Intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text on the opening pages of each chapter.
Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.
Do not use for reproduction, copying, pasting, or reading; exclusively for search engines.
OCR for page 1
EVALUATING THE NATIONAL RESEARCH SERVICE AWARD PROGRAM:
A REVIEW AND RECOMMENDATIONS FOR THE FUTURE
Georgine M. Pioni
The focus of this paper is (1) to review previous evaluation
activities of the National Research Service Award (NRSA) program
and (2) to propose an agenda describing the types of evaluation
activities that should be carried out over the next 5 years. In
line with this emphasis, a description of the major evaluation
questions of interest to key program constituencies will be
presented. Then, previous evaluation efforts will be discussed
in terms of whether they addressed these questions and provided
answers that could be viewed with a reasonable degree of
confidence. The ''match"' between the questions of interest and
the avaiJability.of sound evidence for answering these questions
as gleaned from previous evaluations to date will serve as the
basis for recommending future evaluation priorities.
Throughout this paper, two considerations should be kept in
mind. One concerns the diversity of the NRSA program itself. At
first glance the overarching goal of this program is relatively
straightforward: to train individuals for health-related
research and teaching careers. However, in achieving this
mandate, several different components and activities are
encompassed by the program. For example, NRSA awards support
training in a heterogeneous group of disciplines, ranging from
genetics to health services research, and activities are
administered by a variety of federal agencies and institutes,
each with varying levels of experience in supporting research
training. The training sponsored by these agencies also differs
in terms of academic level (undergraduate, predoctoral, or
postdoctoral), target populations, e.g., M.D.s, Ph.D.s, or ethnic
minorities), and strategies (e.g., short-term training versus
for-mar degree programs or disciplinary versus multidisciplinary
approaches). Further, distinct funding mechanisms (individual)
fellowships versus institutional training grants) are used to
PI would like to thank David Cordray, Peter Rossi, Robert
McGinnis, Grace Carter, and Robert Boruch for their critical and
insightful comments on previous versions of this paper. Also,
all the individuals interviewed, particularly Charles Sherman and
Walter Schaffer, deserve special thanks for their willingness to
answer questions, identify and locate materials, and discuss
issues. The opinions expressed in this paper are the author's
and do not reflect those of either the author's affiliation--
Vanderbilt (University) Institute for Public Policy Studies, the
Committee on Biomedical and Behavioral Research Personnel, or the
National Research Council.
OCR for page 2
support activities, incorporating different selection procedures
and educational strategies (e.g., ''one-to-one" individually
negotiated, student/mentor apprenticeships versus more formally
structured degree programs within an institution). Consequently,
previous evaluations in the NRSA program differ in terms of the
specific program of interest, the target populations examined,
the training activities invest ved, and the outcomes studied.
It also must be remembered that the NRSA program is but one
benefactor of research training. Both the National Institutes or
Health (NIH) and the Alcohol, Drug Abuse, and Mental Health
Administration (ADAMHA) sponsor other programs with quite similar
aims (i.e., increasing the supply of productive researchers in
health-related areas). These programs either directly or
indirectly sponsor research training and/or research career
development (e.g., Research Career Development Awards, Clinical
Investigator Awards, the Minority Biomedical Research Support
program, and individual investigator RO-1 grants). In addition,
other federal agencies and nonfederal organizations support
research training in biomedical and behavioral research at some
levee (e.g., the National Science Foundation's Graduate
Fellowship Program and the Robert Wood Johnson Clinical Scholars
Program). Thus, the goals, functioning, outcomes and effects of
NRSA programs must be viewed within the larger context of
research training occurring in university departments, medical
schools, faculty laboratories, and independent research canters.
WHAT TYPES OF EVALUATION QUESTIONS
ARE OF CURRENT INTEREST?
In reviewing the quality of NRSA evaluation efforts, a major
issue concerns the extent to which previous evaluations have
addressed questions posed by major constituencies. Given that
evaluations are intended to provide useful results, studies
should speak to the key concerns expressed by the various
stakeholders involved with the program.
Four major constituencies for the NRSA program can be
identified. They include (1) Congress, which is responsible for
authorizing the program and appropriating funds; (2) NIH and
ADAMHA, along with their individual institutes, which are in
charge of administering the programs; (3) the individual fellows,
trainees, and faculty involved in NRSA-supported training
activities; and (4) other audiences with vested interests in
training researchers (e.g., professional associations, scientific
societies, and national ''blue ribbon" committees concerned with
research and science policy). In order to identify the issues of
primary interest to these constituencies, relevant legislation
and evaluation reports were reviewed. Individual interviews with
congressional staff, federal agency personnel responsible for
NRSA policies and evaluation activities, and individuals in
2
OCR for page 3
charge of specific NRSA programs (N = 16) also yielded insight
into the questions for which evaluative data are sought. 2
It should be noted that neither is interest in these
questions always generated independently by each constituency nor
is each constituency equal in terms of the urgency with which its
demands are accommodated. For example' questions dictated by
reauthorizing legislation and formal requests from congressional
oversight committees to federal agencies mandate a response;
Congress indeed is the holder of the purse strings, and given
limited time and resources, its requests often rank higher on the
list of agency priorities for evaluation. In addition, the
evaluation questions of most interest to a particular group often
depend on the extent of its nature with the NRSA program. For
example, agency staff whose major responsibility lies in
administering institutional training grants may be most
enthusiastic about collecting data that could improve their
ability to monitor and guide programs; in contrast, scientific
societies' demands may stem primarily from their desire to
develop stronger arguments for increased NRSA funding in their
respective discipliners).
The major evaluation questions that have been and/or
currently are of interest to key NRSA constituencies can be
categorized into seven generic types. These include questions
about3
o the demand for the NRSA program (e.g., the adequacy of
the current supply of for biomedical researchers);
o
o
o
levels of program participation, including numbers and
characteristics of awardees;
characteristics of program operation and functioning,
such as whether payback requirements affect the
attractiveness of the NRSA program to qualified
applicants;
program outcomes (e.g., the research career
accomplishments of awardees) and/or program
effectiveness (e.g., whether the subsequent success of
awardees in obtaining federal grants is directly
attributable to the program);
2A list of individuals interviewed is available from the
author upon request.
3Because of time constraints, attention was focused on those
constituencies most involved in setting priorities and
administering policies for NRSA programs (i.e., Congress and
federal agencies).
3
OCR for page 4
o outcomes and/or effectiveness of individual NRSA
components (e.g., whether the Medical Scientist
Training Program is more successful in training
physician/investigators than extramural postdoctoral
traineeships, intramural fellowships,
physician/scientist awards, and/or a combination of
training support mechanisms);
o cost-effectiveness of the NRSA program; and
o
the development and maturation of scientific careers in
General and the role of research training in this
process (e.g., the components and determinants of
scientific productivity).
Appendix A provides detailed examples of the questions that
emerged during interviews with congressional staff, federal
agency personnel, and others involved in research training
activities and policy.
DEMAND FOR THE PROGRAM AND LEVELS OF PARTICIPATION
Having definitive data on the need for research training
support and levels of participation in NRSA programs are "bottom
line" demands of all major stakeholders. For example, both the
authorizing and reauthorizing legislation for the NRSA (e.g.,
P.L. 93-348 and P.L. 100-607) specify that awards are to be made
only in areas/fields that have demonstrated a need for
researchers. As such, Congressional appropriation committees
traditionally have sought to base their fiscal decisions on this
information, and actual "numerical recommendations' that indicate
the number of training slots necessary to address shortages of
researchers in specific areas have been frequently requested. 4
Agency staff share this predilection for reasonably precise
estimates of researchers needed in specific fields, disease
categories, or problem areas. Other groups, including both those
who lobby Congress regularly for NRSA funding in individual
disciplines and those concerned with the overall health of the
scientific enterprise, also clamor for better projections of
supply and demand. Occasionally, these stakeholders have even
launched their own data collection efforts in an attempt to
obtain this information (e.g., Barries, 1986; Porter, 1979~.
4In addition to specifying the number of training slots
needed in a field or research area, there was a consensus among
congressional staff that better explanations about the ways in
which recommendations were derived (e.g., the assumptions
underlying supply and demand models) were needed.
4
OCR for page 5
Related to supply and demand issues are questions about the
"niche" occupied by NRSA programs in the overall landscape of
research training support. All constituencies interviewed want
to know the types of sponsors, the levels of their investment,
and major priority areas for funding. Congress, in particular,
wants such descriptive information so as to ascertain what the
appropriate role of the federal government should be in the
research training enterprise.
A third question of perennial interest, frequently arising
at congressional hearings, centers around the distribution of
NRSA programs and funds. All constituencies want an accounting
of awarded fellowships and traineeships, the research fields
supported (e.g., nursing or primary care research), and changes
over time. Such data are perceived as crucial to determining
whether NRSA expenditures are targeted at "shortage" areas, to
ascertaining whether agencies have responded to specific
congressional directives, and to identifying where changes in
NRSA program priorities or policies may be warranted. Also
viewed as important is information on the characteristics of
awardees, typically in terms of their sex, race/ethnicity, and
institutional affiliation. Many of these questions have been
spurred by disappointment in the low rates of participation by
women and ethnic minorities in science, coupled with a concern
that the nation's pool of scientists and engineers may prove
inadequate to meet future challenges (e.g., Office of Technology
Assessment 1985; Vetter, 1989~.
PROGRAM CHARACTERISTICS AND OPERATION
Of primary interest to federal agency staff who administer
NRSA programs and policies are questions related to program
functioning. These questions are quite diverse in their scope
and content. They include requests for information on how
institutional review groups (IRGs) make decisions about training
grant awards, the amount and types of research training received
by predoctoral and postdoctoral trainees, and whether faculty
mentors indeed have active research programs in areas most
relevant to an institute's goals and objectives.
Program officers, however, are not the only source of these
questions. The legislation for the NRSA program itself (e.g.,
P.L. 100-607) speaks to the general need for program monitoring
so as to "determine what modifications in the [NRSA] programs are
required to meet the needs E for research personnel]." More
explicitly framed "operational" questions also have been posed by
Congress, including how the payback requirement and current
stipends for NRSA awards affect participation in the program.
5
OCR for page 6
PROGRAM OUTCOMES AND PROGRAM EFFECTIVENESS
Questions related to both absolute outcomes (the
accomplishments of NRSA trainees and fellows) and comparative
outcomes (e.g., the performance of NRSA-funded predoctoral
students in the life sciences as contrasted with those supported
by the National Science Foundation) are specified clearly in the
legislation. All authorizing and reauthorizing language states
that National Academy of Sciences shall "identify the kinds of
research positions available to and held by individuals
completing NRSA and other current training programs]" (e.g.,
P.L. 100-607, Part F. Section 489~. Another example appears in
the Health Research Extension Act of 1985 (P.L. 99-1258), which
requested data on the "number of persons who receive NRSA awards
and who engage in health research or training as a career."
On the other hand, questions related to program
effectiveness (i.e., whether outcomes are directly attributable
to NRSA-supported training) are much less frequent and clearly
enunciated. A broad and relatively vague mandate for
effectiveness data appears in the Jaw (P.L. 100-607~; NAS is
directed to "assess Current NRSA programs] and other current
training programs available for . . . such personnel." Aside
from this fairly global injunction, however, being able to
confidently link NRSA training with specific achievements ranks
lower on Congress's list of evaluation priorities.
Agency staff members also express less interest in
effectiveness questions, particularly if the attention paid to
them is at the expense of other data collection efforts. What
does generate enthusiasm among this group, is obtaining better
information on program outcomes--both in absolute and comparative
terms. Training officers at the various institutes want to know
what happens to their awardees--for example, whether they remain
active in research, whether their research is in the area of
their NRSA training, and whether they have been instrumental in
training other researchers. If these outcomes ultimately can be
contrasted with the performance of individuals who received other
types of research training that is supported by either their own
agencies or by other sponsors, this would be an additional
asset.
It is likely that the lower priority assigned to addressing
effectiveness issues stems from an array of factors. For
example, there is an awareness of the enormous difficulty and
cost involved in obtaining unequivocal data on the effects of
research training, given the current structure of these programs
(e.g., the heterogeneity of training experiences and the lack of
uniformly applied selection criteria). Related to this is the
strong sentiment, based on the substantial erosion in NRSA
training monies over the last decade, that the first priority for
spending any additional funds appropriated by Congress must be to
6
OCR for page 7
increase the number of training slots rather than to initiate
rigorous impact evaluations. Further, in many programs staff
members maintain that the necessary data for answering more basic
questions about program demand and operation are not available
and that this situation must be corrected before such "second-
order" questions as program effectiveness are considered.
When questions as to the effectiveness of NRSA programs do
surface, they typically center around issues of relative
effectiveness. For example, data that can "tease out" the
effects of NRSA programs in producing biomedical researchers
relative to the performance of other research training programs
with similar goals are deemed more salient than evaluations aimed
at understanding whether NRSA training is more effective than no
research training at all or research training that is entirely
financed by the individual through loans or other personal
sources.
OUTCOMES AND/OR EFFECTIVENESS
OF INDIVIDUAL PROGRAM COMPONENTS
As previously mentioned, the NRSA comprises a heterogeneous
group of programs, many of which also have distinct program
components. These include different funding mechanisms,
different target populations, and different training philosophies
and strategies. Outcomes associated with these individual
components and their relative effectiveness have comprised the
focus of congressional and administrative inquiries. For
example, the Health Research Extension Act of 1985 requested a
study on "the effectiveness of Ache training grant] mechanism in
encouraging individuals to engage in health research and training
as a career." Of constant concern to agency staff is "what works
best" among or within NRSA components. Illustrative of this
interest are such questions as "Are M.D./Ph.D. programs or
postdoctoraI traineeships more efficacious in producing
physician/investigators?" and "Is predoctoral training that is
grounded in a particular discipline more successful for
increasing the number of researchers attacking alcohol-related
health problems than predoctoral training that incorporates
several disciplinary perspectives and methodologies?"
Cost-Effectiveness
For the most part, cost-effectiveness questions do not
constitute a high priority among major constituencies. The few
questions that emerged in the interviews pertained to identifying
ways to "best use the training buck," particularly if research
training funds continue to erode. Somewhat redated to this
concern are more global questions associated with the personal,
disciplinary, and social costs incurred from having an
insufficient amount of research monies available to support the
7
OCR for page 8
number of high-quality applications for individual investigator
awards from researchers who have been trained in NRSA programs.
Development and Maturation of Scientific Careers
More frequently, the questions of interest to key NRSA
constituencies are those that address research training,
scientific productivity, and scientific career development in
general rather than with regard to NRSA programs in particular.
These questions span a variety of topics, including the
relationship between research training and the quality of
research, the factors governing an individual's choice to pursue
a scientific career, and the resources required to most
successfully maintain a productive research career. Also
included in this category are those questions posed by agency
staff about how to best measure relevant outcomes of research
training (e.g., "active involvement in research" and "quality of
researched. Although such questions are important for guiding
and improving future evaluations of the NRSA program and can
indeed be addressed by wel1-designed studies, it must be kept in
mind that providing answers is neither the sole responsibility of
NRSA nor the evaluation efforts connected with this program.
Where Improvements in Evaluation Activities Are Needed?
In the previous section the major evaluation questions of
current interest to key constituencies were identified. Although
these questions covered all aspects of the program, the priority
areas centered around those issues associated with demand for the
program, levels of participation, characteristics of training and
recipients, and program outcomes.
To date, past evaluation efforts have provided a wealth of
data about National Research Service Awards, but many questions
basic to understanding how these programs operate and what
happens to awardees remain. Most individuals in charge of NRSA
programs often continue to find themselves operating in almost a
vacuum with regard to having sound, empirical data about how
awardees are selected, the characteristics of participants, and
the training environments and activities supported. Further,
knowledge about the subsequent performance of awardees currently
is confined to a limited set of indicators that vary considerably
across individual NRSA programs and that incompletely
characterize the intended outcomes. Improving this situation
(i.e., "filling the gaps") is what must drive the individual
items included in any portfolio of future evaluation activities.
Four major gaps exist in terms of having an adequate
knowledge base about NRSA programs:
8
OCR for page 9
Basic questions about program participation and
functioning, although of great interest to key
constituencies, have remained inadequately addressed.
2. Our understanding of program outcomes, let
alone program effects, is still limited.
3. Insufficient attention has been given to determining
what works best across and within program components.
Evaluation efforts have been sparse in many fields and
research problem areas.
The first three gaps focus on "points of slippage" between the
types of evaluation questions currently of interest to major
constituencies and those that have comprised the thrust of prior
evaluation efforts. An examination of the generic questions
addressed by the 16 evaluation studies/reports reviewed indicated
that 56 percent (N = 9) addressed program participation issues,
and 50 percent (N = 8) collected data on program characteristics
and operation. The overwhelming majority (94 percent, N = 15)
presented information on one or more outcomes for programs or
program elements, and 38 percent (N = 6) attempted to address in
some way the effectiveness of NRSA programs or distinct
components. None of the studies reviewed dealt with issues of
r
cost-effectiveness. ~
At first glance these percentages suggest that many of the
questions of interest to NRSA stakeholders (e.g.' program
outcomes) indeed have been addressed. However, it must be
remembered that within each of these generic evaluation issues
lie a variety of subquestions. For example, questions about
program operation encompass the nature of the trainee selection
process, characteristics of training, and the relationship of the
payback requirement to participation levels. As shall be seen,
the match between constituency priorities and evaluation efforts
5Given that studies could focus on more than one type of
evaluation question, these percentages do not sum to 100 percent.
Information on how the evaluations were chosen for review and on
the broad categories of questions addressed by each is presented
in Appendixes B and C, respectively.
In classifying these evaluations, distinguishing between
"outcome" studies and "effectiveness" studies has been in many
cases a matter of judgment. For the purposes of this paper, an
effectiveness study is one that incorporated either
methodological (e.g., matching) and/or statistical procedures so
as to control for selectivity bias. Those studies float attempted
to compare outcomes but did not include any real consideration of
selectivity bias were designated, rightly or wrongly, outcome
studies.
9
OCR for page 10
on these more specific questions is where the discrepancies
surface. (See Appendix A for a detailed enumeration of the
questions posed by constituencies and Appendix D for a listing of
those addressed by evaluation activities.)
GAPS IN UNDERSTANDING PROGRAM
PARTICIPATION AND OPERATION
Although past evaluation efforts have addressed aspects of
program participation and operation, several issues have escaped
careful examination, including some that are basic to
understanding any discrete program or intervention (e.g., program
implementation). This situation is partly an outgrowth of the
limited amount of resources that have been allotted for
evaluation activities. Consequently, some programs (e.g., those
sponsored by ADAMHA) have received little scrutiny. Another
problem has concerned the fact that when evaluations were
initiated, the short timelines imposed often dictated that the
focus be on collecting outcome data (no mean feat by itself),
with only secondary attention given to examining participation
levels or program characteristics. As a result, summary profiles
describing NRSA applicants, awardees, and program activities are
either nonexistent, sketchy, or idiosyncratic in terms of the
populations covered, the variables of interest, and the time
periods examined.
Needed Information on the Demand for the Program
Although development of better supply and demand indicators
for biomedical and behavioral science research personnel is
covered more thoroughly in the full committee report, one related
component deserves special mention in this paper. This concerns
the extent of our knowledge about the research training
enterprise as a whole (e.g., the total amount of funds, training
opportunities, and types of training provided by all sponsors).
Congress' motivation for having such information stems from its
desire to ascertain what its role should be in financing research
training and then to apply this understanding when making
decisions about the NRSA program. Similarly, the interest of NIH
and ADAMHA staff arises from their wish to better understand
their own agency's total involvement in research training,
particularly by mechanisms other than those covered under the
NRSA umbrella (e.g., research assistantships paid by grants to
individual investigators).
Answers to these questions also are requested in the charge
for evaluation specified in the authorizing and reauthorizing
legislation for the NRSA program: to "assess current MESA
programs] and other current training programs available for the
training of such personnel" (P.L. 100-607, Part F. Section 489~.
To carry out this charge, a map of the geography and topography
10
OCR for page 11
of non-NRSA funding sources and mechanisms for research training
must be constructed.
Previous committee reports (e.g., National Research Council,
1977, 1981) have attempted to survey this terrain, but this is no
easy task. Currently, the best sources of data are the annual
Survey of Earned Doctorates (SED) conducted by the National
Research Council and the National Science Foundation's (NSF)
Survey of Graduate Science Students and Postdoctorals. However,
each has certain limitations. For example, the NSF survey
requires institutions to indicate only one source of support for
a graduate student. Although respondents to the SED are
instructed to identify all sources of support and estimate the
percentage of support received from each source, their ability to
reconstruct these data accurately is unclear.
Understanding the variety of research training activities
sponsored by NIH and ADAMHA via non-NRSA mechanisms represents
one step toward mapping the terrain, however. Of particular
interest here is predoctoral and postdoctoral research training
paid by research grants to individual investigators. Available
data suggest that the use of this mechanism in supporting
research training is not infrequent; research assistantships paid
by federal and other grants were a source of predoctoral support
for 16 percent of the 1987 Ph.D. recipients in the life sciences-
-an almost equal percentage to that reported for NIH traineeships
(Coyle and Thurgood, 1989~.
Developing this capacity to obtain detailed training support
data for all Public Health Service agencies may be more feasible
than one might expect, given earlier and more recent efforts by
NIH and ADAMHA. Information on all paid and unpaid personnel
working on research grants was collected on a sample basis for
all PHS grants in 1963 and NIH grants in 1969; beginning in 1973,
these data were again requested of all NIH grantees in the NIH
Research Grants Manpower Survey. Unfortunately, this effort was
abolished in 1980 despite a reasonably favorable evaluation
(Williams, 1979).
ADAMHA staff members currently are investigating the
feasibility of implementing a similar system for their own
research grants and have already developed the system
specifications, along with conducting some preliminary pilot
tests (see Tjioe, 1989, for a description of this system).
Variables in this system include name, social security number,
role on the grant, type of position, sex, highest degrees), year
of degree, birth date, field (e.g., surgery), and research
discipline (e.g., brain damage) for all personnel connected with
awards made by ADAMHA institutes. If resources were available to
establish and maintain this data base for all PHS awards,
questions relating to the various ways in which research training
11
OCR for page 30
greater confidence which types of training strategies may work
better than others (e.g., multidisciplinary programs versus those
that focus on one discipline or specialty).
Efforts need to be initiated to rectify this situation.
A1 though retrospective studies of former trainees and fellows
have provided useful insight into training gaps and deficiencies
(e.g., Gentile et al., 1987), they are limited. Surveys of
recent Ph.D.s and M.D.s/Ph.D.s upon receipt of their doctorate
could provide some preliminary insight into the types of
variables that should be examined and how they can be measured
best (e.g., the phrasing of survey questions, if appropriate).
The same can be said for surveys of NRSA-supported postdoctoral
trainees and fellows who have just completed their postdoctoral
appointment.
The most preferred strategy is to collect data from
individuals at key intervals during the training process so as to
avoid the problems of selective memory and to obtain a more
complete picture of how and when training actually occurs. Such
efforts also could provide data on many other issues. Program
implementation could be examined (i.e., did individuals actually
receive the training set forth in the funded applications. If
information on trainee satisfaction were collected, the need for
modifications in training policies or regulations also might be
identified. Finally, if such efforts were extended to a sample
of predoctoral students or postdoctoral associates without
traineeships in the same academic program and possibly to
students in similar programs in departments without training
grants, we would begin to develop a better sense of the strength
and integrity of various NRSA "treatments" (e.g., whether NRSA
programs provide training experiences sufficiently different from
those received by students in the same program or in other
programs).
The manner in which trainees are selected is another
important area worthy of close scrutiny. We know, at least for
the biomedical sciences, that predoctoral trainees tend to
receive their training in top-ranked institutions/departments;
presumably, the training grant application review process is
selecting the best programs, and the departments in which these
programs are based are highly selective in terms of graduate
student admitting policies. However, we do not know whether
7These programs have been described as "contests" (Ross),
personal communication, May, 1989) in that the most promising
applicants are chosen to receive awards. It appears that the
main contestants are institutions; the top-ranked research
universities clearly are given preference. However, within the
departments that house the NRSA programs, it is not clear whether
the contest rules are the same, given the problems described in
30
OCR for page 31
those who actually receive traineeships are awarded them because
they are the best "match" in terms of the requirements of the
NRSA-supported program, because it is a convenient source of
money at a particular period of time; or for other reasons. The
same problem plagues applicants for postdoctoral appointments,
particularly in medical schools. Needless to say, the ability to
ultimately provide answers to questions seeking evidence on
effectiveness is hampered by our lack of knowledge about how
individuals are selected to receive NRSA support.
INCREASED ATTENTION TO A8SE88ING PROGRAM OUTCOMES
Those associated with NRSA programs, particularly at the
program administrative level, want to know what works best in
research training, either in terms of specific training models
(e.g., broad versus specialized training) or in terms of specific
populations (e.g., predoctoral support versus postdoctoral
support versus both types of support in producing physician/
scientists). Previous large-scale evaluation efforts that have
had to rely on retrospective assessment strategies have been
unable to confidently provide unequivocal answers to these
questions because of the probe ems associated with drawing solid
comparison groups. Given this substantial interest in program
outcomes, particularly comparative outcome data, concerted
attention should be devoted to examining the full range of
outcomes implied by the goals of the NRSA program (i.e., both
research and teaching activities as specified in program
announcements and payback requirements for NRSA programs) and to
pilot testing new data collection strategies. The issues and
variables warranting consideration are discussed by Fox (1983)
and Gee (1989) and should be guided by previous research on
scientific careers (e.g., Long et al., 1979; McGinnis et al.,
1982; McGinnis and Long, 1988; Stephan, 1986~. Not only would
efforts to accurately measure outcomes and test various
measurement strategies help to better examine scientific
productivity and career development, but they also would allow an
understanding of the marginal gain associated with using these
measures as contrasted to those previous approaches that have had
to rely on existing archival data bases.
Outcome assessments also could benefit from efforts geared
at improving the use of existing archival data bases. For
example, rather than using archival data bases to obtain a
"snapshot" at one point in time of research grant success, the
feasibility of using such data bases to track individuals over
time, similar to that performed by Biddle et al. (1989) in
this paper. This area requires further investigation, modeled on
the work by Carter et al. (1987) for Research Career Development
Awards.
31
OCR for page 32
examining the career accomplishments of Research Scientist
Development awardees, could be explored. Another possibility
would be to assess the feasibility of using relevant data from
other archival sources that could augment the information on
traditionally used measures (e.g., research grant success) or
provide data on outcomes that have received only minimal
attention (e.g., teaching others to be researchers). For
example, Yasumura (1986) found that recipients of the KO-6 awards
were more likely to receive PHS training grants--a relevant
outcome, based on the request made by Congress for obtaining
information on the number of awardees who engage in health
research or teaching as a career (P.L. 99-1258~.
In addition, exploring relationships between program
characteristics and outcomes also is needed. For example, recent
data on NHLBI awardees suggest that those M.D.s who were
postdoctoral trainees in programs designed to train both M.D.s
and Ph.D.s were more likely to be receive a NIH grant than those
who were postdoctoral trainees in programs only aimed at M.D.s
(C. Roth) personal communication, June, 1989~. More efforts of
this kind can guide subsequent outcome studies in terms of
identifying key programmatic variables that should be considered.
Another focus should be on designing outcome studies where
the entire "dosage" of training is measured. At the very least,
for outcome studies of ADAMHA, NIH, or other PHS support, this
information should be available from the TFF currently in
existence. However, given that NRSA research training is just
one part of funding for scientific personnel development,
measuring the length of training must take into account all types
of research training and development (e.g., short-term research
training, predoctoral and postdoctoral traineeships and
fellowships, and career development awards). If the previously
described efforts for developing and/or reinstituting data bases
on personnel working on PHS grants also are initiated, this would
provide additional information on dosage by getting data on
predoctoral or postdoctoral support financed by faculty research
grants from these agencies. The completeness and accuracy of
this information then could be checked by actually contacting the
individuals chosen for examining outcomes to determine what types
of training support they actually received--both from PHS and
other sponsors.
The final point is a call for evaluation efforts that
incorporate a more longitudinal perspective. To date, the
majority of the studies have been of the snapshot variety--
career accomplishments at one point in time. Future evaluation
activities ideally must track individuals from the time they
begin training (or apply for training opportunities) through the
completion of their training and during their scientific careers.
Even more limited efforts would benefit from following the Carter
et al. (1987) and Biddle et al. (1988) approach to measuring
32
OCR for page 33
outcomes (e.g., PHS research grant activity) at several points
after the individual has completed training. Only within a
longitudinal perspective can a specific achievement or pattern of
achievements be interpreted and yield information that can
illuminate our understanding of these programs and research
training in general.
INCREASING THE AVAILABILITY OF RE80URCES
FOR RESEARCH ON RESEARCH TRAINING
Developing better outcome strategies is at least partially
dependent on better understanding the nature of scientific
careers, what influences productivity, and other similar
relationships. Currently, funding for research on scientific
careers, resources, and so forth is not abundant. The NSF's
Division of Science Resources devotes some funds to supporting
studies in these areas (e.g., National Science Foundation, 1986~.
However, monies for this program are limited, and proposals often
must include all scientific and engineering fields rather than
those most relevant to NRSA programs or must focus on using
preexisting survey data collected by NSF (data that typically
includes, for example, only small numbers of NRSA-supported
individuals in various categories).
Research on scientific career development and enhancement
can go a long way toward addressing basic questions about
productivity, motivations for choosing a scientific career, the
factors that facilitate or inhibit research productivity, and so
on. In conjunction with these research programs, efforts need to
be targeted at developing new measures of scientific productivity
and quality of research and testing their feasibility. Work on
assessing the quality of training programs, apart from the
accomplishments of their graduates, is another component that
requires attention, particularly if future evaluation efforts
aimed at judging the quality of training grant recipients (i.e.,
programs/departments within institutions) are considered worthy
of exploration.
The list of research questions is endless, and reasonably
strong arguments could be made for choosing any particular
question or methodological strategy as the initial starting
point. The major point is that resources for these efforts
should be increased so that our ability to carry out evaluation
activities that can both meet the needs of major NRSA
constituencies and assist in the continual improvement of
training activities in all fields of health-related research can
be enhanced.
33
OCR for page 34
EXPLORING TRE FEA8IBILITY OF EVALUATTON DESIGN8
FOR ASSE8SING EFFECTIVENESS
Given our lack of knowledge about the nature of the
selection and training processes, improving the quality of future
designs certainly represents a technical challenge. Perhaps the
first step toward obtaining better estimates of the relative
effectiveness of different types of research training programs
and determining the training strategies or mechanisms that work
best is to identify where the use of high quality designs can be
implemented.
One way to begin this process is to focus initially on
exploring opportunities that may exist for assessing specific
components. For example, the MSTP appears to be a promising
candidate for consideration, given that training program
directors often vigorously recruit students. Thus, a pool of
applicants may be available to use as a comparison group. The
feasibility of "beating the bushes" to augment the size of the
applicant pool, of convincing directors to collect necessary data
on applicant characteristics, and of persuading them to adhere to
a standard selection process and criteria should be explored.
Even if random assignment is not possible (a likely event), the
use of such designs as regression-discontinuity is worth
examining. This latter type of design was used successfully in
Carter et al.'s (1987) evaluation of the Research Career
Development Award.
In addition, exploring the feasibility of implementing high
quality evaluation designs for the MST program is attractive,
given the level of interest in this component. For example, some
concern has been expressed over the available supply of
physician-researchers, and MD-Ph.D. training programs are being
established by other agencies to resolve this problem. At the
same time, however, the best way to produce physician-scientists
is a point of controversy. Further, questions have been raised
about the fact that the total expenditures per graduate for MST
programs have been increasing over the last decade and are now
significantly higher than for graduates supported by other
predoctoral training grants (National Research Council, 1983~.
Well-designed evaluations of the MST program, focusing on the
outcomes of its graduates and its effectiveness on certain
outcomes and as compared to other research training alternatives
for physician-scientists, would do much towards addressing these
issues.
The lower level of interest in effectiveness questions
expressed by constituencies, coupled with the difficulty in
carrying out high quality studies to address these questions,
does not, however, imply that exploring the feasibility of
implementing rigorous designs for estimating program effects
not important. The long-term objective for NRSA evaluation
34
IS
OCR for page 35
efforts should be to gain an understanding of what works in
training, which programs and program elements work better, and
how training should be assessed. Small-scale, pilot tests of
more rigorous approaches can be quite instructive in terms of
identifying where more better designs can be implemented and
ultimately yield an understanding of research training itself.
Many of the concerns being expressed today (e.g., future
shortages of trained scientists, issues of scientific misconduct,
and the lack of interdisciplinary research efforts in many major
problem areas) have their core issues surrounding human resources
and training. If we are successful in providing a better
understanding of NRSA research training and how it operates and
contributes to the development of outstanding researchers and
research mentors, one step toward addressing these issues will
have been taken.
REFERENCES
Alcohol, Drug Abuse, and Mental Health Administration (ADAMHA).
1989a. ADAMHA NRSA Research Training Tables FY 1988.
Rockvil~e, MD.
Alcohol, Drug Abuse, and Mental Health Administration. 1989b.
Addressing the Shortage of Young ADAMHA Principal
Investigators. (Available from Walter Schaffer, ADAMHA,
Parklawn Building, 5600 Fishers Lane, Rockvil~e, MD 20857.)
Barries, J. 1986. Projected supply and demand for Ph.D.
Biologists. Paper presented at the meeting of the American
Zoological Society, December 1986.
Bicke1, J. W., C. R. Sherman, J. Ferguson, L. Baker, and T. E.
Morgan. 1981. The Role of M.D.-Pl~.D. Training in
Increasing the Supply of Physician-Scientists. New England
Journal of Medicine 304:1265-1268.
Biddle, A. R., G. M. Carter, J. S. Uebersax, and J. D. Winkler.
1988. Research Careers of Recipients of the Research
Scientist and Research Scientist Development Award. Santa
Monica, CA: The RAND Corporation.
Carter, G. M., A. Robyn, and A. M. Singer. (1983~. The Supply
of Physician Researchers and Support for Research Training:
Part I of an Evaluation of the Hartford Fellowship Program
(N-2003-HF). Santa Monica, CA: The RAND Corporation.
Carter, G. M., J. D. Winkler, and A. K. Biddle. 1987. An
Evaluation of the NIH Research Career Development Award.
Santa Monica, CA: The RAND Corporation.
35
OCR for page 36
Clouet, D. H. 1986. The Career Achievements of Trainees and
Fellows Supported by the National Institute of Drug Abuse.
Rockville, MD: NIDA.
Coggeshall, P. E., and P. W. Brown. 1984. The Career
Achievements of NIH Predoctora~ Trainees and Fellows.
Washington, D.C.: National Academy Press.
Coyle, S. L., and Thurgood, D. H. 1989. Summary Report 1987:
Doctorate Recipients from United States Universities.
Washington, D.C.: National Academy Press.
Fox, M. F. 1983. Publication Productivity Among Scientists: A
Critical Review. Social Studies of Science 13:285-305.
Garrison, H. H., and P. W. Brown, 1986. Minority Access to
Research Careers: An Evaluation of the Honors Undergraduate
Research Training Program. Washington, D.C.: National
Academy Press.
Gee, H. 1989. Productivity. Paper commissioned by the
Biomedical and Behavioral Research Personnel Committee,
National Academy of Sciences, Washington, D.C.
Gentile, N. 0., G. S. Levey, P. Jolly, and T. H. Dial. 1987.
Post-doctoral Research Training of Full-Time Faculty in
Departments of Medicine. Washington, D.C.: Association of
Professors of Medicine and Association of American Medical
Colleges.
Gerstein, D. R., Luce, R. D., N. J. Smelser, and S. Sperlich
(eds.~. 1988. The Behavioral and Social Sciences:
Achievements and Opportunities. Washington, D.C.: National
Academy Press.
Levey, G. S., C. R. Sherman, N. O. Gentile, L. J. Hough, T. H.
Dial, and P. Jolly. 1988. Postdoctoral research training
of full-time faculty in academic departments of medicine.
Annals of Internal Medicine 109:414-418.
Long, J. S., P. D. Allison, and R. McGinnis. 1979. Entrance
into the academic career. American Sociological Review,
44:816-830.
Look, M. 1989. Final Report on the Proposed Data Matrix for the
Biomedical and Behavioral Research Personnel Committee.
Bethesda, MD: Quantum Research Corporation.
36
OCR for page 37
Martin, M. R. 1986. The Results of a Manual Look-up of 300
References from Non-foreign Medical School Faculty
Clinicians from the B/I/D Journal Set and an Analysis of
Report by the AAMC on the Number and Character of Physical
Researchers. (Available from Division of Program Analysis,
National Institutes of Health, 9000 Rockville Pike,
Bethesda, MD 20092.)
McGinnis, R., and J. S. Long. 1988. Entry into academia:
Effects of stratification, geography, and ecology. In D. W.
Brenneman and T. I. K. Youn teds.) Academic Labor Markets
and Careers, New York: Falmer Press.
McGinnis, R., P. D. Allison, and J. S. Long. 1982. Postdoctoral
training in bioscience: Allocation and outcomes. Social
Forces 60:701-723.
National Institute of Dental Research (NIDR) Ad Hoc Consultant
Panel. 1988. The National Research Service Award Program:
Report of the NIDR Ad Hoc Consultant Panel. Bethesda, MD:
NIDR.
National Institutes of Health (NIH). 1986. Effects of the
National Research Service Award Program on Biomedical
Research and Teaching Careers. Washington, D.C.: NIH.
National Institutes of Health. 1987. NIH Data Book 1987.
Washington, D.C.
National Institutes of Health. 1988. NHLBI Fact Book Fiscal
Year 1988. Washington, D.C.: NTH.
National Research Council (NRC). 1976. Research Training and
Career Patterns of Bioscientists: The Training Programs of
the National Institutes of Health. Washington, D.C.:
National Academy of Sciences.
National Research Counci1. 1977. Personnel Needs and Training
for Biomedical and Behavioral Research. Washington, D.C.:
National Academy of Sciences.
National Research Council. 1978.
for Biomedical and Behavioral
National Academy of Sciences.
National Research Council. 1979.
for Biomedical and Behavioral
National Academy of Sciences.
National Research Council. 19Bla.
for Biomedical and Behavioral
National Academy of Sciences.
37
Personnel Needs and Training
Research. Washington, D.C.:
Personnel Needs and Training
Research. Washington, D.C.:
Personnel Needs and Training
Research. Washington, D.C.:
OCR for page 38
National Research Council. 198Ib. Postdoctoral Appointments and
Disappointments. Washington, D.C.: NRC.
National Research Council. 1983. Personnel Needs and Training
for Biomedical and Behavioral Research. Washington, D.C.:
National Academy Press.
National Research Council. 1985. Personnel Needs and Training
for Biomedical and Behavioral Research. Washington, D.C.:
National Academy Press.
National Research Council. 1989. Biomedical and Behavioral
Research Scientists: Their Training and Supply.
Washington, D.C.: NRC.
National Science Foundation. 1988.
(NSF 88-336~. Washington, D.C.
Office.
Project Summaries: FY 1988
: U.S. Government Printing
Office of Technology Assessment. 1985. Demographic Trends and
the Scientific and Engineering Work Force (OTA-TM-SET-35~.
Washington, D.C.: U.S. Government Printing Office.
Porter, B. F. 1979. Transition: A follow-up study of 1973
postdoctorals. In the American Physical Society, The
Transition in Physics Doctoral Employment 1960-1990. New
York, NY: APS.
Primos, M. E. 1989a. Minority Access to Research Careers
Program: Briefing on Program Activities, FY 1980 to 1986.
(Available from M. Primos, Minority Research Resources
Branch, NIMH, Parklawn Building, 5600 Fishers Lane,
Rockville, MD 20857.)
Primos, M. E. 1989b. Minority Fellowship Program: Summary of
Program Activities, FY 1973 to FY 1987. (Available from M.
Primos, Minority Research Resources Branch, NIMH, Parklawn
Building, 5600 Fishers Lane, Rockville, MD 20857.)
Schneider, S. 1980. Positions of Psychologists Trained for
Research. American Psychologist 35:861-866.
Sherman, C. R. 1983a. Notes on the NIH Role in Support of
Postdoctoral Research Training of Two Groups of Physicians.
(Available from Charles Sherman, NIH, 9000 Rockville Pike,
Bethesda, MD 20092.) -
Sherman, C. R. 1983b. Training and Manpower Development.
Presentation at the Meeting of the Advisory Committee to the
Director, NIH, Bethesda, MD.
38
OCR for page 39
Sherman, C. R. 1984. Perspectives on Short-Term Training in
Schools of the Health Professions. (Available from Charles
Sherman, NTH, 9000 Rockville Pike, Bethesda, MD 20092.)
Sherman, C. R. 1989. The NIH Role in the Training of Individual
Physician Faculty: A Supplementary Analysis. (Available
from Charles Sherman, Office of Science Policy and
Legislation, NIH, 9000 Rockvi1le Pike, Bethesda, MD 20092.)
Sherman, C. R., H. P. Jolly, T. E. Morgan, E. J. Higgins, D.
Hollander, T. Bry1l, and E. R. Sevilla III. 1981. On the
Status of Medical School Faculty and Clinical Research
Manpower 1968-1990. Washington, D.C.: National Institutes
of Health.
Snyder, J. 1988. Early Career Achievements of National Science
Foundation Graduate Fellows, 1967-1976. Washington, D.C.:
National Research Counci1.
Stephan, P. 1986. Age publishing patterns in science:
Estimation and measurement issues. In Anthony F. J. van
Raan fed.), A Handbook of Quantitative Studies of Science
and Technology. Amsterdam, Holland: Elsevier Science
Publishing Division of North Holland.
Task Force on Minority Research Training. 1986. Minority
Research Training Programs at the National Institute of
Mental Health: Descriptions and recommendations.
Rockville, MD: National Institute of Mental Health.
Tjioe, P. 1989. Researchers' Tracking System (RTS) User's
Guide. Laurel, MD: General Sciences Corporation.
Velletri, P. A., C. R. Sherman, and G. Bowden. 1985. A
Comparison of the Career Achievements of NIH Predoctoral
Trainees and Fellows. Washington, D.C.: National
Institutes of Health.
Vetter, B. M. 1989. Recruiting Doctoral Scientists and
Engineers Today and Tomorrow (Occasional Paper 89-2~.
Washington, D.C.: Commission on Professionals in Science
and Technology.
Williams, P. C. 1979. Evaluation of the Annual Manpower Survey
of NIH Research Grants. Sumner, MD: Cooper-Wi1liams
Associates.
Yasumura, S. 1984. The Research Career Award (R06~: A 20-Year
Perspective and an Analysis of Research Productivity.
Washington, D.C.: National Institutes of Health.
39
OCR for page 40
Zumeta, W. 1985. Extending the Educational Ladder: The
Changing Quality and Value of Postdoctoral Study.
Lexington, Mass.: D.C. Heath.
40
Representative terms from entire chapter:
percent versus
