|
Items in cart [0] |
Questions? Call 888-624-8373 |
| Copyright © 2009. National Academy of Sciences. All rights reserved. Terms of Use and Privacy Statement |
Below are the first 10 and last 10 pages of uncorrected machine-read text (when available) of this chapter, followed by the top 30 algorithmically extracted key phrases from the chapter as a whole.
Intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text on the opening pages of each chapter.
Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.
Do not use for reproduction, copying, pasting, or reading; exclusively for search engines.
OCR for page 25
4
Quantitative Measures
This chapter proposes and describes the quantitative
measures relevant to the assessment of research-doctorate
programs. These measures are valuable because they
· Permit comparisons across programs,
· Allow analyses of the correlates of the qualitative
reputational measure,
· Provide potential students with a variety of dimensions
along which to compare program characteristics, and
· Are easily updateable so that, even if assessing reputation
is an expensive and time-intensive process, updated quanti-
tative measures will allow current comparisons of programs.
Of course, quantitative measures can be subject to distor-
tion just as reputational measures can be. An example would
be a high citation count generated by a faulty result, but these
distortions are different from and may be more easily iden-
tified and corrected than those involving reputational
measures. Each quantitative measure reflects a dimension
of the quality of a program, while reputational measures are
more holistic and reflect the weighting of a variety of factors
depending on rater preferences.
The Panel on Quantitative Measures recommended to the
Committee several new data-collection approaches to
address concerns about the 1995 Study. Evidence from
individuals and organizations that corresponded with the
Committee and the reactions to the previous study both show
that the proposed study needs to provide information to
potential students concerning the credentials required for
admission to programs and the context within which gradu-
ate education occurs at each institution. It is important to
present evidence on educational conditions for students as
well as data on faculty quality. Data on post-Ph.D. plans are
collected by the National Science Foundation and, although
inadequate for those biological sciences in which post-
doctoral study is expected to follow the receipt of a degree,
they do differentiate among programs in other fields and
25
should be reported in this context. It is also important to
collect data to provide a quantitative basis for the assessment
of scholarly work in the graduate programs.
With these purposes in mind, the Panel focused on quan-
titative data that could be obtained from four different groups
of respondents in universities that are involved in doctoral
education:
University-wide. These data reflect resources avail-
able to, and characteristics of, doctoral education at the
university level. Examples include: library resources,
health care, child care, on-campus housing, laboratory
space (by program), and interdisciplinary centers.
Program-specific. These data describe the characteris-
tics of program faculty and students. Examples include:
characteristics of students offered admission, informa-
tion on program selectivity, support available to
students, completion rates, time to degree, and demo-
graphic characteristics of faculty.
Faculty-related. These data cover the disciplinary sub-
field, doctoral program connections, Ph.D. institution,
and prior employment for each faculty member as well
as tenure status and rank.
Currently enrolled students. These data cover pro-
fessional development, career plans and guidance,
research productivity, research infrastructure, and
demographic characteristics for students who have been
admitted to candidacy in selected fields.
In addition to these data, which would be collected
through surveys, data on research funding, citations, publi-
cations, and awards would be gathered from awarding
agencies and the Institute for Scientific Information (ISI), as
was done in the 1995 Study.
OCR for page 26
26
ASSESSING RESEARCH-DOCTORATE PROGRAMS
The mechanics of collecting these data have been greatly lute for Scientific Information (ISI) and the NSF Doctorate
simplified since 1993 by the development of questionnaires Records File (DRF). For the humanities, it gathered data on
and datasets that can be made available on the Web as well honors and awards from the granting organizations. Most of
as software that permits easy analysis of large datasets. This the institutional coordinators did a conscientious and
technology makes it possible to expand the pool of potential thorough job, but the Committee believes that it would be
raters of doctoral programs.
_
helpful to pursue a more complex data-collection strategy
that would include a program data collector (usually the
director of graduate studies) in addition to the key institu-
tional coordinator, a questionnaire to faculty, and question-
naires to students in selected programs. This approach was
tested with the help of the pilot institutions. The institutional
coordinator sent the NRC e-mail addresses of respondents
for each program. The NRC then provided the respondent a
password and the Web address of the program questionnaire.
A similar procedure was followed for faculty whose names
were provided by the program respondents. Copies of the
questionnaires may be found in Appendix D.
In 1995, programs were asked for the number of faculty
engaged in doctoral education and the percentage of faculty
who were full professors. They were also asked for the
numbers of Ph.D.s granted in the previous 3 years, their
graduate enrollment both full-time and part-time, and the
percentage of females in their total enrollment. Data on
doctoral recipients, such as time to degree and demographic
characteristics, came entirely from the DRF and represented
only those who had completed their degrees.
The Committee believed that more informative data could
be collected directly from the program respondents. Follow-
ing the 1995 Study, a number of questions had been raised
about the DRF data on time to degree. More generally, the
Committee observed that data on graduates alone gave a
possibly biased picture of the composition and funding of
students enrolled in the program. The program question-
naire contains questions that are directly relevant to these
concerns.
In the area of faculty characteristics, the program ques-
tionnaire requests the name, e-mail address, rank, tenure
status, and demographic characteristics (gender, race/
ethnicity, and citizenship status) of each faculty member
associated with the program. Student data requested include
characteristics of students offered admission, information on
program selectivity, support available to students, comple-
tion rates, and time to degree. It also asks whether the
program requires a master's degree prior to admission to the
doctoral program, since this is a crucial consideration affect-
ing the measurement of time to degree. The questionnaire
also permits construction of a detailed profile of the percent-
age of students receiving financial aid and the nature of that
aid. Finally, the questionnaire asks a variety of questions
related to program support of doctoral education: whether
student teaching is mentored, whether students are provided
with their own workspaces, whether professional develop-
ment is encouraged through travel grants, and whether
excellence in the mentoring of graduate students by faculty
is rewarded. These are all "yes/no" questions that impose
little respondent burden.
MEASURABLE CHARACTERISTICS OF DOCTORAL
PROGRAMS
The 1995 Study presented data on 17 characteristics of
doctoral programs and their students beyond reputational
measures. These are shown in Table 4-1. Although these
measures are interesting and useful, it is now possible to
gather data that will paint a far more nuanced picture of doc-
toral programs. Indicators of what data would be especially
useful have been pointed out in a number of recent discus-
sions and surveys of doctoral education.
Institutional Variables
In the 1995 Study, data were presented on size, type of
control, level of research and development funding, size of
the graduate school, and library characteristics (total volumes
and serials). These variables paint a general picture of the
environment in which a doctoral program exists. Does it
reside in a big research university? Does the graduate school
loom large in its overall educational mission? The Com-
mittee added to these measures that were specifically related
to doctoral education. Does the institution contribute to
health care for doctoral students and their families? Does it
provide graduate student housing? Are day care facilities
provided on campus? All these variables are relevant to the
quality of life of the doctoral student, who is often married
and subsisting on a limited stipend.
The Committee took an especially hard look at the quan-
titative measures of library resources. The number of books
and serials is not an adequate measure in the electronic age.
Many universities participate in library consortia and digital
material is a growing portion of their acquisitions. The Com-
mittee revised the library measures by asking for budget data
on print serials, electronic serials, and other electronic media
as well as for the size of library staff.
An addition to the institutional data collection effort is
the question about laboratory space. Although this is a pro-
gram characteristic, information about laboratory space is
provided to the National Science Foundation and to govern-
ment auditors at the institutional level. This is a measure of
considerable interest for the laboratory sciences and engi-
neering, and the Committee agreed that it should be collected
as a possible correlate of quality.
Program Characteristics
The 1995 Study included data about faculty, students, and
graduates gathered through institutional coordinators, Insti-
OCR for page 27
QUANTITATIVE MEASURES
TABLE 4-1 Data Recommended for Inclusion in the Next Assessment of Research-Doctorate Programs.
Bolded Elements Were Not Collected for the 1995 Study.
Institutional Characteristics
27
Variable
Description
Year of First Ph.D. The year in which the Doctorate Records File (DRF) first recorded a Ph.D. Since the DRF information dates back only
to 1920, institutions awarding Ph.D.s prior to 1920 were identified by other sources, such as university catalogs or direct
inquiries to the institutions. Because of historic limitations to this fife, this variable should be considered a general
indicator not an institutional record.
Control Type of "Institutional Control": PR=private institution; PU=public institution.
Enrollment Total Totalfull- and part-time students enrolled in Fall 2003 in courses creditable toward a diploma.
Graduate Full- and part-time students in Fall 2003 in nonprofessional programs seeking a graduate degree.
Total R&D Average annual expenditure for research and development at the institution for the previous 5 years in constant dollars.
Federal R&D Average annual federal expenditure for research and development at the institution for the previous 5 years in constant
dollars.
Professional Library Staff Number of library staff (FTE).
Total Library Expenditures Total library expenditure offends from regular institutional budgets and other sources, such as research grants, special
projects, gifts, endowments, and fees for services for the previous academic year.
Library Expenditures: Total library expenditure offends for book acquisition from regular institutional budgets and other sources, such as
Acquisition of Books research grants, special projects, gifts, endowments, and fees for services for the previous academic year.
Library Expenditures: Total library expenditure offends for print serials from regular institutional budgets and other sources, such as research
Print Serials grants, special projects, gifts, endowments, andfees for services for the previous academic year.
Library Expenditures: Total library expenditure offends for serials in electronic media from regular institutional budgets and other sources,
Electronic Serials such as research grants, special projects, gifts, endowments, andfees for services for the previous academic year.
Library Expenditures: Total library expenditure offends for microprint and electronic databases from regular institutional budgets and other
Microprint and Electronic sources, such as research grants, special projects, gifts, endowments, and fees for services for the previous academic
Databases year.
Health Care Insurance Whether health care insurance is available to enrolled doctoral students under an institutional plan. Whether andfor
whom (TAB, RAs, all) percentage of premium cost is covered.
Childcare Facilities Available to graduate students? Subsidized? Listings made available?
University-Subsidized Available to doctoral students?
Student Housing
University Awards/ Teaching or research by doctoral students? Mentoring of doctoral students by faculty?
Recognition
University-Level Support Available for travel to professional meetings? For research off-campus? Available to help students improve their
for Doctoral Students teaching skills? Placement assistance? Available for travel to professional meetings? Available to help students improve
their teaching skills? Placement assistance?
Doctoral Program Characteristics
Variable
Description
Total Students The number offull- and part-time graduate students enrolled in the Fall of the survey year.
Student Characteristics Numbers, full-time and part-time status, gender, race/ethnicity, citizenship status.
Ph.D. Production Numbers of Ph.D.s awarded in each of the previous 5 years.
Program Median Time Year by which half the entering cohort had completed, averaged overfive cohorts. For programs for which half never
to Degree complete, the percentage completing within 7 years.
Master's Required Whether the program requires completion of a master's degree prior to admission.
Financial Support Proportion of first-year students who receive full support. Number of years for which students may expect full financial
support (including Fellowships, RAships, and TAships). Whether summer support is available. Percent receiving
externally funded support. Percent receiving university-funded support.
continues
OCR for page 28
28
ASSESSING RESEARCH-DOCTORATE PROGRAMS
TABLE 4-1 Data Recommended for Inclusion in the Next Assessment of Research-Doctorate Programs.
Bold elements were not collected for the 1995 Study.
Doctoral Program Charactenstics (continued)
Variable
Description
Teaching Assistant Average number of courses per term that a TA is expected to teach.
Work Load
Individually Assigned For allstudents? ForTAs? ForRAs?
Student Workspace
GRE Scores Whether GRE is required for admission. Average and minimum verbal and quantitative scores.
Number of Acceptances Number of applicants who are accepted into the program for the past 3 years.
Number Who Enroll Number of accepted students who enroll in the program for the past 3 years.
Awards for Teaching/ Whether the program gives awards for graduate student teaching/research or for faculty mentoring of doctoral students.
Research, Faculty Mentoring
Program Support for Whether the program provides some form of travel support for doctoral students to attend professional meetings.
Student Travel to Meetings
Teaching Skill Support Whether there is an organized program to help doctoral students improve their teaching skills.
Laboratory Space For each doctoral program in science and engineering, the number of net available square feet of laboratory space in the
previous academic year. Also, an indication of whether this space is shared with other doctoral programs.
Related Interdisciplinary Listing of centers in which program doctoral students may carry out research.
Centers
Data Collection on Student Whether the program collects such data. Whether it makes data available to potential students.
Outcomes
Competitor Programs List of up to five programs with which the program normally competes for graduate students.
Post-doctoral Plans by Obtainedfrom the NSF Survey of Doctoral Recipients, the percentage of Ph.D.s over the previous 5 years by type of
Type of Employer employer.
Program Faculty
Variable
Description
Total Faculty Total number offaculty participating in the program (teaching or supervising dissertations).
Faculty Characteristics Rank, tenure status, gender, race/ethnicity, citizenship status, years since Ph.D., time at this institution.
Percent Full Percentage offull professors participating in the program.
Percent Support Percentage of program faculty with research support (1999-2003).
Percent Engaged in More Percentage of program faculty who teach graduate courses or supervise dissertations in other programs.
than One Doctoral Program
Number of Awardsa Total number of awards and honors attributed to program faculty for the period 1999-2003.
Awarded Facultya Percentage of program faculty that have received at least one honor or award for the period 1999-2003.
Percent Publicationsb Percentage of program faculty (totalfaculty) publishing in the period 1999-2003.
Publications/Facultyb The ratio of the total number of program publications in the period 1999-2003 to the number of program faculty (total
faculty).
Citations/Facultyb The ratio of the total number of program citations in the period 1999-2003 to the number of program faculty (total
faculty).
Faculty Characteristics Listing by name and e-mail address offaculty who are affiliated with the program through either teaching or supervision
of dissertations. Rank, tenure status, whether US citizen/permanent resident, gender, race/ethnicity, date and year of
highest degree.
aFor Arts and Humanities.
bFor the fields in Engineering and the Sciences. Arts and Humanities may be included, depending on whether adequate book and monograph citations and
publication sources exist.
OCR for page 29
QUANTITATIVE MEASURES
Faculty Characteristics
In the 1995 Study, a brief faculty questionnaire was
administered to the raters who produced the reputational
rankings. These raters were drawn from a sample of faculty
nominated by their institutional coordinators. The sample
size reflected the number of programs in each field. The
brief questionnaire asked raters the year, institution, and date
of their highest degree as well as their current field of
specialization. The Committee believes that the faculty
questionnaire should be modified to collect certain other
data. For example, the university origins of current faculty
are a direct measure of which graduate programs are training
Ph.D.s who become faculty at research universities. Data on
date of degree would also permit a comparison of origins of
recently hired faculty as compared to faculty hired, for
example, more than 20 years ago. Although subfield data
were collected for the 1995 Study, they were not used. They
could be useful in improving program descriptions for
potential graduate students and for assuring that specialist
programs are rated by knowledgeable peers in the same
specialty.
The Committee also believes that additional questions
asked of faculty could permit a richer description of
interdisciplinarity. For example, faculty could list all pro-
grams in which they have participated, either by teaching or
serving on dissertation committees. Many faculty would be
listed as members of more than one graduate program, and
for the purposes of the reputational survey, the Committee
recommends that they be listed as program faculty for all
programs with which they are associated. To avoid the pos-
sibility of double counting the output of productive faculty,
objective measures should be attributed pro rata among the
various programs in which they are listed. The decision as
to how to prorate an effort should be made by the faculty
member with guidance that they should try to describe how
time devoted to doctoral education (teaching and student
mentoring) has been allocated among the programs for the
past 3-year period.
The Committee was concerned that programs might want
to associate a well-known faculty member with as many pro-
grams as possible in order to boost its rating, even if he or
she were not involved with the program. Allocation of pub-
lications should serve to discourage this behavior.
Student Characteristics and Views
Student observations have not been a part of past assess-
ments of research-doctorate programs. Past studies have
included data about demographic characteristics and about
sources of financial support of Ph.D. recipients drawn from
the DRF and about graduate student enrollment collected
from the doctoral institutions. Another student measure was
"educational effectiveness of the doctoral program," and for
reasons discussed in Chapter 6, the Committee is recom-
mending the elimination of this measure. The approach for
29
measuring student processes and outcomes is discussed in
Chapter 5.
PILOT TRIAL FINDINGS
The pilot trials were conducted over a 3-month period.
The most important finding was that 3 months was barely
sufficient for dealing with the study questionnaires. The full
study should probably allow at least 4-6 months for data
submission. The answers to many of the questions are pre-
pared for other data collection efforts, but additional time is
needed to customize answers to fit the taxonomy and to
permit time for follow-up with nonrespondents.
All institutions carried out the trial through a single point
of contact for the campus. This single point of contact
worked with institutional research offices and program con-
tacts to answer questions as well as interacted with NRC
staff to assure that data definitions were uniform.
Electronic data collection worked well for institutions,
programs, and faculty. We learned that it was better not to
provide a hard copy alternative (as contrasted to Web
response), since hard copy data simply had to be re-entered
in databases once it was received by the NRC. All the pilot
institutions store and access institutional and program data
electronically. E-mail is the standard mode of communica-
tion with faculty and the rates of faculty response (60 per-
cent) were high for a one-wave administration.
The Committee also learned that more precise definitions
are needed to guide respondents. For example, when asking
for data about "first-year doctoral students" a distinction may
be needed about whether the students have a master's in the
field. Care needs to be taken not to include terminal master's
students, and precise definitions of "full-time" and "part-
time" should be included.
The Committee learned the following from the question-
na~re responses:
Institutional Questionnaire.
· Library expenditures: Not all institutions separate
e-media expenditures from print expenditures.
· Space: The questionnaire needs to provide guidance
about how to allocate shared space. Answers to space ques-
tions also depend on how well the institution's programs fit
the taxonomy. If the fit is poor, the allocation of space is
arbitrary.
· Graduate student awards and support are more appro-
priately queried at the program level.
Program questionnaire.
· Programs had difficulty filling out the inception cohort
matrix but believed they could have done it if they had had
more lead time.
· Programs knew who their competitors were for doc-
toral students.
OCR for page 30
30
· Programs that required GREs knew the averages and
minima. For programs that do not require GREs, it would
be helpful to ask what percentage of applicants submit GRE
scores as well as report averages and minima only for those
programs that are above a certain level (e.g., 80 percent).
· Requests for faculty lists and faculty data should be
separate from requests for other program data.
Faculty questionnaire.
· E-mail notifications must have a sufficiently informa-
tive subject heading so that they are not mistaken for spam.
· Questionnaires should contain a due date.
· Faculty associated with more than one program should
be asked to fill out only one questionnaire. The NRC needs
to develop procedures to duplicate information for the other
programs with which a faculty member is associated.
· Some faculty identified their program by a name other
than that of the program that submitted their name. A proce-
dure must be developed to resolve this problem.
Each pilot institution was asked to provide comments on
the questionnaires. These comments, some of which are
reported above, will be used as background material for the
committee that conducts the full study. Draft questionnaires
ASSESSING RESEARCH-DOCTORATE PROGRAMS
for the full study should be reviewed by a number of institu-
tional researchers from a diverse set of institutions as well as
by survey researchers.
Data Collectecl from Other Sources
The Committee recommends that most of the quantitative
data presented in the 1995 Study from other sources be col-
lected again. These include: publication and citation data
from ISI, data on research grants from government agencies
and large private foundations, data on books from the Arts
and Humanities Citation Index, and data on awards and
honors from a large set of foundations and professional
societies. Student data from the Doctorate Record File
should be considered for inclusion but checked for inconsis-
tencies against institutional and program records. In the case
of inconsistencies, a validation process should be designed.
RECOMMENDATIONS
The Committee recommends that the data listed in Bold
type in Table 4-1 be added to the quantitative measures that
were collected for the 1995 Study.
Representative terms from entire chapter:
program faculty
