Below are the first 10 and last 10 pages of uncorrected machine-read text (when available) of this chapter, followed by the top 30 algorithmically extracted key phrases from the chapter as a whole.
Intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text on the opening pages of each chapter.
Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.
Do not use for reproduction, copying, pasting, or reading; exclusively for search engines.
OCR for page 11
3
Ratings in Specific Dimensions:
The Dimensional Measures
The dimensional measures are provided to assure that measures of a broad range of
characteristics of doctoral programs are available. They are divided into three categories:
(1) research activity, (2) student support and outcomes, and (3) diversity of the academic
environment. Each of the dimensional measures begins with the measures relating to one
dimension of doctoral program performance, applies the weights from the faculty survey
about what program characteristics contribute to quality, and then constructs a range17 of
rankings for each program based on this dimension of the data, taking into account variability
in the data and in the choice of raters. They are dimensional in the sense that they provide
more focused measures than the overall range of rankings, but they are central to the
calculation of this range.
Some specifics about the calculation of these measures follow.
• How the weights are obtained—As part of the NRC faculty questionnaire, we asked
faculty to indicate the relative importance of different characteristics of doctoral programs;
this was done through the multipart question that makes up Section G of the faculty
questionnaire (see Appendix B). Faculty were questioned about faculty quality, student
characteristics, and program characteristics. First they were asked to indicate up to four
characteristics in each category that they thought were important to program quality. Each
characteristic that was listed received an initial score of 1. These preferences were then
narrowed by asking the faculty members to identify a maximum of two characteristics in
each category that they thought were most important. These characteristics each received a
score of 2. A final question asked faculty members to indicate the relative importance of each
category by assigning category weights whose values summed to 100. For each individual
faculty member, the weight for a variable was calculated as the sum of the “votes” that it
received times the importance assigned to the category that contained it. The weight for a
variable in a discipline was the average weight taken across all faculty members in it. We
17
When we use the term “range,” we are referring to the inter-quartile range. This is the range that contains half of
the observations or estimates of the quantity of interest.
11
PREPUBLICATION COPY—UNEDITED PROOFS
OCR for page 12
took into account variability in raters’ opinions and uncertainties due to missing data and the
fact that some measures were sampled at one point in time.18 Approximately 86 percent of
the faculty responded. Their responses permitted calculation of the set of “direct” weights.
Although there was some variation in the faculty responses, they were generally in agreement
that publications and citations were the most important factors in program quality. Every
variable, however, received some weight19. These weights were used to construct the
dimensional measures. The average weights for programs in each broad field are shown in
Appendix F, and an example of ranges of rankings for programs in economics is shown in
Appendix G.
• Research activity—This dimensional measure relates to various ways to gauge the
contribution of research: publications, citations (except for the humanities), the percent of the
faculty holding research grants, and recognition of scholarship as evidenced by honors and
awards. Specifically, the components of the research activity dimensional measure are:
average publications per allocated faculty member,20 average citations per publication,
percent of core and new doctoral faculty respondents holding grants, and awards per
allocated faculty member. Publishing patterns and the availability of research funding and
awards for scholarship vary by field, but the weight placed on publications per faculty
member is remarkably consistent—about 30 percent—across fields. Research activity is the
dimensional measure that most closely tracks the overall measure of program quality,
because in all fields, both the direct measure—based on abstract faculty preferences—and the
regression-based measure put high weight on these measures.
• Student support and outcomes—This measure combines data on the percent of
students fully funded in the first year, the percent of students completing their degrees in a
given time period, time to degree, placement in academic positions (including academic
postdoctoral positions), and whether a program collects data about the employment outcomes
for its students. We found that faculty typically placed a larger weight on student support and
completion rates than on median time to degree, academic placement, or whether a program
18
There is some uncertainty in the values of the program variable values themselves. Some of the 20 program
variables used to calculate the ratings also vary or have an error associated with their values due to year-to-year
fluctuations. Data for five of the variables (publications per faculty, citations per publications, GRE scores, Ph.D.
completion, and number of Ph.D.’s) were collected over time, and averages over a number of years were used as the
values of these program variables. If a different time period had been used, the values would have been different. To
express this type of uncertainty, a relative error term, ejk, was associated with each variable value. For details, see
Appendix A.
19
All “direct” weights are used in the calculation of the Dimensional Measures. This differs from the case of the
Overall Rating of Program Quality, in which some coefficients might be set to zero if the result of combining the
direct and the regression-based weights was not statistically significantly different from zero.
20
Because many faculty members supervise dissertations in more than one program, faculty members were
allocated across these programs so that the total, taken across all programs, equaled one or less (in the case in which
the faculty member was in a professional school).
12
PREPUBLICATION COPY—UNEDITED PROOFS
OCR for page 13
follows the employment outcomes of it students.21 There is surprising uniformity across
broad fields on the weights, which are shown in Appendix F.
• Diversity of the academic environment—The diversity measures did not appear as
major factors in determining the overall perceived quality of programs. Taken separately,
there are definite patterns for variables that faculty thought were more important, and these
vary by field. The measures that are included in this dimensional measure are: the percent of
faculty and percent of students who are from underrepresented minority groups, the percent
of faculty and the percent of students who are female, and the percent of students who are
international (that is, in the United States on a temporary visa). In terms of field differences,
most fields place the highest weight on the percentage of students from underrepresented
minority groups. In the health sciences, social sciences, and humanities, relatively high
weights are also placed on the percentage of faculty who are underrepresented minorities.
The percentage of international students was not highly weighted, except for the physical
sciences. These weights, by broad field, are shown in Appendix F.
What is interesting about the dimensional ratings is that, with the exception of the
research activity measure, they produce program rankings that are quite different from the
overall ratings. This can be seen in the table in Appendix G. Excellence in doctoral programs is
not uni-dimensional. Some students may prefer a program where they can be assured of steady
funding and a short time to degree, even if it is not a program that is perceived as stellar in terms
of the productivity of its faculty. Similarly, a program that is more diverse may be preferable to
many students, although diversity bears only a tenuous relation with the usual measures of
scholarly productivity. Users of the assessment should be aware of these different dimensions,
because each presents the characteristics of an individual doctoral program from a different
perspective.
21
Ideally, we would have used a measure such as employment in one’s field 5 years after receipt of Ph.D., but many
programs did not collect such data. The committee hoped that including this measure would encourage more
programs to pay attention to post-degree outcomes for their graduates
13
PREPUBLICATION COPY—UNEDITED PROOFS
