Below are the first 10 and last 10 pages of uncorrected machine-read text (when available) of this chapter, followed by the top 30 algorithmically extracted key phrases from the chapter as a whole.
Intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text on the opening pages of each chapter.
Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.
Do not use for reproduction, copying, pasting, or reading; exclusively for search engines.
OCR for page 19
5
An Example
Shortly before the assessment is released, each institutional coordinator will receive three
tables for each program that was ranked. These will reflect the following: (1) the values that
they submitted or were calculated from their data for each of the 20 variables with their
corresponding standardized values, and (2) a pair of combined coefficients (plus and minus one
standard deviation from the average value) used in weighting the variables (see Table 5-1); and
(3) the standardized program values and the actual combined coefficients that were used to
calculate the rating corresponding to each endpoint of the inter-quartile range of rankings for that
program, as well as the program ranking corresponding to those ratings (see Tables 5-2a and 5-
2b). Examples of these tables for an economics program are presented and discussed below.
Table 5-1 shows the values submitted by an unidentified program in economics and the
range of combined coefficients for the entire field. Columns 1 and 2 name and label the
variables. Column 3 gives the program value for each of the 20 variables used in the overall
rating (see Appendix E for a description of these variables). Column 4 presents the standardized
value of each variable in column 3; scores are standardized across all programs in the field, using
a mean of 0 and variance of 1. Thus, the relative strengths and weaknesses of a program (in
terms of these 20 variables) can be seen by comparing the standardized values in column 4.
Columns 5 and 6 give the pairs of combined coefficients (weights) assigned to each variable
used in rating all economics programs.29 Each coefficient is a combination of both the direct and
regression-based weights, the derivation of which is described in detail in Appendix A. In
economics, variables V1, V2, and V14 (publications per allocated faculty, cites per publication
and average number of Ph.D.’s) were assigned the largest weights.
Although it would be relatively easy to calculate a single rating for the program using the
data in Table 5-1, the result could be misleading, because it would not reflect the variability (i.e.,
uncertainty) in each of the program measures or the variability in the estimation of the weights.
The process for taking into account these sources of variability is described in detail in Appendix
A.
29
Five hundred regressions are run using half of the raters each time and 500 draws are made from randomly
selected halves of the pool of direct ratings in order to construct the combined coefficients. The values presented
show the range encompassed by plus or minus one standard deviation for each coefficient. See Appendix A for
details.
19
PREPUBLICATION COPY—UNEDITED PROOFS
OCR for page 20
TABLE 5-1 Data and Coefficient Table for a Program in Economics
Standardized Program Values and Range of Combined Coefficients
Institution Name: xxx
Program Name: yyy
Column 1 Column 2 Column 3 Column 4 Column 5 Column 6
Combined Coefficients**
Program Program Value
Standardized* Minus 1 SD Plus 1 SD
Value*
Description Variable
0.118 to 0.132
Publications per Allocated Faculty V1 1.074 2.180
0.276 to 0.307
Cites per Publication V2 1.171 -0.234
0.084 to 0.091
Percent of Faculty with Grants V3 25.50% -0.583
n.s.# n.s.#
Percent Faculty Interdisciplinary V4 5.90% -0.641
n.s.# n.s.#
Percent Non-Asian Minority Faculty V5 7.70% 0.547
n.s.# n.s.#
Percent Female Faculty V6 12.50% -0.440
0.043 to 0.060
Awards per allocated faculty V7 0 -0.546
0.092 to 0.096
Average GRE-Q V8 746 -0.165
0.036 to 0.056
V9 100.00% 0.980
Percent 1st yr. students w/ full support
Percent 1st yr students with portable fellowships 0.021 to 0.033
V10 0.00% -0.544
n.s.# n.s.#
Percent Non-Asian Minority Students V11 10.00% 0.069
-0.038 to -0.030
Percent Female Students V12 44.40% 0.678
#
n.s.#
n.s.
Percent International Students V13 53.30% -0.509
0.120 to 0.144
Average PhDs 2002 to 2006 V14 5.4 -0.355
n.s.# n.s.#
Percent Completing within 6 years V15 27.60% -0.638
-0.028 to -0.017
Time to Degree Full and Part Time V16 5.67 0.232
0.049 to 0.065
Percent students in Academic Positions V17 11.10% -1.405
n.s.# n.s.#
Student Work Space V18 1 1
#
n.s.#
n.s.
Health Insurance V19 1 1
0.026 to 0.037
Number of student activities offered V20 17 0.439
*Col 3 is based on data submitted by the program or calculated from these data.
+
Col 4 is standardized across all program values in the field, with mean of 0 and variance of 1.
** Col 5 is Minus 1 Standard Deviation from the Mean for the combined coefficients for the field as a whole
** Col 6 is Plus 1 Standard Deviation from the Mean for the combined coefficients for the field as a whole
# n.s. in a cell means the coefficient was not significantly different from 0 at the p=.05 level.
20
PREPUBLICATION COPY—UNEDITED PROOFS
OCR for page 21
Tables 5-2a and 5-2b show the calculations of the first and third quartile rankings,
respectively, for a particular program.30 First, a randomly sampled set of regression coefficients
and direct weights is used to obtain a set of 20 combined weights (column 5). These weights are
multiplied by a sampled set of standardized program values (column 4) to generate a program
rating (sum of column 6). This process is repeated another 499 times, generating 500 ratings for
each of the 117 economics programs. Each of these 500 ratings for the program is ranked by
comparing it with the ratings for the other 116 economics programs, based on the same selection
of weights. The 500 rankings for the program are then ordered from best to worst, with the 125th
being the Quartile 3 ranking (45) and the 375th being the Quartile 1 ranking (56). These values
determine the inter-quartile range of rankings for the program. Half of the 500 randomly
generated rankings for the program fall within this range31. The ratings that produced these first
and third quartile rankings are -0.054 and 0.085, as shown in Tables 5-2a and 5-2b.32
30
The first quartile ranking is the highest value of the lowest quarter of rankings. The third quartile ranking is the
highest value of the third quarter of rankings.
31
Use of the inter-quartile range means that we “throw away” half of the possible rankings for the program. The
tails of the distribution can be very long, however, and the inter-quartile range is useful in making meaningful
comparisons, while illustrating the point that any point estimate of a ranking is inexact.
32
We do not show the 117 x 500 matrix of all the ordered ratings for all the economics programs, although it will be
available when the final report is released. However, the ranking is obtained from that table.
21
PREPUBLICATION COPY—UNEDITED PROOFS
OCR for page 22
Table 5-2a Sample First Quartile Ranking Calculation
Institution Name: xxx
Program Name: yyy
Col 1 Col 2 Col 3 Col 4 Col 5 Col 6
Standardized
Program Value Product Col
Combined
Program
with Variation+ 4 x Col 5
Coefficient@
Value*
Description Variable
1.784 0.130
Publications per Allocated Faculty V1 1.074 0.231
-0.269 0.294
Cites per Publication V2 1.171 -0.079
-0.596 0.085
Percent of Faculty with Grants V3 25.5% -0.051
n.s. # n.c. #
-0.581
Percent Faculty Interdisciplinary V4 5.9%
n.s. # n.c. #
0.444
Percent Non-Asian Minority Faculty V5 7.7%
n.s. # n.c. #
-0.511
Percent Female Faculty V6 12.5%
-0.290 0.038
Awards per allocated faculty V7 0 -0.011
-0.286 0.091
Average GRE-Q V8 746 -0.026
Percent 1st yr. students w/ full support 1.432 0.044
V9 100% 0.064
Percent 1st yr students with portable fellowships -0.489 0.023
V10 0.0% -0.011
n.s. # n.c. #
Percent Non-Asian Minority Students 0.062
V11 10.0%
0.561 -0.029
Percent Female Students V12 44.4% -0.016
#
n.c. #
n.s.
-0.018
Percent International Students V13 53.3%
-0.379 0.152
Average PhDs 2002 to 2006 V14 5.4 -0.058
#
n.c. #
n.s.
-0.574
Percent Completing within 6 years V15 27.6%
0.017 -0.026
Time to Degree Full and Part Time V16 5.67 0.000
-1.365 0.063
Percent students in Academic Positions V17 11.1% -0.086
n.s. # n.c. #
1.000
Student Work Space V18 1
n.s. # n.c. #
1.000
Health Insurance V19 1
-0.427 0.025
Number of student activities offered V20 17 -0.011
Average Rating (total of column 6) -0.054
Program Ranking for this rating = 56
*Col 3 is based on data submitted by the program or calculated from these data.
+Col 4 is standardized value for the set of perturbed program values that produced the 1st quartile ranking.
Standardized values have a mean of 0 and variance of 1.
@
Col 5 is the combined direct and regression-based weights for each variable (see Appendix A).
# n.s. in a cell means the coefficient was not significantly different from 0 at the p=.05 level.
# n.c. means the product was not calculated for these coefficients because the coefficient was not significant at the p=.05 level.
22
PREPUBLICATION COPY—UNEDITED PROOFS
OCR for page 23
Table 5-2b Sample Third Quartile Ranking Calculation
Institution Name: xxx
Program Name: yyy
Col 1 Col 2 Col 3 Col 4 Col 5 Col 6
Standardized
Program Value with Product
Combined
Program
Variation+ Col 4 x Col 5
Coefficient@
Value*
Description Variable
2.765 0.134
Publications per Allocated Faculty V1 1.074 0.371
-0.246 0.267
Cites per Publication V2 1.171 -0.066
-0.709 0.073
Percent of Faculty with Grants V3 25.5% -0.051
n.s. # n.c. #
-0.669
Percent Faculty Interdisciplinary V4 5.9%
n.s. # n.c. #
0.515
Percent Non-Asian Minority Faculty V5 7.7%
n.s. # n.c. #
-0.314
Percent Female Faculty V6 12.5%
-0.439 0.050
Awards per allocated faculty V7 0 -0.022
-0.305 0.089
Average GRE-Q V8 746 -0.027
Percent 1st yr. students w/ full support 0.385 0.054
V9 100% 0.021
Percent 1st yr students with portable fellowships -0.585 0.031
V10 0.0% -0.018
n.s. # n.c. #
Percent Non-Asian Minority Students 0.226
V11 10.0%
0.083 -0.043
Percent Female Students V12 44.4% -0.004
#
n.c. #
n.s.
-0.190
Percent International Students V13 53.3%
-0.196 0.121
Average PhDs 2002 to 2006 V14 5.4 -0.024
#
n.c. #
n.s.
-0.725
Percent Completing within 6 years V15 27.6%
-0.439 -0.031
Time to Degree Full and Part Time V16 5.67 0.014
-1.293 0.083
Percent students in Academic Positions V17 11.1% -0.108
n.s. # n.c. #
1.000
Student Work Space V18 1
n.s. # n.c. #
1.000
Health Insurance V19 1
-0.058 0.024
Number of student activities offered V20 17 -0.001
Average Rating (total of column 6) 0.085
Program Ranking for this rating = 45
*Col 3 is based on data submitted by the program or calculated from these data.
+Col 4 is standardized value for the set of perturbed program values that produced the 3rd quartile ranking.
Standardized values have a mean of 0 and variance of 1.
@
Col 5 is the combined direct and regression-based weights for each variable (see Appendix A).
# n.s. in a cell means the coefficient was not significantly different from 0 at the p=.05 level.
# n.c. means the product was not calculated for these coefficients because the coefficient was not significant at the p=.05 level.
23
PREPUBLICATION COPY—UNEDITED PROOFS
OCR for page 24
In interpreting the range of rankings a program received, the first thing to note is which
variables have the highest coefficients. These variables can be determined by examining the
combined coefficients and identifying the largest ones. In the case of economics, the important
variables are citations per publication, publications per allocated faculty, average Ph.D.’s in
2002-2006, and average GRE-Q, each of which has a combined coefficient value of 0.089 or
greater. The rest of the variables are less heavily weighted, and a number of the variables don’t
enter into the determination of the overall rating at all because their coefficients were not
statistically different from 033. The program values in column 3 of Table 1 can be contrasted with
the values taken across all the values in the field, shown in Table 5-3. The importance of
correcting for collinearity 34 is evident from the correlation matrix that follows the variable
listing for each field, and is shown in Table 5-4. Citations per publication, for example, have a
correlation .7 with awards, and .5 with GRE-Q, with average Ph.D.’s and with percent
completing within six years. This interdependence is corrected for by the principal components
adjustment described in Appendix A.
33
The procedure for setting nonsignificant coefficients to 0 is discussed in Appendix A.
34
That is, high degrees of correlation among some of the independent variables.
24
PREPUBLICATION COPY—UNEDITED PROOFS
OCR for page 25
TABLE 5-3 Descriptive Statistics for the Variables used in the Ratings: All Economics Programs
1st 3rd Standard
Quartile Quartile Maximum Deviation
Minimum
Publications per Allocated Faculty 0.049 0.369 0.655 1.357 0.246
Cites per Publication 0.153 0.684 1.771 5.485 1.002
Percent of Faculty with Grants 0.0% 24.0% 50.0% 100.0% 19.9%
Percent Faculty Interdisciplinary 0.0% 2.1% 26.9% 68.4% 16.3%
Percent Non-Asian Minority Faculty 0.0% 0.0% 7.7% 25.0% 5.3%
Percent Female Faculty 0.0% 10.5% 21.1% 66.7% 9.9%
Awards per allocated faculty 0.000 0.000 0.462 5.131 0.890
Average GRE-Q 353 740 790 800 55
Percent 1st yr. students w/ full support 0.0% 50.0% 100.0% 100.0% 29.2%
Percent 1st yr students with portable
fellowships 0.0% 0.0% 9.1% 66.7% 14.2%
Percent Non-Asian Minority Students 0.0% 0.0% 14.3% 50.0% 9.8%
Percent Female Students 0.0% 28.6% 42.9% 76.9% 12.0%
Percent International Students 0.0% 52.4% 76.3% 98.2% 19.6%
Average Ph.D.s 2002 to 2006 1.00 3.20 9.80 26.40 5.73
Percent Completing within 6 years 0.0% 28.3% 51.0% 91.7% 19.1%
Time to Degree Full and Part Time 3.00 5.00 6.00 8.00 0.80
Percent students in Academic Positions 4.2% 17.6% 39.6% 56.5% 12.5%
Student Work Space -1 -1 1 1 0.985
Health Insurance -1 1 1 1 0.672
Number of student activities offered 4 15 18 18 2.161
25
PREPUBLICATION COPY—UNEDITED PROOFS
OCR for page 26
Highest Lowest
Highest Lowest Quartile Quartile Highest Lowest Highest Lowest
Quartile Quartile Student Student Quartile Quartile Quartile Quartile
Table 5-4 Correlations Matrix- Research Research Diversity Diversity Overall Overall
Support Support
Economics Rating Rating Rating Rating Rating Rating Rating Rating
3rd Quartile Research Rating 1.00
1st Quartile Research Rating 1.00 1.00
3rd Quartile Student Support Rating 0.39 0.39 1.00
1st Quartile Student Support Rating 0.38 0.38 1.00 1.00
3rd Quartile Diversity Rating -0.24 -0.25 -0.24 -0.23 1.00
1st Quartile Diversity Rating -0.25 -0.25 -0.24 -0.24 1.00 1.00
3rd Quartile Overall Rating 0.95 0.94 0.45 0.44 -0.23 -0.24 1.00
1st Quartile Overall Rating 0.95 0.95 0.44 0.43 -0.23 -0.23 1.00 1.00
Publications per Allocated Faculty 0.78 0.79 0.32 0.31 -0.23 -0.24 0.64 0.64
Cites per Publication 0.86 0.84 0.33 0.32 -0.16 -0.17 0.90 0.90
Percent of Faculty with Grants 0.55 0.56 0.09 0.08 -0.12 -0.12 0.51 0.52
Percent Faculty Interdisciplinary 0.05 0.05 0.14 0.14 -0.12 -0.13 0.10 0.11
Percent Non-Asian Minority Faculty 0.03 0.03 0.10 0.10 0.39 0.37 0.00 0.00
Percent Female Faculty -0.19 -0.20 -0.26 -0.26 0.45 0.45 -0.21 -0.21
Awards per allocated faculty 0.77 0.76 0.41 0.40 -0.21 -0.22 0.77 0.77
Average GRE-Q 0.52 0.51 0.23 0.22 -0.08 -0.08 0.66 0.65
Percent 1st yr. students w/ full support 0.21 0.21 0.58 0.58 0.00 -0.01 0.24 0.24
Percent 1st yr students with portable fellowships 0.34 0.34 0.21 0.22 -0.13 -0.12 0.37 0.37
Percent Non-Asian Minority Students -0.12 -0.12 0.02 0.02 0.49 0.47 -0.13 -0.13
Percent Female Students -0.29 -0.28 -0.33 -0.33 0.40 0.41 -0.35 -0.34
Percent International Students -0.10 -0.10 -0.14 -0.14 0.67 0.69 -0.03 -0.03
Average Ph.D.s 2002 to 2006 0.58 0.57 0.04 0.03 -0.11 -0.11 0.70 0.70
Percent Completing within 6 years 0.50 0.50 0.57 0.56 -0.32 -0.32 0.54 0.54
Time to Degree Full and Part Time -0.14 -0.16 -0.36 -0.34 -0.07 -0.07 -0.10 -0.10
Percent students in Academic Positions 0.20 0.20 0.71 0.73 -0.20 -0.20 0.27 0.27
26
PREPUBLICATION COPY—UNEDITED PROOFS
OCR for page 27
Student Work Space 0.13 0.15 0.28 0.28 -0.11 -0.11 0.12 0.12
Health Insurance 0.23 0.23 0.19 0.17 -0.11 -0.12 0.25 0.25
Number of student activities offered 0.05 0.04 0.01 0.01 0.04 0.03 0.16 0.15
Percent
Percent Percent 1st yr
Publications Percent of Awards
Non- 1st yr. students
Correlations- Economics cont'd (2)
per Faculty Asian Percent per students with
Allocated Cites per with Percent Faculty Minority allocated Average w/ full portable
Female
Faculty Grants Faculty faculty support
Publication Interdisciplinary Faculty GRE-Q fellowships
3rd Quartile Research Rating
1st Quartile Research Rating
3rd Quartile Student Support Rating
1st Quartile Student Support Rating
3rd Quartile Diversity Rating
1st Quartile Diversity Rating
3rd Quartile Overall Rating
1st Quartile Overall Rating
Publications per Allocated Faculty 1.00
Cites per Publication 0.40 1.00
Percent of Faculty with Grants 0.36 0.37 1.00
Percent Faculty Interdisciplinary -0.08 0.08 0.04 1.00
Percent Non-Asian Minority Faculty 0.06 -0.02 0.12 -0.02 1.00
Percent Female Faculty -0.29 -0.06 0.01 -0.08 0.10 1.00
Awards per allocated faculty 0.42 0.70 0.28 0.20 -0.03 -0.21 1.00
Average GRE-Q 0.37 0.52 0.16 0.10 -0.19 -0.26 0.38 1.00
Percent 1st yr. students w/ full support 0.20 0.16 0.06 0.08 0.17 -0.17 0.20 0.23 1.00
Percent 1st yr students with portable
fellowships 0.13 0.36 0.14 0.05 -0.14 -0.19 0.43 0.14 0.12 1.00
Percent Non-Asian Minority Students -0.09 -0.11 -0.18 0.01 0.42 0.07 0.01 -0.21 -0.01 0.04
Percent Female Students -0.24 -0.22 -0.06 0.00 0.05 0.18 -0.30 -0.15 -0.04 -0.08
27
PREPUBLICATION COPY—UNEDITED PROOFS
OCR for page 28
Percent International Students -0.09 -0.06 -0.10 -0.14 -0.10 0.06 -0.09 0.25 0.02 -0.03
Average Ph.D.s 2002 to 2006 0.33 0.53 0.38 0.11 -0.04 -0.08 0.55 0.44 -0.07 0.20
Percent Completing within 6 years 0.31 0.52 0.09 0.10 -0.17 -0.13 0.52 0.34 0.24 0.28
Time to Degree Full and Part Time -0.19 -0.09 -0.06 0.06 -0.12 -0.03 -0.06 0.04 -0.05 -0.03
Percent students in Academic Positions 0.15 0.15 0.03 0.14 0.13 -0.27 0.28 0.10 0.11 0.11
Student Work Space 0.17 0.12 0.12 -0.02 0.04 -0.13 -0.06 0.12 0.23 0.06
Health Insurance 0.19 0.21 0.13 0.03 0.10 -0.12 0.12 0.22 0.17 -0.21
Number of student activities offered -0.07 0.12 0.12 0.12 0.15 -0.04 0.00 0.13 -0.05 -0.08
28
PREPUBLICATION COPY—UNEDITED PROOFS
OCR for page 29
Time to Percent
Percent Average Percent Degree students Number
Correlations- Economics cont'd (3) Non-Asian Percent Percent Ph.D.s Completing Full and in Student of student
Minority Female International 2002 to within 6 Work activities
Part Academic Health
Students Students Space
Students 2006 years Time Positions Insurance offered
3rd Quartile Research Rating
1st Quartile Research Rating
3rd Quartile Student Support Rating
1st Quartile Student Support Rating
3rd Quartile Diversity Rating
1st Quartile Diversity Rating
3rd Quartile Overall Rating
1st Quartile Overall Rating
Publications per Allocated Faculty
Cites per Publication
Percent of Faculty with Grants
Percent Faculty Interdisciplinary
Percent Non-Asian Minority Faculty
Percent Female Faculty
Awards per allocated faculty
Average GRE-Q
Percent 1st yr. students w/ full support
Percent 1st yr students with portable
fellowships
Percent Non-Asian Minority Students 1.00
Percent Female Students 0.05 1.00
Percent International Students -0.04 0.05 1.00
Average Ph.D.s 2002 to 2006 -0.10 -0.27 0.06 1.00
Percent Completing within 6 years -0.13 -0.36 -0.13 0.28 1.00
Time to Degree Full and Part Time -0.08 0.01 -0.01 0.10 -0.33 1.00
Percent students in Academic Positions 0.11 -0.25 -0.18 0.03 0.22 -0.03 1.00
29
PREPUBLICATION COPY—UNEDITED PROOFS
OCR for page 30
Student Work Space -0.04 -0.04 -0.08 -0.15 0.26 -0.26 0.09 1.00
Health Insurance -0.17 -0.11 -0.01 0.16 0.06 0.01 0.10 -0.05 1.00
Number of student activities offered 0.02 -0.11 0.05 0.16 -0.02 0.09 0.10 -0.05 0.19 1.00
30
PREPUBLICATION COPY—UNEDITED PROOFS
OCR for page 31
The overall range of rankings should be looked at in the context of the dimensional
measures for economics shown in Appendix G. Typically, programs that score well on the
overall rankings will also do well on the research activity ranking, because the two have a
number of highly weighted components in common. It is also worthwhile to look at the student
support and outcomes ranking and the diversity ranking, because these may be of importance to
students in selecting a program. The economics program’s overall measure—it is program
number 62 in the table in Appendix G—places it between the 45th and 56th of the 117
programs. Looking at the dimensional rankings, its research activity is highly ranked—between
the 21st and 31st—primarily because of a relatively high rate of publications per allocated
faculty member. It does less well in terms of student support and outcomes, where it ranks
between the 74th and 87th. Nor does it perform especially well on the diversity dimensional
measure—its rank is between the 64th and 77th. The dimensional measures, then, indicate the
specific areas in which programs are performing well or poorly, as separate from the overall
range of rankings.
The example is intended to explain to the reader how ratings are calculated, and how a
range of rankings is constructed. Shortly before the study results are released, each institutional
coordinator will receive tables similar to the tables above, showing the program data, the range
of coefficients for each variable, the calculation of the first and third quartile rating, and the
corresponding ranking for each rated program at the institution. The user should be aware,
however, that he or she cannot duplicate all 500 samples of combined coefficients. After the
report is released, software will be provided that will permit simulations of ratings with user-
supplied weights and alternative data values. Because the ratings depend on program data and
weights, both of which have uncertainties associated with them, the ranking resulting from a
simulation can only be approximate. The committee would advise that the calculations are more
useful in a qualitative sense. That is, for the numerous programs that fall in the middle range of
rankings, it doesn’t make sense to focus on an exact range. It does make sense to identify the
variables that are important to the ranking of each program and, where possible, improve them35.
35
An example would be working to shorten time to degree.
31
PREPUBLICATION COPY—UNEDITED PROOFS
OCR for page 32
32
PREPUBLICATION COPY—UNEDITED PROOFS
