Click for next page ( 20


The National Academies | 500 Fifth St. N.W. | Washington, D.C. 20001
Copyright © National Academy of Sciences. All rights reserved.
Terms of Use and Privacy Statement



Below are the first 10 and last 10 pages of uncorrected machine-read text (when available) of this chapter, followed by the top 30 algorithmically extracted key phrases from the chapter as a whole.
Intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text on the opening pages of each chapter. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.

Do not use for reproduction, copying, pasting, or reading; exclusively for search engines.

OCR for page 19
5 An Example Shortly before the assessment is released, each institutional coordinator will receive three tables for each program that was ranked. These will reflect the following: (1) the values that they submitted or were calculated from their data for each of the 20 variables with their corresponding standardized values, and (2) a pair of combined coefficients (plus and minus one standard deviation from the average value) used in weighting the variables (see Table 5-1); and (3) the standardized program values and the actual combined coefficients that were used to calculate the rating corresponding to each endpoint of the inter-quartile range of rankings for that program, as well as the program ranking corresponding to those ratings (see Tables 5-2a and 5- 2b). Examples of these tables for an economics program are presented and discussed below. Table 5-1 shows the values submitted by an unidentified program in economics and the range of combined coefficients for the entire field. Columns 1 and 2 name and label the variables. Column 3 gives the program value for each of the 20 variables used in the overall rating (see Appendix E for a description of these variables). Column 4 presents the standardized value of each variable in column 3; scores are standardized across all programs in the field, using a mean of 0 and variance of 1. Thus, the relative strengths and weaknesses of a program (in terms of these 20 variables) can be seen by comparing the standardized values in column 4. Columns 5 and 6 give the pairs of combined coefficients (weights) assigned to each variable used in rating all economics programs.29 Each coefficient is a combination of both the direct and regression-based weights, the derivation of which is described in detail in Appendix A. In economics, variables V1, V2, and V14 (publications per allocated faculty, cites per publication and average number of Ph.D.’s) were assigned the largest weights. Although it would be relatively easy to calculate a single rating for the program using the data in Table 5-1, the result could be misleading, because it would not reflect the variability (i.e., uncertainty) in each of the program measures or the variability in the estimation of the weights. The process for taking into account these sources of variability is described in detail in Appendix A. 29 Five hundred regressions are run using half of the raters each time and 500 draws are made from randomly selected halves of the pool of direct ratings in order to construct the combined coefficients. The values presented show the range encompassed by plus or minus one standard deviation for each coefficient. See Appendix A for details. 19 PREPUBLICATION COPY—UNEDITED PROOFS

OCR for page 19
TABLE 5-1 Data and Coefficient Table for a Program in Economics Standardized Program Values and Range of Combined Coefficients Institution Name: xxx Program Name: yyy Column 1 Column 2 Column 3 Column 4 Column 5 Column 6 Combined Coefficients** Program Program Value Standardized* Minus 1 SD Plus 1 SD Value* Description Variable 0.118 to 0.132 Publications per Allocated Faculty V1 1.074 2.180 0.276 to 0.307 Cites per Publication V2 1.171 -0.234 0.084 to 0.091 Percent of Faculty with Grants V3 25.50% -0.583 n.s.# n.s.# Percent Faculty Interdisciplinary V4 5.90% -0.641 n.s.# n.s.# Percent Non-Asian Minority Faculty V5 7.70% 0.547 n.s.# n.s.# Percent Female Faculty V6 12.50% -0.440 0.043 to 0.060 Awards per allocated faculty V7 0 -0.546 0.092 to 0.096 Average GRE-Q V8 746 -0.165 0.036 to 0.056 V9 100.00% 0.980 Percent 1st yr. students w/ full support Percent 1st yr students with portable fellowships 0.021 to 0.033 V10 0.00% -0.544 n.s.# n.s.# Percent Non-Asian Minority Students V11 10.00% 0.069 -0.038 to -0.030 Percent Female Students V12 44.40% 0.678 # n.s.# n.s. Percent International Students V13 53.30% -0.509 0.120 to 0.144 Average PhDs 2002 to 2006 V14 5.4 -0.355 n.s.# n.s.# Percent Completing within 6 years V15 27.60% -0.638 -0.028 to -0.017 Time to Degree Full and Part Time V16 5.67 0.232 0.049 to 0.065 Percent students in Academic Positions V17 11.10% -1.405 n.s.# n.s.# Student Work Space V18 1 1 # n.s.# n.s. Health Insurance V19 1 1 0.026 to 0.037 Number of student activities offered V20 17 0.439 *Col 3 is based on data submitted by the program or calculated from these data. + Col 4 is standardized across all program values in the field, with mean of 0 and variance of 1. ** Col 5 is Minus 1 Standard Deviation from the Mean for the combined coefficients for the field as a whole ** Col 6 is Plus 1 Standard Deviation from the Mean for the combined coefficients for the field as a whole # n.s. in a cell means the coefficient was not significantly different from 0 at the p=.05 level. 20 PREPUBLICATION COPY—UNEDITED PROOFS

OCR for page 19
Tables 5-2a and 5-2b show the calculations of the first and third quartile rankings, respectively, for a particular program.30 First, a randomly sampled set of regression coefficients and direct weights is used to obtain a set of 20 combined weights (column 5). These weights are multiplied by a sampled set of standardized program values (column 4) to generate a program rating (sum of column 6). This process is repeated another 499 times, generating 500 ratings for each of the 117 economics programs. Each of these 500 ratings for the program is ranked by comparing it with the ratings for the other 116 economics programs, based on the same selection of weights. The 500 rankings for the program are then ordered from best to worst, with the 125th being the Quartile 3 ranking (45) and the 375th being the Quartile 1 ranking (56). These values determine the inter-quartile range of rankings for the program. Half of the 500 randomly generated rankings for the program fall within this range31. The ratings that produced these first and third quartile rankings are -0.054 and 0.085, as shown in Tables 5-2a and 5-2b.32 30 The first quartile ranking is the highest value of the lowest quarter of rankings. The third quartile ranking is the highest value of the third quarter of rankings. 31 Use of the inter-quartile range means that we “throw away” half of the possible rankings for the program. The tails of the distribution can be very long, however, and the inter-quartile range is useful in making meaningful comparisons, while illustrating the point that any point estimate of a ranking is inexact. 32 We do not show the 117 x 500 matrix of all the ordered ratings for all the economics programs, although it will be available when the final report is released. However, the ranking is obtained from that table. 21 PREPUBLICATION COPY—UNEDITED PROOFS

OCR for page 19
Table 5-2a Sample First Quartile Ranking Calculation Institution Name: xxx Program Name: yyy Col 1 Col 2 Col 3 Col 4 Col 5 Col 6 Standardized Program Value Product Col Combined Program with Variation+ 4 x Col 5 Coefficient@ Value* Description Variable 1.784 0.130 Publications per Allocated Faculty V1 1.074 0.231 -0.269 0.294 Cites per Publication V2 1.171 -0.079 -0.596 0.085 Percent of Faculty with Grants V3 25.5% -0.051 n.s. # n.c. # -0.581 Percent Faculty Interdisciplinary V4 5.9% n.s. # n.c. # 0.444 Percent Non-Asian Minority Faculty V5 7.7% n.s. # n.c. # -0.511 Percent Female Faculty V6 12.5% -0.290 0.038 Awards per allocated faculty V7 0 -0.011 -0.286 0.091 Average GRE-Q V8 746 -0.026 Percent 1st yr. students w/ full support 1.432 0.044 V9 100% 0.064 Percent 1st yr students with portable fellowships -0.489 0.023 V10 0.0% -0.011 n.s. # n.c. # Percent Non-Asian Minority Students 0.062 V11 10.0% 0.561 -0.029 Percent Female Students V12 44.4% -0.016 # n.c. # n.s. -0.018 Percent International Students V13 53.3% -0.379 0.152 Average PhDs 2002 to 2006 V14 5.4 -0.058 # n.c. # n.s. -0.574 Percent Completing within 6 years V15 27.6% 0.017 -0.026 Time to Degree Full and Part Time V16 5.67 0.000 -1.365 0.063 Percent students in Academic Positions V17 11.1% -0.086 n.s. # n.c. # 1.000 Student Work Space V18 1 n.s. # n.c. # 1.000 Health Insurance V19 1 -0.427 0.025 Number of student activities offered V20 17 -0.011 Average Rating (total of column 6) -0.054 Program Ranking for this rating = 56 *Col 3 is based on data submitted by the program or calculated from these data. +Col 4 is standardized value for the set of perturbed program values that produced the 1st quartile ranking. Standardized values have a mean of 0 and variance of 1. @ Col 5 is the combined direct and regression-based weights for each variable (see Appendix A). # n.s. in a cell means the coefficient was not significantly different from 0 at the p=.05 level. # n.c. means the product was not calculated for these coefficients because the coefficient was not significant at the p=.05 level. 22 PREPUBLICATION COPY—UNEDITED PROOFS

OCR for page 19
Table 5-2b Sample Third Quartile Ranking Calculation Institution Name: xxx Program Name: yyy Col 1 Col 2 Col 3 Col 4 Col 5 Col 6 Standardized Program Value with Product Combined Program Variation+ Col 4 x Col 5 Coefficient@ Value* Description Variable 2.765 0.134 Publications per Allocated Faculty V1 1.074 0.371 -0.246 0.267 Cites per Publication V2 1.171 -0.066 -0.709 0.073 Percent of Faculty with Grants V3 25.5% -0.051 n.s. # n.c. # -0.669 Percent Faculty Interdisciplinary V4 5.9% n.s. # n.c. # 0.515 Percent Non-Asian Minority Faculty V5 7.7% n.s. # n.c. # -0.314 Percent Female Faculty V6 12.5% -0.439 0.050 Awards per allocated faculty V7 0 -0.022 -0.305 0.089 Average GRE-Q V8 746 -0.027 Percent 1st yr. students w/ full support 0.385 0.054 V9 100% 0.021 Percent 1st yr students with portable fellowships -0.585 0.031 V10 0.0% -0.018 n.s. # n.c. # Percent Non-Asian Minority Students 0.226 V11 10.0% 0.083 -0.043 Percent Female Students V12 44.4% -0.004 # n.c. # n.s. -0.190 Percent International Students V13 53.3% -0.196 0.121 Average PhDs 2002 to 2006 V14 5.4 -0.024 # n.c. # n.s. -0.725 Percent Completing within 6 years V15 27.6% -0.439 -0.031 Time to Degree Full and Part Time V16 5.67 0.014 -1.293 0.083 Percent students in Academic Positions V17 11.1% -0.108 n.s. # n.c. # 1.000 Student Work Space V18 1 n.s. # n.c. # 1.000 Health Insurance V19 1 -0.058 0.024 Number of student activities offered V20 17 -0.001 Average Rating (total of column 6) 0.085 Program Ranking for this rating = 45 *Col 3 is based on data submitted by the program or calculated from these data. +Col 4 is standardized value for the set of perturbed program values that produced the 3rd quartile ranking. Standardized values have a mean of 0 and variance of 1. @ Col 5 is the combined direct and regression-based weights for each variable (see Appendix A). # n.s. in a cell means the coefficient was not significantly different from 0 at the p=.05 level. # n.c. means the product was not calculated for these coefficients because the coefficient was not significant at the p=.05 level. 23 PREPUBLICATION COPY—UNEDITED PROOFS

OCR for page 19
In interpreting the range of rankings a program received, the first thing to note is which variables have the highest coefficients. These variables can be determined by examining the combined coefficients and identifying the largest ones. In the case of economics, the important variables are citations per publication, publications per allocated faculty, average Ph.D.’s in 2002-2006, and average GRE-Q, each of which has a combined coefficient value of 0.089 or greater. The rest of the variables are less heavily weighted, and a number of the variables don’t enter into the determination of the overall rating at all because their coefficients were not statistically different from 033. The program values in column 3 of Table 1 can be contrasted with the values taken across all the values in the field, shown in Table 5-3. The importance of correcting for collinearity 34 is evident from the correlation matrix that follows the variable listing for each field, and is shown in Table 5-4. Citations per publication, for example, have a correlation .7 with awards, and .5 with GRE-Q, with average Ph.D.’s and with percent completing within six years. This interdependence is corrected for by the principal components adjustment described in Appendix A. 33 The procedure for setting nonsignificant coefficients to 0 is discussed in Appendix A. 34 That is, high degrees of correlation among some of the independent variables. 24 PREPUBLICATION COPY—UNEDITED PROOFS

OCR for page 19
TABLE 5-3 Descriptive Statistics for the Variables used in the Ratings: All Economics Programs 1st 3rd Standard Quartile Quartile Maximum Deviation Minimum Publications per Allocated Faculty 0.049 0.369 0.655 1.357 0.246 Cites per Publication 0.153 0.684 1.771 5.485 1.002 Percent of Faculty with Grants 0.0% 24.0% 50.0% 100.0% 19.9% Percent Faculty Interdisciplinary 0.0% 2.1% 26.9% 68.4% 16.3% Percent Non-Asian Minority Faculty 0.0% 0.0% 7.7% 25.0% 5.3% Percent Female Faculty 0.0% 10.5% 21.1% 66.7% 9.9% Awards per allocated faculty 0.000 0.000 0.462 5.131 0.890 Average GRE-Q 353 740 790 800 55 Percent 1st yr. students w/ full support 0.0% 50.0% 100.0% 100.0% 29.2% Percent 1st yr students with portable fellowships 0.0% 0.0% 9.1% 66.7% 14.2% Percent Non-Asian Minority Students 0.0% 0.0% 14.3% 50.0% 9.8% Percent Female Students 0.0% 28.6% 42.9% 76.9% 12.0% Percent International Students 0.0% 52.4% 76.3% 98.2% 19.6% Average Ph.D.s 2002 to 2006 1.00 3.20 9.80 26.40 5.73 Percent Completing within 6 years 0.0% 28.3% 51.0% 91.7% 19.1% Time to Degree Full and Part Time 3.00 5.00 6.00 8.00 0.80 Percent students in Academic Positions 4.2% 17.6% 39.6% 56.5% 12.5% Student Work Space -1 -1 1 1 0.985 Health Insurance -1 1 1 1 0.672 Number of student activities offered 4 15 18 18 2.161 25 PREPUBLICATION COPY—UNEDITED PROOFS

OCR for page 19
Highest Lowest Highest Lowest Quartile Quartile Highest Lowest Highest Lowest Quartile Quartile Student Student Quartile Quartile Quartile Quartile Table 5-4 Correlations Matrix- Research Research Diversity Diversity Overall Overall Support Support Economics Rating Rating Rating Rating Rating Rating Rating Rating 3rd Quartile Research Rating 1.00 1st Quartile Research Rating 1.00 1.00 3rd Quartile Student Support Rating 0.39 0.39 1.00 1st Quartile Student Support Rating 0.38 0.38 1.00 1.00 3rd Quartile Diversity Rating -0.24 -0.25 -0.24 -0.23 1.00 1st Quartile Diversity Rating -0.25 -0.25 -0.24 -0.24 1.00 1.00 3rd Quartile Overall Rating 0.95 0.94 0.45 0.44 -0.23 -0.24 1.00 1st Quartile Overall Rating 0.95 0.95 0.44 0.43 -0.23 -0.23 1.00 1.00 Publications per Allocated Faculty 0.78 0.79 0.32 0.31 -0.23 -0.24 0.64 0.64 Cites per Publication 0.86 0.84 0.33 0.32 -0.16 -0.17 0.90 0.90 Percent of Faculty with Grants 0.55 0.56 0.09 0.08 -0.12 -0.12 0.51 0.52 Percent Faculty Interdisciplinary 0.05 0.05 0.14 0.14 -0.12 -0.13 0.10 0.11 Percent Non-Asian Minority Faculty 0.03 0.03 0.10 0.10 0.39 0.37 0.00 0.00 Percent Female Faculty -0.19 -0.20 -0.26 -0.26 0.45 0.45 -0.21 -0.21 Awards per allocated faculty 0.77 0.76 0.41 0.40 -0.21 -0.22 0.77 0.77 Average GRE-Q 0.52 0.51 0.23 0.22 -0.08 -0.08 0.66 0.65 Percent 1st yr. students w/ full support 0.21 0.21 0.58 0.58 0.00 -0.01 0.24 0.24 Percent 1st yr students with portable fellowships 0.34 0.34 0.21 0.22 -0.13 -0.12 0.37 0.37 Percent Non-Asian Minority Students -0.12 -0.12 0.02 0.02 0.49 0.47 -0.13 -0.13 Percent Female Students -0.29 -0.28 -0.33 -0.33 0.40 0.41 -0.35 -0.34 Percent International Students -0.10 -0.10 -0.14 -0.14 0.67 0.69 -0.03 -0.03 Average Ph.D.s 2002 to 2006 0.58 0.57 0.04 0.03 -0.11 -0.11 0.70 0.70 Percent Completing within 6 years 0.50 0.50 0.57 0.56 -0.32 -0.32 0.54 0.54 Time to Degree Full and Part Time -0.14 -0.16 -0.36 -0.34 -0.07 -0.07 -0.10 -0.10 Percent students in Academic Positions 0.20 0.20 0.71 0.73 -0.20 -0.20 0.27 0.27 26 PREPUBLICATION COPY—UNEDITED PROOFS

OCR for page 19
Student Work Space 0.13 0.15 0.28 0.28 -0.11 -0.11 0.12 0.12 Health Insurance 0.23 0.23 0.19 0.17 -0.11 -0.12 0.25 0.25 Number of student activities offered 0.05 0.04 0.01 0.01 0.04 0.03 0.16 0.15 Percent Percent Percent 1st yr Publications Percent of Awards Non- 1st yr. students Correlations- Economics cont'd (2) per Faculty Asian Percent per students with Allocated Cites per with Percent Faculty Minority allocated Average w/ full portable Female Faculty Grants Faculty faculty support Publication Interdisciplinary Faculty GRE-Q fellowships 3rd Quartile Research Rating 1st Quartile Research Rating 3rd Quartile Student Support Rating 1st Quartile Student Support Rating 3rd Quartile Diversity Rating 1st Quartile Diversity Rating 3rd Quartile Overall Rating 1st Quartile Overall Rating Publications per Allocated Faculty 1.00 Cites per Publication 0.40 1.00 Percent of Faculty with Grants 0.36 0.37 1.00 Percent Faculty Interdisciplinary -0.08 0.08 0.04 1.00 Percent Non-Asian Minority Faculty 0.06 -0.02 0.12 -0.02 1.00 Percent Female Faculty -0.29 -0.06 0.01 -0.08 0.10 1.00 Awards per allocated faculty 0.42 0.70 0.28 0.20 -0.03 -0.21 1.00 Average GRE-Q 0.37 0.52 0.16 0.10 -0.19 -0.26 0.38 1.00 Percent 1st yr. students w/ full support 0.20 0.16 0.06 0.08 0.17 -0.17 0.20 0.23 1.00 Percent 1st yr students with portable fellowships 0.13 0.36 0.14 0.05 -0.14 -0.19 0.43 0.14 0.12 1.00 Percent Non-Asian Minority Students -0.09 -0.11 -0.18 0.01 0.42 0.07 0.01 -0.21 -0.01 0.04 Percent Female Students -0.24 -0.22 -0.06 0.00 0.05 0.18 -0.30 -0.15 -0.04 -0.08 27 PREPUBLICATION COPY—UNEDITED PROOFS

OCR for page 19
Percent International Students -0.09 -0.06 -0.10 -0.14 -0.10 0.06 -0.09 0.25 0.02 -0.03 Average Ph.D.s 2002 to 2006 0.33 0.53 0.38 0.11 -0.04 -0.08 0.55 0.44 -0.07 0.20 Percent Completing within 6 years 0.31 0.52 0.09 0.10 -0.17 -0.13 0.52 0.34 0.24 0.28 Time to Degree Full and Part Time -0.19 -0.09 -0.06 0.06 -0.12 -0.03 -0.06 0.04 -0.05 -0.03 Percent students in Academic Positions 0.15 0.15 0.03 0.14 0.13 -0.27 0.28 0.10 0.11 0.11 Student Work Space 0.17 0.12 0.12 -0.02 0.04 -0.13 -0.06 0.12 0.23 0.06 Health Insurance 0.19 0.21 0.13 0.03 0.10 -0.12 0.12 0.22 0.17 -0.21 Number of student activities offered -0.07 0.12 0.12 0.12 0.15 -0.04 0.00 0.13 -0.05 -0.08 28 PREPUBLICATION COPY—UNEDITED PROOFS

OCR for page 19
Time to Percent Percent Average Percent Degree students Number Correlations- Economics cont'd (3) Non-Asian Percent Percent Ph.D.s Completing Full and in Student of student Minority Female International 2002 to within 6 Work activities Part Academic Health Students Students Space Students 2006 years Time Positions Insurance offered 3rd Quartile Research Rating 1st Quartile Research Rating 3rd Quartile Student Support Rating 1st Quartile Student Support Rating 3rd Quartile Diversity Rating 1st Quartile Diversity Rating 3rd Quartile Overall Rating 1st Quartile Overall Rating Publications per Allocated Faculty Cites per Publication Percent of Faculty with Grants Percent Faculty Interdisciplinary Percent Non-Asian Minority Faculty Percent Female Faculty Awards per allocated faculty Average GRE-Q Percent 1st yr. students w/ full support Percent 1st yr students with portable fellowships Percent Non-Asian Minority Students 1.00 Percent Female Students 0.05 1.00 Percent International Students -0.04 0.05 1.00 Average Ph.D.s 2002 to 2006 -0.10 -0.27 0.06 1.00 Percent Completing within 6 years -0.13 -0.36 -0.13 0.28 1.00 Time to Degree Full and Part Time -0.08 0.01 -0.01 0.10 -0.33 1.00 Percent students in Academic Positions 0.11 -0.25 -0.18 0.03 0.22 -0.03 1.00 29 PREPUBLICATION COPY—UNEDITED PROOFS

OCR for page 19
Student Work Space -0.04 -0.04 -0.08 -0.15 0.26 -0.26 0.09 1.00 Health Insurance -0.17 -0.11 -0.01 0.16 0.06 0.01 0.10 -0.05 1.00 Number of student activities offered 0.02 -0.11 0.05 0.16 -0.02 0.09 0.10 -0.05 0.19 1.00 30 PREPUBLICATION COPY—UNEDITED PROOFS

OCR for page 19
The overall range of rankings should be looked at in the context of the dimensional measures for economics shown in Appendix G. Typically, programs that score well on the overall rankings will also do well on the research activity ranking, because the two have a number of highly weighted components in common. It is also worthwhile to look at the student support and outcomes ranking and the diversity ranking, because these may be of importance to students in selecting a program. The economics program’s overall measure—it is program number 62 in the table in Appendix G—places it between the 45th and 56th of the 117 programs. Looking at the dimensional rankings, its research activity is highly ranked—between the 21st and 31st—primarily because of a relatively high rate of publications per allocated faculty member. It does less well in terms of student support and outcomes, where it ranks between the 74th and 87th. Nor does it perform especially well on the diversity dimensional measure—its rank is between the 64th and 77th. The dimensional measures, then, indicate the specific areas in which programs are performing well or poorly, as separate from the overall range of rankings. The example is intended to explain to the reader how ratings are calculated, and how a range of rankings is constructed. Shortly before the study results are released, each institutional coordinator will receive tables similar to the tables above, showing the program data, the range of coefficients for each variable, the calculation of the first and third quartile rating, and the corresponding ranking for each rated program at the institution. The user should be aware, however, that he or she cannot duplicate all 500 samples of combined coefficients. After the report is released, software will be provided that will permit simulations of ratings with user- supplied weights and alternative data values. Because the ratings depend on program data and weights, both of which have uncertainties associated with them, the ranking resulting from a simulation can only be approximate. The committee would advise that the calculations are more useful in a qualitative sense. That is, for the numerous programs that fall in the middle range of rankings, it doesn’t make sense to focus on an exact range. It does make sense to identify the variables that are important to the ranking of each program and, where possible, improve them35. 35 An example would be working to shorten time to degree. 31 PREPUBLICATION COPY—UNEDITED PROOFS

OCR for page 19
32 PREPUBLICATION COPY—UNEDITED PROOFS