A Guide to the Methodology of the National Research Council Assessment of Doctorate Programs (2009) / Chapter Skim
Currently Skimming:
APPENDIX A A Technical Discussion of the Process of Rating and Ranking Programs in a Field
APPENDIX A A Technical Discussion of the Process of Rating and Ranking Programs in a Field
Pages 33-52
The Chapter Skim interface presents what we've algorithmically identified as the most significant single chunk of text within every page in the chapter.
Select key terms on the right to highlight them within pages of the chapter.
From page 33... ...
The topics in this appendix include: • a summary of the sources of data used in the rating and ranking process, • the direct weights, the regression-based weights, the methods used to calculate the regression-based weights, • the simulation of the uncertainty in the weights by random-halves sampling, • the construction of the combined weights using an optimal fraction to combine the simulated values of the direct and regression-based weights, • the elimination of variables with nonsignificant combined weights, • the simulation of the uncertainty in the values of the program variables, • the combination of the simulated combined weights for the significant program variables with the simulated standardized values of the program variables to obtain simulated rankings, and • the resulting inter-quartile ranges of rankings that are the primary rating and ranking quantities we report.
|
From page 34... ...
Perform backwards stepwise version of P, denoted by regression to obtain a stable fitted % combined weights, f0. equation predicting average ratings from P
|
From page 35... ...
The original 21 program characteristics listed on the faculty questionnaire are shown in Table A-1, and they were divided into three categories -- faculty, student, and program characteristics. Of the original 21, there are 20 for which adequate data were deemed to be available to use in the rating process, and these 20 data values for each program became the 20 program variables used in this study to which we repeatedly refer.
|
From page 36... ...
each of the three categories by assigning them values that summed to 100 over the three categories.38 For each faculty respondent, his or her importance measure for each program characteristic was calculated as the product of the score that it received times the relative importance value assigned to its category. Finally, the 20 importance measures for each faculty respondent were transformed by dividing each one by the sum of his or her importance measures across the 20 program variables.
|
From page 37... ...
Gender diversity of the student population vii. A high percentage of international students Program characteristics i.
|
From page 38... ...
The averages, { xk }, are the direct weights of the faculty respondents because they directly give the average relative importance of each program variable, as indicated by the faculty questionnaire responses in the field of study. Thus, the final 20 importance measures of the program characteristics for each faculty respondent are nonnegative and sum to 1.0.
|
From page 39... ...
Because each program's average rating is determined by a different random sample of graduate faculty raters, it is highly unlikely that any two programs will be evaluated by exactly the same set of raters. Denote the vector of the average ratings for the sampled programs in a field by r .
|
From page 40... ...
The resulting unstable regression coefficients would have been unusable for our purposes. For example, as expected, when we fit a linear model that included all 20 of the program variables, we found that for a number of the variables, the coefficients and their signs did not make intuitive sense.
|
From page 41... ...
This was regarded as a virtue, because we did not necessarily eliminate any of the original program variables from the prediction equation used to find the regression-based weights. By proceeding this way, we are not forced to give a zero weight to one of two collinear variables in the step-wise procedure.
|
From page 42... ...
When there were fewer than 30 programs in a field, it was combined with a larger discipline with similar direct weights for the purposes of estimating the regression-based weights.40 In one case, computer engineering, there were fewer than 25 39 See Appendix G of Assessing Research Doctorate Programs: A Methodology Study, National Research Council (2003) 40 The fields for which this was done were: Small Field Surrogate Field Aerospace engineering Mechanical engineering Agricultural economics Economics American studies English literature Astrophysics and astronomy Physics Entomology Plant science Forestry Plant science Food science Plant science Engineering science and mechanics Mechanical engineering Theatre and performance English literature 42 PREPUBLICATION COPY -- UNEDITED PROOFS
|
From page 43... ...
This is not necessarily true of the regression ˆ coefficients, { mk }. The scale of mk depends on both the scale of pjk and the scale of the average ratings, { rj }.We decided, because our intent was to combine these two sources of the importance of the various program variables, that they needed to be on similar scales.
|
From page 44... ...
ˆ However, because both mk and xk are subject to uncertainty, we made one additional adjustment to Equation 10 that is described below, following the discussion of how we simulated the uncertainty in both the direct weights and in the average ratings used to form the regressionbased weights. 43 We have throughout estimated linear regressions.
|
From page 45... ...
Disagreement among the graduate faculty on the relative importance of the 20 program variables is the source of the uncertainty of the direct weights. The average ratings of the sampled faculty in r are also subject to uncertainty; a different sample of raters or programs would have produced different values in r .
|
From page 46... ...
This gives us 500 replications of the direct weights and 500 replications of the regression-based weights that we then combined into 500 replications of the combined weights, which we describe next.
|
From page 47... ...
(12) ˆ By construction of the RH procedure, the mean of the distribution of mk is mk (the regression coefficients that are obtained when the data from all n faculty raters are used)
|
From page 48... ...
denotes the variance of the individual direct weights given to the kth program variable by these faculty respondents. The value of σ2( mk )
|
From page 49... ...
The eliminated program variables are ignored in calculating the direct and regression-based weights for the other variables. New RH samples are drawn, the direct weights are retransformed so that the absolute sum of the remaining direct weights was 1.0, the regressions are re-run using the reduced set of program variables as predictors, and new optimal fractions are computed to combine the direct and regression-based weights.
|
From page 50... ...
This standard error was then divided by the value of the publications per faculty variable to get the relative error factor for this program variable. 47 Examination of the effect of this procedure gave correlations between the median rankings with and without the elimination of nonsignificant variables of .99.
|
From page 51... ...
Now we use Equations 17, 19,and 20 to combine the replications of the combined weights with the replications of the standardized perturbed program variables to obtain 500 replications of the combined rating Rj for each program, j. Denote the kth replication of Rj by R (j k )
|
From page 52... ...
distribution. The interpretation of the inter-quartile range is that it is the middle of the distribution of rankings and reflects the uncertainty in the direct and regression-based weights and in the program data values, twenty-five percent of a program's rankings in our process are less than this interval and 25 percent are higher.
|
Key Terms
This material may be derived from roughly machine-read images, and so is provided only to facilitate research.
More
information on Chapter Skim is available.