Skip to main content

A Guide to the Methodology of the National Research Council Assessment of Doctorate Programs (2009) / Chapter Skim
Currently Skimming:

4 The Overall Rating of Program Quality
Pages 14-18

The Chapter Skim interface presents what we've algorithmically identified as the most significant single chunk of text within every page in the chapter.
Select key terms on the right to highlight them within pages of the chapter.


From page 14...
... In addition to reflecting the faculty preferences in each field as derived from the faculty questionnaire, it includes the results of the importance measures derived from the rating survey. This section describes in nontechnical terms how the overall rating of a program is calculated.
Read the entire page →
From page 15...
... The panel investigated two related methods, one based on Bootstrap re-sampling and another closely related method based on Random Halves, and found that either method would be appropriate." (p.
Read the entire page →
From page 16...
... Institutional practices, program characteristics, and faculty and student demographics. Obtained through a combination of original surveys and existing data sources (NSF surveys and ISI publication and citation data)
Read the entire page →
From page 17...
... As is explained below, each rating is recalculated 500 times using different samples of raters. The program ratings obtained from all these calculations can then be arranged 23 The question given raters about program quality was: On a scale from 1 to 6, where 1 equals not adequate for doctoral education and 6 equals a distinguished program, how would you rate this program?
Read the entire page →
From page 18...
... This method of calculating ratings and rankings takes into account variability in rater assessment of what contributes to program quality within a field, variability in values of the measures for a particular program, and the range of error in the statistical estimation. It is important that these techniques give us a range of rankings for most programs.
Read the entire page →

Key Terms

  • program quality
  • overall rating
  • particular program
  • program data
  • doctoral programs

  • direct weights
  • program ratings
  • specific program
  • doctoral education
  • overall approach

  • perceived quality
  • combined weight
  • section describes
  • doctoral program
  • actual program

  • program faculty
  • measured program
  • provide program
  • views faculty
  • dimensional views

  • overall view
  • faculty questionnaire
  • faculty survey
  • direct approach
  • faculty feel

  • resulting direct
  • asking faculty
  • faculty provided
  • faculty raters
  • faculty hold


    • This material may be derived from roughly machine-read images, and so is provided only to facilitate research.
    More information on Chapter Skim is available.