A Data-Based Assessment of Research-Doctorate Programs in the United States (with CD) (2011)

Below is the uncorrected machine-read text of this chapter, intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text of each book. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.

5 Faculty Values as Reflected in the Two Illustrative Rankings This study is valuable for both the comparative data it makes available and the importance it attaches to some of the collected data by conducting a survey of faculty and relating program ratings to measured characteristics. The values used throughout this report, for the two overall rankings and, taken separately, for the dimensional rankings, derive in part from faculty members’ answers to questions designed to measure faculty perceptions of the relative importance of program characteristics to the quality of doctoral programs. The 21 characteristics identified by the committee and in the literature as important were divided into three categories are shown in Box 5-1. BOX 5-1 Characteristics Included in the Faculty Weighting Process CATEGORY I—Program Faculty Quality a. Number of publications (books, articles, etc.) per faculty member b. Number of citations per faculty member c. Receipt of extramural grants for research d. Involvement in interdisciplinary work e. Racial and ethnic diversity of the program faculty f. Gender diversity of the program faculty g. Reception by peers of a faculty member’s work, as measured by honors and awards CATEGORY II—Student Characteristics a. Median GRE scores of entering students b. Percentage of students receiving full financial support c. Percentage of students with portable fellowships d. Number of student publications and presentationsa e. Racial and ethnic diversity of the student population f. Gender diversity of the student population g. A high percentage of international students 65 

66 A DATA-BASED ASSESSMENT OF RESEARCH-DOCTORATE PROGRAMS IN THE U.S. CATEGORY III—Program Characteristics a. Average number of Ph.D.’s granted over the previous five years b. Percentage of entering students who complete a doctoral degree c. Time to degree d. Placement of students after graduation e. Percentage of students with individual work space f. Percentage of health insurance premiums covered by the institution or program g. Number of student support activities provided at either the institutional or program levelb a The committee initially believed this variable should be included, but later found no data to support it. It was eliminated from the calculation of weights, and the data tables in this report include 20, not 21, variables. b This variable is a tally of whether the following services are provided to graduate students at either the institutional or program level: orientation for new students, prizes or awards to doctoral students for teaching or research, formal training in academic integrity/ethics, travel funds to attend professional meetings, grievance and dispute resolution procedures, annual review of all enrolled doctoral students, training to improve teaching skills, institutionally supported graduate student association, information about employment outcomes of graduates, and on-campus graduate student research conference. Faculty respondents were asked to choose up to four characteristics in each category that they thought were important. They were then asked to indicate which one or two of the four they found the most important. The final task was to assign an importance score of from 0–100 to each category; the sum of the importance scores over all categories was to equal 100.1 The five characteristics given the highest rating on each measure are shown in Table 5-1. Specifically, it shows the average ranking of the characteristic in the field across all 20 measures.2 This table makes the differences in the two rating methods clear. On the general survey (S measure) in all fields, the publication measure was very important. It was less important in the regression-based R measure, where for all fields size, as measured by the average number of Ph.D.’s was important. The percentage of faculty with grants was highly ranked on the S measure in all fields but the humanities. Awards per allocated faculty, a measure that may reflect reputation, was important in all fields but the agricultural sciences, and it was highly ranked for both the R and S measures in three of the five broad fields. None of the diversity measures appeared to be important in either methodology. GRE scores were important for R measures, but not for S measures, while placement of students in academic positions was important for S measures, but not for R measures. 1 This was a forced choice. Faculty could not enter a characteristic beyond the ones given. 2 To calculate the ranking of the importance weight of a characteristic, the rank order (1–20, with 1 being the highest) of the median weight was calculated for each characteristic, and this rank was averaged across all the fields in the each broad field. 

FACULTY VALUES AS REFLECTED IN THE TWO ILLUSTRATIVE RANKINGS 67 TABLE 5-1 Most Highly Rated Characteristics of Doctoral Programs on R and S Measures Biological and Physical and Social and Humanities Measure Agricultural Health Mathematical Behavioral Characteristic Type Sciences Sciences Engineering Sciences Sciences R 2.67 6.23 3.00 --- --- --- Publications per allocated faculty S 1.17 1.85 1.38 1.67 1.00 1.00 R --- 4.31 --- 4.88 6.60 n.a. Cites per publication S 4.00 3.77 3.00 2.63 2.40 n.a. R --- --- --- --- 8.30 --- Percentage of faculty with grants S 1.83 1.15 1.63 1.67 3.20 --- R --- --- --- --- --- --- Percentage of interdisciplinary faculty S 5.67 --- --- --- --- 4.77 R 5.17 5.31 3.63 4.11 3.10 4.54 Awards per allocated faculty S 1.83 1.15 1.63 1.67 3.20 --- Average GRE (GRE-V for the R 5.50 5.69 5.63 7.33 4.20 3.62 humanities, GRE-Q otherwise) S --- 5.00 --- --- --- --- Percentage of first-year students with R --- --- --- --- --- 8.08 full support S --- --- --- --- --- 4.77 Average number of Ph.D.’s, 2002– R 1.00 3.46 1.00 1.44 4.20 3.85 2006 S --- --- --- --- --- --- Percentage of students in academic R --- --- --- 6.78 --- 4.77 positions S 3.17 4.23 5.25 4.78 4.20 2.92 R --- --- --- --- --- --- Health insurance S --- --- --- --- --- --- R 5.33 --- 7.63 --- --- --- Number of student activities offered S --- --- --- --- --- --- Note: Number shown is average rank of the characteristic taken across the disciplines in the broad field. The five categories given the highest rankings are shown for each field. “---“ indicates the characteristic was not one of the top five for the field. “n.a.” indicates not collected; GRE-Q = GRE-Quantitative Reasoning; GRE-V = GRE-Verbal. 

68 A DATA-BASED ASSESSMENT OF RESEARCH-DOCTORATE PROGRAMS IN THE U.S. Faculty values are also reflected in the relative importance of each category measured on the faculty questionnaire. For all fields the importance score for the faculty productivity variables was highest, followed by the student support and outcomes category, with program demographic characteristics coming in last. These category importance values are shown in Table 5-2. One interesting observation is that although these weights are different from one another in a statistical sense, they are remarkably similar regardless of the field of the respondents. TABLE 5-2 Faculty Importance Weights by Broad Field Faculty Productivity and Associated Student Support and Program Diversity Characteristics (%) Outcome Characteristics (%) Characteristics (%) Agricultural sciences 45.2 30.5 25.1 Biological and health sciences 45.1 31.9 23.7 Physical and mathematical sciences 48.9 29.7 22.2 Engineering 46.5 31.8 22.5 Social and behavioral sciences 49.1 28.2 23.6 Humanities 46.4 28.9 25.6 DIMENSIONAL MEASURES Despite the relatively moderate importance that faculty placed on the student treatment and program diversity dimensions of doctoral education, the committee felt it was very important to measure and discuss these dimensions, in part because they have figured prominently in national discussions of doctoral education.3 The dimensional measures were obtained by means of the faculty responses to Section G (see Table 5-1).4 These measures take a subset of all the characteristics and recalculate the weights so that the total of the weights for the subset adds up to 1. The dimensional measures used in this study—research activity, student support and outcomes, and diversity of the academic environment—are described in the sections that follow. Research Activity This dimensional measure relates to the various ways in which to gauge the contribution of research: publications, citations (except for the humanities), the percentage of the faculty holding research grants, and recognition of scholarship as evidenced by honors and awards. The importance weights are shown in Table 5-2a. Specifically, the components of the research activity dimensional measure are average publications per 3 See, for example, Ronald G. Ehrenberg, Harriet Zuckerman, Jeffrey A. Groen, and Sharon M. Brucker, Educating Scholars: Doctoral Education in the Humanities (Princeton, N.J.: Princeton University Press, 2010). 4 These dimensional weights are different from the S weights, which take all 20 variables into account. 

FACULTY VALUES AS REFLECTED IN THE TWO ILLUSTRATIVE RANKINGS 69 TABLE 5-2A Average Faculty Importance Weights on Components of Research Activity Dimensional Measure Publications per Percentage of Allocated Cites per Allocated Faculty Awards per Broad Field Facultya Publication Holding Grants Allocated Faculty Agricultural sciences 0.349 0.175 0.348 0.128 Biological and health sciences 0.314 0.192 0.377 0.118 Physical and mathematical sciences 0.281 0.258 0.294 0.167 Engineering 0.291 0.238 0.304 0.167 Social and behavioral sciences 0.376 0.250 0.216 0.158 a Humanities 0.591 0.124 0.284 a For the humanities, publications are measured by books and articles per allocated faculty member. There are no data for citations. allocated faculty member,5 average citations per publication, percentage of core and new doctoral faculty respondents holding grants, and awards per allocated faculty member.6 Publishing patterns and the availability of research funding and awards for scholarship vary by field, but the weight placed on publications per faculty member is remarkably consistent—about 30 percent—across fields. Research activity is the dimensional measure that most closely tracks the overall measure of program quality, because in all fields both the S measure—based on abstract faculty preferences—and the R measure place high weights on these characteristics. For the research activity measures, faculty in the sciences and engineering place the greatest weight on grants per faculty member. In some fields research funding is common, and grants are an important source of support for faculty and doctoral students. In the social and behavioral sciences and the humanities, the greatest weight is placed on publications. In one social science discipline, economics, the weight placed on citations is almost equal to that placed on publications, but in all other fields publication is more highly valued. Grants are a less important source of funding in the humanities, and for those fields publications and awards are the most important visible signs of research activity. The values evinced in the broad fields are, with the exception of the humanities, very similar. Student Support and Outcomes This measure combines data on the percentage of students fully funded in the first year, the percentage of students completing their degrees in a given time period, time to degree, and placement in academic positions (including academic postdoctoral positions). 5 Because many faculty members supervise dissertations in more than one program, faculty members were allocated across these programs so that the total, taken across all programs, equaled 1 or less (when the faculty member was in a professional school). 6 In constructing this measure, a distinction was made between “highly prestigious” and “prestigious” awards, with the former given a weight of 5 and the latter given a weight of 1. The committee reviewed 1,393 awards and honors from various scholarly organizations. Highly prestigious awards were identified by the committee. 

70 A DATA-BASED ASSESSMENT OF RESEARCH-DOCTORATE PROGRAMS IN THE U.S. The committee found that faculty typically placed a larger weight on student support and completion rates than on median time to degree or academic placement.7 Surprising uniformity appears across broad fields on the weights, which are shown in Table 5-2B. TABLE 5-2B Average Faculty Importance Weights on Components of the Student Support and Outcomes Dimensional Measure Percentage Percentage of First-Year Completing Degree Time to Degree, Graduates Students with Within Six or Full- and Part- in Broad Field Full Support Eight Yearsa Timeb Academic Positions Agricultural sciences 0.304 0.231 –0.109 0.357 Biological and health sciences 0.259 0.264 –0.135 0.342 Physical and mathematical 0.306 0.221 –0.114 0.359 sciences Engineering 0.346 0.200 –0.099 0.356 Social and behavioral sciences 0.291 0.229 –0.110 0.370 Humanities 0.316 0.245 –0.102 0.337 a For the humanities, eight years are used in the completion measure. This completion measure is measured as the fraction of the entering cohort that has received a Ph.D. within six or eight years. b Time to degree has a negative weight reflecting that a shorter time is better. The sum of the absolute values of the weights is 1. The percentage of graduates obtaining academic positions dominates these measures, and, interestingly, the weight given to this variable (0.34 –0.37) is essentially the same in all of the broad academic fields. The negative sign on time to degree indicates that the shorter the time to degree, the better. Student support in the first year is also an important variable in all fields. Percentage of completion and time to degree are less important, and although these variables have been discussed within the community of graduate deans, they are not variables that faculty feel are important in determining the quality of a doctoral program. Diversity of the Academic Environment The diversity measures—percentage of faculty and of students from underrepresented minority groups, percentage of faculty and of students who are female, and percentage of students who are international (that is, in the United States on a temporary visa)—did not appear to be major factors in determining the overall perceived quality of doctoral programs.8 When these measures are taken separately, definite patterns emerge for variables that faculty thought were more important, and these patterns vary by field. Most fields place the highest weight on the percentage of students from underrepresented 7 Ideally, the committee would have used a measure such as employment in one’s field five years after receipt of a Ph.D., but many programs did not collect such data. The committee hoped that including this measure would encourage more programs to pay attention to postdegree outcomes for their graduates. 8 In other words, the weights on these characteristics were small relative to other characteristics in the R and S measures. 

FACULTY VALUES AS REFLECTED IN THE TWO ILLUSTRATIVE RANKINGS 71 minority groups. In the biological and health sciences, social and behavioral sciences, and humanities, relatively high weights are also placed on the percentage of faculty who are underrepresented minorities. The percentage of international students was not highly weighted relative to the other diversity weights, except for the physical and mathematical sciences. These weights, by broad field, are shown in Table 5-2C. TABLE 5-2C Average Faculty Importance Weights on Components of the Diversity Dimensional Measure Non- Non-Asian Asian Minority Female Minority Female International Broad Field Faculty Faculty Students Students Students Agricultural sciences 0.101 0.124 0.348 0.231 0.196 Biological and health sciences 0.115 0.173 0.362 0.235 0.115 Physical and mathematical 0.059 0.144 0.200 0.318 0.279 sciences Engineering 0.083 0.107 0.281 0.295 0.234 Social and behavioral sciences 0.156 0.150 0.298 0.166 0.230 Humanities 0.172 0.212 0.212 0.192 0.213 The preferences of faculty in the broad fields are very similar across fields. The physical and mathematical sciences place a greater weight on the percentage of students who are female than the percentage of students who from a underrepresented minority. This weighting is reversed for the other fields. None of the fields places a large weight on faculty diversity, although generally a slightly higher weight is placed on the percentage of faculty who are female. The physical sciences and engineering and, to some extent, the social sciences faculty indicate that a higher percentage of international students is beneficial and important to program quality. The relatively high weight for this measure for the humanities reflects high weighting in the foreign language fields and comparative literature. SUMMARY OF THE FINDINGS The findings of the committee fall into three areas: 1. Indicators of research activity are of the greatest importance to faculty in determining program quality by means of the S measures, which are based on the program characteristics that faculty say explicitly are important. In many cases program size is very important when quality is measured by the regression-based, or R measures. 2. Of the student support and outcome characteristics, placement in an academic position and support in the first year are highly weighted. Completion rates and time to degree are not. 3. Faculty view student diversity as important, when considered with other diversity measures, but not as a direct predictor of overall program quality.