A Data-Based Assessment of Research-Doctorate Programs in the United States (with CD) (2011)

Below is the uncorrected machine-read text of this chapter, intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text of each book. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.

6 Some Uses of the Data Students and their families, faculty members, administrators, boards of higher education, trustees, and departments of education, both state and federal, as well as private sector employers and policy analysts and scholars, among others, should find the data in this study of interest. This chapter provides examples of some uses of the study data. POTENTIAL USERS Students Once students know the discipline in which they want to pursue doctoral study, they usually take the next step of discussing with their adviser or a professor in that field what that doctoral study would involve. Here are some examples of the questions that might be asked: • Do I know what I want to specialize in? • Do I want a program in a particular region—for example, near my home? • Do I want a large program or a small program? • Do I want a program in which a high proportion of the faculty has grants? • Are my GRE scores competitive with those of other students in programs that interest me? • Do I want a program in which a high proportion of students complete their degrees in a reasonable period of time? • Do I want a program in which I am likely to find other students and faculty like myself (e.g., who are female or from underrepresented minority groups)? • Do I want a program that funds most of its students in their first year? • Do I want a program that is interdisciplinary? • Do I want a program whose faculty are highly cited? After choosing among the doctoral programs suggested by their adviser, students can then create a spreadsheet of those programs from the online data available from this study, which will allow them compare the programs on the various measures of interest. For example, Table 6-1 shows the rankings and data for 5 of the 34 chemistry programs in universities in the mid-Atlantic area. 73 

74 A DATA-BASED ASSESSMENT OF RESEARCH-DOCTORATE PROGRAMS IN THE U.S. TABLE 6-1 Ranges of Rankings and Data for Five Mid-Atlantic Chemistry Programs Institution Institution Institution Institution Institution A B C D E Overall measures at 5th and 95th percentile points R5 6 6 17 53 32 R95 29 24 49 112 83 S5 5 10 9 88 33 S95 18 38 31 155 89 Dimensional measures at 5th and 95th percentile pointsa RA5 5 8 10 62 20 RA95 21 37 53 146 91 SS5 17 35 5 75 25 SS95 113 134 66 156 114 D5 87 103 109 81 44 D95 141 158 162 138 103 Characteristic Publications per allocated faculty 4.413 4.601 4.136 1.891 2.896 Cites per publication 2.971 2.933 2.504 1.578 2.513 Faculty with grants (%)b 100.0% 88.6% 95.4% 85.9% 90.5% Interdisciplinary faculty (%) 0.0% 71.4% 38.1% 6.3% 18.8% Non-Asian minority faculty (%) 5.0% 0.0% 0.0% 3.7% 2.9% Female faculty (%) 13.6% 16.2% 8.0% 13.8% 17.9% Awards per allocated faculty 13.837 6.475 3.802 0.067 1.907 Average GRE-Q 772 712 769 700 703 First-year students with full support (%) 100.0% 100.0% 100.0% 100.0% 100.0% First-year students with external funding (%) 0.0% 0.0% 22.2% 0.0% 0.0% Non-Asian minority students (%) 1.9% 2.8% 3.2% 5.4% 8.0% Female students (%) 39.1% 39.3% 39.8% 39.7% 42.2% International students (%) 42.7% 23.0% 37.2% 35.8% 45.1% Average Ph.D.’s, 2002–2006 17.4 31.6 20.2 12.8 11.4 Completing within six years (%), 77.8% 49.3% 67.6% 43.3% 41.6% 

SOME USES OF THE DATA 75 Time to degree, full- and part- time (years) 5 5.7 4.9 6 4.3 Students in academic positions (%) 57.0% 44.7% 54.3% 37.8% 48.7% Student work space 1 1 1 1 1 Health insurance 1 1 1 1 1 Number of student activities offered. 16 18 16 16 18 a RA = research activity; SS = student support and outcomes; D = diversity of academic environment. b All percentages are the percentage of total in the relevant group (faculty, students, or Ph.D.’s). The data reveal clearly that these programs are different. The chemistry program in Institution B is large—it graduates almost 32 students a year compared with 20 or fewer for the remaining four programs. In terms of research activity, the first three programs are highly productive, and their range of rankings would likely place them among the top 20 programs in the field. One of the institutions has a prestigious (and likely older) faculty, as measured by awards. With one exception, all of the programs support all of their first-year students. All of the institutions have a time to degree of between four and six years, but in institutions A, D, and E less than 50 percent of their students complete their degrees within six years. Institution A places almost a third of its graduates in academic positions. The comparisons based on the diversity variables are mixed. Women make up more than one-third of the students at all the programs, but the gender diversity of faculty varies. In all of the programs more than 20 percent of the students are international, but none of the programs have more than 10 percent of students from racial or ethnic minorities, and two programs have no minorities in their teaching faculty. This example illustrates that for many uses the data themselves may be more useful than any range of rankings. The temptation, however, will be to use the ranges of rankings. Faculty and Administrators It is hoped faculty and administrators will look at the data, ask what characteristics are important to the purpose at hand, and rank programs accordingly. They may, however, want to look at the illustrative R and S ranges, which are after all constructed from faculty opinions, and try to understand where their programs fall in these illustrative rankings. A detailed understanding of the generation of the rankings of a single program in biochemistry can be obtained in part through Table 6-2, which shows the overall and dimensional rankings for a program in that field of study. TABLE 6-2 An Example of Selected Ranges of Rankings for a Doctoral Program in Biochemistry Range of Overall Rankings Dimensional Rankings Diversity of the Student Support Educational R Rankings S Rankings Research Activity and Outcomes Environment 

76 A DATA-BASED ASSESSMENT OF RESEARCH-DOCTORATE PROGRAMS IN THE U.S. R5 R95 S5 S95 RA5 RA95 SS5 SS95 D5 D95 Program name 11 19 7 31 5 29 7 107 89 134 It is natural to ask at this point: where did these rankings come from? Table 6-3 provides the details for this program, which can be obtained by clicking on the link in the online spreadsheet. 

SOME USES OF THE DATA 77 TABLE 6-3 Calculation of the R and S Rankings for a Single Program Standardized Values of the Variables, for Characteristic the Programa R Coefficients S Coefficients R5 R95 S5 S95 R5 R95 S5 R5 Publications per allocated faculty 1.807 1.619 1.755 1.668 0.059 0.111 0.144 0.139 Cites per publication 1.222 1.221 1.274 1.216 0.102 0.118 0.102 0.103 Faculty with grants (%) -0.437 -1.204 2.071 -1.068 0.018 0.024 0.171 0.172 Interdisciplinary faculty (%) -0.387 -0.425 -0.457 -0.280 0.027 -0.016 0.042 0.039 Non-Asian minority faculty (%) -0.837 -0.818 -0.461 -0.275 -0.059 -0.015 0.009 0.010 Female faculty (%) 0.315 0.675 0.379 1.042 -0.002 0.045 0.015 0.015 Awards per allocated faculty 3.249 3.064 2.973 2.686 0.093 0.140 0.062 0.062 Average GRE-Q 0.015 0.261 0.031 0.390 0.101 0.092 0.081 0.079 First-year students with full support (%) 0.433 -0.233 0.158 0.779 0.027 -0.012 0.057 0.056 First-year students with portable fellowships (%) 0.846 1.044 0.696 1.006 0.064 0.037 0.047 0.046 Non-Asian minority students (%) -0.922 -0.961 -1.097 -1.001 0.023 0.008 0.020 0.020 Female students (%) -0.242 -0.266 0.080 -0.368 -0.051 -0.064 0.017 0.017 International students (%) 0.196 0.630 0.194 0.070 -0.022 -0.008 0.008 0.009 Average Ph.D.’s, 2002–2006 -0.494 -0.967 -0.552 -0.405 0.117 0.121 0.026 0.027 Completing degree within six years (%) -0.387 2.088 -0.381 -0.362 0.035 -0.035 0.055 0.056 Time to degree, full- and part- time -0.764 -1.145 -0.636 -0.260 0.016 0.018 -0.031 -0.030 Students in academic positions (%) -0.610 -1.287 -1.235 -1.493 -0.047 0.004 0.074 0.078 Student work space 1 1 1 1 -0.025 -0.067 0.004 0.005 Health insurance 1 1 1 1 -0.028 0.003 0.005 0.005 Number of student activities offered 1.950 0.293 0.876 0.495 0.083 0.061 0.032 0.034 R rating 0.671 0.578 S rating 0.858 0.322 R ranking 11 19 S ranking 7 31 a Because of the Monte Carlo technique used to generate the ranking ranges, in some cases the R95 (or S95) standardized measure may be smaller than the R05 (or S05) measure for a particular variable. 

78 A DATA-BASED ASSESSMENT OF RESEARCH-DOCTORATE PROGRAMS IN THE U.S. Table 6-3 shows the standardized values of the variables for the particular estimation that resulted in the program’s rank for the 5th percentile and the 95th percentile for each methodology. Uncertainty has been taken into account for the value of each variable, and 500 separate sets of half-samples of the raters have been selected for each measure. The values of the variables have been selected from a random distribution within which the variable values can vary by plus or minus 10 percent or by the extent of their actual variation, if data were collected on that variation.1 It is apparent from Table 6-2 that the range of S rankings is larger than the range of R rankings for this program, and that it does especially well on the research activity dimensional measure and less well on the other two measures. Closer examination reveals that the coefficients on both measures place a high weight on the research characteristics, where this program is well above average. Administrators with budget allocation responsibility will be particularly interested in an analysis of all the programs of one university or in one division of a university. Keeping in mind that each discipline is different, administrators may find the NRC data useful in making allocation decisions designed to shorten the time to degree, for example, or to improve completion rates in particular programs, or to enhance support for faculty research. But, again, the NRC data are just a start, and universities will want to supplement and update them. Comparison of a given program with those in a higher echelon will likely indicate that ratings will improve if the research impact of the faculty improves. Although hiring “stars” is one obvious solution, another is to enable faculty to increase their research productivity by seeding new research programs or giving faculty more time to spend on research. These data provide some quantitative measures to inform decisions about the balance between research and teaching for a given program. But, again, the NRC dataset is only part of the information needed for budget allocations. Indeed, the NRC data are only one way of measuring a doctoral program, and in all fields the ranges of rankings are heavily influenced by metrics about the research productivity of the faculty in terms of publications and citations as well as grants and awards. No effort has been devoted to assessing the outcomes of graduate education or to determining the effectiveness of the doctoral research experience in preparing students for a life of scholarly inquiry. Nor has effort been given to measuring or assessing the need for doctoral studies in any given area, or of the career outcomes of students who follow any particular course of study. These considerations also influence administrative decisions about resource allocations to graduate programs. Additional central university support may target not the highest-ranked programs or the lowest, but rather the units best equipped to use additional funding in a productive fashion. Boards of Higher Education and Trustees As the ultimate authority on which doctoral programs should be offered by a university or universities in each state, boards of higher education and boards of trustees are ultimately responsible for the quality of doctoral programs. They are also aware of the full range of doctoral programs in a state or university. They may find that a program does not rank highly, but it plays an important role in producing Ph.D.’s who well serve an industry that 1 The questionnaires in Appendix D reveal whether variable values for multiple years were collected. 

SOME USES OF THE DATA 79 is important to the state’s economy. Some programs offered in the same field by various institutions may display a measurable quality differential that is supported by differences in one or more selected characteristics, thereby indicating that consolidation could be the answer. However, resolving such matters requires fine-grained study of the programs and more detailed analysis than can be provided by the NRC data alone. Private Sector Employers Academia is no longer the majority employer of Ph.D.’s. They are employed by industry, nonprofit organizations, government, and consulting firms. Rather than just the programs with the highest rankings, these employers may wish to look for programs that tend to aim at the nonacademic sector, as well as programs that are more diverse and focused, as evidenced by higher completion rates and shorter times to degree. Policy Analysts and Scholars The data from this study contain a wide selection of information needed for policy analysis of graduate education. Additional data from the program, student, and faculty questionnaires will be made available in a public-use dataset. How to obtain access to the dataset will be announced following release of this report. These data will contain tabulations of the answers to most questions on the questionnaires aggregated by program, provided that this aggregation preserves individual confidentiality. Upon publication of this report, the analytic tables showing the derivations of the rankings for each program, and the master tables for programs by discipline, will be available. If a researcher needs individual-level records, these will be made available to researchers who agree to respect the confidentiality of respondents and sign a confidentiality agreement with the NRC. Researchers are warned, however, that the data not shown in the master tables have spottier response rates. A WORKSHOP ON ANALYTIC USES OF THE DATA A workshop on the ways in which universities and researchers have used the data was held after the release of the report and data tables. The committee believes it is very important that ways of analyzing the data become widely known throughout the graduate and higher education research community and to enhance their usefulness.