Now Available: Excel Data Table (revised 4/29/2011)

See Summary of Changes in right menu for more information.

A Data-Based Assessment of Research-Doctorate Programs in the United States provides an unparalleled dataset that can be used to assess the quality and effectiveness of doctoral programs based on measures important to faculty, students, administrators, funders, and other stakeholders.

The data, collected for the 2005-2006 academic year from more than 5,000 doctoral programs at 212 universities, covers 62 fields. Included for each program are such characteristics as faculty publications, grants, and awards; student GRE scores, financial support, and employment outcomes; and program size, time to degree, and faculty composition. Measures of faculty and student diversity are also included.

The book features analysis of selected findings across six broad fields:  agricultural sciences, biological and health sciences, engineering, physical and mathematical sciences, social and behavioral sciences, and humanities, as well as a discussion of trends in doctoral education since the last assessment in 1995, and suggested uses of the data . It also includes a detailed explanation of the methodology used to collect data and calculate ranges of illustrative rankings.

Included with the book is a comprehensive CD-ROM with a data table in Microsoft Excel. In addition to data on the characteristics of individual programs, the data table contains illustrative ranges of rankings for each program, as well as ranges of rankings for three dimensions of program quality: (1) research activity, (2) student support and outcomes, and (3) diversity of the academic environment.

As an aid to users, the data table is offered with demonstrations of some Microsoft Excel features that may enhance the usability of the spreadsheet, such as hiding and unhiding columns, copying and pasting columns to a new worksheet, and filtering and sorting data. Also provided with the data table are a set of scenarios that show how typical users may want to extract data from the spreadsheet.

PhDs.org, an independent website not affiliated with the National Research Council, incorporated data from the research-doctorate assessment into its Graduate School Guide. Users of the Guide can choose the weights assigned to the program characteristics measured by the National Research Council and others, and rank graduate programs according to their own priorities.

.

Purchase and Free Download Options
Purchase Options
cover image

Paperback
322 pages, 8.5 x 11

0-309-16030-8
978-0-309-16030-8

Price:$99.95
add to cart

A Data-Based Assessment of Research-Doctorate Programs in the United States with CD (2011)

The book features analysis of selected findings across six broad fields: agricultural sciences, biological and health sciences, engineering, physical and mathematical sciences, social and behavioral sciences, and humanities, as well as a discussion of trends in doctoral education since the last assessment in 1995, and suggested uses of the data . It also includes a detailed explanation of the methodology used to collect data and calculate ranges of illustrative rankings. Included with the book is a comprehensive CD-ROM with a data table in Microsoft Excel.

Free Download Options
cover image

Final
322 pages

download PDF

Related Links:

 A Data-Based Assessment of Research-Doctorate Programs in the United States (2011)
Jeremiah P. Ostriker, Charlotte V. Kuh, and James A. Voytuk, editors; Committee to Assess Research-Doctorate Programs; National Research Council

A Data-Based Assessment of Research-Doctorate Programs in the United States presents findings from data collected for the 2005-2006 academic year from more than 5,000 doctoral programs at 212 universities. This report also includes illustrations of how the data can be used to produce rankings of doctoral programs based on measures important to faculty, students, administrators, funders, and other stakeholders.
xl icon

Excel Spreadsheet
35 MB
2011

Sign In

A Data-Based Assessment of Research-Doctorate Programs in the United States: Excel Data Table (revised 4/29/2011)
Committee to Assess Research-Doctorate Programs

The Data Table in Excel includes data from more than 5,000 doctoral programs offered at 212 universities across the United States. This rich resource allows evaluation and comparison of programs in areas such as faculty research activity, student support and outcomes, and diversity of the academic environment. Three formats of the spreadsheet are available. The Windows and Excel 2004 and 2011 for Mac versions are optimized for users through the use of macros that enable customized filtering and click-through to background data. As Excel 2008 for Mac does not allow for the use of macros, this version offers reduced functionality but contains the same extensive data set. It can be used by anyone whose personal or institutional computer security settings make it difficult to work with files containing macros.

DATA TABLE DEMOS
The functionality described in these demos applies to the Excel for Windows and Excel 2004 and 2011 for Mac versions of the spreadsheet only.

Spreadsheet Basics

How to Hide and Unhide Columns

 Watch Demo

How to Copy and Paste Columns to a New Worksheet

 Watch Demo

How to Filter and Sort Data

Watch Demo

User Scenarios

Scenario 1
The chair of a microbiology department wants to compare her program with other microbiology programs nationally, with an emphasis on faculty research productivity.

Watch Demo

Scenario 2
A dean of engineering is tasked with identifying ways to increase racial and gender diversity in his institution's engineering programs. He wants to begin by investigating how his Ph.D. programs compare in faculty and student diversity with engineering programs across the country, field by field.

Watch Demo


Scenario 3
A college junior is exploring Ph.D. programs in English literature. She is willing to go anywhere in the United States, but she wants a small or mid-sized program.

Watch Demo

Scenario 4
A state board of higher education in the Midwest is evaluating the performance of the public institutions in its state compared to other public schools in the region. The board is particularly interested in the support doctoral programs give to help students complete their degrees.

Watch Demo

Scenario 5
A provost at a large research university is considering a request for two new tenure-track faculty positions in his chemistry program in response to a surge in graduate enrollment. Before deciding, he wants to know more about the number and composition of the faculty in other chemistry programs of comparable size.

Watch Demo
On This Site

Technical Support
  • For technical support for the Data Table spreadsheet, please:
  • E-mail resdoctechsupport@nas.edu

  • Call
    202-630-0140
    Monday-Friday, 8:30 a.m. – 5:00 p.m.

Share Tools
E-mail This Facebook Digg StumbleUpon Twitter More






















Summary

A Data-Based Assessment of Research Doctorate Programs in the United States provides an unparalleled dataset collected from doctoral institutions, doctoral programs, doctoral faculty and public sources that can be used to assess the quality and effectiveness of doctoral programs based on measures important to faculty, students, administrators, funders, and other stakeholders. The committee collected 20 measures that include characteristics of the faculty, such as their publications, citations, grants, and diversity; characteristics of the students, such as their GRE scores, financial support, publications, and diversity; and characteristics of the program, such as number of Ph.D.s granted over five years, time to degree, percentage of student completion, and placement of students after graduation. The data were collected for the academic year 2005-6 from over 4,000 doctoral programs at 212 universities. These observations span 62 fields, and the research productivity data are based typically on a five-year interval. Some datasets (such as publications and citations) go as far back as 1981. Information on enrollments and faculty size were also collected for 14 emerging fields.
The program-level data, collected using questionnaires, reflect the size, scope, and other components of each program, as well as, financial aid and training practices. In addition, data were collected about time to degree and completion rates and whether the program followed the progress of its students after completion. The faculty questionnaire, which was sent to all faculty identified as doctoral faculty by their institutions, collected data on funding, work history, and publications, as well as on demographic characteristics. One section of the questionnaire asked the respondent to rate the relative importance of program, faculty productivity, and demographic characteristics in assessing program quality, and then to rate the relative importance of components within these larger categories. The student questionnaire asked about student educational background and demographic characteristics, as well as research experiences in the program, scholarly productivity, career objectives, and satisfaction with a variety of aspects of the program.
This report also includes illustrations of how the dataset can be used to produce rankings of doctoral programs, based on the importance of individual measures to various users. Two of the approaches provided in the report are intended to be illustrative of constructing data-based ranges of rankings that reflect values to assess program quality determined by the faculty who teach in these programs. Other ranges of rankings can also be produced reflecting the values of the users. The production of rankings from measures of quantitative data turned out to be more complicated and to have greater uncertainty than originally thought. As a consequence, the illustrative rankings are neither endorsed nor recommended by the National Research Council (NRC) as an authoritative conclusion about the relative quality of doctoral programs. Nevertheless, the undertaking did produce important insights that are useful as stakeholders use the dataset and the illustrations to draw conclusions for their own purposes. The illustrative approaches illuminate the interplay between program characteristics and the weights based on values of users that go into constructing rankings. The ranges of rankings that are shown convey some, but not all, of the uncertainties that can be estimated in producing rankings based on assigning values to quantitative measures.
The reader who seeks a single, authoritative declaration of the ?best programs? in given fields will not find it in this report. The reason for this outcome is that no single such ranking can be produced in an unambiguous and rigorous way. To create illustrative rankings, the committee explored several approaches to evaluate and rate programs, with the subsequent rankings reflecting an ordered list of ratings from high to low. Program ratings depend on two things, namely the characteristics of the program (e.g., number of faculty, number of publications, citations, and other quantifiable measures) and the weighting, or value, that faculty assigned to each characteristic. The committee determined the weights to apply to important characteristics by two different methods based on faculty inputs. One method involved asking direct questions about what characteristics are important and how they should be weighed, while the second used an implicit method to determine the weights based on evaluations of programs by faculty raters. The results of these two approaches are different, and are presented separately in the report.
The committee also developed three other rankings based on separate dimensions of the doctoral programs. All five approaches, which are explained in more detail in the following paragraphs, have strengths and deficiencies. The committee is not endorsing any one approach or any one measure or combination of measures as best.�� Rather, the user is asked to consider the reason a ranking is needed and what measures would be important to that ranking.�The different measures should then be examined by the user and given appropriate weights, and the user should choose an approach that weights most heavily what is important for that user?s purpose.� As the committee has stressed repeatedly, the user may take the data that the study provides and construct a set of rankings based on the values that the specific user places on the measures.
The faculty survey on the relative importance of various measures yielded weights that are used to develop one illustrative ranking, the S-ranking (for survey-based), for which we present ranges for each program. On a separate questionnaire, smaller groups of randomly selected faculty in each field were asked to rate programs from a sample of doctoral programs. The results of the regression of these ratings on the measures of program characteristics are used to develop another range of illustrative rankings, the R-rankings (for regression based). The ranges and weights for these two ways of calculating rankings ? one direct (S-ranking) and one indirect (R ranking) -- are reported separately and provided in on-line spreadsheets (http://www.nap.edu/rdp) with a guide to the user.
The ranking methodology utilized by the committee in these illustrative approaches has been chosen to be based on faculty values. This decision was made because perceived quality of the graduate doctoral program in a field is typically based on the knowledge and views of scholars in that field. Dimensional measures in three areas --research activity, student support and outcomes, and diversity of the academic environment -- are also provided to give additional illustrative ranges of rankings of separate aspects of doctoral programs.
An earlier version of the methodology is described in the Methodology Guide. 1 The primary change made since the Guide was prepared was the decision to provide separate R and S rankings as illustrations rather than combining them into one overall ranking. Although the relative importance of measures varies in different fields, per capita measures for publications, citations, grants, and awards are strongly preferred by faculty as key measures of program quality. One interesting and important difference between the weights that result in the R and S rankings is that the one measure of program size ? the average number of Ph.D.s granted over the previous five years ? is often the largest weight in the R rankings and relatively small in the S rankings. Faculty appear not to assign as much importance to program size when assigning weights directly compared to the importance of program size in weights assigned indirectly based on their rating of programs. Program size, while not likely to be a direct cause of higher program quality, may serve as a surrogate for other program features that do exert positive influences on quality.
The illustrative ranges of rankings are instructive for several reasons. Most importantly, they allow for comparison of programs in a field in a way that recognizes some?but not all--of the uncertainties and variability inherent in any ranking process. This uncertainty and variability come partially from variability in rater opinions, variability in data from year to year, and the error that accompanies the estimation of any statistical model. The ranges that are provided cover a broad interval of 90%, which is another change from the original methodology report. There are other sources of uncertainty that are not captured in the ranges presented in the illustrative rankings. These additional sources include uncertainty in the model for assessing quality based on quantitative measures as well as the uncertainty that the 20 measures capture the most relevant factors that are needed to assess quality in a particular field.
The current approach does have the advantage of collecting exactly the same categories of data from all programs being assessed, and uses those data to calculate ratings based on the relative importance of measures as established by doctoral faculty. This approach, however, entails a key vulnerability. In the current methodology, when program measures in a field are similar, program differences in the range of rankings can depend strongly on the precise values of the input data, and so are quite sensitive to errors in those data. We have worked to assure the quality of the measures used to generate rankings and have tried to minimize such errors in the data collection. But errors can arise from clerical mistakes and possible misfit between the measures and the data. They can be caused by misunderstandings by our respondents concerning the nature of the data requested from them, or they may be embedded in the public data-bases that we have utilized. Some of the key publication sources in a field or subfield may not be included in the public database that was used.2 Despite our efforts we are certain that errors in input data remain, and these errors will propagate through to any rankings.
We believe, however, that careful error-checking both by the NRC and by the doctoral programs being assessed has produced a collection of data of great usefulness in understanding doctoral education, both by providing a means for users to assess the quality of doctoral programs and for what the detailed analyses of the data themselves can tell us. The data permit comparisons of programs on the basis of several program characteristics, each of which provides an important vantage point on doctoral education. The ranges of illustrative rankings, because of the values expressed in the faculty questionnaires, emphasize measures of faculty productivity. But the data enable comparisons using any of the categories in which data were collected. Doctoral programs can readily be compared, not only on measures of research activity in a program, but, for example, on measures of student support, degree completion, time to degree, and diversity of both students and faculty. These data will become even more valuable if they are updated periodically and made current, which the committee strongly encourages.3
The work that has gone into producing this assessment of doctoral programs has raised the level of data collection vital to understanding the broad dimensions of doctoral education in the United States. It would be difficult to overstate the efforts required of universities and doctoral programs to produce, check, and recheck the data collected for this assessment. The extensive reliance on data in this assessment called for the collection of an enormous amount of information that has not been routinely or systematically collected by doctoral programs in the past. Graduate schools, institutional researchers, program administrators, and individual doctoral faculty all contributed countless hours to compiling, clarifying, and substantiating the information on which this assessment is based. As a result, we believe that this focus on data collection in and of itself by participating universities and their programs has created new standards, and improved practices, for recording quantitatively information on which qualitative assessments of doctoral programs can be based.
With the abundance of data included in this assessment comes a great deal of freedom in determining which information is most useful to individual users. We are particularly hopeful that the wealth of data collected for this assessment will encourage potential applicants to doctoral programs to decide what characteristics are important to them and will enable them to compare programs with respect to those characteristics. Potential doctoral applicants, and, indeed, all users of this assessment, are invited to create customized assessment tables that reflect their own values.4
In attempts to rank doctoral programs, sports analogies are especially inappropriate.
Finally and most importantly, this study is a tool that can be useful to administrators, faculty, students, and others with an interest in the characteristics of doctoral programs.
doctoral programs have attracted students from around the world for many years.
Whatever their interest, all users will find that they have access to information about doctoral programs that was not available in the past.
have developed targeted programs to stimulate the recruitment, retention, and success of women and minorities in doctoral programs.
Since then, however, reputational measures have been repeatedly used to assess the quality of doctoral programs.
The committee has worked for more than three years on arriving at a satisfactory methodology for generating rankings for doctoral programs.
Doctoral programs that do not have storied histories may find it difficult to demonstrate their current strengths.
It is highly unlikely that rankings calculated from composite measures will serve all or even most purposes in comparing the quality of doctoral programs.
33 His survey, which appeared in a report to the Association of American Colleges, was aimed at constructing rankings of doctoral programs.
These changes include the increasing interdisciplinarity of U.S doctoral programs.
presents data that provide, for the first time in one place, basic information about many aspects of doctoral education in the United States.
WHAT THIS STUDY HAS REVEALED Doctoral education in the United States is a far-flung and varied enterprise.
Following this release, further analyses will be conducted by the committee and other researchers and discussed at a conference focusing on doctoral education in the United States.
DIVERSITY OF FACULTY AND STUDENTS IN DOCTORAL PROGRAMS
The years since 1993 have been characterized by the increasing interdisciplinarity in doctoral programs and the blurring of the boundaries across fields, which has been manifested in a variety of ways.
Students from other countries enrolled in the entire range of doctoral programs were surveyed for this study.
Both of these trends point to a change in the composition of participants in doctoral education in the United States.
As universities across the globe compete with increasing intensity for the faculty and students who will advance the knowledge economy of the future, it is important that we take stock of the enormous value represented by the United States research doctorate programs.
WHY ASSESS DOCTORAL PROGRAMS
Several important dimensions of doctoral programs become much clearer when viewed from the vantage point of reliable data.
The data were collected for the academic year 2005-6 from over 4,000 doctoral programs at 212 universities.
Perceptions of the quality of doctoral programs are built
One measure, the S ranking, is based on a survey of the importance to faculty in a given field of the general characteristics of doctoral programs.
As a result, most of the data in this study are related to doctoral programs and their faculty.
Many successful international doctoral students stay in the United States.
doctoral education, is the responsibility of research universities, of state and federal government policy makers, of powerful funding agencies and foundations, and of all other stakeholders in the American doctorate and the vast research enterprise that depends on it.
More broadly, since the 1995 study doctoral education has benefited greatly from dramatic increases in enrollments of international students and of domestic minorities and women.
It became clear, however, that if doctoral education was to serve the population equitably, and if domestic doctoral production was to adequately meet the research and professorial needs of the nation, universities must increase the participation in doctoral education of underrepresented minorities and women.
Even when the NRC study findings cannot answer all the important questions that the many constituencies of doctoral education will bring to it, the study will put them in a better position to know the questions that they then need to pose to the programs they are considering.
As a consequence, the illustrative rankings are neither endorsed nor recommended by the National Research Council (NRC)
The data permit comparisons of programs on the basis of several program characteristics, each of which provides an important vantage point on doctoral education.
A table showing the correlation of the medians of the two measures for programs in each discipline appears in Appendix G Meanwhile, measures other than the range of R rankings and S rankings may be important to others engaged in doctoral education, such as granting agencies and the students themselves, and as such should not be ignored.
Still in use today, the traditional measures for assessment of doctoral education have been time to degree and completion rates35
It contrasts the methodology and results from the 1995 study with the current study and then presents a description of important findings about doctoral education in 2006
or that have emerged from earlier interdisciplinary work present knotty problems with program classification and with the variety of ways in which different universities organize doctoral education.
Because of differences between the definition of faculty in 1993 and 2006, we cannot strictly compare faculty sizes, but it appears that the number of faculty involved in doctoral education has also grown in most programs.
As this committee understands, not only must an enterprise of this significance be operated effectively, but also constituencies crucial to the support of doctoral education must have access to the information that can help provide reliable assessments of its effectiveness.
Likewise, using such information, policy makers must be sensitive to the changing characteristics, or evolution, of doctoral education, because such changes are likely to be a consequence of the prominence of doctoral education in the national system of innovation.
position is facing substantial challenges, from a growing emphasis on doctoral education in other countries to financial constraints stemming both from the economic downturn of 2008
For doctoral education another administrative unit is of central importance: the graduate program.
Indeed, the domestic nonminority male population in doctoral education has decreased in both numbers and as a percentage of total doctoral enrollments.
universities for the last half-century, has introduced ambitious programs to expand and strengthen doctoral education in its own universities.
Taken together, these criteria ensure that the field is a significant presence in doctoral education and that there are enough programs nationwide to make comparison meaningful.
They also demonstrate the capacity to bring into doctoral education vital components of the national citizenry historically underrepresented in Ph D.
Finally, efforts to determine whether doctoral education is living up to its promise call for an evaluation of whether it has done so by expanding domestic sources of talent, improving time to degree, and raising rates of completion.
Doctoral education is intimately involved in the creation of scholars whose ideas will shape both future innovations and how Americans use and understand innovation as it changes their lives.
For example, the European Union nations have recognized how important the knowledge and skills developed through doctoral education are to building a twenty-first century economy, and so those nations have given high priority to strengthening the doctoral education they offer.
The biological and health sciences taxonomy recognizes this fact and provides for the inclusion of such programs among the basic biological research doctorate programs.
Some fields have relatively few subspecialties, and the basic predissertation years of doctoral education are similar for all students in the program.
This finding may reflect a deeper faculty involvement in doctoral education, or it may be partially a result of definitional changes between the two studies29
For both public and private universities, doctoral education is expensive in the commitment of time and dollars both by those engaged in the enterprise and by its funders.
In addition, assessments were conducted of three separate dimensions of doctoral education: (1)
In the more than 80 years since the Hughes report, doctoral education has changed tremendously in size, number of fields, and the nature of employment destinations of Ph D.
Far more women are in doctoral education now than in the 1980s and 1990s, and in some fields once dominated by males, women doctoral candidates now are the majority.
However, doctoral education is in fact concentrated in relatively few institutions whose programs have many students and faculty.
This quantitative criterion is designed to ensure that doctoral education and research are a central part or a mission of any included program.
These rankings address three specific dimensions of doctoral education: (1)
With the enormous importance of and investment in doctoral education comes the need for accountability, because many different sectors of the U.S.
A major challenge faced by this study was to find measures that do justice to the growth of interdisciplinarity in doctoral education.
Doctoral education trains the professors of the future
The demographic group that has not shown gains during this period is the group that was long dominant in doctoral education
Doctoral education is at the heart of the U.S.
This database should prove to be of great interest to researchers on doctoral education9.
Through the 1970s doctoral education in almost all fields was largely a man
These data have been used to produce illustrative ranges of rankings of research doctorate programs aimed at reflecting the values of the faculty who teach in these programs.
According to the Institute for International Education, the absolute numbers of enrollments of international doctoral students increased from 100,092 in 2003
Many excellent research doctorate programs in the basic biomedical sciences are located in colleges or schools of medicine.
Compounding the issue was an escalation in the level of competition worldwide for the best international doctoral students.
However, the committee recognizes that research doctorate programs have grown and diversified since 1993 and that research doctorates are not limited to the arts and sciences.
This study cannot, of course, provide a comprehensive understanding of these research doctorate programs.
Committee on an Assessment of Research Doctorate Programs directed its research at fulfilling the following task:
But there are limits to the reliability of a picture of graduate program quality and opportunity that is based on reputation alone.
Race, Ethnicity, and Gender of American Doctoral Students
How the study was designed to achieve these objectives is described briefly in Chapter 2 and in considerable technical detail in A Guide to the Methodology of the National Research Council Assessment of Doctorate Program.
leadership in innovation, including recently from the National Research Council, Rising Above the Gathering Storm: Energizing and Employing America for a Brighter Economic Future, which focuses especially on the need to improve U.S.
With that in mind, six months after the release of this report and the accompanying data, the National Research Council will hold a conference at which researchers and others who have used the data will report on the uses they have made of them.
Ranking programs based on quantitatively based estimates of program quality is a highly complex task.
Furthermore, countries that provide doctoral study in English, such as Great Britain, Canada, and Australia, capitalized on the situation in the United States by moving rapidly to recruit more international students.
The student questionnaire asked about student educational background and demographic characteristics, as well as research experiences in the program, scholarly productivity, career objectives, and satisfaction with a variety of aspects of the program.
One section of the questionnaire asked the respondent to rate the relative importance of program, faculty productivity, and demographic characteristics in assessing program quality, and then to rate the relative importance of components within these larger categories.
The National Research Council
The faculty questionnaire collected data on funding, work history, and publications, as well as on demographic characteristics.
Citizens or Permanent Resident Males, in Selected Broad Fields, 1993 and 2006 Broad Field 1993 2006 Engineering 1,608 1,269 Physical sciences, math and computer sciences, and
Program size, while not likely to be a direct cause of higher program quality, may serve as a surrogate for other program features that do exert positive influences on quality.
In addition to international competitive forces, strong drivers in the United States are underlying efforts to improve the quality and efficiency of graduate programs.
One section of the questionnaire asked the respondent to rate the relative importance of program, faculty, and demographic characteristics to program quality.
Measures put in place after the terrorists attacks of September 11, 2001, not only made entry into the United States for study more difficult, but may also have had a chilling effect on the interest of international students in pursuing graduate study in the U.S.
An assessment of the quality and characteristics of research-doctorate programs in the United States will be conducted.
The student questionnaire, administered to advanced21 students in physics, chemical engineering, neuroscience, economics, and English, asked about student educational background and demographic characteristics, as well as research experiences in the program, scholarly productivity, career objectives, and satisfaction with a variety of aspects of the program.
Yet on balance the committee believes that the core educational experience of doctoral students takes place within a program that embraces both the course work that they experience with multiple members of the faculty and the concentrated research experience within the laboratory of one or several faculty members or in seminars or in individual discussion with faculty.
American graduate education draws students from across the United States and around the globe, particularly in the science, technology, engineering, and mathematics (STEM)
Faculty asked to fill out questionnaires providing information about their scholarly records responded at an exceptionally high rate, as did doctoral students in the five disciplines selected for an experimental student survey.
Although the relative importance of measures varies in different fields, per capita measures for publications, citations, grants, and awards are strongly preferred by faculty as key measures of program quality.
the numbers of students enrolled in the programs that participated in both studies have increased in some broad fields (in engineering by 4 percentage points, and in the physical sciences by 9 percentage points)
Like all large organizations, research universities in the United States consist of many related parts.
The program questionnaire collected data relevant to many of these questions.
When international graduates return to their country of origin, they take back with them an understanding of American culture and values that is important in clarifying and stabilizing the place of the United States in the global political and economic culture.
Furthermore, in an illustrative manner, the assessment analyzes and combines these data to create two ranges of rankings based on overall measures of program quality that were derived from faculty perceptions of program quality approached in different ways.
The first NRC assessment, published in 1982,15 was a rich source of data for educational planners and policy makers, as well as a source of reputational ratings of perceived program quality obtained from raters who were acquainted with the programs in the discipline16
Graduate schools, institutional researchers, program administrators, and individual doctoral faculty all contributed countless hours to compiling, clarifying, and substantiating the information on which this assessment is based.
It is this portion of the charge that called for constructing program ratings and deriving rankings from them that reflect program quality.
research universities have benefited greatly from the influx of doctoral students from other nations.
Which universities provide adequate health insurance programs and child care services for their doctoral students
The committee calculated ranges of rankings for three of these aspects: research activity, student support and outcomes, and diversity of the academic environment.
Federal agencies also invest considerable sums of money to support doctoral students as fellows, trainees, and research assistants, as do private foundations.
Thus this study includes six agricultural fields in the agricultural sciences category and one agricultural field (agricultural and resource economics)
Corporations depend on highly trained doctoral students ready to bring cutting-edge technology and science to their labs and offices.
Will newly enrolled doctoral students be expected to join a graduate employees union if they attend University X
plus expanded coverage of completion, student service in teaching and research, support services provided, student costs and financial support, interdisciplinarity, and postdoctoral study a Earlier studies relied on a program questionnaire and a faculty
The continued success of the United States in this increasingly competitive arena, which is crucial to sustaining the excellence of U.S.
All are essential to scientific discovery, technological innovation, and cultural understanding in the United States and across the globe.
Faculty members would demonstrate the effort they put into each program primarily by stating the number of doctoral students whose dissertations they advised or on whose doctoral committees they served.
The faculty questionnaire, which was sent to all faculty identified as doctoral faculty by their institutions, collected data on funding, work history, and publications, as well as on demographic characteristics.
The rankings were calculated using the opinions of faculty in each program area of both what was important to program quality in the abstract and, separately, how experts implicitly valued the same measures when asked to rate the quality of specific programs.
And it seems increasingly true that faculty sponsors of doctoral students have a greater influence on the next steps in their careers than the program faculty as a whole.
In absolute numbers, in the broad fields in this study the total number of doctoral degrees awarded to white, non-Hispanic U.S.
In terms of completion, over 50% of students complete in six years or less in the agricultural sciences and in engineering, but a smaller percentage does so in the other broad fields.
And China, which has provided large numbers of superb doctoral students for U.S.
Questionnaires were sent to advanced8 doctoral students in programs in five fields
Two of the approaches provided in the report are intended to be illustrative of constructing data-based ranges of rankings that reflect values to assess program quality determined by the faculty who teach in these programs.
Design specific measures along separate dimensions of program quality.
Are funds available to doctoral students for travel to conferences or for research
Because faculty may not be in complete agreement on the importance of the different measures or the rating of sampled programs, differences in views were reflected by repeatedly resampling the ratings and, for each resampling, calculating the resulting weights or overall program ratings.
For these emerging fields the committee collected data only on the number of faculty (core, new, and associated)
Faculty appear not to assign as much importance to program size when assigning weights directly compared to the importance of program size in weights assigned indirectly based on their rating of programs.
These measures summarize the program characteristics of research activity, student treatment and outcomes, and diversity of the academic environment.
Despite these significant gains, underrepresented minorities are still a small proportion of students in many areas of doctoral study
Reputation, although it has the advantage of reflecting dimensions of program quality that are difficult to quantify, may also be dated and include halo effects
Programs do well in supporting students to attend professional and scholarly meetings and, in the science and engineering fields, over 35% have published articles in refereed journals while still enrolled in their doctoral program.
Faculty were asked, on the one hand, to identify the measures that were important to program quality and then asked, on the other, to rate a stratified sample of programs in their fields.
To be included, a measure had to be one that the participating universities either collected in the course of regular institutional research and management, such as enrollment counts, or that the committee felt should be known by a responsible doctoral program, such as the percentage of entering students who complete a degree in a given amount of time.
In addition to these, the current study fielded an institutional questionnaire, a student questionnaire in five fields, and an extensive faculty questionnaire.
Faculty appear to not assign as much importance to program size when assigning weights directly as when assigning them indirectly based on their rating of programs.
For fields in engineering, physical sciences, humanities, and social sciences, where comparisons between the previous study and this one are possible, we find that:
The measures viewed as most important to the quality of a doctoral program are related primarily to faculty research productivity.
Even though the data collected in 2006 for this NRC study are already dated, they will increase in usefulness as long as they are regularly updated.
the collection of quantitative data through questionnaires administered to institutions, programs, faculty, and admitted to candidacy students (in selected fields)
Outside of the biological and health sciences, the program as the unit of analysis also does some injustice to new, interdisciplinary programs that have not been sufficiently filtered into a standard curriculum and a standard method of organizing the educational experience.
For example, the plant sciences include programs that may be named agronomy, horticulture, plant pathology, or crop sciences at different institutions; the animal sciences include programs that might be named dairy science, animal science, or poultry sciences at different institutions.
For the fields surveyed in this study, NSF data indicate that the largest gains in underrepresented minorities have been in the humanities and the biological sciences42
has produced rapidly evolving and highly differentiated ways of organizing graduate education in this field.
Other questions might be: what is the record of research productivity as determined by objective measures among the faculty in any given doctoral program
The ranges of illustrative rankings, because of the values expressed in the faculty questionnaires, emphasize measures of faculty productivity.
A Panel on Data Collection composed of graduate deans and institutional researchers was tasked with drafting questionnaires for this study.
The current approach does have the advantage of collecting exactly the same categories of data from all programs being assessed, and uses those data to calculate ratings based on the relative importance of measures as established by doctoral faculty.
capacity to develop international research partnerships is greatly strengthened by the presence of former students from American universities in key positions in laboratories and universities around the world.
level, the committee found evidence of graduate students identifying their own intellectual roots or heritage with the laboratory supported by their graduate thesis adviser or the professor who organizes a laboratory.
Universities do not follow one standard method of organizing graduate education.
Since the last NRC study was published in 1995 (based on data collected in 1993)
Programs with a high level of research activity have more faculty with research funding, and they typically exist in research universities with higher levels of available support.
The results of the regression of these ratings on the measures of program characteristics are used to develop another range of illustrative rankings, the R-rankings (for regression based)
In most programs students apply directly for admission to the doctoral program without having first obtained a master
Other programs, however, do not admit students to the doctoral program until they have satisfactorily completed a master
International students are well over 40% of students in the agricultural sciences (42%)
These programs correspond to six broad fields and 59 different academic disciplines.
The rate of growth from 1993 to 2006 for the humanities was 6.4 percent and 5.2 percent for biological sciences, compared with only 0.4 percent growth rate in the agricultural sciences for the same period.
Even when reputations for high quality are soundly based and current, they may not help guide prospective students to the best fit for the needs and ambitions they bring to doctoral study.
How many students enroll for each one accepted for doctoral study in electrical engineering at University C, and what range of Graduate Record Examination (GRE)
In contrast to classifying graduate programs, classifying academic disciplines is comparatively straightforward because of the reasonably high level of consensus within a field about its general boundaries and its major subspecialties and subcategories.
student support and outcomes, a measure that reflects program characteristics that are specifically relevant to the student experience; and (3)
This decision was made because perceived quality of the graduate doctoral program in a field is typically based on the knowledge and views of scholars in that field.
the survey research firm engaged for data collection, spent many months refining the questionnaires that organized the information collected from universities.
This result holds despite our reliance in the current study on per capita measures of scholarly productivity.
This NRC study has engaged the surveyed universities in an unprecedented effort to identify the most appropriate data categories and the most effective ways to collect and organize data within them.
Additional measures used include race and gender diversity, test scores, financial support of students, percentage completing, relationship with mentor, and overall socialization.
The larger programs have significantly longer median times to degree in all fields except the biological sciences, and this is particularly true in the humanities (7.4 years as compared to 6.1 years for the broad field as a whole)
Student treatment and outcomes is related to research activity, because programs with a high level of research activity have the resources to treat students better.
The illustrative approaches illuminate the interplay between program characteristics and the weights based on values of users that go into constructing rankings.
The need for enhanced performance and accessibility is also highlighted in the recent report from the Council of Graduate Schools Graduate Education: The Backbone of American Competitiveness and Innovation32
The decision to enroll in a doctoral program represents an enormous personal commitment.
In most broad fields, the programs with the largest number of Ph D.s publish more per faculty member, have more citations per publication, and receive more awards per faculty member than the average program.
diversity of the academic environment, a measure that includes the gender, racial, and ethnic diversity of the faculty and of the student body, as well as a measure of the percentage of international students.
An active group of institutional representatives, institutional researchers, and staff from the NRC and Mathematica Policy Research (MPR)
The committee collected 20 measures that include characteristics of the faculty, such as their publications, citations, grants, and diversity; characteristics of the students, such as their GRE scores, financial support, publications, and diversity; and characteristics of the program, such as number of Ph D.s granted over five years, time to degree, percentage of student completion, and placement of students after graduation.
For one thing, public universities are experiencing a sustained decline in state support that is forcing institutions to increase tuition and raise funds privately, thereby mirroring some of the features of private universities.
universities is closely tied to their ability to recruit and retain talented students who come to this country to pursue doctoral study.
Interdisciplinarity means that a plethora of faculty members from several disciplines and programs have multiple responsibilities for training graduate students and identify with several of the programs offered at the university.
In its data collection, the committee asked universities to exclude their clinical programs even if they awarded a Ph D.
To be included in the study, a doctoral program meeting these criteria must also have produced at least five doctorates between 2001
in considerable detail in the 2003 study already noted, Assessing Research-Doctorate Programs: A Methodology Study.
To create illustrative rankings, the committee explored several approaches to evaluate and rate programs, with the subsequent rankings reflecting an ordered list of ratings from high to low.
from the relative importance of weights calculated from answers to the faculty questionnaire, and (2)
The faculty survey on the relative importance of various measures yielded weights that are used to develop one illustrative ranking, the S-ranking (for survey-based)
Based on data from the National Science Foundation Survey of Doctorate Recipients, the committee found that less than 50 percent of Ph D.
According to Science and Engineering Indicators, from 1996 to 2004 the percentage of doctoral degrees awarded to white, non-Hispanic U.S.
We have worked to assure the quality of the measures used to generate rankings and have tried to minimize such errors in the data collection.
It does expect that, as a result of this data collection effort, updating will be easier next time for the respondents and will result in fewer errors.
And the selection of a doctoral program is a life choice of great importance.
In asking about the student experience within these programs, the committee had to remember that students at some universities are admitted to biological and health sciences programs without having to choose an area of specialization until the second or third year of study.
It could not lump all biological and health science programs together and get an accurate representation of the experiences of students in various parts of the biological and health sciences at a university.
To be sure, there are also many smaller programs with high rates of completion and times to degree similar to highly ranked programs.
The ratio of faculty to students has changed since the 1995 NRC study.
The increasingly interdisciplinary character of the biological and health sciences is both a cause and a consequence of these academic and institutional changes.
The data gathered for this study show considerable progress in these areas since the 1995 NRC study was conducted.
The biological and health sciences, a broad field that proved difficult to address in prior assessments, again proved the most problematic in this assessment.
Because the biological science fields have been extensively reorganized since 1993, when the last NRC assessment was carried out, it is difficult to make comparisons in these areas over time.
As additional resources are being considered for graduate programs, it becomes increasingly important to have structures in place to continually assess these investments.
The committee asked programs which definition they used and, under that definition, how many students they enrolled.
Similarly, domestic students gain a more global perspective and benefit from collaborations with graduate students from other countries and are thus better prepared for the global workplace they will encounter after graduation.
The methodology for the study will be a refinement of that described by the Committee to Examine the Methodology for the Assessment of Research-Doctorate Programs, which recommended that a new assessment be conducted.
Perhaps the most vexing issue the committee faced was how to reconcile the various ways that universities structure their graduate educational experiences.
Answers were posted by both NRC staff and the survey contractor, Mathematica Policy Research.
The current study continues that tradition but uses a methodology that directly relates a measure of perceived reputation to quantified variables.
In response to this situation, many universities have introduced programs to emphasize the importance of recruiting and supporting underrepresented minorities in virtually all fields, as well as the need to increase the presence of women in many of them.
The ranges of rankings that are shown convey some, but not all, of the uncertainties that can be estimated in producing rankings based on assigning values to quantitative measures.
But one important purpose of this NRC study is to make a very large amount of information
Diversity among the faculty has improved impressively since the 1995 NRC study.
These additional sources include uncertainty in the model for assessing quality based on quantitative measures as well as the uncertainty that the 20 measures capture the most relevant factors that are needed to assess quality in a particular field.
Growth rates in the underrepresented minority population outstrip those of the majority, and more women now are awarded bachelor
The faculty is more diverse in terms of gender, with women making up over 30% of the doctoral faculty in the biological sciences (32%)
The data collected for this study represent an unprecedented wealth of information, and the committee hopes that they will be updated and used for further analysis10
Meanwhile, the significant gains in minority and female enrollments in undergraduate education are broadening the base from which these students may be recruited.
Every field has its highly ranked and renowned programs, which are typically characterized by a large size and the high research productivity of faculty.
The ranking methodology is based on faculty values, expressed either explicitly through the questionnaire results that are used to calculate S rankings or implicitly through the ratings of a sample of programs that are used to calculate the R rankings.
program; or reminders of the traditional high regard for the university in which a program is housed These reputational dimensions can make a program look very attractive to prospective students, to prospective donors, and to funding agencies.
Women make up nearly 50% or more of students in the agricultural, biological and social sciences and the humanities.
As the committee has stressed repeatedly, the user may take the data that the study provides and construct a set of rankings based on the values that the specific user places on the measures.
Unfortunately, there was no consensus about the nomenclature for programs within the biological and health sciences, because different universities classify their biological programs differently.
In the physical sciences, the six-year completion percentage is 44 percent.
As early as 1996 a planning meeting was held to consider a separate study of the agricultural sciences, because they were not included in the 1995 study.
In the social sciences the percentage is 37%, which is the same percentage completion for the humanities after eight years.
The next NRC study, published in 1995, expanded the coverage of fields and types of data17
Rankings should be based both on data that reflect the relative importance to the user of the available measures and on the uncertainty inherent in them.
Women have made striking gains in some biological science and social science fields, but they remain underrepresented in many areas, especially in engineering and the physical and mathematical sciences.
As was found in the 1982 and 1995 reports, program size continues to be positively related to program ranking.
A final challenge inherent in making the program the unit of analysis was how to measure the workload of faculty members, whose appointment generally lies in a single department but who participate in more than one graduate program.
Each of the four questionnaires was also reviewed by the Nation Research Council
QUESTIONNAIRE DEVELOPMENT AND DATA COLLECTION
The ranking methodology utilized by the committee in these illustrative approaches has been chosen to be based on faculty values.
Graduate schools, or other institutional units asked to submit the data to the NRC, mobilized exceptional efforts to complete the forms.
They named 4,838 for which the committee calculated ranges of rankings.
All the common programs have experienced a growth in the percentage of female students with the smallest growth (3.4 percentage points)
Users are warned that, because of fundamental changes in the methodology, comparisons between 1993 rankings and ranges of rankings from the current study may be misleading.
Distinct from the earlier studies, the primary purpose of the current study, as outlined in the study
Other programs that were not included in 1993 are included in this assessment, including many programs in the field of agricultural sciences.
In addition, it has placed more emphasis on studies that extend beyond a single field, and so 14 emerging fields are included to recognize the growth of multi-, cross- and interdisciplinary study.
While enrolled, these students characteristically show higher than average completion rates and shorter than average times to degree39
The largest decrease was in the social sciences and psychology: from 2,501 to 2,048.
One method involved asking direct questions about what characteristics are important and how they should be weighed, while the second used an implicit method to determine the weights based on evaluations of programs by faculty raters.
Areas in which increases in underrepresented minorities and women have been most prominent include some fields not included in this study.
For this study the committee chose to define faculty as those who had directed doctoral research dissertations within the last five years.
Although overall measures are useful, some users may be particularly interested in measures that focus on specific aspects of a graduate program.
In the current methodology, when program measures in a field are similar, program differences in the range of rankings can depend strongly on the precise values of the input data, and so are quite sensitive to errors in those data.
The committee chose to preserve the criteria from the 1995 study for the selection of fields to be included in the current study.
In addition to these overall rankings, the study provides ranges of program rankings, based on the weights obtained for subsets of the S measure in each field.
This information should be useful for future benchmarking studies and may assist prospective students in the identification of these programs.
Details of the illustrative rankings can be obtained by clicking on links provided in these worksheets.
The data reported by the programs reflected the size, scope, and practices of each program, as well as financial aid and training practices that affect students.
The program was chosen as the primary unit for the study because programs admit students, offer degrees, and are the obvious target of student interest.
By contrast, institutional data such as time to degree, levels of student support, and infrastructure investment are not uniform and not always available or as easily compared.
Starting from survey instruments drafted originally by the 2003 study committee that developed a methodology for the assessment, the panel drafted questionnaires for four groups of respondents: institutions, programs, faculty, and students.
The swift growth of knowledge in the biological and health sciences
Some of the currently emerging fields identified by the committee are nanoscience, systems biology, urban studies and planning, and film studies.
The method of obtaining rankings through two separate ways of calculating ranges of overall rankings is discussed briefly in Chapter 4and in far more detail in the methodology guide.
Finally, when the committee developed the taxonomy it expected that each field would have enough programs to be ranked, but after it administered the program questionnaires it found that three fields could not be ranked: languages, societies, and cultures (LSC)
Program ratings depend on two things, namely the characteristics of the program (e.g., number of faculty, number of publications, citations, and other quantifiable measures)
How much financial support will likely be available for a doctoral student in history at University B, and for how long and in what form
Potential doctoral applicants, and, indeed, all users of this assessment, are invited to create customized assessment tables that reflect their own values4
The full dataset contains all the responses to all the questions on the program questionnaire, but response rates to some questions may be lower.
Of course, this observation also applies to human activity outside of research universities.
These observations span 62 fields, and the research productivity data are based typically on a five-year interval.
These and other reports lay out clear frameworks for a focused commitment to improving graduate education.
Thus admittedly, interdisciplinary programs, even though they are becoming increasingly important at universities, are shortchanged in the evaluation of more standard scholarly and scientific programs.
There are other sources of uncertainty that are not captured in the ranges presented in the illustrative rankings.
In addition, data were collected about time to degree and completion rates and whether the program tracked its students after completion.
In addition, data were collected about time to degree and completion rates and whether the program followed the progress of its students after completion.
Examples are stories of a long heritage of powerful research findings in a distinguished department; recollections of the accomplishments of famous graduates of years past; recounts of new faculty appointments made to strengthen particular areas of studies; lists of faculty publications that have shaped, changed, or even brought into existence whole fields of scholarship; recitations of the high hopes and aspirations engendered by the development of a new Ph D.
It is the process that generates highly educated scholars and researchers, significant research results, and avenues for innovation, thereby creating the leaders needed to produce the research advances that will create new careers and economic vitality for the nation30
For students, these comparisons may be along the lines of funding and completion rates, or characteristics of programs near their homes.
Because this is the first study to make such extensive demands on programs to provide so much comparable data, nearly all programs had to adapt existing practices, or devise new ones, to produce the information required by the questionnaires.
The formula eventually developed related allocation to the number of dissertations chaired by individual faculty members46
There is no significant difference based on size in the percentage of students who have definite plans for an academic position upon graduation.
What does the university do to facilitate interdisciplinary study, how fully are the faculty engaged in it, and how is such work across disciplinary lines reflected in the degree structure
The intent is to illustrate how individuals can use the data and apply their own values to the quantitative measures to obtain rankings suitable for their specific purposes.
The program-level data, collected using questionnaires, reflect the size, scope, and other components of each program, as well as, financial aid and training practices.
Computer engineering was put forward as a field that was separate from electrical engineering and computer engineering, but the universities in the study reported only 20 computer engineering programs.
in the humanities fields, which were already heavily female, and the greatest growth in the engineering fields (9 percentage points, increasing to 22% overall)
One interesting and important difference between the weights that result in the R and S rankings is that the one measure of program size
One interesting and important difference between the weights that result in the R and S rankings is that the one measure of program size
Chapter 5 discusses the ways in which the study ascertains faculty values, which are key to understanding the rankings in the study.
Both of these approaches reflect faculty values, which are discussed in Chapter 5.
the collection of quantitative data through questionnaires administered to institutions, programs, faculty, and
It also recognizes the maturation of several interdisciplinary programs, such as neuroscience, into established independent fields.
As an illustration of the kinds of data-based rankings that can be produced, the committee explains and reports rankings based on two measures.
It largely agreed on one principle: it should not allow double counting and should try to prevent universities from assigning their most prolific and distinguished faculty to multiple programs unless they actually expended
It then asked institutional coordinators to consult with the programs to judge whether this numerical allocation adequately reflected how a faculty member
A self-allocation procedure in which faculty assigned themselves or were assigned by their institutions was deemed unacceptable by most of the committee, because that procedure allowed allocations that did not accurately reflect the strength of programs.
After approval by the committee, the questionnaires were posted on the project web site and participating institutions were asked to comment on them.
English stood out as a field where fewer than 40% of the students reported that their research facilities and their workspaces were
Despite these problems of classification and assignment, the committee believes that the program continues to represent the unit that most accurately defines the range of experiences of the graduate student once admitted to a specific department or program.
Faced with the practical question of whether the allocations of faculty time were realistic, the committee counted the dissertations that faculty members were directing and allocated their time among the programs in which they served.
By becoming university faculty, by establishing start-up companies, and by contributing to the research enterprise of corporate America, these international Ph D.
The names of these interdisciplinary programs often vary, and it is not altogether evident that what is being taught in the new programs is in fact comparable.
The production of rankings from measures of quantitative data turned out to be more complicated and to have greater uncertainty than originally thought.
American doctoral programs, individual doctoral programs, assessing doctoral programs, american doctoral education, considering doctoral education, doctoral programs, doctoral education, international doctoral students, american doctoral students, doctoral students, research doctorate programs, graduate program quality, considering doctoral study.
Doctoral program, program quality, health sciences, agricultural sciences, physical sciences, research doctorate, broad fields, data collection, social sciences, doctoral study, current study, doctorate programs, broad field, underrepresented minorities, individual doctoral faculty, questionnaire collected data, faculty questionnaire collected, demographic characteristics, research activity, program characteristics, organizing graduate education, data collected, computer engineering, biological science fields, basic biomedical sciences, permanent resident males, students considering doctoral, doctoral faculty, program questionnaire collected, graduate programs, international students, faculty questionnaire, chemical engineering civil, highly ranked programs, biological sciences, emerging fields, resident males decreased, doctoral degrees awarded, mathematica policy research, graduate students, program size, universities provided data, graduate education, student educational background, national science foundation, illustrative rankings, biomedical sciences, comparative literature english, literature german language, literature english language, rankings based, evolutionary biology ecology, developmental biology ecology, policy makers, international doctoral, american doctoral, considering doctoral, graduate program, research universities, program questionnaire, research productivity, graduate schools, engineering science, chemical engineering, electrical engineering, student questionnaire, science fields, biological science, postdoctoral study, mathematical sciences, public universities, illustrative ranges, molecular biology, highly ranked, institutional researchers, earlier studies, permanent resident, academic environment, financial support, scholarly productivity, funding agencies, individual doctoral, assessing doctoral, doctoral degrees, policy research, research experiences, organizing graduate, biomedical engineering, engineering biomedical, prospective students, engineering electrical, public health, industrial engineering, mechanical engineering, aerospace engineering, engineering aerospace, engineering civil, faculty productivity, faculty involved, ranked programs, rate programs, student experience, animal sciences, plant sciences.
Underrepresented minority, literature german, comparative literature, literature english, universities provided, constructing rankings, reputational measures, materials science, private universities, literature french, quantitative measures, developmental biology, evolutionary biology, structural biology, capita measures, educational experience, academic disciplines, educational background, objective measures, biology ecology, standard method, degrees awarded, portuguese language, career objectives, german language, francophone language, english language, doctoral research, own doctoral, domestic doctoral, provide doctoral, assess doctoral, organize doctoral, doctoral studies, support doctoral, enable doctoral, total doctoral, characterize doctoral, pursue doctoral, offering doctoral, superb doctoral, strengthen doctoral, admits doctoral, trained doctoral, directed doctoral, motivation doctoral, understanding doctoral, women doctoral, enrolled doctoral, doctoral institutions, doctoral committees, doctoral production, doctoral enrollments, doctoral candidates, faculty research, program faculty, international research, biological research, doctorate program, program rankings, states research, research funding, program measures, overall program, survey research, assessing program, research training, excellent research, assess program, research doctorates, theater research, concentrated research, operations research, program ratings, research assistants, research enterprise, research experience, research partnerships, research dissertations, program admits, reflects program, program ranking, graduate study, program meeting, program classification, program features, program administrators, program differences, program solidity, health engineering, engineering programs, pursuing graduate, engineering fields, underrepresented students, appropriate graduate, training graduate, graduate thesis, quantitative data, graduate employees, schools graduate, graduate record, graduate deans, reliable data, graduate school, graduate applications, sciences programs, university faculty.
Selected faculty, domestic students, biological programs, sciences sciences, engineering chemical, female students, health science, agricultural fields, faculty survey, engineering materials, compare faculty, faculty questionnaires, environmental engineering, mentoring faculty, exceptional students, science programs, engineering operations, civil engineering, supporting students, former students, faculty sponsors, admit students, faculty sizes, faculty involvement, faculty perceptions, engineering indicators, matriculating students, students increased, treat students, students enrolled, draws students, talented students, attracted students, students apply, faculty opinion, faculty identified, agricultural field, computer sciences, students identifying, students stay, field agricultural, field programs, separate study, students enroll, american doctorate, international education, interdisciplinary study, interdisciplinary programs, social science, programs overall, biology sciences, environmental health, student completion, successful student, agricultural animal, developed programs, health physiology, health insurance, student survey, health biochemistry, experimental student, compare programs, basic biological, student costs, insurance programs, interdisciplinary fields, clinical programs, student body, renowned programs, student service, ambitious programs, student treatment, integrated programs, american universities, programs directed, public education, programs based, programs demonstrate, programs conducted, study reported, programs effectively, study findings, life sciences, programs nationwide, successful international, programs offered, current strengths, supporting underrepresented, sciences entomology, national citizenry, current approach, behavioral sciences.
International graduates, current practices, crop sciences, reputational rankings, forest sciences, sciences nutrition, poultry sciences, develop international, national academies, sciences taxonomy, separate questionnaire, historically underrepresented, graduating underrepresented, relative quality, overall rankings, stem fields, literature history, literature art, illustrative manner, animal science, doctorate recipients, produce rankings, rankings provided, biology molecular, increasingly interdisciplinary, literature spanish, science materials, rankings including, fields defined, fields represented, nursing public, government policy, overall measures, earlier interdisciplinary, educational world, subsequent rankings, individual measures, generate rankings, food science, public sources, dairy science, core educational, undergraduate education, demographic trends, key measures, domestic minorities, public administration, surveyed universities, science forestry, american competitiveness, participating universities, assessing quality, institutional units, american colleges, perceived quality, interdisciplinary work, institutional representatives, educational experiences, reputational ratings, educational planners, assess quality, increasingly competitive, questionnaire gathered, summary measures, attend university, field coverage, biology cell, scholarly records, unranked emerging, academic position, domestic sources, traditional measures, field institutions, biology biochemistry, scholarly meetings, disciplines selected, financial constraints, global competition, reputational dimensions, interdisciplinary character, economic future, completion percentage, philosophy religion, percentage completion, global perspective. A DATA-BASED ASSESSMENT OF RESEARCH-DOCTORATE PROGRAMS IN THE UNITED STATES
Committee on an Assessment of Research Doctorate Programs
Jeremiah P. Ostriker, Charlotte V. Kuh, and James A. Voytuk, Editors

Board on Higher Education and Workforce
Policy and Global Affairs

THE NATIONAL ACADEMIES PRESS
Washington, D.C.
www.nap.edu

This project was supported by the Andrew W. Mellon Foundation, the Alfred P. Sloan
Foundation, the U.S. Department of Energy (Grant DE-FG02-07ER35880), the National
Institutes of Health (Grant N01-OD-4-2139, TO#170), the National Science Foundation (Grant OIA-0540823), the National Research Council, and contributions from 212 U.S. universities. Any opinions, findings, conclusions, or recommendations expressed in this publication are those of the author(s) and do not necessarily reflect the views of the organizations or agencies that provided support for the project.
Committee on an Assessment of Research-Doctorate Programs
Jeremiah P. Ostriker, Committee Chair, Charles A. Young Professor of Astronomy and Provost Emeritus, Princeton University
Virginia S. Hinshaw, Vice Chair, Chancellor, University of Hawai�i at M?noa
Elton D. Aberle, Dean Emeritus of the College of Agricultural and Life Sciences, University of Wisconsin�Madison
Norman Bradburn, Tiffany and Margaret Blake Distinguished Service Professor Emeritus, University of Chicago
John I. Brauman, J. G. Jackson�C. J. Wood Professor of Chemistry, Emeritus, Stanford University
Jonathan R. Cole, John Mitchell Mason Professor of the University, Columbia University (resigned June 2010)
Paul W. Holland, Frederic M. Lord Chair in Measurement and Statistics (retired), Educational Testing Service
Eric W. Kaler, Provost and Senior Vice President for Academic Affairs, Stony Brook University
Earl Lewis, Provost and Executive Vice President for Academic Affairs and Asa Griggs Candler Professor of History and African American Studies, Emory University
Joan F. Lorden, Provost and Vice Chancellor for Academic Affairs, University of North Carolina at Charlotte
Carol B. Lynch, Dean Emerita of the Graduate School, University of Colorado, Boulder
Robert M. Nerem, Parker H. Petit Distinguished Chair for Engineering in Medicine, and former director, Parker H. Petit Institute for Bioengineering and Bioscience, Georgia Institute of Technology
Suzanne Ortega, Provost and Executive Vice President for Academic Affairs, University of New Mexico
Robert J. Spinrad, Vice President (retired), Technology Strategy, Xerox Corporation (resigned January 2008; deceased September 2009)
Catharine R. Stimpson, Dean, Graduate School of Arts and Science, and University Professor, New York University
Richard P. Wheeler, Vice Provost, University of Illinois at Urbana-Champaign

Staff
Charlotte V. Kuh, Study Director
Peter H. Henderson, Senior Program Officer
James A. Voytuk, Senior Program Officer
John Sislin, Program Officer
Michelle Crosby-Nagy, Research Assistant
Kara Murphy, Research Assistant
Rae E. Allen, Administrative Coordinator
Sabrina Hall, Program Associate

Contents
Letter from the Presidents
Preface and Acknowledgements

Summary 1
1 Introduction 8
2 Context and Motivation 15
3 Study Design 24
4 The Methodologies Used to Derive Two Illustrative Rankings 40
5 Faculty Values as Reflected in the Two Illustrative Rankings 55
6 Some Uses of the Data 62
7 The Data and Principal Findings 69
8 Looking Ahead 94
Appendixes
A Committee Biographies
B Taxonomy of Fields
C Participating Institutions
D Questionnaires
Institutional Questionnaire
Program Questionnaire
Program Questionnaire for Emerging Fields
Survey of Program Quality
Faculty Questionnaire
Admitted-to-Candidacy Doctoral Student Questionnaire
E List of Variables
F R and S Coefficients by Field
G Correlation for Median R and S Rankings by Broad Field
H Detail for the Rating Study

Boxes, Figures and Tables

BOXES
3-1 Variables Used in Data Cleaning, 38
4-1 Fields for Which the Correlation or the Median R and S Ranking is Less than 0.75, 43
5-1 Characteristics Included in the Faculty Weighting Process, 55
FIGURES
4-1 Steps in Calculating Two Types of Overall Program Rankings, 43
7-1 Minority and Nonminority Ph.D.�s, 1993 and 2006, 74
7-2 Percentage of Underrepresented Minority Faculty and Students by broad field,
2006, 82
7-3 Percentage of Faculty and Students Female by broad field, 2006, 84
TABLES
1-1 Coverage of NRC Studies, 1982�2006, 9
1-2 Study Questionnaires and Response Rates, 10
2-1 Ph.D.�s Awarded to White, Non-Hispanic U.S. Citizens or Permanent Resident Males, in Selected Broad Fields, 1993 and 2006, 21
3-1 Numbers of Programs and Institutions in Each Broad Field, 28
3-2 Fields in 1993 and 2006 Data Collection, 30
3-3 Characteristics Listed in the Online Database, 35
4-1 Summary of Differences Between 1995 and 2006 Studies, 47
5-1 Most Highly Rated Characteristics of Doctoral Programs on R and S
Measures, 57
5-2 Faculty Importance Weights by Broad Field, 58
5-2A Average Faculty Importance Weights on Components of Research Activity Dimensional Measure, 59
5-2B Average Faculty Importance Weights on Components of the Student Support and Outcomes Dimensional Measure, 60
5-2C Average Faculty Importance Weights on Components of the Diversity Dimensional Measure, 61
6-1 Ranges of Rankings and Data for Five Mid-Atlantic Chemistry Programs, 64
6-2 An Example of Selected Ranges of Rankings for a Doctoral Program in Biochemistry, 65
6-3 Calculation of the R and S Rankings for a Single Program, 66
7-1 Weighted Measures for Faculty and Students, 2006 and 1993, 70
7-2 Changes in Ph.D.�s, Enrollment, and Gender Composition, Common Programs, 1993�2006, 71
7-3 Average of Total Faculty, All Common Programs, 1993 and 2006, 72
7-4 Percentage Change in Number of Doctoral Recipients, Common Programs, 1993 and 2006, 74
7-5 Number of Postdocs by Broad Field, 2006, 73
7-6 Number of Ph.D's in 2006 NRC Study Compared with Ph.D.'s in NSF Doctorate Record File, 75
7-7 Institutions with 50 Percent of Ph.D.�s in Ranked Programs, by Control (Public or Private), 2002�2006 (average number of Ph.D.�s), 76
7-8 Research Measures and Program Size, 78
7-9 Student Characteristics by Broad Field Average and for the Largest Quartile, 80
7-10 Diversity Measures, 81
7-11 Fields with More than 10 Percent of Enrolled Students from Underrepresented Minority (URM) Groups, 83
7-12 Science and Engineering Fields with More than 15 Percent of Doctoral Faculty Female, 85
7-13 Faculty Data: Selected Measures, 2006, 87
7-14 Response Rates: Student Survey, Five Fields, 2006, 88
7-15 Student Satisfaction: Programs with More Than 10 Students Responding, 2006, 89
7-16 Student Productivity: Programs with More Than 10 Student Responses, 90
7-17 Students: Advising and Academic Support (percent), 91
7-18 Student Career Objectives at Program Entry and at Time of Response (percent), 92


Preface and Acknowledgments
Doctoral education, a key component of higher education in the United States, is performing well. It educates future professors, researchers, innovators and entrepreneurs. It attracts students and scholars from all over the world and is being emulated globally. This success, however, should not engender complacency. It was the intent of this study to measure characteristics of doctoral programs that are of importance to students, faculty, administrators, and others who care about the quality and effectiveness of doctoral programs in order to permit comparisons among programs in a field of study and to provide a basis for self-improvement within the disciplines. To this end, the Committee on an Assessment of Research-Doctorate Programs collected a large amount of data relating to research productivity, student support and outcomes and program diversity from over 4800 doctoral programs in 62 fields at 212 U.S. universities. Some of these data, such as the percent of entering students who complete in a given time period, the percent of students funded in the first year, and the diversity of program faculty and students have not been collected in earlier studies. These data appear in the online spreadsheets that accompany this report and can easily be selected, downloaded, and compared. The most important benefits of this study will flow from examination and analysis of the data that were collected.
In addition to making new data available, the committee addressed the issue of program rankings from an illustrative standpoint. Rankings based on faculty opinions of program quality had been produced in earlier NRC reports in 1982 and 1995. In these studies, the ratings and rankings were derived from surveys in which faculty members were asked to assess the scholarly quality and effectiveness in education of individual doctoral programs in their own fields, i.e. they were based on reputation. There was a widespread reaction, after the completion of the 1995 study, that the one, reputation based, measure was inadequate to represent the many important characteristics that are needed to describe and assess the full range of US doctoral programs.
The present NRC study, A Data-based Assessment of Research-Doctorate Programs, differs significantly from these earlier studies in that it uses objective data to estimate overall quality of doctoral programs using values important to faculty, and does so in two different ways as illustrations. It also creates measures of program strength along three separate dimensions, that is, five separate measures have been developed. Using a much broader range of collected data and information from new surveys, this data-based assessment obtains faculty importance weights for 20 program characteristics, and designs two specific techniques to obtain weights that relate these characteristics to perceived program quality. It has also incorporated the uncertainty that comes from differences in faculty views, variability in program data, and statistical variation to produce ranges of rankings for each program in 59 disciplines. The committee considers these ranges of rankings to be illustrative. Other ranges could have been obtained with different weights. One example of alternative ranges of rankings with weights obtained from the surveys are ranges of rankings along the separate dimensions of 1)research activity, 2)student support and outcomes, and 3)diversity of the academic environment. These dimensional measures are all examples of ways that the data and weights can be combined. Users are encouraged to develop additional measures employing weights that reflect their values. None of these ranges of rankings should be considered NRC-endorsed.
The committee believes that the concept of a precise ranking of doctoral programs is mistaken. How doctoral programs are ranked will depend on what raters are chosen, on year-to-year variability in program characteristics, and on the statistical error involved in any estimation. These sources of uncertainty imply that rankings of programs are intrinsically imprecise. The committee has tried to take into account these sources of variation in its illustrative rankings and, in order to convey that variation, has presented ranges of rankings. The two overall measures illustrate that data-based rankings may vary, depending on the weights applied to program characteristics, and that these may vary depending on which raters are chosen and the techniques used to obtain their rankings.
As noted earlier, it is the comparison of the program characteristics that will, in the end, be more valuable than any range of rankings. The analysis of these characteristics will help direct faculty in academic programs to areas of potential improvement and will expand what students understand about the programs that interest them.
Because some of the data collected for this study had not been collected previously, time had to be spent on data validation and assurance. The statistical techniques also required time to develop and test. As a result, much of the data presented here come from 2005-2006. Programs and faculty may have changed in the intervening period. In the on-line spreadsheets, each program has a url. Users are encouraged to go to these program websites to obtain the latest information about programs of interest. Now that the statistical machinery and the data structure are in place, it should be easier to replicate this study with current data in the near future.
ACKNOWLEDGEMENT OF REVIEWERS AND SPONSORS
This report has been reviewed in draft form by individuals chosen for their diverse perspectives and technical expertise, in accordance with procedures approved by the National Academies� Report Review Committee. The purpose of this independent review is to provide candid and critical comments that will assist the institution in making its published report as sound as possible and to ensure that the report meets institutional standards for objectivity, evidence, and responsiveness to the study charge. The review comments and draft manuscript remain confidential to protect the integrity of the process.
We wish to thank the following individuals for their review of this report and the accompanying data: John Bailar, University of Chicago; Diane Birt, Iowa State University; Craig Calhoun, New York University; Alicia Carriquiry, Iowa State University; Joseph Cerny, University of California, Berkeley; Gill Clarke, University of Bristol; David Donoho, Stanford University; Ronald Ehrenberg, Cornell University; Daniel Fogel, University of Vermont; George Langford, Syracuse University; Risa Palm, The State University of New York; William Press, University of Texas, Austin; Raul Ramos, University of Houston; Lydia Snover, Massachusetts Institute of Technology; Stephen Stigler, University of Chicago; Patrick Stover, Cornell University; Andrew Wachtel, Northwestern University; and John Wiley, University of Wisconsin-Madison. The review of this report was overseen by Lyle Jones, University of North Carolina, Chapel Hill and Stephen Fienberg, Carnegie Mellon University.
Although the reviewers listed above have provided many constructive comments and suggestions, they were not asked to endorse the conclusions or recommendations, nor did they see the final draft of the report before its release. Responsibility for the final content of this report rests entirely with the authoring committee and the institution.
In addition to the reviewers, the sponsors of this study deserve both recognition and thanks for their patience and resources. First, thanks to the 212 universities that provided not only significant financial support but also staff time to obtain and validate high quality data. Second, our thanks to the foundations and agencies that provided support to this project: the Andrew W. Mellon Foundation, the Alfred P. Sloan Foundation, the U.S. Department of Energy, the National Institutes of Health, the National Science Foundation, and the National Research Council. As the case with the reviewers, responsibility for the final content of this report rests entirely with the authoring committee and the institution.
I would also like to thank our data and statistical contractor, Mathematica Policy Research, and particularly David Edson, who oversaw countless runs and data revisions with calm and good sense. I am also grateful to the Council of Graduate Schools, the American Council of Learned Societies, and the Association of Graduate Schools of the American Association of Universities, which heard progress reports at various stages of the project and offered suggestions that, we hope, have resulted in a better report. Thanks also to the scholarly societies and councils of department chairs who have helped us shape both the data for this report and its presentation. Finally, I would like to thank the staff of the project--Charlotte Kuh, James Voytuk, Michelle Crosby-Nagy, Sabrina Hall, Rae Allen, and Kara Murphy-- for their tireless efforts to bring to completion what has been a lengthy but rewarding project.
Jeremiah P. Ostriker, chair
Committee to Assess Research-Doctorate Programs