The data used in this study were collected as part of the National Research Council’s Data-Based Assessment of Research-Doctorate Programs, and the data collection procedures and caveats are described in detail in that report.1 The committee authoring the Assessment identified several sources of errors in the data that could not be eliminated, including classification errors and data collection errors. The omission of field-specific measures, such as books, patents, and articles presented at refereed conferences in some science and engineering fields, means that the data do not capture the full scope of a program’s research productivity (see Box 2-1).
The data on research productivity that were collected during the study were analyzed in specific ways in the Assessment report, but the full database available to researchers could extend this analysis to explore alternate measures of research productivity by the faculty. For example, less emphasis could be placed on a count of journal articles, which were not judged on the basis of their impact, and greater emphasis could be placed on the citation measure. Alternately, only articles with citations could be counted. These are only a few suggestions for further analysis.
Once the data were released, institutions and others identified additional problems, which led to the release of a corrected data table in April, 2011.2 It is important for the reader to understand some of the limitations of the data used to produce the correlations and other analysis in this report.
______________
1 see Chapter 3 of the Assessment, “Study Design.”
2 A summary of the changes made to the data table and a log of individual corrections are available at www.nap.edu/rdp.
BOX 2-1 Sources of Data Errors in the Assessment of Research-Doctorate Programs
1) Classification errors. The taxonomy of fields may not adequately reflect distinctions that the field itself considers to be important. For example, in anthropology physical anthropology is a different scholarly undertaking from cultural anthropology, and each subfield has different patterns of publication. By lumping together these subfields into one overall field, the committee is implying comparability. Were they separate, different weights might be given to publications or citations. Anthropology is not alone in this problem. Other fields are public health, communications, psychology, and integrated biological science. Although this study presents ranges of rankings across these fields, the committee encourages users to choose comparable programs and use the data, but apply their own weights or examine ranges of rankings only within their peer group.
2) Data collection errors. The committee provided detailed definitions of important data elements used in the study, such as doctoral program faculty, but not every program that responded paid careful attention to these definitions. The committee carried out broad statistical tests, examined outliers, and got back to the institutions when it had questions, but that does not mean it caught every mistake. In fields outside the humanities it counted publications by matching faculty names to Thomson Reuter’s data and tried to limit mistaken attribution of publications to people with similar names. Despite these efforts, some errors may remain.
3) Omission of field-specific measures of scholarly productivity. The measures of scholarly productivity used were journal articles and, in the humanities, books and articles. Some fields have additional important measures of scholarly productivity. These were included in only one field, the computer sciences. In that field peer-reviewed conference papers are very important. A discussion of data from the computer sciences with its professional society led to further work on counting publications for the entire field. In the humanities the committee omitted curated exhibition volumes for art history. It also omitted books for the science fields and edited volumes and articles in edited volumes for all fields, since these were not indexed by Thomson-Reuters. All of these omissions result in an undercounting of scholarly productivity. The committee regrets them, but it was limited by the available sources. In the future it might be possible to obtain data on these kinds of publication from résumés, but that is expensive and time-consuming.
NOTE: The computer sciences count as publications articles that are presented at refereed conferences, but until recently few of these papers were indexed by Thomson Reuters. To deal with this practice, the committee compiled a list of such conferences that were not indexed and counted these publications from faculty résumés, as it did in the humanities.
SOURCE: A Data-Based Assessment of Research-Doctorate Programs in the United States, p. 7.
In addition to data from the Assessment, data on training grants and training slots were collected from the NIH website.3 Using these two sources, the panel has identified correlations among many of the characteristics of doctoral programs in the biomedical sciences mentioned in the statement of task:
Average Publications per Faculty Member
Average Citations per Publication
Percent of Faculty with Grants
Percent of Non-Asian Minority Faculty
Percent of Female Faculty
Awards per Faculty Member
Average GRE Scores
Percent of Non-Asian Minority Students
Percent of Female Students
Average PhDs per Year, 2002-2006
Average Cohort Completion Rate
Median Time to Degree
Appendix D provides the correlations for these 12 variables for each field. With the exception of Awards per Faculty Member, all are discussed in Chapter 3.
In addition to the above list, other variables, such as the percent of first-year students with research assistantships or the percent with external fellowships, were used in analyses in later chapters (e.g., Chapter 4 ). Appendix C contains definitions of all of the relevant variables from the Assessment; data on these variables for each biomedical program are included in the Excel table available with this report. Appendix E contains the statistical summary of each variable by field.
Finally, the panel relied on other results from the Assessment surveys of doctoral programs, faculty, and students for more targeted analysis. Data on doctoral student satisfaction, productivity, and changes in career objectives in neuroscience and neurobiology (Chapter 6 ) came from the survey conducted of doctoral students in that and four other sample fields (chemical engineering, physics, economics, and English)4. Data on postdoctorates in Chapter 7 were drawn from primarily unpublished results of the program and faculty surveys. Although not all of these data are discussed in the Assessment report, they are available in the online Excel data table that accompanies this report or in the full database available for public use.
______________
3 See http://grants.nih.gov/training/outcomes.htm#fundedgrants; data are from the version posted in 2009. Using NIH data, we were unable to associate training grant funding with particular programs. We were, however, able to tie them to particular institutions, and this is the approach we take in the analysis in this report.
4 See “Data from Student Questionnaires” in Chapter 7 of the Assessment.