National Academies Press: OpenBook
« Previous: VIII. Zoology Programs
Suggested Citation:"IX. Summary and Discussion." National Research Council. 1982. An Assessment of Research-Doctorate Programs in the United States: Biological Sciences. Washington, DC: The National Academies Press. doi: 10.17226/9779.
×
Page 163
Suggested Citation:"IX. Summary and Discussion." National Research Council. 1982. An Assessment of Research-Doctorate Programs in the United States: Biological Sciences. Washington, DC: The National Academies Press. doi: 10.17226/9779.
×
Page 164
Suggested Citation:"IX. Summary and Discussion." National Research Council. 1982. An Assessment of Research-Doctorate Programs in the United States: Biological Sciences. Washington, DC: The National Academies Press. doi: 10.17226/9779.
×
Page 165
Suggested Citation:"IX. Summary and Discussion." National Research Council. 1982. An Assessment of Research-Doctorate Programs in the United States: Biological Sciences. Washington, DC: The National Academies Press. doi: 10.17226/9779.
×
Page 166
Suggested Citation:"IX. Summary and Discussion." National Research Council. 1982. An Assessment of Research-Doctorate Programs in the United States: Biological Sciences. Washington, DC: The National Academies Press. doi: 10.17226/9779.
×
Page 167
Suggested Citation:"IX. Summary and Discussion." National Research Council. 1982. An Assessment of Research-Doctorate Programs in the United States: Biological Sciences. Washington, DC: The National Academies Press. doi: 10.17226/9779.
×
Page 168
Suggested Citation:"IX. Summary and Discussion." National Research Council. 1982. An Assessment of Research-Doctorate Programs in the United States: Biological Sciences. Washington, DC: The National Academies Press. doi: 10.17226/9779.
×
Page 169
Suggested Citation:"IX. Summary and Discussion." National Research Council. 1982. An Assessment of Research-Doctorate Programs in the United States: Biological Sciences. Washington, DC: The National Academies Press. doi: 10.17226/9779.
×
Page 170
Suggested Citation:"IX. Summary and Discussion." National Research Council. 1982. An Assessment of Research-Doctorate Programs in the United States: Biological Sciences. Washington, DC: The National Academies Press. doi: 10.17226/9779.
×
Page 171
Suggested Citation:"IX. Summary and Discussion." National Research Council. 1982. An Assessment of Research-Doctorate Programs in the United States: Biological Sciences. Washington, DC: The National Academies Press. doi: 10.17226/9779.
×
Page 172
Suggested Citation:"IX. Summary and Discussion." National Research Council. 1982. An Assessment of Research-Doctorate Programs in the United States: Biological Sciences. Washington, DC: The National Academies Press. doi: 10.17226/9779.
×
Page 173
Suggested Citation:"IX. Summary and Discussion." National Research Council. 1982. An Assessment of Research-Doctorate Programs in the United States: Biological Sciences. Washington, DC: The National Academies Press. doi: 10.17226/9779.
×
Page 174
Suggested Citation:"IX. Summary and Discussion." National Research Council. 1982. An Assessment of Research-Doctorate Programs in the United States: Biological Sciences. Washington, DC: The National Academies Press. doi: 10.17226/9779.
×
Page 175
Suggested Citation:"IX. Summary and Discussion." National Research Council. 1982. An Assessment of Research-Doctorate Programs in the United States: Biological Sciences. Washington, DC: The National Academies Press. doi: 10.17226/9779.
×
Page 176
Suggested Citation:"IX. Summary and Discussion." National Research Council. 1982. An Assessment of Research-Doctorate Programs in the United States: Biological Sciences. Washington, DC: The National Academies Press. doi: 10.17226/9779.
×
Page 177
Suggested Citation:"IX. Summary and Discussion." National Research Council. 1982. An Assessment of Research-Doctorate Programs in the United States: Biological Sciences. Washington, DC: The National Academies Press. doi: 10.17226/9779.
×
Page 178
Suggested Citation:"IX. Summary and Discussion." National Research Council. 1982. An Assessment of Research-Doctorate Programs in the United States: Biological Sciences. Washington, DC: The National Academies Press. doi: 10.17226/9779.
×
Page 179
Suggested Citation:"IX. Summary and Discussion." National Research Council. 1982. An Assessment of Research-Doctorate Programs in the United States: Biological Sciences. Washington, DC: The National Academies Press. doi: 10.17226/9779.
×
Page 180
Suggested Citation:"IX. Summary and Discussion." National Research Council. 1982. An Assessment of Research-Doctorate Programs in the United States: Biological Sciences. Washington, DC: The National Academies Press. doi: 10.17226/9779.
×
Page 181
Suggested Citation:"IX. Summary and Discussion." National Research Council. 1982. An Assessment of Research-Doctorate Programs in the United States: Biological Sciences. Washington, DC: The National Academies Press. doi: 10.17226/9779.
×
Page 182
Suggested Citation:"IX. Summary and Discussion." National Research Council. 1982. An Assessment of Research-Doctorate Programs in the United States: Biological Sciences. Washington, DC: The National Academies Press. doi: 10.17226/9779.
×
Page 183
Suggested Citation:"IX. Summary and Discussion." National Research Council. 1982. An Assessment of Research-Doctorate Programs in the United States: Biological Sciences. Washington, DC: The National Academies Press. doi: 10.17226/9779.
×
Page 184
Suggested Citation:"IX. Summary and Discussion." National Research Council. 1982. An Assessment of Research-Doctorate Programs in the United States: Biological Sciences. Washington, DC: The National Academies Press. doi: 10.17226/9779.
×
Page 185
Suggested Citation:"IX. Summary and Discussion." National Research Council. 1982. An Assessment of Research-Doctorate Programs in the United States: Biological Sciences. Washington, DC: The National Academies Press. doi: 10.17226/9779.
×
Page 186
Suggested Citation:"IX. Summary and Discussion." National Research Council. 1982. An Assessment of Research-Doctorate Programs in the United States: Biological Sciences. Washington, DC: The National Academies Press. doi: 10.17226/9779.
×
Page 187
Suggested Citation:"IX. Summary and Discussion." National Research Council. 1982. An Assessment of Research-Doctorate Programs in the United States: Biological Sciences. Washington, DC: The National Academies Press. doi: 10.17226/9779.
×
Page 188
Suggested Citation:"IX. Summary and Discussion." National Research Council. 1982. An Assessment of Research-Doctorate Programs in the United States: Biological Sciences. Washington, DC: The National Academies Press. doi: 10.17226/9779.
×
Page 189
Suggested Citation:"IX. Summary and Discussion." National Research Council. 1982. An Assessment of Research-Doctorate Programs in the United States: Biological Sciences. Washington, DC: The National Academies Press. doi: 10.17226/9779.
×
Page 190
Suggested Citation:"IX. Summary and Discussion." National Research Council. 1982. An Assessment of Research-Doctorate Programs in the United States: Biological Sciences. Washington, DC: The National Academies Press. doi: 10.17226/9779.
×
Page 191
Suggested Citation:"IX. Summary and Discussion." National Research Council. 1982. An Assessment of Research-Doctorate Programs in the United States: Biological Sciences. Washington, DC: The National Academies Press. doi: 10.17226/9779.
×
Page 192
Suggested Citation:"IX. Summary and Discussion." National Research Council. 1982. An Assessment of Research-Doctorate Programs in the United States: Biological Sciences. Washington, DC: The National Academies Press. doi: 10.17226/9779.
×
Page 193
Suggested Citation:"IX. Summary and Discussion." National Research Council. 1982. An Assessment of Research-Doctorate Programs in the United States: Biological Sciences. Washington, DC: The National Academies Press. doi: 10.17226/9779.
×
Page 194
Suggested Citation:"IX. Summary and Discussion." National Research Council. 1982. An Assessment of Research-Doctorate Programs in the United States: Biological Sciences. Washington, DC: The National Academies Press. doi: 10.17226/9779.
×
Page 195
Suggested Citation:"IX. Summary and Discussion." National Research Council. 1982. An Assessment of Research-Doctorate Programs in the United States: Biological Sciences. Washington, DC: The National Academies Press. doi: 10.17226/9779.
×
Page 196
Suggested Citation:"IX. Summary and Discussion." National Research Council. 1982. An Assessment of Research-Doctorate Programs in the United States: Biological Sciences. Washington, DC: The National Academies Press. doi: 10.17226/9779.
×
Page 197
Suggested Citation:"IX. Summary and Discussion." National Research Council. 1982. An Assessment of Research-Doctorate Programs in the United States: Biological Sciences. Washington, DC: The National Academies Press. doi: 10.17226/9779.
×
Page 198

Below is the uncorrected machine-read text of this chapter, intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text of each book. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.

1X Summary and Discussion In the six preceding chapters results are presented of the assess- ment of 616 research-doctorate programs in biochemistry, botany, cellu- lular/molecular biology, microbiology, physiology, and zoology. In- cluded in each chapter are summary data describing the means and intercorrelations of the program measures in a particular discipline. In this chapter a comparison is made of the summary data reported in the six disciplines. Also presented here are an analysis of the reli- ability (consistency) of the reputational survey ratings and an exami- nation of some factors that might possibly have influenced the survey results. This chapter concludes with suggestions for improving studies of this kind--with particular attention given to the types of measures one would-like to have available for an assessment of reseaxch-doctor- ate programs. This chapter necessarily involves a detailed discussion of various statistics (means, standard deviations, correlation coefficients) de- scribing the measures. Throughout, the reader should bear in mind that all these statistics and measures are necessarily imperfect attempts to describe the real quality of research-doctorate programs. Quality and some differences in quality are real, but these differences cannot be subsumed completely under any one quantitative measure. For example, no single numerical ranking--by measure 08 or by any weighted average of measures--can rank the quality of different programs with precision. However, the evidence for reliability indicates considerable sta- bility in the assessment of quality. For instance, a program that comes out in the first decile of a ranking is quite unlikely to "really" belong in the third decile, or vice versa. If numerical ranks of programs were replaced by groupings (distinguished, strong, etc.), these groupings again would not fully capture actual differences in quality since there would likely be substantial ambiguity about the borderline between adjacent groups. Furthermore, any attempt at linear ordering (best, next best, . . .) may also be inaccurate. Programs of roughly comparable quality may be better in different ways, so that there simply is no one best program--as will also be indicated in some analyses. However, these difficulties of formulating ranks should not hide the underlying reality of differences in quality or the importance of high quality for effective doctoral education. of the numerical 163

164 SUMMARY OF THE RESULT S Displayed in Table 9.1 are the numbers of programs evaluated (bot- tom line) and the mean values for each measure in the six biological science disciplines. As can be seen, the mean values reported for individual measures vary considerably among disciplines. The pattern of means on each measure is summarized below, but the reader interested in a detailed comparison of the distribution of a measure should refer to the second table in each of the preceding chapters · 2 Program Size (Measures 01-03~. Based on the information provided to the committee by the study coordinator at each university, cellular/ molecular biology programs had, on the average, the largest number of faculty members {26 in December 1980), followed by zoology (211. Zo- ology programs graduated the most students (26 Ph.D. recipients in the FY1975-79 period), and cellular/molecular programs had the largest en- rollment (34 doctoral students in December 1980~. In contrast, physi- ology programs were reported to have an average of 19 faculty members, 14 graduates, and 16 doctoral students. Program Graduates (Measures 04-07~. The mean fraction of FY1975-79 doctoral recipients who as graduate students had received some national fellowship or training grant support (measure 04) ranges from .21 for graduates of botany programs to .59 for graduates in cellular/molecular biology. The relatively high fraction of support in cellular/molecular biology, biochemistry, microbiology' and physiology may be explained by the availability of National Institutes of Health (AIR) training grant support for graduate students in these disciplines. With respect to the median number of years from first enrollment in a graduate pro- gram to receipt of the doctorate (measure 05), graduates in biochemis- try, cellular/molecular biology, microbiology, and physiology typically earned their degrees approximately a year sooner than graduates in zo- ology. In terms of employment status at graduation (measure 06), an average of 80 percent or more of the Ph.D. recipients in these same four disciplines reported that they had made firm job commitments by the time they had completed requirements for their degrees, contrasted with 65 percent of the program graduates in zoology and 67 percent in botany. A mean of 56 percent (or higher) of the program graduates in biochemistry, cellular/molecular biology, and physiology reported that they had made firm commitments to take positions in Ph.D.-granting in- stitutions (measure 07), while only 30 percent of those in zoology and 32 percent in botany had made such plans. t Means for measure 16, publication "influence," are omitted since arbitrary scaling of this measure prevents meaningful comparisons across disciplines. 2 The second table in each of the six earlier chapters presents the standard deviation and decile values for each measure.

165 TABLE 9.1 Mean Values for Each Program Measure, by Discipline Bio- Cell./Molec. Micro- Physi- chemistry Botany Biology biology ology Zoology Program Size 01 19 19 26 16 19 21 02 20 19 23 16 14 26 03 25 20 34 20 16 33 Program Graduates 04 .47 .21 .59 .48 .53 .33 05 6.0 6.5 6.1 6.1 6.2 7.1 06 .81 .67 .80 .80 .80 .65 07 .56 .32 .56 .48 .57 .30 Survey Results 08 - 2.6 3.0 2.9 2.8 3.Q 2.7 09 1.7 1.8 1.9 1.8 1.9 1.7 10 1.2 1.2 1.2 1.2 1.2 1.2 11 .7 .7 .7 .6 .6 .7 University Library 12 .2 .3 .3 .3 .3 .3 Research Support 13 .63 .28 .64 .45 .52 .36 14 8419 8406 10243 8449 8687 8806 Publication Records 15 92 60 133 46 17 16 Total Programs 139 83 89 134 101 70

166 Survey Results {Measures 08-11~. Differences in the mean ratings de- rived from the reputational survey are small. In all-six disciplines the mean rating of scholarly quality of program faculty (measure 08) is at or slightly below 3.0 {"good"), and programs were judged to be, on the average, "reasonably" (2.0) to "moderately" (1.0) effective in educating research scholars/scientists (measure 09~. In the opinions of the survey respondents, there has been "slight improvement" (an av- erage of 1.2 in each discipline on measure 10) in the last five years in the overall average quality of programs. The mean rating of an evaluator's familiarity with the work of program faculty (measure 11) falls below 1.0 ("some familiarity") in every discipline--about which more will be said later in this chapter. The reader should be reminded that the distribution of ratings may vary from one discipline to another. If one examines, for example, the top program ratings recorded for measure 08 in each discipline, one finds noticeably higher top ratings in biochemistry (two programs with ratings above 4.8) and cellular/molecular biology (three programs with ratings above 4.7} than in either botany or physiology (no pro- grams with ratings above 4.5~. The study committee does not have an explanation of this observation but wishes to emphasize that many differences may be found in the distributions of survey ratings in the various disciplines and that the determinants of these differences are not known. As discussed in Chapter II, the survey ratings reflect each program's standing relative to other programs in the same discipline and provide no basis for making comparisons across disciplines. University Library (Measure 12~. Measure 12, based on a composite in- _ dex of the sizes of the library in the university in which a program resides, is calculated on a scale from -2.0 to 3.0, with means ranging from .2 in biochemistry to .3 in the other five disciplines. As men- tioned earlier in this report, these data are not available for many of the smaller universities. Were the program coverage complete for this measure, it is likely that the reported means would be signifi- cantly lower. Research Support (Measures 13-14~. Measure 13, the proportion of pro- gram faculty who had received NIH, NSF, or ADAMHA4 research grant awards during the FY1978-80 period, has mean values ranging from as high as .64 and .63 in cellular/molecular biology and biochemistry, respectively, to .28 in botany. It should be emphasized that this measure does not take into account research support that faculty mem- 3 The index, derived by the Association of Research Libraries, re- flects a number of different measures, including number of volumes, fiscal expenditures, and other factors relevant to the size of a uni- versity library. See the description of this measure presented in Ap- pendix D. National Institutes of Health; National Science Foundation; and A1- cohol, Drug Abuse, and Mental Health Administratione

167 hers have received from sources other than these three federal agen- cies. In terms of total university expenditures for R&D in-the biological sciences (measure 14), the mean values are reported to range from $8,406,000 in botany to $10,243,000 in cellular/molecular biology. In considering this measure it must be remembered that it reflects the overall university expenditures in the biological sciences and not expenditures in individual disciplines within the biological sciences. Publication Records (Measures 15 and 16~. Considerable diversity is also found in the mean number of articles associated with a research- doctorate program (measure 15~. An average of 133 articles published in the 1978-79 period is reported for programs in cellular/molecular biology and 92 articles for programs in biochemistry; in physiology and zoology the mean number of articles is fewer than 20. These large differences reflect several factors, including the program size in a particular discipline (i.e., the total number of faculty and other staff members involved in research), the frequency with which scien- tists in that discipline publish, and the length of a typical paper in a discipline. Another important factor is the fact that journals in biochemistry and cellular/molecular biology far outnumber those in the other four biological disciplines. (As decribed in Chapter II, data on published articles were identified by field of journal--not by matching the names of program faculty with authors' names.) Mean scores are not reported on measure 16, the estimated "overall influ- ence" of the articles attributed to a program. Since this measure is calculated from an average of journal influence weights,5 normalized for the journals covered in a particular discipline, mean differences between disciplines are uninterpretable. CORRELATIONS AMONG MEASURES Relations among the program measures are of intrinsic interest and are relevant to the issue of validity of the measures as indices of the quality of a research-doctorate program. Measures that are logically related to program quality are expected to be related to each other. To the extent that they are, a stronger case might be made for the validity of each as a quality measure. A reasonable index of the relationship between any two measures is the Pearson product-moment correlation coefficient. A table of corre- lation coefficients of all possible pairs of measures is presented in each of the six preceding chapters. This chapter presents selected correlations to determine the extent to which coefficients are compa- rable in the six disciplines. Special attention is given to the corre- lations involving the number of FY1975-79 program graduates (measure 02), survey rating of the scholarly quality of program faculty (measure ssee Appendix F for a description of the derivation of this measure.

168 08), university R&D expenditures in a particular discipline (measure 14), and the influence-weighted number of publications-(measure 16~. These four measures have been selected because of their relatively high correlations with several other measures. Readers interested in corre- lations other than those presented in Tables 9.2-9.S may refer to the third table in each of the preceding six chapters. Correlations with Measure 02. Table 9.2 presents the correlations of measure 02 with each of the other measures used in the assessment. As might be expected, correlations of this measure with the other two mea- sures of program size--the number of faculty {01) and doctoral student enrollment (03~--are moderately high in all six disciplines. Of greater interest are the strong positive correlations between measure 02 and measures derived from the reputational survey ratings (08, 09, and 111. In biochemistry and zoology these coefficients are all above .60; in cellular/molecular biology, microbiology, and physiology most of the coefficients are above .40. In botany the correlations are somewhat lower. It is quite apparent that the programs that received high survey ratings and with which evaluators were more likely to be familiar were also ones that had larger numbers of graduates. Although the committee gave serious consideration to presenting an alternative set of survey measures that were adjusted for program size, a satisfac- tory algorithm for making such an adjustment was not found. In at- tempting such an adjustment on the basis of the regression of survey ratings on measures of program size, it was found that some exception- ally large programs appeared to be unfairly penalized and that some very small programs received unjustifiably high adjusted scores. Measure 02 also has positive correlations in most disciplines with measure 12, an index of university library size, with measures 13 and 14, which pertain to the level of support for research in a program, and with measures 15 and 16, which reflect publication productivity. Of particular note are the moderately large coefficients--in disci- plines other than botany and physiology--for the latter two measures. The relation of the number of program graduates and the publication records of that program is especially strong in biochemistry and zool- ogy. The correlations of measure 02 with measures 04, 05, 06, and 07 are below .20 in all disciplines except biochemistry and zoology. From this analysis it is apparent that the number of program graduates tends to be positively correlated with all other variables except those per- taining to recent program graduates (04-07~. It is also apparent that the relationship of measure 02 with the other variables tends to be strongest for programs in biochemistry and zoology. Correlations with Measure 08. Table 9.3 shows the correlation coeffi- cients for measure 08, the mean rating of the scholarly quality of pro- gram faculty, with each of the other variables. The correlations of measure 08 with measures of program size (01, 02, and 03) are typically greater than .40. Not surprisingly, the larger the program, the more likely its faculty is to be rated high in quality. Correlations of measure 08 with measure 04, the fraction of stu- dents with national fellowship or traineeship awards, are .47 or

169 TABLE 9.2 Correlations of the Number of Program Graduates (Measure 02) with Other Measures, by Discipline Sio- Cell./Mo~ec. Micro- Physi- chemistry Botany Biology biology ology Zoology Program Size 01 .48 .42 .54 .61 .43 .66 03 .72 .55 .83 .80 .54 .78 Program Graduates 04 .47 -.14 .16 .18 -.09 .27 05 .16 .04 -.11 .02 .19 .2S 06 .23 -.06 .12 .07 .14 .19 07 .27 -.07 -.02 .12 .15 .44 Survey Results 08 .63 .29 .42 .48 .46 .66 09 .67 .34 .47 .55 .S1 .68 10 .14 .11 .09 .19 .19 .09 11 .63 .27 .40 .58 .32 .62 University Library 12 .46 -.01 .19 .27 .2S .60 Research Support 13 .35 .03 .18 .20 .04 .50 14 .46 .09 .02 .18 .30 .42 Publication Records 15 .66 .23 .32 .44 .16 .56 16 .65 .24 .34 .29 .17 .59

170 TABLE 9.3 Correlations of the Survey Ratings of Scholarly Quality of Program Faculty (Measure 08) with Other Measures, by Discipline Bio- chemistry Botany Cell./Molec. Micro- Biology biology Pbysi- ology Zoology Program Size 01 .58 .56 .39 .50 .64 .53 02 .63 .29 .42 .48 .46 .66 03 .60 .51 .43 .42 .42 .52 Program Graduates 04 .70 .31 .58 .57 .47 .58 05 .15 .26 .38 .08 -.11 .39 06 .24 .36 .33 .23 .27 .25 07 .35 .59 .41 .38 .30 .63 Survey Results 09 .96 .97 .96 .96 .95 .98 10 .21 .29 .33 .46 .38 .19 11 .96 .83 .94 .91 .89 .95 University Library 12 .63 .66 .47 .54 .49 .78 Research Support 13 .62 .49 .58 .64 .57 .72 14 .69 .62 .57 .68 .51 .65 Publication Records 15 .83 .60 .69 .72 .69 .59 16 .83 .62 .71 .75 .71 .64

171 greater in all disciplines except botany. For programs in the mathe- matical and physical sciences and in engineering, the corresponding coefficients (reported in earlier volumes) are found to be considerably lower, typically in the range of .10 to .30. In the biological sci- ences (especially in the biomedical fields), there is a far greater re- liance on training grant and fellowship support, and fewer graduate students are supported by research assistantships or teaching assist- antships. Correlations of rated faculty quality with measure 05, the shortness of time from matriculation in graduate school to award of the doctorate, are small but positive for programs in botany, cellular/ molecular biology, and zoology and close to zero for programs in the other three disciplines. Correlations of ratings of faculty quality with measure 06, the fraction of program graduates with definite employment plans, range be- tween .23 and .36 in the six biological disciplines. In every disci- pline the correlation of measure 08 is higher with measure 07, the fraction of graduates having agreed to employment at a Ph.D.-granting institution. These coefficients are approximately .60 in botany and zoology and .30 or above in the other four disciplines. Thus, those programs with the larger fractions of graduates intending to take aca- demic positions tended to receive higher survey ratings. The correlations of measure 08 with measure 09, the rated effec- tiveness of doctoral education, are uniformly very high, at or above .95 in every discipline. This finding is consistent with results from the Cartter and Roose-Andersen studies .6 The coefficients describing the relationship between measure 08 and measure 11, the familiarity with the work of program faculty, are also very high, ranging from .83 to .96. In general, evaluators were more likely to have high regard for the quality of faculty in those programs with which they were most familiar. That the correlation coeffients are as large as observed may simply reflect the fact that "known" programs tend to-be those that have earned strong reputations. Correlations of ratings of faculty quality with measure 10, the ratings of perceived improvement in program quality, are much smaller-- ranging from .19 in zoology to .46 in microbiology. One might have expected that a program judged to have improved in quality would have been somewhat more likely to receive high ratings on measure 08 than would a program judged to have declined--thereby imposing a small posi- tive correlation between these two variables. Moderately high correlations are observed in most disciplines be- tween measure 08 and university library size (measure 12), support for research (measures 13 and 14), and publication records (measures 15 and 16~. With few exceptions these coefficients are .S0 or greater in all disciplines. Of particular note are the strong correlations with the two publication measures--as high as .83 in biochemistry. In all dis- ciplines the correlations with measure 16 are as high as or slightly higher than those with measure 15; the "weighted influence" of journals 6 Roose and Andersen, p. 19.

172 in which articles are published yields an index that tends to relate more closely to faculty reputation than does an unadjusted count of the number of articles published. Although the observed differences be- tween the coefficients for measures 15 and 16 are not large, this re- sult is consistent with earlier findings of Anderson et al.7 Correlations with Measure 14. Correlations of measure 14, the reported dollars of support for R&D, with other measures are shown in Table 9.4. The reader is reminded that this measure reflects total university ex- penditures in the biological sciences and not expenditures in the six separate biological science disciplines. The pattern of relations is quite similar for programs in all six disciplines: moderately high correlations with reputational survey results {except measure 10), uni- versity library size, and publication measures. Measure 14 is also positively correlated with measures of program size {01-03), the frac- tion of recent graduates with fellowship/traineeship support (04), the fraction with definite commitments for employment in Ph.D.-granting universities (07), and the fraction of program faculty with federal research grants (13~. In interpreting these relationships one must keep in mind the fact that the research expenditure data have not been adjusted for the number of faculty and other staff members involved in research in a program. Correlations with Measure 16. Measure 16 is the number of published articles attributed to a program and adjusted for the "average influ- ence" of the journals in which the articles appear. The correlations of this measure with all others appear in Table 9.5. Of particular interest are the high correlations with the reputational survey results (excluding measure 10~. All of those coefficients exceed .60, and for biochemistry programs the coefficients are approximately .80. Moder- ately high correlations are also observed between measure 16 and mea- sures 12, 13, and 14; with few exceptions these correlations are .40 or higher. It should be pointed out that the exceptionally large coef- ficients reported for measure 15 result from the fact that the two pub- lication measures are logically as well as empirically interdependent. Despite the appreciable correlations between reputational ratings of quality and program size measures, the functional relations between the two probably are complex. If there is a minimum size for a high- quality program, this size is likely to vary from discipline to disci- pline. Increases in size beyond the minimum may represent more high- quality faculty, or a greater proportion of inactive faculty, or a faculty with heavy teaching responsibilities. In attempting to select among these alternative interpretations, a single correlation coeffi- cient provides insufficient guidance. Nonetheless, certain similari- ties across disciplines may be seen in the correlations among the measures. High correlations consistently appear among measures 08, 09, and 11 from the reputational survey, and these measures also are prom- 7Anderson, Narin, and McAllister, p. 95.

173 TABLE 9.4 Correlations of the University Research Expenditures in a Discipline (Measure 14) with Other Measures, by Discipline Bio- Cell./Molec. Micro- Physi- chemistry Botany Biology biology ology Zoology Program Size 01 .33 .31 .16 .24 .36 .20 02 .46 .09 .02 .18 .30 .42 03 .27 .17 .02 .08 .37 .28 Program Graduates 04 .50 .20 .30 .43 .31 .21 05 .03 .12 .25 .05 -.16 .08 06 .12 .04 .18 .12 .14 -.04 07 .29 .28 .20 .27 .23 .19 Survey Results 08 .69 .62 .57 .68 .51 .65 09 .64 .61 .47 .65 .51 .63 10 .03 -.07 -.16 .12 -.01 -.16 11 .69 .48 .58 .50 .46 .62 University Library 12 .56 .SS .42 .48 .S0 .63 Research Support 13 .30 .36 .24 .29 .28 .37 Publication Records 15 .71 .47 .58 .73 .43 .43 16 .71 .52 .59 .78 .48 .46

174 TABLE 9.5 Correlations of the Influence-Weighted Number of Publications (Measure 16) with Other Measures, by Discipline Bio- Cell./Molec. Micro- chemistry Botany Biology biology Physi- ology Zoology Program Size 01 .58 .55 .37 .38 .52 .36 02 .65 .24 .34 .29 .17 .59 03 .55 .32 .35 .18 .14 .34 Program Graduates 04 .62 -.11 .42 .51 .50 .16 05 .12 .16 .29 -.03 -.12 .18 06 .22 .18 .26 .13 .29 .11 07 .34 .26 .32 .27 .33 .41 Survey Results 08 .83 .62 .71 .75 .71 .64 09 .79 .63 .61 .68 .65 .66 10 .28 .24 .17 .27 .20 .24 11 .85 .56 .74 .62 .81 .68 University Library 12 .56 .37 .59 .54 .37 .47 Research Support 13 .47 .04 .4u .40 .~i .46 14 .71 .52 .59 .78 .48 .46 Publication Records 15 .Yd .96 .~Y .90 .Y/ .94

175 inently related to program size (measures 01, 02, and 03), to publica- tion productivity (measures IS and 16), to R&D expenditures=tmeasure 14), and to library size (measure 12~. These results show that for all disciplines the reputational rating measures (08, 09, and 11) tend to be associated with program size and with other correlates of size: publication volume, R&D expenditures, and library size. Also, for most disciplines the reputational measures 08, 09, and 11 tend to be posi- tively related to fellowship/traineeship support {measure 04), to em- ployment prospects of program graduates (especially measure 07), and to the fraction of faculty holding research grants (measure 13~. ANALYSI S OF THE SURVEY RESPONSE Measures 08-11, derived from the reputational survey, may be of particular interest to many readers since measures of this type have been the most widely used (and frequently criticized) indices of qual- ity of graduate education. In designing the survey instrument for this assessment the committee made several changes in the form that had been used in the Roose-Andersen study. The modifications served two pur- poses: to provide the evaluators with a clearer understanding of the programs that they were asked to judge and to provide the committee with supplemental information for the analysis of the survey response. One change was to restrict to 50 the number of programs that any indi- vidual was asked to evaluate. Probably the most important change was the inclusion of lists of names and ranks of individual faculty members involved in the research-doctorate programs to be evaluated~on the sur- vey form, together with the number of doctoral degrees awarded in the previous five years. Ninety percent of the evaluators were sent forms with faculty names and numbers of degrees awarded; the remaining 10 percent were given forms without this information, so that an analysis could be made of the effect of this modification on survey results. Another change was the addition of a question concerning an evaluator's familiarity with each of the programs. In addition to providing an in- dex of program recognition (measure 11), the inclusion of this question permits a comparison between the ratings furnished by individuals who had considerable familiarity with a particular program and the ratings by those not as familiar with the program. Each evaluator was also asked to identify his or her own institution of highest degree and cur- rent field of specialization. This information enables us to compare, for each program, the ratings furnished by alumni of that institution with the ratings by other evaluators, as well as to examine differences in the ratings supplied by evaluators in certain specialty fields. Before examining factors that may have influenced the survey re- sults, some mention should be made of the distributions of responses to the four survey items and the reliability {consistency) of the rat- ings. For example, in judging the scholarly quality of faculty (mea- sure 08), survey respondents in each discipline rated between 5 and 8 percent of the programs as being "distinguished" and approximately 1 percent as "not sufficient for doctoral education" (see Table 9.6~. In evaluating the effectiveness in educating research scholars/scientists,

176 Survey Measure TABLE 9.6 Distribution of Responses to Each Survey Item, by Discipline Bio- Cell./ Micro- Physi- Total chem. Botany Molec. biology ology Zoology 08 SCHOLARLY QUALITY OF PROGRAM FACULTY Distinguished 6.3 7.0 5.9 8.1 5.3 6.0 5.4 Strong 14.5 12.7 16.1 16.9 14.1 14.4 14.5 Good 16.9 17.0 16.7 17.8 17.0 14.7 18.5 Adequate 10.3 12.0 8.8 8.6 10.5 8.1 12.8 Marginal 3.9 5.5 2.8 3.1 3.5 2.9 4.9 Not Sufficient for Doctoral Education 1.1 1.4 1.1 1.0 1.0 .8 1.3 Don't Know Well Enough to Evaluate 47.0 44.5 48.5 44.4 48.5 53.3 42.6 TOTAL 100.0 100.0 100.Q 100.0 100.0 100.0 100.0 O9 EFFECTIVENESS OF PROGRAM IN EDUCATING SCIENTISTS Extremely Effective 7.2 7.2 7.6 8.1 6.8 6.7 6.9 Reasonably Effective 24.6 22.3 27.7 22.4 25.4 22.0 28.8 Minimally Effective 8.4 8.6 8.1 6.7 8.4 7.4 ll.o Not Effective 1.4 1.6 1.1 1.3 1.2 1.1 2.4 Don't Know Well Enough to Evaluate 58.5 60.3 55.5 61.5 58.2 62.7 51.0 TOTAL 100.0 100.0 100.0 100.0 100.0 100.0 100.0 10 CHANGE IN PROGRAM QUALITY IN LAST FIVE YEARS Better 9.8 8.5 10.2 11.1 9.9 8.4 11.5 Little or No Change 19.6 19.3 22.6 18.3 18.4 17.6 22.1 Poorer 3.8 3.6 3.8 3.8 3.6 3.7 4.5 Don't Know Well Enough to Evaluate 66.9 68.6 63.4 66.7 68.1 70.3 61.9 TOTAL 100.0 100.0 100.0 100.0 100.0 100.0 100.0 11 FAMILIARITY WITH WORK OF PROGRAM FACULTY Considerable 15.2 13.8 17.4 18.4 15.1 12.1 15.0 Some 34.3 36.3 31.5 34.0 33.5 32.6 37.5 Little or None 47.8 46.2 48.3 44.3 50.3 50.9 45.7 No Response 2.8 3.6 2.8 3.3 1.1 4.4 1.8 TOTAL 100.0 100.0 100.0 100.0 100.0 100.0 100.0 NOTE: For survey measures 08, 09, 10 the "don't know" category includes a small number of cases for which the respondents provided no response to the survey item.

~ 7" they rated 7 to 8 percent of the programs as being "extremely effec- t~ve" and approximately 1 to 2 percent as "not effective.n -Of~partic- ular interest in this table are the frequencies with which evaluators failed to provide responses to measures 08, 09, and 10. Approximately 47 percent of the total number of evaluations requested for measure 08 were not furnished because survey respondents in the biological sci- ences felt that they were not familiar enough with a particular program to evaluate it. The corresponding percentages of "don't know" re- sponses for measures 09 and 10 are considerably larger--58 and 67 per- cent, respectively--suggesting that survey respondents found it more difficult (or were less willing) to judge program effectiveness and change than to judge the scholarly quality of program faculty. The percentages of "don't know" responses are greater in the biological disciplines than in most other science, engineering, or humanities disciplines. This finding may be attributed, in part, to the fact that many different academic units or departmental fields are represented in the assessment in each biological discipline and that survey evalu- ators were less likely to be familiar with programs outside their own fields of specialty. The large fractions of "don't know" responses are a matter of some concern. However, given the broad coverage of research-doctorate pro- grams, it is not surprising that faculty members would be unfamiliar with many of the less distinguished programs. As shown in Table 9.7, survey respondents in each discipline were much more likely to furnish evaluations for programs with high reputational standing than they were for programs of lesser distinction. For example, for biological sci- ence programs that received mean ratings of 4.0 or higher on measure 08, almost 85 percent of the evaluations requested on measure 08 were provided; 72 and 57 percent, respectively, were provided on measures 09 and 10. In contrast, the corresponding response rates for programs with mean ratings below 2.0 are much lower--30, 22, and 16 percent re- sponse on measures 08, 09, and 10, respectively. Of great importance to the interpretation of the survey results is the reliability of the responses. How much confidence can one have in the reliability of a mean rating reported for a particular program? In the second table in each of the preceding six chapters, estimated standard errors associated with the mean ratings of every program are presented for all four survey items (measures 08-111. While there is some variation in the magnitude of the standard errors reported in every discipline, they rarely exceed .20 for any of the four measures and typically range from .10 to .15. For programs with higher mean ratings the estimated errors associated with these means are generally smaller--a finding consistent with the fact that survey respondents were more likely to furnish evaluations for programs with high reputa- tional standing. The "split-half" correlations presented in Table 9.8 For a discussion of the interpretation of "split-half" coefficients, see Robert L. Thorndike and Elizabeth Hagan, Measurement and Evaluation in Psychology and Education, John Wiley & Sons, New York, 1969, pp. 182-185.

178 TABLE 9.7 Survey Item Response Rates, by Discipline and Mean Rating on Measure 08 Survey Measure Total Bio- Cell./ Micro- Physi- chem. Botany Molec. b iology ology Zoology 08 SCHOLARLY QUALITY OF PROGRAM FACULTY Mean Rating on Measure 08 4.0 or Higher 84.7 95.9 67.7 89.9 82.0 81.6 91.1 3.0 - 3.9 65.9 76.0 57.9 72.6 64.2 57.0 75.4 2.0 - 2.9 44.8 48.8 43.9 43.8 44.1 36.3 50.1 Less than 2.0 29.9 32.2 34.2 23.4 27.6 21.8 35.3 O9 EFFECTIVENESS OF PROGRAM IN EDUCATING SCIENTISTS Mean Rating on Measure 08 4.0 or Higher 72.2 78.3 61.5 72.3 69.9 70.9 83.4 3.0 - 3.9 52.3 56.1 51.3 49.7 52.5 45.1 65.2 2.0 - 2.9 33.S 33.5 36.5 27.1 35.1 28.1 41.6 Less than 2.0 21.6 19.8 26.7 15.6 21.9 18.1 29.1 10 CHANGE IN PROGRAM QUALITY IN LAST FIVE YEARS Mean Rating on Measure 08 4.0 or Higher S7.3 62.0 51.4 60.5 51.3 56.0 64.4 3.0 - 3.9 42.5 45.3 43.2 44.1 39.9 36.1 51.7 2.0 - 2.9 27.1 26.8 29.2 24.6 27.3 22.6 32.2 Less than 2.0 15.9 14.0 20.0 11.7 16.5 13.4 21.6 give an indication of the overall reliability of the survey results in-each discipline and for each measure. In the derivation of these correlations individual ratings of each program were randomly divided into two groups (A and B), and a separate mean rating was computed for each group. The last column in Table 9.8 reports the correlations be- tween the mean program ratings of the two groups and is not corrected for the fact that the mean ratings of each group are based on only half rather than a full set of responses.9 As the reader will note, the coefficients reported for measure 08, the scholarly quality of program faculty, are in the range of .91 to .96--indicating a high degree of consistency in evaluators' judgments. The correlations reported for measures 09 and 11, the rated effectiveness of a program and the eval- uators' familiarity with a program, are somewhat lower but still at a level of .87 or higher in every discipline, except physiology. Not surprisingly, the reliability coefficients for ratings of change in program quality in the last five years (measure 10) are considerably lower, ranging from .48 to .68 in the six biological science disci- 9To compensate for the smaller sample size the "split-half" coeffi- cient may be adjusted using the Spearman-Brown formula: r' = 2r/~1 ~ r). This adjustment would have the effect of increasing a correlation of .70, for example, to .82, a correlation of .80 to .89, a correlation of .90 to .95, and a correlation of .95 to .97.

179 TABLE 9.8 Correlations Between Two Sets of Average Ratings from Two Randomly Selected Groups of Evaluators in the Biological Sciences MEASURE 08: SCHOLARLY QUALITY OF PROGRAM FACULTY Discipline Mean Rating Std. Deviation Correlation . Group A Group ~ Group A Group B N r Biochemistry 2.65 2.62 .94 .98 138 .96 Botany 3.02 3.03 .84 .82 83 .95 Cell/Mblec. Biol. 2.86 2.90 1.02 .97 88 .96 Microbiology 2.83 2.81 .79 .82 134 .93 Physiology 2.93 2.94 .83 .77 101 .91 Zoology 2.71 2.74 .84 .78 70 .95 MEASURE 09: EFFECTIVENESS OF PROGRAM IN EDUCATING SCHOLARS Discipline Mean Rating Std. Deviation Correlation , Group A Group B Group A Group B N r Biochemistry 1.67 1.63 .52 .54 138 .88 Botany 1.82 1.84 .45 .45 83 .90 Cell/Molec. Biol. 1.73 1.75 .52 .53 88 .89 Microbiology 1.75 1.76 .45 .42 134 .87 Physiology 1.76 1.76 .43 .46 101 .81 Zoology 1.66 1.69 .45 .45 70 .89 MEASURE 10: IMPROVEMENT IN PROGRAM IN LAST FIVE YEARS Discipline Mean Rating Std. Deviation Correlation Group A Group B Group A Group ~ N r Biochemistry 1.11 1.12 .27 .24 138 .55 Botany 1.15 1.17 .20 .22 83 .64 Cell/Molec. Biol. 1.17 1.17 .29 .27 88 .48 Microbiology 1.14 1.15 .28 .28 134 .68 Physiology 1.13 1.12 .27 .28 101 .55 Zoology 1.17 1.15 .21 .24 70 .64 MEASURE 11: FAMILIARITY WITH WORK OF PROGRAM FACULTY Discipline Mean Rating Std. Deviation Correlation Group A Group B Group A Group ~ N r Biochemistry .67 .6S .42 .42 138 .95 Botany .68 .68 .30 .29 83 .87 Cell/Molec. Biol. .72 .73 .43 .44 88 .9S Microbiology .64 .64 .34 .33 134 .90 Physiology .58 .61 .32 .32 101 .87 Zoology .68 .69 .36 .36 70 .92

180 plines. While these coefficients represent tolerable reliability, it is quite evident that the responses to measure 10 are-not as reliable as the responses to the other three items. In interpreting program ratings for this measure the reader is urged to refer to the standard error estimates reported with the ratings. Further evidence of the reliability of the survey responses is presented in Table 9.9. As mentioned in Chapter VI, 11 mathematics programs selected at random were included on a second form sent to 178 survey respondents in this discipline, and 116 individuals (65 percent) furnished responses to the second survey. A comparison of the overall results of the two survey administrations (columns 2 and 4 in Table 9.9) demonstrates the consistency of the ratings provided for each of the 11 programs. The average, absolute observed difference in the two sets of mean ratings is less than 0.1 for each measure. Col- umns 6 and 8 of Table 9.9 report the results based on the responses of only those evaluators who had been asked to consider a particular pro- gram in both administrations of the survey. (For a given program ap- proximately 40-45 percent of the 116 respondents to the second survey had been asked to evaluate that program in the prior survey.) It is not surprising to find comparable small differences in the mean ratings provided by this subgroup of evaluators. Critics of past reputational studies have expressed concern about the credibility of reputational assessments when evaluators provide judgments of programs about which they may know very little. As al- ready mentioned, survey participants in this study were offered the explicit alternative, "Don't know well enough to evaluate." This re- sponse option was quite liberally used for measures 08, 09, and 10, as is shown in Table 9.6. In addition, evaluators were asked to indicate their degree of familiarity with each program. Respondents reported "considerable" familiarity with an average of only one program in every six or seven. While this finding supports the conjecture that many program ratings are based on limited information, the availability of reported familiarity permits us to analyze how ratings vary as a func- tion of familiarity. This issue can be addressed in more than one way. It is evident from the data reported in Table 9.10 that mean ratings of the scholarly quality of program faculty tend to be higher if the evaluator has con- siderable familiarity with the program. There is nothing surprising or, for that matter, disconcerting about such an association. When a particular program fails to provoke more than vague images in the evaluator's mind, he or she is likely to take this as some indication that the program is not an extremely lustrous one on the national scene. While visibility and quality are scarcely the same, the world of research in higher education is structured to encourage high quality to achieve high visibility, so that any association of the two is far from spurious. Mathematics is the only discipline in which results were obtained from two separate administrations of the survey.

181 TABLE 9.9 Comparison of Mean Ratings for 11 Mathematics Programs Included in Two Separate Survey Administrations Survey All Evaluators Measure First N X Second N X Evaluators Rating the Same Program in Both Surveys First N X Second N X Program A 08 100 4.9 114 4.9 50 4.9 50 4.9 09 90 2.7 100 2.8 42 2.7 43 2.7 10 74 1.2 83 1.2 38 1.1 34 1.2 11 100 1.6 llS 1.6 50 1.5 50 1.6 Program B 08 94 4.6 115 4.6 48 4.6 50 4.5 09 81 2.6 91 2.5 40 2.6 39 2.5 10 69 1.0 82- 1.0 37 1.0 36 0.9 11 98 1.4 116 1.4 50 1.5 50 1.5 Program C 08 86 3.4 103 3.6 42 3.4 44 3.5 09 56 2.0 66 2.1 28 2.1 29 2.0 10 55 1.1 62 1.3 30 1.2 27 1.4 11 99 1.0 116 1.1 50 1.1 50 1.0 Program D 08 74 3.0 93 3.0 37 2.8 38 2.9 09 50 1.8 48 1.6 27 1.7 16 1.6 10 46 1.4 52 1.5 24 1.4 23 1.5 11 90 1.0 113 0.9 46 1.0 46 0.9 Program E 08 69 3.0 95 3.1 39 3.0 46 3.1 09 40 1.8 60 1.9 25 1.8 30 1.8 10 36 0.8 58 0.9 24 0.8 29 0.9 11 96 0.8 115 0.9 52 0.9 52 1.0 Program F 08 63 2.9 90 3.D Z6 3.O 3-c 3.l 09 35 1.8 46 1.7 10 1.6 13 1.8 10 32 1.1 43 1.1 11 1.3 12 1.2 11 95 0.7 115 0.8 43 0.7 44 0.7 Program ~ 08 bY -c.~ Y-c -c.~ dY c.-/ dY d.U 09 35 1.7 45 1.6 17 1.7 19 1.7 10 36 1.1 43 1.2 17 1.1 19 1.2 11 85 0.9 116 0.8 46 0.9 46 0.9 Program ~ 08 ~u c.c In c.~ ~o c.c ~' c 09 32 1.3 43 1.3 22 1.2 19 1.3 10 30 1.5 39 1.5 20 1.7 17 1.4 11 90 0.7 116 0.6 51 0.7 52 0.6 Program I 08 55 2.0 74 1.9 30 1.9 30 2.0 09 33 1.0 41 0.9 19 1.0 18 0.8 10 27 1.2 31 1.1 15 1.1 13 1.2 11 99 0.5 115 0.5 50 0.5 50 0.5 rLv~,".l. ~ 08 ~1 1.~ a, 1.~ no 1.~ c. 1." 09 31 0.8 36 0.7 14 0.6 14 0.7 10 26 1.2 23 1.1 14 1.2 12 1.3 11 96 0.5 113 0.3 49 0.4 48 0.4 ¢~v~"ll~ ~ 08 ~~ ..^ "~ ..^ . , .. _ ^- _. 09 19 0.8 21 0.5 11 0.6 8 0.4 10 12 0.8 15 0.9 5 1.0 5 0.8 11 99 0.2 114 0.2 48 0.2 47 0.2

182 TABLE 9.10 Mean Ratings of Scholarly Quality of Program Faculty, by Evaluator's Familiarity with Work of *acuity MEAN RATINGS Consid- Some/ erable Little CORRELATION r N Biochemistry 2.99 2.59 .86 133 Botany 3.30 2.90 .85 82 Cellular/Molecular Biology 3.33 2.88 .90 83 Microbiology 3.17 2.73 .89 130 Physiology 3.54 2.83 .77 97 Zoology 3.09 2.62 .88 69 NOTE: N reported in last column represents the number of programs with a rating from at least one evaluator in each of the two groups. From the data presented in Table 9.10 it is evident that if mean ratings were computed on the basis of the responses of only those most familiar with programs, the values reported for individual programs would be increased. A largely independent question is whether a re- striction of this kind would substantially change our sense of the relative standings of programs on this measure. Quite naturally, the answer depends in some degree on the nature of the restriction imposed. For example, if we exclude evaluations provided by those who confessed "little or no" familiarity with particular programs, then the revised mean ratings would be correlated at a level of at least .99 with the mean ratings computed using all of the data. (This similarity arises, in part, because only a small fraction of evaluations are given on the basis of no more than ~little" familiarity with the program.) The third column in Table 9.10 presents the correlation in each discipline between the array of mean ratings supplied by respondents claiming "considerable" familiarity and the mean ratings of those in- dicating "some" or "little or no" familiarity with particular programs. This coefficient is a rather conservative estimate of agreement since there is not a sufficient number of ratings from those with "consider- able" familiarity to provide highly stable means. Were more such rat- ings available, one might expect the correlations to be higher. How- ever, even in the form presented, the correlations, which are at least .85 in all disciplines but physiology, are high enough to suggest that the relative standing of programs on measure 08 is not greatly affected by the admixtures of ratings from evaluators who recognize that their knowledge of a given program is limited. As mentioned previously, 90 percent of the survey sample members were supplied the names of faculty members associated with each program to be evaluated, along with the reported number of program graduates (Ph.D. or equivalent degrees) in the previous five years. Since ear- lier reputational surveys had not provided such information, 10 percent

i83 TABLE 9.11 Item Response Rate on Measure 08, by Selected Characteristics of Survey Evaluators in the Biological Sciences Bio- Cell./ Micro- Physi- Total them. Botany Molec. biology ology Zoology EVALUATOR'S FAMILIARITY WITH PROGRAM Considerable 99.9 99.9 99.5 100.0 100.0 100.0 100.0 Some 95.3 95.3 94.8 94.4 95.5 95.1 96.7 Little or None 10.3 14.3 8.5 10.7 8.6 6.5 12.8 TYPE OF SURVEY FORM Names 53.1 56.0 51.4 56.1 51.8 47.0 56.1 No Names 52.2 52.0 52.0 48.0 48.1 43.2 68.2 INSTITUTION OF HIGHEST DEGREE Alumni 91.6 96.0 84.0 94.1 91.4 95.1 96.4 Nonalumni 52.6 55.3 50.9 55.2 51.2 46.5 57.0 EVALUATOR'S PROXIMITY TO PROGRAM Same Region 64.4 Outside Region 51.4 66.1 63.3 64.1 64.0 56.7 7S.3 54.1 50.0 54.3 49.6 45.3 S5.3 NOTE: The item response rate is the percentage of the total ratings requested from survey participants that included a response other than "don't know. n of the sample members, randomly selected, were given forms without fac- ulty names or doctoral data, as a "control group." Although one might expect that those given faculty names would have been more likely than other survey respondents to provide evaluations of the scholarly qual- ity of program faculty, no appreciable differences were found {see Table 9.11) between the two groups in their frequency of response to this survey item. {The reader may recall that the provision of faculty names apparently had a positive effect on survey sample members' wil- lingness to complete and return their questionnaires. The mean ratings provided by the group furnished faculty names are generally lower than the mean ratings supplied by other respondents (Table 9.12~. Although the differences are small, they attract atten- tion because they are reasonably consistent from discipline to disci- pline {except in physiology and zoology) and because the direction of the differences was not anticipated. After all, those programs more familiar to evaluators tended to receive higher ratings, yet when steps were taken to enhance the evaluator's familiarity, the resulting rat- ings are somewhat lower. One post hoc interpretation of this finding leas shown in Table 2.3, the survey response rate for those furnished faculty names is approximately 7 percentage points higher than that for those not given this information.

184 TABLE 9.12 Mean Ratings of Scholarly Quality of Program Faculty, by Type of Survey Form Provided to Evaluator MEAN RATINGS Names No Names r N CORRELATION Biochemistry 2.63 2.83 .76 137 Botany 3.01 3.19 .75 83 Cellular/Molecular Biology 2.94 3.33 .80 79 Microbiology 2.84 3.14 .57 126 Physiology 3.06 2.86 .52 84 Zoology 2.72 2.71 .88 70 NOTE: N reported in last column represents the number of programs with a rating from at least one evaluator in each of the two groups. is that a program may be considered to have distinguished faculty if even only a few of its members are considered by the evaluator to be outstanding in their field. However, when a full list of program fac- ulty is provided, the evaluator may be influenced by the number of in- dividuals whom he or she could not consider to be distinguished. Thus, the presentation of these additional, unfamiliar names may occasionally result in a lower rating of program faculty. However interesting these effects may be, one should not lose sight of the fact that they are small at best and that their existence does not necessarily imply that a program's relative standing on measure 08 would differ much whichever type of survey form was used. Since only about 1 in 10 ratings was supplied without the benefit of faculty names, it is hard to establish any very stable picture of relative mean ratings of individual programs. However, the correlations between the mean ratings supplied by the two groups are reasonably high--.75 or greater in four disciplines and .S7 and .52 in microbiology and physi- ology, respectively (see Table 9.12~. Were these coefficients adjusted for the fact that the group furnished forms without names constituted only about 10 percent of the survey respondents they would be substan- tially larger. From this result it seems reasonable to conclude that differences in the alternative survey forms used are not likely to be responsible for any large-scale reshuffling in the reputational ranking of programs on measure 08. It also suggests that the inclusion of faculty names in the committee's assessment need not prevent compari- sons of the results with those obtained from the Roose-Andersen survey. Another factor that might be thought to influence an evaluator's judgment about a particular program is the geographic proximity of that program to the evaluator. There is enough regional traffic in academic life that one might expect proximate programs to be better known than

185 TABLE 9.13 Mean Ratings of Scholarly Quality of Program Faculty, by Evaluator's Proximity to Region of Program MEAN RATINGS Nearby Outside r N CORRELATION Biochemistry 2.71 2.63 .92 134 Botany 2.99 3.03 .88 80 Cellular/Molecular Biology 2.91 2.91 . 84 83 Microbiology 2.89 2.84 .87 131 Physiology 2.99 2.97 .76 97 Zoology 2.80 2.73 .86 68 NOTE: N reported in last column represents the number of programs with a rating from at least one evaluator in each of the two groups. those in distant regions of the country. This hypothesis may apply especially to the smaller and less visible programs and is confirmed by the survey results. For purposes of analysis, programs were as- signed to one of nine geographic regionsi2 in the United States, and ratings of programs within an evaluator's own region are categorized in Table 9.13 as "nearby." Ratings of programs in any of the other eight regions were put in the "outside" group. Findings reported elsewhere in this chapter confirm that evaluators were more likely to provide ratings if a program was within their own region of the country,~3 and it is reasonable to imagine that the smaller and the less visible programs received a disproportionate share off thmi r racing "; Thor Prom evaluators within their own region or from ~ another were particularly familiar with programs in that region. Although the data in Table 9.13 show that "nearby" programs were given higher ratings than those outside the evaluator's region in four of the disciplines, the differences in reported means are quite small and probably represent no more than a secondary effect that might be expected, because, as we have already seen, evaluators tended to rate higher those programs with which they were more familiar. Furthermore, the reasonably high correlations found between the mean ratings of the two groups indicate that the relative standings of programs are not dramatically influenced by the geographic proximity of those evaluating them. Another consideration that troubles some critics is that large programs may be unfairly favored in a faculty survey because they are ~ ^~— — — ~~~ others who for one reason or Resee Appendix I for a list of the states included in each region. Resee Table 9.11.

186 TABLE 9.14 Mean Ratings of Scholarly Quality of Program Faculty, by Evaluator's Institution of Highest Degree MEAN RATINGS NUMBER OF PROGRAMS WITH ALUMNI RATINGS N Alumn i Nonalumni Biochemistry 3.80 3.60 31 Botany 3.76 3.43 46 Cellular/Molecular Biology 3.69 3.46 35 Microbiology 3.36 3.11 51 Physiology 3.82 3.30 29 Zoology 3.56 3.32 30 NOTE: The pairs of means reported in each discipline are computed for a subset of programs with a rating from at least one alumnus and are substantially greater than the mean ratings for the full set of programs in each discipline. likely to have more alumni contributing to their ratings who, it would stand to reason, would be generous in the evaluations of their alma maters. Information collected in the survey on each evaluator's insti- tution of highest degree enables us to investigate this concern. The findings presented in Table 9.14 support the hypothesis that alumni provided generous ratings--with differences in the mean ratings (for measure 08) of alumni and nonalumni ranging from .20 to .S2 in the six disciplines. Given the appreciable differences between the ratings furnished by program alumni and other evaluators, one might ask how much effect this has had on the overall results of the survey. The answer is "very little." As shown in the table, in biochemistry only one program in every four received ratings from any alumnus; in botany slightly more than half of the programs were evaluated by one or more alumni.) 4 Even in the latter discipline, however, the fraction of alumni providing ratings of a program is always quite small and should have had minimal impact on the overall mean rating of any program. To be certain that this was the case, mean ratings of the scholarly qual- ity of faculty were recalculated for every biological science program-- with the evaluations provided by alumni excluded. The results were compared with the mean scores based on a full set of evaluations. Out of the 616 biological science programs evaluated in the survey, only 1 program (in physiology) had an observed difference as large as 0.2, and for 574 programs (93 percent) their mean ratings remain unchanged (to because of the small number of alumni ratings in every discipline, the mean ratings for this group are unstable and therefore the correla- tions between alumni and nonalumni mean ratings are not reported.

187 the nearest tenth of a unit). On the basis of these findings the com- mittee saw no reason to exclude alumni ratings in the calculation of program means. Another concern that some critics have is that a survey evaluation may be affected by the interaction of the research interests of the evaluator and the areats) of focus of the research-doctorate program to be rated. It is said, for example, that some narrowly focused pro- grams may be strong in a particular area of research but that this strength may not be recognized by a large fraction of evaluators who happen to be unknowledgeable in this area. This is a concern more dif- ficult to address than those discussed in the preceding pages since little or no information is available about the areas of focus of the programs being evaluated (although in certain disciplines the title of a department or academic unit may provide a clue}. To obtain a better understanding of the extent to which an evaluator's field of specialty may have influenced the ratings he or she has provided, an analysis was made of ratings provided by evaluators in physics and statistics/ biostatistics. In each discipline the survey participants were divided into two groups according to specialty field (as reported on the survey questionnaire). The results of the analysis, which are presented in the mathematical and physical sciences volume of the committee's re- port, indicate that there is a high degree of correlation in the mean ratings provided by those in differing specialty fields within these two disciplines. Although one cannot conclude from these findings that an evaluator's specialty field has no bearing on how he or she rates a program, these findings do suggest that the relative standings of pro- grams in physics and statistics/biostatistics would not be greatly al- tered if the ratings by either group were discarded. In the biological science disciplines there is a corollary concern that a survey evaluator may be influenced by the interaction of the university setting in which the evaluator works and the setting in which the program to be evaluated resides. For example, one might sus- pect that medical school faculty members would tend to rate research- doctorate programs in schools of medicine higher than would evaluators from other university settings. To test this hypothesis evaluators in each of the four biomedical disciplines--biochemistry, cellular/molec- ular biology, microbiology, and physiology--were separated into two groups: (1) those included on the faculties of programs located in medical schools and (2) those in other programs. For purposes of analysis, if a program spans both the medical and nonmedical school settings, the survey evaluators selected from that program have been included in the first group. The percentage of evaluators from medical school programs varies considerably by disciplines: biochemistry (60 percent), cellular/molecular biology (32 percent), microbiology (46 percent), and physiology (68 percent). As shown in Table 9.15, there is a high degree of correlation between the mean ratings provided by medical school and other faculty members. For the four biomedical disciplines the coefficients range from .83 to .96, suggesting that the relative standings of programs in each discipline would not be greatly affected if the survey responses of one of these two groups were ig- nored.

188 TABLE 9.15 Mean Ratings of Scholarly Quality of Program Faculty, by School in Which Evaluator's Own Program IS Located MEAN RATINGS CORRELATION Medical School Other r N Biochemistry 2.61 2.67 .96 138 Cellular/Molecular Biology 2.80 2.91 .94 88 Microbiology 2.89 2.77 .93 134 Physiology 2.87 3.01 .83 101 NOTE: N reported in last column represents the number of programs with a rating from at least one evaluator in each of the two groups. INTERPRETATION OF REPUTATIONAL SURVEY RATINGS It is not hard to foresee that results from this survey will re- ceive considerable attention through enthusiastic and uncritical re- porting in some quarters and sharp castigation in others. The study committee understands the grounds for both sides of this polarized re- sponse but finds that both tend to be excessive. It is important to make clear how we view these ratings as fitting into the larger study of which they are a part. The reputational results are likely to receive a disproportionate degree of attention for several reasons, including the fact that they reflect the opinions of a large group of faculty colleagues and that they form a bridge with earlier studies of graduate programs. But the results will also receive emphasis because they alone, among all of the measures, seem to address quality in an overall or global fashion. While most recognize that "objectives program characteristics (i.e., publication productivity, research funding, or library size) have some bearing on program quality, probably no one would contend that a single one of these measures encompasses all that need be known about the quality of research-doctorate programs. Each is obviously no more than an indicator of some aspect of program quality. In contrast, the repu- tational ratings are global from the start because the respondents are asked to take into account many objective characteristics and to arrive at a general assessment of the quality of the faculty and the effec- tiveness of the program. his generality has self-evident appeal. On the other hand, it is wise to keep in mind that these reputa- tional ratings are measures of perceived program quality rather than of "quality" in some ideal or absolute sense. What this means is that, just as for all of the more objective measures, the reputational rat- ings represent only a partial view of what most of us would consider quality t; be; hence, they must be kept in careful perspective.

189 Some critics may argue that such ratings are positively misleading because of a variety of methodological artifacts or because they are supplied by "judges" who often know very little about the programs they are rating. The committee has conducted the survey in a way that per- mits the empirical examination of a number of the alleged artifacts and, although our analysis is by no means exhaustive, the general con- clusion is that their effects are slight. Among the criticisms of reputational ratings from prior studies are some that represent a perspective that may be misguided. This perspec- tive assumes that one asks for ratings in order to find out what ~qual- ity" really is and that to the degree that the ratings miss the mark of "quintessential quality, n they are unreal, although the quality that they attempt to measure is real. What this perspective misses is the reality of quality and the fact that impressions of quality, if widely shared, have an imposing reality of their own and therefore are worth knowing about in their own right. After all, these perceptions govern a large-scale system of traffic around the nation's graduate institu- tions--for example, when undergraduate students seek the advice of pro- fessors concerning graduate programs that they might attend. It is possible that some professors put in this position disqualify theme selves on grounds that they are not well informed about the relative merits of the programs being considered. Most faculty members, how- ever, surely attempt to be helpful on the basis of impressions gleaned from their professional experience, and these assessments are likely to have major impact on student decision-making. In short, the impres- sions are real and have very real effects not only on students shopping for graduate schools but also on other flows, such as job-seeking young faculty and the distribution of research resources. At the very least, the survey results provide a snapshot of these impressions from disci- pline to discipline. Although these impressions may be far from ideally informed, they certainly show a strong degree of consensus within each discipline, and it seems safe to assume that they are more than passingly related to what a majority of keen observers might agree program quality is all about. COMPARI SON WITH RESULTS OF THE ROOSE—ANDERSEN STUDY An analysis of the response to the committee's survey would not be complete without comparing the results with those obtained in the sur- vey by Roose and Andersen 12 years earlier. Although there are obvious similarities in the two surveys, there are also some important differ- ences that should be kept in mind in examining individual program rat- ings of the scholarly quality of facultye Already mentioned in this chapter is the inclusion, on the form sent to 90 percent of the sample members in the committee's survey, of the names and academic ranks of faculty and the numbers of doctoral graduates in the previous five years. Other significant changes in the committee's form are the identification of the university department or academic unit in which each program may be found, the restriction of requesting evaluators to

190 make judgments about no more than 50 research-doctorate programs in their discipline, and the presentation of these programs in random se- quence on each survey form. The sampling frames used in the two sur- veys also differ. The sample selected in the earlier study included only individuals who had been nominated by the participating universi- ties, while more than one-fourth of the sample in the committee's sur- vey were chosen at random from full faculty lists. (Except for this difference the samples were quite similar--i.e., in terms of the number of evaluators in each discipline and the fraction of senior scholars.~5) Several dissimilarities in the coverage of the Roose-Andersen study and this committee's reputational assessments should be mentioned. m e former included a total of 130 institutions that had awarded at least 100 doctoral degrees in two or more disciplines during the FY1958-67 period. The institutional coverage in the committee's assessment was based on the number of doctorates awarded in each discipline (as de- scribed in Chapter I) and covered a total population of 228 universi- ties. Most of the universities represented in the present study but not the earlier one are institutions that offered research-doctorate programs in a limited set of disciplines. In the Roose-Andersen study, programs in 10 biological science disciplines were rated: biochemis- try, botany, developmental biology, entomology, microbiology, molecular biology, pharmacology, physiology, population biology, and zoology. For reasons explained in Chapter I, the committee in its assessment was able to include programs in the six disiplines with the largest numbers of doctoral awards in recent years. 6 Programs in the other four disciplines--developmental biology, entomology, pharmacology, and popu- lation biology--were not evaluated in the present assessment. Finally, in the Roose-Andersen study only one set of ratings was compiled from each institution represented in a discipline, whereas in the commit- tee's survey separate ratings were requested if a university offered more than one research-doctorate program in a given discipline. In the committee's survey, evaluations were requested for a total of 616 re- search-doctorate programs in the six biological science disciplines, compared with 618 programs in these same disciplines in the Roose- Andersen study. Although the total numbers of programs included in the studies are nearly equal, there are many differences in the program coverage in each discipline. Figures 9.1-9.6 plot the mean ratings of scholarly quality of fac- ulty in programs included in both surveys; sets of ratings are graphed for 90 programs in biochemistry, 52 programs in botany, 53 programs in cellular/molecular biology, 82 programs in microbiology, 63 programs in physiology, and 48 programs in zoology. Since in the Roose-Andersen for a description of the sample group used in the earlier study, see Roose and Andersen, pp. 28-31. wit should be noted that the ''molecular biology" category used in the Roose-Andersen study was expanded in the committee's assessment to include cellular and molecular biology programs.

191 s . o++ * * * * 4.0++ + + + + + + + + Measure + 3.0++ 08 + + + 2. 0++ + + 1.0++ + + + + * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * r = .92 0.0 + + + + + + + + + + + + + + + + + + + + + + + + + + + + + 1.0 2.0 3.0 4.0 5.0 Roose-Andersen Rating (1970) FIGURE 9.1 Mean rating of scholarly quality of faculty (measure 08) versus mean rating of faculty in the Roose-Andersen study--90 programs in biochemistry.

192 s . o++ + + + 4.0++ . Measure + 3.0++ 0 8 + + + + + 2. 0++ + + + + 1. 0++ + + + * * * * * * * * * * * * * * * *** : * * * * * * * * * * * 0.0 + + + + + + + + + + + + + + + + + + + + + + + + + + + + 1.0 2.0 3.0 Roose-Andersen Rating (1970) * * * * * * * * * * * + + + + + + + + t + + + + + + + + + + + + + + 4.0 5.0 FIGURE 9.2 Mean rating of scholarly quality of faculty (measure 08) versus mean rating of faculty in the Roose-Andersen study--52 programs in botany.

193 s.o++ + + + + + 4. 0++ + + + Measure + 3.0++ 08 + + + + + 2. 0++ + 1.0++ + + + + + 0.0 + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + 1.0 2.0 3.0 4.0 5.0 * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * r = .87 * * * Roose-Andersen Rating (1970) FIGURE 9.3 Mean rating of scholarly quality of faculty (measure 08) versus mean rating of faculty in the Roose-Andersen study--53 programs in cellular/molecular biology.

194 Measure 5 . 0++ 4.0+ 3 . 0++ + 2. 0++ + 1. 0++ * * * * * * * * * * * 0.0 + + + + + + + + + + + + + + 1.0 * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * + * * * * * r = .87 + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + 2.0 3.0 4.0 5.0 Roose-Andersen Rating (1970 ) FIGURE 9.4 Mean rating of scholarly quality of faculty (measure 08) versus mean rating of faculty in the Roose-Andersen study--82 programs in microbiology.

195 s.o++ + + + + + 4.0++ + Measure + 3.0++ 08 + + + 2.0++ + + + 1 .0++ + * * * * * * * * t * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * 0.0 + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + 1.0 2.0 3.0 4.0 5.0 Roose-Andersen Rating (1970) FIGURE 9.5 Mean rating of scholarly quality of faculty (measure 08) versus mean rating of faculty in the Roose-Andersen study--63 programs in physiology.

196 s. o++ 4. 0++ + Measure + 3.0++ 08 + + + + 2.0+ 1 . 0+ * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * r = .83 0.0 + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + 1.0 2.0 3.0 4.0 5.0 Roose-Andersen Rating ( 197 0 ) FIGURE 9.6 Mean rating of scholarly quality of faculty (measure 08) versus mean rating of faculty in the Roose-Andersen study--48 programs in zoology.

197 study programs were identified by institution and discipline (but not by department), the matching of results from this survey with those from the committee's survey is not precise. For universities repre- sented in the latter survey by more than one program in a particular discipline, the mean rating for the program with the largest number of graduates (measure 02) is the only one plotted here. Although the re- sults of both surveys are reported on identical scales, some caution must be taken in interpreting differences in mean ratings a program re- ceived in the two evaluations. It is impossible to estimate what ef- fect all of the differences described above may have had on the results of the two surveys. Furthermore, one must remember that the reported scores are based on the opinions of different groups of faculty members and were provided at different time periods. In 1969, when the Roose- Andersen survey was conducted, graduate departments in most universi- ties were still expanding and not facing the enrollment and budget re- ductions that many departments have had to deal with in recent years. Consequently, a comparison of the overall findings from the two surveys tells us nothing about how much graduate education has improved (or de- clined) in the past decade. Nor should the reader place much stock in any small differences in the mean ratings that a particular program may have received in the two surveys. On the other hand, it is of particu- lar interest to note the high correlations between the results of the evaluations. For programs in biochemistry, cellular/molecular biology, and microbiology, the correlation coefficients range between .87 and .92; in botany, physiology, and zoology, the coefficients range between .78 and .83. The high correlations found here may suggest to some readers that reputational standings of programs in these disciplines have changed very little in the last decade. However, differences are apparent for some institutions. Also, one must keep in mind that the correlations are based on the reputational ratings of only approxi- mately two-thirds of the programs evaluated in this assessment in these disciplines and do not take into account the emergence of many new pro- grams that did not exist or were too small to be rated in the Roose- Andersen study. FUTURE STUDIES One of the most important objectives in undertaking this assessment was to test new measures not used extensively in past evaluations of graduate programs. Although the committee believes that it has been successful in this effort, much more needs to be done. First and fore- most, studies of this kind should be extended to cover other types of programs and other disciplines not included in this effort. AS a con- sequence of budgeting limitations, the committee had to restrict its study to 32 disciplines, selected on the basis of the number of doctor- ates awarded in each. Among those omitted were programs in develop- mental biology, entomology, pharmacology, and population biology, all of which were included in the Roose-Andersen study; a multidimensional assessment of research-doctorate programs in these and many other im- portant disciplines would be of value. Consideration should also be given to embarking on evaluations of programs offering other types of

198 graduate and professional degrees. AS a matter of fact, plans for in- cluding master 's-degree programs in this assessment were originally contemplated, but because of a lack of available information about the resources and graduates of programs at the master's level, it was de- cided to focus on programs leading to the research doctorate. Perhaps the most debated issue the committee has had to address concerned which measures should be reported in this assessment. In fact, there is still disagreement among some of its members about the relative merits of certain measures, and the committee fully recognizes a need for more reliable and valid indices of the quality of graduate programs. First on a list of needs is more precise and meaningful in- formation about the product of research-doctorate programs--the gradu- ates. For example, what fraction of the program graduates have gone on to be productive investigators--either in the academic setting or in government and industrial laboratories? What fraction have gone on to become outstanding investigators--as measured by receipt of major prizes, membership in academies, and other such distinctions? How do program graduates compare with regard to their publication records? Also desired might be measures of the quality of the students applying for admittance to a graduate program (e.g., Graduate Record Examination scores, undergraduate grade point averages). If reliable data of this sort were made available, they might provide a useful index of the reputational standings of programs, from the perspective of graduate students. A number of alternative measures relevant to the quality of program faculty were considered by the committee but not included in the as- sessment because of the associated difficulties and costs of compiling the necessary data. For example, what fraction of the program faculty were invited to present papers at national meetings? What fraction had been elected to prestigious organizations/groups in their field? What fraction had received senior fellowships and other awards of distinc- tion? In addition, it would be highly desirable to supplement the data presented on NSF, NIH, and ADAMHA research grant awards (measure 13) with data on awards from other federal agencies (e.g., Department of Agriculture, Department of Defense, Department of Energy, National Aeronautics and Space Administration) as well as from major private foundations. As described in the preceding pages, the committee was able to make several changes in the survey design and procedures, but further im- provements could be made. Of highest priority in this regard is the expansion of the survey sample to include evaluators from outside the academic setting (in particular, those in government and industrial laboratories who regularly employ graduates of the programs to be evaluated). To add evaluators from these sectors would require a major effort in identifying the survey population from which a sample could be selected. Although such an effort is likely to involve considerable costs in both time and financial resources, the committee believes that the addition of evaluators from the government and industrial settings would be of value in providing a different perspective to the reputa- tional assessment and that comparisons between the ratings supplied by academic and nonacademic evaluators would be of particular interest.

Next: Minority Statement »
An Assessment of Research-Doctorate Programs in the United States: Biological Sciences Get This Book
×
Buy Paperback | $60.00
MyNAP members save 10% online.
Login or Register to save!
Download Free PDF
  1. ×

    Welcome to OpenBook!

    You're looking at OpenBook, NAP.edu's online reading room since 1999. Based on feedback from you, our users, we've made some improvements that make it easier than ever to read thousands of publications on our website.

    Do you want to take a quick tour of the OpenBook's features?

    No Thanks Take a Tour »
  2. ×

    Show this book's table of contents, where you can jump to any chapter by name.

    « Back Next »
  3. ×

    ...or use these buttons to go back to the previous chapter or skip to the next one.

    « Back Next »
  4. ×

    Jump up to the previous page or down to the next one. Also, you can type in a page number and press Enter to go directly to that page in the book.

    « Back Next »
  5. ×

    To search the entire text of this book, type in your search term here and press Enter.

    « Back Next »
  6. ×

    Share a link to this book page on your preferred social network or via email.

    « Back Next »
  7. ×

    View our suggested citation for this chapter.

    « Back Next »
  8. ×

    Ready to take your reading offline? Click here to buy this book in print or download it as a free PDF, if available.

    « Back Next »
Stay Connected!