The National Academies Press

Currently Skimming:

IX. Summary and Discussion
Pages 163-198

The Chapter Skim interface presents what we've algorithmically identified as the most significant single chunk of text within every page in the chapter.
Select key terms on the right to highlight them within pages of the chapter.

From page 163... ... of the reputational survey ratings and an examination of some factors that might possibly have influenced the survey results. This chapter concludes with suggestions for improving studies of this kind -- with particular attention given to the types of measures one would-like to have available for an assessment of reseaxch-doctorate programs. Read the entire page →
From page 164... ... As can be seen, the mean values reported for individual measures vary considerably among disciplines. The pattern of means on each measure is summarized below, but the reader interested in a detailed comparison of the distribution of a measure should refer to the second table in each of the preceding chapters � 2 Program Size (Measures 01-03~. Read the entire page →
From page 165... ... 165 TABLE 9.1 Mean Values for Each Program Measure, by Discipline Bio- Cell./Molec. Micro- Physichemistry Botany Biology biology ology Zoology Program Size 01 19 19 26 16 19 21 02 20 19 23 16 14 26 03 25 20 34 20 16 33 Program Graduates 04 .47 .21 .59 .48 .53 .33 05 6.0 6.5 6.1 6.1 6.2 7.1 06 .81 .67 .80 .80 .80 .65 07 .56 .32 .56 .48 .57 .30 Survey Results 08 - 2.6 3.0 2.9 2.8 3.Q 2.7 09 1.7 1.8 1.9 1.8 1.9 1.7 10 1.2 1.2 1.2 1.2 1.2 1.2 11 .7 .7 .7 .6 .6 .7 University Library 12 .2 .3 .3 .3 .3 .3 Research Support 13 .63 .28 .64 .45 .52 .36 14 8419 8406 10243 8449 8687 8806 Publication Records 15 92 60 133 46 17 16 Total Programs 139 83 89 134 101 70 Read the entire page →
From page 166... ... In all-six disciplines the mean rating of scholarly quality of program faculty (measure 08) is at or slightly below 3.0 {"good") Read the entire page →
From page 167... ... , survey rating of the scholarly quality of program faculty (measure ssee Appendix F for a description of the derivation of this measure. Read the entire page →
From page 168... ... Table 9.3 shows the correlation coefficients for measure 08, the mean rating of the scholarly quality of program faculty, with each of the other variables. The correlations of measure 08 with measures of program size (01, 02, and 03) Read the entire page →
From page 169... ... Micro- Physi chemistry Botany Biology biology ology Zoology Program Size 01 .48 .42 .54 .61 .43 .66 03 .72 .55 .83 .80 .54 .78 Program Graduates 04 .47 -.14 .16 .18 -.09 .27 05 .16 .04 -.11 .02 .19 .2S 06 .23 -.06 .12 .07 .14 .19 07 .27 -.07 -.02 .12 .15 .44 Survey Results 08 .63 .29 .42 .48 .46 .66 09 .67 .34 .47 .55 .S1 .68 10 .14 .11 .09 .19 .19 .09 11 .63 .27 .40 .58 .32 .62 University Library 12 .46 -.01 .19 .27 .2S .60 Research Support 13 .35 .03 .18 .20 .04 .50 14 .46 .09 .02 .18 .30 .42 Publication Records 15 .66 .23 .32 .44 .16 .56 16 .65 .24 .34 .29 .17 .59 Read the entire page →
From page 170... ... Micro Biology biology Pbysi ology Zoology Program Size 01 .58 .56 .39 .50 .64 .53 02 .63 .29 .42 .48 .46 .66 03 .60 .51 .43 .42 .42 .52 Program Graduates 04 .70 .31 .58 .57 .47 .58 05 .15 .26 .38 .08 -.11 .39 06 .24 .36 .33 .23 .27 .25 07 .35 .59 .41 .38 .30 .63 Survey Results 09 .96 .97 .96 .96 .95 .98 10 .21 .29 .33 .46 .38 .19 11 .96 .83 .94 .91 .89 .95 University Library 12 .63 .66 .47 .54 .49 .78 Research Support 13 .62 .49 .58 .64 .57 .72 14 .69 .62 .57 .68 .51 .65 Publication Records 15 .83 .60 .69 .72 .69 .59 16 .83 .62 .71 .75 .71 .64 Read the entire page →
From page 171... ... Correlations of ratings of faculty quality with measure 06, the fraction of program graduates with definite employment plans, range between .23 and .36 in the six biological disciplines. In every discipline the correlation of measure 08 is higher with measure 07, the fraction of graduates having agreed to employment at a Ph.D.-granting institution. Read the entire page →
From page 172... ... The pattern of relations is quite similar for programs in all six disciplines: moderately high correlations with reputational survey results {except measure 10) , university library size, and publication measures. Read the entire page →
From page 173... ... Micro- Physi chemistry Botany Biology biology ology Zoology Program Size 01 .33 .31 .16 .24 .36 .20 02 .46 .09 .02 .18 .30 .42 03 .27 .17 .02 .08 .37 .28 Program Graduates 04 .50 .20 .30 .43 .31 .21 05 .03 .12 .25 .05 -.16 .08 06 .12 .04 .18 .12 .14 -.04 07 .29 .28 .20 .27 .23 .19 Survey Results 08 .69 .62 .57 .68 .51 .65 09 .64 .61 .47 .65 .51 .63 10 .03 -.07 -.16 .12 -.01 -.16 11 .69 .48 .58 .50 .46 .62 University Library 12 .56 .SS .42 .48 .S0 .63 Research Support 13 .30 .36 .24 .29 .28 .37 Publication Records 15 .71 .47 .58 .73 .43 .43 16 .71 .52 .59 .78 .48 .46 Read the entire page →
From page 174... ... Micro chemistry Botany Biology biology Physi ology Zoology Program Size 01 .58 .55 .37 .38 .52 .36 02 .65 .24 .34 .29 .17 .59 03 .55 .32 .35 .18 .14 .34 Program Graduates 04 .62 -.11 .42 .51 .50 .16 05 .12 .16 .29 -.03 -.12 .18 06 .22 .18 .26 .13 .29 .11 07 .34 .26 .32 .27 .33 .41 Survey Results 08 .83 .62 .71 .75 .71 .64 09 .79 .63 .61 .68 .65 .66 10 .28 .24 .17 .27 .20 .24 11 .85 .56 .74 .62 .81 .68 University Library 12 .56 .37 .59 .54 .37 .47 Research Support 13 .47 .04 .4u .40 .~i .46 14 .71 .52 .59 .78 .48 .46 Publication Records 15 .Yd .96 .~Y .90 .Y/ .94 Read the entire page →
From page 175... ... Probably the most important change was the inclusion of lists of names and ranks of individual faculty members involved in the research-doctorate programs to be evaluated~on the survey form, together with the number of doctoral degrees awarded in the previous five years. Ninety percent of the evaluators were sent forms with faculty names and numbers of degrees awarded; the remaining 10 percent were given forms without this information, so that an analysis could be made of the effect of this modification on survey results. Read the entire page →
From page 176... ... biology ology Zoology 08 SCHOLARLY QUALITY OF PROGRAM FACULTY Distinguished 6.3 7.0 5.9 8.1 5.3 6.0 5.4 Strong 14.5 12.7 16.1 16.9 14.1 14.4 14.5 Good 16.9 17.0 16.7 17.8 17.0 14.7 18.5 Adequate 10.3 12.0 8.8 8.6 10.5 8.1 12.8 Marginal 3.9 5.5 2.8 3.1 3.5 2.9 4.9 Not Sufficient for Doctoral Education 1.1 1.4 1.1 1.0 1.0 .8 1.3 Don't Know Well Enough to Evaluate 47.0 44.5 48.5 44.4 48.5 53.3 42.6 TOTAL 100.0 100.0 100.Q 100.0 100.0 100.0 100.0 O9 EFFECTIVENESS OF PROGRAM IN EDUCATING SCIENTISTS Extremely Effective 7.2 7.2 7.6 8.1 6.8 6.7 6.9 Reasonably Effective 24.6 22.3 27.7 22.4 25.4 22.0 28.8 Minimally Effective 8.4 8.6 8.1 6.7 8.4 7.4 ll.o Not Effective 1.4 1.6 1.1 1.3 1.2 1.1 2.4 Don't Know Well Enough to Evaluate 58.5 60.3 55.5 61.5 58.2 62.7 51.0 TOTAL 100.0 100.0 100.0 100.0 100.0 100.0 100.0 10 CHANGE IN PROGRAM QUALITY IN LAST FIVE YEARS Better 9.8 8.5 10.2 11.1 9.9 8.4 11.5 Little or No Change 19.6 19.3 22.6 18.3 18.4 17.6 22.1 Poorer 3.8 3.6 3.8 3.8 3.6 3.7 4.5 Don't Know Well Enough to Evaluate 66.9 68.6 63.4 66.7 68.1 70.3 61.9 TOTAL 100.0 100.0 100.0 100.0 100.0 100.0 100.0 11 FAMILIARITY WITH WORK OF PROGRAM FACULTY Considerable 15.2 13.8 17.4 18.4 15.1 12.1 15.0 Some 34.3 36.3 31.5 34.0 33.5 32.6 37.5 Little or None 47.8 46.2 48.3 44.3 50.3 50.9 45.7 No Response 2.8 3.6 2.8 3.3 1.1 4.4 1.8 TOTAL 100.0 100.0 100.0 100.0 100.0 100.0 100.0 NOTE: For survey measures 08, 09, 10 the "don't know" category includes a small number of cases for which the respondents provided no response to the survey item. Read the entire page →
From page 177... ... As shown in Table 9.7, survey respondents in each discipline were much more likely to furnish evaluations for programs with high reputational standing than they were for programs of lesser distinction. For example, for biological science programs that received mean ratings of 4.0 or higher on measure 08, almost 85 percent of the evaluations requested on measure 08 were provided; 72 and 57 percent, respectively, were provided on measures 09 and 10. Read the entire page →
From page 178... ... b iology ology Zoology 08 SCHOLARLY QUALITY OF PROGRAM FACULTY Mean Rating on Measure 08 4.0 or Higher 84.7 95.9 67.7 89.9 82.0 81.6 91.1 3.0 - 3.9 65.9 76.0 57.9 72.6 64.2 57.0 75.4 2.0 - 2.9 44.8 48.8 43.9 43.8 44.1 36.3 50.1 Less than 2.0 29.9 32.2 34.2 23.4 27.6 21.8 35.3 O9 EFFECTIVENESS OF PROGRAM IN EDUCATING SCIENTISTS Mean Rating on Measure 08 4.0 or Higher 72.2 78.3 61.5 72.3 69.9 70.9 83.4 3.0 - 3.9 52.3 56.1 51.3 49.7 52.5 45.1 65.2 2.0 - 2.9 33.S 33.5 36.5 27.1 35.1 28.1 41.6 Less than 2.0 21.6 19.8 26.7 15.6 21.9 18.1 29.1 10 CHANGE IN PROGRAM QUALITY IN LAST FIVE YEARS Mean Rating on Measure 08 4.0 or Higher S7.3 62.0 51.4 60.5 51.3 56.0 64.4 3.0 - 3.9 42.5 45.3 43.2 44.1 39.9 36.1 51.7 2.0 - 2.9 27.1 26.8 29.2 24.6 27.3 22.6 32.2 Less than 2.0 15.9 14.0 20.0 11.7 16.5 13.4 21.6 give an indication of the overall reliability of the survey results in-each discipline and for each measure. In the derivation of these correlations individual ratings of each program were randomly divided into two groups (A and B) Read the entire page →
From page 179... ... Biol. 2.86 2.90 1.02 .97 88 .96 Microbiology 2.83 2.81 .79 .82 134 .93 Physiology 2.93 2.94 .83 .77 101 .91 Zoology 2.71 2.74 .84 .78 70 .95 MEASURE 09: EFFECTIVENESS OF PROGRAM IN EDUCATING SCHOLARS Discipline Mean Rating Std. Read the entire page →
From page 180... ... It is evident from the data reported in Table 9.10 that mean ratings of the scholarly quality of program faculty tend to be higher if the evaluator has considerable familiarity with the program. There is nothing surprising or, for that matter, disconcerting about such an association. Read the entire page →
From page 181... ... 181 TABLE 9.9 Comparison of Mean Ratings for 11 Mathematics Programs Included in Two Separate Survey Administrations Survey All Evaluators Measure First N X Second N X Evaluators Rating the Same Program in Both Surveys First N X Second N X Program A 08 100 4.9 114 4.9 50 4.9 50 4.9 09 90 2.7 100 2.8 42 2.7 43 2.7 10 74 1.2 83 1.2 38 1.1 34 1.2 11 100 1.6 llS 1.6 50 1.5 50 1.6 Program B 08 94 4.6 115 4.6 48 4.6 50 4.5 09 81 2.6 91 2.5 40 2.6 39 2.5 10 69 1.0 82- 1.0 37 1.0 36 0.9 11 98 1.4 116 1.4 50 1.5 50 1.5 Program C 08 86 3.4 103 3.6 42 3.4 44 3.5 09 56 2.0 66 2.1 28 2.1 29 2.0 10 55 1.1 62 1.3 30 1.2 27 1.4 11 99 1.0 116 1.1 50 1.1 50 1.0 Program D 08 74 3.0 93 3.0 37 2.8 38 2.9 09 50 1.8 48 1.6 27 1.7 16 1.6 10 46 1.4 52 1.5 24 1.4 23 1.5 11 90 1.0 113 0.9 46 1.0 46 0.9 Program E 08 69 3.0 95 3.1 39 3.0 46 3.1 09 40 1.8 60 1.9 25 1.8 30 1.8 10 36 0.8 58 0.9 24 0.8 29 0.9 11 96 0.8 115 0.9 52 0.9 52 1.0 Program F 08 63 2.9 90 3.D Z6 3.O 3-c 3.l 09 35 1.8 46 1.7 10 1.6 13 1.8 10 32 1.1 43 1.1 11 1.3 12 1.2 11 95 0.7 115 0.8 43 0.7 44 0.7 Program ~ 08 bY -c.~ Y-c -c.~ dY c.-/ dY d.U 09 35 1.7 45 1.6 17 1.7 19 1.7 10 36 1.1 43 1.2 17 1.1 19 1.2 11 85 0.9 116 0.8 46 0.9 46 0.9 Program ~ 08 ~u c.c In c.~ ~o c.c ~' c 09 32 1.3 43 1.3 22 1.2 19 1.3 10 30 1.5 39 1.5 20 1.7 17 1.4 11 90 0.7 116 0.6 51 0.7 52 0.6 Program I 08 55 2.0 74 1.9 30 1.9 30 2.0 09 33 1.0 41 0.9 19 1.0 18 0.8 10 27 1.2 31 1.1 15 1.1 13 1.2 11 99 0.5 115 0.5 50 0.5 50 0.5 rLv~,".l. Read the entire page →
From page 182... ... Quite naturally, the answer depends in some degree on the nature of the restriction imposed. For example, if we exclude evaluations provided by those who confessed "little or no" familiarity with particular programs, then the revised mean ratings would be correlated at a level of at least .99 with the mean ratings computed using all of the data. Read the entire page →
From page 183... ... biology ology Zoology EVALUATOR'S FAMILIARITY WITH PROGRAM Considerable 99.9 99.9 99.5 100.0 100.0 100.0 100.0 Some 95.3 95.3 94.8 94.4 95.5 95.1 96.7 Little or None 10.3 14.3 8.5 10.7 8.6 6.5 12.8 TYPE OF SURVEY FORM Names 53.1 56.0 51.4 56.1 51.8 47.0 56.1 No Names 52.2 52.0 52.0 48.0 48.1 43.2 68.2 INSTITUTION OF HIGHEST DEGREE Alumni 91.6 96.0 84.0 94.1 91.4 95.1 96.4 Nonalumni 52.6 55.3 50.9 55.2 51.2 46.5 57.0 EVALUATOR'S PROXIMITY TO PROGRAM Same Region 64.4 Outside Region 51.4 66.1 63.3 64.1 64.0 56.7 7S.3 54.1 50.0 54.3 49.6 45.3 S5.3 NOTE: The item response rate is the percentage of the total ratings requested from survey participants that included a response other than "don't know. n of the sample members, randomly selected, were given forms without faculty names or doctoral data, as a "control group." Although one might expect that those given faculty names would have been more likely than other survey respondents to provide evaluations of the scholarly quality of program faculty, no appreciable differences were found {see Table 9.11) Read the entire page →
From page 184... ... However interesting these effects may be, one should not lose sight of the fact that they are small at best and that their existence does not necessarily imply that a program's relative standing on measure 08 would differ much whichever type of survey form was used. Since only about 1 in 10 ratings was supplied without the benefit of faculty names, it is hard to establish any very stable picture of relative mean ratings of individual programs. Read the entire page →
From page 185... ... Although the data in Table 9.13 show that "nearby" programs were given higher ratings than those outside the evaluator's region in four of the disciplines, the differences in reported means are quite small and probably represent no more than a secondary effect that might be expected, because, as we have already seen, evaluators tended to rate higher those programs with which they were more familiar. Furthermore, the reasonably high correlations found between the mean ratings of the two groups indicate that the relative standings of programs are not dramatically influenced by the geographic proximity of those evaluating them. Read the entire page →
From page 186... ... To be certain that this was the case, mean ratings of the scholarly quality of faculty were recalculated for every biological science program-with the evaluations provided by alumni excluded. The results were compared with the mean scores based on a full set of evaluations. Read the entire page →
From page 187... ... . The results of the analysis, which are presented in the mathematical and physical sciences volume of the committee's report, indicate that there is a high degree of correlation in the mean ratings provided by those in differing specialty fields within these two disciplines. Read the entire page →
From page 188... ... his generality has self-evident appeal. On the other hand, it is wise to keep in mind that these reputational ratings are measures of perceived program quality rather than of "quality" in some ideal or absolute sense. Read the entire page →
From page 189... ... Although there are obvious similarities in the two surveys, there are also some important differences that should be kept in mind in examining individual program ratings of the scholarly quality of facultye Already mentioned in this chapter is the inclusion, on the form sent to 90 percent of the sample members in the committee's survey, of the names and academic ranks of faculty and the numbers of doctoral graduates in the previous five years. Other significant changes in the committee's form are the identification of the university department or academic unit in which each program may be found, the restriction of requesting evaluators to Read the entire page →
From page 190... ... Although the total numbers of programs included in the studies are nearly equal, there are many differences in the program coverage in each discipline. Figures 9.1-9.6 plot the mean ratings of scholarly quality of faculty in programs included in both surveys; sets of ratings are graphed for 90 programs in biochemistry, 52 programs in botany, 53 programs in cellular/molecular biology, 82 programs in microbiology, 63 programs in physiology, and 48 programs in zoology. Read the entire page →
From page 191... ... FIGURE 9.1 Mean rating of scholarly quality of faculty (measure 08) versus mean rating of faculty in the Roose-Andersen study -- 90 programs in biochemistry. Read the entire page →
From page 192... ... * + + + + + + + + t + + + + + + + + + + + + + + 4.0 5.0 FIGURE 9.2 Mean rating of scholarly quality of faculty (measure 08) Read the entire page →
From page 193... ... FIGURE 9.3 Mean rating of scholarly quality of faculty (measure 08) versus mean rating of faculty in the Roose-Andersen study -- 53 programs in cellular/molecular biology. Read the entire page →
From page 194... ... FIGURE 9.4 Mean rating of scholarly quality of faculty (measure 08) versus mean rating of faculty in the Roose-Andersen study -- 82 programs in microbiology. Read the entire page →
From page 195... ... FIGURE 9.5 Mean rating of scholarly quality of faculty (measure 08) versus mean rating of faculty in the Roose-Andersen study -- 63 programs in physiology. Read the entire page →
From page 196... ... FIGURE 9.6 Mean rating of scholarly quality of faculty (measure 08) versus mean rating of faculty in the Roose-Andersen study -- 48 programs in zoology. Read the entire page →
From page 197... ... For universities represented in the latter survey by more than one program in a particular discipline, the mean rating for the program with the largest number of graduates (measure 02) is the only one plotted here. Read the entire page →
From page 198... ... What fraction have gone on to become outstanding investigators -- as measured by receipt of major prizes, membership in academies, and other such distinctions? How do program graduates compare with regard to their publication records? Read the entire page →

From page 163...

... of the reputational survey ratings and an examination of some factors that might possibly have influenced the survey results. This chapter concludes with suggestions for improving studies of this kind -- with particular attention given to the types of measures one would-like to have available for an assessment of reseaxch-doctorate programs.

IX. Summary and Discussion Pages 163-198

IX. Summary and Discussion
Pages 163-198