National Academies Press: OpenBook

Assessing Research-Doctorate Programs: A Methodology Study (2003)

Chapter: 4. Quantitative Measures

« Previous: 3. Taxonomy
Suggested Citation:"4. Quantitative Measures." National Research Council. 2003. Assessing Research-Doctorate Programs: A Methodology Study. Washington, DC: The National Academies Press. doi: 10.17226/10859.
×
Page 25
Suggested Citation:"4. Quantitative Measures." National Research Council. 2003. Assessing Research-Doctorate Programs: A Methodology Study. Washington, DC: The National Academies Press. doi: 10.17226/10859.
×
Page 26
Suggested Citation:"4. Quantitative Measures." National Research Council. 2003. Assessing Research-Doctorate Programs: A Methodology Study. Washington, DC: The National Academies Press. doi: 10.17226/10859.
×
Page 27
Suggested Citation:"4. Quantitative Measures." National Research Council. 2003. Assessing Research-Doctorate Programs: A Methodology Study. Washington, DC: The National Academies Press. doi: 10.17226/10859.
×
Page 28
Suggested Citation:"4. Quantitative Measures." National Research Council. 2003. Assessing Research-Doctorate Programs: A Methodology Study. Washington, DC: The National Academies Press. doi: 10.17226/10859.
×
Page 29
Suggested Citation:"4. Quantitative Measures." National Research Council. 2003. Assessing Research-Doctorate Programs: A Methodology Study. Washington, DC: The National Academies Press. doi: 10.17226/10859.
×
Page 30

Below is the uncorrected machine-read text of this chapter, intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text of each book. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.

4 Quantitative Measures This chapter proposes and describes the quantitative measures relevant to the assessment of research-doctorate programs. These measures are valuable because they · Permit comparisons across programs, · Allow analyses of the correlates of the qualitative reputational measure, · Provide potential students with a variety of dimensions along which to compare program characteristics, and · Are easily updateable so that, even if assessing reputation is an expensive and time-intensive process, updated quanti- tative measures will allow current comparisons of programs. Of course, quantitative measures can be subject to distor- tion just as reputational measures can be. An example would be a high citation count generated by a faulty result, but these distortions are different from and may be more easily iden- tified and corrected than those involving reputational measures. Each quantitative measure reflects a dimension of the quality of a program, while reputational measures are more holistic and reflect the weighting of a variety of factors depending on rater preferences. The Panel on Quantitative Measures recommended to the Committee several new data-collection approaches to address concerns about the 1995 Study. Evidence from individuals and organizations that corresponded with the Committee and the reactions to the previous study both show that the proposed study needs to provide information to potential students concerning the credentials required for admission to programs and the context within which gradu- ate education occurs at each institution. It is important to present evidence on educational conditions for students as well as data on faculty quality. Data on post-Ph.D. plans are collected by the National Science Foundation and, although inadequate for those biological sciences in which post- doctoral study is expected to follow the receipt of a degree, they do differentiate among programs in other fields and 25 should be reported in this context. It is also important to collect data to provide a quantitative basis for the assessment of scholarly work in the graduate programs. With these purposes in mind, the Panel focused on quan- titative data that could be obtained from four different groups of respondents in universities that are involved in doctoral education: University-wide. These data reflect resources avail- able to, and characteristics of, doctoral education at the university level. Examples include: library resources, health care, child care, on-campus housing, laboratory space (by program), and interdisciplinary centers. Program-specific. These data describe the characteris- tics of program faculty and students. Examples include: characteristics of students offered admission, informa- tion on program selectivity, support available to students, completion rates, time to degree, and demo- graphic characteristics of faculty. Faculty-related. These data cover the disciplinary sub- field, doctoral program connections, Ph.D. institution, and prior employment for each faculty member as well as tenure status and rank. Currently enrolled students. These data cover pro- fessional development, career plans and guidance, research productivity, research infrastructure, and demographic characteristics for students who have been admitted to candidacy in selected fields. In addition to these data, which would be collected through surveys, data on research funding, citations, publi- cations, and awards would be gathered from awarding agencies and the Institute for Scientific Information (ISI), as was done in the 1995 Study.

26 ASSESSING RESEARCH-DOCTORATE PROGRAMS The mechanics of collecting these data have been greatly lute for Scientific Information (ISI) and the NSF Doctorate simplified since 1993 by the development of questionnaires Records File (DRF). For the humanities, it gathered data on and datasets that can be made available on the Web as well honors and awards from the granting organizations. Most of as software that permits easy analysis of large datasets. This the institutional coordinators did a conscientious and technology makes it possible to expand the pool of potential thorough job, but the Committee believes that it would be raters of doctoral programs. _ helpful to pursue a more complex data-collection strategy that would include a program data collector (usually the director of graduate studies) in addition to the key institu- tional coordinator, a questionnaire to faculty, and question- naires to students in selected programs. This approach was tested with the help of the pilot institutions. The institutional coordinator sent the NRC e-mail addresses of respondents for each program. The NRC then provided the respondent a password and the Web address of the program questionnaire. A similar procedure was followed for faculty whose names were provided by the program respondents. Copies of the questionnaires may be found in Appendix D. In 1995, programs were asked for the number of faculty engaged in doctoral education and the percentage of faculty who were full professors. They were also asked for the numbers of Ph.D.s granted in the previous 3 years, their graduate enrollment both full-time and part-time, and the percentage of females in their total enrollment. Data on doctoral recipients, such as time to degree and demographic characteristics, came entirely from the DRF and represented only those who had completed their degrees. The Committee believed that more informative data could be collected directly from the program respondents. Follow- ing the 1995 Study, a number of questions had been raised about the DRF data on time to degree. More generally, the Committee observed that data on graduates alone gave a possibly biased picture of the composition and funding of students enrolled in the program. The program question- naire contains questions that are directly relevant to these concerns. In the area of faculty characteristics, the program ques- tionnaire requests the name, e-mail address, rank, tenure status, and demographic characteristics (gender, race/ ethnicity, and citizenship status) of each faculty member associated with the program. Student data requested include characteristics of students offered admission, information on program selectivity, support available to students, comple- tion rates, and time to degree. It also asks whether the program requires a master's degree prior to admission to the doctoral program, since this is a crucial consideration affect- ing the measurement of time to degree. The questionnaire also permits construction of a detailed profile of the percent- age of students receiving financial aid and the nature of that aid. Finally, the questionnaire asks a variety of questions related to program support of doctoral education: whether student teaching is mentored, whether students are provided with their own workspaces, whether professional develop- ment is encouraged through travel grants, and whether excellence in the mentoring of graduate students by faculty is rewarded. These are all "yes/no" questions that impose little respondent burden. MEASURABLE CHARACTERISTICS OF DOCTORAL PROGRAMS The 1995 Study presented data on 17 characteristics of doctoral programs and their students beyond reputational measures. These are shown in Table 4-1. Although these measures are interesting and useful, it is now possible to gather data that will paint a far more nuanced picture of doc- toral programs. Indicators of what data would be especially useful have been pointed out in a number of recent discus- sions and surveys of doctoral education. Institutional Variables In the 1995 Study, data were presented on size, type of control, level of research and development funding, size of the graduate school, and library characteristics (total volumes and serials). These variables paint a general picture of the environment in which a doctoral program exists. Does it reside in a big research university? Does the graduate school loom large in its overall educational mission? The Com- mittee added to these measures that were specifically related to doctoral education. Does the institution contribute to health care for doctoral students and their families? Does it provide graduate student housing? Are day care facilities provided on campus? All these variables are relevant to the quality of life of the doctoral student, who is often married and subsisting on a limited stipend. The Committee took an especially hard look at the quan- titative measures of library resources. The number of books and serials is not an adequate measure in the electronic age. Many universities participate in library consortia and digital material is a growing portion of their acquisitions. The Com- mittee revised the library measures by asking for budget data on print serials, electronic serials, and other electronic media as well as for the size of library staff. An addition to the institutional data collection effort is the question about laboratory space. Although this is a pro- gram characteristic, information about laboratory space is provided to the National Science Foundation and to govern- ment auditors at the institutional level. This is a measure of considerable interest for the laboratory sciences and engi- neering, and the Committee agreed that it should be collected as a possible correlate of quality. Program Characteristics The 1995 Study included data about faculty, students, and graduates gathered through institutional coordinators, Insti-

QUANTITATIVE MEASURES TABLE 4-1 Data Recommended for Inclusion in the Next Assessment of Research-Doctorate Programs. Bolded Elements Were Not Collected for the 1995 Study. Institutional Characteristics 27 Variable Description Year of First Ph.D. The year in which the Doctorate Records File (DRF) first recorded a Ph.D. Since the DRF information dates back only to 1920, institutions awarding Ph.D.s prior to 1920 were identified by other sources, such as university catalogs or direct inquiries to the institutions. Because of historic limitations to this fife, this variable should be considered a general indicator not an institutional record. Control Type of "Institutional Control": PR=private institution; PU=public institution. Enrollment Total Totalfull- and part-time students enrolled in Fall 2003 in courses creditable toward a diploma. Graduate Full- and part-time students in Fall 2003 in nonprofessional programs seeking a graduate degree. Total R&D Average annual expenditure for research and development at the institution for the previous 5 years in constant dollars. Federal R&D Average annual federal expenditure for research and development at the institution for the previous 5 years in constant dollars. Professional Library Staff Number of library staff (FTE). Total Library Expenditures Total library expenditure offends from regular institutional budgets and other sources, such as research grants, special projects, gifts, endowments, and fees for services for the previous academic year. Library Expenditures: Total library expenditure offends for book acquisition from regular institutional budgets and other sources, such as Acquisition of Books research grants, special projects, gifts, endowments, and fees for services for the previous academic year. Library Expenditures: Total library expenditure offends for print serials from regular institutional budgets and other sources, such as research Print Serials grants, special projects, gifts, endowments, andfees for services for the previous academic year. Library Expenditures: Total library expenditure offends for serials in electronic media from regular institutional budgets and other sources, Electronic Serials such as research grants, special projects, gifts, endowments, andfees for services for the previous academic year. Library Expenditures: Total library expenditure offends for microprint and electronic databases from regular institutional budgets and other Microprint and Electronic sources, such as research grants, special projects, gifts, endowments, and fees for services for the previous academic Databases year. Health Care Insurance Whether health care insurance is available to enrolled doctoral students under an institutional plan. Whether andfor whom (TAB, RAs, all) percentage of premium cost is covered. Childcare Facilities Available to graduate students? Subsidized? Listings made available? University-Subsidized Available to doctoral students? Student Housing University Awards/ Teaching or research by doctoral students? Mentoring of doctoral students by faculty? Recognition University-Level Support Available for travel to professional meetings? For research off-campus? Available to help students improve their for Doctoral Students teaching skills? Placement assistance? Available for travel to professional meetings? Available to help students improve their teaching skills? Placement assistance? Doctoral Program Characteristics Variable Description Total Students The number offull- and part-time graduate students enrolled in the Fall of the survey year. Student Characteristics Numbers, full-time and part-time status, gender, race/ethnicity, citizenship status. Ph.D. Production Numbers of Ph.D.s awarded in each of the previous 5 years. Program Median Time Year by which half the entering cohort had completed, averaged overfive cohorts. For programs for which half never to Degree complete, the percentage completing within 7 years. Master's Required Whether the program requires completion of a master's degree prior to admission. Financial Support Proportion of first-year students who receive full support. Number of years for which students may expect full financial support (including Fellowships, RAships, and TAships). Whether summer support is available. Percent receiving externally funded support. Percent receiving university-funded support. continues

28 ASSESSING RESEARCH-DOCTORATE PROGRAMS TABLE 4-1 Data Recommended for Inclusion in the Next Assessment of Research-Doctorate Programs. Bold elements were not collected for the 1995 Study. Doctoral Program Charactenstics (continued) Variable Description Teaching Assistant Average number of courses per term that a TA is expected to teach. Work Load Individually Assigned For allstudents? ForTAs? ForRAs? Student Workspace GRE Scores Whether GRE is required for admission. Average and minimum verbal and quantitative scores. Number of Acceptances Number of applicants who are accepted into the program for the past 3 years. Number Who Enroll Number of accepted students who enroll in the program for the past 3 years. Awards for Teaching/ Whether the program gives awards for graduate student teaching/research or for faculty mentoring of doctoral students. Research, Faculty Mentoring Program Support for Whether the program provides some form of travel support for doctoral students to attend professional meetings. Student Travel to Meetings Teaching Skill Support Whether there is an organized program to help doctoral students improve their teaching skills. Laboratory Space For each doctoral program in science and engineering, the number of net available square feet of laboratory space in the previous academic year. Also, an indication of whether this space is shared with other doctoral programs. Related Interdisciplinary Listing of centers in which program doctoral students may carry out research. Centers Data Collection on Student Whether the program collects such data. Whether it makes data available to potential students. Outcomes Competitor Programs List of up to five programs with which the program normally competes for graduate students. Post-doctoral Plans by Obtainedfrom the NSF Survey of Doctoral Recipients, the percentage of Ph.D.s over the previous 5 years by type of Type of Employer employer. Program Faculty Variable Description Total Faculty Total number offaculty participating in the program (teaching or supervising dissertations). Faculty Characteristics Rank, tenure status, gender, race/ethnicity, citizenship status, years since Ph.D., time at this institution. Percent Full Percentage offull professors participating in the program. Percent Support Percentage of program faculty with research support (1999-2003). Percent Engaged in More Percentage of program faculty who teach graduate courses or supervise dissertations in other programs. than One Doctoral Program Number of Awardsa Total number of awards and honors attributed to program faculty for the period 1999-2003. Awarded Facultya Percentage of program faculty that have received at least one honor or award for the period 1999-2003. Percent Publicationsb Percentage of program faculty (totalfaculty) publishing in the period 1999-2003. Publications/Facultyb The ratio of the total number of program publications in the period 1999-2003 to the number of program faculty (total faculty). Citations/Facultyb The ratio of the total number of program citations in the period 1999-2003 to the number of program faculty (total faculty). Faculty Characteristics Listing by name and e-mail address offaculty who are affiliated with the program through either teaching or supervision of dissertations. Rank, tenure status, whether US citizen/permanent resident, gender, race/ethnicity, date and year of highest degree. aFor Arts and Humanities. bFor the fields in Engineering and the Sciences. Arts and Humanities may be included, depending on whether adequate book and monograph citations and publication sources exist.

QUANTITATIVE MEASURES Faculty Characteristics In the 1995 Study, a brief faculty questionnaire was administered to the raters who produced the reputational rankings. These raters were drawn from a sample of faculty nominated by their institutional coordinators. The sample size reflected the number of programs in each field. The brief questionnaire asked raters the year, institution, and date of their highest degree as well as their current field of specialization. The Committee believes that the faculty questionnaire should be modified to collect certain other data. For example, the university origins of current faculty are a direct measure of which graduate programs are training Ph.D.s who become faculty at research universities. Data on date of degree would also permit a comparison of origins of recently hired faculty as compared to faculty hired, for example, more than 20 years ago. Although subfield data were collected for the 1995 Study, they were not used. They could be useful in improving program descriptions for potential graduate students and for assuring that specialist programs are rated by knowledgeable peers in the same specialty. The Committee also believes that additional questions asked of faculty could permit a richer description of interdisciplinarity. For example, faculty could list all pro- grams in which they have participated, either by teaching or serving on dissertation committees. Many faculty would be listed as members of more than one graduate program, and for the purposes of the reputational survey, the Committee recommends that they be listed as program faculty for all programs with which they are associated. To avoid the pos- sibility of double counting the output of productive faculty, objective measures should be attributed pro rata among the various programs in which they are listed. The decision as to how to prorate an effort should be made by the faculty member with guidance that they should try to describe how time devoted to doctoral education (teaching and student mentoring) has been allocated among the programs for the past 3-year period. The Committee was concerned that programs might want to associate a well-known faculty member with as many pro- grams as possible in order to boost its rating, even if he or she were not involved with the program. Allocation of pub- lications should serve to discourage this behavior. Student Characteristics and Views Student observations have not been a part of past assess- ments of research-doctorate programs. Past studies have included data about demographic characteristics and about sources of financial support of Ph.D. recipients drawn from the DRF and about graduate student enrollment collected from the doctoral institutions. Another student measure was "educational effectiveness of the doctoral program," and for reasons discussed in Chapter 6, the Committee is recom- mending the elimination of this measure. The approach for 29 measuring student processes and outcomes is discussed in Chapter 5. PILOT TRIAL FINDINGS The pilot trials were conducted over a 3-month period. The most important finding was that 3 months was barely sufficient for dealing with the study questionnaires. The full study should probably allow at least 4-6 months for data submission. The answers to many of the questions are pre- pared for other data collection efforts, but additional time is needed to customize answers to fit the taxonomy and to permit time for follow-up with nonrespondents. All institutions carried out the trial through a single point of contact for the campus. This single point of contact worked with institutional research offices and program con- tacts to answer questions as well as interacted with NRC staff to assure that data definitions were uniform. Electronic data collection worked well for institutions, programs, and faculty. We learned that it was better not to provide a hard copy alternative (as contrasted to Web response), since hard copy data simply had to be re-entered in databases once it was received by the NRC. All the pilot institutions store and access institutional and program data electronically. E-mail is the standard mode of communica- tion with faculty and the rates of faculty response (60 per- cent) were high for a one-wave administration. The Committee also learned that more precise definitions are needed to guide respondents. For example, when asking for data about "first-year doctoral students" a distinction may be needed about whether the students have a master's in the field. Care needs to be taken not to include terminal master's students, and precise definitions of "full-time" and "part- time" should be included. The Committee learned the following from the question- na~re responses: Institutional Questionnaire. · Library expenditures: Not all institutions separate e-media expenditures from print expenditures. · Space: The questionnaire needs to provide guidance about how to allocate shared space. Answers to space ques- tions also depend on how well the institution's programs fit the taxonomy. If the fit is poor, the allocation of space is arbitrary. · Graduate student awards and support are more appro- priately queried at the program level. Program questionnaire. · Programs had difficulty filling out the inception cohort matrix but believed they could have done it if they had had more lead time. · Programs knew who their competitors were for doc- toral students.

30 · Programs that required GREs knew the averages and minima. For programs that do not require GREs, it would be helpful to ask what percentage of applicants submit GRE scores as well as report averages and minima only for those programs that are above a certain level (e.g., 80 percent). · Requests for faculty lists and faculty data should be separate from requests for other program data. Faculty questionnaire. · E-mail notifications must have a sufficiently informa- tive subject heading so that they are not mistaken for spam. · Questionnaires should contain a due date. · Faculty associated with more than one program should be asked to fill out only one questionnaire. The NRC needs to develop procedures to duplicate information for the other programs with which a faculty member is associated. · Some faculty identified their program by a name other than that of the program that submitted their name. A proce- dure must be developed to resolve this problem. Each pilot institution was asked to provide comments on the questionnaires. These comments, some of which are reported above, will be used as background material for the committee that conducts the full study. Draft questionnaires ASSESSING RESEARCH-DOCTORATE PROGRAMS for the full study should be reviewed by a number of institu- tional researchers from a diverse set of institutions as well as by survey researchers. Data Collectecl from Other Sources The Committee recommends that most of the quantitative data presented in the 1995 Study from other sources be col- lected again. These include: publication and citation data from ISI, data on research grants from government agencies and large private foundations, data on books from the Arts and Humanities Citation Index, and data on awards and honors from a large set of foundations and professional societies. Student data from the Doctorate Record File should be considered for inclusion but checked for inconsis- tencies against institutional and program records. In the case of inconsistencies, a validation process should be designed. RECOMMENDATIONS The Committee recommends that the data listed in Bold type in Table 4-1 be added to the quantitative measures that were collected for the 1995 Study.

Next: 5. Student Education and Outcomes »
Assessing Research-Doctorate Programs: A Methodology Study Get This Book
×
Buy Paperback | $48.00
MyNAP members save 10% online.
Login or Register to save!
Download Free PDF

How should we assess and present information about the quality of research-doctorate programs? In recommending that the 1995 NRC rankings in Assessing the Quality of Research-Doctorate Programs: Continuity and Change be updated as soon as possible, this study presents an improved approach to doctoral program assessment which will be useful to administrators, faculty, and others with an interest in improving the education of Ph.D.s in the United States. It reviews the methodology of the 1995 NRC rankings and recommends changes, including the collection of new data about Ph.D. students, additional data about faculty, and new techniques to present data on the qualitative assessment of doctoral program reputation. It also recommends revision of the taxonomy of fields from that used in the 1995 rankings.

  1. ×

    Welcome to OpenBook!

    You're looking at OpenBook, NAP.edu's online reading room since 1999. Based on feedback from you, our users, we've made some improvements that make it easier than ever to read thousands of publications on our website.

    Do you want to take a quick tour of the OpenBook's features?

    No Thanks Take a Tour »
  2. ×

    Show this book's table of contents, where you can jump to any chapter by name.

    « Back Next »
  3. ×

    ...or use these buttons to go back to the previous chapter or skip to the next one.

    « Back Next »
  4. ×

    Jump up to the previous page or down to the next one. Also, you can type in a page number and press Enter to go directly to that page in the book.

    « Back Next »
  5. ×

    To search the entire text of this book, type in your search term here and press Enter.

    « Back Next »
  6. ×

    Share a link to this book page on your preferred social network or via email.

    « Back Next »
  7. ×

    View our suggested citation for this chapter.

    « Back Next »
  8. ×

    Ready to take your reading offline? Click here to buy this book in print or download it as a free PDF, if available.

    « Back Next »
Stay Connected!