Appendix A
Study Methodology
INTRODUCTION
In carrying out this study, the committee—with staff from the NRC's Office of Scientific and Engineering Personnel (OSEP) and Center for Education —conducted a national survey and a series of telephone interviews.
To obtain data on the willingness of Ph.D.s to consider positions in K-12 science and mathematics education and on the various conditions that might enhance their recruitment, OSEP staff conducted a national survey of graduate students and postdoctorates. To inform development of the survey questionnaire, staff conducted five focus groups on career aspirations and perceptions of K-12 science and mathematics education careers with advanced graduate students and recent Ph.D.s. Led by a professional facilitator, focus groups obtained reactions to hypothetical incentives to attract Ph.D.s to secondary school positions and also elicited additional ideas that were incorporated into the survey questionnaire.
Focus group participants represented the spectrum of fields in the biological sciences, physical sciences, and mathematics. The focus groups were also located geographically diversified, with sessions held at Rutgers University/University of Medicine and Dentistry of New Jersey, Duke University, the University of Texas at Austin, the University of California, San Francisco, and the Fred Hutchison Cancer Research Center in Seattle.
The survey questionnaire reflected the input from these focus groups as well as input from committee members and staff from several NRC units. It assessed four broad areas of interest: (1) demographic characteristics of the respondents; (2) their short-term and long-term career aspirations, including salary expectations; (3) conditions under which the respondent might consider K-12 science and mathematics education as a career; and (4) incentives that might be required for respondents to consider careers in K-12 science and mathematics education.
In addition to this survey, staff conducted a series of telephone interviews with interested individuals in K-12 science and mathematics education during summer 1999. Interviews were conducted with 18 science and mathematics Ph.D.s currently teaching in secondary schools to synthesize their experiences, including barriers encountered, strategies adopted to overcome these barriers, areas of success and achievement, and tips on recruitment and retention of other science Ph.D.s. Interviews were also conducted with high school principals, school district superintendents, and chief state school officers, to obtain information on concerns with
employment of Ph.D.s in secondary school, certification issues, and funding of Ph.D.s in secondary education. In addition, telephone interviews were conducted with graduate deans to ascertain the kinds of programmatic changes required in graduate education to prepare Ph.D.s for careers teaching in secondary schools.
SURVEY OF GRADUATE STUDENTS AND RECENT PH.D.S
The survey was fielded in July and August 1999 to a national sample of 2,000 graduate students and recent Ph.D.s stratified by field of study. This survey gathered data on demographic characteristics, short-term and long-term career aspirations, salary expectations, and conditions under which respondents would consider employment in secondary school education. The NRC received 713 responses to the survey. A copy of the survey instrument is located in Appendix C.
Focus Groups
As a first step in survey development, five focus groups were conducted to gain greater insight to the attitudes of science and mathematics graduate students and postdocs on secondary school education activities. The focus groups were conducted in sites that included both life scientists and physical scientists/mathematicians, graduate students and postdocs, and represented a broad geographical spread. The focus group protocol focused on six broad areas, including:
-
Career aspirations.
-
Consideration of aspects of careers other than in a university.
-
Considerations of careers in secondary education.
-
Perceptions of teacher certification.
-
Perceptions of teacher compensation.
-
Perceptions of work the environment in secondary education.
A copy of the focus group protocol is located in Appendix B. An integrated summary of the five focus groups is located in Appendix C.
Questionnaire Development
Based on input from the focus groups, a questionnaire was designed to ascertain conditions under which graduate students and recent postdocs in science and mathematics would consider employment in secondary school education. In addition to questions on demographic and educational characteristics, the questionnaire proposed a number of scenarios under which the respondents would consider secondary school teaching. Questions were addressed to other aspects of secondary school employment such as curriculum and professional development.
The questionnaire was pretested with a small group of graduate students and recent Ph.D.s in Washington, D.C. and revised again. A copy of the questionnaire is located in Appendix C.
Sampling
The questionnaire was then sent to a random sample of 2,000 science and mathematics graduate students and postdocs. NRC staff developed a stratified random sample of science and mathematics graduate students and postdocs for this survey. The sampling frame was derived from the 1993 edition of Research-Doctorate Programs in the United States: Continuity and Change. The population of sampling units consisted of science and mathematics departments listed in Research-Doctorate Programs—a total of 935 life science and 570 physical science/mathematics departments. The committee had concluded that astrophysics/astronomy, computer science, and statistics/biostatistics departments were not germane to this survey and these departments were excluded from the sampling frame. The sample was stratified into life science and physical science/mathematics categories using proportional allocation with proportionality based on the number of graduate students in physical science/mathematics and life science departments as listed in the 1993 Research-Doctorate Programs appendixes.
A two-stage sampling frame (with replacement) was developed from the listing of Ph.D. granting departments in Research-Doctorate Programs in the United States. In the initial stage, departments were selected using systematic sampling, stratified by discipline. Systematic sampling selects items from an ordered array by selecting a random starting point and selecting items on periodic basis. For example, for the life sciences, we identified a random starting point between 1 and 39 and then selected that and every 39th item in the array. Sampling with replacement provides an exact replacement sample item for each nonresponding department by identifying a specific replacement unit for each sampling unit. This was accomplished by identifying pairs of departments in the systematic sample. The first department was designated as the primary sampling unit, and the nest department on the listing was identified as the replacement-sampling unit.
Field of Study |
Number of Departments |
Life Sciences |
|
Biochemistry and Molecular Biology |
187 |
Cell and Developmental Biology |
165 |
Ecology, Evolution, and Behavior |
127 |
Molecular and General Genetics |
102 |
Neuroscience |
98 |
Pharmacology |
121 |
Physiology |
135 |
Natural Sciences and Mathematics |
|
Chemistry |
168 |
Geosciences |
95 |
Mathematics |
135 |
Oceanography |
26 |
Physics |
146 |
According to Research-Doctorate Programs, there were an estimated 30,500 life science and 37,594 natural science and mathematics graduate students in the identified departments in 1993. Life science departments had an average of 32.6 graduate students while physical science/mathematics departments had an average of 65.9 graduate students. The sampling frame consisted of 47 departments —17 physical science/mathematics and 30 life science departments. Approximately one-fourth of the departments is ranked in each quartile of scholarly quality. However, because higher ranked departments have more graduate students, there was a disproportionately larger number graduate students in the sample from higher ranked departments. For example, among chemistry departments top-ranked departments had an average of 180 students. For chemistry departments ranked in the second, third, and fourth quartiles, the average number of graduate students were 88, 53, and 31, respectively. Approximately 42 percent of graduate students/postdocs in the sample were attending universities ranked in the top quartile. For the second, third, and fourth quartiles, the percentages were 23, 27, and 8 percent, respectively. At the second stage of sampling, all doctoral graduate students and postdocs at a selected department are included in the sample and received a questionnaire.
The chairs of the 47 departments were contacted by a letter that explained the nature of the survey and requested a listing of the names and addresses of all graduate students and postdocs. A total of 38 departments, from either the primary or nonrespondent lists, submitted rosters in time to be included in the survey, resulting in 2,713 graduate students and postdocs identified in the first stage of sampling. Approximately 55 percent of these were in life sciences. Using systematic sampling, questionnaires were sent to 2,000 of the graduate students and postdocs. Depending on the locator data provided by the department, graduate students and postdocs were contacted by either mail (1,144) or email (856). Two departments were unwilling to provide individual student locator data, but agreed to submit questionnaires sent in bulk through campus mail. Consequently, 241 of the graduate students/postdocs contacted by mail
had the packet of questionnaire, cover letter, and return envelope distributed through campus mail rather than by direct mail.
Survey Response
A total of 2,000 science and mathematics graduate students and postdocs were surveyed in three waves on July 15, August 9, and September 2 of 1999. A total of 715 responses were received; 680 responses were received before the cut-off date for data entry. For 169 subjects, no delivery could be made by either mail or email. An additional, nine subjects were out-of-the scope of the survey. The response rate was 39 percent.
There are two possible reasons for the low response rate. First, if subjects were biased against secondary school education activities, they might refuse to respond to the survey. Second, nearly all of the subjects were contacted through their university addresses (by either mail or email) and if these respondents were not located at their university locations during the summer, this could account for the low response rate.
To test whether there was a nonresponse bias, NRC staff conducted a telephone survey with a random sample of nonrespondents. Nonrespondent candidates were selected using systematic sampling from a listing of nonrespondents, a method of selection designed to ensure that the sample was representative of all nonrespondents. A sample of 200 nonrespondents was selected and telephone numbers were obtained for 174 of them.
The focus of the survey was to determine why the subject had not responded to the survey. The telephone survey results are shown below.
Response |
Number |
Already responded |
22 |
Conducted survey on telephone |
1 |
Never received instrument |
10 |
Received, but misplaced – send new survey |
14 |
Too busy to respond |
20 |
Out of the office – did not respond |
12 |
Does not apply |
6 |
Not interested in responding |
3 |
Other response |
4 |
Colleague answered – subject on vacation |
21 |
Left voice mail message |
36 |
No answer |
25 |
Total |
174 |
We believe that the low level of response is result of the timing of the mailout rather that any systematic bias of subjects toward the survey and the survey's content. During two days of
calling, NRC staff were able to make contact with 113 persons. Voice mail messages, urging completion of the questionnaire, were left for 36 persons. There was no answer to repeated calls for 25 persons. Among those with whom NRC staff spoke, nearly half indicated that they had not responded to the questionnaire because they had not been at the office or had been too busy to respond. About 20 percent indicated that they had already responded (one completed the survey over the telephone). Only three persons indicated that they were not interested in K-12 education and had not responded to the survey for that reason. The largest category, 61 persons, represented those with whom NRC staff were unable to establish contact, even after repeated calls. Eventually NRC staff left voice mail messages when possible. However, there was never any contact with 25 persons.
On the basis of the telephone survey, we concluded that the low response rate was probably a consequence of survey administration during the summer when many graduate students are not at their office locations. The telephone survey found that practically none of the persons contacted refused to participate because of a bias towards secondary school teaching.
INTERVIEWS
In addition to fielding the national survey, the committee also collected information for the study through a series of interviews with Ph.D.s already in K-12 education careers and with administrators in K-12 and higher education.
Interviews with Ph.D.s
To complement the survey of graduate students and recent Ph.D.s about their potential interest in K-12 education, we interviewed 18 Ph.D.s already working at the K-12 level to draw insights from their experiences about the opportunities for and obstacles to Ph.D.s taking positions in K-12 education. A copy of the interview protocol is located in Appendix C.
These interviews were conducted by telephone during August 1999. NRC staff compiled a short list of Ph.D.s working in teaching or curriculum development positions and these individuals were contacted for an interview. At the end of each interview, we asked the individual being interviewed if he or she could think of someone else who would be suitable for interviewing. This “snowball” sampling method yielded more than 32 names, of which 20 were available for interviews in the time allotted. Two of the 20 did not hold doctorates, so the total number of Ph.D.s interviewed was 18.
This was not a random sample of persons with science and mathematics Ph.D.s who hold positions in K-12 education. The goal of these telephone interviews was not to obtain scientifically accurate survey data. The goal of these interviews was to provide background for and insights to the data collected through The Survey of Graduate Students and Recent Ph.D.s.
In interviews typically lasting 35 to 50 minutes, we asked these individuals to talk about their experiences in K-12 education. We asked them to comment on what attracted them to secondary school teaching or to curriculum development. We asked them whether colleagues or mentors were supportive when they decided to enter K-12 education, to describe any barriers they encountered in taking K-12 education positions, and how they handled teacher certification. We also asked for the effects of financial or family considerations on their career paths. Individuals interviewed were asked to comment on their current work environment, their relationships with other faculty and administrators, and whether they were actively engaged in research. Finally, we asked them whether they were happy with their choice of a career working in K-12 education, what advice they would give a Ph.D. considering such a career, and what the prospects for such a career would be at this time.
While the interviewer asked these specific questions, the interview was also conducted in such a manner to encourage the interviewee to explore the reasons why they went into teaching and to discuss what were the most significant issues involving that career choice.
Interviews with Administrators
In addition, telephone interviews were conducted with a number of state and local school administrators and higher education administrators, including:
-
Chief state school officers.
-
School district superintendents.
-
High school principals.
-
Principals of science and technology magnet schools.
-
Graduate deans.
Interviews were conducted with high school principals, school district superintendents, and chief state school officers, to obtain information on concerns with employment of Ph.D.s in secondary school, certification issues, and funding of Ph.D.s in secondary education. In addition, telephone interviews were conducted with graduate deans to ascertain the kinds of programmatic changes required in graduate education to prepare Ph.D.s for careers teaching in secondary schools.
A professional skilled in in-depth interviewing conducted all of these interviews. A semi-structured interview schedule was developed that paralleled the interview guide for Ph.D. teachers. A copy of the interview schedule is located in Appendix B.