Final Report to the U.S. Department of Defense on the Defense Reinvestment Initiative (1999)

External Evaluation of the Defense Reinvestment Initiative Project

Final Report

1994 - 1998

Submitted to:

Center for Science, Mathematics, and Engineering Education

National Research Council

Submitted by:

Glenn F. Nyre, Ph.D.

Evaluation Consultant

December 31, 1998

Because I had the privilege of serving as the external evaluator during the entire four years of the Defense Reinvestment Initiative, the evaluation process has benefitted from a continuity of approaches, an understanding of programmatic nuances, and the ability to obtain the trust of the DRI fellows and their colleagues in the schools where they observed, participated in cooperative teaching, student taught, and were eventually full-fledged teachers themselves.

Throughout this time, the evaluation benefitted from the efforts of many others. The thorough and impeccable records maintained by the DRI program office, the willingness of CSULB project director and the mathematics and education professors to discuss issues with me, and the teachers and administrators at the schools who spoke with me during my many site visits to their campuses made my job both easier and enjoyable. In addition, the DRI Advisory Board provided valuable guidance and feedback regarding the evaluation activities and findings, and were responsive to recommendations that were included in each of the four annual evaluation reports. The DRI program director and I maintained in close communication throughout the program. She kept me fully informed of the status of the fellows and any programmatic changes, and could not have been more supportive of and responsive to the evaluation component of the program.

The fellows gave a great deal of their time and effort to the evaluation, enduring multiple interviews, classroom observations, surveys and telephone calls. Viewing me from the beginning, in their engineering language, as “the quality control guy,” their cooperation was invaluable to my work. I am indebted to them, and wish them well in their new careers.

Glenn F. Nyre, Ph. D.

During the four years of the DRI program, the following evaluation activities were undertaken:

• Both formal and informal contact and reporting was maintained with the DRI Program Office on a routine basis throughout the program.

• All program documents were routinely reviewed, from the minutes of organizational meetings and applications for participation in the program through the status of the fellows’ attainment of teaching certification requirements, employment status, and participation in the follow-on support components of the program after the formal thirteen-week training program had been completed.

• Fellows were formally interviewed twice every year.

• Fellows were asked to complete a survey every year.

• Informal contact was maintained with the fellows by means of telephone and e-mail conversations.

• Fellows were observed in classrooms settings at least once every year, including their observation, cooperative teaching and student teaching phases, as well as after they became full-fledged teachers themselves.

• The fellows’ teaching and administrative colleagues were interviewed during each school site visit.

• Both formal and informal contact was maintained with the CSULB project administrator during the initial two years of the program.

• CSULB mathematics and education faculty involved with the program were interviewed during the initial two years of the program.

• Oral evaluation briefings were given at all DRI Advisory Board meetings.

• Four annual written evaluation reports were submitted, as well as this final report.

Twelve of the 15 fellows who began the program completed it (80%) -- an outstanding completion rate when compared against both traditional and alternative teacher preparation programs. Of the non-completers, one had health problems that precluded his continuance; one received several negative reviews from schools to which he was assigned during the classroom observation and cooperative teaching phases of the program, and was asked to leave; and one left voluntarily after experiencing repeated difficulties dealing with what some might call the “second-class status” of being a student again after having held an executive-level position in private industry.

All 12 of the DRI fellows who completed the program were teaching in the semester immediately following program completion. Ten had full-time mathematics or science positions; one was substituting as a mathematics teacher; and one was teaching non-science/mathematics subjects in an adult school program.

The National Center for Educational Statistics reports that 22 percent of teachers leave their jobs within the first three years. This percentage is greater in urban areas, where the attrition rate approaches 50 percent in five years. As they entered their third year of teaching in the fall of 1998, only two of the 12 DRI fellows had left the field -- an 83 percent retention rate. One of those 12 is currently on hiatus from teaching due to a medical emergency, but plans to return when his health improves.

As the fall 1998 semester progressed, a third fellow was experiencing a great deal of stress from the large middle school mathematics classes that he was teaching. He had experienced problems with classroom management throughout the DRI program, and had attempted to deal with this situation by taking two courses and attending workshops on the topic. However, he decided that his health was suffering too much and that he would probably be better suited to teaching at the adult school level. Although the urban schools that were to be the beneficiaries of the program have lost him, the profession has not. I therefore do not consider him to be an unsuccessful participant in the program, but rather, a “diverted success.”

The two fellows who left teaching after having been successfully placed in schools both returned to private industry -- one after not having her teaching contract renewed, and one for personal reasons. Both remain very positive about their DRI experience. The former is very grateful to the DRI program for renewing the confidence and discipline she thought she had lost in the intervening years since her previous college experiences. One of the comments from that fellow was: “I think it takes a special ‘gene’ to be a teacher, and I just didn’t have it and couldn’t develop it.”

That fellow had inadequate support from the principal of the school, her assigned mentor teacher taught English at another school in the district, and the mathematics department held no formal meetings -- basically leaving her “alone” during her first year of teaching. She was employed in a district well beyond the boundaries of the one with which DRI had a formal relationship, and the program had little recourse to intervene on her behalf. The second fellow who left maintains his relationships with several other DRI fellows strongly asserts that he will return to teaching when it again becomes a feasible option for him, given personal considerations.

The fellows were asked to provide their own opinions as to why the DRI completion and retention rates have been so positive. Their most frequent answers were as follows:

• Support from the DRI Program Office;

• Their gradual immersion into teaching through the sequential observation, cooperative teaching, and student teaching phases; and

• Their early and continuous exposure to the types of schools in which they would eventually work, and to the types of students they would be teaching.

It is important to the DRI program that the fellows are aware of, use, and advocate the national standards in mathematics and science. All of them report using the standards, but to somewhat varying degrees depending upon circumstances in their school or district. As one said, “When the department chair hands you a book and says ‘cover these chapters this semester and those chapters next semester,’ you must be creative to make sure you integrate the content and teaching techniques required by the standards.”

Most of the DRI fellows are not only basing the majority of their preparation, delivery and assessment practices on the standards, but many are encouraging other teachers to do so as well. Although some were hired by schools in which new curricula was not in place and new teaching methods were not

evident, as these materials and concepts are increasingly introduced, the fellows are more receptive than many other teachers and champion the cause. Most are supplementing old textbooks by bringing in newer materials on their own, and sharing them with other teachers as well. There is a good flow of pedagogical, content and student exercise and activity information and materials among the fellows themselves.

As reported in the 1995 Annual Evaluation Report, based on interviews and a survey as the fellows had just completed their first summer session and were two months into the fall semester of their thirteen-month formal training program, they “only moderately endorse reform learning, and are not confident that they understand how to carry it out.” That same report found that 40 percent of the fellows “have negative attitudes about the teaching methods endorsed by state and national reform standards,” and also commented on their “cynicism” and “decreasing enthusiasm” for reform methods. To use the vernacular, the fellows “didn’t get it,” didn’t think they ever would, and weren’t sure it was worth “getting.” Major strides were obviously made in this regard in the subsequent two years, as the fellows have now become “champions ” of the cause. This was a result of increased classroom exposure and experience, coupled with the additional learning and information exchanges facilitated by the “Saturday Session” seminars and other follow-on activities provided by the program.

At certain stages of the program, the fellows were asked to rate various components of the DRI program in terms of their perceived value of each. Their responses have been included in previous annual reports, and the most notable finding across the years is that the longer they were in the program, and especially after they became teachers themselves, the less they valued all aspects of the program that were connected to the college -- education classes and student teaching observations and feedback from CSULB education professors. On the other hand, the school-based activities, support from the DRI Program Office and the professional development aspects of the program (i.e., “Saturday Seminars”, professional memberships, attendance at conferences, etc.) were highly valued.

In the fall of 1998, the fellows were asked about their plans to remain in teaching, and their responses were compared to those of a national sample of traditionally credentialed (TC) and alternatively credentialed (AC) teachers. All of the DRI fellows said that they plan to remain in the field for the rest of their career, whereas only 60 percent and 57 percent of the national sample of TC and AC teachers, respectively, gave that response. By contrast, fourteen percent of both comparison groups held out the possibility that they might leave teaching “if something better comes along;” four percent of both the TC and AC teachers said they “intend to leave as soon as I can;” and 22 percent of the TC and 26 percent of the AC teachers in the national sample were undecided about their future in relation to a teaching career.¹

Near the end of the four years, the fellows were asked the following question: “Knowing what you know now, if you could go back to your college days and start over again, would you chose to become a teacher?” All but one said either “absolutely yes” or “yes.” The remaining one selected the option “possibly,” and subsequently opted to teach in an environment other than the middle and high school level. None gave a response of “no” or “absolutely not.”

By the fall of 1998, it was obvious that the fellows, as a group, had “blossomed” as teachers and leaders within their schools. It had become a pleasure to observe them (which was not always the case previously), as they became much more self-assured and in control of their classrooms. They were also using a broader base of materials and student activities in their instruction, and those who have teaching assistants were using them more effectively.

The evaluation site visits became increasingly like professional exchanges, as we talked about things such as pedagogy, new curricular materials they were introducing, additional small group activities they were implementing, and field trips they were arranging. Previously compelling concerns about classroom management, lesson plan preparations, and grading student homework were rarely raised.

The fellows have also gained seniority and are exerting more influence within their departments. As one of them joyously said, “They actually gave me a choice of what courses and grade level I would like to teach this year!” Another is developing his school’s first honors science course. With only a couple of exceptions, the fellows are moving into leadership roles not only within their departments, but within their schools at large. Three are doing volunteer mathematics tutoring, and two are chairing school-wide committees.

Some fellows are showing advanced aspects of leadership – introducing colleagues to new textbooks, materials and instructional strategies; had brought them along to DRI “Saturday Sessions;” and are encouraging them to accompany them to off-site subject matter and pedagogical meetings. Most are contributing personal resources to their classrooms, including manipulatives and software, as well as bringing in their own computers, modems, scanners, televisions, videotape machines, telescopes, etc.

Some fellows are in schools that have excellent support systems for innovative instruction: One is in a school that divides instructional time into three distinct block-class periods per day, with each consisting of almost two hours, thus allowing him more time for hands-on activities; one is teaching in a school in which he spent much of his DRI training time, and, according to his department chair, was viewed as a “fixture ” in the mathematics department for more than a year before being hired; four others are enjoying teaching integrated mathematics classes; one was asked by a special magnet program in his school to develop and teach their specially-focused mathematics course; and another is in a school that provides stipends for participation in four-week summer courses for curriculum and professional development.

Principals, department chairs and fellow teachers have pointed out that new teachers who have been through more traditional preparation programs typically have more problems dealing with the “routine” of schooling, interacting with today’s multicultural and multi-talented students, and becoming active participants in the school and surrounding community.

Because of the opportunities awarded them through the Defense Reinvestment Program, all appearances are that public schools in the greater Los Angeles area will continue to benefit from the

fellows’ DRI experiences for some time to come.

The many positive program outcomes detailed above are testimony to the overall success of the program and point to several of its components that contributed the most to them. For example:

• The early classroom observation phase was invaluable in introducing the fellows to the types of settings in which they would eventually work and the variety of youth with whom they would be interacting.

• The cooperative instruction phase was one of the more innovative aspects of the program, and critical in easing the fellows into the practice of teaching under the guidance of a seasoned teacher.

• The two-year follow-up phase after the formal pilot program was completed facilitated the fellows’ transition from learner to teacher, and helped several of them succeed in environments where resources were inadequate and faculty in their field were not using standards and reform methods of instruction.

• The DRI Program Office was viewed by the fellows as one of the most important elements of the program, bringing general program advocacy, “troubleshooting,” and individual support to bear as needed. As one of the fellows said in a group interview, with the others concurring, “Dr. Shiflett was the glue that held this program together administratively – and some of us psychologically.”

If subsequent programs based upon the DRI pilot model are implemented, the following recommendations should prove helpful in ensuring their success:

1. Unless the program is large enough to provide adequate coverage in more than one content area, it should focus on only one.

2. Given the tremendous anxiety created among the fellows, and the hurdles that were encountered by the need to pass subject matter examinations in order to obtain a full credential, consideration should be given to enacting at least one of the following suggestions to mitigate this problem:

   • Applicants should provide evidence of passing any tests required for this purpose in advance;

   • Applicants should be required to take a “predictive” test to determine their knowledge of the subject as it is currently being taught and/or will be presented in such examinations;

   • Applicants should be required to take a diagnostic test so that targeting tutoring can be provided to meet their needs; and/or

   • General subject-matter tutoring and specialized coaching should be a built-in component of the program.

3. The full cooperation and support of the school district in which the program is operating should be obtained, and any specific support required of certain positions within the district should be detailed and agreed to in writing prior to the commencement of program operations.

4. The selection of school sites and teachers to work with the program should be made a high priority, and the selection of each should be presented as an honor. Requirements in this regard should include the following:

   • The school, or at least the participating department, should provide evidence of its understanding of and commitment to the practice of reform-based instruction and assessment; and

   • Teacher interest in and ability to constructively work with program fellows should be evidenced by a written application to participate in the program, and an agreement to welcome

observation of their instruction by program personnel prior to their possible selection.

4. The district, the schools, and the teachers should be engaged and treated as program partners, with the teachers being provided with opportunities to participate in special, out-of-school professional development activities with the fellows.

5. Partner expectations, roles and responsibilities should be detailed, and lines and methods of communication spelled out and agreed to prior to the fellows’ initial visits to the schools.

6. College personnel involved with the program should maintain a close working relationship with school personnel, and visit the schools and classrooms throughout the observation and cooperative teaching phases of the program, in addition to the student teaching phase.