Below are the first 10 and last 10 pages of uncorrected machine-read text (when available) of this chapter, followed by the top 30 algorithmically extracted key phrases from the chapter as a whole.
Intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text on the opening pages of each chapter.
Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.
Do not use for reproduction, copying, pasting, or reading; exclusively for search engines.
OCR for page 73
7
Evaluation of
Professional-Development Programs
Large amounts of federal and private money are being spent on innovative
programs for the professional development of K-12 science teachers, and many
people are devoting tremendous amounts of time and energy to this critical com-
ponent of excellent science education. How do we know whether these resources
are productive whether a program's objectives have been accomplished? We
need to be able to determine which programs are working best and which are not,
and we need to know whether a program has provided participants with new
insights, knowledge, or skills and whether it has led to improved achievement
among students. We have few data on the long-term impact of the programs;
obtaining objective data on such programs is daunting and expensive. The prob-
lem is not new. If we can learn from the past, we should be able to do better in the
future.
Much of what we know about the effectiveness of professional-development
programs is based on anecdotes and on reports from teachers, principal investiga-
tors, and program directors involved with the programs themselves. In our re-
view of professional-development programs, we heard repeatedly that teachers
felt empowered by their participation and gained an enhanced sense of profes-
sionalism. We also heard that many teachers particularly elementary-school
teachers felt that they had increased their content knowledge and were more
comfortable in using inquiry-based methods of instruction in their classrooms.
That kind of subjective information is important and useful, but the overwhelm-
ing majority of programs we examined had no formal mechanisms for determin-
ing the effectiveness of programs by assessing how students fared after partici-
pating in programs in which their teachers participated.
73
OCR for page 74
74
PROFESSIONAL DEVELOPMENT OF SCIENCE TEACHERS
We are concerned about that situation because of the lessons learned in the
1980s. In 1984, the General Accounting Office concluded that although the
NSF-sponsored teacher institutes of the 1960s and 1970s had favorable effects on
teachers, as measured by attitudinal surveys, there was no empirical evidence to
support the contention that the institutes had favorable effects on students. At the
time, the costs of evaluation were not allowed in education grants and there
might have been a dearth of instruments available to measure effects on students.
A decade later, there are still an inadequate number of techniques to measure and
evaluate the effects. At a time of federal budget cutbacks, the lack of evidence
linking spending to student achievement was used as one excuse for the elimina-
tion of science-education programs. Federal appropriations for science educa-
tion practically disappeared from the political landscape until another "crisis in
education" had begun again, this time driven by consensus about science literacy
and economic competitiveness rather than national security.
We need to know the effects of professional-development activities on the
classroom behavior of teachers, such as the extent to which they incorporate the
content and process elements of their training into their classroom teaching.
Obviously, we must evaluate student performance what they know and are able
to do as a result of their teachers' professional-development activities. We need
to develop measures of the cost effectiveness of various programs on classroom
behavior. Ultimately, we need to design ways to collect longitudinal data to
measure the effects of professional-development programs for teachers on their
students, including how they learn and make decisions beyond high school.
Obtaining such data will require perseverance to collect and analyze comparable
data over periods of 5-10 years. Funding to support those long-term efforts must
also be obtained. But if the barriers can be surmounted, we will gain better
insight into how professional development affects teacher and student outcomes
over time.
This chapter presents ideas and recommendations for the development of
effective evaluation tools that will yield the data we need. The primary question
to be answered is, "Has the professional-development program being evaluated
helped teachers to create a high-quality learning environment so that the students
are doing better in science?" To develop evaluation strategies that answer that
primary question, we must know what forms of evaluation are required, who
should conduct evaluations, what tools and programs we need to conduct useful
evaluations, what levels of funding are needed to conduct useful evaluations, and
how the administrative structure supports program evaluation.
FORMS OF PROGRAM EVALUATION
Program evaluation can take many forms. Not all professional-develop-
ment programs need to be evaluated in the same way. For example, a lecture
OCR for page 75
EVALUATION OF PROFESSIONAL-DEVELOPMENT PROGRAMS
75
series does not require as extensive an evaluation as a program designed to foster
systemic reform. In its most general form, however, we believe that evaluation
will be most effective if it is designed in the initial planning stages of a program,
if it measures the success of a program against its stated goals, and if it continues
throughout the life of the program and, for students, beyond.
Concurrent Design and Evaluation of
Professional-Development Programs
It- program planners include evaluation in program design, they will find that
it will help them to
Define specific, realistic, important, and measurable program goals.
Identify scientific content and science-process skills that are appropriate
for teachers and their students.
· Choose instructional strategies and followup activities that are consistent
with the objectives of the program and reinforce core concepts.
· Establish mechanisms for receiving continuing participant feedback.
· Establish, before the program begins, procedures and instruments for col-
lecting overall program-evaluation data.
.
Examine a program's cost effectiveness or efficiency.
The last issue is perplexing because it addresses the age-old problem of
"comparing apples and oranges": how can one compare the relatively high cost
of a high-school biotechnology program, with its expensive equipment, to the
relatively low cost of an elementary-school science program that serves hundreds
of teachers? Is the elementary-school science program more cost-efficient sim-
ply because it has a lower per-teacher cost?
Table 7.1 shows how the planning of a professional-development program
and its evaluation are conceptualized as a seamless process. The sought-after
goals of professional-development programs are listed in the first column and are
based on the characteristics of effective programs identified in Chapter 2. The
remaining columns show the steps toward the goals and what might be evaluated
to assess the progress of the program.
Continuing Evaluation
Continuing evaluation (as contrasted with summary evaluation on comple-
tion of a program) can include both informal and formal mechanisms to help
program facilitators to identify problems (and successes) during various stages of
program implementation. Informal evaluation can be conducted by program
staff. Continuing evaluation often uses questionnaires, interviews with partici-
pants, or self-reports in the form of journal excerpts; these types of evaluation
OCR for page 76
76
Cq
be
o
a'
o
a'
a'
o
.=
Cq
a'
o
o
o
.=
o
3
a'
a'
be
.Y
¢
EM
o
sit
: - o.
o
v
m
4=
o
4=
o
Ed
ca
V,
o
.^
ca
0 ~
ca ~
·0 t,.4
ca
0 ca
=.4 ~
0 ~ 4=
O
o
C)
~ ~ ca
0 lo ~
ca ca
hi · ~ Ct
8 ..= ~ ~ ~ ~
~ ~ ° ~ ~) ca 0~e
an ° R O R ~ a R
R .- R O ~ o ~ R
ca ~ ~d O
S~
O ~
~;
ca
ca ~,4
~; ~
SO~ ~
ca ~
~ O
·0 ._4
S~ 4._,
S-.
~; ~
O ·0
;^ ~
C) ca
~ U.
S~ 4._,
C)
._4
·0
S-.
;^ ~
C) P
C) ~
ca
S~
O
O C)
~;
U:~ ·-~
ca 4=
ca _
ca 4=
¢
O · -
·0
cd d
R V,
.~ ~
C) d
~ .~
·4=
._4 ~
C) t,.!
~ .~
~ ·0
3 d
ca ca
S~ d
O
.~
, 4=
C)
C)
ca
ca
· _4
4=
4=
ca
1 · -
~;
ca
·_4
S~
ca
.>
i,
4=
~ d
1 · ~ tM4
~ .
1 Cd
O
S~
d 4=
O
S~
C)
4= ·0 O
_ ~ ca
t~4
, ~ ~
_4 · _4
~ _
_ _
V, ~
·0 O
_
~ ~ s~
O
~ca
4= ~ s~
.0
O ~ -
.
.O
V,
.^ O
~ca 4=
4=
p~ ~ ~
s°- ~ ~0 ca4=
O ~ ~ ~ ~
4= ~ ~P
~ ~ O ~ ~
~ o U E ° ~ ~
O d 4= . ~ v~ t-~ ~
o t~o cr' R V~ R R
o P~ ~ ~ ~ ~ ~
R ~ · t.~ ~ R ,~ tD
Z ~
d o
R C
_ ~ ~ C R
~ ~ ~ d ~
R ~ 't ~
_ ~ ~ R
R - ~ ~ O
V,
·,4
~ ~ _ ~
4= ~C)
~ ca
P4 =) . ~4 ~
= O C) C)
ca _ 4= Cd
|4 V, 14 0
c,,
3
ca ~
_4 ~
i~ ~
C) 14
O
d
O . ~·,4
-4 V, ~
- 4 {t {t
O P44 4 ~P4
O · C) ~ · ~4
~ Ct Cd
"R~ O cr,
.^ ca
£ 3 ° 8
_
~4
C C ~
ca ~O
~,4 ·,4 ~ 4
~ V, P4
,4 4,) 4=
~ ~4
·4= ~
O C) P4
. ~ o] tM4
C) ca ~I
~ 4 ~ V, ~
_4 {t ~,4 ·,O4
~ ~ .~4 ~
,= O O P4
Ct ~ ~
4,) .,4 · -4 _
_4
~O
=4 V,
~ca
4 0
O P4 ce
·,4 =) -t
O
~ O
- 4 V, o
.4= R Ce
~ >4 4~
O O d
P4 · ~4 ~
O E~
~ _
·,4 C,)
ca S
~4 5 ·,4
4= ~
d 4= C'
~ 4= ~4
{t X ~ 4
o 54
~ 4 ct {t
P4 ~ V,
·,O4
P4 ~ ~ ~d
~ P4 ~ t~
0 -4 ~ O
O O p
O ~ O
V, O 4-)
;^
· ,4
~ O
0 14
d
4=
~ ~4
0 14
~4 .,4
~ 4 4=
_4 ~
.,4 0
~ O
_ ·,4 -
'~ c~ ca
4 ~ O
O .4~4 ·,4
_ ca
- 4 C,) ·,4
O Ct 4.)
4= ~
4') .' 4 d
_ ~ 4
4= "4
4= 0 4=
ca
·,4
· ,4
·~4
4=
ca
^C
{t ~ 4
~ ~0
4
{t =4
p4 4=
~ ·4=
.,4 ~
C) ~
.r _4
P4 ·,4
OCR for page 77
EVALUATION OF PROFESSIONAL-DEVELOPMENT PROGRAMS
77
should rely heavily on participants' comments so that appropriate changes can be
incorporated into the program. Often, continuing evaluation leads both to better
ways to achieve the initial goals and to changes in the goals themselves as
perceived needs change. The evolution and improvements in programs that
result from observations made during continuing evaluation are desirable, and all
the most-effective programs that we studied used it. However, the changes and
improvements in programs that result from continuing evaluation complicate
long-term evaluation of program effectiveness because it is aiming at a moving
target.
Long-Term Evaluation
Formal evaluation of the impact of an overall program requires long-term
strategies for data-gathering and analysis that begin with the program's design
and continue throughout the life of the program. Most evaluation stops when a
program ends, although it can take years for the impact on students to become
apparent. Usually, long-term data are not collected, although their collection
might be as simple as tallying the number of science electives taken by students
of a teacher in a middle-school program. Such data provide a quick indicator of
students' interest in science, which might or might not reflect good science teach-
ing in earlier grades.
We found a lack of overall program evaluations linking teacher participation
in professional development with enhancement of teaching skills or student per-
formance. To determine the ultimate impact of a program, long-term evaluation
is needed to keep track of program participants and how they incorporate new
information and techniques into their classroom activities.
Whether evaluation is intended to be continuing or summary, fundamental
questions must be addressed: What are the teachers learning? Is effective peda-
gogy being modeled in the professional-development sessions? Does the pro-
gram address "real needs" of teachers? Does the program hold promise of favor-
ably affecting student learning in science? Other questions that one should
consider in the evaluation are the following:
· Is the program built on models of other well-tested programs? Answers
to this question can reveal whether program planners have educated themselves
about other professional-development programs around the country rather than
inventing new programs from scratch.
· Does the program engage participants in open discussion about educa-
tional practices, reform, curriculum, and policies? Over the years, educational
reforms have been piled on other educational reforms. Veteran teachers have
seen them come and go. For strategic reasons or out of skepticism, many teachers
wait in silence for each fad to pass so that they can go back to what has worked
best for them. Open discussion is a sign of active participation and can reflect a
OCR for page 78
78
PROFESSIONAL DEVELOPMENT OF SCIENCE TEACHERS
deep structural change in how schools work (that is, adjustment of a structure to
accommodate participating teachers' input).
· Does the program have or plan to develop a relationship with systemwide
initiatives in the region? Answers to this question will provide evidence about
the extent to which a program is complementary to or coordinated with
systemwide efforts.
· Are participants particularly teachers being intellectually stimulated
and inspired by the program? Answers to this question might shed light on
whether teachers are motivated to participate in other professional-development
activities or whether, in the long run, they choose to stay in the field.
· Who is participating in the program and why? Answers to this question
will reveal whether the program is reaching all teachers or only those who are
already motivated. Evaluators should also examine the diversity of the partici-
pant pool with respect to the age, race, and ethnicity of students taught and
teachers' race, social class, work habits, age, teaching experience, and stage of
career. Answers to this question might lead to the development of strategies to
involve less-motivated teachers in professional-development activities.
Does the program result in changes in teachers' teaching practices and in
student learning? Answers to this question require that data be collected that
could show changes in teachers' behavior and student performance. Evaluators
need to seek creative indicators of these changes. For example, the adoption of
hands-on or innovative laboratories could be monitored through records of the
purchase of necessary supplies and equipment assuming that these are being
used in the classroom. Changes in lesson plans and laboratory activities might
also indicate that teachers have altered their practices, as might videotapes of
classroom activities when "before and after" tapes exist.
.
EVALUATORS
As noted above, informal, continuing evaluation can be conducted by pro-
gram staff using questionnaires, interviews with participants, or self-reports in
the form of journal excerpts. We found that formal evaluation is usually con-
ducted by principal investigators or program directors. For science-based pro-
grams, the principal investigator, hence the evaluator, will probably be a scientist.
Special problems face him or her. Scientists are trained to plan careful experi-
ments that generate data to support or refute a hypothesis. In education, objective
data are much harder to generate because the variables are harder to identify and
control. Scientists are accustomed to having responsibility for the implementa-
tion of a program but not for its evaluation. They are unlikely to have training in
the evaluation of educational programs. Unlike scientific research, whose prod-
uct is a peer-reviewed paper, the "product" we are dealing with an education
program involves human interactions and is not as readily subject to peer re-
view. Scientists therefore must be aware of the complexities inherent in analyz
OCR for page 79
EVALUATION OF PROFESSIONAL-DEVELOPMENT PROGRAMS
79
ing educational programs. Finally, it is difficult for any principal investigator,
whether scientist or science educator, to oversee an objective analysis of his or
her own program.
Evaluation does not have to be only a means to determine the "success" or
"failure" of a program or individuals. If education researchers who are trained in
evaluation are involved from the beginning of a program, evaluation can also be
a means of conducting research on professional-development programs. Educa-
tion researchers who serve as evaluators can help to bridge the gap between
education research and practice. They can help to design programs that have
measurable components. And they can produce overall program reports that
highlight difficulties faced during program implementation as a means of letting
other program directors know how to improve their programs or avoid mistakes.
When evaluation is seen as a means for educating others in the field, program
staff might feel that they can be honest in their evaluation of their own programs.
Having an evaluator involved in the planning of the program can help to
ensure that program objectives are clear and focused, that the evaluator will begin
to think about evaluation strategies and instruments before the program begins,
and that the program will include appropriate points for the evaluation of progress
and midcourse correction. One warning: As always, balance is necessary. As
one increases one's focus on program evaluation, one needs to be careful not to
contrive neat evaluations by looking for easily measured outcomes or easily
administered tests at the expense of effective program design and implementa-
tion.
EVALUATION TOOLS
In theory, well-designed evaluations of professional-development programs
for science-education reform should be based on testable hypotheses. In reality,
most current evaluation strategies fall into the category of using "How did you
like the course?" questionnaires before teachers have had the chance to try new
techniques in the classroom. Instead, long-term followup evaluations of teachers
to determine how they are implementing their new knowledge and techniques in
the classroom are needed, as are new evaluation instruments that can measure the
long-term effects of programs on both teachers and students.
Much research on ways to conduct useful evaluations is still needed. Re-
search could be conducted on existing programs to produce a comprehensive
guide to evaluating professional development (including a computer database for
ease of access and updating) that would provide program directors and other
interested parties with specific examples of effective evaluation techniques and
tools. The document could address such issues as measurement of teacher and
student conceptual understanding, application of inquiry-based learning, effects
of professional-development programs on students (with samples of assessment
tools for measuring student outcomes), and change in teaching practices or school
OCR for page 80
80
PROFESSIONAL DEVELOPMENT OF SCIENCE TEACHERS
reform. The document should include evaluation research being done interna-
tionally.
FUNDING AND ADMINISTRATIVE ISSUES
Most funding agencies and private funders request information about pro-
grams' effectiveness, but few provide funds or guidance to conduct full-scale
evaluations. Effective evaluation procedures are expensive, and available fund-
ing is inadequate for the kinds of evaluation that are necessary to assess the
linkages between teacher preparation and student performance. Budgets for
programs normally do not include a line item for evaluation. Evaluation of a
program's effects will be most valid when time has passed since the program
ended. Only then can the impact on students be noted. However, most funding
agencies require a grant-program report before an appropriate evaluation time.
Funding agencies should allocate more of their resources to support both evalua-
tion of selected existing programs and evaluation research.
NSF previously has placed the responsibility for evaluating programs on
principal investigators, most of whom do not have a background in survey or
social-science research. NSF's program officers, overwhelmed with large num-
bers of grants, have little time to provide feedback to grantees about their grants
or to follow up on programs. The average tenure of a program officer in the
Education and Human Resources Directorate is 2 years, but most grants run for 3
years. Considering that a grant is in preparation, review, and approval for about
9-12 months, it is possible to be assigned to three different program officers over
a single grant period. Rarely does an NSF program officer stay long enough to
see a round of grants through to completion, much less the next round of proposal
submissions as those programs are submitted for renewal. Newer NSF policies
recognize the need for outside evaluation.
RECOMMENDATIONS
· Those involved in professional-development programs should recognize
the importance of establishing an evaluation plan from the onset of program
design. That process forces a clarity of thinking about program objectives and
activities in much the same way that careful planning of one's scientific research
forces clarity of thinking about experimental design.
· Every professional-development program should include evaluation
mechanisms from the beginning. The evaluation should continue for the life of
the program and seek feedback from all participants.
· Because formal long-term evaluation of an entire program requires more
extensive analysis than informal evaluation, it is not cost-effective for every
program. Instead, we recommend that clusters of similar kinds of programs, such
as biotechnology or systemic programs, or all programs in a geographic region be
OCR for page 81
EVALUATION OF PROFESSIONAL-DEVELOPMENT PROGRAMS
81
reviewed as a group. The goal is to learn about the net effect of various kinds of
programs on teachers and students and to identify strengths, weaknesses, and
gaps in content, pedagogy, or geographic distribution.
· Program directors and principal investigators should not necessarily have
primary responsibility for long-term evaluation of their own programs; these
persons often do not have sufficient background in the methods of social-science
and education research and might not be objective about their own programs.
Ideally, professional-development programs include a budget item for a qualified
evaluator, who should be involved from the onset of program. We concur in
previous recommendations to NSF that continuing evaluations (and evaluation
specialists) be considered "legitimate uses of project funds" (SRI, 1988~.
· Scientists and other program developers should work with colleagues in
the social sciences and education to identify experienced program evaluators who
can help to plan and participate in overall program evaluation.
.
Funding agencies should support more research about effective evalua-
tion methods. NSF should fund the development of a comprehensive guide to
evaluating professional development that includes a compendium of educational
research and examples of effective evaluation. The task is complex and requires
evaluators who have formal training in program evaluation. A recent publication
by the Council on Foundations, which includes nine extensive case studies of
evaluation projects, is a start in this direction (Johnson, 1993~.
Representative terms from entire chapter:
continuing evaluation
