Below are the first 10 and last 10 pages of uncorrected machine-read text (when available) of this chapter, followed by the top 30 algorithmically extracted key phrases from the chapter as a whole.
Intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text on the opening pages of each chapter.
Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.
Do not use for reproduction, copying, pasting, or reading; exclusively for search engines.
OCR for page 25
2
The Selecton and
Interpretation of Indicators
In order to develop effective policy for precollege
education in science and mathematics, information is
needed on its current condition and on the effects of
efforts to improve it. Given, however, that there are
limitations to the resources that can be devoted to data
collection, what aspects of science and mathematics
education is it most important to monitor? And what kind
of information is most useful for the lay governing
bodies and professionals involved in making decisions
about these critical areas of education?
AVAILABLE DATA AND INFORMATION ON EDUCATION
A large supply of statistical data and research
information is available on education in general. At the
national level, the National Center for Education
Statistics (NCES) of the Department of Education has as
its main responsibility "to report full and complete
statistics on the conditions of education in the United
States . . ." (General Education Provisions Act, as
amended (20 U.S.C. 1211e-1)). The Center publishes two
major compilations annually: The Digest of Education
Statistics, issued since 1962, which provides an abstract
of statistical information on United States education
from prekindergarten through graduate school, and The
Condition of Education, issued since 1975, which presents
the statistics in charts accompanied by discussion. The
NCES and other components of the Department of Education
also sponsor periodic surveys, for example, High School
and Beyond Study, a study of 1980 high school graduates
and 1980 sophomores (National Center for Education
Statistics, 1981a), which was extended to 1982 graduates,
25
OCR for page 26
26
and the earlier National Longitudinal Study of 1972 high
school graduates (National Center for Education Statis-
tics, 1981b). These studies provide information on
student enrollment and achievement, although information
specific to mathematics and science education is limited.
The Department of Education also supports the National
Assessment of Educational Progress (NAEP), which since
1969 has provided data on scholastic achievement and
student attitudes, one of the few such sources that
involve well-designed national samples.
Another source of information is the International
Project for the Evaluation of Educational Achievement
(TEA) in which the United States has participated. A
comparison of mathematics achievement and schooling
variables in 12 countries was carried out using data
collected in 1964 (Husdn, 1967); an assessment of science
education involving 14 countries was done in 1970 (Wolf,
1977). New data on mathematics achievement in 24 coun-
tries were collected in 1981-1982 and their analysis is
in progress; a summary report on findings in the United
States is available (Travers, 1984). The science
assessment is also being repeated, with 30 countries
participating. Both the Department of Education and the
National Science Foundation (NSF) as well as private
foundations have provided support for these international
assessments.
The NSF has special responsibility in the area of
science and mathematics education, but most of its data
collection activities focus on higher education and
scientific and engineering personnel rather than on
precollege education. However, NSF does support some of
TEA ~ ~ work and has sponsored special studies on science
and mathematics in the SChOOlS, most recently a national
science assessment using the NAEP framework (Hueftle et
al., 1983). Three landmark studies were carried out in
1977-1978 with NSF support: a review of the literature
on science and mathematics improvement efforts between
1955-1975 (Helgeson et al., 1978; Suydam and Osborne,
1978); a survey in 1977 of the current status of
education in these fields (Weiss, 1978), which will be
repeated in 1985; and a series of case studies of schools
(Stake and Easley, 1978). Some of the information
resulting from these NSF-supported studies and data from
other sources have been compiled in a data book (also
covering higher education and employment in science and
engineering), which was first issued in 1980 and revised
in 1982 (National Science Foundation, 1980, 1982a).
OCR for page 27
27
Every state also has its own data collection system,
much of it devoted to fiscal, demographic, and managerial
information, but also including data on enrollments,
personnel, and student achievement. There is, however,
considerable variation in the types of data collected by
states and in the manner of collection, which is not
surprising in view of the organizational diversity among
the states (and, within each state, among school dis-
tricts) with respect to their educational practices and
institutions (see Tables A2 and AS in the Appendix).
Larger local education agencies also collect information
that they find useful for their internal operation as
well as data requested by the state agencies. The data
from local education agencies exhibit an even greater
diversity than do those of the state systems.
In addition to the governmental sources of information,
some data are available from private organizations.
Educational associations collect relevant data, usually
on the supply and demand, pay, and characteristics of
teachers (see, for example, Graybeal, 1983). Some
scientific societies occasionally survey or study the
substance of what is taught in their disciplines at the
precollege level and publish their findings.
THE CONCEPT OF INDICATORS
The existence of potentially relevant information does
not necessarily make it possible to formulate conclusions
about the state of mathematics education or science
education--or any other field. For one thing, the data
often are not comparable; see, for example, the critique
by Gray (1984) of the comparison of state data made by
the Department of Education (Bell, 1984).
For another,
the quality of the data is sometimes too low to permit
robust findings. Lastly, due to the massive amount of
data, it is difficult to summarize the information or
draw implications. The use of suspect data or selective
interpretations of data may lead to inappropriate policy,
as pointed out by Peterson (1983) and by Stedman and
Smith (1983) in their articles on the recent reform
proposals for education.
To provide focus to the problem of having to picture
complex systems with massive amounts of diverse data, the
concept of indicators has emerged. An indicator is a
measure that conveys a general impression of the state or
nature of the structure or system being examined. While
OCR for page 28
28
it is not necessarily a precise statement, it gives suf-
ficient indication of a condition concerning the system
of interest to be of use in formulating policy. Johnstone
(1981) uses the analogy of the litmus test in chemistry,
which gives an indication of the acidity or alkalinity of
a liquid, but does not provide a precise measure of pH
(the concentration of hydrogen ions, the condition that
determines acidity or alkalinity). Optimally, an indi-
cator combines information on conceptually related
variables, so that the number of indicators needed to
describe the system of concern can be kept reasonably
small. Limiting the number of indicators is important
for two reasons. First, individuals involved in making
decisions about such a complex endeavor as education
require information that is relevant and easily under-
stood. To achieve the necessary clarity requires reduc-
tion and simplification of pertinent information, together
with a discussion of the selected indicators that inter-
prets their values and explains their meaning and limita-
tions. Second, since the progress of any field, such as
science or mathematics education, can be tracked only if
measures are repeated periodically, the feasibility and
cost of indicators become critical factors. There are
advantages, then, in adopting a small number of indica-
tors, carefully selected to highlight major aspects of
education in the areas of interest, so as to encourage
continuing data collection.
There are four stages in the development of indicators:
identifying the central concepts relevant to the system
in question; deciding what measurable variables best
represent those concepts; analyzing and combining the
data collected on the variables into informative indi-
cators; and presenting the results in succinct and clear
form. Regarding the first step, education systems have
generally been modeled in terms of inputs, processes, and
outputs. A conceptual framework that follows this model
but more specifically maps the domain of science education
has been proposed by Welch (1983) and is outlined in
Table 1.
While the use of such a framework highlights the major
areas to be covered, it does not specify the combination
of variables that will best portray prevalent conditions
in each area. For this purpose, the most important out-
comes desired from mathematics and science education must
first be specified. Next, the schooling variables that
are related to these outcomes and that can be affected by
educational policy must be identified. Third, in order
OCR for page 29
29
TABLE 1 Domain of Science Education
Context or Antecedent Transactions
Conditions (Inputs) (Processes)
Outcomes
(Outputs)
Student Student Student
characteristics behaviors achievement
Teacher Teacher Student
characteristics behaviors attitudes
Curriculum Classroom Career choices
materials environment Teacher changes
Public attitudes External Institutional
Goals influences effects
Advances in science . . . etc. . . . etc.
School climate
Home environment
. . . etc.
SOURCE: Adapted from Welch (1983).
to assess current conditions and monitor changes, appro-
priate measures for the identified variables must be
selected (or developed). Using these measures and
carrying out a variety of analyses will lead to results
that can be displayed in the form of statistical indi-
cators portraying the condition of mathematics and science
education. The choice of analyses and indicators, like
the selection of variables, should be guided by relevance
to policy. The next three sections discuss the selection
of variables; succeeding chapters discuss how the selected
variables might be measured and analyzed.
SELECTING INDICATOR VARIABLES
Outcomes
The outcome most clearly expected of instruction in
mathematics and science is the acquisition of knowledge,
abilities, and skills in those fields. Some degree of
proficiency is deemed essential for all high school gradu-
ates so that they can function effectively in society and
manage their personal and family lives; additional prepar-
ation may be needed to take advantage of further education
OCR for page 30
30
or to participate successfully in the world of work. The
importance given to student achievement as an outcome of
education is documented by the many measures developed to
assess it, ranging from quizzes constructed by individual
teachers to standardized tests with norms based on nation-
ally representative samples of students, from minimum-
competency tests that are expected to be passed by all
students to tests of material in advanced curricula.
Most states have their own student assessment programs
(see Table 5, in Chapter 3, and Table As, in the
Appendix), as do many of the larger school districts. As
noted above, NAEP was established some 15 years ago to
provide information on educational achievement for the
country as a whole. Although not nationally representa-
tive, the scores made by students from year to year on
college entrance tests are frequently interpreted by the
media and the public as indicators of academic perfor-
mance. Public interest has extended to international
data on student achievement; the results of the tests
administered through the TEA have been used to document
the achievement of U.S. students compared with that of
students in other countries. Since these various means
of assessing student achievement do not always yield
consistent results, syntheses and interpretations are
necessary; see, for example, the one done for mathematics
and science achievement by Jones (1981).
The emphasis and resources invested on assessing
student achievement demonstrate the importance attached
to this outcome--in fact, the acquisition of knowledge is
the main reason for the existence of formal education.
Hence, student achievement must be considered as the
primary indicator of the condition of science and mathe-
matics education.
A second outcome often stated as a goal of science and
mathematics education is the development of favorable
attitudes of students toward these fields. Thus, for
example, the most recent national science achievement
assessment (Hueftle et al., 1983) included items on
student attitudes toward science activities and science
classes, science teachers, and science careers and about
the usefulness of science. It is not clear, however,
whether favorable attitudes are to be considered a desired
outcome of schooling in and of themselves or whether they
are considered important because they are believed to
mediate such other desirable outcomes as increased
involvement with mathematics and science activities and
therefore increased achievement.
OCR for page 31
31
Research evidence on the relationship between psycho-
logical factors and achievement indicates that classroom
morale and encouragement at home correlate rather highly
with student achievement (Walberg, 1985), but the correla-
tion between favorable attitudes toward a particular
subject and success in learning that subject is fairly
low (Welch, 1983; Horn and Walberg, 1984). In an analysis
of research results from a number of studies on the rela-
tionship between science achievement and science attitude,
Willson (1983) also found only a modest correlation of
.16 across all grade levels, including college. In the
same study, causal ordering results supported the hypothe-
sis that achievement affects attitude rather than the
other way around, at least for grades 3 to 8. One problem
in the assessment of attitudes and interpretation of
results is the lack of adequate theory: as a consequence,
some of the instruments and test items that have been
used to assess attitudes toward science have given incon-
sistent and ambiguous results, raising doubt as to what
is really being measured (Munby, 1983). Given the uncer-
tainties about the significance of favorable attitudes
toward a particular field of study and about some of the
measures used, the committee in this report has not
treated them as a primary indicator of science and
mathematics education.* The committee believes that the
question of developing and using an indicator representing
student attitudes towards science and mathematics deserves
reconsideration in any further work on indicators.
Other outcomes of education generally considered to be
important include college attendance, choice of college
majors, choice of careers, and later career paths, includ-
ing life income and job satisfaction. Each of these has
received the attention of researchers seeking to assess
the benefits of education; each is important to indi-
vidual and societal goals and to the development of human
resources. However, each is mediated by many variables
other than those associated with schooling. For example,
it has been suggested that plans for college attendance
and field of study might be taken as a proxy for student
attitudes, but economic conditions and perceptions of
future employability strongly affect such plans.
One school variable, additional years of schooling,
has been found to be correlated with increases in overall
*Wayne W. Welch dissents from this decision.
OCR for page 32
32
lifetime income and with job satisfaction, but neither of
these outcomes has been tied to instructional variations
within the precollege experience, given the same number
of years of school completed. (Student achievement, how-
ever, does predict years in school.) Despite the lack of
strong correlations between school achievement and work
performance, employers continue to resort to secondary
indicators such as academic degrees achieved and schooling
records for applicants without prior experience (Spence,
1973), because degrees and schooling records can be more
readily assessed than nonschool variables that might be
related to job performance. This use of school variables
to select new employees does not imply that career out-
comes should be used as an indicator of schooling quality.
In general, the more distant an outcome from the
immediate purpose of instruction, the more tenuous the
link and the more likely that nonschool variables will
affect that outcome. Pending research findings that more
clearly link schooling variables to career achievement
and other life outcomes, the committee has not chosen to
include in this preliminary review indicators representing
such outcomes.
Schooling Inputs and Processes
The selection of student achievement as the outcome
variable of greatest interest determines to a consider-
able extent what schooling input and process variables
need to be selected, namely, those that seem to have some
causal relationship to student achievement. The landmark
study by Coleman et al. (1966) and several succeeding
studies appeared to throw into question the intuitively
obvious connection between differences in schooling and
student performance. More recent work, however, has
consistently shown significant positive associations
between certain schooling variables and cognitive achieve-
ment by students. The most robust effects are correlated
with "opportunity to learn": that is, whether and for
how long students are exposed to particular subject
matter. Opportunity to learn in school consists of the
instructional time spent on a subject together with the
content of that instruction. To a considerable extent,
both time and content are controlled by the teacher,
although in secondary school students themselves decide
at least in part how many units of a subject to study.
OCR for page 33
33
School Processes: Instructional Time
Educational practice assumes that exposure to a subject
will lead to students' acquiring knowledge and skills per-
taining to that subject.
Recent evidence supporting this
assumption comes from major cross-sectional studies anct
assessment efforts. One such assessment, an extensive
study of elementary school teachers in California, found
increases in academic learning time strongly associated
with increases in student learning (Fisher et al., 1980).
Similar results have also been found for mathematics and
science.
Using data from the 1977-1978 NAEP study of student
performance in mathematics, Welch et al. (1982) found
that, while background variables (such antecedent con-
ditions as home and community environment and previous
mathematics learning) accounted for 25 percent of the
variance in mathematics achievement, exposure to mathe-
matics courses explained an additional 34 percent. The
study was replicated by the authors on three different
national samples with similar results. Using another
NAEP sample, Horn and Walberg (1984) also obtained a
sizable correlation (.62) between the number of mathe-
matics courses taken and student achievement for 17-year-
olds. In a somewhat different analysis, using data from
a special 1975-1976 NAEP study on mathematics achievement,
Jones (1984) found that the average mathematics score of
17-year-olds varied from 47 percent correct for those
having taken no algebra or geometry courses in high school
to 82 percent correct for those having taken at least 3
of such courses. While some of the difference may
years
be accounted for by the fact that more proficient students
tend to take more mathematics courses, part of the dif-
ference remains even after adjusting for initial profici
ency (see Wisconsin Center for Education Research, 1984).
The relation between amount of schooling and science
achievement is also positive. Welch (1983) has shown a
correlation of .35 between achievement and semesters of
science. Similarly, Wolf (1977) found a correlation of
.28 between science test scores and course exposure.
Based on educational practice and experience and the
available research evidence, the committee believes that
time given to a subject in elementary school and course
enrollment in secondary school ought to be considered key
process variables in developing indicators of mathematics
and science education. This is not to say that instruc-
tional time is the only factor affecting learning or that
OCR for page 34
34
increases in instructional time will yield equivalent
increases in student achievement. Clearly, the quality
of instruction as exemplified by such process variables
as teacher behaviors, student behaviors, and classroom
environment also influence student achievement to a
considerable degree. However, given the limited knowledge
available about these variables and the constraints
inherent in this preliminary review, the committee does
not recommend their use as indicators at this time. The
process variable of instructional time or course enroll-
ment can be considered a proxy for process variables in
general until others can be documented and measured with
greater certainty.
Input Variables
_ontent The content of instruction is obviously
another dimension of opportunity to learn. The research
that has been done confirms what common sense would
predict: emphasis on specific subject matter increases
student performance on tests of that subject. Thus, both
Husen (1967) and Wolf (1977), summarizing the TEA mathe-
matics and science assessments, report that student test
scores in all participating countries are correlated with
the teachers' ratings on whether the topics on the tests
had been covered in instruction. The correlation of
student achievement with number of mathematics courses
taken becomes even stronger when the content of the
mathematics courses is taken into account: with the
variables controlled for one another, Horn and Walberg
(1984) found that an index of the number of advanced
mathematics courses taken correlated somewhat more highly
with mathematics achievement than did just the number of
all mathematics courses taken. The common-sense idea
that subject matter content, not only amount of time, is
important to student learning has been further documented
in an analysis of 105 studies on the effects of alterna-
tive curricula: Shymansky et al. (1983:387) found that
students exposed to new science curricula (i.e., those
developed during the school science and mathematics
reforms that followed the launching of Sputnik in 1957)
"performed better than students in traditional courses in
general achievement, analytic skills, and process skills
[i.e., the skills stressed in the materials]. . . . On a
composite basis, the average student in new science
curricula exceeded the performance of 63 percent of the
students in traditional science courses."
OCR for page 35
35
Teachers The second schooling input deemed critical
by the committee is the number and qualifications of
teachers with instructional responsibilities in science
and mathematics. Classroom teachers are the single most
costly resource component in schooling. Although the
teacher share of the school dollar has dropped in the
last decade--in part because teacher salaries have not
kept pace with inflation--those salaries still repre-
sented 38 to 44 percent of total direct operating costs
for public schools during 1982-1983, even without counting
pension payments or fringe benefits (Feistritzer, 1983;
Educational Research Service, 1984, personal communica-
tion). Moreover, even though the extent of their control
over instructional time and content may vary, teachers do
determine the nature of classroom instruction.
At the elementary level, the number of teachers is not
now an issue, but it may become one as student enrollments
increase again in the mid-1980s. Even now, however, the
competence of elementary school teachers with respect to
mathematics and science is of major concern. Assessing
the competence of teachers for grades 7 and 8 poses a
special problem. In several states, teachers certified
for elementary school are automatically certified to
teach those grades as well without the subject-matter
preparation usually required of secondary school
teachers; yet those are the grades when differentiation
of the curriculum into disciplinary courses begins and
one would expect the need for greater subject-matter
knowledge by teachers than for grades 1 to 6. At the
secondary school level, both the quantity and the
qualifications of the teachers responsible for teaching
mathematics and sciences determines what courses are
offered and how well they are taught.
Expenditures and Other Cost Factors In addition to
content and the number and qualifications of teachers,
other input variables were considered by the committee.
One input variable often used to try to explain educa-
tional outcomes is the amount of money invested in
schools. An effort has been made to determine dollar
costs of "adequate n education, state by state (Miner,
1983), that
Differences
bound to be
perspective.
shows wide variability over the states.
among communities within states also are
large, and are less tractable from a national
OCR for page 36
36
Some cost factors, especially per-pupil expenditures,
teacher salaries, expenditures on books and materials,
and acquisition of computers and laboratory equipment
have been separately tracked as important inputs.
Attempts to relate such expenditures to student achieve-
ment have yielded mixed results. In a review of quanti-
tative studies of school effectiveness, Murnane (1980:14)
concluded that the primary school resources are teachers
and students and that such other inputs as physical
facilities and class size "can be seen as secondary
resources that affect student learning through their
influence on the behavior of teachers and students. n
Little is known, however, about the ways in which teacher
and student behaviors are related to alternative invest-
ments, say, in teacher salaries, materials and equipment,
school plant, specialist teachers, and the like.
A major cost factor is class size, yet the evidence
indicates that marginal (if costly) decreases in class
size of two or three students (e.g., from 33 to 30)
hardly affect achievement (Glass et al., 1982). In a
study of achievement gains in grades 3 to 6, Summers and
Wolfe (1977) found that large classes (more than 28
pupils) were detrimental for low-achieving students but
were beneficial for high achievers, a finding that might
explain the inconsistency of results of research on class
size that fails to consider the achievement levels of
students. Another major cost factor is that associated
with teacher salaries. While salary level might be a
good indicator of public attitudes about education, it
has not consistently been found to be related to student
achievement. Salary levels are related both to the
seniority of teachers and to the extent of teachers'
education beyond the B.A. level. But neither teacher
seniority nor post-baccalaureate education seems to show
a simple positive relationship to student learning.
Indeed, under some circumstances, a negative relation
between student achievement and post-baccalaureate
education is reported (e.g., Summers and Wolfe, 1977;
Hilton et al., 1984). Since teachers with advanced
degrees command higher salaries than those without such
degrees, this finding would lead to the expectation that
teacher salaries would also relate negatively to student
achievement.
In a review of 130 studies that analyzed the relation-
ship between student performance and school expenditures,
Hanushek (1981:30) concluded that "higher school expendi-
tures per pupil bear no visible relationship to higher
OCR for page 37
37
student performance. n Walberg and Rasher (1979) conjec-
ture that it may not be total educational expenditure
that may make a difference, but highly targeted and
selective investments. Yet school budgets, whether local
or state, are not constructed nor reported to provide the
kind of detail needed to track expenditures for specific
subject areas such as science or mathematics. Even if it
were feasible--probably at considerable cost--to disaggre-
gate budgets in this manner, the expenditures would still
need to be related to student achievement before they
could be accepted as a useful indicator. So far, adequate
evidence is lacking.
Another approach might be to track federal support.
There is evidence that the post-Sputnik federal investment
in science and mathematics education helped increase both
enrollment and performance in those subjects. But while
the programs supporting science and mathematics education
within the National Science Foundation and the Department
of Education are generally identifiable, some others of
considerable magnitude--for example, those sponsored by
the Department of Defense and by the National Aeronautics
and Space Administration--are not.
In the absence of relevant budgetary information and
without further evidence on the relationship between
educational spending and student performance, the com-
mittee, in this preliminary review, decided not to recom-
mend use of expenditure data as an indicator. Given
interest in the funding of education, however, financial
data and research on the economics of education should be
. — _
_
. ~ ~
reexamined in any future consideration ot ~na~cators.
Public Attitudes One other indicator of input was
considered by the committee: =~ =
science and mathematics education. Perception of these
fields appears to have discernible effect on the emphasis
they receive in school, as witness the current wave of
increases in requirements for high school graduation (see
Table 5, in Chapter 3). Federal funding may be another
indication of public attitudes; for example, the share of
Public attitudes coward
the total NSF budget allocated for science education rose
to nearly 50 percent in the late 1950s, decreased to
about 30 percent in the 1960s, has been 10 percent or
less over the last decade, and is now on the rise again
(Klein, 1982). But these fluctuations are not mirrored
in measures of public opinion. The results of 15 years
of polling by the Gallup Organization on attitudes toward
OCR for page 38
38
education do not show parallel swings: mathematics has
ranked high in importance as a school subject throughout
this period; science generally has ranked near the average
of school subjects (see, e.g., Gallup, 1981, 1983).
Given little change in public attitudes over the last 15
years, at least as demonstrated by this measure, and the
uncertainty of the relationship between public attitudes
and schooling outcomes, the committee did not use this
variable and is not recommending its development as an
indicator.
Conclusion
In sum, the committee has identified a minimal set of
key schooling variables that should be monitored, shown
in Figure 1. Assessing the condition of each of these
variables will set the stage for the development of
indicators. For example, counting the number of cer-
tified mathematics teachers actively teaching in a
particular school year provides a datum that could be
displayed against other pieces of information: total
secondary school enrollment, enrollment in mathematics
courses, total number of secondary school teachers,
expected demand for mathematics teachers, numbers of
mathematics teachers in some previous year, or--if there
are separate counts for different geographic entities--
comparisons of the density of mathematics teachers
related to student population.
Education System
INPUTS PROCESS OUTCOME
Teachers
quantity
qual ity ~
Curricul urn
content
Instructional
ti me/cou rse ~
ach tenement
en rol I me nt
Student
FIGURE 1 Areas of science and mathematics education to
be monitored.
OCR for page 39
39
COLLECTING INFORMATION
Most of the information available on the variables
selected by the committee in the first phase of its work
has been collected through surveys and student tests,
although occasionally case studies have been employed to
describe classroom processes in greater detail (e.g.,
Stake and Easley, 1978).
_ Some surveys and tests use
whole populations, others are based on national (or
state) samples, still others are characterized by
self-selection of participants, as in the case of the
College Board's Scholastic Aptitude Tests (SATs). Some
surveys are planned to document conditions at a single
point in time (e.g., Weiss, 1978); some, such as several
of the NCES data collections, are repeated annually;
others--IEA, for example--are repeated at irregular
intervals; still others are designed as longitudinal
studies that follow a cohort population over a number of
years.
Methods for collecting information pertinent to the
selected variables depend on the nature of a particular
variable and on the types of analyses appropriate for
portraying values associated with it. For example, data
on the time allocated to each subject in elementary
school can be collected through questionnaires to school
personnel, but the use of instructional time in the
classroom can best be documented by observation. Since
this entails time-consuming research procedures, only a
limited number of cases can be studied in detail. Case
studies are also useful for uncovering problems with data
collected through surveys. Thus, data on enrollments in
high school courses can be collected from student trans-
cripts, self-reports by students on questionnaires, or
reports by school personnel--likely with significant
discrepancies among these three sources.
Examination of
individual course syllabi and observation of the subject
matter actually taught under given course titles can
clarify such discrepancies. In general, a mix between
sample surveys, full population censuses, and case studies
seems optimal, with studies linked over time by a consis-
tent set of defined indicators.
Periodic replication of studies is necessary if tem-
poral trends are to be identified, but this does not
necessarily mean annual surveys. Careful thought must be
given to reducing the response burden entailed in surveys
and the disruption that sometimes accompanies case
studies. For some purposes, especially for preparing
OCR for page 40
40
budgets, annual data may be necessary, but for the purpose
of documenting changes over time in the state of science
and mathematics education, periods between surveys can be
2 or 3 years, or even 10 years, as in the case of the
complex TEA studies. One way of limiting both the expense
and the disruption and response burden of periodic surveys
and case studies may be to set up a carefully selected
panel of schools, with systematic rotation of schools
into and out of the panel, to provide a consistent data
base.
DISAGGREGATING DATA
Collecting Data at the State and Local Levels
Much of the data used to document the several recent
reports on education that have given impetus to various
reform efforts come from national surveys or nationally
administered tests. Such information may be useful for
developing federal education policy and for following
general national trends. However, education in the
United States is decentralized and, despite some
tendencies toward conformity, quite diverse in inputs,
processes, and outcomes. Each state education system
represents a unique combination of factors; so does each
local system. The richness and sometimes even the mean-
ing of information is obscured by reporting only national
averages. Indeed, nationally aggregated statistics are
of limited use in formulating state and local policy: it
is states--and localities--that carry the authority for
education. Therefore, if the condition of science and
mathematics education is to be portrayed so as to inform
all the people and policy makers involved in education,
indicators must be selected to be useful at the state and
local level as well as at the national level. Moreover,
the appropriateness of the indicators must be tested
against the burden of collecting the requisite information
at each level. For these reasons, this report presents
data relevant to the selected indicators for several
states as well as nationally aggregated data. Each of
the states cooperating with the committee already has
good data systems in place; the inclusion of information
from these states is intended to demonstrate both the
feasibility of the committee's suggested indicators and
some of the problems to be overcome in obtaining the
pertinent data. In addition, even though the included
OCR for page 41
41
states were not selected on the basis of being representa-
tive or exhibiting particular contrasts, the data show
considerable variation from the national data as well as
from state to state. By analyzing such variations,
analogous data on the same indicators that come from
different reporting groups greatly add to the value of
the information available.
Disaggregating Data by Demographic Descriptors
To serve the national goal of equal educational
opportunity, it is important to collect certain data by
gender and minority status. The reason for this type of
disaggregation is to obtain information on critical dis-
tributional issues; for example, different enrollment
rates by members of different minority groups in advanced
mathematics and science courses may provide at least a
partial explanation for different achievement levels.
Data for a whole school population (or any age cohort)
cannot be used to identify such distributional differ-
ences. The underrepresentation in the sciences and
mathematics of individuals from some minority groups and
of females makes it important to collect data pertinent
to input and process indicators in such a way as to
illuminate existing differences. Other demographic
descriptors may be important for a given indicator.
Within a state, for example, the density of population
may affect, say, the number of science teachers per
number of students in different parts of the state, as
may the economic characteristics of different communities.
Separating Data by Educational Level
Since the teaching of science and mathematics in
elementary school is not generally provided by specialist
teachers and enrollment is not recorded by specific
courses, some indicators may have to be represented by
different measures at different levels of education.
Exposure to science instruction, for example, may be
represented in minutes per week in elementary school and
by student enrollment in physics, chemistry, biology, and
other specific courses in secondary school. Similarly,
measures of achievement will need to be different for
elementary and for secondary education. A special prob-
lem in this regard is the middle or junior high school,
OCR for page 42
42
which may comprise any 2, 3, or 4 years between grades 5
to 9 and may be considered part of either the elementary
or secondary school.
INTERPRETING INDICATORS
An indicator acquires meaning according to the inter-
pretation given to its measured value. There are several
bases for interpretation, all using comparisons of some
sort. Most commonly, the value of an indicator at a given
time is compared with its value at some earlier time. For
example, changes over time may be observed in the indi-
cator "the percentage of students graduating from high
school who have taken three or more years of science," or
in the indicator "the percentage of students who achieve
within a given range of scores on comparable tests. n
Another basis of comparison is among groups or geographic
entities: this basis is appropriate to address distribu-
tional issues. Thus, it is illuminating to examine the
supply in various states of certified teachers of science
or mathematics as a proportion of the total number of
teachers in each of these states assigned to science or
mathematics classes, or the proportion of female students
enrolled in high school physics classes compared with the
proportion of male students. Changes in observed differ-
ences among geographic entities or population groups
can, of course, also be related to changes over time. A
third basis for comparison is to establish an ideal value
for an indicator and record the difference between it and
the observed value; for instance, the number of qualified
mathematics teachers available might be compared with the
supply needed. The problem with this method is that
determining the ideal value is usually difficult. For
example, a higher demand for teachers might be estimated
if it is assumed that higher teacher/pupil ratios are
desirable because they yield higher student achievement
than if the estimate is based on current teacher/pupil
ratios. Establishing ideal values often involves judg-
ments about goals and priorities; it is therefore best
left to those making policy about education rather than
to those providing information.
For indicators for which ideal values cannot be
established, international comparisons (a variation of
comparing geographic regions) are sometimes used, as in
the case of student achievement. Such comparisons are
subject to major methodological criticism because of
OCR for page 43
43
social, cultural, economic, and political dissimilarities
in the purposes and practices of education in different
countries. Yet, in the absence of ideal values, student
achievement in science and mathematics in other indus-
trialized nations continues to be used as a benchmark
against which to assess student achievement in this
country. The most responsible of the international
studies, including those carried out under the TEA
auspices, have collected information on differences in
cultural traditions, family variables, forms of educa-
tional organization, and schooling processes, so that the
ways in which these differences affect student achievement
might be examined. Also, the tests used to assess
achievement in science and mathematics (as well as in
other fields) are carefully standardized. They are based
as much as possible on a common core of the various cur-
ricula in use in the different countries and thus repre-
sent agreement on what students ought to know, even
though much of the content of advanced courses may not be
included in the tests. Hence, international comparisons
of the performance scores on these tests are relatively
free of the kinds of cultural bias that would vitiate
comparability in other studies less carefully designed
and controlled, and the wealth of accompanying information
has served to explain some of the differences in results
All three methods of interpreting indicator values--
comparisons over time, comparison among groups or geo-
graphic entities, and comparison to an ideal value--are
used in this report. These interpretations are accom-
panied by commentary on their appropriateness and
associated difficulties in given instances.
.
Representative terms from entire chapter:
mathematics education
