Below is the uncorrected machine-read text of this chapter, intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text of each book. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.
- 7 Indicators of Curriculum The curriculum interacts with teachers and students in complex and important ways. Classroom behavior is inseparable from curricu- lum. By providing incentives that stimulate effective teaching and learning, or by creating constraints on study and understanding, the curriculum affects the choices of students and teachers in every cIass- room. A curriculum may or may not provide incentives for teachers to master specific teaching techniques, such as laboratory experi- ments or the use of current events in creating applied mathematics problems. A curriculum might create opportunities for students to do extra work on questions raised in school, for example, by focusing attention on the evolutionary implications of insect species diversity. And a curriculum can act as a constraint on both teachers and stu- dents when the information conveyed through textbooks or tests is inaccurate, explanations are confusing or misleading, the logic of a concept and its derivation is lost, or mathematics or science is viewed as the memorization of facts and technical vocabulary. These examples suggest the importance of the idea that a cur- riculum, by itself, does not cause teachers and students to behave in a certain way. Teachers or students can ignore a textbook, correct its errors, fad] to carry out its inappropriate methods and in so doing, create a learning experience that is better or worse, or simply differ- ent, from the one envisioned in the formal curriculum. But curricula still matter. By providing materials, encouragement, points of view, 119
120 INDICATORS OF SCIENCE AND MATHEMATICS EDUCATION evaluations, and other pressures for certain approaches to teaching and learning, as well as discouragement and sanctions for others, curricula shape behavior. By portraying science or mathematics as it is actually practiced, or by substituting a dogmatic, rigid version, curricula signal to teachers and students how they are expected to behave if they continue their work in science and mathematics. As important as curriculum is to the quality of science and math- ematics education, no indicators exist to assess curriculum quality (Raizen and Jones, 19853. Science texts are reviewed from time to time by professional bodies, for example, by the American Associ- ation for the Advancement of Science (1985a, 1986a, 1986b), but this represents only a small slice of the curriculum and can address only partially the kinds of policy questions that confront teachers, educators, and others who need to make decisions about curricula. DEFINING THE CONCEPTS This chapter recommends the development of indicators to assess the coverage and quality of the mathematics and science curricula in the schools. Before considering how and why such indicators might be formed, two prior questions need to be addressed: What is meant by curriculum? Who will use curriculum indicators and how will they use them? What Is Meant by Curriculum? The curriculum is usually visualized as an operational plan that includes the substantive content, the expected actions and behav- iors of teachers, the expected actions and behaviors of students, and the technology (textbooks, laboratory exercises, computer programs, tests, explicit pedagogic strategies) for conveying subject matter and structuring teacher and student activities. Indicators for two com- ponents of the curriculum thus broadly defined-the actions and behaviors of students and of teachers are discussed in the two pre- ceding chapters. Therefore, the term curriculum as used in this chapter (and generally throughout the report) refers primarily to the subject matter, the content of the curriculum. In mathematics and science, this includes theories, facts, algorithms, concepts, methods of inquiry, and procedural knowledge. Unless the text specifically notes otherwise, this definition is not concerned with much of the paraphernalia of curriculum that turns it into instructional chunks,
INDICATORS OF CURRICULUM 121 including directions for sequencing and presenting the content. Al- though the committee recognizes that such instructional directions may also provide incentives and constraints, they ought to be matters determined as much as possible by the teacher, with whatever guid- ance is needed from other teachers and local and state curriculum specialists. The substantive content of the curriculum generally represents a joining together of many different influences: historical precedent, views of professional educators, market forces determining the sales of textbooks and related instructional aids, the wishes of parents and other interest groups in the community, recommendations by state and national bodies, and changing perceptions of what students need to know. Moreover, the curriculum is expressed in several different forms: the plans and guidelines of state and local policy makers, the content of textbooks and such other materials as related workbooks and laboratory manuals, the actual content presented to the student, and the content learned by the student. These distinctions have been widely recognized. In this chapter, the committee, following the prac- tice of lEA, refers to guidelines, textbooks, tests, and other written or programmed materials to be used for instruction as the intended curriculum; all of this material as constructed and presented by the teacher as the actual or implemented curriculum; and the content and skills learned by the student as the achieved curriculum. In the committee's view, it is important to have indicators of all three of these forms of the curriculum, since they would provide substantially different information and might be used to answer different policy questions. The intended curriculum itself takes on many different expres- sions. In a concession to practicality, the committee decided to limit the scope of our recommendations to three manifestations of the intended curriculum: (1) the content of state plans and guidelines; (2) the content in textbooks and directly related workbooks, labora- tory exercises, computer software, and other materials; and (3) the content of examinations. In a few states, for example, New York, state guidelines have always been an important determinant of the intended curriculum. As states become more active in school reform and assessment, state guidelines can be expected to play an impor- tant role in an increasing number of states. Regarding the second aspect of the intended curriculum, there is much research evidence that the content of textbooks importantly influences the content presented to the student (Goldstein, 1978; Stake and EasTey, 1978;
122 INDICATORS OF SCIENCE AND MATNEMA TICS EDUCATION Goodiad, 1984~. Therefore, an indicator of the content of textbooks needs to be part of any monitoring system for science and mathemat- ics education. As for the content of examinations, it also is believer] to influence classroom instruction to a considerable degree (Resnick and Resnick, 1985; Romberg, 1986~; hence, test content needs to be monitored an well. When the term intended curriculum is used alone in this chapter, it refers to any or all of these three levels of the formalized expression of curriculum, unless specific reference is made to a particular form, such as state guidelines. In parallel with assessing the intended curriculum, it is neces- sary to assess the implemented curriculum, the curriculum that the student actually experiences. However, this is considerably more dif- ficult. Whereas assessing the intended curriculum can be done by analyzing written materials apart from the classroom, assessing the implemented curriculum requires classroom observation. The third expression of the curriculum, the content learned by the student or the achieved curriculum, has already been considered in Chapter 4. Another question regarding the meaning of the term curriculum concerns the grade-level span over which curricula are defined: Is it a school term, a grade in school, or a longer period of time, such as all the elementary grades? The committee believes that grade-level groups have greater validity for assessment than single grades because of the interrelated nature of much of the content of mathematics and science and because of the fact that there are many ways of teaching to reach productive educational goals. Prescriptions for attainment for each year of school would generate the kinds of Tockstep curricula that constrain the creativity of teachers and are likely to lead to mediocrity. However, the committee believes that challenging, yet sensible, goals for defensible grade spans are critical for upgrading the general quality of mathematics and science education. After some consideration, the committee suggests the following curriculum blocks as useful to consider as integral units: grades K-5, grades 6-8, the high school literacy curriculum, and the high school curriculum for college-bound students. The comrn~ttee proposes that indicators for mathematics and science be developed for each of these grade clusters. Because the emphasis in this report is on mathematics and science literacy for all students, the committee views the need for assessing elementary and middle school curricula to be of the highest priority of the four areas, with the high school literacy curricula next in order of priority.
INDICATORS OF CURRICULUM 123 Some variations in grade clusters may be appropriate, for ex- ample, mathematics is often structured K-4, 5-8, ~12, as in the forthcoming standards for school mathematics being prepared by the National Council of Teachers of Mathematics. In mathematics in particular, it may be of value to overlap the curriculum blocks so as to allow for greater flexibility of topic placement; blocks rep- resenting grades K-6, 5-9, and 9-12 have been suggested at recent international meetings of mathematicians and mathematics educa- tors. There generally are two major options for the high school mathematics and science curricula for college-bound students: one for students expecting to major in mathematics and science-related fields and one for students expecting to major in other fields, another variation that should be considered in constructing the curriculum frameworks proposed below. Indicators for Whom? A second issue that concerned the committee had to do with the audience for the indicators of curriculum quality and their use by that audience. It is one thing to design a way of assessing the scientific quality of a textbook for a committee of scientists. It is another thing to design a way of capturing the quality of the science curriculum in a state for a state legislator with little science back- ground. The committee concluded that the ultimate audience for its work should be federal, state, and local policy makers responsible for thinking about the overall quality of the educational program under their jurisdictions, even though specific judgments on the quality of the science or mathematics being taught will be made by scientists and mathematicians. Indicators should be developed to allow policy makers to address the following kinds of questions: . How much attention is paid to complex problem solving by the schools in our state? Has this changed over time? Is it more or less than in other states? ~ Do some kinds of children receive more mathematics content than others? By race? By social class? By sex? . Do children in our schools receive as comprehensive an imple- mented science (or mathematics) curriculum as children in schools in Japan? . What relationship do the state curriculum guidelines have to the actual instruction that goes on in the state?
124 INDICATORS OF SCIENCE AND MATHEMATICS EDUCATION . Have the recent state reforms in education changed the con- tent and nature of science and mathematics education in our state and other states? What Kinds of Inclicators? There is little question about the importance of the content of the implemented curriculum in determining the achievement of stu- dents in mathematics and science. At the most superficial level, it seems clear that few students would learn anything about geometry or the conservation of energy, for example, unless they received sys- tematic instruction. One reason that Japanese 13-year-olds outscore their U.S. counterparts in mathematics is that all Japanese students are exposed to a year of algebra in seventh grade, while most U.S. students have to wait until ninth grade (McKnight et al., 1987~. At a more detailed level, there is a solid literature that relates the teaching of particular concepts, knowledge, and skills to their ac- quisition by students (Walker and Schaffarzick, 1974; Wolf, 1977; Peterson, 1979; Romberg, 1986~. lEA's second international math- ematics study found that, in the United States, eighth-grade math- ematics students are typically placed in one of four kinds of classes (remedial, typical, enriched, aIgebra). The amount covered is low- est in the remedial classes, about 25 percent greater in the typical classes, and another 25 percent greater again in the enriched classes. (Algebra classes were not included in this analysis because of their entirely different content.) The achievement gains of the students in the four types of classes correspond directly to the amount covered in the classes (Crosswhite et al., 1985~. The extent of variability among curricula in content coverage, even given presumed variabil- ity in student ability, may well foreclose the possibility of attaining desirable levels of student achievement for some student populations (McKnight et al., 1987~. A description of the content coverage of a curriculum is only a beginning; in addition, descriptors of curriculum quality are needed. After all, topics can be included in a curriculum briefly or superfi- cially. At one level, it seems obvious that students will have a better chance of learning something if sufficient time is allocated to learning it. This is the driving notion behind some instructional approaches, such as mastery learning (Bloom, 1976; Brophy and Good, 1986~. Similarly, if a concept is introduced a number of different times during the school experience of a child, in different contexts and in
INDICATORS OF CURRICULUM 125 increasing complexity, it is more likely to be well learned. This strat- egy leads to an approach the spiraled curriculum favored by many science curriculum specialists. (See, however, warnings by McKnight et al. t1987] against a poorly implemented spiral curriculum that can lead to shallow repetition of topics and attenuation of the cur- riculum.) At present, there is an emerging literature that relates the depth of coverage of subject matter to student understanding of the content (Glaser, 1984; Sizer, 19843. Deeper, more complex coverage of a concept or set of concepts increases the opportunity for students to be engaged in effective complex problem solving (Chi et al., -1981; Resnick, 1987~. Not surprisingly, these researchers have also found that people's capacity to understand and remember new information in an area is related to their prior level of understanding of the area, and that experts in a field approach the solution of problems differ- ently and more efficiently than do novices. This discussion suggests that the depth of coverage of material in a curriculum is an impor- tant aspect of its quality and needs to be assessed, in addition to the assessment of the extent of concept coverage. The quality of a mathematics or science curriculum is influenced by two other factors: the scientific and mathematical accuracy of the content and the pedagogical logic or way it is presented. Cur- ricula act as unwelcome constraints on the teacher's effectiveness to the extent that they embody inaccuracies, inadequate explana- tions, or poor sequencing of concepts or when they misrepresent the methods of science and mathematics, for example, by presenting sci- entific inquiry as a dogmatic and rigid procedure. No matter how comprehensive or deep the coverage of a content area, there will be little gained if it is confusing or inaccurate. Similarly, materials that are poorly organized or sequenced or exhibit other poor pedagogic strategies also constrain the teacher's ability to present the subject well. This suggests that an assessment of curricular quality needs to address the mathematical or scientific accuracy and the pedagogical quality of a curriculum, in addition to its depth. To summarize, the committee suggests the development of two types of measures to capture and assess the content of mathematics and science curricula: measures of the extent of content covered in the curriculum and measures of quality including the depth of coverage of the content in the curriculum, the scientific or mathematical accuracy of the content of the curriculum, and the pedagogical quality of the curriculum. Measures of these two types should be developed for both the intended curriculum and the implemented curriculum.
126 INDICATORS OF SCIENCE AND MATHEMATICS EDUCATION Indicators to assess the kinds of policy questions set out earlier should be developed from these measures. For example, to assess breadth of coverage throughout a set of schools, the ratings of the different textbooks being used could be weighted by the number of students using each textbook. MEASURES OF CURRICULUM CONTAINED IN OTHER CHAPTERS There are clear relationships between the topics of concern treated in this chapter and several topics discussed in earlier chapters. One crude way of assessing coverage of content in high school, for ex- ample, is to measure the number of mathematics and science courses taken by a student, as recommended in Chapter 5. As noted there, a slightly more sophisticated strategy is to use the information in course titles; thus, one might expect a student to be exposed to more algebra content In a course called "Algebra" than in a course called "General Mathematics." A variety of analysts have related course- taking to individual achievement and found consistent and important relationships, independent of other measured student characteristics such as prior achievement and social class (see Ralzen and Jones, 1985; Rock et al., 1985~. These effects of course-taking swamp the effects of variables such as sex, race, public or private schooling, and teacher characteristics. Using the approach of logging the number of courses taken, the National Longitudinal Survey of 1972 and the High School and Beyond Survey of 1980 have provided data for very crude national or state indicators of content coverage in mathematics and science (National Center for Education Statistics, 1981, 19843. Because participation in high school courses is often up to students, the development of this indicator is discussed in Chapter 5. In elementary school, the analogous measure to course-taking is a measure of the time devoted by the teacher to instruction in mathematics or science. This might be expressed on an absolute scale, such as number of minutes, or on a relative scale, such as percentage of the school day. Each would supply somewhat different information. A national or state indicator would require aggregating information across a representative sample of classrooms. Sometimes the nature of this information can be shocking: the most recently available survey of the time elementary school teachers (K-3) spend teaching science revealed that the average time per week was 17 min- utes (Weiss, 1978~. Also surprising is the variation from class to class
INDICATORS OF CURRICULUM 127 in the amount of time spent on mathematics, generally considered a core subject in elementary school from 23 to 61 minutes per day in two different fifth grades (Berliner, 1978~. Because the amount of time allocated to mathematics or science instruction in a classroom is often the choice of the teacher, this measure is discussed in the chapters on both student behavior (Chapter 5) and teaching quality (Chapter 6~. Once measures of the coverage of the actual curriculum are developed, measures of course-taking and time spent in instruc- tion in mathematics and science might become superfluous. In the meantime, however, these measures are useful and relatively easy to gather. A final, related measure discussed in Chapter 5 is the amount of time students spend on mathematics and science homework. A substantial body of literature finds that the careful use of homework enhances the learning of students; the amount of homework, the way that it is treated by the teacher, and its relationship to the curriculum all influence its effects (see Walberg, 1984; Redden and Jones, 1985~. Because homework is an expression of student behavior but also strongly affected by the teacher, this measure is first discussed in Chapter 5 on student behavior but further amplified in Chapter 6 on teaching quality. DEVELOPING INDICATORS OF CONTENT COVERAGE This section describes the approach that we recommend for de- veloping indicators of content coverage. Succeeding sections consider indicators of curriculum quality including content depth, scientific accuracy, and pedagogical quality. Curriculum Frameworks We envisage the development of indicators for content coverage as starting with the development of exemplary curriculum frame- works. The frameworks would be intended to capture an "ideal" conception of the curriculum. They would be designed by a na- tional group or groups; a variety of science organizations have the expertise and have produced curriculum recommendations over the years (Harms and Yager, 1981; American Chemical Society, 1984; Joint Committeee on Geographic Education, 1984~. Also relevant may be state guidelines (California Department of Education, 1984; South Carolina Department of Education, 1986; Virginia Depart- ment of Education, 1986) and curricula from other countries (Klein
128 INDICATORS OF SCIENCE AND MATNEMA TICS EDUCATION and Rutherford, 1985; Travers et al., 1985~. The mathematical com- munity has been particularly active in thinking through the content of the school mathematics curriculum, as exemplified by the efforts of the Conference Board of the Mathematical Sciences (1983), the Mathematical Sciences Education Board (1987), and the National Council of Teachers of Mathematics (1987~. California, Illinois, and Wisconsin also have constructed detailed frameworks for mathemat- ics instruction in their school districts. The objective envisaged by the committee would be to have a single national framework by grade cluster for each subject without dictating the placement of specific topics. Revised on a regular basis to reflect changes in the subject and advances in pedagogy, the frameworks would act as templates against which the content of existing and planned curricula could be matched. They would operate as a standard that is, when a curriculum was mapped onto the framework, its content could be expressed in a measure representing the comprehensiveness of coverage of the curriculum. The measures might be expressed as a percentage of the content coverage represented by the framework. Taken alone, such a measure might be of some use to local and state policy makers. For example, suppose that local school officials wanted to buy a new K-5 textbook series and related materials in mathematics. They might use the measures of coverage of different textbook series in making their decision. Or, if a particular textbook series with limited content coverage were widely used in a certain state, the measure of content coverage might help explain to a state legislator why the children of the state scored badly on mathematics achievement tests compared with children in other states, although such an inference might or might not be correct. Once measures of the content coverage of textbooks are devel- oped for the major textbooks, for example, they could be weighted by the number of students using the textbooks to develop aggregate measures of content coverage for a state or local district, or even for the nation. Such aggregate measures might help a state legislator somewhat more in the quest to understand the low scores in the state. Moreover, if the content of the tests used in the states were also matched against the framework, then the content of the text- book series and the content of the tests would be expressed in the same nomenclature and a measure could be developed of the degree of overlap between the two. This might go even further in explaining the differences between the states in their test scores. As noted, a
INDICATORS OF CURRICULUM 129 comparison of different eighth-grade curricula and their match with the content of the lEA tests explained much of the differences in results on the lEA tests (Crosswhite et al., 19853. Finally, a state legislator might want to know how the state's guidelines compared with the framework. Perhaps more important than measures of the content coverage of the intended curriculum would be measures of the coverage of the content of the actual curriculum taught to students. If such measures were also available, the state legislator could compare the degree of match between the content of the textbook curriculum, the test, and the actual curriculum. If the content of the textbook and the test matched and they both fit with the state guidelines but the actual curriculum did not match, perhaps the suggested policy would- be to increase teacher in-service education rather than to change the textbooks or tests. And, as another example, if the content of the state guidelines were mapped against the framework, analyses could be carried out of how fan' hfu] the content of the textbooks and the actual curriculum were to the wishes of the state policy makers who developed the guidelines. Establishing Subject-Matter Frameworks The usefulness of comparisons of curriculum materials with ex- emplary frameworks is directly related to the quality of these frame- works. We recommend that the effort to develop such frameworks be started immediately, be well funded, and not be hurried. If de- veloped in the way that we propose, they would serve as touchstones (or ideal descriptions) for the development of future guidelines, text- books, and tests. They deserve the best effort that the nation's scientific and educational community can give, including a continual process of review and revision. In priority order, the committee recommends first establishing frameworks for science in grades K-5 and 6-8. These two areas of schooling require immediate attention. It is suggested that this be followed with establishing frameworks for mathematics in grades K-5 and 6-8 (or K-4 and 5-8), then with frameworks for science literacy and mathematics literacy in grades 9-12, and finally with the frame- works for college-bound youth in mathematics and science, grades 9-12. An important caveat needs to accompany these suggestions:
130 INDICATORS OF SCIENCE AND MATHEMATICS EDUCATION optimally, instruction in science and mathematics ought not be sep- arated in such a rigid manner, quite the contrary. Particularly in the lower grades, an integrated curriculum would be highly desirable. Constructions of such curricula, however, need to build on efforts to define in some detail the substantive core of subject matter from each discipline that is appropiately taught preferably in a related, if not integrated way at the given grade levels. The frameworks must represent the structures of the subject matter and desirable learning goals, or alternatives among desirable goals. The frameworks should meet some general criteria: they should array, in a two-dimensional or more complex formal, major processes, emphases, or principles in the curriculum against content topics, rather than simply list detailed topics; they should represent the best thinking of a combination of disciplinary specialists and specialists in the design of curricula and in teaching the subject; they should be conceived to "lead" practice, rather than representing a least common denominator of current practice; and they should be flexible, presenting a commonly agreed-on core and allowing for major options or alternatives in the content presented in states, localities, schools, and classrooms. The core should represent a detailed explication of the standards for scientific and mathematical literacy set out in Chapter 2, adjusted for the appropriate grade levels. Over time, these frameworks should be regularly and critically revisited, so they reflect developments in the discipline, in pedagogy, and in the nation's aspirations for its youth. It seems reasonable to expect that a major review of each of the frameworks be made every decade. As noted, work on the frameworks could build on existing efforts. In mathematics, for example, a framework based on the taxonomy developed by Romberg (1983) might have three dimensions: one con- cerned with the activities common to all mathematics, one with the specific processes entailed in doing mathematics, and one with the conceptual strands of mathematics that represent the historical de- velopment and core of the field. Romberg lists the essential activities common to all mathematics as: . abstracting (i.e., dealing with quantitative relations and spa- tial forms to the exclusion of all other properties of objects); inventing (e.g., dealing with complex tasks with nonobvious solutions, making guesses or assertions and then demonstrating their logical validity);
INDICATORS OF OURRICULUM 131 . proving (using fundamental concepts to deduce a theorem through logical argument); and ~ applying (using mathematics in the sciences, in engineering, in business and industry, in private and social life). Four basic sets of processes involved in doing mathematics need to be included in the curriculum: ~ relation processes-describing, classifying, comparing, order- ing, separating, grouping, and partitioning; . representation processes going from the concrete to the ab- stract (or from the abstract to the concrete) in solving problems; . symbolic-procedure processes for example, the common al- gorithms learned in elementary school; and . validation processes- carried out through empirical or logical deductive determinations. The seven strands of substantive content suggested by Romberg for the core curriculum are: . whole numbers arithmetic counting, addition and subtrac- tion, and multiplication and division; spatialrelations basic concepts of geometry; measurement relating numbers to geometry; . fractions extending the concept of number from whole num- bers to fractional numbers; . coordinate geometry procedures for assigning numbers to · . Points In any space; and .. . . . algebra dealing with abstractions from concrete numbers; . statistics bridging the world of mathematics and the world of practical problems through data analysis and the interpretation of various forms of data collection and display. Often added to these strands are two more topics (see, for example, Conference Board of the Mathematical Sciences, t1983~: . discrete mathematics basic combinatorics, graph theory, discrete probability and . . iteration. · computer science programming, introduction to algorithms, The advent of the personal computer and related technology makes it particularly critical to rethink the mathematics curriculum at this
132 INDICATORS OF SCIENCE AND MATHEMATICS EDUCATION time. It calls for the introduction of such new subject matter as computer science; it makes possible new ways of teaching traditional mathematics; and it forces reconsideration of the place of many traditional topics and instructional strategies. The computer has materially changed the ways in which mathematics is being done. Therefore, its use is integral for teaching the kinds of curricula being suggested by mathematicians and mathematics educators. The development of K-5 science curricula is far behind efforts in mathematics. The committee is particularly concerned about this area, since the foundation for scientific literacy must be laid in those years. Several states are now taking steps to remedy the virtual absence of science teaching from elementary school. An example is the Oregon Framework for Science Programs (Northwest Evaluation Association, 1986~. The main components of this framework are: . Scientific concepts (e.g., cause-effect, change, cycle) . Scientific problem-solving and inquiry processes (e.g., classi- fying, hypothesizing, inferring) Applications of hand skills (e.g., measuring, constructing) Interests in science (e.g., scientific avocation, confidence) Values that underlie science (e.g., questioning, searching for data, considering consequences) . Interactions between science and society (e.g., science's influ- ence on society, limitations of science) bilistic) Characteristics of science (e.g., tentative, replicable, proba These components encompass the latter three dimensions of scien- tific literacy (see Chapter 2). An adequate framework would also need to include a common core and optional components of factual knowledge, concepts, and theories representative of the first dimen- sion of the literacy model, called in Chapter 2 "the nature of the scientific world view." (See, for example, the longer-term effort of the American Association for the Advancement of Science t1985b] to examine what science and technology is most worth learning.) Obtaining Measures of Content Coverage Once the exemplary frameworks are established, the next task is to map the content of the various exemplifications of curricu- lum onto the framework to derive measures of content coverage. For the most part, this rating task would have to be carried out jointly by
INDICATORS OF CURRICULUM 133 expert scientists and educational experts. As suggested in Chapter 3, a considerable amount of training should be given to the raters, and acceptable levels of coding reliability should be established. In par- ticular, there may be legitimate reasons for giving different weights to the coverage of different topics and to the emphasis on facts, princi- ples, and procedural knowledge. These reasons should be made quite explicit ahead of time; weights should be agreed to by any particular panel before rating begins or spelled out by individual raters; both the process of rating and the weights used in the application of a particular framework should be described in detail in reporting the panel's findings. State Guidelines The task of mapping the 50 state guidelines onto any framework seems relatively straightforward. Indeed, in a number of states, it would be a trivial task since very sparse, or in some cases no, guidelines exist for some of the curriculum areas. One issue that would arise would be what document to take as the state guidelines if multiple sets occurred in legislative, regulatory, and subregulatory material. The answer to this would have to be worked out on a state-by-state basis. It would also be useful to map the national guidelines of countries such as Japan, West Germany, and France, and the guidelines of the provinces of Canada for later purposes of comparison. Textbook Series The task of mapping the content of textbook series and their related materials (laboratory exercises, computer software, films, workbooks) would be more tedious but straightfor- ward. One decision here would have to do with which textbooks to assess. Two types seem important for national purposes: textbooks that are widely used and textbooks that are reputed to be exemplary themselves. States and local districts may wish to select textbooks and ancillary materials for mapping that are being considered for adoption or local purchase. Tests Similarly, mapping test content onto a framework and deriving measures of coverage appear routine. Again, we suggest that frequency of use and reputation determine the tests to be chosen for analysis at the national level. Tests being considered for use at the state or local level should be chosen for analysis at these levels.
134 INDICATORS OF SCIENCE AND MATHEMATICS EDUCATION Implemented Curriculum The methodology for assessing the curriculum as actually implemented is more problematic. One way to do this would be to observe classes, an expensive and time-consuming method. Some members of the committee believe that teachers could supply adequate information. Presented with a list of items in the subject, a sample of teachers would be asked whether they covered the topic; items in the list would be drawn purposefully from the content framework but would be presented to teachers as an unorganized list. Teachers would be asked if they covered the topic that year, whether it was covered previously, or whether it is not covered at all in their school. This is similar to the approach used to assess the "opportunity to learn" measures in the lEA studies (Crosswhite et al., 1985; Jacobson, 1985) and by several investigators studying classroom processes (Barr, 1985; McLean, 1985~. (For a description of this methodology, see Raizen, 1987.) A simple matrix sampling design would make it possible for each teacher to respond to only a few of the questions regarding the coverage of subject matter in his or her classroom and yet make possible estimates of coverage for the total framework. Other members of the committee agreed that observation would be expensive but believed that teachers might not respond accurately or might forget what they had or had not taught. Research needs to be carried out to establish the validity of the approach of using teacher-reported information by conducting cross-checks with ciass- room observation conducted by outside observers. Research based on classroom observation could also probe the current ambiguity surrounding the meaning of "coverings a topic or subject. Frequency of Mapping The mapping of state guidelines, text- book series, and test areas would require periodic updating as the content is changed or new materials are developed. The sampling of the actual content of instruction should be carried out nationally at least every four years on a cycle that is synchronized with the cycles for student assessment, so that the resulting indicators could be used together. Recommendations Research and Development: In order to develop indi- cators of breadth of content coverage in the science and
INDICATORS OF CURRICULUM mathematics curriculum, the committee recommends that exemplary frameworks be constructed for the following cur- ricuTum blocks: grades K-5 science, grades K-5 mathemat- ics, grades 6-8 science, grades 6-8 mathematics, grades 9- 12 literacy in science, grades 9-12 literacy in mathematics, grades 9-12 science for college-bound students, and grades 9-12 mathematics for college-bound students. The frame- works for grades K-5 and 6-8 science should be accorded the highest priority. 135 The frameworks must represent the structures of the subject matter and desirable learning goals, or alternatives among desirable goals. Key Indicator: Once the frameworks are constructed, the committee recommends that three elements of the intended curriculum should be matched and rated against them for content coverage: state guidelines, textbooks and such as- sociated materials as computer software and laboratory ex- ercises, and tests. The frameworks should also be used to analyze the content coverage of the implemented curriculum (i.e., the content presented to the student as reported by classroom teachers). The ratings obtained through analysis of the three elements of the intended curriculum and analysis of the implemented curriculum will provide the raw material for the construction of indicators of content coverage. The ratings should be carried out every four years at the national level in synchronization with the student assessments recommended in Chapter 4 so that the indicators can be used to- gether. Ratings could be aggregated in different ways for different uses and different policy makers. Research and Development: The committee recommends that research be carried out to establish the validity of teacher-reported information regarding content coverage in the classroom.
136 INDICATORS OF SCIENCE AND MATHEMATICS EDUCATION DEVELOPING INDICATORS OF CURRICULUM QUALITY All three dimensions of curriculum quality- depth of topic treat- ment, scientific accuracy, and pedagogic quality present difficult problems of measurement since they do not lend themselves to the kind of detailed analysis suggested for assessing breadth of coverage. The judgment of experts is required. For each of these quality dimen- sions, we discuss below why and how it should be assessed, together with some implications for developing assessment criteria. Depth of Treatment Discussions of science education have emphasized that a major goal is to give students a basic understanding and appreciation of the structure of, say, physics as a scientific discipline, the process of doing physics, and some of the complex problems solved and created by its applications. The ability to pass tests emphasizing the recall of science facts is not a sufficient foundation either for general scientific literacy or for further study. Therefore, the curriculum needs to concentrate on a limited number of topics to be studied in depth, in contrast to the makeup of many textbooks that, with every revision, add more topics to an already overburdened curriculum (Hurd et al., 1981; Taylor, 1984~. The topics ought to be carefully chosen so that their presentation forms a coherent body of knowledge. Not only will this enhance learning with understanding; research suggests that in-depth study of particular topics as well as the use of laboratory or hands-on experiences are related to the engagement of students and to their interest in a course (Harms and Yager, 1981~. Arguably, positive experiences in science classes engendered by these instructional strategies influence later interests and involvement in science. The burden of this argument is that frameworks need to ac- commodate judgments on depth of coverage as well as breadth of coverage. How might this be accomplished? The most important requirement is that depth of coverage be an explicit evaluation cri- terion. Then, once a framework is in hand and the tasks of mapping of content coverage are complete, sets of judgments on the depth of coverage of text and other materials (or of reported or observed classroom practices) can be made depending on the weights assigned by the judges to the importance of various topics, concepts, and processes.
INDICATORS OF CURRICULUM 137 One sort of measure that might be used to assess depth of treat- ment is the number of pages devoted to a topic in a textbook, number of items on a test, or the amount of instructional time suggested in state guidelines or reported by teachers. Admittedly, these may turn out to be superficial measures; they certainly would need validation. It may well be that different judges might differ on the weights to be assigned to the treatment of particular topics and even to the need for broad coverage as contrasted to the depth of coverage of key topics and concepts. Scientists and science educators may place greater emphasis on depth as contrasted to the extent of coverage than do state and local authorities charged with developing overall curriculum guidelines that need to be endorsed by practicing edu- cators and politicians. This underscores the importance of making explicit the weights assigned to various topics, concepts, and process skills by different expert groups. Ratings of depth of treatment should be constructed for all three elements of the intended curriculum state guidelines, texts and as- sociated materials, and tests as well as for the content actually presented to the student. As with the analysis of extent of con- tent coverage, the latter will require special surveys of teachers and students supported by classroom observation. Scientific Accuracy and Pedagogic Quality The assessment of scientific accuracy of the content of the in- tended curriculum (state and local guidelines, textbooks and associ- ated computer software and laboratory materials, and tests) would appear to be relatively straightforward. Panels of scientists could be convened periodically to review the content of these materials to en- sure scientific accuracy. Optimally, these judgments would be made in conjunction with the ratings of materials for content coverage and depth, so that information on all three factors regarding a particular textbook or test would become available simultaneously. Assessing the scientific accuracy of the content of the imple- mented curriculu~what the students actually receive in class is much more difficult, for the reasons already stated. Classroom ob- servation may be an appropriate tool, but at best it could provide information for only a limited number of classrooms. Another ap- proach may be to establish some sort of threshold: minimally, one would expect a teacher to have the subject-matter knowledge neces- sary to teach the content defined by the framework at a particular
138 INDICATORS OF SCIENCE AND MATHEMATICS EDUCATION level or for a particular subject. Thus, performance on the teacher tests of subject-matter competence recommended in Chapter 6 might be considered a minimum by which to judge the scientific accuracy of instruction, particularly if these tests were to be based on the con- tent of the relevant framework. If tests of student learning were also based on the framework, it would be reasonable to expect teachers to perform well on the very same tests as a necessary, if not sufficient, indicator of subject-matter competence. Similar problems arise in assessing pedagogic quality as in as- sessing scientific accuracy, even though the aspects of the curriculum to be judged are different. As to the intended curriculum and its components, panels of relevant experts could judge their pedagogic strengths and weaknesses: the appropriateness of the instructional strategies, given the subject matter and the grade level; the design and sequencing of topics to be taught; consonance with what is known about learning various scientific or mathematical constructs, processes, and skills; specific approaches to learners with different backgrounds and interests; and the like. Indicators of the pedagogic quality of actual classroom practice, however, would be difficult and expensive to obtain. The limitations of teacher surveys and cIass- room observation already discussed apply to this area with even more cogency, since classroom practice is more difficult to distill and de- scribe succinctly and teachers' professional competence is involved. Moreover, as pointed out in Chapter 6, there are additional obstacles in the relative lack of consensus on best pedagogic practice, despite years of research on teaching effectiveness (e.g., Darling-Hammond and Hudson, 1986~. Despite a considerable body of work, little of it has focused on the teaching of specific subject matter, at spe- cific grade levels, to student populations with specific competencies, even though it is probably the case that effective teaching strate- gies are closely linked to context. One would hesitate at this time even to suggest some sort of test of pedagogic competence. Perhaps the follow-up work to the reports by the Holmes Group Consortium (1984), which recommends a year of professional education after graduation with a liberal arts degree, and by the Carnegie Forum on Education and the Economy (1986), which will aim to create a national board for teaching standards and teacher certification, will help build the needed understanding and consensus on what peda- gogic knowledge teachers need to have and be able to orchestrate in given settings.
INDICATORS OF CURRICULUM Developing Criteria for Assessing Quality 139 We have suggested above that criteria for judging curriculum quality would be developed by the different panels of experts as they assess the depth with which topics are treated, the scientific accuracy of the content, and the pedagogic strategies used in presenting it. As experience with such judgments accumulates, criteria can be expected to become more finely honed. An approach to establishing standards that would facilitate the work of the panels is the analysis of high-quality programs by sci- entists and educators with a view toward providing models of excel- lence. First, outstanding science and mathematics programs would be selected, somewhat in the fashion of the Focus on Excellence series (National Science Teachers Association, 1983-1984), but in a more systematic manner to cover adequately the several curricu- Jum blocks from grades K-S through high school. The programs would then be described in some detail, with particular attention to the three quality dimensions. In preparation for selecting candidate programs and developing the descriptions, professionals (scientists, science educators, teachers, cognitive researchers) would be surveyed for judgments on the characteristics of a high-quaTity curriculum. Through the suggested selection and analyses, the characteristics of acknowledged high-quality programs would be made explicit and perhaps synthesized to provide several models. As the models and descriptions of their quality characteristics became available, panels could use them as a basis for creating criteria in assessing depth of treatment, scientific accuracy, and pedagogic quality. The impor- tance of this strategy is that it would encourage panels to base their judgments on leading curricula rather than on the average content of science and mathematics instruction. Recommendations Research and Development: Standards of exceldence should be developed based on the best of curricula in current use. High-quality programs encompassing the curriculum blocks sug- gested above should be selected, profiled, and analyzed to provide models of excellence in depth of content coverage, scientific accuracy, and pedagogic soundness of science and mathematics curricula.
140 INDICATORS OF SCIENCE AND MATHEMATICS EDUCATION :Key Indicator: The quality of the curriculum should be assessed by expert panels along three dimensions: depth of content treatment, scientific accuracy, and pedagogic sound- ness. Ratings for each of these quality dimensions should be assigned to the three elements of the intended curriculum (i.e., state guidelines, texts and associated materials, and tests). Assessments regarding depth of treatment should also be made of the implemented curriculum through teacher and student surveys and classroom observation. To assess the depth of content treatment, the frameworks devel- oped according to the recommendation made above should be used to identify the critical topics that constitute a coherent curriculum. Weights assigned by each rating panel regarding the depth of treat- ment desired for a given topic must be made explicit in reporting results. The assessment of the scientific accuracy of the intended cur- riculum should be carried out by scientists in the relevant disciplines. The scientific content of the frameworks should be used to construct the tests of teacher competency of subject matter recommended in Chapter 6 and such tests used as a minimum measure of the scientific accuracy of the actual curriculum experienced by students. Research and Developn~ent: The committee recommends research to provide validity checks on the standards being used to assess depth of treatment, scientific accuracy, and pedagogic soundness of science and mathematics curricula. For example, research should be undertaken to establish what pedagogic knowledge teachers need to have and need to know how to use in order to teach science or mathematics effectively to students of different ages, backgrounds, and competencies. IMPLICATIONS FOR STATE EDUCATION AGENCIES Many state education agencies, for example, California (1984) and South Carolina (1986) in science and Texas in mathematics, are moving toward the curriculum framework concept described in this chapter. The Council of Chief State School Officers, representing
INDICATORS OF CURRICULUM 141 all the state superintendents, has implicitly moved as an organiza- tion to a national framework by its endorsement of a state-by-state assessment system scheduled to be implemented in 1989 (Council of Chief State School Officers, 1984~. Thus, a national framework could have an important function in making possible comparison and evaluations of the content of various state assessment tests in specific subjects. The concept of a commonly agreed-on curriculum core allow- ing flexibility for alternatives reflects the commonality that really exists among schools, but preserves the cherished local and state freedom from federal curriculum control. The distinction between a "national" curriculum framework and a "federal" one is critical to states and localities that is, the distinction between a set of guide- lines developed by one or more nationally recognized groups and a prescribed course of studies mandated by a central authority. Be- cause the proposed frameworks would be developed and applied for assessing curriculum content not just within a grade level or course, but over a reasonable period of schooling (e.g., the intermediate grades), there could be latitude regarding the sequencing of units. For example, a core topic might be taught in either sixth, seventh, or eighth grade. The framework concept would lead to a national grid of science and mathematics subject matter that would identify key concepts and processes to be included in the curriculum, but not the exact placement. When such frameworks are developed, they can serve as a guide for a review of state level analyses including: ~ Equal educational opportunity for all students regardless of socioeconomic status, location, race, ethnicity, or gender. Without accurate information on the breadth and depth of curriculum cov- erage as well as the variability among school systems, the first state indicator of a problem may be the number of small, rural school valedictorians who have to complete remedial mathematics courses before being accepted by the state's four-year colleges or the number of urban minority high school graduates who score at low levels in mathematics ~d science college placement tests. Guides for textbook analyses. Profiles matching the content analyses of textbooks with the appropriate framework could inform state and local decision makers about how well the textbooks will assist teachers in meeting the intended curriculum. ~ Foundation for test development. The proposed curriculum frameworks could define the parameters for what a state believes all r--- .
142 INDICATORS OF SCIENCE AND MATNEMA TICS EDUCATION students as well as the college-bound should know and be able to do as a result of their school experience in mathematics and science. Therefore, the frameworks could serve as the basis for a common understanding of what should be tested in a diagnostic way at the local school level and in a broader snapshot sense at the state level. At that point, the test results could serve as a basis for evaluating not only curriculum implementation but also whether the testing is sensitive enough to assess accurately the elements of the framework. State and school district policies affect to some degree each ciass- room teacher's decisions on the implementation of a mathematics or science curriculum. The approach recommended by the committee provides a foundation for individuals at all levels to make informed decisions about what is working in the curriculum and about future directions.
