Below is the uncorrected machine-read text of this chapter, intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text of each book. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.
Summary and Recommendations The Committee on Indicators of Precollege Science and Mathe- matics Education was established by the National Research Council to develop indicators of the condition of science and mathematics education in the nation's schools. The committee's first report con- centrated on conventional indicators. In this report, the committee makes recommendations for improved ways of monitoring the condi- tion of education in these critical fields. Our recommendations are based on two premises: . All students need the knowledge and reasoning skills that good science and mathematics education provides. Not only should students leave school scientifically and mathematically literate, but they should also have acquired the mental tools with which they can renew that literacy throughout their lives. . What teachers and students do in schools determines how much learning takes place. Student and teacher behaviors are influ- enced by a variety of incentives and constraints. Among the many influences on behavior are curriculum mandates and curricular sup- port materials, working conditions for teachers, and resources at the classroom level. This simplified model of the educational process provides the framework for our report. This chapter presents the committee's recommendations. Sup- port for the recommendations, definitions of what the committee 1
2 INDICATORS OF SCIENCE AND MATNEMA TICS EDUCATION means by high-quality education and by the term indicators, and caveats about the interpretation of indicators are provided in the subsequent chapters. INDICATORS OF SCIENCE AND MATHEMATICS EDUCATION The committee makes three kinds of recommendations. We rec- ommend a number of key indicators, to which we assign the highest priority. We recommend supplementary indicators, which, although of lesser importance, would also improve knowledge of the quality of mathematics and science education. The key indicators we recom- mend are listed below, and the supplementary indicators are listed on page 4. We also make a number of proposals for research, either to validate the recommended indicators or to lead to the development of additional indicators. For two of the key types of indicators dealing with assessment of student learning and assessment of curriculum content important development work needs to be done before they can become useful in- dicators. For the assessment of learning in mathematics and science, tests and exercises need to be developed that will allow assessment of conceptual knowledge, process skills, and higher-order thinking in addition to the factual knowledge and skills assessed by tests in cur- rent use. For the assessment of curriculum, exemplary frameworks containing substantive content and desirable learning goals need to be constructed, each spanning several grade levels, to provide refer- ents against which textbooks and other curriculum components can be evaluated. Recommended Key Indicators Extent of student learning in mathematics and sci- ence Extent of scientific and mathematical literacy of adults Enrollment data for mathematics and science courses taken by students in high school and the amount of time spent on the study of science and mathematics in elementary and middIe/junior high school Nature of student activities during science and math- ematics instruction
SUMMARY AND RECOMMENDATIONS 3 Extent of teachers' knowledge in the subject matter that they are expected to teach Salaries paid to college graduates with particular subject-matter specialties who choose to enter var- ious occupations Quality of the curriculum content in state guidelines, textbooks and associated materials, tests, and actual classroom instruction in science and mathematics through matching to exemplary curriculum frame- works along four dimensions: breadth and depth of treatment and scientific and pedagogic soundness For each key indicator, unless otherwise noted in a specific rec- ommendation, data should be collected in four-year cycles. For in- dicators dealing with student learning, student behavior, teaching effectiveness, and quality of the curriculum, information should be collected and analyzed so it can be reported by student subgroup- that is, data should be aggregated not only by region (or state, if current efforts in that direction proceed), but also by students' age or grade level, gender, race, ethnicity, socioeconomic status, and type of community (urban, suburban, rural). The reason for aggregating by student demographic variables is to establish to what extent there are systematic inequities in the distribution of resources devoted to science and mathematics education and systematic differences in student learning. Some of the proposed indicators are most appropriate at the national level, for example, assessment of the scientific and math- ematical literacy of the general population. Others may be most policy-relevant at the school level, for example, the information oh tained by observing student activities during instruction. Still others are relevant at the national, state, and local levels, for example, assessments of student learning or teacher knowledge of subject mat- ter. When appropriate, the recommendations note the policy level for which an indicator is intended. Recommendations for research are addressed both to the research community and to those fund- ing agencies concerned with better understanding and monitoring of science and mathematics education.
4 INDICATORS OF SCIENCE AND MATHEMATICS EDUCATION Recommended Supplementary Indicators Amount of time spent on science and mathematics homework Teacher preparation college courses in mathematics and science, majors and minors, advanced degrees Teachers' use of time outside the classroom spent on professional activities related to their teaching of mathematics and science Materials, facilities, and supplies available and used by teachers in mathematics and science instruction Level of federal financial support for science and mathematics education Commitment of resources by scientific bodies for the improvement of mathematics and science education in the schools In the chapters that follow, our recommendations for improving indicators in current use and developing new ones appear within the appropriate domain of science and mathematics education: student learning (Chapter 4) and student behavior (Chapter 5), teaching quality (Chapter 6), quality of the curriculum (Chapter 7), and financial and leadership support (Chapter 83. A consequence of this organization is that recommendations for key indicators are intermingled with recommendations for supplementary indicators and with recommendations for research. The remainder of this chapter presents all the recommendations with amplifying material as they appear in the report, spelling out the recommended key and supplementary indicators in greater detail as well as needed research. The recommendations are organized by domain of mathematics and science education, as they are in the chapters of this report. RECOMMENDATIONS Indicators of Learning in Science and Mathematics Indicators of student learning at the national, state, and local levels should be based on scores on tests that are consonant with the
SUMA£4RY AND RECOMMENDATIONS 5 curriculum and all major curricular objectives, including the learning of factual and conceptual knowledge, process skills, and higher-order thinking in specific content areas. Measuring progress toward this last objective is especially important, since it is possible that pres- sures on school practitioners to increase student scores on multiple- choice tests emphasizing recall of factual information may result in diminished attention paid to the development of higher-order think- ing skills. In order to establish how well major curricular objectives are being met, test items used to assess students' mathematics and science learning should not be exclusively in a multiple-choice format. A significant number of items using an open-ended pencil-and-paper format and a hands-on problem-solving format should also be used. Research and Development: To provide the requisite tests for use as indicators of student learning, the committee recommends that a greatly accelerated program of research and development be undertaken aimed at the construction of free-response materials anct techniques that measure skills not measured with multiple-choice tests. The committee urges that the development of science tests at the K-5 level receive immediate attention. Techniques to be developed include problem-solving tasks, as exemplified by the College Board Advanced Placement Tests; pencil- and-paper tests of hypothesis formulation, experimental design, and other tasks requiring productive-thinking skills, as exemplified by questions in the British Assessment of Performance Unit Series; hands-on experimental exercises, as exemplified by some test materi- als administered by the National Assessment of Educational Progress (NAEP) and the International Association for the Evaluation of Edu- cational Achievement (TEA); and simulations of scientific phenomena with classroom microcomputers that give students opportunities for experunental manipulations and prediction of results. The creation of new science tests for grades K-5 should be done by teams that include personnel from the school districts that have been developing hands-oI1 curricula to ensure that the new tests match the objectives of this type of instruction. In addition to providing valid national indicators of learning in areas of great im- portance, such new assessment materials for science in grades K-5
6 INDICATORS OF SCIENCE AND MATHEAfA TICS EDUCATION will provide models of tests that state and local school officials may want to adopt and use. Eey Indicator: The committee recommends that assess- ment of student learning using the best available tests and testing methods continue to be pursued in order to provide periodic indicators of the quality of science and mathematics education. Tests should be given to students in upper-elementary, middle, and senior high school (for example, in grades 4, 8, and 12~. Because of the rapid changes taking place in science instruction in grades K-S, assessment at this level should be carried out every two years, using exercises developed according to the preceding recommendation. For higher levels, a four-year cycle is appropriate. The tests should be given to a national sample, using matrix-sampling techniques. Test scores should be available for each test item or exercise and should be reported over time and by student subgroups (e.g., gender, race, ethnicity, type of community). Similar procedures are appropriate for indicators of state or district assessments of student learning. Research and Development: The committee recommends that a research and development center be established to provide for the efficient production, evaluation, and distribu- tion of assessment materials for use as indicators of student learning at district, state, and national levels and for use by teachers in instruction. The center should function as a centralized resource and clear- inghouse that would make it possible for school people to survey the available assessment materials and obtain those desired. The center might be called the National Science and Mathematics Assessment Resource Center. Key Indicator: The committee recommends that, starting in 1989, the scientific and mathematical literacy of a random sample of adults (including 17-year-olds) be assessed. The
SUMMARY AND RECOMMENDATIONS assessment should tap the dimensions of literacy discussed in Chapter 2 and should be carried out every four years. 7 To make the desired types of assessment possible, effort should be devoted over the next two years to developing interim assessment tools that use some free-response and some problem-solving compo- nents; these assessment tools should be used until more innovative assessment techniques are available. The data collected should be aggregated and reported by age, gender, race, ethnicity, socioeco- nomic status, and geographic region so as to establish to what extent there are systematic inequities in the distribution of scientific and mathematical literacy. Indicators of Student Behavior Key Indicator: The committee recommends that data on secondary school course enrollment be gathered on a four- year cycle for both mathematics and science. The specific data to be gathered are the number of semesters of science and mathematics taken by students and total enrollment in the variety of science and mathematics courses offered in secondary schools. Courses should be identified as to level of difficulty (e.g., for eighth-grade mathematics: remedial, typical, enriched, algebra). The indicators to be constructed from these data are the average number of mathematics and science courses taken and the percent- age of students enrolled in specific courses. Key Indicator: The committee recommends that the data to be gathered at the elementary- and middle-school level, equivalent to course enrollment data, be the number of min- utes per week devoted to the study of science and mathemat- ics. The indicator should also be expressed both as a ratio of all instructional time and of total time spent in school. At each policy level-national, state, and local experts may wish to define the minimum amount of class time necessary in each grade, particularly for science. Because of the importance of possible
8 INDICATORS OF SCIENCE AND MATHEMATICS EDUCATION differences among various groups (ethnic and racial, gender, socio- economic status, etc.) we recommend that the data be collected at the level of both the school and the individual student. Key Indicator: The committee recommends development of a time-use study involving external observers to obtain some indication of the quality of the science and mathemat- ics instruction being received. In science classes, this would include, in addition to the teaching of conceptual and fac- tual knowledge, the percentage of time spent by students involved in the processes of science (observing, measuring, conducting experiments, asking questions, etc.~. A similar study is recommended for mathematics classes; a panel of mathematics educators should deterrn~ne the nature of the student behaviors sought. Supplementary Indicator: The committee recommends the collection of information on minutes per week spent on science and mathematics homework. The frequency and detail necessary for gathering data on home- work are the same as for in-school activities-that is, the information should be gathered every four years and allow analysis by ethnicity, race, gender, grade level, and size and type of community. Na- tional data are important for comparisons over time and with other countries; states and local districts may also wish to have this infor- mation. Care must be taken that homework done in school is not double counted as both homework time and instructional time. Research and Development: The committee recommends further research and development on possible supplemen- tary indicators in the following three areas of out-of-school behaviors, with the goal of clarifying their relationships to student mathematics and science learning: . Amount of time (minutes) devoted to out-of-school sci- ence and mathematics activities, for example, going to zoos and science museums, watching science programs on television, reading science books, playing with a
SUMMARY AND RECOMMENDATIONS . computer at home, voluntarily doing science projects or mathematics puzzles. Percentage of students reporting that they use (apply) the concepts of science and mathematics from time to time in their own lives. One way to implement this in- dicator is to conduct a survey on the number of times students faced a personal decision and relied on some- thing that they learned in science or mathematics to help them make that decision. Percentage of students reporting that they use the con- cepts of science and mathematics to help them address some persistent societal problem. Research and Development: The committee recommends continued research on linkages between student learning and various student activities, on more effective ways of assess- ing activities that affect learning, and on the factors that influence individuals to engage in these activities. Research and Development: Given the importance at- tached by science and mathematics educators to the devel- opment of attitudes that will foster continuing engagement with science and mathematics, the committee recommends that research be conducted to establish which attitudes af- fect future student and adult behavior in this regard and to develop unambiguous measures for those that matter most. Research and Development: The committee recommends research to identify and validate constructs related to the continuing involvement of students and adults with science and mathematics throughout their lives. In addition to the refinement of these constructs, strategies should be explored for obtaining indicators of the relevant constructs and asso- ciated behaviors. Indicators of Teaching Quality Key Indicator: The committee recommends that samples of teachers be selected to take tests that probe the same 9
10 INDICATORS OF SCIENCE AND ~TH~TICS EDUCATION content and skills that their students are expected to master. For this purpose, tests for teachers should be developed to include the same kinds of improvements that the committee recommends for tests of student learning. The distribution of teachers' test scores should be reported by student background and characteristics. This will provide informa- tion about the distribution across different student groups of teachers who are in command of the mathematics and science they are ex- pected to teach. Both current distribution and change over time are of interest; therefore, tests should be given every four years to a sample of all teachers and every two years to a sample of newly hired secondary school mathematics and science teachers. Supplementary Indicator: The committee recommends reorganization of the information currently being collected on teacher preparation (college courses in mathematics and science, majors and minors, advanced degrees), using various student groups taught as the reporting groups of interest. The information reported should display the percentage of stu- dents with particular backgrounds and characteristics who are being taught mathematics and science in elementary school as well as courses in these domains in secondary school by teachers with spe- cific college preparation. For this indicator also, four-year cycles are appropriate for collection and analysis of information. Research and Development: The committee recommends that research be undertaken on two issues: the impact of teachers' knowledge of subject matter on their effectiveness in teaching these subjects to students, and the role of early home and school experiences in determining the decisions to become a teacher and on how and what to teach. Supplementary Indicator: The committee recommends that time-budget studies be conducted, asking teachers to record how they spend time related professionally to their
SUMMARY AND RECOMMENDATIONS present or future classroom activities, other than in the classroom itself, during a particular period, perhaps a week. 11 The information collected should be evaluated against sets of activities identified by experts as advancing effectiveness in the cIass- room in teaching mathematics or science. Investigations of the relationships between professional activities reported by teachers and teaching effectiveness should be conducted to help refine this indicator. Research and Development: The committee recommends research on the following aspects of the behavior of teachers in science and mathematics instruction: . the factors affecting teacher responses to changes in the intended curriculum; the use of hands-on experiences involving concrete ma- terials, laboratory experiments, and computers; and allowing an adequate period of time for students to for- mulate responses to questions. Supplementary Indicator: The committee recommends that data be collected on a four-year cycle through open- ended surveys on the materials, facilities, and supplies avail- able and used by teachers in mathematics and science in- struction. An indicator can be constructed from this information by report- ing on the levels of resources being used in the classroom by student subgroups of different backgrounds and competencies. Key Indicator: The committee recommends collection at least every three years (preferably every two years) of de- tailed information on the salaries paid to college graduates with particular subject matter specialties who choose to en- ter various occupations.
12 INDICATORS OF SCIENCE AND MATHEMATICS EDUCATION The information should include data on starting salaries and on salaries after 15 years of experience. These data should be reported in a manner that facilitates comparisons of salaries in teaching with salaries in other occupations for college graduates trained in partic- ular sciences and mathematics. indicators of curriculum Quality Research and Development: In order to develop indica- tors of breadth of content coverage in the science and math- ematics curriculum, the committee recommends that exem- plary frameworks be constructed for the following curriculum blocks: grades K-5 science, grades K-5 mathe- matics, grades - 8 science, grades 6-8 mathematics, grades 9-12 literacy in science, grades - 12 literacy in mathematics, grades 9-12 science for college-bound students, and grades 9-12 mathematics for college-bound students. The frame- works for grades K-5 and 6-8 science should be accorded the highest priority. The frameworks must represent the structures of the subject matter and desirable learning goals, or alternatives among desirable goals. Key Indicator: Once the frameworks are constructed, the committee recommends that three elements of the intended curriculum should be matched and rated against them for content coverage: state guidelines, textbooks and such as- sociated materials as computer software and laboratory ex- ercises, and tests. The frameworks should also be used to analyze the content coverage of the implemented curriculum (i.e., the content presented to the student as reported by classroom teachers). The ratings obtained through analysis of the three elements of the intended curriculum and analysis of the implemented curriculum will provide the raw material for the construction of indicators of content coverage. The ratings should be carried out every four years
SUMMARY AND RECOMMENDATIONS 13 at the national level in synchronization with the student assessments recommended above so that the indicators can be used together. Research and Development: The committee recommends that research be carried out to establish the validity of teacher-reported information regarding content coverage in the classroom. Research and Development: Standards of excellence should be developed based on the best of curricula in current use. High-quality programs encompassing the curriculum blocks sug- gested above should be selected, profiled, and analyzed to provide models of excellence in depth of content coverage, scientific accuracy, and pedagogic soundness of science and mathematics curricula. Key Indicator: The quality of the curriculum should be assessed by expert panels along three dimensions: depth of content treatment, scientific accuracy, and pedagogic sound- ness. Ratings for each of these quality dimensions should be assigned to the three elements of the intended curriculum (i.e., state guidelines, texts and associated materials, and tests). Assessments regarding depth of treatment should also be made of the implemented curriculum through teacher and student surveys and classroom observation. To assess the depth of content treatment, the frameworks devel- oped according to the recommendation made above should be used to identify the critical topics that constitute a coherent curriculum. Weights assigned by each rating panel regarding the depth of treat- ment desired for a given topic must be made explicit in reporting results. The assessment of the scientific accuracy of the intended cur- riculum should be carried out by scientists in the relevant disciplines. The scientific content of the frameworks shoud be used to construct the tests of teacher competency of subject matter recommended in
14 INDICATORS OF SCIENCE AND MATHEMATICS EDUCATION Chapter 6 and such tests used as a minimum measure of the scientific accuracy of the actual curriculum experienced by students. Research and Develop~nent: The committee recommends research to provide validity checks on the standards being used to assess depth of treatment, scientific accuracy, and pedagogic soundness of science and mathematics curricula. For example, research should be undertaken to establish what pedagogic knowledge teachers need to have and need to know how to use in order to teach science and mathematics effectively to students of different ages, backgrounds, and competencies. [r`dicators of Financial and Leadership Support Supplementary Indicator: The committee recommends the construction of a set of accounts detailing the level and type of support for science and mathematics education from all departments and agencies of the federal government that fund relevant programs. The importance of having reliable annual data on the level of federal financial support merits the investment necessary to con- struct such a set of accounts. Agencies should be encouraged to report budget and funding data by categories identifiable as precol- lege mathematics and science education, and funds should be made available (possibly through NSF) to perform the necessary analy- ses. The kind of disaggregation of financial support for science and mathematics education found in the NSF budget could be used as a mode! for developing the recommended cross-agency indicator of federal support. Supplementary Indicator: The committee recommends that indicators be designed using budgetary data of scien- tific bodies and information on stab time and volunteer time devoted to education and that these indicators be routinely available to reflect the commitment of resources by scien- tific bodies for the improvement of mathematics and science education in the schools.
