5
ASSESSING TO SUPPORT EQUITY AND OPPORTUNITY IN MATHEMATICS LEARNING

High-quality mathematics assessments must focus on equity concerns to ensure that all students are well served by assessment practices. This fundamental concept is embodied in the third educational principle

THE EQUITY PRINCIPLE

Assessment should support every student's opportunity to learn important mathematics.

The tendency in education has been to think about equity in terms of groups of children who by some definition are not well served by educational institutions. Quite often in these discussions the focus has been on groups that are defined in terms of status variables such as sex, ethnicity, native language, and socioeconomic indicators, because these factors influence access, expectations, and the systems of rewards. Increasingly, however, educators have recognized that although status characteristics of learners may be important, they are far less important than functional characteristics are in the design of instructional strategies.1 Reforms in mathematics and other areas of schooling must aim to ensure that each student, regardless of achievement level or demographic characteristics, has the opportunity to study challenging subject matter. The example on the following page, a policy statement adopted by the National Council of Teachers of Mathematics (NCTM) in April 1990,2 reflects this perspective on equity.

Some proponents of educational reform maintain that use of more flexible assessment methods and more performance-oriented



The National Academies | 500 Fifth St. N.W. | Washington, D.C. 20001
Copyright © National Academy of Sciences. All rights reserved.
Terms of Use and Privacy Statement



Below are the first 10 and last 10 pages of uncorrected machine-read text (when available) of this chapter, followed by the top 30 algorithmically extracted key phrases from the chapter as a whole.
Intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text on the opening pages of each chapter. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.

Do not use for reproduction, copying, pasting, or reading; exclusively for search engines.

OCR for page 91
Measuring What Counts: A Conceptual Guide for Mathematics Assessment 5 ASSESSING TO SUPPORT EQUITY AND OPPORTUNITY IN MATHEMATICS LEARNING High-quality mathematics assessments must focus on equity concerns to ensure that all students are well served by assessment practices. This fundamental concept is embodied in the third educational principle THE EQUITY PRINCIPLE Assessment should support every student's opportunity to learn important mathematics. The tendency in education has been to think about equity in terms of groups of children who by some definition are not well served by educational institutions. Quite often in these discussions the focus has been on groups that are defined in terms of status variables such as sex, ethnicity, native language, and socioeconomic indicators, because these factors influence access, expectations, and the systems of rewards. Increasingly, however, educators have recognized that although status characteristics of learners may be important, they are far less important than functional characteristics are in the design of instructional strategies.1 Reforms in mathematics and other areas of schooling must aim to ensure that each student, regardless of achievement level or demographic characteristics, has the opportunity to study challenging subject matter. The example on the following page, a policy statement adopted by the National Council of Teachers of Mathematics (NCTM) in April 1990,2 reflects this perspective on equity. Some proponents of educational reform maintain that use of more flexible assessment methods and more performance-oriented

OCR for page 91
Measuring What Counts: A Conceptual Guide for Mathematics Assessment Mathematics for All Students In April 001990, the NCTM Board of Directors endorsed the following statement: As a professional organization and as individuals within that organization, the Board of Directors sees the comprehensive mathematics education of every child as its most compelling goal. By every child we mean specificallly students who have been denied access in any way to educational opportunities as well as those who have not; students who are African American, Hispanic, American Indian, and other minorities as well as those who are considered to be a part of the majority; students who are female as well as those who are male; and students who have not been successful in school and in mathematics as well as those who have been successful. tasks can produce fairer measures of all students' intellectual development.3 Others have cautioned that new assessment content and methods alone can not assure equity: The call for performance assessment did not derive from concerns for fairness or equal access… Well designed assessment systems might act as a lens to more clearly reveal existing and ongoing inequalities, and to inform policy and practice… but only if explicitly designed to do so in all its parts.4 Designing for equity requires conscientious rethinking of not just what we assess and how we do it, but how different individuals and groups are affected by assessment design and procedures. It also requires conscientious attention to how we use assessment results to make decisions about individual children and the schools they attend. DEVELOPING ASSESSMENTS TO INCREASE EQUITY To meet the equity principle, tasks must be designed to give children a sense of accomplishment, to challenge the upper reaches of each child's mathematical understanding, and to provide a window on each student's mathematical thinking. Just as good instruction accommodates differences in the ways learners construct knowledge, good assessments accommodate differences in the ways that students think about mathematics and display mathematical understanding. Although all students are to be assessed on important mathematical concepts and skills, in accord with the content and learning principles, the equity principle implies that assessments must be sufficiently flexible to allow all students to show what they know and can do.

OCR for page 91
Measuring What Counts: A Conceptual Guide for Mathematics Assessment DESIGN FEATURES Just as good instruction accommodates differences in the ways learners construct knowledge, good assessments accommodate differences in the ways that students think about and display mathematical understanding. Some ways of accommodating differences among learners include permitting multiple entry and exit points in an assessment and allowing students to respond in ways that reflect different levels of mathematical knowledge or sophistication. These design characteristics are critical to assessment equity just as they are critical to the content and learning principles. Consider the two apparently parallel problems illustrated below and on the following page.5 Close inspection indicates that the first problem may be less accessible to many students. In the first mosaic there is no indication of how the colored tiles are arranged to form the picture. Therefore, to find how many of the different colored tiles would be needed for the enlarged picture, one must already known the general proposition that doubling the length of a figure quadruples its area. On the other hand, the figure in the second problem, with its explicit specification of the color of each part, opens alternative avenues for approaching the task. For example, the student might draw a sketch Pythagoras—A Mosaic Problem Your school's math club has designed a fancy mosaic in honor of Pythagoras, whose work united algebra and geometry. The mosaic includes a picture of the Pythagorean Theorem and its algebraic statement, as indicated below. The figure appears inside a square. You have constructed a 2' × 2' scale model of the mosaic, which used 120 red mosaic pieces, 200 blue mosaic pieces, and 150 yellow pieces How many red, blue, and yellow pieces would be required for a 4' × 4' mosaic? Explain. How many red, blue, and yellow pieces would be required for a 6' × 6' mosaic? Explain. The school's principal likes the model of the mosaic so much that she's willing to have you build a big version that will go on permanent display in the auditorium. The only thing—she hasn't yet decided how big the mosaic should be. She wants the math club to write her a letter explaining how many tiles of each color she'll need, as soon as she makes up her mind how big the side of the square should be. Write the letter, providing an explanation of how the number of tiles can be calculated, and also why your answer is correct.

OCR for page 91
Measuring What Counts: A Conceptual Guide for Mathematics Assessment Tri-Tex Logo—A Mosaic Problem Tri-Tex Corporation has the following logo. It's going to put a large-scale mosaic of the logo on the side of its corporate headquarters building. A 2' × 2' scale model of the logo uses: 144 yellow tile pieces 144 blue tile pieces 288 red tile pieces When the logo is placed on the side of the building it will also have a border that consists of a single band of long black tiles. It takes 40 of the black tiles to make a border for the 2' × 2' scale model. How many tiles of each color will the purchasing department have to order if the full-scale mosaic is 4' × 4'? What if it's 6' × 6'? Tri-Tex's president says he'll decide later just how big the mosaic will be, but he wants everything set up in advance so the purchasing department can send out the order as soon as he makes up his mind. Your job is to write a two-part memo to the purchasing department. In part I of the memo, tell how the purchasing department can do a simple calculation to find out how many yellow, blue, red, and black tiles they need to purchase, for any size design. The instructions should be as simple and direct as possible. In part 2 of the memo, explain how you arrived at the answer you did, so the people in the purchasing department can understand why your instructions give the right answer. of the enlarged figure and discover how many congruent copies of the original figure will be needed to cover it. In fact, by using an informal, experimental approach to the task, the student has the opportunity to gain insight into the relation between length and area, thus potentially learning some important mathematics in the process of doing the task. As the preceding illustration suggests, assessments designed with multiple entry and exit points permit each student—even those who had mastered only the most elementary knowledge and skills tapped by a problem—to experience some success. However, this goal is not always easily accomplished. Consider the response on the following page of a Ute Indian student to an assessment task.6 Rather than focusing on the mathematical question asked, the student responded to the hypothetical nature of its premise. Children who live in harsh environments in the inner cities and in remote rural areas often respond to hypothetical problems in much the same way. Hypothetical situations representing options that do not exist in their daily lives may be treated as meaningless or nonsensical. Therefore, their responses may reveal little about their-

OCR for page 91
Measuring What Counts: A Conceptual Guide for Mathematics Assessment A Cultural Convention An elementary school student from the Northern Ute Indian reservation in Utah was interviewed as he solved some word problems. He was asked to determine how much his brother would have to spend on gasoline if he wanted to drive his truck from the reservation to Salt Lake City. Instead of estimating (or generalizing) a response, or attempting to calculate an answer based on the information presented in the request, the student responded quite simply: "My brother does not have a pickup." He was assessing the truth value of the conditions presented by the problem. Such assessments are a common part of everyday life of this reservation. Decisions regarding personal conduct as well as innovations and changes affecting tribal membership as a whole are made only if the proposed line of action seems consistent with recognized cultural convention. Proposals that conflict with the way things are may be briefly considered, then set aside for action at some later time, or may simply be given no serious discussion at all. understanding of the mathematical problem posed or the mathematical knowledge and skills required by the problem. As assessments become more complex and more connected to real-world tasks, there is a greater chance that the underlying assumptions and views may not apply equally to all students, particularly when differences in background and instructional histories are involved. In such instances special care will be needed to determine whether differences in performance stem from differences in unrelated characteristics or genuinely reflect differences in mathematical development. The evolving agenda for research on assessment must reflect as much concern for equity as for content, learning, and technical issues. SUPPORTING STRATEGIES When assessments are administered by the teacher in a classroom setting to gauge student progress or diagnose a particular difficulty, differences in students' backgrounds that might affect results are likely to be known. In such cases skillful probing can be used to clarify the basis of the students' responses. Questions can be reframed in ways that better reflect students' understanding of the mathematics required. In many assessment settings, particularly those used for accountability, it is less possible to accommodate differences in students' ways of understanding and responding to a task. To help overcome this difficulty, it is important that varied perspectives be represented at every phase of assessment development and use. The teams that create the intellectual frameworks from which specific tasks are generated should include experts with different views and life experiences. Broad-based representation is also vital on the research teams that pose study

OCR for page 91
Measuring What Counts: A Conceptual Guide for Mathematics Assessment questions, devise data collection protocols and strategies, and oversee analyses and reporting of results. It is equally important in pilot research: sites should be selected to provide realistic variation among students, teachers, and schools. The raters who evaluate student responses should also represent varied backgrounds.7 The evolving agenda for research on assessment must reflect explicit concern for equity as well. Several projects of major national significance have taken steps in this direction. The New Standards Project (NSP), for example, has created special equity initiatives to ensure that the standards and tasks adopted reflect the experiences and aspirations of varied groups. NSP has conducted focus groups with local and national advocacy representatives to gain broad perspectives on how equity concerns might be addressed. It has also established advisory groups of scholars and practitioners with expertise in culture, gender, and language issues to advise the project on development of assessment tasks and procedures. In addition, a special advisory group was created to address issues involved in administering appropriate assessment tasks and procedures for students who are not native speakers of English.8 The Center for Research, Evaluation, and Student Testing (CRESST) at the University of California at Los Angeles also has special equity initiatives. CRESST created a task force of consultants and scholars to investigate the equity challenges confronting the assessment community. The task force has a broad agenda, encompassing very practical issues as well as a range of theoretical issues and basic research problems: What dimensions of diversity have relevance for education and the assessment of educational achievement? How do group identity and status influence learning of varied cultural groups and their responses to assessment probes? How can problems of reliability, validity, and calibration posed by cultural diversity be addressed? How can procedures and criteria be developed so that appropriate and inappropriate responses are defined that are sensitive to differences among the groups being assessed? On what kinds of tasks do specific groups tend to excel or have difficulty?9

OCR for page 91
Measuring What Counts: A Conceptual Guide for Mathematics Assessment INTERPRETING ASSESSMENT RESULTS Despite efforts such as those described above and the hopes of the educational reform community that new assessments will be fairer to every student than traditional forms of testing were, preliminary research does not confirm the corollary expectation that group differences in achievement will disappear. Surely, there are many anecdotal reports from classroom teachers and researchers of some students who perform quite poorly on traditional tests exhibiting advanced reasoning and mathematical understanding on more flexible assessments. However, several studies suggest that differences in average group performance are essentially unchanged with the switch from multiple-choice to open-ended assessments. Still other research suggests that group differences might be magnified when performance assessment tasks are used.10 Although the results may vary depending on the content area assessed, the nature of the specific tasks used, and whether group or individual performance is the focus of comparisons, it seems clear that reformers' hopes must be balanced by a spirit of empiricism: there is much more to be learned about how changes in assessment will affect long-standing group differences, one facet of equity concerns. Equity implies a commitment to seek and explore the sources of any systematic performance differences that may be observed on new assessments. In any event, equity implies a commitment to seek and explore the sources of any systematic performance differences that may be observed on new assessments. It is important to know whether observed differences among groups reflect genuine differences in mathematical development or can be attributed to factors such as those identified in earlier chapters as special challenges for assessment design: The context in which a task is presented can influence mathematics performance.11 The words and concepts used, the expressions of knowledge required or permitted, will always reflect some particular set of values or life experiences. Not all children will be equally familiar with any given context. Use of genuinely novel contexts or pre-assessment experience with a given context will allow students to enter on an equitable footing. Language can be a potent factor in mathematics assessments.12 Better readers may more easily grasp what they are being asked to do on an assessment task. Better writers may more often present their ideas well, even when of marginal mathematical quality. Because communication is a legitimate facet of mathematical power, some balance must be sought in determining how to weigh students' general command of language in evaluating their expressions of mathematical ideas.

OCR for page 91
Measuring What Counts: A Conceptual Guide for Mathematics Assessment Open-ended assessment procedures may exacerbate the influence of resource inequities that exist in the real world. If parents or teachers participate in rather than support students' work, assessments may overestimate the student's mathematical fluency. Rater influences may distort judgments of performance on more open-ended tasks. Many factors can come into play: the rater's expertise and knowledge of the specific mathematics content of the task, the rater's interpretation of task requirements and scoring criteria, or even similarities or differences in the backgrounds of the rater and the student being evaluated.13 When any of these factors contribute to systematic differences among definable groups, assessment results have meanings or implications that are different for some groups than for others. Inferences based on results from such an assessment are said to be biased, that is, not equally valid for all groups.14 Solid documentation on the students assessed, the schools they attend, and the instruction they have received makes it possible to identify inequities in opportunity. Solid documentation on the students assessed, the schools they attend, and the instruction they have received makes it possible to identify inequities in opportunity. Pilot testing provides an opportunity to ferret out biases and to try out potential remedies. Thorough documentation also is important when assessments are fully implemented so that appropriate cautions and conditions can be applied to interpretations and uses of the results.15 Remedies for group differences depend on the source of the difference. Not all differences will be due to bias. Some, in fact, may be due to systematic differences in the quality of mathematics instruction provided. The following example outlines some of the kinds of information needed to form inferences about individuals and schools.16

OCR for page 91
Measuring What Counts: A Conceptual Guide for Mathematics Assessment Equity of Opportunity Indices CRESST has identified the kinds of process information that will be needed about the quality of students' educational experiences to ensure that clear inferences can be drawn from assessment results: What evidence is there that the students have had opportunity to learn the assessed material? What evidence is there of the quality of those experiences? What is the cumulative experience of transient or frequently absent students? What evidence is there that poorly performing groups of students have been taught by teachers of the same quality, training, and experience as more successful students? What are the net educational resources available to students, including compensatory family and community benefits? Are comparable books, materials, and the other educational supports available across groups? Is there evidence that the affective environments of education are comparable? How safe are the schools? What size are the schools? How are individual rights and needs accommodated? How balanced is the exposure of students across the full range of desired educational outcomes, both measured and unmeasured? USING ASSESSMENTS TO COMMUNICATE NEW EXPECTATIONS Assessments represent an unparalleled tool for communicating the goals and substance of mathematics education reform to various stakeholders. Teachers, students, parents, policymakers, and the general public will need to understand clearly where mathematics reform will take America's children and why they should support the effort. Assessments can be enormously helpful in this reeducation campaign. Public dialogue is an important strategy for ensuring that assessments are designed and used equitably. PROMOTING PUBLIC INVOLVEMENT Increasingly, public dialogue is seen as an important strategy for ensuring that assessments are designed and used equitably.17 The Urban District Assessment Coalition (UDAC), a joint effort of Boston College and the American Federation of Teachers, organized a series of conversations about assessment reform among various groups before designing a new assessment system for the Boston schools. Over several months, parents, teachers, administrators, and members of the business community worked to arrive at a

OCR for page 91
Measuring What Counts: A Conceptual Guide for Mathematics Assessment consensus on the values, goals, and information needs the system would serve. Representatives of each group have been involved in all stages of the development process, from planning to scoring responses during the first round of pilot tests. That process is being replicated in other UDAC sites.18 Another strategy for increasing public engagement involves making assessment blueprints public. CRESST has proposed that the public be given quite extensive information in blueprints,19 as indicated in the example below. Assessment Blueprints CRESST views public reporting of assessment blueprints as an equity remedy for assessment design. They have recommended that the following questions be addressed in such blueprints. World knowledge: What common experiences and understandings are required of students to make sense of the task and to productively undertake its solution? Which of these are more or less likely for students of different cultural and socioeconomic backgrounds? Prior knowledge: What specific types of information or resources are essential for successful performance? Language demands: If the focus of the task is not language facility, are alternative options for display of understanding available for students with limited English proficiency. Task structure and topics: How were task structure and topics created? Is there reason to believe that all groups of children will be motivated by the topics provided? Are sufficient numbers of topics represented to draw safe inferences about the task domain of interest. Scoring criteria: What criteria will be used to judge student performance? Are these criteria specific enough to overcome biases potential in global ratings of performance. Judges: How are raters selected? Do they possess high degrees of relevant knowledge of the dominant to be assessed? How have they been trained? Have raters' preferences for performances of rates of like ethnicity been estimated? Access: Who is excluded from assessments and on what basis? How comparable are exclusion rules? What special provisions for access are available to students with special needs? Equity of educational assessment settings: What is the quality of educational experiences to which students have been exposed? PROMOTING COMMUNITY UNDERSTANDING OF ASSESSMENT REFORM For mathematics education reform to reach each child, parents and other community members will need to understand the new vision of school mathematics. Their involvement is needed to support the efforts of the teachers, school administrators, and others who, with them, share responsibility for the education of their children. Because many students are "doing well" or "showing improvement" in mathematics achievement as measured by many of

OCR for page 91
Measuring What Counts: A Conceptual Guide for Mathematics Assessment the tests in use today, members of each school community will need to be assured that new assessments and the standards they reinforce are not arbitrary. They need to understand that "doing better" by old standards is no longer good enough to secure children's futures. Assessment tasks and their corresponding scoring criteria can be used to make this point quite convincingly. They illustrate that the knowledge and proficiency that students are being asked to demonstrate are of value of all students. Parents and community members need to understand that doing well by old standards is no longer good enough to secure children's future. Supervisors in the San Francisco Unified School District have reported that working with assessments in PTA meetings helped parents in one community see exactly how the mathematics that their children will need differs from the mathematics the parents learned.21 These sessions helped assure parents that much of the mathematics that they value (e.g., numbers and computation) remains important but that it is being learned and assessed as a part of meaningful problem solving and reasoning. After preliminary discussion of the kinds of mathematics students would need in the future and an overview of K–12 program content, parents met in small groups to discuss the mathematics required by several problems. One of the more challenging problems is shown below.22 Scoring rubrics and sample responses were also made available to the small groups. The program coordinator reported several interesting reactions from parents. Many were awed by the kinds of problems that students are now expected to do and the sophisticated mathematical problem solving and reasoning required. Most believed it important for their children to develop the kinds of skills the problem tapped. Most also believed their children could do the kind of work required but stressed the importance of students getting an early start in primary grades. Many parents asked questions about the quality of mathematics instruction at the school and sought reassurance that their children were getting the right kind of teaching. Most significantly, perhaps, no parent asked why children were not drilling on multiplication facts, a rare Multiplication Task Used in Parent Workshop The five digits—1, 2, 3, 4, and 5—are placed in the boxes above to form a multiplication problem. If they are placed to give a maximum product, the product will fall between: A. 10,000 and 22,000 B. 22,001 and 22,300 C. 22,301 and 22,400 D. 22,401 and 22,500

OCR for page 91
Measuring What Counts: A Conceptual Guide for Mathematics Assessment rather than to manipulate test scores, rational policies must be developed for using assessments. Policies that promote equity may challenge well-entrenched school practice and the educational philosophies upon which it rests. Clearly, teachers' evaluations play a strong role in grouping and tracking decisions, but the sorting would not take place without shared beliefs to sustain it. The practice of sorting rests on a general societal preoccupation with individual differences in ability, particularly in ability to learn and achieve in mathematics. Assessment also plays a role in perpetuating the practice. The scientific mystique that surrounds tests and testing procedures helps to convince teachers, parents, the public, and students themselves that this sorting is both rational and efficient. Assessments should not be used to channel students into dead-end mathematics courses. Assessments should not be used to channel students into dead-end mathematics courses that will leave them ill prepared to meet the challenges of the twenty-first century. The lessons of Chapter I remedial programs make this point clearly. Chapter I of Title I of the Elementary and Secondary Education Act provides funds for programs to improve the achievement of low-income students who are functioning below grade level in mathematics and reading. Eligibility for Chapter I services is determined by scores on standardized achievement tests that do not target higher-order skills. Program effectiveness is judged by performance gains on those same tests. Although the basic skills gap separating Chapter I students from other students has declined in the past 15 years, critics have charged that "A continued focus on remediation denies the richness of learning to those who need more, not less, of what makes education engaging and exciting."32 In mathematics, many of the children who begin remediation in third grade never catch up to their peers. By ninth grade many are so far behind that they opt out of more challenging mathematics courses if given a choice. Others are programmed out of such courses by teachers and counselors who judge them poorly prepared for gateway subjects, such as algebra and geometry. Many of the proposals for improving Chapter I programs focus on assessment, because Chapter I eligibility is based on students' performance on tests that reflect very low standards, and on students' relative standing. The tests typically used provide no information on how students measure up to standards of what they need to know to function in the adult world. Although proposals differ, most recommend the use of assessments consonant with the

OCR for page 91
Measuring What Counts: A Conceptual Guide for Mathematics Assessment ideas embedded in the content and learning principles. Most also recommend that assessment results not be used for individual placements that deny children access to powerful and engaging learning experiences.33 PROVIDING SAFEGUARDS FOR STUDENTS The equity principle requires that opportunity to learn be considered whenever assessment results are reported or used. Obviously, students who have had opportunities to reflect on the mathematics they are learning, to present and defend their ideas, or to organize, execute, and report on a complex piece of work will have an advantage when called upon to do so in an assessment situation. There is a legal basis for this requirement as well. The courts have held, in several landmark cases, that schools are obligated to see that all students have a chance to learn content that is assessed for high-stakes purposes. The courts have held that schools are obligated to provide all students a chance to learn content for high-stakes assessments. In Florida, for example, the state legislature imposed a testing program to guarantee that all students would graduate with the minimum skills needed to function in society. To be granted a high school diploma, students had to pass functional literacy examinations in reading, writing, and mathematics in addition to completing all required courses. Average test performance was poor across the board when the program began, but minority students failed the test at disproportionately high rates. This practice was successfully opposed in the case of Debra P. v. Turlington, where the court ruled that the state "may not constitutionally so deprive its students [of a high school diploma based upon test performance] unless it has submitted proof of the curricular validity of the test which the court defined as 'a fair test of that which was taught.'" The ruling in that case and several others that dealt with high-stakes use of tests for placement established two important guidelines for fairness in educational assessment. First, assessments used to allocate different instructional treatments must be demonstrably fair tests of what students have been taught. Second, schools must show that they have taken affirmative steps to remove the effects of any past discrimination that might influence the results of the assessments.34 These rulings have been construed to mean that when assessments are used to make high-stakes decisions on matters such as graduation and promotion, certain basic safeguards must be built

OCR for page 91
Measuring What Counts: A Conceptual Guide for Mathematics Assessment into the system for students. Social compacts, school delivery standards, and a variety of other mechanisms have been proposed to provide the needed safeguards. The National Council on Educational Standards and Testing recommended that schools be held to standards of instructional quality (termed school delivery standards) to ensure that students have a fair chance to develop the knowledge and skills targeted by the more challenging assessments now being proposed.35 The council was formed to advise the Congress, the Secretary of Education, and the National Education Goals Panel on the desirability and feasibility of instituting a national testing system. Other groups, while not endorsing a specific proposal for school delivery standards, have endorsed the notion that opportunity to learn must be taken into account in interpreting and making decisions based on assessment results.36 NSP Social Compact We pledge to do everything in our power to ensure all students a fair shot at reaching the new performance standards, and to prevent students' performance on the new assessments from being used as the basis for awarding a diploma or any other form of credential unless all students in the jurisdiction awarding the credential have had an opportunity to prepare themselves well. This means that they will be taught a curriculum that will prepare them for the assessments, their teachers will have the preparation to enable them to teach it well, and there will be an equitable distribution of the resources the students and their teachers need to succeed. The NSP requires the states and school districts working with them to enter into a social compact (see left) designed to address issues of fairness in accountability.37 Social compacts and school delivery standards are straightforward enough in conception. Both attempt to ensure that all students have a chance to learn the mathematics that is assessed. Neither proposal has found universal support among educators and policymakers, however. Challenges have been raised on technical grounds mainly related to the difficulty of developing and validating appropriate indicators and cutoff scores. Opposition has also been forthcoming on policy grounds: the burdens delivery standards might impose on states, unresolved consequences for schools that fail to satisfy standards, and unresolved questions about consequences for individual rights in high-stakes contexts when schools fail to meet delivery standards. Some of the strongest opposition to delivery standards, however, rests on the observation that remedies designed to improve schools in the past often failed precisely because the emphasis was placed on the resources schools should provide rather than the outcomes that schools should achieve.38 The equity principle requires that due consideration be given to the consequences of assessments for individual students.

OCR for page 91
Measuring What Counts: A Conceptual Guide for Mathematics Assessment PROVIDING SAFEGUARDS FOR SCHOOLS The equity principle also requires some consideration of assessment consequences for schools. This implies, first of all, that fair comparisons can be made of what schools are accomplishing only when assessment data include information on the nature of the students served and the adequacy of resources. Second, the principle implies that assessment results must be compiled and reported in ways that support efforts to achieve greater equity among schools. Schools educating large numbers of poor and minority students have further to go and have fewer discretionary resources with which to pursue reform than do more affluent schools. Those schools that have more to do need greater resources. Without equity in resources—not the same as numerical equality—there can be no equity in assessment or any other sphere of education.39 This does not absolve schools of their responsibility to educate students to the level of new standards. Rather, it obligates educational reformers and the general public to take quite strong measures to ensure that every school has the resources needed to educate its students well. Such efforts are under way in a number of states, as evidenced by recent litigation and legislative proposals in Alabama, Texas, and Massachusetts aimed at reducing disparities in the funding of poor and more affluent schools.40 Without equity in resources there can be no equity in assessment. HOLDING ALL STUDENTSTO HIGH EXPECTATIONS Assessments can contribute to students' opportunities to learn important mathematics only if they reflect and are reinforced by high expectations for every student. Performance standards communicate expectations for achievement. They provide concrete examples and explicit definitions of what students must do to demonstrate that they have learned required content, process, and skills to a satisfactory level.41 If we want excellence from every student, performance standards must be set high enough so that, with effort and good instruction, every student will learn important mathematics. Some students may take longer than others to master the mathematics that they will need throughout their lives. Some may require different kinds of instructional support to meet performance standards, but there can be no equity in assessment as long as excellence is not demanded of every student.

OCR for page 91
Measuring What Counts: A Conceptual Guide for Mathematics Assessment Performance standards must be high enough to ensure that doing well on an assessment means that the student has learned enough of the right kinds of mathematics to function as an informed citizen. Meeting performance standards should mean that students have an adequate foundation on which to build, regardless of their eventual chosen career. There can be no equity in assessment as long as excellence is not demanded of every student. We have much to learn about how to ensure that performance standards are uniformly high for everyone while also ensuring that students are adequately prepared to pursue their chosen field. We also have much to learn about how to maintain uniformly high performance standards while allowing for assessment approaches that are tailored to varied backgrounds. Consistent application of standards to a varied set of tasks and responses poses an enormous challenge that we do not yet know how to meet fairly and effectively. Nonetheless, the challenge is surely worth accepting. To meet this challenge, equity must be viewed in the broadest terms possible. Equity considerations must be a part of every decision affecting the development, interpretation, and use of assessments. This view is expressed quite forcefully in a statement of equity principles adopted by a number of educational leaders, reproduced here on the following page.42

OCR for page 91
Measuring What Counts: A Conceptual Guide for Mathematics Assessment Leadership Statement of Nine Principles on Equity and Educational Testing and Assessment March 12, 1993 The following statement was developed by a group of educational leaders seeking to ensure that concerns for equity would be reflected in all efforts at assessment reform. Since the statement was articulated, it has been signed by educational policymakers, measurement scientists, educational researchers, school administrators, and experts on equity and diversity who represent a variety of perspectives and expertise. Although these leaders have not always agreed on the particulars of assessment reform, they have unanimously supported the need to address equity and excellence in tandem as assessment reform moves forward. As policymakers move forward to develop new standards and assessments, they should consider including the following principles, which will help to ensure that both equity and quality are dominant themes: New assessments should be field tested with the nation's diverse population in order to demonstrate that they are fair and valid and that they are suitable for policymakers to use as levers to improve outcomes before they are promoted for widespread use by American society. New standards and tests should accurately reflect and represent the skills and knowledge that are needed for the purposes for which they will be used. New content standards and assessments in different fields should involve a development process in which America's cultural and racial minorities are participants. New policies for standards and assessments should reflect the understanding that standards and assessments represent only two of many interventions required to achieve excellence and equity in American education. Equity and excellence can only be achieved if all educators dedicate themselves to their tasks and are given the resources they need. New standards and assessments should offer a variety of options in the way students are asked to demonstrate their knowledge and skills, providing a best possible opportunity for each student to perform. New standards and assessments should include guidelines for intended and appropriate use of the results and a review mechanism to ensure that the guidelines are respected. New policies should list the existing standards and assessments that the new standards and assessments should replace (e.g., Chapter I standards and tests, state-mandated student standards and tests) in order to avoid unnecessary and costly duplication and to avoid overburdening schools, teachers and students who already feel saturated by externally mandated tests. New policies need to reflect the understanding by policymakers of the tradeoff between the types of standards and assessments needed for monitoring the progress of school systems and the nation versus the types of standards and assessments needed by teachers to improve teaching and learning. The attention and resources devoted to the former may compete for the limited resources available for research and development for the latter. New policies to establish standards and assessments should feature teachers prominently in the development process.

OCR for page 91
Measuring What Counts: A Conceptual Guide for Mathematics Assessment ENDNOTES 1   Edmund Gordon, "Assessment Challenges: Changing Views of Learning, Instruction and Assessment" (Paper presented at the annual conference of the National Center for Research on Evaluation, Standards, and Student Testing, Los Angeles, CA, 10 Sept. 1992). 2   2. National Council of Teachers of Mathematics, Professional Standards for Teaching Mathematics (Reston, VA: Author, March 1991), 4. 3   See D. A. Archbald and F. M. Newmann, The Nature of Authentic Academic Achievement in the Secondary School (Reston, VA: National Association of Secondary School Principals, 1988); Joseph P. McDonald, "Three Pictures of an Exhibition: Warm, Cool, and Hard" (Substantially revised version of a paper presented at the annual meeting of the American Education Research Association, Chicago, IL, April 1991); Grant Wiggins, "Teaching to the (Authentic Test)," Educational Leadership 46:7 (1989), 41-47; Grant Wiggins, "Creating Tests Worth Taking," Educational Leadership 49:8 (1992), 26-33; Dennie P. Wolf et al., "To Use their Minds Well: Investigating New Forms of Student Assessment," in Gerald Grand, ed., Review of Research in Education (Washington, D.C.: American Educational Research Association, 1991), 31-74. 4   Dennie P. Wolf and Sean F. Reardon, "Equity in the Design Performance Assessments: A Handle to Wind Up the Tongue With?" (Paper presented at the Ford Foundation Symposium on Equity and Educational Testing and Assessment, Washington, D.C., 11-12 March 1993). 5   This exercise was created by the Balanced Assessment Project at the University of California, Berkeley, which is funded by the National Science Foundation. 6   William L. Leap, "Assumptions and Strategies Guiding Mathematics Problem Solving by Ute Indian Students," in Rodney R. Cocking and José P. Mestre, eds., Linguistic and Cultural Influences on Learning Mathematics (Hillsdale, NJ: Lawrence Erlbaum Associates, Inc., 1988), 176-177. 7   Eva L Baker and Harold F. O'Neil, Jr., "Diversity, Assessment, and Equity in Educational Reform" (Paper presented at the Ford Foundation Symposium on Equity and Educational Testing and Assessment, Washington, D.C., 11-12 March 1993); Related papers presented at the same symposium were George F. Madaus, "A Technological and Historical Consideration of Equity Issues Associated with Proposals to Change Our Nation's Testing Policy"; Warren Simmons and Daniel P. Resnick, "National Standards, Assessment and Equity." 8   Warren Simmons and Lauren Resnick, "Assessment as the Catalyst of School Reform," Educational Leadership 50:5 (1993), 11-15. 9   "Authentic Assessments—The Challenge of Diversity and Equity," The CRESST Line, Newsletter of the UCLA Center for Research on Evaluation, Standards, and Student Testing, Spring 1992, 6. 10   Robert L. Linn, Eva L. Baker, and Stephen B. Dunbar, "Complex, Performance-Based Assessment: Expectations and Validation Criteria" Educational Researcher 20:8 (1991), 15-21; Stephen B. Dunbar, Daniel M. Koretz, and H. D. Hoover, "Quality Control in the Development and Use of Performance Assessments," Applied Measurement in Education 4:4 (1991), 289-303; R. J. Shavelson, G. P. Baxter, and J. Pine, "Performance Assessments: Political

OCR for page 91
Measuring What Counts: A Conceptual Guide for Mathematics Assessment     Rhetoric and Measurement Reality," Educational Researcher 21:4 (1992), 22-27; R. J. Shavelson et al., "New Technologies for Large-Scale Science Assessments: Instruments of Educational Reform" (Paper presented at the annual meeting of the American Educational Research Association, Chicago, IL, April 1991). 11   Monty Neill, "Some Pre-Requisites for the Establishment of Equitable, Inclusive Multicultural Assessment Systems" (Paper presented at the Ford Foundation Symposium on Equity and Educational Testing and Assessment, Washington, D.C., 11-12 March 1993); J. R. Mercer, "Alternative Paradigms for Assessment in a Pluralistic Society," in J. A. Banks and C.A.M. Banks, eds., Multicultural Education: Issues and Perspectives (Boston, MA: Allyn and Bacon, 1989), 289-304; Asa G. Hilliard, III, "The Strengths and Weaknesses of Cognitive Tests for Young Children," in J. D. Andrews, ed., One Child Indivisible (Washington, D.C.: The National Association for the Education of Young Children, 1975), 17-33; O. Taylor and D. L. Lee, "Standardized Tests and African Americans: Communication and Language Issues," in Asa G. Hilliard, III, ed., Testing African American Students: Special Re-Issue of the Negro Educational Review (Morristown, NJ: Aaron Press, 1991), 67-80. 12   David J. Clarke, "Open-Ended Tasks and Assessment: The Nettle or the Rose" (Paper presented to the Research Pre-Session of the 71st Annual Meeting of the National Council of Teachers of Mathematics, Seattle, WA, 29-30 March 1993); Stephen B. Dunbar and Elizabeth A. Witt, "Design Innovations in Measuring Mathematics Achievement" (Paper commissioned by the Mathematical Sciences Education Board, September 1993, appended to this report); See Patricia Ann Kenney and H. Tang, "Conceptual and Operational Aspects of Rating Student Responses to Performance Assessments" (Paper presented at the annual meeting of the American Educational Research Association, San Francisco, CA, April 1992) for a discussion of the issues involved in scoring written products in mathematics; Edward Silver and Suzanne Lane, "Balancing Considerations of Equity, Content Quality and Technical Excellence in Designing, Validating and Implementing Performance Assessments in the Context of Mathematics Instructional Reform: The Experience of the QUASAR Project (Pittsburgh, PA: Learning Research and Development Center, University of Pittsburgh, 1993). 13   See Eva L. Baker, Harold F. O'Neil, and Robert b Linn, "What Works in Alternative Assessment? (Sherman Oaks, CA: Advance Design Information, Inc., draft final report, September 1992). For discussion of broader cultural perspectives, see "Alternative Paradigms for Assessment in a Pluralistic Society"; "Some Pre-Requisites for the Establishment of Equitable, Inclusive Multicultural Assessment Systems"; "The Strengths and Weaknesses of Cognitive Tests for Young Children"; "Standardized Tests and African Americans: Communication and Language Issues." 14   Lloyd Bond, "Bias in Mental Tests," in B. F. Green, ed., New Directions for Testing and Measurement: Issues in Testing—Coaching, Disclosure and Ethnic Bias, no. 11 (San Francisco, CA: Jossey-Bass, 1981), 55-75; Nancy Cole and Pamela Moss, "Bias in Test Use," in R. L. Linn, ed., Educational Measurement, 3rd. ed. (New York, NY: American Council on Education/MacMillan Publishing Company, 1989), 201-217. 15   For more information on the kinds of collateral information needed to support inferences about equity from alternative assessments, see "Diversity, Assessment and Equity in Educational Reform"; "A Technological and

OCR for page 91
Measuring What Counts: A Conceptual Guide for Mathematics Assessment     Historical Consideration of Equity Issues Associated with Proposals to Change our Nation's Testing Policy"; "To Use their Minds Well: Investigating New Forms of Student Assessment." 16   "Diversity, Assessment and Equity in Educational Reform." 17   Ibid. 18   Center for the Study of Testing, Evaluation and Educational Policy, Boston College, "Progress Report: Urban District Assessment Coalition," September 1991-93; John Cawthorne, personal communication, April 1993. 19   "Diversity, Assessment and Equity in Educational Reform"; "What Works in Alternative Assessment?" 20   Maria Santos, personal communication, April 1993. 21   California Assessment Program, A Sampler of Mathematics Assessment (Sacramento, CA: California Department of Education, 1991), 16. 22   Richard Lesh and Susan Lamon, "Assessing Authentic Mathematical Performance," in Richard Lesh and Susan J. Lamon, eds., Assessment of Authentic Performance in School Mathematics (Washington, D.C.: American Association for the Advancement of Science, 1992), 35. 23   Richard Lesh et al., "Future Directions for Mathematics Assessment," in Assessment of Authentic Performance in School Mathematics, 417. 24   Ibid., 418. 25   Ibid., 416-417. 26   Stephen Jay Gould, The Mismeasure of Man (New York, NY: Norton, 1981), 146-158. 27   M.S. Smith and J. O'Day, "Systemic School Reform," in Susan H. Fuhrman and Betty Malen, eds., The Politics of Curriculum and Testing, the 1990 Year­book of Politics of Education Association (London, England: The Falmer Press, 1990), 233-267; Congressional Budget Office, Educational Achievement: Explanations and Implications of Recent Trends (Washington, D.C.: Government Printing Office, 1987). 28   For discussion of issues related to sanctions, see Richard L. Allington and Anne McGill-Franzen, "Does High-Stakes Testing Improve School Effectiveness?'' ERS Spectrum, 10:2 (1992), 3-12; Linda Darling-Hammond, "The Implications of Testing Policy for Quality and Equality," Phi Delta Kappan 73:3 (1991), 220-225; Linda Darling-Hammond, "Equity Issues in Performance-Based Assessment" (Paper prepared for the Symposium on Equity and Educational Testing and Assessment, Washington, D.C., 11-12 March 1993); Lorrie A. Shephard and Mary L. Smith, "Escalating Academic Demands in Kindergarten: Counterproductive Policies," The Elementary School Journal 89 (1988): 135-145; Lorrie Shepard, "Inflated Test Scores: Is the Problem Old Norms or Teaching to the Test?" Educational Measurement Issues and Practices 8 (1990): 15-22; Richard L. Allington and Anne McGill-Franzen, "Unintended Effects of Educational Reform in New York," Educational Policy 6:4 (1992), 397-414. 29   The National Council on Education Standards and Testing, Raising Standards for American Education: A Report to Congress, The Secretary of Education, The National Education Goals Panel, and the American People (Washington, D.C.:

OCR for page 91
Measuring What Counts: A Conceptual Guide for Mathematics Assessment     Author, 1992); M. Fleming and B. Chambers, "Teacher-Made Tests: Windows on the Classroom," in W. E. Hathaway, ed., Testing in the Schools: New Directions for Testing and Measurement (San Francisco, CA: Jossey Boss, 1983), 29-38; Leslie Salmon-Cox, "Teachers and Standardized Achievement Tests: What's Really Happening?" Phi Delta Kappan 62 (1981): 631-634; R. J. Stiggins and N. J. Bridgeford, "The Ecology of Classroom Assessment," Journal of Educational Measurement 22:4 (1985), 271-286; Linda Darling-Hammond and Arthur Wise, "Beyond Standardization: State Standards and School Improvement," The Elementary School Journal 83 (1985): 315-336. 30   "Equity Issues in Performance-Based Assessment"; "A Technological and Historical Consideration of Equity Issues"; Raising Standards for American Education; National Commission on Testing and Public Policy, From Gatekeeper to Gateway: Transforming Testing in America (Chestnut Hill, MA: Author, 1990). 31   Commission on Chapter I, Making Schools Work for Children in Poverty: A New Framework Prepared by the Commission on Chapter I (Washington, D.C.: American Association for Higher Education, December 1992). 32   Ibid.; National Forum on Assessment, "Criteria For Evaluation of Student Assessment Systems, Educational Measurement: Issues and Practice 11:1 (1992), 32. 33   Diana Pullin, "Legal and Ethical Issues in Mathematics Assessment" (Paper commissioned by the Mathematical Sciences Education Board, September 1993, appended to this report). 34   Raising Standards for American Education. 35   Michael T. Nettles et al., Leadership Statement of Nine Principles on Equity and Educational Testing and Assessment, 12 March 1993; Diversity and Equity in Assessment Network, Guidelines for Equitable Assessment (Cambridge, MA: Fair Test, 1993). 36   Learning Research and Development Center, University of Pittsburgh and National Center on Education and the Economy, New Standards Project (Pittsburgh, PA: Author, 1993), 11. 37   Andrew C. Porter, "School Delivery Standards," Educational Researcher 22:3 (1993), 24-30. 38   "A Technological and Historical Consideration of Equity Issues Associated with Proposals to Change Our Nation's Testing Policy; "Equity Issues in Performance-Based Assessment"; "Some Pre-Requisites for the Establishment of Equitable, Inclusive Multicultural Assessment Systems." 39   Lonnie Harp, "Texas Finance Bill Signed Into Law, Challenges Anticipated," Education Week, 9 June 1993; Lonnie Harp, "Impact of Texas Finance Law, Budget Increase Gauged," Education Week, 16 June 1993; Millicent Lawton, "Alabama Judge Sets October Deadline for Reform Remedy," Education Week, 23 June 1993. 40   "National Standards, Assessment and Equity," 7. 41   Leadership Statement of Nine Principles on Equity and Educational Testing and Assessment.

OCR for page 91
Measuring What Counts: A Conceptual Guide for Mathematics Assessment This page in the original is blank.