IV. A Framework for Addressing Concerns

By this point, it should be apparent that there is a particular complexity to assessment-based professional development. Participating teachers learn about new directions in assessment while they engage in repeated applications of the assessment process.

From the start in assessment-focused groups, teachers are challenged to sharpen their skills in gathering and using evidence to make valid inferences. However, the varied underlying concerns that teachers bring to their group work will have a bearing on their attention to the challenges and their capacities to meet the challenges. For example, it would be unwise for a group facilitator to concentrate on teachers' judgment about the appropriateness of tasks if the dominant concern in the group is to get more information about the different kinds of tasks being used in alternative-assessment initiatives.

Those who design assessment working groups need to account for the variety of concerns in their planning. This section describes some common teacher concerns about alternative assessment and a framework to use in listening to and responding to the concerns.

Assuming that there will be a group of teachers working together on assessment, it may be that experience with assessment, and therefore the concerns, will vary within the group. And, no matter what the variation, concerns in a group will change over the course of time. Planning for the group activities needs to take variation and change into account. Whether the driving purpose is to improve classroom instruction or to align what happens in



The National Academies | 500 Fifth St. N.W. | Washington, D.C. 20001
Copyright © National Academy of Sciences. All rights reserved.
Terms of Use and Privacy Statement



Below are the first 10 and last 10 pages of uncorrected machine-read text (when available) of this chapter, followed by the top 30 algorithmically extracted key phrases from the chapter as a whole.
Intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text on the opening pages of each chapter. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.

Do not use for reproduction, copying, pasting, or reading; exclusively for search engines.

OCR for page 33
Learning About Assessment, Learning Through Assessment IV. A Framework for Addressing Concerns By this point, it should be apparent that there is a particular complexity to assessment-based professional development. Participating teachers learn about new directions in assessment while they engage in repeated applications of the assessment process. From the start in assessment-focused groups, teachers are challenged to sharpen their skills in gathering and using evidence to make valid inferences. However, the varied underlying concerns that teachers bring to their group work will have a bearing on their attention to the challenges and their capacities to meet the challenges. For example, it would be unwise for a group facilitator to concentrate on teachers' judgment about the appropriateness of tasks if the dominant concern in the group is to get more information about the different kinds of tasks being used in alternative-assessment initiatives. Those who design assessment working groups need to account for the variety of concerns in their planning. This section describes some common teacher concerns about alternative assessment and a framework to use in listening to and responding to the concerns. Assuming that there will be a group of teachers working together on assessment, it may be that experience with assessment, and therefore the concerns, will vary within the group. And, no matter what the variation, concerns in a group will change over the course of time. Planning for the group activities needs to take variation and change into account. Whether the driving purpose is to improve classroom instruction or to align what happens in

OCR for page 33
Learning About Assessment, Learning Through Assessment the classroom with externally administered testing, assessment-focused professional development will be experienced by many teachers as an innovation that is nudging, if not requiring, them to change their practice. There is a well-established framework (Loucks-Horsley and Stiegelbauer, 1991) for thinking about the stages of concern that adopters of innovations typically pass through, and that framework, the Concerns-Based Adoption Model (CBAM) is relevant here. Briefly, the framework's stages of concern span transitions from initial awareness of an innovation all the way to exploring how to refocus and improve the innovation. The stages are shown in Table 2. To capture the flavor of concerns that teachers express about assessment, we interviewed ten people with extensive experience in designing and conducting assessment-focused staff development. The interviews yielded a set of concerns often heard by these group leaders, which we have sorted using the CBAM framework. (See Table 3.) Some of the concerns could sit in more than one category. For example, the bottom, italicized quote seems primarily to be concerned about consequence for the students, but also seems to foreshadow concerns about collaboration with other teachers. No concerns were expressed that seemed to fit under Refocusing. An example of a refocusing concern voiced by teachers once they have worked with innovations in assessment for a while might be, "Will the district support us in using grade-by-grade student work analyses to redefine the district's performance standards?" Table 2. Stages of concern Stage 0. Awareness concerns: Basic awareness about the innovation. Stage 1. Informational concerns: Focus on learning more detail about the innovation. Stage 2. Personal concerns: Focus on individual's role, the demands of the innovation, and adequacy to meet demands. Stage 3. Management concerns: Focus on efficiency, organization, management, time, best use of resources. Stage 4. Consequence concerns: Focus on impact on students. Stage 5. Collaboration concerns: Coordination and cooperation with others in use of the innovation. Stage 6. Refocusing concerns: Exploration of more powerful alternatives. From Loucks-Horsley and Stiegelbauer, 1991.

OCR for page 33
Learning About Assessment, Learning Through Assessment Stages 0 and 1. Awareness and Informational concerns How might a group facilitator plan for addressing needs for awareness and information? One economical way is to organize the information around the four shifts in assessment mentioned earlier. These were shifts away from basing inferences on single sources of evidence and toward basing inferences on multiple and balanced sources of evidence; away from reliance on comparing students' performance with that of other students and toward reliance on comparing students' performance with established criteria; away from relying on outside sources of evidence and toward a balance between these sources and evidence compiled by teachers; and away from a preponderance of assessment items that are short, skill-focused, single-answer, and decontextualized, toward a greater use of tasks that are context-based; open to multiple approaches and, in some cases, to multiple solutions; complex in the responses they demand—e.g., in communication, Table 3. Assessment-related concerns 0. Awareness   1. Informational ''What are rubrics?'' "Portfolios?" "What are examples of the kinds of alternative tasks that are being used?" 2. Personal "What are the newspapers going to do with the information?" "What conclusions are going to come out of this, and how and where is the information going to be used?" "How do I talk with parents about the changes?" "Will my administrators support me?" "How do I deal with transition? I'm using fewer traditional tests, but don't yet have an adequate system to replace it." 3. Management "Where can I find the time to fit these tasks in? to score using rubrics? to handle portfolios?" "If a simple grade isn't sufficient, how do I report in a way that is clear, concise, but not backbreaking for me?" "With all the open-ended tasks, how can I be sure what's a right answer?" 4. Consequence "Will there be some continuity between grades, so that my students don't only see something different in my classroom and not in others?" "Do the new assessments really tell us what we need to know?" "I'm afraid we're de-emphasizing rigor in favor of student inventiveness." "How can you score a piece 'proficient' if the student doesn't get the right answer?" 5. Collaboration "Will there be some continuity between grades, so that my students don't only see something different in my classroom and not in others?" 6. Refocusing  

OCR for page 33
Learning About Assessment, Learning Through Assessment representation, and level of generalization; and drawn from a wide spectrum of mathematics concepts and processes. The first implies that attention should be given to defining and providing examples of different sources of evidence—e.g., portfolios, short-answer tasks, performances, student self-assessments, observations. The second implies that attention be given to understanding and reaching consensus about the criteria that are used to judge quality; therefore, examples of scoring rubrics and performance standards need to be provided and discussed. The third implies that information should be provided about efforts in several states and districts to complement test data with other data sources, such as portfolios. The fourth implies that teachers should see and discuss a variety of mathematics tasks. "Another question teachers have is, where do the problems come from? Where can I find good performance tasks? What do I do if I don't have one that fits what I'm teaching? Often teachers don't feel they have access to the resources they need. That's the biggest hit in workshops, is giving people collections of problems for them to use as resources in getting started." District leader of test-change efforts Stage 2. Personal concerns Once concerns for basic information are addressed, and as assessment-focused groups progress, teachers express other, more personal concerns related to the impact on their own experience and on the values they hold dear. The challenges to teachers' judgments about the quality of mathematics, discussed earlier, are particularly likely to raise personal concerns. Understandably, individual feelings of inadequacy about mathematical understandings or skills will raise personal concerns in settings where open discussion is invited and expected. Drawn from the experience of the educators interviewed for this document, the teachers' expressed concerns reveal a particular set of worries about how vulnerable new assessments will make them. Concerns about support are common, and they include the administrative net underneath, the buy-in by teachers in grades before and after, and the availability of helpful resources. Concerns about smooth transition from an old system to a new one are strongly felt. This is true of individual teachers trying to make changes in their own practice, but it is especially true in settings where widespread assessment change is underway or pending, where concerns include the use of information in and by the community, the handling of community questions, and the use of information to improve programs. "Designing alternative forms of assessment is a real tricky thing. If you believe, if you've been convinced, that these forms of as-

OCR for page 33
Learning About Assessment, Learning Through Assessment sessment are the way to go, then getting the skills to do that kind of assessment is very necessary and requires time. You can't just say, I believe this is a good way to go and so it's going to just happen. We didn't see these forms of assessment when we were in school, we didn't see them in teacher preparation programs, and we haven't ever used them in our classrooms before. Teachers need to not give up when at first an assessment doesn't succeed. A lot of the time, you try something, and it doesn't work at first and it turns out it wasn't assessing what you thought you were assessing. These assessments are new, and different, and teachers ought to have the right to work on it, and improve on it. They need the space to try it without being evaluated on it right away, for instance." Teacher veteran of portfolio-scoring teams "I find it important to talk about taking a long-range view of assessment change. I don't talk about it in the usual one-year time frame, but more in terms of three to five years. I tell teachers, If this stuff is really profound, it may not show up right away." District associate superintendent "Another question teachers have is what are parents going to say, related to how their students will do on standardized tests. If we don't test that way, when students get to standardized tests, how are they going to deal with the multiple choice questions? What if the scores aren't there? These are concerns of both parents and administrators." High-school mathematics teacher "One big question from teachers is, how do we handle concerns and questions from the parents and the community, especially in California or other states where testing is a big political issue?" Urban district mathematics supervisor "My biggest concern right now is how do we educate the public, especially when the assessment is a visible thing. When changes are happening in the classroom, it can be handled, especially through parent-teacher communication: work gets done, feedback is given, work goes home, and there's a gradual awareness that builds." Urban district mathematics supervisor For those facilitating assessment groups, there are a few important considerations in planning to address these personal concerns: Provide regular opportunities for participating teachers to discuss their progress and concerns. Use small-group explorations of the mathematics, with an emphasis on the building of knowledge in the groups, to alleviate individual feelings of inadequacy around the mathematics. Plan to advocate for the longer view in setting the tone with the teachers, to advocate for a commitment to exploration and experiment, and to resist requests to "show us how."

OCR for page 33
Learning About Assessment, Learning Through Assessment Find ways to elicit concerns and make them explicit, but be candid and realistic about the possibilities of addressing them within the group. Separate the teachers' capacity building—especially the sharpening of judgment about tasks, student work, and consequent actions—from techniques for public relations. The latter may be of interest to participants, but they are no substitute for the former. Give concrete reasons for becoming proficient in assessing students' learning. Plan that, over time, the group will select illustrative examples of tasks and student work for purposes of communication about assessment changes. Good examples and informed commentary can be the most effective kind of public relations for assessment in mathematics. Stage 3. Management concerns Once teachers invest themselves in trying to incorporate alternative approaches to assessment in their own practice, management concerns arise, related to time management; the application of criteria; and reporting procedures. "The question, and I hear this one all the time, is where do you get the time, as a classroom teacher, to score all these papers? And not only to score them, but read them and process what the kids are saying? To be honest, when I was a teacher way back when, most of the time, I'd just check the 'answer column.' I hardly ever looked at the student work. And I think that's what most teachers do. Where do you find the time to read responses, make comments, give feedback? Like using student journals is a terrific idea, but teachers want to know, when am I going to find time to do this?" Urban district mathematics supervisor "The most frequent question I hear from other teachers is, how is this going to impact everything else I have to do, since this is going to take longer? . . . The big issue is time, and time in the perspective of, if I spend time on this stuff, how am I going to be able to cover the curriculum? . . . The time issue usually comes up in terms of the coverage of the curriculum. And I usually have to answer that by saying, ultimately we're going to have to grab stuff out of the curriculum. Because it's a legitimate concern, how can you do this stuff, and still cover the same curriculum." High-school mathematics teacher In our experience and in the experience of those we consulted, one of the most effective ways to address management concerns, once they have been elicited, is to ask teachers to address the group who have found ways to address particular management concerns. This approach increases the number of effective strategies available to the group, going beyond the personal experience of the facilitator.

OCR for page 33
Learning About Assessment, Learning Through Assessment In addition, there are helpful print resources available, such as Petit (1992), Stenmark (1991), and Tsuruda (1994). Stages 4 and 5. Consequence and collaboration concerns Even as they grapple with management concerns, teachers who are invested in learning through assessment and adopting alternative approaches to assessment develop concerns related to how their changes in assessment will affect the quality of their classroom work and the learning of their students. Two prominent clusters of such concerns are content and equity. Content. An essential component of learning through assessment is exposure to a variety of mathematics tasks. As they become familiar with tasks that require student performance, or that are open or complex, teachers often express concerns about validity, rigor, and quality. "As we move to using new forms of assessment, the question I hear most often from teachers, and from parents, is 'Where's the math?' I think people can often see how these open-ended problems do address the first four NCTM standards: communication, problem solving, reasoning, and connections. But a lot of the time, they have this question: How do you know that students know the basic skills or are able to compute? They have a sense that there's not a good balance." Urban district mathematics supervisor "Teachers ask, where do the basic skills get taught in relation to performance assessments? Before, during, after? Do performance assessments address the teaching of basic skills? In other words, and this is a question for a lot of parents, can we assume that a child that can perform on a high level on a performance assessment knows thebasic skills?" Director of assessment-based teacher program "This is a more subjective way of assessing than multiple choice, in the sense that it relies on a subjective judgment. So a question teachers have is, is that fair? And of course, students have that question too. I think teachers can get around that by talking about the rubric before the assessment. On the other hand, . . . I have yet to write a rubric that I didn't need to revise once I saw the students' papers." Teacher veteran of portfolio-scoring teams Here is an instance where a concern about assessment has aspects of more than one category to it: consequence (Will the students get shortchanged?) and informational (Where is the rigor in these new assessment systems?). Both aspects would need attention in the professional development. Equity. The simplicity of the 4-part, cyclical diagram for the assessment process belies the complexity of assessment as a continuum of practice. On one end of the continuum, teachers must

OCR for page 33
Learning About Assessment, Learning Through Assessment attend to students' individual interests and ways of thinking, and learn to incorporate portfolios and project work in such areas as data study, mathematical model of physical systems or phenomena, design of physical structures, management and planning analysis, pure mathematical investigations, and histories of mathematical ideas. (See, e.g., the performance standards categories developed by the New Standards Project, 1995.) While portfolios and projects provide avenues for individual expression in assessment, there is another end to the continuum, where assessors must attend to things that society has deemed important and to standard ways of knowing. For example, society expects that, no matter their background, students will be able to perform well on tasks that relate to desired levels of literacy and numeracy in the citizenry, such as the task in Figure 12, administered to 17 year-olds in the National Assessment of Educational Progress (NAEP) and on which only three percent responded at the satisfactory level or better (Lindquist et al., 1995). In between these two ends of the continuum are a range of individualized expressions of socially important ideas. This range appears to be growing in most classrooms, but it is especially wide in classrooms where teachers see a broad diversity in individual interests or, especially where many cultures are represented, they see a variety of socially important ideas. It becomes important for designers of professional development to look at assessment innovations through the lens of teachers' equity concerns. These concerns become especially relevant, and important to elicit, when judgments about task appropriateness and accessibility are at issue. For example, we have found it advantageous for teachers to discuss equity issues around the use of open tasks that invite students to explore and construct responses. Some teachers believe that some students require structured tasks all the time, and so should be protected from open tasks. This belief can, and should, be questioned and alternative perspectives should be considered. "I'm an advocate of open-ended problems because I think they allow for more equity. It's still a new experience for all students to have problems that could have multiple entry points and multiple solutions, problems that allow for sophisticated response or a basic response. If the problems are presented correctly, then everyone can have a stab at it. In contrast, if a problem depends on a certain technique, then if a student can't do it, that's it? Urban high-school mathematics teacher Alternative assessment as an innovation is relatively young. Using the stages-of-concern lens on the concerns reported in our interviews, however, we can infer by the quantities of concerns in the management and consequences categories that the innovation is beginning to take root. At the same time, the educators we

OCR for page 33
Learning About Assessment, Learning Through Assessment One plan for a state income tax requires those persons with incomes of $10,000 or less to pay no tax and those persons with income greater than $10,000 to pay a tax of 6 percent only on the part of their income that exceeds $10,000. A person's effective tax rate is defined as the percent of total income that is paid in tax. Based on this definition, could any person's effective tax rate be 5 percent? Could it be 6 percent? Explain your answer. Include examples if necessary to justify your conclusions. Figure 12. A problem from NAEP From the National Assessment of Educational Progress, as cited in Lindquizt, Dossey, & Mullis, 1995. interviewed report a range of concerns in the earlier categories, as well, implying that planning for assessment-based staff development should proceed from a concerns-based perspective, and allow participating teachers over time to begin reconsidering their perspectives on assessment, and to consider more powerful alternatives. The optimal staff-development event is one that addresses current concerns while it foreshadows future concerns. In any case, it takes time to adopt and internalize new perspectives on assessment, and it takes considerable support. In the final section of the document, we advocate for actions that school administrators and others can take so that teachers get the necessary time and support.