Grading the Nation's Report Card Research from the Evaluation of NAEP (2000) / Chapter Skim
Currently Skimming:
11 Developing Classroom Process Data for the Improvement of Teaching
11 Developing Classroom Process Data for the Improvement of Teaching
Pages 229-264
The Chapter Skim interface presents what we've algorithmically identified as the most significant single chunk of text within every page in the chapter.
Select key terms on the right to highlight them within pages of the chapter.
From page 229... ...
Indeed, all attempts to improve education must of necessity at some point be mediated through the classroom. This is obvious because classroom practice represents the most direct means for affecting student outcomes.
|
From page 230... ...
If change in student learning outcomes is observed in the tests, we still need to know whether change is due to something going on in the classroom or something independent of that. In this paper, we make the assumption that classroom process data, especially when collected in conjunction with student achievement data, can play a critical role in efforts to improve education.
|
From page 231... ...
In the process-product research of the 1970s and 1980s, it was demonstrated, across many studies, that student learning of mathematics was significantly associated with rapid coverage of a large number of problems during the lesson: the more problems the teacher led students through, and the faster the pace, the more students learned as measured by achievement tests (Leighton, 1994; Leinhardt and Putnam, 1987~. As often as this effect was found, however, it turned out not to hold up in cross-cultural comparisons.
|
From page 232... ...
What are the processes by which students learn from classroom instruction, and what characteristics of different instructional systems affect how much students learn? These are key questions, as our interest in instruction rests on the assumption that student learning is affected by instruction.
|
From page 233... ...
Not only do we need both qualitative and quantitative data, we also need a way to link the two kinds of data together. As we will see, this has been a problem with more traditional approaches to the study of classroom processes.
|
From page 234... ...
Questionnaires and rating scales Questionnaires and rating scales are often used to tap classroom processes. Questionnaires and rating scales used for these purposes typically request information from teachers about the activities taking place in their classrooms.
|
From page 235... ...
Generally, Perry found that problem assignment was related to student learning; more specifically, she found that spending a great deal of time on a few difficult problems led to better student achievement than covering many problems, especially problems that most students could solve before receiving instruction. In this study, a diary of what instruction consisted of was used to make inferences about teaching practices that were related to learning outcomes.
|
From page 236... ...
In general, the process-product approach assesses classroom processes or their proxies and relates these to student outcomes.
|
From page 237... ...
Survey data can describe the major dimensions of classroom processes and how they vary across course levels and types of schools. National survey data, collected periodically, can document trends in teachers' use of generic instructional strategies.
|
From page 238... ...
Lack of Shared Language Related to the problem of not obtaining a valid picture of classroom practices with typical paper-and-pencil survey instruments is that these instruments require an evaluation of whether teachers understand the items in the way they were intended. However, for this we need a common language that we really do not have.
|
From page 239... ...
Why not? Because to improve practice concrete new ideas about classroom practice are needed.
|
From page 240... ...
The narrative notes typically are summarized and/or coded for the occurrence or absence of specified or interesting events that emerge from reading the narrative descriptions. If the observers take notes with enough detail, this method has the potential to yield multiple analyses on a variety of classroom practices.
|
From page 241... ...
Relative to survey methods, where the questions for investigation are fairly well specified before data are collected, the questions for investigation often emerge from careful reading of the data when using methods relying on narrative observation. This makes the cost of analysis including developing coding systems, training coders to be reliable, and so forth very expensive.
|
From page 242... ...
VIDEO RECORDS OF CLASSROOM INSTRUCTION Video has been used for many years for the study of classroom processes. However, until recently, it was primarily used for small-scale qualitative studies, often focusing on a single teacher.
|
From page 243... ...
Psychologists, anthropologists, sociologists, and others interested in understanding classroom processes can all make some use of a single video dataset.
|
From page 244... ...
Going back to the video enabled us to see that Japanese teachers asked their students for complete descriptions of how they solved a problem, whereas U.S. teachers asked students to justify specific steps in a solution.
|
From page 245... ...
Once video is digital, many tasks that were nearly impossible to accomplish using videotape can now be accomplished easily. Digital video, in contrast to analog video, can be stored in various formats and storage mediums.
|
From page 246... ...
It is also possible to restrict the use of video datasets and require researchers who wish to access the data to sign nondisclosure agreements. This is not an ideal solution, however, as it means that video images cannot be used as a means of communicating study results to the public.
|
From page 247... ...
In the TIMSS video study we used two forms of intermediate representations: a transcript and a lesson table. The transcript simply consists of a written
|
From page 248... ...
Similarly, the lesson table provides a more content-oriented representation of the lesson as it unfolds over time. In the lesson tables constructed for the TIMSS video study, we wrote down the organization of the classroom (e.g., classwork or seatwork)
|
From page 249... ...
If, however, we need a picture of teaching that is reliable at the individual teacher level, it probably will be too expensive. In general, video data and the more traditional kinds of data can both play an important role in a portfolio of classroom process data.
|
From page 250... ...
Given that policy makers and researchers share these critical features, which means that the ways in which they can use data are distinctively different from the way in which teachers use data, policy makers and researchers are considered together in this section. Researchers studying teaching and learning in classrooms are interested in such questions as: What kinds of instructional practices lead to improvements in student learning?
|
From page 251... ...
To understand why this happens, Cohen and Hill developed a model. Their model has student learning, as measured by student achievement, as the ultimate dependent measure of instructional policy.
|
From page 252... ...
Based on Cohen's work, a new model of policy implementation one that includes adding and deleting teaching practices should be put forth. Of course, we would not have come to this conclusion without collecting data on classroom practices.
|
From page 253... ...
In this example of generating models that are useful for policy makers, reliable outcome measures need to be used, too. Thus, for this example to work well, we would recommend that the video records be combined with survey methods that include student achievement measures, such as the NAEP, to support the inferences that could be used by policy makers.
|
From page 254... ...
In sum, video data can provide a shared set of examples for building language and theories for analyzing classroom practices. Data Needed to Test and Validate Theoretical Models The data most useful to policy makers are probably those that say whether or not teachers have implemented the stated policy and, if so, what the impact of the implementation has been on student achievement.
|
From page 255... ...
The general point we wish to make is that the data we share with the public need to be accessible and the data need to communicate or demonstrate clearly what can be learned. Recommendations Classroom process data relevant to the needs of researchers and policy makers are scant.
|
From page 256... ...
Student outcome data must be linked into this effort, but outcome data alone will not be enough to understand how policies work. In particular, we propose that video surveys be used, in conjunction with more traditional surveys, to study classroom processes.
|
From page 257... ...
Video data, especially those that are collected outside our own country, can serve this function of generating new ideas and new hypotheses about teaching. DATA FOR CLASSROOM PRACTITIONERS We have described the role that data can play in helping researchers and policy makers understand the chain of influence that relates policy to classroom practice to student learning.
|
From page 258... ...
Generating localized theories of teaching will require prolonged reflection and discussion of examples of classroom practice. Video can play a central role in these discussions because it allows what is normally a complex and transitory phenomenon to be slowed down and replayed for study.
|
From page 259... ...
This shared language is critical as it becomes the foundation on which localized theories of teaching can be stored, accessed, and communicated about with other teachers. In the process we envision by which teachers could use classroom videos, it is interesting to ponder what kinds of examples ought to be collected.
|
From page 260... ...
Of course, it is not feasible to videotape in every classroom included in NAEP, but collecting video records of lessons in a substantial subsample of NAEP classrooms is both practical and useful. Using techniques similar to those in the TIMSS video study, videotaping in national samples of classrooms can provide the first reliable means of tracking changes in instructional practices over time.
|
From page 261... ...
Another use of video surveys in NAEP should be to aid in the development and validation of better traditional measures of classroom practices such as questionnaires. A well-designed sample of video data could serve both immediate research purposes and instrument development purposes, provided the two are integrated in their conception and design.
|
From page 262... ...
We must, before we collect any data at all, develop an understanding of how the data will be used, and by whom, to improve education. We have ruminated on how classroom process data might be used by policy makers, researchers, and classroom practitioners, but this is only the beginning.
|
From page 263... ...
Unpublished manuscript, Policy Studies Associates, Inc., Washington, D C Leinhardt, G., and R.T. Putnam 1987 The skill of learning from classroom lessons.
|
From page 264... ...
1996 Large-scale video surveys for the study of classroom processes.
|
Key Terms
This material may be derived from roughly machine-read images, and so is provided only to facilitate research.
More
information on Chapter Skim is available.