CONCRETE BRAIN STORM

I decided that rather than wishing for the moon, I should try to be constructive. So we have come to the section of my outline headed “Concrete Brain Storm.” You can decide whether to put a hyphen before “Brain.”

I suggest that we think in terms of site visiting teams. I envision a cross between the kind of team that the National Institutes of Health have sent out to research sites, and whatever the National Aeronautics and Space Administration must send around to make sure that its contractors are doing sensible things. I think of a group of perhaps five visitors—three technical, one curriculum, and one policy type. They would have to work half-time; no one could stand to do the work full-time. There is a lot of hard work to do at each site. It is not a matter of one plane in and the next plane out. The visitors would have to read in advance anything that could be supplied on the goals, the instrument development, the scoring rubrics, the statistics from pilot work, and the sorts of decisions that are being made by the assessment. They might be able to specify some additional studies to be run in advance of the visit. When they get to the site, the visit is sure to change their image of what is going on; one cannot get everything in proportion from a distance. New questions will surface for research follow-up.

The team would be engaged in cross-pollination. Going from site to site, it can warn State X of the mistakes it saw in State Q. It would be doing in-service education, both for the members of the state agency that is running the assessment and for the contractors. Contractors are a key part of today's testing enterprise, and they are badly overstrained, just as the state testing agencies are overstrained.

We have an enormous amount of educating to do because even the experts are off balance. My story about the standard errors for CLAS could surely be duplicated dozens of times. As we get insight, there is going to be a tremendous need to disseminate it to all assessments. We don't have a natural mechanism for the transfer of so much information about an enterprise in which the stakeholders are so diverse. So this is a major challenge. I am glad you are working on the job.



The National Academies | 500 Fifth St. N.W. | Washington, D.C. 20001
Copyright © National Academy of Sciences. All rights reserved.
Terms of Use and Privacy Statement



Below are the first 10 and last 10 pages of uncorrected machine-read text (when available) of this chapter, followed by the top 30 algorithmically extracted key phrases from the chapter as a whole.
Intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text on the opening pages of each chapter. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.

Do not use for reproduction, copying, pasting, or reading; exclusively for search engines.

OCR for page 15
A Valedictory: Reflections on 60 Years in Educational Testing CONCRETE BRAIN STORM I decided that rather than wishing for the moon, I should try to be constructive. So we have come to the section of my outline headed “Concrete Brain Storm.” You can decide whether to put a hyphen before “Brain.” I suggest that we think in terms of site visiting teams. I envision a cross between the kind of team that the National Institutes of Health have sent out to research sites, and whatever the National Aeronautics and Space Administration must send around to make sure that its contractors are doing sensible things. I think of a group of perhaps five visitors—three technical, one curriculum, and one policy type. They would have to work half-time; no one could stand to do the work full-time. There is a lot of hard work to do at each site. It is not a matter of one plane in and the next plane out. The visitors would have to read in advance anything that could be supplied on the goals, the instrument development, the scoring rubrics, the statistics from pilot work, and the sorts of decisions that are being made by the assessment. They might be able to specify some additional studies to be run in advance of the visit. When they get to the site, the visit is sure to change their image of what is going on; one cannot get everything in proportion from a distance. New questions will surface for research follow-up. The team would be engaged in cross-pollination. Going from site to site, it can warn State X of the mistakes it saw in State Q. It would be doing in-service education, both for the members of the state agency that is running the assessment and for the contractors. Contractors are a key part of today's testing enterprise, and they are badly overstrained, just as the state testing agencies are overstrained. We have an enormous amount of educating to do because even the experts are off balance. My story about the standard errors for CLAS could surely be duplicated dozens of times. As we get insight, there is going to be a tremendous need to disseminate it to all assessments. We don't have a natural mechanism for the transfer of so much information about an enterprise in which the stakeholders are so diverse. So this is a major challenge. I am glad you are working on the job.

OCR for page 15
A Valedictory: Reflections on 60 Years in Educational Testing This page in the original is blank.