CONCRETE BRAIN STORM
I decided that rather than wishing for the moon, I should try to be constructive. So we have come to the section of my outline headed “Concrete Brain Storm.” You can decide whether to put a hyphen before “Brain.”
I suggest that we think in terms of site visiting teams. I envision a cross between the kind of team that the National Institutes of Health have sent out to research sites, and whatever the National Aeronautics and Space Administration must send around to make sure that its contractors are doing sensible things. I think of a group of perhaps five visitors—three technical, one curriculum, and one policy type. They would have to work half-time; no one could stand to do the work full-time. There is a lot of hard work to do at each site. It is not a matter of one plane in and the next plane out. The visitors would have to read in advance anything that could be supplied on the goals, the instrument development, the scoring rubrics, the statistics from pilot work, and the sorts of decisions that are being made by the assessment. They might be able to specify some additional studies to be run in advance of the visit. When they get to the site, the visit is sure to change their image of what is going on; one cannot get everything in proportion from a distance. New questions will surface for research follow-up.
The team would be engaged in cross-pollination. Going from site to site, it can warn State X of the mistakes it saw in State Q. It would be doing in-service education, both for the members of the state agency that is running the assessment and for the contractors. Contractors are a key part of today's testing enterprise, and they are badly overstrained, just as the state testing agencies are overstrained.
We have an enormous amount of educating to do because even the experts are off balance. My story about the standard errors for CLAS could surely be duplicated dozens of times. As we get insight, there is going to be a tremendous need to disseminate it to all assessments. We don't have a natural mechanism for the transfer of so much information about an enterprise in which the stakeholders are so diverse. So this is a major challenge. I am glad you are working on the job.
