SCHECHTER: Well, I would say this. [For] the review process OMB wants to see as many of those scenarios as possible, from those reviews of the instrument, since all of that influences burden. So we’re interested in those observations from the field. But I don’t know how that would solve the documentation problem; it’s more of an enhancement to the process.
BANKS: I think it’s a part of the documentation; on top of the original, you have an “excuse list” explaining why a change got made.
GROVES: A completely different question: on your slides on testing, you had a role for methods researchers and for analysts. Is that a role that goes beyond just checking that the specs they delivered are being executed?
DOYLE: That’s correct; there is a role beyond that.
GROVES: What is it? Are you letting them re-design in the middle of the go?
DOYLE: In the course of developing, our experience has been that we start out and say, “Do this question.” And we implement the question. And then we start to test it and find that, “Oooooo, this isn’t quite working like I thought it would.” So let’s go back and change the specification so it works slightly differently. It also may happen that we’re implementing the automated instrument at the same time that we’re doing cognitive testing, so that while we’re doing cognitive testing and find that these words don’t mean exactly what we thought they would, let’s change them to these other words. And you have to go back and embed it in there. So some of it is an overlap, to try to reduce the overall length of time it takes to do all of this work.
MANNERS: Just a comment; I think it’s important, to follow up on Cathy’s question, to build in your documentation plans while you’re planning and building the instrument, and to build in links to external databases that give rationale and reasons for items.
DOYLE: Absolutely. And one of the benefits that we derive from using the database management systems to develop specifications is that they are bigger than just the specs for the instrument—they have instructions for post-collection processing, so you can tie in that work. And eventually they’re going to be able to accept back in statistics from the fielding, so you get all your frequency counts. There [are] also ways to link back to other documents that provide the research for, say, the cognitive testing for question x. It’s more than just solving the instrument documentation problem.
PARTICIPANT: I think that one thing we seem to be increasingly coming to [is that] at the same time that we’re trying to get better at enforcing having all the specs, and having better specification, I think it