patterns or one question with five variations on wording, depending on the characteristics of the response—it could go either way. One of those is connected to this type of testing, the other is one that takes a lot of time. Would you recommend that we choose the five-question version in order to adapt your type of model-based testing? Or would you …

ROBINSON: Well, I think that they both can be modelled. What you would end up doing is …

DOYLE: Having to model all the many variations?

ROBINSON: … The variation on the verbiage; can you give me an example?

DOYLE: There was a question, if you remember, on the thing Tom Piazza put out that if TCOUNT=1 you get asked this question and if greater than 1 you get asked that question? And on the screen it has the same name so it’s technically the same question but the words that appear on the page are going to be different depending on whether there’s an actual TCOUNT.

ROBINSON: Yes, those both sound very modelable. What you could do is choose one. And then if later on you decide to change it it is very easy to change the model to adapt. Because what you do is change that part and run it all, just as—I’m a tester congenitally here—somebody before gave a choice that said is your income between 500 and 1000 or between 1000 and 2000, or something …

DOYLE: And missing something in between …

ROBINSON: Yeah, where does 1000 go? No tester would have let you go on that...

DOYLE: Cognitive psychologists would put that there …

PARTICIPANT: I think this is a very important question about the demographics . . .

ROBINSON: Oh, my God! I wasn’t even here! [laughter]

PARTICIPANT: What do you do with your English majors?

ROBINSON: What do I do with the English majors? I’m a religion major.

PARTICIPANT: Do you rephrase it?

ROBINSON: No; what I’m doing is putting together a talk about how liberal arts majors make the best testers . . .

PARTICIPANT: … because they’re logical?

ROBINSON: They’re logical, and they delve. Remember when you used to sign up for a course because, you know, we’re going to delve into Shakespeare or something. That’s what you need—somebody who will get in there and say, “what are we actually doing?” Those are the right questions, and the right questions asked early enough can save a lot of trouble downstream.



The National Academies | 500 Fifth St. N.W. | Washington, D.C. 20001
Copyright © National Academy of Sciences. All rights reserved.
Terms of Use and Privacy Statement