at times during the development process that are not opportune. And that’s where we get into trouble.

I think we have a sense of an appropriate design-develop-test model that you provided, and we would love to implement it. But, in our lives, the requirements are really not set at the beginning. And not at the level of all the questions, and not at the level of all the flows.

So, just within the instrument piece, we live in a world where we are constrained by reality—the laws change when we’re halfway through developing an instrument and suddenly our questions are irrelevant. That’s the kind of thing that I can see—from your [Poore’s] description, it sounds like that’s controlled better when you’re developing a product, and know what the product is. But in our case it doesn’t seem to be under control. We need to find a way to live within that unpredictability.

GROVES: My third meeting like this one was a wonderful day spent in Palo Alto, overlooking the ocean as I recall, with a set of software engineers for expert systems. And the question on the table was, “Why can’t we develop an expert system for questionnaire construction?” That was what went on. It was a great day, we had a lot of fun. The conclusion of the expert systems engineers, at the end of the day, was, “Perhaps you people would like to define how to do a questionnaire before you talk to us next time.” [laughter] And partly that’s it. And the immediate reaction is, “Why don’t you standardize this stuff? That’s stupid; you’ve been doing this for thirty years, or fifty years. Well, that’s your problem, and when you fix that, come back to me.” Well, it doesn’t quite work like that.

I think the other thing to note is that there’s a real distinction in our discussion so far between developing software of generalizable use versus developing an application within a software framework. So some of what Pat was talking about is of the ilk: “What does a user need to know to develop a document in Microsoft Word?” And some of what she was saying was, “What do you need to do to develop Microsoft Word as a software product?” And those are very different things. So my friend at CBS News doesn’t design Microsoft Word; they do documents in Microsoft Word—to use this metaphor—nightly to get a survey out. So we need to be careful in our discussions on that point.

CORK: This is a good point to stop here, if we can break things off and pick up after lunch.

[The workshop stopped for a lunch break.]



The National Academies | 500 Fifth St. N.W. | Washington, D.C. 20001
Copyright © National Academy of Sciences. All rights reserved.
Terms of Use and Privacy Statement