looked like, for providing the product in new release. I think that’s different from what you’re saying, but it reminded me of that. And, as I understand, how you would be dealing with that would be kind of cut the development right here, it stops, and send to external organizations using different approaches.

MCCABE: [very faint on recording] One of the problems with this, it turns out, with universities is that it’s way too expensive to use as these operational sites…. But an operational company will usually be able to field a couple of operational sites, and typically those focus on different priorities. Or they pick one site that is known to get something done quickly; that information is relevant. And then you get some comparisons among the methodologies.

GROVES: I see that as an interesting comment because it would be great to have something that allows us to judge which of those sites. The question is the criterion … [trails off to inaudible] I wonder what kind of evidence you can produce to say: I prefer this method to that one? What do you fit into the criterion?

MCCABE: I think it’s really important to realize that there’s no single testing method used at any of these places. [inaudible until closing comments] I think that the place that test well do so because they have at least four or different methods that they use at various stages. I don’t know of any place that has just one thing going in testing.

SMITH: I think that a lot of those methods are very good, and I’ve also found that if you integrate them in a regular way, then they smooth out and the problems with them start to diminish. For example, code review can be a threat to people who have never done it before, because they remember back in college when their work was being graded or something. And after it becomes commonplace and everybody’s had that chance to go around the table, it becomes a very different thing. And you find a lot of problems that way, and you understand the code, and you do some cross-training to some extent. And I think that everything else you mentioned—unit testing, regression testing—if regularized does kind of smooth out. And costs become less. Would you concur?

MCCABE: One thing we didn’t talk about that we ought to comment on is relative cost of testing software versus surveys. In software, it’s quite high; in software, it’s often, in the total budget, maybe 60 percent. Now places like Microsoft, where you’re shipping millions of products, the testing cost tends to be very, very high because the cost of errors is so high. Versus more of an engineering shop. But, still, we’re in the software business and producing, and so the costs of testing can go up to 60 percent. Now how does that compare with surveys?

DOYLE: It’s a small portion of the current budget. Of the total budget.



The National Academies | 500 Fifth St. N.W. | Washington, D.C. 20001
Copyright © National Academy of Sciences. All rights reserved.
Terms of Use and Privacy Statement