So, yes, like my confreres from the software world, I know nothing about surveys. So let’s just get that out right away. But what I do know about is testing, and from what I see you folks are leaving the world of surveys to some degree and moving to software, and so expertise on software testing may inform your efforts to test your questionnaires.

So I work for Microsoft; I’m currently with the Six Sigma Productivity Team. I’m in charge of test productivity initiatives and, before Microsoft, I was with HP for three years and before that with Bell Labs for ten.

OK—the theme. I test software, you test surveys. But your surveys are becoming software. And there [are] actually models that underlie them both. And what’s been fascinating to me is that even though I have nothing to do with surveys during my normal life the number of things that people have said about how you do surveys keep resonating with me and how we do software testing.

Once you left the world of paper, you entered the software world. And in much the same way that we are very feature-driven in the industry, you folks sound like you’re feature-driven because you can do it. And so what you’re ending up with is the same kind of tension we have between features that you’d really like to get in and your ability to validate that you haven’t introduced bugs into your system along the way. Lots of bugs, evidently, from survey instruments; it sounds like a lot to me.

Just in terms of the cost of bugs, just for a second, in keeping with the trend, $300—I mean, 300 times—is a low estimate. [laughter] Because if you find a bug in the field—which is part of the reason why Microsoft really doesn’t need the reputation of releasing things to have people test, because once you’ve released it, by the time you get it back you have to regression test it through all those systems. And then you have to redeploy. Oh, it’s terrible. So let’s use 300 as a low estimate.

And another thing is that I’m fascinated to hear the work on complexity. I know of a project before my time at Microsoft where—if you’re familiar with software bug trends—usually the bug trend towards the early part of a project starts up here and then comes down here, and you kind of say, here’s where we’re going to release. There was a project where it was essentially a big bowl of mud and every time you fixed something you broke more things. So what they actually had—so, this is basically a zero-defect type approach … what they actually had was an infinite-defect approach. [laughter] Every time they fixed it they broke more stuff, and they did what was really the sensible thing—they shipped. [laughter]

Software testing problems—[first,] time is limited. You have some time—in between when there is something to test and when you actually have to give it to somebody—[in which] you need to be able to find as many bugs as you can. And what you need to do is address your efforts



The National Academies | 500 Fifth St. N.W. | Washington, D.C. 20001
Copyright © National Academy of Sciences. All rights reserved.
Terms of Use and Privacy Statement