Testing
Less progress has been made in the area of comprehensive testing of CAPI instruments. Typically saved until the end of the design process, testing of CAPI instruments is—in the Census Bureau’s experience— often tightly constrained by scant available time before due dates of deliverables and the remaining time and staff resources.
Current practice in testing CAPI instruments—the protocols used to test not only software functionality but also usability and human interface factors—differs across survey organizations. As indicated at the workshop, the Census Bureau relies to a large extent on the manual entry of a relatively limited number of sample answer scenarios. Some amount of direct manual testing of CAPI instruments is clearly necessary; cognitive factors, interviewer usability, and even mundane design choices like font size and screen color are things that can be properly appreciated only by directly interacting with the instrument. But hand input of dozens or even hundreds of prepackaged scenarios cannot possibly certify that all logical paths through a questionnaire work correctly. Manual scenarios must also be adjusted as the content of the questionnaire changes, making the process even more inefficient.
Part of the problem faced by the Census Bureau and other survey organizations in establishing a more rigorous test regime of a computerized survey’s functionality lies in the limitations of the current software used to produce CAPI instruments. Neither CASES nor Blaise currently has the capacity for automated entry of randomly or probabilistically generated data to test paths through the instrument. At the workshop, it was mentioned that some prototype capacity has been developed to write scripts that can at least automate the task of entering data from canned answer scenarios into a Blaise-coded instrument, but automated test routines are not yet available.
SHIFT FROM SURVEY RESEARCH TO SOFTWARE ENGINEERING
Although the presentations at the Workshop on Survey Automation covered diverse topic areas, several unifying themes do emerge from reviewing the workshop proceedings. One of these is perhaps most fundamental: the advent of computer-assisted interviewing thrust survey research organizations and federal statistical agencies into the business of software development, a business for which the extant management styles and design processes of the survey world are ill suited. The major problems in documenting and testing computerized survey instruments
are symptomatic of the need to reexamine the basic process of designing, constructing, and implementing a survey in the computer age.
This is not to say that the process of fielding surveys has become strictly an exercise in developing software. The root of a survey remains a communication process between interviewer and respondent, and the unique features of that communication interface—the sensitivity of respondents to factors like question wording and ordering, the need for interviewers to be able to convey questions with proper grammar in a language that the respondent can understand, and so forth—prevent survey research from being a truly mechanical science. Moreover, the translation from survey work to the arena of software development is not entirely direct. The number of end users of a commercially developed software package may number in the millions, dwarfing the total number of interviewers and respondents who encounter a typical CAPI implementation. Likewise, the number of iterations of a survey is usually quite small, while the latitude for continuous revision—reissues, updates, and patches—is greater in commercial software environments. But the uniqueness of the survey context does not give license to conclude that the documentation and testing problems in CAPI are so unique or so challenging that external expertise in software development cannot help address the problems.
It is perhaps telling that the methods currently in use have been termed “computer-assisted interviewing” or “computer-assisted survey information collection.” These labels convey an implicit supremacy of the interview over the computer, of the end over the means. To a large extent, this implied supremacy is altogether appropriate; the credibility of survey research organizations and federal statistical agencies rests on the timely and accurate production of data, and highest priority is understandably put on the direct interface with respondents and on the final data products.
But an underlying message of the Workshop on Survey Automation is that the survey industry needs to begin thinking more in terms of “interview-oriented computing”—of the mechanics of designing effective software for data elicitation, capture, and processing. The process by which a modern survey is designed and developed must also result in a quality piece of software, and so that process must ensure that proper software engineering standards are met. The importance of rethinking the mechanics and the organizational structure of modern survey design becomes increasingly important as CAPI and CATI systems become more dominant features of the survey world, and more important still as computer-assisted surveys are implemented on new platforms, such as the Internet or handheld computing devices.