National Academies Press: OpenBook

Survey Automation: Report and Workshop Proceedings (2003)

Chapter: Shift from Survey Research to Software Engineering

« Previous: Current Practice in Documentation and Testing
Suggested Citation:"Shift from Survey Research to Software Engineering." National Research Council. 2003. Survey Automation: Report and Workshop Proceedings. Washington, DC: The National Academies Press. doi: 10.17226/10695.
×

Testing

Less progress has been made in the area of comprehensive testing of CAPI instruments. Typically saved until the end of the design process, testing of CAPI instruments is—in the Census Bureau’s experience— often tightly constrained by scant available time before due dates of deliverables and the remaining time and staff resources.

Current practice in testing CAPI instruments—the protocols used to test not only software functionality but also usability and human interface factors—differs across survey organizations. As indicated at the workshop, the Census Bureau relies to a large extent on the manual entry of a relatively limited number of sample answer scenarios. Some amount of direct manual testing of CAPI instruments is clearly necessary; cognitive factors, interviewer usability, and even mundane design choices like font size and screen color are things that can be properly appreciated only by directly interacting with the instrument. But hand input of dozens or even hundreds of prepackaged scenarios cannot possibly certify that all logical paths through a questionnaire work correctly. Manual scenarios must also be adjusted as the content of the questionnaire changes, making the process even more inefficient.

Part of the problem faced by the Census Bureau and other survey organizations in establishing a more rigorous test regime of a computerized survey’s functionality lies in the limitations of the current software used to produce CAPI instruments. Neither CASES nor Blaise currently has the capacity for automated entry of randomly or probabilistically generated data to test paths through the instrument. At the workshop, it was mentioned that some prototype capacity has been developed to write scripts that can at least automate the task of entering data from canned answer scenarios into a Blaise-coded instrument, but automated test routines are not yet available.

SHIFT FROM SURVEY RESEARCH TO SOFTWARE ENGINEERING

Although the presentations at the Workshop on Survey Automation covered diverse topic areas, several unifying themes do emerge from reviewing the workshop proceedings. One of these is perhaps most fundamental: the advent of computer-assisted interviewing thrust survey research organizations and federal statistical agencies into the business of software development, a business for which the extant management styles and design processes of the survey world are ill suited. The major problems in documenting and testing computerized survey instruments

Suggested Citation:"Shift from Survey Research to Software Engineering." National Research Council. 2003. Survey Automation: Report and Workshop Proceedings. Washington, DC: The National Academies Press. doi: 10.17226/10695.
×

are symptomatic of the need to reexamine the basic process of designing, constructing, and implementing a survey in the computer age.

This is not to say that the process of fielding surveys has become strictly an exercise in developing software. The root of a survey remains a communication process between interviewer and respondent, and the unique features of that communication interface—the sensitivity of respondents to factors like question wording and ordering, the need for interviewers to be able to convey questions with proper grammar in a language that the respondent can understand, and so forth—prevent survey research from being a truly mechanical science. Moreover, the translation from survey work to the arena of software development is not entirely direct. The number of end users of a commercially developed software package may number in the millions, dwarfing the total number of interviewers and respondents who encounter a typical CAPI implementation. Likewise, the number of iterations of a survey is usually quite small, while the latitude for continuous revision—reissues, updates, and patches—is greater in commercial software environments. But the uniqueness of the survey context does not give license to conclude that the documentation and testing problems in CAPI are so unique or so challenging that external expertise in software development cannot help address the problems.

It is perhaps telling that the methods currently in use have been termed “computer-assisted interviewing” or “computer-assisted survey information collection.” These labels convey an implicit supremacy of the interview over the computer, of the end over the means. To a large extent, this implied supremacy is altogether appropriate; the credibility of survey research organizations and federal statistical agencies rests on the timely and accurate production of data, and highest priority is understandably put on the direct interface with respondents and on the final data products.

But an underlying message of the Workshop on Survey Automation is that the survey industry needs to begin thinking more in terms of “interview-oriented computing”—of the mechanics of designing effective software for data elicitation, capture, and processing. The process by which a modern survey is designed and developed must also result in a quality piece of software, and so that process must ensure that proper software engineering standards are met. The importance of rethinking the mechanics and the organizational structure of modern survey design becomes increasingly important as CAPI and CATI systems become more dominant features of the survey world, and more important still as computer-assisted surveys are implemented on new platforms, such as the Internet or handheld computing devices.

Suggested Citation:"Shift from Survey Research to Software Engineering." National Research Council. 2003. Survey Automation: Report and Workshop Proceedings. Washington, DC: The National Academies Press. doi: 10.17226/10695.
×
Page 12
Suggested Citation:"Shift from Survey Research to Software Engineering." National Research Council. 2003. Survey Automation: Report and Workshop Proceedings. Washington, DC: The National Academies Press. doi: 10.17226/10695.
×
Page 13
Next: Changing Survey Management Processes to Suit Software Design »
Survey Automation: Report and Workshop Proceedings Get This Book
×
Buy Paperback | $80.00 Buy Ebook | $64.99
MyNAP members save 10% online.
Login or Register to save!
Download Free PDF

For over 100 years, the evolution of modern survey methodology—using the theory of representative sampling to make interferences from a part of the population to the whole—has been paralleled by a drive toward automation, harnessing technology and computerization to make parts of the survey process easier, faster, and better. The availability of portable computers in the late 1980s ushered in computer-assisted personal interviewing (CAPI), in which interviewers administer a survey instrument to respondents using a computerized version of the questionnaire on a portable laptop computer. Computer assisted interviewing (CAI) methods have proven to be extremely useful and beneficial in survey administration. However, the practical problems encountered in documentation and testing CAI instruments suggest that this is an opportune time to reexamine not only the process of developing CAI instruments but also the future directions of survey automation writ large.

  1. ×

    Welcome to OpenBook!

    You're looking at OpenBook, NAP.edu's online reading room since 1999. Based on feedback from you, our users, we've made some improvements that make it easier than ever to read thousands of publications on our website.

    Do you want to take a quick tour of the OpenBook's features?

    No Thanks Take a Tour »
  2. ×

    Show this book's table of contents, where you can jump to any chapter by name.

    « Back Next »
  3. ×

    ...or use these buttons to go back to the previous chapter or skip to the next one.

    « Back Next »
  4. ×

    Jump up to the previous page or down to the next one. Also, you can type in a page number and press Enter to go directly to that page in the book.

    « Back Next »
  5. ×

    Switch between the Original Pages, where you can read the report as it appeared in print, and Text Pages for the web version, where you can highlight and search the text.

    « Back Next »
  6. ×

    To search the entire text of this book, type in your search term here and press Enter.

    « Back Next »
  7. ×

    Share a link to this book page on your preferred social network or via email.

    « Back Next »
  8. ×

    View our suggested citation for this chapter.

    « Back Next »
  9. ×

    Ready to take your reading offline? Click here to buy this book in print or download it as a free PDF, if available.

    « Back Next »
Stay Connected!