Click for next page ( 216


The National Academies | 500 Fifth St. N.W. | Washington, D.C. 20001
Copyright © National Academy of Sciences. All rights reserved.
Terms of Use and Privacy Statement



Below are the first 10 and last 10 pages of uncorrected machine-read text (when available) of this chapter, followed by the top 30 algorithmically extracted key phrases from the chapter as a whole.
Intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text on the opening pages of each chapter. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.

Do not use for reproduction, copying, pasting, or reading; exclusively for search engines.

OCR for page 215
APPENDIX D Laboratory Methods and Document Redesign Leda Kydoniefs and Marie van Melis-Wright Office of Survey Methods Research Bureau of Labor Statistics BACKGROUND In response to the 1983 Advanced Seminar on Cognitive Aspects of Survey Methodology (CASM), interdisciplinary survey research centers and laboratories have been established at a number of governmental organiza- tions (Jabine et al., 1984~. In keeping with the CASM mission, such centers actively promote collaboration among behavioral scientists from various fields, statisticians, and survey researchers in order to improve data quality and the data collection process (Dippo and Norwood, 1992~. One such research center is the Behavioral Science Research Center (BSRC) at the Bureau of Labor Statistics (BLS). Founded in 1988, its professional staff includes psychologists from various specialties (e.g., so- cial, cognitive, educational psychology, artificial intelligence), statisticians, sociologists, a psychological anthropologist, and research assistants. Cur- rently, primary areas of investigation include questionnaire development and design, form redesign, interviewer training, interviewer-participant in- teractions, interview administration methodologies, and basic research in the areas of memory, proxy reporting, confidentiality, and privacy. Since 1989, BSRC has undertaken two major form redesign studies: an effort sponsored by the Internal Revenue Service (IRS) to redesign selected tax forms and the redesign of the BLS Survey of Occupational Illnesses and Injuries. Among the goals of the redesign efforts are to reduce respondent burden, improve data quality, reduce cost and processing time, and develop formats suited for use with different technologies (e.g., optical scanners, administration by computer). 215

OCR for page 215
216 QUALITY IN STUDENT FINANCIAL AID PROGRAMS A distinct feature of the BLS form-redesign project is the application of behavioral science methods to increase understanding of the cognitive, so- cial, and organizational factors in the information-gathering process. Re- searchers have many data collection methods from which to choose, for example, expert analysis, interactional behavior coding, laboratory and field observational techniques, focus groups, vignette classifications, rating tech- niques, free and dimensional sorting, ethnographic and cognitive interviews, and concurrent and retrospective think-alouds (Forsythe and Lessler, 1990~. This paper illustrates the utility of some of these methods using examples from the two redesign projects mentioned above. IRS SCHEDULE C STUDY Schedule C (Profit or Loss from Business) is an attachment to IRS Form 1040. It is used by taxpayers who classify themselves as "sole proprietorships." In 1987, taxpayers filed 14.5 million Schedule C forn;~s. The data collected from Schedule C are used in aggregate form by the IRS and other governmental agencies to, for example, help determine estimates of personal income at the local level. One of the goals of the redesign effort was to simplify the taxpayer's task, which included locating the description of his or her business or pro- fessional activity in the listing on the back of the form, finding the corre- sponding 4-digit code, and writing the description and the 4-digit code on the front of the form. To meet this goal, an interdisciplinary research team was created, which consisted of IRS representatives and behavioral scien- tists from the American Institutes for Research (AIR) and the BSRC labora- tory. The research team used a variety of behavioral science methods to obtain different perspectives on taxpayers' problems with the form and the potential solutions. The methods they used included in-depth telephone interviews with IRS editors of Schedule C, observations and in-depth debriefings of taxpayers, concurrent and retrospective think-alouds with taxpayers, and an experiment evaluating taxpayers' performance under the old and revised versions of Schedule C. Rather than focus only on the taxpayer, the research team decided to collect input from other users of Schedule C, specifically IRS editors of Schedule C and primary and secondary data users (e.g., Statistics of Income Division, IRS; and Bureau of the Census and Bureau of Economic Analysis, U.S. Department of Commerce). These groups regularly handle the com- pleted forms, analyze or disseminate the data, or serve as gatekeepers and trackers/monitors of the flow of forms. Thus, they are in an excellent position to identify typical errors made by the taxpayer. The BSRC team conducted a number of in-depth, semistructured tele- phone interviews with a subsample of editors of Schedule C from among

OCR for page 215
APPENDIX D 217 the 26 national IRS Service Centers; the AIR team conducted observations and used concurrent think-alouds with a small sample of taxpayers; and IRS representatives used an open-ended written questionnaire to interview end users from the Census Bureau and the Bureau of Economic Analysis. All the methods combined yielded a wealth of qualitative and quantitative data pinpointing problems and errors, including lack of or unclear instructions, errors occurring during transfer of information from one side of the form to the other, small print and cluttered physical layout, missing or hard-to- understand principle business and professional code descriptions, use of invalid or outdated codes, and a tendency of taxpayers to favor the "Other" or "Unable to Classify" categories over more specific descriptions. Before making revisions, the research team also collected information from IRS representatives to make sure the revisions would not interfere with impor- tant organizational constraints (e.g., policies with regard to mandated state- ments, paper size, number of pages, format of instructions, and so on). Based on the information gathered during the first stage of the research, the AIR and IRS focus of the revisions crystallized into: alphabetizing the main- and subcategories of the principal business codes on the back of Schedule C, increasing the size of the print, improving the clarity of and completeness of instructions printed on the form, moving the four-digit codes originally located to the left of the cat- egory description to the right, and removing visual "clutter." The rationale for these changes was to facilitate searching, reduce scanning time and errors, and facilitate the taxpayer's task. The final phase of the project involved two experiments. The first compared the existing Schedule C with a revised version on which the 4- digit code was listed to the right of the description of the activity. In experiment 2, three revised versions were compared. In both experiments, subjects in each condition were given a list of 48 scenarios (business or professional activities) and asked to complete a Schedule C for each of them. The majority of scenarios were descriptions of activities taken di- rectly from the current form, synonyms of descriptions listed on the form, and ambiguous descriptions. The last group were descriptions for which several reasonable responses, including "Other" and "Unable to Classify," were options each subject might consider. Outcome measures included accuracy, response time, and a measure of preference by taxpayers for the various versions. In addition, behavioral observations, concurrent and ret- rospective think-alouds, and in-depth debriefings were conducted to investi- gate taxpayers' rationale for their performance.

OCR for page 215
218 QUALITY IN STUDENT FINANCIAL AID PROGRAMS The scope of this paper does not allow for an in-depth discussion of the results. Briefly, however, the findings from experiment 1 did not reveal any significant differences. In experiment 2, alphabetization and format changes yielded higher accuracy and shorter completion times than for the old ver- sion. Findings also showed that respondents did not read the directions, even when they were printed directly above the activity codes, and moving the 4-di~it codes from the left to the right of the description caused more, Not surprisingly, the accuracy levels for the various scenarios variedfrom 85 percent correct for exact descriptions, 78 percent for synonyms, and 42 percent for ambiguous descriptions. An in-depth error analysis was conducted to determine the type of errors made by the taxpayer. On the basis of the results of the experiments, a set of recommendations for the redesign of Schedule C codes was developed, as well as for large- scale testing on a representative sample of taxpayers. Feedback from key participants in the research was very favorable re- garding the methods and techniques used. In addition to the information necessary for the development of revised forms, the close collaboration between key players resulted in a firm commitment to the task at hand and future form development. - not fewer, errors." SURVEY OF OCCUPATIONAL ILLNESSES AND INJURIES A major redesign of the Occupational Safety and Health Statistics (OSHS) system, the nation's source of information on job-related illnesses and inju- ries, was begun in 1987. The OSHS Survey of Occupational Illnesses and Injuries is a mandatory survey and is conducted by mail. The redesign effort included a review of all aspects of the survey: forms design, collection, and processing. . . content, sampling, (, , , ~ ~ The forms redesign portion of the OSHS project was begun in 1989 and consisted of several pilot/feasibility studies investigating alternative survey formats. Once the decision was made in 1991 to pursue the design of a booklet format, the forms redesign efforts intensified. The goal was fourfold: Develop a booklet survey form with accompanying instructions that was "user friendly." . Design a questionnaire format with a built-in flexibility that would permit certain minor variations in item content. (These variations are re- quired in order to maximize the survey's utility across the nation.) 1 This likely happened because the description of some activities took up two lines of text, in which case, the 4-digit code was put at the end of the second line. However, as it turned out, participants did not take that code, but rather selected the one below it (i.e., the code for the next category).

OCR for page 215
APPENDIX D 219 Test and evaluate old (i.e., rephrased) questionnaire content as well as new survey items. Test and evaluate the redesigned OSHS survey that was developed. To achieve these goals, a variety of behavioral science and test devel- opment methods were used, each of which added a different perspective to the knowledge base and forms redesign efforts. Focus groups, consisting of volunteers from nearby establishments, were used to test the newly devel- oped items and instructions for clarity, comprehension, and respondent bur- den. Some respondents were videotaped in their establishments while com- pleting the questionnaire, and retrospective think-aloud interviews were later conducted with them; others were videotaped using the concurrent think- aloud technique.2 These "think alouds" permitted the researchers to view the respondent's interaction with the questionnaire and the respondent within the respondent's working environment. After the focus group data were analyzed and the appropriate changes were made in the survey booklet, it was decided to evaluate the data collec- tion and processing aspect of this newly developed survey format. The questionnaire was mailed to approximately 1,500 respondents located in four states. The data were then processed and analyzed within each state, using the state's "standard operating procedures." This method revealed certain deficiencies in the sampling portion of the survey document that were not evident when each respondent was observed individually. Once again, the test results were analyzed and the survey was revised accord- ingly. Once the focus groups, concurrent and retrospective think-alouds, and test mailings were completed and all appropriate changes implemented, the forms redesign process was approaching completion. To ensure that the changes reflected "true" improvement and had not caused or created other errors or difficulties, the survey was mailed to approximately 200 respon- dents located in five states. The focus was again on evaluating the survey contents (i.e., comprehension of items and response options, clarity of in- structions, and respondent burden). Establishments were sampled in such a way as to permit assessment of differences in data quality between large and small establishments and rural and urban establishments. In addition to the OSHS questionnaire, respondents were asked to answer a number of questions designed to collect information on certain complex items (e.g., employment average, total hours worked by all employees in a year), time required to complete the OSHS survey, and the "readability" of the survey booklet. As the completed forms were received, debriefing interviews were con- ducted by telephone with approximately 50 percent of the respondents. The 2 For further information on this and other techniques, see Mullin et al (1991).

OCR for page 215
220 QUALITY IN STUDENT FINANCIAL AID PROGRAMS focus of the interviews was to investigate the respondents' feelings regard- ing the new survey format and the additional items, achieve a better under- standing of respondent burden, and obtain an estimate of the validity of the data received. The data obtained were analyzed and the survey item content and/or the design of the booklet was revised accordingly. SUMMARY The BSRC laboratory used behavioral science methods to investigate errors and to test revisions of the IRS Schedule C and the BLS OSHS survey. The methods used included focus groups, behavioral observations, concurrent and retrospective think-alouds, in-depth debriefings, semistructured telephone interviews, open-ended written interviews, and expert analysis. All of the methods yielded important information and enhanced the overall redesign efforts. While each has its own strengths and weaknesses in terms of psychometric properties, cost and time requirements, and so on, they all share the characteristic of providing information from perspectives that have often been ignored in the past. REFERENCES Dippo, C. and J. Norwood 1992 A review of research at the Bureau of Labor Statistics. Pp. 271-290 in Questions about Questions: Inquiries into the Cognitive Bases of Surveys. New York: Russell Sage. Forsythe, B. H. and J. T. Lessler 1990 Cognitive Laboratory Methods: A Taxonomy. Paper presented at the Interna- tional Conference on Measurement Errors in Surveys, Tucson, Arizona. Jabine, T. B., M. L. Straf, J. M. Tanur, and R. Tourangeau, eds. 1984 Cognitive Aspects of Survey Methodology: Building a Bridge Between Disci- plines. Washington, D.C.: National Academy Press. Mullin, P. A., L. A. Miller, M. van Melis-Wright, D. B. Stone 1991 Laboratory Research Methods for Improving Surveys. Paper presented at the GAO Technical Conference, University of Maryland, College Park. Available from the Behavioral Science Research Center, Bureau of Labor Statistics.