Below are the first 10 and last 10 pages of uncorrected machine-read text (when available) of this chapter, followed by the top 30 algorithmically extracted key phrases from the chapter as a whole.
Intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text on the opening pages of each chapter. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.
Do not use for reproduction, copying, pasting, or reading; exclusively for search engines.
OCR for page 221
APPENDIX E The TRS Experience with Cognitive Labs and Forms Design Mary Batcher Statistics of Income Division Internal Revenue Service The problems with income tax forms are legendary. Not only are they very complex, but they come complete with difficult instructions. In addi- tion, very few people can totally avoid interacting with the forms. The content and format of tax forms are of interest to many parts of the Internal Revenue Service (IRS), outside agencies, the practitioner commu- nity, and taxpayers. Despite what may be impressions to the contrary, the IRS pays a great deal of attention to improving the design of the more common forms, especially the 1040 family of forms. Many other less fa- miliar IRS forms and schedules are extremely important to users of tax data. They contribute key components to the data bases used in tax policy and statistical analyses. Consequently, the Statistics of Income (SOI) Division of the IRS, which has primary responsibility for those data bases, has been supporting work by the Bureau of Labor Statistics (BLS) cognitive lab, the Behavioral Science Research Center, to improve some of the less well known forms. Specific details and reports on the forms redesign work are available from the author upon request, but the focus of this paper is the effective use of cognitive lab technology to improve complex forms and instructions and the implementation of suggested improvements in an environment that is simultaneously bureaucratic and hands-on operational. USE OF THE LAB In general, the SOI Division is still working out the best way to use the resources of the BLS cognitive lab. Some areas that make a difference in 221
OCR for page 221
222 QUALITY IN STUDENT FINANCIAL AID PROGRAMS the effective use of the cognitive lab for forms design are specification of the problem; communication among the processing staff, data users, and the lab researchers; and availability of resources, in terms of time and funding, to allow for proper problem formulation, exploratory work, and testing of alternatives. Problem Specification The importance of having a well-defined problem is clearly not unique to forms design. However, problem specification has been particularly trouble- some in the division's use of the BLS cognitive lab to improve forms. The lab's SOI customers are often not very knowledgeable about the lab's capa- bilities and methodology. Forms improvement can appear deceptively simple; many in the SOI Division, for example, have their own commonsense rules about how to clarify instructions or make forms more user friendly. This lack of familiarity with the lab's methodology, coupled with a sense that the task is relatively straightforward, contributes to a misconception that the only problem specification needed is to ask the lab to "fix" a particular form. It is important that specific goals and limitations are communicated to the lab researchers as part of problem definition. Forms can be improved with a number of goals in mind: error reduction, appearance, ease of pro- cessing, and so on. The primary objectives should be agreed upon by the customers and specified for the lab. The limitations should also be well defined. These include the usual cost and time constraints but may also include specification of untouchable portions of the form, page or line lim- its, and limits on how many of the feeder items, forms, and instructions are to be addressed. Defining limitations has been a difficult task on occasion because of the interrelated nature of the IRS forms and the many uses for them beyond simple determination of taxes owed. Finally, part of problem specification should be the identification of any known problems with the form under consideration, including high- error items. This seems obvious, but in a large bureaucratic environment with multiple users of forms, researchers may have to make contact with several areas of the agency, including both field and headquarters staff, to locate all the relevant information. Communication Even under the best of circumstances, it is not likely that the cognitive lab staff and the customer will have a completely clear mutual understand- ing of the problem and desired outcome. It is, therefore, essential that there be frequent communication between the lab and its customers throughout
OCR for page 221
APPENDIX E 223 the conduct of the study. This is especially critical given the lack of famil- iarity of the customers with the lab's capabilities and limitations. Expecta- tions must be established and continually reestablished as each of the par- ticipants learns more about the problem and the others' capabilities. Resources _ A ~ The IRS is a production-oriented agency. As a result, there is not a general appreciation of the difference between data collection for explor- atory purposes and for testing of alternatives. Often exploratory work is viewed as testing, and thus, time and resources are not allocated for a testing stage. For example, focus group testing using forms to identify problem areas is frequently viewed by the IRS customers as adequate test- ing to support recommendations. They do not see the need to continue to a second stage and test the performance of alternative versions of the form according to defined criteria using larger groups of respondents. Frequently, despite suggestions by the lab, no provision is made for actually testing the focus group recommendations. General Observations on Use of the Lab In general, from the perspective of the SOI Division, researchers in the cognitive lab must learn to understand and deal with results-oriented, opera- tional customers, who will probably never fully appreciate the complexity of the forms design task. However, the operational customers do know what they want, although they may need some persistent prompting to specify it. In identifying customer wants and needs, it is important that the cogni- tive lab researchers avoid imposing their definition of what the customer needs and ignore what the customer wants. The relationship will not suc- ceed if the customer does not receive some version of what he or she wants. As the lab and the customer learn more about each other's skills and unique needs, the customer's wants should come into closer alignment with his or her needs as seen by the forms design and cognitive experts. IMPLEMENTATION ISSUES Some factors affecting the successful IRS implementation of the recom- mendations of the cognitive lab are the multiplicity of IRS customers and elements of the IRS culture itself. As noted, there are multiple users of each of the tax forms throughout the IRS. Although the request for forms design work generally comes from only one of the many IRS users, each of the other stakeholders has an opinion about proposed forms changes and, often, the potential to block proposed changes. The titular IRS owner of the
OCR for page 221
224 QUALITY IN STUDENT FINANCIAL AID PROGRAMS tax forms is the Tax Forms and Publications Division. The SOI Division is supporting cognitive lab work on the forms that are important in the SOI data bases. The division has kept the Tax Forms and Publications Division involved with SOI on this research. This has been an important element in securing support for the recommendations of the lab. In addition to control- ling forms changes, the Tax Forms and Publications Division is the only part of the IRS that is fully aware of the complexity of the task and the importance of conducting research on forms and basing decisions on the results of appropriate testing. Although the SOI Division has been able to involve its primary external customers and the major IRS owner of the tax forms in the forms improve- ment effort, it has depended on the Tax Forms and Publications Division to coordinate the involvement of the other stakeholders. The traditional ve- hicle for this coordination is a meeting at which stakeholders are given an opportunity to comment on proposed forms changes. There are organiza- tional and cost limitations that support this approach, but less formal con- tact with the major stakeholders at an earlier stage in the process might reduce resistance to proposed changes and ensure that their perspectives are considered. The IRS is not a research or statistical agency; it has a specific opera- tional mission to administer the tax laws. Hence, the broad IRS culture is very strong and strongly operational. On the positive side, regarding ease of implementation, the IRS has the ability to move more quickly than other governmental agencies once an objective is determined. On the other hand, there is often a tendency within the IRS to listen to advice, understand it, and choose to follow it or not depending on how well it fits conventional operational wisdom. Advice linked to operational considerations, like error or cost reduction, is more appealing to most in the IRS than advice linked to cognitive theory. Results based on any kind of testing are valued; however, there is little critical assessment of the testing methodology or differentia- tion among types and purposes of testing. In general, in any application of cognitive lab technology to forms design, attention to the organizational culture should pay off in identifying the best techniques to aid implementation. A standard tactic that has worked well for the SOI Division has been to begin with a few small projects whose results can be seen quickly and publicized. Once users within the IRS see some of the improvements, they tend to become enthusiastic about using the lab to improve forms and instructions.