Below are the first 10 and last 10 pages of uncorrected machine-read text (when available) of this chapter, followed by the top 30 algorithmically extracted key phrases from the chapter as a whole.
Intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text on the opening pages of each chapter.
Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.
Do not use for reproduction, copying, pasting, or reading; exclusively for search engines.
OCR for page 221
APPENDIX
E
The TRS Experience with Cognitive
Labs and Forms Design
Mary Batcher
Statistics of Income Division
Internal Revenue Service
The problems with income tax forms are legendary. Not only are they
very complex, but they come complete with difficult instructions. In addi-
tion, very few people can totally avoid interacting with the forms.
The content and format of tax forms are of interest to many parts of the
Internal Revenue Service (IRS), outside agencies, the practitioner commu-
nity, and taxpayers. Despite what may be impressions to the contrary, the
IRS pays a great deal of attention to improving the design of the more
common forms, especially the 1040 family of forms. Many other less fa-
miliar IRS forms and schedules are extremely important to users of tax data.
They contribute key components to the data bases used in tax policy and
statistical analyses. Consequently, the Statistics of Income (SOI) Division
of the IRS, which has primary responsibility for those data bases, has been
supporting work by the Bureau of Labor Statistics (BLS) cognitive lab, the
Behavioral Science Research Center, to improve some of the less well known
forms.
Specific details and reports on the forms redesign work are available
from the author upon request, but the focus of this paper is the effective use
of cognitive lab technology to improve complex forms and instructions and
the implementation of suggested improvements in an environment that is
simultaneously bureaucratic and hands-on operational.
USE OF THE LAB
In general, the SOI Division is still working out the best way to use the
resources of the BLS cognitive lab. Some areas that make a difference in
221
OCR for page 222
222
QUALITY IN STUDENT FINANCIAL AID PROGRAMS
the effective use of the cognitive lab for forms design are specification of
the problem; communication among the processing staff, data users, and the
lab researchers; and availability of resources, in terms of time and funding,
to allow for proper problem formulation, exploratory work, and testing of
alternatives.
Problem Specification
The importance of having a well-defined problem is clearly not unique
to forms design. However, problem specification has been particularly trouble-
some in the division's use of the BLS cognitive lab to improve forms. The
lab's SOI customers are often not very knowledgeable about the lab's capa-
bilities and methodology. Forms improvement can appear deceptively simple;
many in the SOI Division, for example, have their own commonsense rules
about how to clarify instructions or make forms more user friendly. This
lack of familiarity with the lab's methodology, coupled with a sense that the
task is relatively straightforward, contributes to a misconception that the
only problem specification needed is to ask the lab to "fix" a particular
form.
It is important that specific goals and limitations are communicated to
the lab researchers as part of problem definition. Forms can be improved
with a number of goals in mind: error reduction, appearance, ease of pro-
cessing, and so on. The primary objectives should be agreed upon by the
customers and specified for the lab. The limitations should also be well
defined. These include the usual cost and time constraints but may also
include specification of untouchable portions of the form, page or line lim-
its, and limits on how many of the feeder items, forms, and instructions are
to be addressed. Defining limitations has been a difficult task on occasion
because of the interrelated nature of the IRS forms and the many uses for
them beyond simple determination of taxes owed.
Finally, part of problem specification should be the identification of
any known problems with the form under consideration, including high-
error items. This seems obvious, but in a large bureaucratic environment
with multiple users of forms, researchers may have to make contact with
several areas of the agency, including both field and headquarters staff, to
locate all the relevant information.
Communication
Even under the best of circumstances, it is not likely that the cognitive
lab staff and the customer will have a completely clear mutual understand-
ing of the problem and desired outcome. It is, therefore, essential that there
be frequent communication between the lab and its customers throughout
OCR for page 223
APPENDIX E
223
the conduct of the study. This is especially critical given the lack of famil-
iarity of the customers with the lab's capabilities and limitations. Expecta-
tions must be established and continually reestablished as each of the par-
ticipants learns more about the problem and the others' capabilities.
Resources
_ A ~
The IRS is a production-oriented agency. As a result, there is not a
general appreciation of the difference between data collection for explor-
atory purposes and for testing of alternatives. Often exploratory work is
viewed as testing, and thus, time and resources are not allocated for a
testing stage. For example, focus group testing using forms to identify
problem areas is frequently viewed by the IRS customers as adequate test-
ing to support recommendations. They do not see the need to continue to a
second stage and test the performance of alternative versions of the form
according to defined criteria using larger groups of respondents. Frequently,
despite suggestions by the lab, no provision is made for actually testing the
focus group recommendations.
General Observations on Use of the Lab
In general, from the perspective of the SOI Division, researchers in the
cognitive lab must learn to understand and deal with results-oriented, opera-
tional customers, who will probably never fully appreciate the complexity
of the forms design task. However, the operational customers do know
what they want, although they may need some persistent prompting to specify
it. In identifying customer wants and needs, it is important that the cogni-
tive lab researchers avoid imposing their definition of what the customer
needs and ignore what the customer wants. The relationship will not suc-
ceed if the customer does not receive some version of what he or she wants.
As the lab and the customer learn more about each other's skills and unique
needs, the customer's wants should come into closer alignment with his or
her needs as seen by the forms design and cognitive experts.
IMPLEMENTATION ISSUES
Some factors affecting the successful IRS implementation of the recom-
mendations of the cognitive lab are the multiplicity of IRS customers and
elements of the IRS culture itself. As noted, there are multiple users of
each of the tax forms throughout the IRS. Although the request for forms
design work generally comes from only one of the many IRS users, each of
the other stakeholders has an opinion about proposed forms changes and,
often, the potential to block proposed changes. The titular IRS owner of the
OCR for page 224
224
QUALITY IN STUDENT FINANCIAL AID PROGRAMS
tax forms is the Tax Forms and Publications Division. The SOI Division is
supporting cognitive lab work on the forms that are important in the SOI
data bases. The division has kept the Tax Forms and Publications Division
involved with SOI on this research. This has been an important element in
securing support for the recommendations of the lab. In addition to control-
ling forms changes, the Tax Forms and Publications Division is the only
part of the IRS that is fully aware of the complexity of the task and the
importance of conducting research on forms and basing decisions on the
results of appropriate testing.
Although the SOI Division has been able to involve its primary external
customers and the major IRS owner of the tax forms in the forms improve-
ment effort, it has depended on the Tax Forms and Publications Division to
coordinate the involvement of the other stakeholders. The traditional ve-
hicle for this coordination is a meeting at which stakeholders are given an
opportunity to comment on proposed forms changes. There are organiza-
tional and cost limitations that support this approach, but less formal con-
tact with the major stakeholders at an earlier stage in the process might
reduce resistance to proposed changes and ensure that their perspectives are
considered.
The IRS is not a research or statistical agency; it has a specific opera-
tional mission to administer the tax laws. Hence, the broad IRS culture is
very strong and strongly operational. On the positive side, regarding ease
of implementation, the IRS has the ability to move more quickly than other
governmental agencies once an objective is determined. On the other hand,
there is often a tendency within the IRS to listen to advice, understand it,
and choose to follow it or not depending on how well it fits conventional
operational wisdom. Advice linked to operational considerations, like error
or cost reduction, is more appealing to most in the IRS than advice linked to
cognitive theory. Results based on any kind of testing are valued; however,
there is little critical assessment of the testing methodology or differentia-
tion among types and purposes of testing.
In general, in any application of cognitive lab technology to forms
design, attention to the organizational culture should pay off in identifying
the best techniques to aid implementation. A standard tactic that has worked
well for the SOI Division has been to begin with a few small projects whose
results can be seen quickly and publicized. Once users within the IRS see
some of the improvements, they tend to become enthusiastic about using the
lab to improve forms and instructions.
Representative terms from entire chapter:
tax forms
