Survey Automation Report and Workshop Proceedings (2003) / Chapter Skim
Currently Skimming:
What Makes the CAI Testing and Documentation Problems So Hard to Solve?
What Makes the CAI Testing and Documentation Problems So Hard to Solve?
Pages 36-62
The Chapter Skim interface presents what we've algorithmically identified as the most significant single chunk of text within every page in the chapter.
Select key terms on the right to highlight them within pages of the chapter.
From page 36... ...
my opening remarks, and Chet apologizes for not being able to be here, but there was the minor issue of the budget for the U.S. Census Bureau, so I guess we can forgive him.
|
From page 37... ...
Each round of interviewing for a particular household is lcnown as a "wave," so that the first time the interviewer contacts the household is called Wave 1, the second contact Wave 2, and so forth. For additional detail on the SIPP and its sample design, see Westat (2001 )
|
From page 38... ...
from the core questionnaire clocument, Wave 3 of the Survey of Income and Program Participation (SIPP) , 1 993 Panel.
|
From page 39... ...
What happens is most of the transitions tend to be reported at the juncture between the rounds of interviewing, instead of evenly pretty milch throllghollt the period that's what's called the "seam bias." Another survey where we're in the process of instituting an extremely complex instrument is the Consumer Expenditure Survey (CES)
|
From page 40... ...
Child's Health Insurance Program (SCHIP) .4 This was instituted a few years ago in response to the concern about lower-income children not having adequate health insurance coverage.
|
From page 41... ...
So we have a fairly simple questionwe want to lcnow what their income was from a particular source; in this case it's an asset-type source. Again, in the original paper version of this, we have a fairly simple question; we have the ellipses where we would fill in the respondent's name, or "you", or "he", depending on the circumstances again, up to the FR to decide, a lot of freedom there on their part.
|
From page 42... ...
We'd lilce to create a friendly rapport between the field representative which is an interviewer, by the way; this is Census Bureau-speak for an interviewer, my apologies we want that interaction to be friendly, so we don't want to force the respondent to answer a question in a way he doesn't want to. In this case, we wouldn't force them to report monthly if they really didn't want to.
|
From page 43... ...
it's the mix of alphanumeric and numeric that causes some data processing procedures to get a little unhappy. Sometimes we want to change our questions slightly depending on the way people answer, so we have these little pop-up things I don't lcnow what you call these things in a cartoon, but they just pop up.
|
From page 44... ...
Blaise and CASES represent perhaps the two dominant survey software packages used by survey organizations (particularly among U.S. federal statistical agencies)
|
From page 45... ...
. DOYLE: Except for the cognitive people of the world, who have now got a gold mine for ways that they can predetermine how to appropriately phrase the question so that the FR isn't using a little bit too much imagination in determining how to phrase it.
|
From page 46... ...
Similarly, reference period could be handled as a fill, so that instead of seeing a parenthetical remark describing a generic reference period, when this question goes up on the screen it would say "April, May, rune, luly," or it would say, "since April 1". In other words the computer has changed the appearance or the wording on-screen so that it meets the need of this particular case.
|
From page 47... ...
. Let's say that just before this interest income amount question we had a question that said, "Do you have an interest-bearing account?
|
From page 48... ...
And the other approach is to basically allow the FR to have a conversation with the respondent and then to fill out the form in an intelligent fashion based on what the respondent said. Is there any experiment that has been done that compares the yield or the accuracy of results under those two lcinds of positions?
|
From page 49... ...
But there's this whole different aspect do you really need to carry the fills out to the nth degree? And can't the field representative with their native intelligence provide the fills themselves?
|
From page 50... ...
That carries a lot into the development process and the testing. DOYLE: The whole systems side is something I haven't dwelled on, and I sort of view the setting of that maybe I'm wrong, but I lcind of put that final outcome code setting more on the systems end.
|
From page 51... ...
First of all, when I talk about documentation now, I'm just talking about documenting the questions that were aslced in the field. I'm not talking about documenting files; I'm not talking about study-level quality profiles.
|
From page 52... ...
DOYLE: It's difficult. One of the reasons we have a couple of attempts to do automated instrument documentation is so that we lcnow what we're getting as a documentation of the instrument as opposed to what we intended to collect.
|
From page 53... ...
. BANICS: But I think this suggestion goes directly to that, because if the 10 pathways that 80 percent of people employ in fact would cause little burden, then that's helpful to lcnow.
|
From page 54... ...
DOYLE: That's right, and in recent times we've moved toward using database management systems to develop the specifications, to automatically feed the software that runs the instruments. What we're finding is that it's very diffiullt for that to flow naturally.
|
From page 55... ...
The IDOC notion was to basically capture all the little pieces of the instrument and put them into HTML documents, put it up in Netscape and let you use all the browsers and what-not to go through and follow the instrument. And if you wanted to lcnow what a fill was you could go click on it and go back and find what it was.
|
From page 56... ...
We have some standards; we have them internally at the Census Bureau for the actual implementation of our instruments. We also have, in the documentation arena, the Data Documentation Initiative, which is producing standards for data documentation which includes standards for formatting instrument documentation.9 That's going to help us a lot once we move toward really getting our instruments documented, having them in a format that can be shared in a machine-readable capacity across all the archives, all the data libraries, all the users, and so forth.
|
From page 57... ...
But it's a real problem, and it's one of those things that will really trip up an FR. If they're just reading what's on there and all of a sudden the grammar is bad that's a really big deal to them; they're going to see that a lot qlliclcer than they're going to see a fundamental flaw in the logic.
|
From page 58... ...
We have the survey methodologists, the cognitive researchers, who are trying to figure out how to phrase the questions they need to look to malce sure that things are getting properly phrased. We often use our field staff for testing our instruments once we have an instrument that's pretty much working, they're excellent sources for finding things lily the grammatical problems.
|
From page 59... ...
We have, at the Census Bureau, a "Tester's Menu" so we can basically post these test instruments and have one of those menu options for processing the output so you can more easily check the output.~2 We have tracking systems that lceep track of where the code is and what version of the code you've got, and that sort of thing. We also have a testing checklist that I thought I would summarize a little bit for you.
|
From page 60... ...
Because, sure enough, if we don't, we'll introduce a bug that we didn't think we did, and it'll destroy a path somewhere out there in the instrument where we didn't loolc and we won't find out about it until after we've collected the data. Our systems testing is basically distinct from our instrument testing, which is part of another linear process.
|
From page 61... ...
For this group, what we're trying to do is to get them to figure out a way just to document what we did do and we'll get another group in here to talk about documenting why we do it. What we're trying to do in the context of routine quality profiles for our surveys is to start incorporating documentation lily that, which says, "We tested these questions, we made these decisions.
|
From page 62... ...
It also may happen that we're implementing the automated instrument at the same time that we're doing cognitive testing, so that while we're doing cognitive testing and find that these words don't mean exactly what we thought they would, let's change them to these other words. And you have to go back and embed it in there.
|
Key Terms
This material may be derived from roughly machine-read images, and so is provided only to facilitate research.
More
information on Chapter Skim is available.