Reengineering the Census Bureau's Annual Economic Surveys (2018) / Chapter Skim
Currently Skimming:
4 Harmonization of Questionnaires and Data Collection Processes
4 Harmonization of Questionnaires and Data Collection Processes
Pages 53-90
The Chapter Skim interface presents what we've algorithmically identified as the most significant single chunk of text within every page in the chapter.
Select key terms on the right to highlight them within pages of the chapter.
From page 53... ...
Even some Census Bureau employees seemed surprised by the many differences that were revealed in the Bureau's presentations to the panel. In recent years, the Census Bureau has taken steps toward streamlining the survey and statistical processes for the annual economic surveys.
|
From page 54... ...
Although the achievable labor savings are likely to vary across organizations, the 2000 redesign of the Annual Structural Surveys at Statistics Netherlands (see discussion in Chapter 8) made it possible to reduce staffing for these surveys by 40 percent.3 4.1 DATA SOURCES An important consideration in reengineering the Census Bureau's annual economic surveys is the choice of data sources, which currently include surveys as the primary source, together with federal administrative data and annual reports and 10-K reports that public companies are required to provide to both shareholders and the Securities and Exchange Commission.4 With the increasing difficulty and cost of maintaining high response rates for the annual economic surveys and the growing availability of alternative data sources, there is the potential to move the surveys and a future ABSS toward a mix of data sources, including sources not in current use, that could better meet users' needs for timeliness, consistency, and additional relevant content.
|
From page 55... ...
cARTS and AWTS use administrative data for sampled nonemployer businesses. They draw IRS administrative records data from the Census Bureau's annual Nonemployer Statistics (NES)
|
From page 56... ...
. Section 6 "requires that the Census Bureau use administrative data from other agencies, state and local governments and other instrumentalities, and private organizations instead of conducting direct inquiries if such data meet the quality and timeliness standards of the Census Bureau" (Gates, 2009)
|
From page 57... ...
Census Bureau, 2013) , administrative data and any other potentially useful sources for the annual economic surveys and an ABSS (e.g., those similar to the annual reports and 10-K forms already being used for public companies)
|
From page 58... ...
As the Census Bureau proceeds to harmonize and align key concepts across the various annual economic surveys, it will be important to keep in mind: (1) the user perspective, related to the relevance of the data; (2)
|
From page 59... ...
, extranet, EDI [Electronic Data Interchange] network, electronic mail, or other comparable online system.
|
From page 60... ...
In addition to the tailored core questionnaires, NAICS subgroup–specific concepts were identified, including data items that only need to be collected for specific sectors of the economy. These TABLE 4-2 Key Concepts Measured in the Census Bureau's Annual Economic Surveys Survey Concepts ASM Total annual employment, payroll, cost of materials, operating expenses, value of shipments, e-commerce shipments, value added, end-of-year inventories, inventory held outside the United States, and value of product shipments
|
From page 61... ...
aSurvey is currently suspended. SOURCE: Key concepts are those identified on the Census Bureau website for each survey and by the panel from an inspection of the survey questionnaires.
|
From page 62... ...
Second, response burden related to the selected concepts and data items can be identified through feasibility studies and the results of any pretests of the annual survey questionnaires. In particular, when identifying concepts, it is important to determine not only whether the information requested is easily available in business records, but also whether the Census Bureau's understanding of the concepts aligns with respondents' understanding.
|
From page 63... ...
Willimack, U.S. Census Bureau, "Testing and Evalu ation of Questions and Instruments," during the panel's June 3, 2016, meeting.
|
From page 64... ...
provide an overview of methods that can be used in the concept and data item definition stage. The results of these investigations can be used to inform the design of a questionnaire, as well as the Census Bureau's communications strategy.
|
From page 65... ...
Although in infor mation provided to the panel some surveys cited input from the Economic TABLE 4-3 Groups Responsible for Content in Questionnaires Used for the Census Bureau's Annual Economic Surveys Subject Matter Subject Matter Subject Matter Survey within EWD within EID within CES ESMD ASM x M3UFO x MOPS x x ARTS x AWTS x SAS x x ACES x ICTSa x ASE x x SQ-CLASS x COS x x NOTES: See text for full names of the surveys. aSurvey is currently suspended.
|
From page 66... ...
We envision that EMD would establish a content harmonization team, separate from the staffs of the individual annual economic surveys, to develop harmonized concepts to be introduced into the various annual economic surveys and ultimately into a new ABSS. The team would have a goal of developing concepts for a core questionnaire and tailored subquestionnaires.
|
From page 67... ...
RECOMMENDATION 4-3: The Census Bureau should involve both subject-matter experts and methodologists in questionnaire redesign for the annual economic surveys, moving toward an A nnual Business Survey System. These experts should be part of a content harmonization team that works on concept harmoniza tion and feasibility studies, through new question construction and pretesting, to final questionnaire design.
|
From page 68... ...
ASE, launched in 2014 as a collaboration between the Census Bureau, the Marion Ewing Kauffman Foundation, and the Minority Business Devel opment Agency, covers all nonfarm employer businesses in most sectors of the economy. In addition to core information about the characteristics of business owners, the length of time the business has been in operation, payrolls, and employment, each year's ASE has contained a supplemental module with questions on some special topic.
|
From page 69... ...
suggests a set of possible questions about how companies use information technology that could be considered for incorporation into ICTS. For the other topics listed above -- organizational practices, the use of workers under alternative employment arrangements, the contracting out of business functions, and offshoring and global value chains -- supplements to one or more of the current annual economic surveys could be developed.
|
From page 70... ...
The Census Bureau should determine which of these topics are appropriate for the annual economic surveys and an Annual Business Survey System and develop new or expanded questions and supplemental rotating modules accordingly. 4.4 DATA COLLECTION MODES Having identified the concepts and sets of data items that will be collected (both core data items and those specific to a subquestionnaire or supplemental module)
|
From page 71... ...
bASE uses mail follow-ups SOURCE: Information provided to the panel by the Census Bureau.
|
From page 72... ...
Some of this variation across the annual economic surveys is likely due to two different in-house web survey software programs having been used as platforms for electronic data collection. Until 2015, ASM and COS were conducted using an electronic instrument software application called Surveyor that required downloading to a respondent's PC, while AWTS, ARTS, SAS, ACES, and ASE have used Centurion, which is web-based.
|
From page 73... ...
Paradata may be less informative if many respondents first enter their information on a worksheet and then transfer it to the web instrument, although at this writing, there is no empirical information about the proportion of respondents to the annual surveys who do this or how it affects web completion. An innovative approach for the Census Bureau to consider for an ABSS is to move toward a data collection system that accesses information 10 Paradata are data collected in real time about survey processes; for an overview of para data related to business survey errors, see Snijkers and Haraldsen (2013, pp.
|
From page 74... ...
RECOMMENDATION 4-5: For the annual economic surveys and a future Annual Business Survey System, the Census Bureau should continue to develop and enhance a data collection strategy that takes maximum advantage of electronic data collection modes at all stages in the survey process. The Bureau should use a consistent software platform across all surveys and make use of paradata to identify questions that are problematic for respondents and would benefit from redesign.
|
From page 75... ...
. Questionnaires also TABLE 4-5 Questionnaire Design Elements for Harmonizing the Annual Economic Surveys and a Future Annual Business Survey System Individual Questions Questions • Question type: open, closed, partially closed • Question wording Response options • Closed questions: wording of options, order of options • Open questions: answer format (e.g., $ or 1,000 $)
|
From page 76... ...
, but encourages the Bureau to review the extensive literature on questionnaire design that has been conducted in the decade since these guidelines were written as input to the process of revising questionnaires for the annual economic surveys and developing an ABSS questionnaire. At the question level, similar questions across questionnaires can be designed as one, with necessary tailoring to industry differences ("tailoring within harmonization")
|
From page 77... ...
SOURCE: Figure provided to the panel by the Census Bureau. version management.
|
From page 78... ...
According to a Census Bureau presentation to the panel, the Bureau currently uses expert reviews by methodologists and subject-matter specialists, exploratory early-stage scoping interviews, cognitive interviews, u sability testing, low-fidelity prototypes, and other methods for development and testing of the agency's annual economic surveys.13 Other methods, such as recordkeeping studies or field pilot data, are used less routinely, while analysis of paradata from web survey instruments is only beginning to be used as the agency has staff members with the appropriate skills. This sort of pretesting has revealed a number of ways in which the current a nnual economic surveys do not match business records.
|
From page 79... ...
During the transition from the current surveys to an ABSS, communications with sampled business units that are or have been sampled for one or more of the current surveys will need to inform them about what has changed and why. 4.6.1 Communication Methods Current Census Bureau practice with regard to communications for the annual economic surveys has been informed by lessons learned through experience.
|
From page 80... ...
The panel welcomes experimentation aimed at improving the efficiency of contact strategies for the annual economic surveys and encourages such 15 Based on a presentation by Charles Brady, U.S. Census Bureau, "Annual Survey Account Manager Program," during the panel's June 3, 2016, meeting.
|
From page 81... ...
Such "tailoring within harmonization" would facilitate effective communication with sampled businesses and continued improvements by Census Bureau staff, since their efforts could be focused on an integrated survey system and not be spread among separate surveys as is currently the case. The current communications strategy for each of the annual economic surveys also includes FAQs available on websites about the surveys.
|
From page 82... ...
Depending on the nature of the data being collected for an industry, however, some tailoring of timing may be required. One Census Bureau concern relayed by Diane Willimack at a panel meeting was that some respondents to the annual economic surveys find the timing of the survey requests inconvenient, as they arrive when respondents are busy with other reporting obligations, such as filings to the Securities and Exchange Commission, tax filings, or preparation of internal financial reports.18 Survey timelines are based in part on when the survey samples can be generated and on data dissemination requirements, but the Census Bureau also attempts to apply best practices for contacting respondents.
|
From page 83... ...
aSurvey is currently suspended. SOURCE: Information provided to the panel by the Census Bureau.
|
From page 84... ...
Paradata for the current surveys can be analyzed as input for developing the fielding strategy for an ABSS. Census Bureau staff presented analyses of this kind to the panel.19 Paradata related to respondents' login experiences, such as the number of login attempts, successful and unsuccessful login attempts, and logins by unique persons, also can be informative.
|
From page 85... ...
The panel commends the Census Bureau's work in developing measures and tools for active monitoring, and encourages the continuation and extension of that work. RECOMMENDATION 4-7: The Census Bureau should work to harmonize its communications and fieldwork strategies across the annual economic surveys, leading to a strategy for an Annual Busi ness Survey System (ABSS)
|
From page 86... ...
SOURCE: Figure provided to the panel by the Census Bureau. 4.7 REFERENCES NOTE: All URL addresses were active as of November 2017.
|
From page 87... ...
Census Bureau. Available: http://citeseerx.ist.psu.edu/viewdoc/download?
|
From page 88... ...
Washington DC: U.S. Census Bureau.
|
From page 89... ...
U.S. Census Bureau Statistical Quality Standards Section A2, Developing Data Collection Instruments and Supporting Materials.
|
Key Terms
This material may be derived from roughly machine-read images, and so is provided only to facilitate research.
More
information on Chapter Skim is available.