Reengineering the 2010 Census: Risks and Challenges (2004)

FOR THE READER’S CONVENIENCE, we list below the specific recommendations that appear in the text of this report. The recommendations are grouped by chapter.

Recommendation 3.1: The Census Bureau must devise a plan and develop effective procedures for updating and correcting the Master Address File (MAF). A complete and accurate Master Address File is critical not only to the success of the 2010 census but also to the effective implementation of the American Community Survey, the other household surveys conducted by the Census Bureau, and the 2008 dress rehearsal. Because the 2000 MAF was not simply discarded following the 2000 census (as occurred in censuses prior to 1990), the 2010 census will have as a base an address file of unprecedented completeness, but that does not obviate the need for continual updating, filtering, unduplicating, and cleaning of the MAF during the years leading to the 2010 census.

The plan for a continually updated 2010 MAF must include, but not be limited to, the following:

1. A clear articulation of how the MAF/TIGER Enhancements Program and other Census Bureau activities will add missing

housing unit addresses, remove duplicate addresses, and generally correct the Master Address File, independent of benefits derived from being cross-referenced to an updated TIGER database;

2. More effective definitions of housing units and methods to obtain accurate address listings for structures containing multiple housing units, as it is not sufficient to know only the address or geographic coordinates of the structure location;

3. Detail on the temporal sequencing and adequacy of address updates from the U.S. Postal Service’s Delivery Sequence File, the Census Bureau’s Community Address Updating System, and as-yet unspecified local partnership programs;

4. More effective means to define, list, and enumerate group quarters living arrangements, which should be done in coordination with the development and maintenance of the MAF; and

5. A detailed plan for Objective Five (quality metrics) of the MAF/TIGER Enhancements Program, including a program of evaluation and assessment of MAF coverage and input to the MAF/TIGER Redesign (Objective Two), so that the revised database structure includes appropriate address source codes and other useful variables for evaluation.

Recommendation 3.2: The Census Bureau should create and staff a position to oversee the development and maintenance of the MAF as a housing unit inventory, with a focus on improving methods to designate, list, and update units. This position should be responsible for development and implementation of plans drawn up consistent with Recommendation 3.1.

Recommendation 3.3: The Census Bureau should pursue more effective partnership and research collaboration with the U.S. Postal Service, including but not limited to further work on “undeliverable as addressed” items from the 2000 census, assessment of the address coverage quality of the Delivery Sequence File (DSF), and possibilities for more accurate translation of

post office box listings and other DSF entries to street addresses and geographic coordinates.

Recommendation 3.4: The Census Bureau should assess how critical the Community Address Updating System (CAUS) is to providing address updates in rural, non-city-style address areas. Such an assessment should include not only estimates of the number of addresses that could be provided and the workload that could be handled by CAUS/American Community Survey staff, but also empirical evidence on coverage gaps in the U.S. Postal Service’s Delivery Sequence File by geographic area or type.

Recommendation 3.5: The Census Bureau should immediately develop and describe plans for partnerships with state, local, and tribal governments in collecting address list and geographic information. Such plans should include a focus on adding incentives for localities to contribute data to the census effort, making it easier for localities and the Bureau to exchange geographic information. Accordingly, plans for partnerships should include:

• clear articulation of realistic schedules for local input and review;

• definition and clear presentation of benchmark standards for local data to be submitted to the Bureau;

• mechanisms for providing effective feedback to local and tribal governments, detailing and justifying the Bureau’s decisions to use or not use the information provided; and

• coordination of efforts across the Bureau so that calls for local and tribal entities to supply input to the Master Address File, TIGER, the Boundary and Annexation Survey, and other Bureau programs are not unduly redundant and burdensome.

Recommendation 3.6: The Census Bureau should evaluate the necessity of its plans to conduct a complete block canvass shortly before the 2010 census. Such justification must include analysis

of extant census operational data and should include, but not be limited to, the following:

1. arguments as to why selective targeting of areas for block canvass is either infeasible or inadequate, and as to how the costs of the complete block canvass square with the benefits; and

2. analysis of how a full block canvass fits into the Census Bureau’s cost assumptions for the 2010 census.

If plans proceed for a complete canvass, the Bureau should also consider how such a mass field deployment prior to 2010 could be used to achieve other improvements or efficiencies, such as the collection of GPS trace data as supplement to or as quality control for the TIGER realignment.

Recommendation 3.7: The Census Bureau must:

• fully exploit the address source information in the MAF Extract in order to complete 2000 census evaluations, fill gaps in knowledge remaining from the 2000 census evaluations, and assess causes of duplicate and omitted housing units; and

• build the capability for timely and accurate address evaluation into the revised MAF/TIGER data architecture, including better ways to code address source histories and to format data sets for independent evaluation.

Recommendation 4.1: The Census Bureau should continue research to understand the differences between and relative quality of ACS estimates and long-form estimates, with particular attention to measurement differences and error from nonresponse and imputation. The Bureau must work on ways to effectively communicate and articulate those findings to interested stakeholders, particularly potential end users of the data.

Recommendation 4.2: The Census Bureau must make ACS data available (protecting confidentiality) to analysts in the 31 ACS

test sites to facilitate the comparison of ACS and census long-form estimates as a means of assessing the quality of ACS data as a replacement for census long-form data. Again, with appropriate safeguards, the Census Bureau should release ACS data to the broader research community for evaluation purposes.

Recommendation 4.3: The Census Bureau must issue a guide for users of ACS data that details the statistical implications of the difference between point-in-time and moving average estimates for various uses.

Recommendation 4.4: As soon as possible, based on the 2006 proof-of-concept test, the Census Bureau should work with Congress and the administration to secure agreement on the overall design for the 2010 census and the American Community Survey (ACS). Extended delay in finalizing an overall design for the 2010 census—such as occurred in the preparation for the 2000 census—would unacceptably heighten the risk associated with the 2010 census. The role of the ACS is of particular concern; failure to secure commitment to the ACS as a replacement for the census long form would severely impair plans for a short-form-only census and undercut the ability to provide reliable small-area characteristics data by 2010.

The Census Bureau should identify the costs and benefits of various approaches to collecting characteristics information if support for the full ACS is not forthcoming. These costs and benefits should be presented for review so that decisions on the ACS and its alternatives can be fully informed.

Recommendation 4.5: The Census Bureau should identify the costs and benefits of various approaches to collecting characteristics information should funding for the full ACS not be forthcoming. These costs and benefits should be presented for review so that decisions on the ACS and its alternatives can be fully informed.

Recommendation 5.1: The Census Bureau should develop and perform a rigorous test of its plans for use of portable computing devices, and this test should compare the performance and outcomes of data collection using:

• devices of the current (Pocket PC) class being developed for use in the 2004 census test;

• high-end devices (e.g., tablet computers) of classes that are very likely to be available at reasonable cost by the time of procurement for 2010; and

• traditional paper instruments.

Such a test is intended to provide fuller information about the costs and benefits of portable computing devices, using paper as a point of comparison. The test should also provide the opportunity to review specifications and requirements for the PCDs, using devices of the caliber likely to be available by 2010.

Recommendation 5.2: By the end of 2004, the Census Bureau should complete requirements design for its portable computing devices, building from the results of the 2004 census test and in anticipation of the 2006 proof-of-concept test. The requirements and specifications for portable computing devices must include full integration with the census system architecture and should include suitability for other, related Census Bureau applications. The Bureau’s requirements design for PCDs must devote particular attention to the human factors underlying use of the devices.

Recommendation 5.3: The Census Bureau must develop a complete engineering and testing plan for the software components of the portable computing devices, with particular attention to the computer-assisted personal interviewing interface, data capture systems, and communication/synchronization capabilities (including assignment of enumerator workload).

Recommendation 5.4: The Census Bureau’s techniques for enumerating the population in special places and group quarters must be completely evaluated and redesigned for the 2010 census. This effort must include (but not be limited to):

• clear definitions of group quarters;

• redesign of questionnaire and data content as appropriate, including a provision for handling data items that might best be provided by group quarters administrators rather than individual residents;

• collection of information, including additional addresses, that will be needed to facilitate unduplication of all census records;

• improvement of the address listing processes for group quarters, including coordination with the development of the Master Address File; and

• specification of enumeration and coverage evaluation plans for group quarters.

Recommendation 5.5: The Census Bureau’s development of tailored enumeration methods for special populations—including irregular urban areas, colonias, gated communities, and rural areas—must begin early, and not be put off for development late in the census planning cycle.

Recommendation 5.6: The Census Bureau must quickly determine ways to implement a second questionnaire mailing to nonresponding households in the 2010 census, in order to improve mail response rates. Such determination should be done in a cost-effective manner that minimizes duplicate enumerations, but must be made early enough to avoid the late problems that precluded such a mailing in the 2000 census.

Recommendation 5.7: The Census Bureau must develop comprehensive plans for unduplication in the 2010 census, in terms of both housing units and person records. Housing unit unduplication research and efforts should be conducted consistent

with objectives outlined in the panel’s recommendations related to the Master Address File. Person-level unduplication efforts should focus on improvements to the methodology developed for the 2000 Accuracy and Coverage Evaluation Program, including national-level matching of records by person name. It is essential that changes in unduplication methodology be tested and evaluated using extant data from the 2000 census and that unduplication methods be factored into the 2006 proof-of-concept test and 2008 dress rehearsal.

Recommendation 5.8: The Census Bureau must pursue research on the trade-off in costs and accuracy between field (enumerator) work and imputation routines for missing data. Such research should be included in the 2006 proof-of-concept test, and census imputation routines should be evaluated and redefined prior to the 2008 dress rehearsal. As appropriate, the American Community Survey research effort should also address the trade-off between imputation and field work.

Recommendation 5.9: The Census Bureau should conduct research into the effects of imputation on the distributions of characteristics, and routines for imputation of specific data items should be completely evaluated and revised as appropriate for use in the American Community Survey.

Recommendation 6.1: In order to achieve the full benefit of architecture modeling, the highest management levels of the Census Bureau should commit to the design and testing of a redesigned logical architecture, so that the most promising model can facilitate the implementation of an efficient technical infrastructure for the 2010 census.

Recommendation 6.2: To ensure the successful integration of new technologies and techniques in the census process, the Census Bureau should create and staff the position of system ar-

chitect for the decennial census. The selected candidate should have expertise in modeling business processes, designing large-scale systems, and conducting reengineering activities. The system architect must be given the authority to work with and coordinate efforts among the organizational divisions within the Census Bureau and should serve as a champion of the importance of architecture reengineering at the highest levels of management within the Bureau.

The Census Bureau should also consider designating a subsystem architect for portable computing devices and related field systems, as has already been done for the MAF/TIGER redesign. The efforts of the MAF/TIGER redesign and PCD subsystem architects should be coordinated in partnership with the system architect for the decennial census.

Recommendation 6.3: As part of the MAF/TIGER redesign, the Census Bureau should consider ways to make its application code for mapping, geocoding, digital exchange, map editing, and other functions openly available in order to facilitate continued ties to and improvement in geographic information systems software applications and to tap the feedback of the broader computer science/software development community.

Recommendation 6.4: The Census Bureau should generally improve its software engineering processes and should pursue its goal of raising its Capability Maturity Model score in software development. In particular, the Bureau should focus on available tools and techniques in rigorously developing and tracking software requirements and specifications. In beginning the task of improving its software practices, though, the Census Bureau must recognize that the effort is a difficult one, requiring high-level commitment in the same manner as architecture reengineering.

Recommendation 6.5: The Census Bureau should evaluate and improve its protocols for hardware and software testing, drawing on expertise from the computer science and software development communities. Rigorous hardware and software testing should be factored into census operational schedules, in addi-

tion to the field testing performed in the 2006 proof-of-concept test, the 2008 dress rehearsal, or such other formal census tests as may arise.

Recommendation 7.1: The Census Bureau should continue to pursue methods of improving demographic analysis estimates, working in concert with other statistical agencies that use and provide data inputs to the postcensal population estimates. Work should focus especially on improving estimates of net immigration. Attention should also be paid to quantifying and reporting uncertainty in demographic estimates. Updated assessments of the assumptions underlying demographic analysis estimates, including the completeness of birth registration, should also be considered.

Recommendation 8.1: The Census Bureau should materially strengthen the evaluation component of the 2010 census, including the ongoing testing program for 2010. Plans for census evaluation studies should include clear articulation of each study’s relevance to overall census goals and objectives; connections between research findings and operational decisions should be made clear. The evaluation studies must be less focused on documentation and accounting of processes and more on exploratory and confirmatory research while still clearly documenting data quality. To this end, the 2010 census evaluation program should:

1. identify important areas for evaluations (in terms of both 2010 census operations and 2020 census planning) to meet the needs of users and census planners and set evaluation priorities accordingly;

2. design and document data collection and processing systems so that information can be readily extracted to support timely, useful evaluation studies;

3. focus on analysis, including use of graphical and other exploratory data analysis tools to identify patterns (e.g., mail return rates, imputation rates) for geographic areas and population groups that may suggest reasons for variations in data quality and ways to improve quality (such tools could also be useful in managing census operations);

4. consider ways to incorporate real-time evaluation during the conduct of the census;

5. give priority to development of technical staff resources for research, testing, and evaluation; and

6. share preliminary analyses with outside researchers for critical assessment and feedback.

Recommendation 8.2: A major focus of the Census Bureau’s ongoing research and evaluation program must be opportunities for targeting and efficiency—tailoring approaches to key population groups and areas rather than pursuing a “one-size-fits-all” approach.

Recommendation 8.3: The Census Bureau must mine and fully exploit data resources currently available in order to build a research base for the 2010 census and to further evaluate the 2000 census. These resources include:

• microdata from the 2000 Accuracy and Coverage Evaluation and its related Person Duplication Studies;

• extracts from the Master Address File;

• the Local Census Office Profile dataset;¹

• a match of census records and the March 2000 Current Population Survey; and

• the Master Trace Sample.

Recommendation 8.4: The Census Bureau should develop a list of studies important to 2010 census planning that can exploit the richness of the Master Trace Sample. These studies should be prioritized and then conducted as resources permit.

Recommendation 8.5: The Master Trace Sample from the 2000 census should be expanded to include data from group quarters enumeration, the Accuracy and Coverage Evaluation, and the Local Update of Census Addresses Program.

Recommendation 8.6: The Census Bureau should explore ways to allow the broader research community to perform analyses using the 2000 Master Trace Sample, subject to confidentiality limitations.

Recommendation 8.7: The Census Bureau should carry out its future development in this area of tracing all aspects of census operations with the ultimate aim of creating a Master Trace System, developing a capacity for real-time evaluation by linking census operational databases as currently done by the Master Trace Sample. Emerging 21st century technology should make it feasible to know almost instantaneously the status of various census activities and how they interact. Such a system should be seriously pursued by the Census Bureau, whether or not it can be attained by 2010 (or even by 2020).