Description and Assessment of Bureau of Transportation Statistics’ Surveys
In this chapter, an overview of the three Bureau of Transportation Statistics (BTS) surveys that the committee reviewed—the National Household Travel Survey (NHTS), the Commodity Flow Survey (CFS), and the Omnibus Survey Program—is provided, and the committee’s assessments of each of these surveys are summarized. The committee’s detailed findings and recommendations are provided in the letter reports reproduced in Appendixes A, B, and C. The purpose of this chapter is to set the context for the committee’s conclusions and recommendations, presented in Chapters 3 and 4, rather than to provide detailed descriptions of BTS’s survey programs. Further information on the surveys is given in the committee’s letter reports and in the accompanying references.
Both the NHTS and CFS are funded and conducted by BTS in conjunction with survey partners, as discussed later in this chapter. Furthermore, both have evolved from surveys that predate the establishment of BTS in 1991. Thus, although the NHTS and CFS are described throughout this report as BTS’s surveys, the agency does not bear the sole responsibility for these survey programs or for the manner in which they have evolved over time.
NATIONAL HOUSEHOLD TRAVEL SURVEY
The NHTS is a personal travel survey of the civilian, noninstitutionalized population of the United States. The survey is conducted by BTS, the Federal Highway Administration (FHWA), and their contractors, and funded by three agencies within the U.S. Department of Transportation (USDOT), namely, BTS, FHWA, and the National Highway Traffic Safety Administration (NHTSA). The total budget for the 2001
NHTS was $10.7 million, of which $300,000 came from NHTSA and the remainder approximately equally from BTS and FHWA. In 2001, the NHTS superseded two earlier federal government surveys of personal travel in the United States. The Nationwide Personal Travel Survey (NPTS) investigated daily travel and was conducted five times between 1969 and 1995. The American Travel Survey (ATS) investigated long-distance travel and was conducted twice, once in 1977 and again in 1995.
The purpose of the NHTS is to provide information on personal travel within the United States. Detailed data from a sample of U.S. households on daily and longer-distance travel for all purposes and by all modes are expanded to provide national estimates of trips and miles by travel mode, trip purpose, and household attributes. Aside from information on journey-to-work trips reported in the Decennial Census and the American Community Survey, the NHTS is the only national source of information on the typical travel of U.S. residents. The survey provides data on the type and amount of travel, the use of various modes, the time and miles spent traveling for various purposes, ownership and use of the vehicle fleet, and relationships among household composition, life stage, and travel.
The NHTS collects data from a nationally representative sample of households to derive statistically reliable travel estimates at the national level. The size of the national sample is insufficient to provide statewide or area-specific estimates, but states and metropolitan planning organizations (MPOs) can purchase additional samples of households in their jurisdictions to support local studies. These add-on samples are surveyed as part of the larger NHTS effort. For the 2001 survey, the national sample comprised approximately 26,000 households. In addition, five state departments of transportation and four MPOs purchased supplemental samples for their local planning efforts. These supplemental samples involved a total of 40,000 additional households.
The 2001 NHTS data were collected using computer-assisted telephone interviewing (CATI) methods and a random digit dialing (RDD),
list-assisted sample. In common with many household travel surveys, data were collected in a two-stage process. A recruitment interview obtained demographic information and rosters of household members and vehicles. Map and diary packages were then mailed to recruited households to help them keep track of their travel. The subsequent data-gathering interview obtained information on household travel on a preassigned travel day as well as on longer-distance travel over a 28-day travel period.
Preliminary 2001 NHTS data for the national sample of 26,000 households were released in January 2003, approximately 8 months after the completion of data collection. Data on households, persons, vehicles, and daily trips can be downloaded from the NHTS website (nhts.ornl. gov/2001/), together with supporting documentation. The data are also available on CD. In addition, an online analysis tool allows users to generate travel statistics without having a detailed knowledge of data file structures. Further data for the national sample, together with data for the 40,000 households in the supplementary regional samples, are scheduled for release in October 2003.
The committee found that data from the NPTS and ATS have proved useful to policy makers and planners at national, state, and local levels, as well as to researchers, industry associations, and public interest groups. The data are used primarily for analyzing policy issues, setting funding priorities, and monitoring trends in travel behavior. Data from the 2001 NHTS are expected to prove similarly useful.
Despite their many uses, data from national surveys such as the NPTS, ATS, and NHTS do not meet the needs of all users. In particular, such national data do not generally provide the level of detail required to inform decisions about location-specific issues. In addition, the lack of contextual information—for example, information on the availability and quality of local transportation services—limits the usefulness and relevance of the data for model estimation and some policy analyses.
The committee identified opportunities for BTS to improve its personal travel surveys in terms of both their value to a wide range of users and the quality of the data provided. The committee’s concerns about data quality focused on the response rate for the 2001 NHTS. Although the final response rate of 41 percent is relatively high compared with response rates for household travel surveys conducted by MPOs across the United States, it is low compared with response rates for other federal policy-related surveys and raises questions about the validity of the data as a basis for decision making. The committee was also concerned about the lack of formal processes for identifying users of BTS’s household travel data and for modifying surveys to meet user needs.
The diversity of analysis and decision needs to be met by the NHTS led the committee to conclude that it may become increasingly difficult to meet user requirements for both quality and subject coverage with a single periodic survey. Therefore, the committee recommended that BTS consider developing a family of personal travel surveys that take advantage of different survey designs and supporting technologies to collect household travel data. These surveys would likely differ from each other in content, coverage, methodology, and frequency, but would be designed such that data from different surveys could be readily combined. To inform both the development of this family of surveys and future enhancements of the NHTS, the committee recommended that BTS develop a formal program for identifying and interacting with current and potential users of its personal travel data. Such a program would help the agency better understand the needs of data users and their perspectives on issues such as data quality.
The committee also recommended that BTS take advantage of a range of design concepts and new technologies in its continuing efforts to improve the response rate and data quality for the NHTS. These efforts could benefit from related research, and the committee suggested, therefore, that BTS assume a leadership role in research into methodologies for conducting transportation surveys. This action would help ensure that current and emerging issues relating to survey quality are investigated and the results incorporated into the agency’s future surveys.
COMMODITY FLOW SURVEY
The CFS is a national survey of business establishments in selected industries, namely, mining, manufacturing, wholesale trade, and certain retail establishments. The survey captures data on shipments of goods originating from a sample of such establishments located in the 50 states of the United States and the District of Columbia. The CFS is conducted by BTS and the Census Bureau of the U.S. Department of Commerce. The Census Bureau administers the survey as part of the 5-yearly Economic Census. The budget for the 5-year cycle of the 2002 CFS is $13 million, of which 80 percent is provided by BTS and 20 percent by the Census Bureau. The CFS has been conducted three times—in 1993, 1997, and 2002.
The purpose of the CFS is to supply information on the flow of goods by mode of transport within the United States. Data are provided on tons, miles, ton-miles, value, shipment distance, commodity, and weight. All major modes of freight transportation (air, motor carrier, rail, water, and pipeline) and intermodal combinations are covered. Despite gaps in shipment and industry coverage, the CFS is the only federal government data source that recognizes the need for such comprehensive information on freight flows.
The CFS sampling frame is drawn from the Census Bureau’s Business Register of 6 million employer establishments, of which approximately 750,000 are in industries covered by the CFS. The sample size has been halved each time the survey has been conducted, falling from 200,000 establishments in 1993 to 100,000 in 1997 and 50,000 in 2002.
For all three editions of the survey, CFS data were collected entirely by mail. Respondents were asked to report the total numbers of their outbound shipments and, for a sample of these shipments, information on value, weight, commodity, domestic destination or port of exit, and mode (or modes) of transport. The survey questionnaire included instructions to respondents on how to take a sample of their shipments.
For the 2002 CFS, each establishment was assigned a 1-week reporting period every quarter, for a total of 4 weeks in the calendar year. By assigning different reporting periods to different establishments, the sample covered all 52 weeks of the year.
The Census Bureau makes a range of CFS data products available to the public in printed reports, on CD, and on the Web.1 The published CFS data at the national level tabulate information on shipment characteristics by mode of transport and by commodity. Additional reports provide geographical breakdowns for flows between census divisions and regions, individual states, and major metropolitan areas. Reports on movements of hazardous materials and on exports are also published. Although such summary tables are useful, many analysts seeking to use the data as input to their own models and calculations would prefer a database providing access to origin–destination flow patterns.
The committee found that analysts and researchers in both the public and private sectors use data from the CFS—often in conjunction with data from other sources—for a variety of purposes. Uses of CFS data include the analysis of trends in goods movement over time, economic analyses, the development of models and other analytical products to inform policy analyses and management and investment decisions, and the analysis and mapping of spatial patterns of commodity and vehicle flows. Despite their many uses, CFS data are inadequate for some applications because of gaps in shipment and industry coverage, a lack of geographic and commodity detail at the state and local levels, and the inability of a 5-yearly survey to capture rapid changes in economic cycles.
The committee also found that the design of the 2002 CFS appears to have been compromised in important ways by the lack of a clear understanding between BTS and the Census Bureau about their respective
roles and responsibilities. In particular, confusion over the responsibility for ensuring sufficient funding to produce a useful, quality product resulted in uncertainty about the availability and level of funding until late in the survey planning process. As a result, key design decisions were delayed, and opportunities for advance preparation and problem solving were limited.
The committee recommended that BTS continue to provide data on the flow of goods by mode of transport within the United States. To this end, it recommended that the CFS be continued—with some modifications—at least until a viable alternative source of national freight data has been established. BTS and the Census Bureau should proceed with planning for the 2007 CFS, and this planning effort should explore opportunities for conducting pilot studies of new methods in parallel with established designs. In the context of efforts to improve survey quality and cost-effectiveness, the committee recommended that BTS and the Census Bureau initiate a research program to investigate survey methods for the CFS and any successor surveys.
The committee also recommended that, in developing future versions of the CFS (or its successors), BTS and the Census Bureau solicit user input to the design process through dialogue with CFS users and other outreach mechanisms. Furthermore, the CFS partnership (BTS and the Census Bureau) needs to ensure that the rationale for major survey design decisions is documented and discussed in such a way as to engage users in decision making.
Finally, the committee recommended that BTS and the Census Bureau reevaluate their roles and responsibilities within the CFS partnership to make the most effective use of the expertise and experience of both parties.
OMNIBUS SURVEY PROGRAM
The Omnibus Survey Program currently comprises a monthly household travel survey that addresses a range of transportation issues, and up to a maximum of four targeted surveys per year that address special
transportation topics.2 The Omnibus surveys are conducted and funded by BTS in conjunction with other administrations in USDOT, such as NHTSA and the Maritime Administration. The division of survey tasks and funding responsibilities among BTS and its survey partners varies on a case-by-case basis. The committee was unable to obtain estimates of the total costs of any of the Omnibus surveys because BTS staff time spent on these surveys is not itemized. Nonetheless, information on survey design features and contractor costs led the committee to conclude that the Omnibus Survey Program is a relatively modest effort compared with the NHTS and CFS.
The Omnibus monthly household survey was initiated in August 2000 and has been conducted on a monthly basis since then, apart from a 3-month hiatus between April and June 2001. The targeted surveys were also initiated in 2000 and, to date, eight surveys have been completed or are in progress.
The core function of the Omnibus program is to assess customer satisfaction with various aspects of the transportation system, although the surveys also include questions designed to obtain factual (behavioral) information on transportation use or other transportation-related issues. The Omnibus program focuses on meeting some of the information needs of customers within USDOT. For example, data from the surveys assist the department in complying with the requirements of the Government Performance and Results Act of 1993 for federal agencies to measure their performance and effectiveness. The survey data also help inform transportation policy decisions. Thus, the monthly household survey delivers timely data on topical subjects such as travelers’ experience with airport security screening, as well as providing a means of monitoring the public’s use of and opinions about the transportation system. To date, the targeted surveys have been used to provide information on populations of special interest, such as mariners, air travelers,
cyclists and pedestrians, and travelers with disabilities. These surveys could also be used to investigate a wider variety of transportation policy issues.
Every month, the Omnibus household survey collects data from approximately 1,000 households nationwide using CATI and an RDD telephone methodology. The survey questionnaire includes a core set of transportation questions (which remain the same from month to month), questions to assess progress in achieving USDOT’s strategic goals, and questions supplied by the USDOT modal administrations. Examples in the latter category include questions from NHTSA about headlight glare and tire pressure measurement.
In contrast to the monthly household survey, which relies on telephone methods to gather data, the targeted surveys use a variety of data collection methods—including mail out/mail back, telephone, and Web-based approaches—depending on the survey objective and the target population. For example, the 2001 Mariner Survey was conducted primarily by mail, but telephone interviews were conducted with some nonrespondents in an effort to increase the overall response rate. The data collection cycle for targeted surveys is determined by information requirements and, in contrast to the monthly household survey, is not routinely constrained by the need for a quick response. The sample size is determined by the purpose of the survey and the availability of resources.
Data from the Omnibus monthly household survey are made available on the BTS website (www.bts.gov) and are also used by the agency to prepare OmniStats, two- or three-page popular reports on items of widespread interest. Recent issues of OmniStats have addressed security screening at the nation’s airports, recreational boating, and bicycle use among adults.
The results of the targeted surveys may be published in reports from BTS or other USDOT administrations. On occasion, the data may be made available to the public on the BTS website, subject to the agreement of the USDOT administration sponsoring the survey.
The committee found that the Omnibus Survey Program has value for USDOT because it delivers timely data to inform decision making. However, the features that make the Omnibus program an attractive tool for policy makers also raise concerns about the potential of the program to damage BTS’s credibility as an independent provider of transportation data. The opportunity to obtain timely public opinion data on topical transportation issues carries a concomitant obligation for BTS, as a federal statistical agency, to make a clear distinction between statistical information and policy interpretation. The committee was particularly concerned that current BTS procedures for approving the Omnibus surveys may not adequately safeguard the agency’s independence because they do not ensure that every survey is subject to rigorous, objective, and informed review of its content and method before being fielded.
The committee found the quality of surveys conducted under the Omnibus program to be variable. While some of the targeted surveys, notably the 2001 Mariner Survey, are of high quality, the monthly household survey gave cause for concern. In particular, there is a risk that the quality of the latter survey will be compromised by the time constraints imposed by the monthly schedule. The committee was also concerned that BTS analyses and reporting of the results of the Omnibus monthly household survey do not consistently meet the quality standards expected of a federal statistical agency.
The committee recommended that BTS continue its Omnibus Survey Program as a relatively low-budget activity providing timely information on a range of transportation issues. However, to safeguard the agency’s independence, the committee also recommended that BTS establish an independent review mechanism for the Omnibus program involving experts from outside the agency. Such a mechanism should ensure that proposed surveys are consistent with BTS’s overall mission and do not address inappropriate questions that could undermine the agency’s independence.
The committee also recommended that BTS implement measures aimed at ensuring that all surveys conducted under the Omnibus program
are of a consistently high quality. These measures should include the establishment of effective procedures for developing and evaluating survey questionnaires and the aggressive pursuit of strategies for increasing the response rate for the monthly household survey. Reducing the frequency of the household survey from monthly to quarterly merits consideration because of the possible resulting improvements in survey quality. The committee further recommended that BTS take steps to ensure that its analyses of Omnibus survey data are technically robust and that the resulting products comply with established guidelines for the reporting of statistical data.
The next chapter draws on the outcomes of the committee’s reviews of individual surveys, together with the evaluation framework described in Chapter 1, to identify seven major themes pertinent to BTS’s current survey programs. In addressing these themes, the committee’s conclusions emphasize crosscutting issues relating to the NHTS, the CFS, and the Omnibus program.