Measuring Personal Travel and Goods Movement: A Review of the Bureau of Transportation Statistics' Surveys -- Special Report 277 (2003)

The Bureau of Transportation Statistics (BTS) was established within the U.S. Department of Transportation (USDOT) by the 1991 Intermodal Surface Transportation Efficiency Act. The agency was charged with providing comprehensive, systemwide transportation data for policy making, planning, and research purposes. Today, BTS’s statistics are used to support transportation decision making by all levels of government, transportation-related associations, private businesses, and consumers.

The Transportation Equity Act for the 21st Century (TEA-21) authorized BTS at an annual funding level of $31 million for the 6-year period from 1998 through 2003. Against the backdrop of the impending reauthorization of TEA-21, BTS asked the National Academies to review the agency’s current survey programs in light of (a) transportation data needs for policy planning and research and (b) the characteristics and functions of an effective statistical agency. In response to this request, the Transportation Research Board and the Committee on National Statistics of the National Academies established a 12-member committee to conduct the review. The committee reviewed BTS’s three major surveys—the National Household Travel Survey (NHTS), the Commodity Flow Survey (CFS), and the Omnibus Survey Program—and issued a letter report on each survey providing specific guidance to BTS on approaches for improving future versions of the surveys. In this report, major themes identified from the reviews of individual surveys are addressed, and crosscutting guidance to BTS about its portfolio of transportation surveys is offered.

The committee characterized the NHTS and CFS as BTS’s flagship personal travel and freight surveys, respectively. These major, multiyear

survey programs with budgets on the order of $10 million to $15 million serve a broad constituency of organizations and individuals interested in transportation, providing essential data that are not available from other sources. Users include USDOT, other federal agencies, the U.S. Congress, state departments of transportation, metropolitan planning organizations, consulting companies, academia, think tanks, and industry associations.

The committee views the flagship surveys as essential to the BTS mission of providing statistical information to support transportation decision making. Therefore, the committee’s analyses and recommendations focus on opportunities for BTS to improve these flagship surveys. The Omnibus surveys, by contrast, are small-scale, quick-response efforts with relatively modest budgets. Initiated in 2000, the Omnibus program of customer satisfaction surveys serves primarily clients within USDOT and, in the committee’s judgment, constitutes a small component of the BTS survey portfolio. Nonetheless, the committee was concerned that the variable quality of surveys conducted under the Omnibus program, combined with inadequate procedures for approving these surveys, could undermine BTS’s credibility as an independent provider of transportation data.

To develop cost-effective, high-quality surveys responsive to the needs of data users, BTS has to communicate effectively with its customers. A better understanding of the types of questions and analytical problems addressed by users would help BTS develop relevant data products. In addition, many users could provide BTS with valuable suggestions about data concepts, methods, and products in the context of a dialogue about the agency’s survey development and design activities.

In general, BTS’s outreach activities for communicating with users of its personal travel and freight surveys have been sporadic. Some initiatives, such as the 1999 conference to discuss the proposed new personal travel survey (the NHTS),¹ have been valuable in facilitating discussions of spe-

cific issues. Nevertheless, the agency does not appear to have a rigorous, systematic strategy for interacting with its customers on a regular basis.

BTS’s efforts to develop its flagship surveys are further complicated by a lack of clearly defined survey objectives. For example, in the case of the CFS, a decision about whether the survey is to provide data on state-to-state flows in addition to general national flows is key to developing a cost-effective sampling design. For transportation surveys in general, parameters such as sample size need to be determined on a rational statistical basis that reflects user requirements for reliable data at specified levels of geographic detail. In the absence of clear objectives, the statistical foundation needed to inform quality/quantity/cost trade-offs inherent in the survey design process is lacking, and the survey scope itself may be ambiguous. As a result, available resources may not be used effectively to meet the needs of data users.

From a user’s perspective, an important feature of the NHTS and CFS is stability. Users count on the data products being made available on a regular, periodic basis, with the quality and content at least as good as that of earlier surveys. However, the history of the flagship personal travel and freight surveys has been characterized by variations in budgets and changes in survey ownership that threaten to undermine survey stability and quality. Budget variations have resulted in irregular survey frequency and reductions in sample size. The former limit the ability to measure trends, while the latter are likely to have adverse effects on data usability. As a result of changes in ownership, both flagship surveys now are funded and conducted by BTS in conjunction with survey partners.² BTS is largely dependent on the institutional memory of these partners to provide continuity and build on experience with previous surveys.

Ensuring the stability and quality of major national surveys such as the NHTS and CFS requires long-term planning and technical development,

and a clear and timely commitment by the survey partners to provide the necessary funding. Given the importance of the flagship personal travel and freight surveys to a broad spectrum of data users, the committee believes measures are needed to prevent a repeat of the 2002 CFS scenario in which delays in committing funds eliminated most opportunities for survey improvement and innovation, and almost resulted in cancellation of the survey.

The purpose of BTS’s portfolio of survey programs is to provide transportation data products that are responsive to customer needs, relevant to policy and investment decisions affecting the transportation enterprise, and appropriate to a federal statistical agency. The development of products such as the CFS and NHTS needs to be driven not only by statistical considerations but also by a broad understanding of the nation’s transportation system and sensitivity to related policy issues. The committee’s reviews of individual survey programs led it to conclude that BTS lacks the balance of expertise needed to guide the development of data products for informing transportation decision making. In particular, a better understanding of transportation issues could have resulted in better survey design and implementation decisions in some instances. For example, the reduced budget for the 2002 CFS was accommodated by halving the sample size to 50,000 establishments, compared with 100,000 in 1997. More informed insights into the uses of freight flow data, and in particular the need for reliable data at specific levels of geographic detail, could have highlighted the importance of seeking additional funds or investigating creative ways to maintain the sample size for the 2002 CFS at a level comparable with that of the 1997 survey.

Continuing to provide useful, high-quality survey products over a period of many years requires an ongoing program to research and implement more effective survey methods. As a result of social and technological changes, survey methods that yielded good data 15 or 20 years ago may no longer give such satisfactory results. For example, defensive measures by consumers to deflect telemarketing calls, combined with the growing number of cell-phone-only households, are reducing the effectiveness

of many telephone surveys. These factors may have contributed to the 41 percent response rate for the 2001 NHTS—a value that gives cause for concern because of the potential for significant nonresponse bias in the results. At the same time, technical developments may provide opportunities for more cost-effective data collection—an important benefit for BTS as it seeks to fulfill users’ data needs in the face of pressure on survey budgets. For example, while the 2002 CFS data were collected entirely by mail, the Census Bureau investigated electronic reporting as part of the 2002 Economic Census and has tentative plans to provide the option of a Web-based questionnaire for the 2007 CFS. Such an approach offers the potential to reduce data entry costs as well as to improve data quality through automated editing that assists respondents while they are in the process of completing the questionnaire.

As a relatively new statistical agency, BTS does not have an established tradition of research into survey methods. Nonetheless, many of the methodological issues the agency faces in developing the NHTS and CFS are common to surveys in general, and much of the extensive technical literature on survey methodology is pertinent to BTS’s flagship surveys. Leveraging existing work on survey methods could allow BTS to focus its limited research budget on efforts to solve its particular survey problems and investigate topics specific to transportation surveys.

The committee identified five main topic areas in which improvements in the effectiveness of BTS’s survey methods could enhance the quality and usefulness of the resulting data products:

• Response rates for household travel surveys,

• Data collection,

• Sample design,

• Questionnaire development and testing, and

• Data dissemination.

Recommendations 1 through 7 identify actions BTS could take to render its flagship surveys more effective in meeting the needs of a broad spectrum of data users. Recommendation 8 addresses the Omnibus program.

BTS should continue to conduct and enhance the NHTS and the CFS, its flagship surveys on personal travel and goods movement in the United States.

BTS, together with its CFS and NHTS partners, should establish a formal process for (a) eliciting and responding to the needs of the community of data users on a regular basis and (b) consulting these users about key decisions affecting future surveys.

BTS should use clear and explicit survey objectives (e.g., scope and scale), developed in conjunction with its survey partners and users, to inform the design and implementation of future editions of the NHTS and CFS.

BTS should establish institutional procedures and long-term financial plans that help ensure the stability and quality of its flagship personal travel and freight surveys.

BTS should work with its survey partners to establish a clear understanding of respective roles and to define clear lines of organization and management.

BTS should enhance and maintain the transportation expertise of its staff to achieve a balance between statistical and transportation knowledge.

BTS should address technical problems associated with its major surveys by making those problems a focus of its applied research program.

BTS should establish a process for conducting the Omnibus surveys that ensures the agency’s credibility as an independent provider of statistical information.

This page intentionally left blank.