The Bureau of Transportation Statistics (BTS) is charged with providing data to support decision making by diverse organizations within the broad transportation enterprise. In light of this charge, the committee identified the National Household Travel Survey (NHTS) and the Commodity Flow Survey (CFS) as key components of the agency’s portfolio of surveys. Seven of the committee’s eight recommendations identify actions BTS could take to render these flagship surveys more effective in meeting the needs of a broad spectrum of data users. The eighth recommendation addresses the Omnibus Survey Program.1
FLAGSHIP PERSONAL TRAVEL AND FREIGHT SURVEYS
Recommendation 1: BTS should continue to conduct and enhance the NHTS and the CFS, its flagship surveys on personal travel and goods movement in the United States.
Given the broad scope of BTS’s mandate and inevitable resource constraints on its programs, the committee urges the agency to adopt a strategy for enhancing the NHTS, CFS, and successor surveys that maximizes the value of the data for users at national, regional, state, and local levels. To do this, BTS must have a good understanding of how the NHTS and CFS are used and how they fit into the broader picture of transportation data.
In particular, the committee encourages BTS to explore opportunities that add value to the NHTS and CFS without involving major increases in survey cost. For example, it may be possible to assist users needing more detailed geographic data on freight movements by modifying the CFS such that national and local data sets can be readily combined.2 Similarly, revisions to the NHTS could help link data to the contextual information needed for policy analysis and model estimation. A recent National Research Council (NRC) report (Martin et al. 2001, 7) recommends that federal statistical agencies seek opportunities to integrate data from more than one survey or from other sources as part of the continuing development of more useful data, and the committee urges BTS to follow this advice. In particular, dialogue with users could address opportunities to coordinate data collection initiatives in an effort to obtain consistent and comparable local, state, and national data.
In developing and enhancing its flagship surveys, BTS will need to take account of user demands for both data consistency to support survey-to-survey trend analysis and expanded survey coverage to include evolving phenomena such as e-commerce. As noted in the letter report on the NHTS (Appendix A), changes in survey scope and design can result in additional complexity in trend analysis incorporating data from earlier surveys. Thus, BTS will need to develop a strategy to meet the potentially conflicting requirements for data consistency over time and expanded survey coverage.
Recommendation 2: BTS, together with its CFS and NHTS partners, should establish a formal process for (a) eliciting and responding to the needs of the community of data users on a regular basis and (b) consulting these users about key decisions affecting future surveys.
While information on the needs of data users does not guarantee good surveys, BTS cannot produce quality products without understanding these needs. Information on how and for what purposes data will be used is extremely important in developing an effective survey design.
Because the market for transportation data is diverse and evolving, BTS faces challenges in eliciting, understanding, and balancing the needs of its various constituencies to inform the development of its flagship surveys. For example, changes in the national transportation agenda, such as the increased focus on security since September 11, 2001, have resulted in new data requirements. Similarly, changes in the business environment, such as the revolution in freight logistics, have resulted in a requirement for more detailed and timely information about freight flows. These challenges are compounded by the fact that, as a relatively new statistical agency, BTS does not have long-standing relationships with its data users.
The committee urges BTS to develop outreach and interaction processes that facilitate open, accessible, responsive, and timely communication between the agency and current and potential users of its personal travel and freight surveys. A variety of processes will likely be required because these users are so diverse. In the case of the CFS, for example, “power” users—those who employ CFS data in their own analyses and models—could provide BTS with detailed technical input. Regular users—those who include CFS-based data in briefing papers and reports but do not themselves undertake data analyses—could provide the agency with more general feedback on the usefulness of survey products. In general, in consulting with a broad spectrum of users, BTS needs to employ “a variety of formal and informal means of communication that are appropriate to the types of input sought” (Martin et al. 2001, 9).
The committee encourages BTS to be proactive in reaching out to users by diverse means so as to better understand how transportation data are being used and for what purposes. Conferences and workshops convened by professional societies, transportation associations, and BTS itself constitute valuable opportunities for discussion of data needs and uses, and survey methods and their application. Nonetheless, because resource limitations preclude some users from traveling to these events, BTS also needs to adopt other approaches, such as surveys of users and a user-friendly feedback opportunity on its website, to broaden the scope of its outreach activities.
BTS may wish to consider using expert committees under the auspices of professional associations to serve as forums for interacting with users
of transportation data, survey researchers, and the transportation and statistics communities in general. Other federal statistical agencies, such as the Energy Information Administration and the Bureau of Justice Statistics, already adopt this approach. Thus, the Committee on Energy Statistics of the American Statistical Association (ASA) is charged with considering energy statistics as they relate to energy policy analysis and the framing of a comprehensive energy data system, and with promoting the integration of energy statistics with other statistical programs and with existing bodies of national statistics. Similarly, the ASA Committee on Law and Justice Statistics helps disseminate information about legal and justice statistics activities throughout the statistics community, and promotes the development of quality statistical activities in civil and criminal justice settings. A number of TRB standing committees, such as the Freight Transportation Data Committee, the Travel Survey Methods Committee, and the National Transportation Data Requirements and Programs Committee, could assist BTS in assessing the data needs of users, obtaining feedback on survey products, and providing a forum for discussing survey methods.
BTS’s outreach and interaction processes also need to include mechanisms for consulting users about decisions affecting the agency’s surveys. Openness in decision making can enhance the usefulness of surveys by providing users the opportunity to intervene and question BTS’s decisions during the survey planning process. Mechanisms for consulting users are also important for maintaining a relationship of mutual respect and trust between the agency and its customers. Thus, implementation of an open, interactive decision-making process is important if a customer-driven agency, such as BTS, is to fulfill its mission of providing data to meet client needs.
Recommendation 3: BTS should use clear and explicit survey objectives (e.g., scope and scale), developed in conjunction with its survey partners and users, to inform the design and implementation of future editions of the NHTS and CFS.
Clearly defined survey objectives provide the robust foundation needed to inform the decisions inherent in any major survey program. Therefore, the committee urges BTS to work with its survey partners and customers
(data users) in developing objectives that provide an explicit basis for defining the scope and content of proposed surveys. These objectives should be used in survey development and design to guide the agency’s examination of trade-offs between resource expenditures and the character and quality of survey products.
Key trade-offs affecting data content, scale, and quality should be identified and taken into account when making decisions about the allocation of resources. For example, a major decision in designing future editions of the CFS is determining the level of geographic detail the data will provide. Because this decision defines a minimum sample size, it will give a reasonable indication of whether resources allocated for the survey are sufficient. Concerted efforts should be made to avoid disadvantageous breakpoints—such as a sample size just below that needed to determine commodity flows at the requisite level of geographic detail. Under such circumstances, efforts to obtain additional resources or to increase the sample size through more cost-effective data collection or other efficiencies could yield a valuable payoff.
Recommendation 4: BTS should establish institutional procedures and long-term financial plans that help ensure the stability and quality of its flagship personal travel and freight surveys.
The establishment of BTS by the 1991 Intermodal Surface Transportation Efficiency Act resulted in the agency’s assuming important responsibilities for the flagship national transportation surveys (the NHTS and CFS). However, BTS currently lacks a history of intellectual investment in these surveys and the accompanying institutional memory needed to provide continuity and build on experience with previous surveys. Thus, the agency, in conjunction with its survey partners, needs to make special efforts to undertake the long-term planning and technical development required to support its NHTS and CFS programs.
In the committee’s view, much-needed stability could result from a clear commitment by BTS to deliver quality flagship survey products to its users and set its priorities accordingly. The committee is concerned that without a long-term commitment to the flagship surveys, and the accompanying staffing and financial resources needed to ensure program continuity, future personal travel and freight surveys may suffer the
same fate as the 2002 CFS. Delays in committing funds eliminated most opportunities for survey improvement and innovation and almost resulted in cancellation of the survey.
The committee urges BTS to recognize its flagship personal travel and freight surveys as core elements of its portfolio of programs and to manage its resources accordingly. Analysis of the costs of earlier surveys can provide BTS and its survey partners with a basis for developing reasonable cost estimates for future surveys. These estimates, together with information about anticipated budgets, could help inform decisions about survey timing and scheduling. For example, avoiding concurrent peaks in effort and expenditure for the two major programs is clearly helpful for budgetary and staffing purposes. As discussed in the committee’s letter reports on the NHTS and CFS (Appendixes A and C, respectively), continuous data collection may offer important benefits in this regard.
Recommendation 5: BTS should work with its survey partners to establish a clear understanding of respective roles and to define clear lines of organization and management.
One of the clearest lessons to emerge from the committee’s review of the CFS was that funding uncertainties can undermine the planning and preparation needed to develop a cost-effective survey design responsive to user needs. In the case of the 2002 CFS, these uncertainties were related to the lack of a clear understanding between BTS and the Census Bureau about the responsibility for ensuring survey funding. The difficulties encountered by the CFS partnership may be partially attributable to the different funding priorities of the two agencies. The Census Bureau is concerned with the budget for the entire Economic Census, whereas BTS is concerned only with the budget for the transportation component (the CFS). Such problems do not arise with the NHTS because this survey is not part of a larger effort with broader objectives outside the transportation arena.
While recognizing the difficulties inherent in survey partnerships in general and the CFS partnership in particular, the committee urges BTS to work with its survey partners to establish a clear understanding of respective roles and responsibilities. When necessary, BTS should take a
leadership role in its survey partnerships to ensure the feasibility of the flagship surveys. Timely efforts to build consensus and establish memorandums of understanding (MOUs) addressing key aspects of a survey are a prerequisite to the development and conduct of major national surveys, such as the NHTS and CFS, that involve two or more survey partners. Dialogue among these partners needs to involve not only agency leadership but also midlevel technical experts. MOUs should address areas such as survey objectives, cost sharing, timing, the sampling frame, the development and testing of survey questionnaires, data collection, the use of subcontractors, data processing, and the dissemination of survey results.
Recommendation 6: BTS should enhance and maintain the transportation expertise of its staff to achieve a balance between statistical and transportation knowledge.
BTS’s data-gathering, analysis, and dissemination activities require not only expertise in statistics and survey methodology but also an understanding of the socioeconomic context of passenger and freight movements; the supply, costs, and performance of transportation services; and all the modes providing these movements. A balance of statistical and transportation expertise is needed to guide the development of useful data products appropriate to a federal statistical agency. Making decisions about such product development is at the heart of effective management of BTS’s survey programs.
An earlier examination of BTS’s statistical programs found the agency’s staff to have experience in the analysis of transportation data but relatively limited statistical expertise (Citro and Norwood 1997, 123). While progress has been made in developing the agency’s statistical strengths, a considerable portion of the necessary transportation expertise appears to have been lost, as evidenced by some recent decisions about the scope and content of survey programs. The committee also observed that many of the BTS staff members participating in committee meetings were relatively new to the agency. Therefore, the committee encourages BTS to take measures aimed at recruiting and retaining staff who will provide the agency with the necessary balance of statistical and transportation expertise at both management and operational levels.
Recommendation 7: BTS should address technical problems associated with its major surveys by making those problems a focus of its applied research program.
The committee urges BTS, a relatively new statistical agency without an established research tradition, to draw on existing survey research and expertise before making major investments in its own research program. Much of the extensive technical literature on survey methodology is relevant to BTS’s activities, and syntheses of research in specific areas could provide useful knowledge to inform the agency’s surveys. In addition, BTS staff could benefit from interactions with other survey researchers through attendance at professional meetings and active participation in the work of professional societies.
BTS could learn from the experiences of other agencies through joining in the activities of the Federal Interagency Council on Statistical Policy, the body coordinating the work of federal statistical agencies, and the Federal Committee on Statistical Methodology (FCSM), an interagency committee dedicated to improving the quality of federal statistics.3 One of the major goals of FCSM is to provide a mechanism for statisticians in different federal agencies to meet and exchange ideas. Recent FCSM statistical policy working papers and seminars have addressed topics relevant to BTS’s flagship surveys, including measuring and reporting on sources of error in surveys and interagency coordination and cooperation.
The committee encourages BTS to focus its own research program on
Solving its particular survey problems, such as declining response rates in household travel surveys; and
Investigating how recent developments in data collection methods and advanced statistical techniques can benefit transportation surveys. Examples include the use of the Global Positioning System (GPS) in personal travel and freight surveys and the use of new disclosure limitation methods to mask the identity of individual respondents in survey microdata.
Further information on FCSM is available on the committee’s website, www.fcsm.gov.
A number of mechanisms are available for conducting applied research of the type envisaged. Studies could be conducted internally, or the services of experts not on the agency’s staff could be enlisted through consulting or other arrangements (Martin et al. 2001, 11). For example, research could be contracted out to universities and small businesses, or qualified persons from academia could be brought into the agency temporarily under the Intergovernmental Personnel Act or BTS-sponsored fellowships to address specific problems. Studies could be conducted as stand-alone projects or could be an integral part of the surveys themselves. For example, pilot projects to investigate alternative data collection strategies could be conducted as part of the survey data collection effort.
According to a recent NRC report (Martin et al. 2001, 33), “the history of the statistical agencies has shown repeatedly that methodological research can lead to large productivity gains in statistical activities at relatively low cost.” The committee identified five main topics within the broad area of survey methodology in which applied research could particularly benefit the quality and usefulness of future surveys.4
Response rates for household travel surveys. The committee urges BTS to investigate ways of increasing response rates in its household travel surveys, both through improved understanding of the causes of non-response and associated bias in telephone surveys and through increasing knowledge about the effectiveness of alternative data collection techniques. As discussed in the letter report on the NHTS (Appendix A), a number of federal policy-related surveys have experienced declining response rates. Thus, in seeking to improve response rates in its household travel surveys, BTS may benefit from the findings of investigations by other statistical agencies. For example, an investigation of response rates for the National Household Education Survey, a telephone survey of the noninstitutionalized civilian population of the United States, found that a number of factors, including survey objectives, approaches to screening households, and interactions between interviewers and respondents, must be considered in assessing the impact of survey design and procedures on response rates in random digit
dialing surveys (National Center for Education Statistics 1997). Much of the recent research on nonresponse has focused on the refusal of potential respondents to participate in a survey. This refusal research has examined issues such as the salience of the survey topic to the respondent, the organization conducting the survey (the Census Bureau or a commercial firm, for example), and the social psychology of responding to a request from a stranger.
Data collection. Investigations of a range of options for gathering data from survey respondents could lead to methods for strengthening BTS’s future personal travel and freight surveys. In particular, technological innovation and development may provide opportunities for new, more effective and efficient approaches. Web-based methods, such as electronic data collection forms with automated editing systems and Internet-based travel diaries, have shown promise and merit further investigation, probably as part of mixed-mode data collection initiatives that reach different respondents in different ways. Personal and in-vehicle GPS data loggers also offer opportunities to gather large amounts of detailed data on the movement of people and goods at low cost. However, issues of privacy and confidentiality associated with such passive data collection techniques, together with some technical problems, remain to be resolved.
Sample design. Investigation of sample design issues could help BTS make its surveys more effective. For example, longitudinal panel surveys can provide useful information about changes in personal travel behavior over time and could be a valuable complement to the more traditional cross-sectional surveys. However, more research is needed to understand the cost–quality–usefulness trade-offs of panel data compared with cross-sectional approaches. Other sample design issues worthy of investigation include sampling approaches for telephone surveys that take account of the increasing number of cell-phone-only households and options for using shipment-based rather than establishment-based sampling for the CFS.
Questionnaire development and testing. Extensive research on the cognitive aspects of surveys has been conducted by government agencies such as the National Center for Health Statistics, the Bureau of Labor Statistics, and the Census Bureau. There may be opportunities for BTS to build on this research in areas specific to transportation surveys as
part of an effort to develop methods for quick and accurate testing of survey instruments.
Data dissemination. A report on the confidentiality and accessibility of government statistics recommended that federal agencies “strive for a greater return on public investment in statistical programs through … expanded availability of federal datasets to external users” (Duncan et al. 1993, 224). Research into ways of making more spatially specific microdata available to data users, while continuing to protect the confidentiality of data providers, could result in opportunities to add value to surveys without increasing data collection costs.
OMNIBUS SURVEY PROGRAM
Recommendation 8: BTS should establish a process for conducting the Omnibus surveys that ensures the agency’s credibility as an independent provider of statistical information.
The committee urges BTS to place greater emphasis on establishing a clear separation between statistical information and political policy in the Omnibus program. The example of the 2001 Mariner Survey—an Omnibus targeted survey—illustrates that such a separation not only is possible but also can result in a quality customer satisfaction survey on transportation-related issues. The objectives of this survey were clearly defined: the Maritime Administration (MARAD) needed information about the numbers of merchant mariners who would be willing to take an afloat position in a national defense emergency and the period of time they would be willing to serve. In designing the survey, BTS’s expertise in survey methodology complemented MARAD’s knowledge of the merchant mariner community—knowledge that is clearly reflected in the survey questionnaire. The report on the survey’s principal findings, prepared by BTS’s Office of Survey Programs, presents a thorough and careful analysis of the data but avoids commenting on policy issues (BTS 2001). Thus, this report is in marked contrast to some editions of OmniStats, BTS’s two- or three-page popular reports on the Omnibus monthly household survey, which sometimes draw interpretations reaching beyond the objective data.
The committee’s letter report on the Omnibus Survey Program (Appendix B) recommends approaches that BTS can take to enhance and ensure the integrity of the surveys:
Establish an independent review mechanism with participation from experts outside BTS,
Implement a range of measures aimed at ensuring that all surveys are of a consistently high quality, and
Take steps to ensure that analyses of survey data are technically robust and that the resulting products comply with established guidelines for the reporting of statistical data.
In addition, the committee urges BTS to consider restricting its role in the Omnibus program to technical guidance and conduct of the surveys, leaving to survey sponsors—MARAD, the National Highway Traffic Safety Administration, and others—the responsibility for survey content and interpretation. In the committee’s view, this approach would be beneficial in helping BTS establish the clear distinction between statistical information and policy interpretation that is vital to the credibility of a federal statistical agency.
BTS Bureau of Transportation Statistics
TRB Transportation Research Board
BTS. 2001. 2001 Mariner Survey: Principal Findings. U.S. Department of Transportation, Washington, D.C.
Citro, C. F., and J. L. Norwood (eds.). 1997. The Bureau of Transportation Statistics: Priorities for the Future. Panel on Statistical Programs and Practices of the Bureau of Transportation Statistics, National Research Council, Washington, D.C.
Duncan, G. T., T. B. Jabine, and V. A. de Wolf (eds.). 1993. Private Lives and Public Policies: Confidentiality and Accessibility of Government Statistics. National Research Council, Washington, D.C.
Martin, M. E., M. L. Straf, and C. F. Citro (eds.). 2001. Principles and Practices for a Federal Statistical Agency, 2nd ed. Committee on National Statistics, National Research Council, Washington, D.C.
National Center for Education Statistics. 1997. An Overview of Response Rates in the National Household Education Survey: 1991, 1993, 1995, and 1996. Report 97-948. nces.ed.gov/pubs97/97948.html.
TRB. 2003. Special Report 276: A Concept for a National Freight Data Program. National Research Council, Washington, D.C.