The committee’s conclusions about the current survey programs of the Bureau of Transportation Statistics (BTS) are presented in this chapter. These conclusions are based on the outcomes of the committee’s reviews of individual survey programs—the National Household Travel Survey (NHTS), the Commodity Flow Survey (CFS), and the Omnibus Survey Program—and address the following seven major themes:
Essential value of the survey data (Conclusion 1),
Substantive expertise to guide the development of data products (Conclusion 2),
Survey stability and quality (Conclusion 3),
Communication with data users (Conclusion 4),
Clear and explicit survey objectives (Conclusion 5),
Effective and efficient methods for conducting the surveys (Conclusion 6), and
Clear separation between statistical information and policy interpretations in the Omnibus program (Conclusion 7).
After its reviews of individual survey programs, the committee observed that the NHTS and CFS differ in both substance and significance from surveys conducted under the Omnibus program. The committee characterized the NHTS and CFS as BTS’s flagship personal travel and freight surveys. These major, multiyear survey programs have budgets on the order of $10 million to $15 million, and survey data are widely used by transportation analysts and researchers in both the public and private sectors. In contrast, the Omnibus surveys, which include a customer satisfaction component, are relatively modest, quick-response efforts serving clients primarily within the U.S. Department of Transportation
(USDOT). While recognizing the value of the Omnibus program, the committee sees the flagship surveys as far more important to the overall BTS mission of supporting decision making by organizations within the broad transportation enterprise. For this reason, the following conclusions draw primarily on the outcomes of the committee’s reviews of the NHTS and CFS, and the recommendations in Chapter 4 focus on opportunities for BTS to improve its flagship surveys.
VALUE OF FLAGSHIP SURVEY DATA
Conclusion 1: BTS’s flagship personal travel and freight surveys provide essential data not available from other sources.
The NHTS and CFS serve a broad constituency of organizations interested in transportation. USDOT, other federal agencies, the U.S. Congress, state departments of transportation, and metropolitan planning organizations (MPOs) are important public-sector data users. For example, the surveys provide policy makers at USDOT with national-level data to inform policy and investment decisions pertaining to the departmental goals of safety, mobility, economic growth, human and natural environment, and national security. Private-sector groups, such as consulting organizations, think tanks, and industry associations, also make extensive use of NHTS and CFS data, as do those in academia.
The widespread use of data from the NHTS and CFS indicates that these surveys provide essential data not available elsewhere. In the absence of the NHTS, nationwide personal travel data available from the federal government would be limited to journey-to-work trips reported in the Decennial Census and the American Community Survey. There would be no source of nationwide data on increasingly important nonwork-related travel, making it difficult to assess trends in this market. While the CFS is one of many sources of freight transportation data, it is the only federal government data source that attempts to provide a comprehensive picture of freight flows across all modes of transportation. Trade databases provide some useful information, but they are intended for tracking economic transactions and provide only limited data on the physical movement of goods.
Information on the origin, quality, and limitations of the NHTS and CFS data is made available as part of the survey documentation. For ex-
ample, the sample size and response rate are reported for both surveys, as are limitations in shipment coverage for the CFS. Thus, users are able to assess the reliability of the data products. Not all sources of transportation data provide the same degree of transparency. For example, commercial data sets on freight movements may incorporate proprietary data, the origin and reliability of which are not reported. This lack of transparency raises concerns about the validity of the data as a foundation for decision making. A federal statistical agency such as BTS is required to “be open about its data and their strengths and limitations” (Martin et al. 2001, 8). Thus, users have come to rely on NHTS and CFS data because they bear the imprimatur of a federal statistical agency.
Conclusion 2: BTS lacks the balance of expertise needed to guide the development of data products for informing transportation decision making.
Some federal statistical agencies have clearly defined core programs providing the data needed to calculate high-profile national performance indicators. For example, the Bureau of Labor Statistics collects the data used in calculating the nation’s monthly employment statistics. In contrast, BTS is tasked with providing data to inform national transportation decision making “by all levels of government, transportation-related associations, private businesses, and consumers” [49 U.S.C. 111(c)(7)]. Thus, the data collected in the agency’s major surveys (the NHTS and CFS) are not defined by requirements for calculating specific metrics or indicators. Although it allows BTS more flexibility than some other federal statistical agencies in deciding what data to collect and how often to collect them, the broad nature of this mandate imposes an added burden on the agency. BTS has to assume responsibility for developing its own portfolio of programs. Development of this portfolio requires judgments about the kinds of data products that will be both responsive to user needs and relevant to policy and investment decisions affecting the transportation enterprise as a whole. Such judgments, in turn, require a broad understanding of the nation’s transportation system, as well as expertise in statistics and survey methods.
The committee’s reviews of the CFS and the Omnibus Survey Program revealed instances in which a better understanding of the transportation
context and its policy implications, and of the problems being addressed by data users in the transportation field, could have resulted in better survey design and implementation decisions. Consequently, the committee concluded that BTS lacks the quality and quantity of transportation expertise needed to inform important decisions about its surveys. The following example illustrates this point.
A greater understanding of transportation issues on the part of BTS could have been beneficial in informing decisions about the sample size for the 2002 CFS, particularly in the face of the reduced budget available for the survey.1 During the early stages of the survey design process, BTS appears to have made few efforts to understand the requirements of transportation analysts for freight flow data at specific levels of geographic detail and reliability—levels that determine the minimum sample size. Consequently, opportunities to investigate creative ways of achieving this minimum sample size with limited resources, or to seek additional funding for the survey, were severely restricted. The eventual reduction in sample size to 50,000 establishments—compared with 200,000 in 1993 and 100,000 in 1997—has adversely affected the usefulness of the data. Furthermore, transportation analysts are concerned that reductions in sample size below 50,000 establishments could seriously compromise the ability of the CFS to provide even general-purpose statistics on commodity flows. Nonetheless, during design of the 2002 survey, BTS asked Census Bureau staff to produce cost estimates for sample sizes of 30,000 and 10,000 establishments.
SURVEY STABILITY AND QUALITY
Conclusion 3: Budget variations and changes in ownership threaten to undermine the stability and quality of BTS’s flagship personal travel and freight surveys.
The history of BTS’s personal travel and freight surveys is characterized by variations in survey frequency and sample size. These variations, which are linked to variations in survey budgets, potentially reduce the
usefulness of survey data in informing transportation decision making. Reductions in frequency limit the availability of timely data and the ability to detect new trends, while reductions in sample size may adversely affect data usability.
The personal travel surveys have been conducted with varying frequency since the late 1960s. The Nationwide Personal Transportation Survey (NPTS) was conducted in 1969, 1977, 1983, 1990, and 1995, while the American Travel Survey (ATS) was conducted in 1977 and 1995. In 2001, the two surveys were combined as the NHTS. The budget for the 2001 NHTS is $10.7 million ($411 per household surveyed). The cost of the 1995 NPTS was $4.1 million ($195 per household) and that of the 1995 ATS was $18 million ($269 per household), giving a combined cost for the 1995 surveys of approximately $22 million. The freight surveys have generally been conducted on a regular 5-yearly basis, although a hiatus occurred in the 1980s and early 1990s before the initiation of the CFS. From 1963 through 1977, the Census Bureau collected data on commodity flows as part of its 5-yearly economic census program. The Census Bureau conducted a smaller commodity transportation survey in 1983 but did not release the results because of problems with data reliability. The CFS, which is linked to the mandatory 5-yearly Economic Census, was conducted in 1993, 1997, and 2002. The CFS budget has ranged from a high of $19 million for the 1997 survey to a low of $13 million for the 2002 survey.
In the case of the personal travel surveys, budget uncertainties appear to have been accommodated either by increasing the time between surveys or by reducing the sample size. For the NPTS, the period between surveys has been as long as 7 or 8 years; the ATS was conducted only twice, with an interval of 18 years between the two surveys. Clearly, such prolonged gaps limit the usefulness of the data for detecting trends and identifying shorter-term changes in travel behavior. The pace of economic, demographic, and technological changes may be such that a snapshot of personal travel behavior taken every 7 or 8 years does not provide an adequate basis for informing transportation policy decisions and related public- and private-sector investments. The reduction in budget for the 2001 NHTS, compared with that for the 1995 predecessor surveys, was accommodated in part by a reduction in sample size for
the long-distance travel component of the combined survey. The 1995 ATS surveyed 67,000 households, whereas the 2001 NHTS surveyed only 26,000 households. Consequently, in contrast to the 1995 ATS, the 2001 NHTS does not provide information on state-to-state travel patterns. While the full implications of this loss of geographic detail are not yet clear, the committee notes that the continuing interest in improving intermediate- and long-distance travel services, such as high-speed rail, defines a public policy need for quality data on longer trips.
The CFS sample size has been reduced by 75 percent over three survey cycles. Uncertainty about the availability and level of funding for the 2002 survey delayed key design decisions until late in the survey planning process. As a result, options for more cost-effective data collection could not be adequately explored because of time constraints—a deficiency that contributed in part to the greatly reduced sample size.
In addition to budget uncertainties, institutional changes have complicated efforts to ensure the stability and quality of the flagship personal travel and freight surveys. Before the establishment of BTS in the 1991 Intermodal Surface Transportation Efficiency Act (ISTEA), national commodity transportation surveys were conducted by the Census Bureau. The successor survey (the CFS) is undertaken through a partnership between BTS and the Census Bureau. Similarly, the NPTS and ATS were the responsibility of other agencies before being assigned to BTS. The NPTS was conducted by the Federal Highway Administration (FHWA), and the ATS was conducted by the Census Bureau, first in 1977 as a component of the Census of Transportation and then in 1995 under a contract from BTS.
Both the NHTS and CFS are now funded and conducted by BTS in conjunction with survey partners. The NHTS is funded approximately equally by BTS and FHWA, with a small contribution (less than 3 percent) from the National Highway Traffic Safety Administration. The survey is conducted by BTS, FHWA, and their contractors. The CFS is funded by BTS (80 percent) and the Census Bureau (20 percent), which together are responsible for conducting the survey. The NHTS partnership appears to be functioning effectively. BTS and FHWA have established a good working relationship that allows them to build on experience with earlier surveys. However, the committee’s review of the CFS revealed the
lack of a clear understanding between BTS and the Census Bureau about ownership of the survey; responsibility for ensuring sufficient funding to produce a useful, quality product; and their respective roles in developing survey methods.2
COMMUNICATION WITH DATA USERS
Conclusion 4: BTS and its survey partners have not adopted a sufficiently thorough and systematic approach for communicating with data users.
The quality of BTS products—notably the data from the agency’s personal travel and freight surveys—must be assessed in relation to clearly defined objectives. In particular, good survey design cannot be defined without reference to the data needs of users. Thus, Richardson et al. (1996, 80) note that “good survey design demands making trade-offs between the competing requirements of good design practice … to arrive at the most cost-effective, high-quality survey meeting the needs of the client within budget constraints” (emphasis added).
A major purpose of communication between BTS and its clients is to assist the agency in ensuring that its products are truly useful. The committee concluded that information from a range of sources—such as mailing lists, publications citing surveys, records of website usage, and the BTS products customer database—provides the agency with some indications of how and by whom its data are used. Meetings convened by BTS to facilitate interactions with survey users have also yielded benefits. For example, the 1999 conference Personal Travel: The Long and Short of It (TRB 2001) stimulated valuable discussion of methodological
and content issues relating to the merging of the NPTS and ATS. Nevertheless, the committee observed that, to date, such outreach activities have been sporadic and have not provided a regular forum for interaction between BTS staff and data users. For example, in November 2000, BTS convened a meeting with public- and private-sector CFS users and Census Bureau representatives to initiate a dialogue on freight data needs for planning and policy purposes. Although this meeting resulted in valuable discussion, the committee is not aware of any efforts to continue the dialogue on a regular basis. The lack of an underlying strategy for eliciting feedback from the broad spectrum of data users also raises concerns that BTS’s outreach activities may not always be effective in reaching interested groups.
The Advisory Council on Transportation Statistics (ACTS), appointed by and reporting to the BTS Director, is charged with advising the agency on transportation statistics and analyses, including whether the statistics and analyses disseminated by BTS are of high quality and based on the best available objective information.3 The ACTS members, five or six high-level managers and researchers, meet two or three times a year. Although ACTS provides some regular guidance on survey product requirements, meeting agendas must cover many topics and may include only limited opportunities for detailed discussion of user needs. Despite its considerable value, ACTS is not in a position to provide the detailed technical input that could result from regular dialogue between BTS staff and “hands-on” data users, such as midlevel professionals in research and consulting organizations, state departments of transportation, and MPOs. This input could assist BTS in developing a better understanding of its customers’ requirements to inform the development of more effective surveys.
Given the opportunity, users can play a dual role in helping BTS make its data products more useful. In addition to sharing information on their data needs with BTS, many users can also advise the agency on data concepts, methods, and products (Martin et al. 2001, 9). However, soliciting such advice requires BTS to share information with users about its survey development and design activities, and related decisions, so
that they can provide feedback. The aforementioned conference on the merging of the NPTS and ATS (TRB 2001) provided a feedback mechanism that enabled BTS to benefit from users’ advice about the proposed new personal travel survey (the NHTS). In contrast, feedback from users on the 2002 CFS was very limited. The Census Bureau conducted fairly extensive investigations of alternative sampling schemes, but the results do not appear to have been shared with data users early in the survey planning process.4 Consequently, the decision to halve the sample size vis-à-vis the 1997 survey was taken by the CFS partnership with apparently little discussion with users of the advantages and disadvantages of the sampling options.
CLEAR SURVEY OBJECTIVES
Conclusion 5: A lack of clearly defined survey objectives complicates BTS’s efforts to develop cost-effective, quality personal travel and freight surveys responsive to the needs of data users.
The committee’s reviews of the NHTS and CFS revealed a lack of clearly defined objectives for these major national surveys. In the absence of such objectives, the robust foundation needed to inform quality/quantity/cost trade-offs inherent in the survey design process is lacking, and the survey scope itself may be ambiguous. As a result, available resources may not be used effectively in meeting the needs of data users. The following examples illustrate this point.
The committee observed that sample sizes for the NHTS and CFS do not appear to have been determined on a rational statistical basis that reflects user needs for statistically reliable data at specified levels of geographic detail, as well as mode and commodity detail as appropriate. Rather, decisions about sample size appear to have been dictated primarily by survey budgets. The example of the 2002 CFS has already been discussed. In the case of the personal travel surveys, it is not clear that reducing the sample size for long-distance travel from 67,000 in the 1995
ATS to 26,000 in the 2001 NHTS will provide sufficiently reliable and comprehensive data on longer trips to inform policy and investment decisions.
In the case of the CFS, a series of sample size thresholds can be identified. These thresholds are determined by the level of geographic detail at which commodity flows can be characterized. For example, the minimum sample size needed to provide useful data on state-to-state flows is much larger than that needed to characterize general national flows. Thus, selecting a sample size just below that needed to determine state-to-state flows would result in a particularly inefficient use of resources. The amount of data collected would be far in excess of that necessary to determine general national flows but insufficient to provide usable data at the level of state-to-state flows. Thus, a decision about survey objectives—namely, whether the survey is to provide data on state-to-state flows or general national flows—is key to developing a cost-effective design.
Conclusion 6: Improvements in the effectiveness of BTS’s survey methods could enhance the quality and usefulness of the resulting data products.
The committee identified five main topic areas in which more effective survey methods could improve the quality and usefulness of BTS’s survey products:
Response rates for household travel surveys,
Questionnaire development and testing, and
BTS is not alone in needing to develop more effective survey methods. Many of the methodological issues the agency faces—including the decline in the effectiveness of telephone surveys, the search for more cost-effective ways of collecting quality data, and the need to expand the availability of survey data without compromising the confidentiality of data providers—are common to the wider survey community.
Response Rates for Household Travel Surveys
Survey methodologists generally agree that the changing characteristics of telephone usage, including defensive measures by consumers to deflect telemarketing calls and the growing number of cell-phone-only households, are reducing the effectiveness of many current telephone surveys and may be increasing bias in the results. Both the 2001 NHTS and the Omnibus monthly household survey, which rely exclusively on telephone interviews to collect data, are indicative of this trend. The low response rates for these surveys (41 percent for the NHTS and 43 percent for the Omnibus survey) give cause for concern because of the likelihood of significant nonresponse bias. There is reason to believe that the travel behavior of survey non-respondents may differ significantly from that of respondents, leading users to question whether data provided by less than half the households surveyed form a valid basis for analysis and decision making. The Office of Management and Budget gave BTS only conditional clearance to proceed with the 2001 NHTS on the understanding that the agency will investigate the high nonresponse rate and find ways to reduce it in the future.
The experience with the NHTS illustrates clearly how changes in respondent behavior (willingness to participate in telephone surveys) can undermine the effectiveness of a chosen data collection method, resulting in low response rates that threaten the survey’s validity. Furthermore, at a time when survey budgets are coming under increasing pressure, BTS is being asked to provide quality data in a timely and cost-effective manner. Thus, new data collection methods could bring important benefits by offering the potential to reduce respondent burden and increase respondents’ willingness to participate, thereby increasing response rates. For example, electronic reporting options developed for the 2002 Economic Census allow businesses to extract data directly from their own spreadsheets and import it into survey software. Businesses can also complete electronic survey questionnaires on their own computers at their own pace.5 Similarly,
Easier, Faster, Smarter: Census Bureau Features Electronic Reporting in the 2002 Economic Census, www.census.gov/epcd/ec02/ec02electronic.htm.
Internet-based travel diaries for use in personal travel surveys allow respondents to complete the diary in a series of work sessions at convenient times. Such diaries also permit the implementation of user-friendly features such as context-sensitive instructions, a help feature, automatic addition of intrahousehold shared trips, and respondent-interactive geocoding to help describe trip origins and destinations.
Improving sampling techniques could make BTS’s surveys more effective, as illustrated by two examples:
Because pricing structures may require users to pay for incoming calls, cell phones are excluded from random digit dialing lists used in telephone surveys such as the NHTS. However, these exclusions may lead to sampling bias because of the increasing number of cell-phone-only households. Acceptable and effective ways to include cell-phone users in the sample are needed.
The CFS requires respondents to report details of a sample of their shipments. Some reporting errors have been linked to respondent confusion in applying the current method by which firms are instructed to select a sample of shipments (Black 1997). An alternative sampling scheme, such as an approach that involves randomly selecting a starting point in terms of shipments and taking the next n records, could result in fewer reporting errors and improved data quality.
Questionnaire Development and Testing
Pilot surveys form an important component of the transportation survey process because they provide an opportunity to correct the inevitable errors in the original design (Richardson et al. 1996). However, because the time available for survey development and testing is always restricted, pilot surveys are sometimes limited or neglected entirely. In the case of the Omnibus monthly household survey, the committee is concerned that schedule constraints imposed by the quick-response nature of the
survey may prevent adequate cognitive testing of questionnaires. Thus, the resulting survey data may be difficult to interpret or inconsistent with survey objectives. Methods for testing survey instruments quickly and accurately could be very useful in improving the quality of both the NHTS and the Omnibus surveys.
A number of data users indicated to the committee that the release of additional survey microdata would greatly enhance the usefulness of the NHTS and CFS by providing greater geographic detail. However, in attempting to expand the availability of its data, BTS faces a dilemma. The requirement to suppress geographic location information that could be linked to individual data providers is at odds with the need to retain sufficient geographic detail for data to be more useful. Techniques such as statistical disclosure limitation methods (see, for example, Duncan et al. 1993) may offer the possibility of expanding the availability of the NHTS and CFS data sets to external users without compromising the confidentiality of data providers. Thus, such methods could add value to the surveys without the need for additional data collection.
STATISTICAL INFORMATION AND POLICY INTERPRETATIONS
Conclusion 7: A clearer separation between statistical information and policy interpretations in the Omnibus program would strengthen BTS’s credibility as an independent provider of transportation data.
A strong position of independence is essential if BTS, as a federal statistical agency, is to be viewed as a source of objective, reliable information. Indeed, an important reason to establish a separate statistical agency is to meet the need for data series to be independent of control by policy makers or regulatory or enforcement agencies (Martin et al. 2001, 3). Any hint that data collection, analysis, and reporting procedures are being influenced to produce a particular outcome or support
a policy initiative can undermine both the trust of data users and the cooperation of data providers.6
The committee recognizes the value of the Omnibus program of customer satisfaction surveys in providing timely information to operating agencies and policy makers in USDOT. In so doing, the surveys also help BTS demonstrate its relevance and utility to major constituencies within the department. Clearly, operating agencies need to measure the impact of their programs and benchmark their progress, and decision makers need to measure the outcomes of their actions. Nevertheless, there is also a natural desire to paint such outcomes in a favorable light, thereby demonstrating the wisdom of decisions made and the success of programs administered and policies adopted. BTS’s role in the Omnibus program involves the agency in measuring public opinion on topical items for internal USDOT customers whose missions differ from the mission of BTS. Consequently, the committee had serious concerns about the potential of the Omnibus surveys to compromise, or appear to compromise, BTS’s independence.7
Continuing to provide the kinds of data obtained in the Omnibus program—while maintaining the essential objectivity and independence of a federal statistical agency—will require much wisdom on the part of BTS. For example, it will require sensitivity to the policy implications of questions included in the survey. Such sensitivity is not possible without a broad understanding of the transportation system and its financing, organization, and operating relationships. It will also require that BTS avoid interpretations directly linked to matters of policy, focusing instead on the technical elements of instrument design, sampling, and data analysis.
As discussed in the 1997 NRC review of BTS, the independence of the head of a federal statistical agency is important in ensuring the independence of the agency itself. Therefore, the review committee recommended continuing the provisions established by ISTEA that “the director of BTS be a presidential appointee with a fixed term of 4 years, who reports directly to the secretary of transportation and is a qualified professional with relevant training and experience” (Citro and Norwood 1997, 98).
In its letter report on the Omnibus surveys (Appendix B), the committee noted that federal agencies have generally tended to avoid quick response customer satisfaction surveys conducted on a continuing basis, such as the Omnibus monthly household survey.
The committee’s recommendations to BTS for addressing the issues discussed above, thereby improving the quality and usefulness of its surveys, are presented in the following chapter.
TRB Transportation Research Board
Black, J. 1997. Can Respondents Construct a Frame and Draw a Sample? Experiences from the 1993 Commodity Flow Survey. Proceedings of the Survey Methods Section, American Statistical Association, Alexandria, Va., pp. 216–221.
Citro, C. F., and J. L. Norwood (eds.). 1997. The Bureau of Transportation Statistics: Priorities for the Future. Panel on Statistical Programs and Practices of the Bureau of Transportation Statistics, National Research Council, Washington, D.C.
Duncan, G. T., T. B. Jabine, and V. A. de Wolf (eds.). 1993. Private Lives and Public Policies: Confidentiality and Accessibility of Government Statistics. National Research Council, Washington, D.C.
Martin, M. E., M. L. Straf, and C. F. Citro (eds.). 2001. Principles and Practices for a Federal Statistical Agency, 2nd ed. Committee on National Statistics, National Research Council, Washington, D.C.
Richardson, A. J., E. S. Ampt, and A. H. Meyburg. 1996. Nonresponse Issues in Household Travel Surveys. In Conference Proceedings 10: Conference on Household Travel Surveys: New Concepts and Research Needs, TRB, National Research Council, Washington, D.C., pp. 79–114.
TRB. 2001. Transportation Research Circular E-C026: Personal Travel: The Long and Short of It. National Research Council, Washington, D.C. gulliver.trb.org/publications/circulars/ec026/ec026.pdf.