Below is the uncorrected machine-read text of this chapter, intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text of each book. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.
4 Ensuring Relevance A statistical agency must not only document, evaluate, and improve the qual- ity of the data within its subject area, but it must also ensure that there are relevant data on topics of importance to policy makers, planners, and researchers in the field. The previous chapter addressed the immediate necessity for the Bureau of Transportation Statistics (BTS) to focus on data quality and to build a strong statistical staff to carry out its responsibilities for quality improvement. This chapter addresses areas that BTS should undertake to improve the relevance of transportation data to meet important user needs. The 1991 Intermodal Surface Transportation Efficiency Act (ISTEA) assigns several functions to BTS that have the goal of ensuring the relevance of transpor- tation data for policy making and other purposes. They include: developing appropriate indicators for the transportation system; coordinating the collection of transportation data by USDOT with other federal agencies and collecting data to fill gaps; and identifying unmet information needs and ways to meet those needs. These functions are commonly undertaken by statistical agencies, but to date BTS has done relatively little on them. Work needs to begin. Central to a statistical agency' s ability to improve data relevance, and more generally to determine priorities for its work, is that it have a broad vision of a comprehensive data system that can serve the information needs of users over the medium and long term. In this chapter, after first defining what we mean by "relevance," we discuss the development by BTS of a vision of a comprehensive transportation data system and how that vision and other considerations should factor into its development of a long-range plan for implementing all aspects of its mandate. We then discuss priority areas for BTS to undertake to improve the relevance 64
ENSURING RELEVANCE 65 of transportation data for policy making and other uses. They include: the devel- opment of key national transportation indicators; an increased role in coordinat- ing the collection of transportation data, in particular, the compilation of an annual statistical budget as a data coordination mechanism for USDOT; the es- tablishment of regular mechanisms for identifying user information needs, in par- ticular, effective two-way communication channels with states and metropolitan planning organizations, building on the work that BTS has under way in this area; and the assessment and further development of BTS's analysis programs and publications. Primary recommendations appear at the end of the chapter. DIMENSIONS OF RELEVANCE "Relevance" concerns substantive aspects of data systems that affect their usefulness. Dimensions of relevance include the following: · The appropriateness of concepts, which means that the concepts a data system is intended to measure are those that can help policy makers and analysts understand trends and behaviors of concern to them and the implications of pro- gram and policy changes. Conceptual appropriateness must be reviewed in light of changing conditions. For example, with concern about the effects of economic growth on the environment and nonrenewable resources, there is growing interest in concepts of national income and gross national product that account for natural resource depletion, pollution, and other environmental costs. More narrowly, beginning in December 1991, the Bureau of Economic Analysis has featured gross domestic product (GDP), and not gross national product (GNP), as the more appropriate concept by which to measure U.S. output for comparative analysis with other countries in today's global economy.) · The match between concepts, operational definitions, and measurements, which implies that the theoretical concepts of importance to data users are operationalized by appropriately defined empirical variables that, in turn, are ac- curately and reliably measured (see Bonnen, 1977:395-396~. Many concepts are difficult to operationalize. For example, the economic cost of transportation fa- talities and injuries is an important concept for which to have data, but it may be difficult to operationalize such a concept with an appropriate proxy variable or combination of variables that can be measured empirically. For example, should the costs include the immediate costs of medical treatment, emergency system use, vehicle repair, etc.? The long-term costs over the lifetime of accident vic- tims of health care, public assistance, lost productivity, foregone tax revenues, etc.? Estimates of the value of lost quality of life? The choice of operational definition affects the relevance of the concept for policy and research use and also iGDP includes the output produced by labor and property located in the United States, including the output of U.S.-located establishments of foreign-owned enterprises; GNP includes the output attribut- able to labor and property supplied by U.S. residents (see Bureau of the Census, 1996b:439-440).
66 BUREAU OF TRANSPORTATION STATISTICS: PRIORITIES FOR THE FUTURE the ability to obtain accurate measurements. In the example, the broader the operationalization of the concept, the more difficult the measurement process. · More generally, the appropriate level of detail in a data system in terms of the specificity and range of subject matter and geographic detail that it pro- vides to inform current and emerging policy and research interests. Appropriate- ness of detail must be reviewed continually in light of changing conditions and policy concerns. As examples, more data are needed for the fast-growing ser- vices sector of the economy to support public- and private-sector policy making and planning than was true in the past, and more data are needed on intermodal transport to address increasingly important transportation policy concerns. · Timeliness of statistical information, which denotes the length of time between the occurrence of some event or the act of measuring some attribute of interest and the availability of statistics to the user. For example, the Bureau of Labor Statistics provides a monthly update of the labor market status of the economy. Wise decisions on the periodicity of data collection are a function of the rate of change and causes of change in estimates as well as the nature of decisions taken on the basis of the estimates. The implications of a statistical agency's focusing on data relevance in terms of appropriateness of concepts and their measurement, level of subject matter and geographic detail, and timeliness are that it must identify needs for data among current users, gaps in available data systems, and possibilities for the agency to inform the policy debates of the future. This process, combined with a compre- hensive understanding of the field, permits the agency to define sets of indicators that offer great relevance to current and future users and to provide other kinds of useful data. In its efforts to ensure relevance, it is important that a statistical agency seek ways to contain the costs and burden of data collection, processing, and analysis by keeping abreast of new methods and technologies that have the potential for cost savings and by looking for ways to cut back on less important data (e.g. through reductions in sample size or frequency). There will always be more data demands than can be satisfied, particularly in an era of increasingly constrained budgets, and it is critical for a statistical agency to evaluate data needs to deter- mine priority areas for new and improved data and also areas for which reduc- tions are possible. A VISION OF A COMPREHENSIVE TRANSPORTATION DATA SYSTEM Key to BTS's ability to ensure the relevance of transportation data and to make wise choices among competing activities to improve both data relevance and quality is that it have a broad vision of transportation data (see recommenda- tion 5 at the end of the chapter). The vision should encompass the information
ENSURING RELEVANCE 67 needs of transportation policy makers, planners, and researchers in the medium and long term and the characteristics of a comprehensive data system that could best serve those needs. BTS will not necessarily or even likely itself develop all of the data that are required to implement the vision. Much of the specific work to be undertaken will be done outside BTS: by other USDOT modal administra- tions, by other parts of the federal statistical system, and by states, localities, and private organizations. However, if BTS is to fill the leadership role set out for it by the 1991 ISTEA, then it must have an overarching vision of the data require- ments in key constituencies in the transportation field. How can BTS, as a priority effort, go about constructing such a vision and refreshing it periodically? One way is to ask relevant constituencies such ques- tions as the following: What are seen as important national policy concerns in transportation, how are they changing, and what are the implications for data? The 1991 ISTEA called for a reorientation of transportation planning to address intermodal and multimodal issues and concerns. The reauthorization of ISTEA is likely to con- tinue a cross-modal planning focus and may single out other important policy issues for the transportation community as well. The continuing public concern with such issues as the safety of the transportation system, the quality of the environment, and the costs and availability of energy sources will also have im- portant implications for transportation planning and investment. In developing a vision of information needs and a data system to address them, BTS must assess the data requirements of continuing and emerging national transportation policy concerns as seen by the Congress, the administration, and others, including states, metropolitan planning organizations, industry, and the general public. What changes are occurring in the economy and society that suggest the needfor new data or the reassignment of priorities among areas? The nature and pace of future social and economic change are hard to foresee with any great precision; however, broad trends are identifiable that are likely to have implica- tions for a comprehensive transportation data system that can serve user needs. Such trends include the aging of the population; the continued suburbanization of people and industry; growing pressures on the environment; the computerization of homes, schools, and businesses; and the globalization of the economy and continued growth in international trade. BTS could elaborate scenarios in these and other areas and consider the possible data implications for transportation. While not making too much of the results of such scenario-building, BTS could identify areas in which modest additional data collection, or somewhat different data collection, could help the transportation community anticipate and respond to important societal trends. As an example, rapid growth over the next few decades in the proportion of employed people who telecommute to work via computer, telephone, and fax at home can be expected to change the kinds of infrastructure investments that are
68 BUREAU OF TRANSPORTATION STATISTICS: PRIORITIES FOR THE FUTURE required for an efficient transportation system, compared with a continuation of current work practices. If the projected differences in the kinds of needed invest- ment are significant, then it could be important to the transportation community to have data with which to assess more accurately the likely rates of change in home-based employment linked with data on residence patterns. Such data in- puts could include not only survey questions about current workplaces, but also survey questions about the likelihood that respondents will work part or all of their hours at home in the next year, or next 5 years, and what factors might cause them to make such a change. What topics and information needs are still relevantfrom the past? In addi- tion to new and modified data to respond to emerging concerns, there is always a need for continuing time series to support trend analysis and provide benchmarks against which to measure change. The question is which series are critically important to continue and which could be reduced in scope, frequency, or sample size or redesigned in other ways (e.g., by converting an administrative records system to sample-based reporting or using new collection technology) in order to free up resources for other areas or to reduce the overall costs and burden of transportation data collection. In the transportation area, data on safety are clearly an important continuing need, particularly for USDOT, given the extensive involvement of the federal government in safety issues and safety regulation in all modes of transportation. There are likely to be other such areas as well. However, there are also likely to be areas in which the data that are currently collected are of less value to continue in the future (e.g., because of lesser policy concern) or that could be collected just as effectively by other organizations or other means. For example, detailed infor- mation on financial and operating characteristics of some kinds of common carri- ers might be one such area. Decisions to reduce or eliminate long-established data series are always difficult to make. However, a statistical agency that is striving to improve data relevance must have a vision of a comprehensive data system that is dynamic and allows for the retirement of obsolescent data series along with the emergence of new and modified series. A BTS IMPLEMENTATION PLAN A critically important task for BTS to undertake in the near future is the development of a long-term strategy for implementing its mission to improve both the relevance and quality of transportation data. BTS's mandate encom- passes a large, almost daunting, array of functions and responsibilities. A struc- tured implementation plan that specifies short-term, intermediate, and long-term goals in each of BTS's main programmatic areas is a necessity in order for BTS to work toward its vision of a comprehensive transportation data system and evolve as a statistical agency for USDOT. Without such a plan, BTS's energies are likely to be dissipated in striving to do more than it reasonably can. Also,
ENSURING RELEVANCE 69 without such a plan, the inevitable pressures from continuing areas of responsi- bility (e.g., for data dissemination) may result in continued deferment of needed initiatives in other areas (e.g., development of quality standards). BTS must have a roadmap with a well-blocked-out route to guide its activities, help it develop the necessary staff capabilities, and build a reputation as an effective agency that, over time, is fulfilling its mandate from the ISTEA. The implementation plan should identify overall priorities among BTS's major functions for the short and longer term and, within each functional area, identify specific activities, goals, and timetables.2 BTS's vision of a comprehen- sive transportation data system should provide the context for the development of the implementation plan. For example, such a vision should help determine a priority sequence for the development of national transportation indicators. An- other source of input to the plan is this report, which identifies broad areas of high priority, including work to develop department-wide quality standards, increased emphasis on documenting and evaluating data quality, and work to develop na- tional indicators, at the same time recommending decreased emphasis on the quantity of data disseminated. Still other sources of input, for both general and specific priorities and goals, include the constituencies or customers for transpor- tation data national policy makers, state and local agencies, private-sector orga- nizations, and academic researchers. The development of a vision of a comprehensive transportation data system and the development of a long-term strategy for implementing BTS's mandate are difficult, time-consuming tasks that represent added responsibilities for BTS staff. As would be true for any attempt at a serious long-range planning process, it is likely that BTS's initial efforts will produce areas for which it is not clear how to proceed or for which there is less complete articulation of ideas and goals than for other areas. Also, there must be flexibility to revise and further develop the vision and plan as circumstances change and new knowledge and experience are gained. Nonetheless, it is critical that BTS make its best attempt to envision the future requirements for transportation data and to plan its own future so that, for the long term, it has an overall sense of direction and, for the short and me- dium term, it has a set of goals that are feasible, contribute to the long-term agenda, and make it possible for the agency to demonstrate a solid record of accomplishment over time. ENSURING RELEVANCE: TRANSPORTATION INDICATORS The 1991 ISTEA mandates BTS to establish and implement, in cooperation with the modal administrations, the states, and other federal officials, a compre 2BTS recently outlined its goals in specific areas for fiscal 1997 and 1998 (provided in a back- ground document for the Advisory Council on Transportation Statistics). These goals should be reexamined as part of a longer-range planning process.
70 BUREAU OF TRANSPORTATION STATISTICS: PRIORITIES FOR THE FUTURE hensive, long-term program for collection and analysis of data relating to the performance of the national transportation system. The Transportation Research Board report, Datafor Decisions (National Research Council, 1992a), urged, as a high priority, that a new transportation data center, which is now BTS, develop a national transportation performance monitoring system. It developed a list of important attributes of the transportation system, for which it suggested one or more types of indicators (this list is reproduced in Table 4-1~. We agree that a high priority for BTS is to develop a consistent, easily under- stood, and useful set of indicators of key aspects of the transportation system (see recommendation 6 at the end of the chapter). Most statistical agencies produce indicators (usually regular time series) in their areas: examples are monthly and quarterly gross domestic product (GDP) estimates produced by the Bureau of Economic Analysis; monthly Consumer Price Index (CPI) and unemployment rate estimates produced by the Bureau of Labor Statistics; monthly retail sales, monthly housing starts, and annual poverty statistics produced by the Census Bureau; annual high school and college completion and dropout rates and peri- odic assessments of levels of student achievement produced by the National Cen- ter for Education Statistics; and annual vital statistics, which include estimates of births and deaths and mortality rates by cause, produced by the National Center for Health Statistics. Each of these indicator series is important in informing the public and contributing to the policy debate in its area; some of them have signifi- cant effects on the economy and public- and private-sector decision making. In other words, appropriately developed indicators provide highly relevant data for policy making and general public awareness. It is a heavy responsibility to produce such important statistics. There are often difficult conceptual, definitional, and measurement issues involved in de- veloping a single reliable and credible indicator to represent a complicated socio- economic phenomenon or construct (e.g., GDP, unemployment, poverty); even two or three indicators may not be adequate. Moreover, the policy use of key indicators can bring unwelcome publicity to a statistical agency, which may be hard pressed to explain the proper interpretation of its statistics and to defend the concepts and methods against politically motivated criticism and misuse. Furthermore, the development of indicators should not be the only focus of a statistical agency in terms of providing relevant data. Thus, indicators cannot serve such important needs as that of researchers for rich, multivariate data sets (e.g., longitudinal surveys) with which to analyze complex trends and behaviors. Nonetheless, key statistical indicators are important for both the public and policy makers, and statistical agencies can gain stature and support from the responsibil- ity to produce them. Also, such responsibility can help an agency set priorities for improvement of key concepts, definitions, and data sources that are needed to support the development of indicators and to support more in-depth analysis as well.
ENSURING RELEVANCE 7 Indicators, Not Performance Measures In discussing BTS's role in developing transportation indicators, we explic- itly use the term "statistical indicator" instead of "performance measure" or a similar term. The latter has a judgmental or regulatory connotation that is inap- propriate for a statistical agency. Indeed, there has been considerable concern among the states about the re- cent interest on the part of the federal government in assessing the performance of the transportation system. The states are wary of jurisdiction-specific perfor- mance measures that might be used for such purposes as allocating federal trans- portation funds, particularly in light of the difficulties of developing valid mea- sures that appropriately take account of measurement problems and varying conditions at state and local levels. (As examples, measures of road conditions should adjust for such factors as types and extent of usage, and measures of high- way traffic congestion should adjust for such factors as population density and the availability of public transit.) Such concerns are not limited to transportation. For many years, the states largely opposed the development of comparable crossjurisdiction indicators of children's educational progress. A major survey, the National Assessment of Educational Progress (NAEP), was originally designed so that only national-level and not also state-level estimates could be produced from the data. However, increasing public concern about educational issues has led to a willingness on the part of the states to compare their performance, and the NAEP was recently rede- signed to provide state-level estimates. What all this means is that a statistical agency must approach the develop- ment of indicators with care. To the extent that meaningful national-level indica- tors can be developed, they should be an important focus of the agency. BTS has already established as a priority goal for fiscal 1998 to begin work on transportation indicators. BTS proposes a cooperative activity with the other USDOT modal administrations, such as the Federal Aviation Administration and the Federal Highway Administration, in which they would first identify appropri- ate topics and concepts for indicators and BTS would then provide technical ad- vice on implementation. A motivation for the development of indicators is the 1993 Government Performance and Results Act, which requires federal agencies to establish performance measures of their output. We urge BTS to move forward with its plans to help the other modal admin- istrations identify performance indicators for their own programs. In general, as BTS builds its statistical staff and capabilities, it should be able increasingly to be helpful to the other modal administrations not only in the development of indica- tors and other kinds of statistical data, but also in advising on ways to improve the cost-effectiveness and usefulness of the large amounts of data that many of them collect for purposes of program administration and regulation. At the same time, we urge BTS to develop a small set of key national statis
72 .o .tq Cq a' be VO Cq so o C) a' Cq VO .~ En ¢ En ca o · ~ o .n ca Cat · ~ o ca ~o o ~s. .= ~^ ~o Y ° o E R R ER 9 as a <, A t>.4 ~ R ~ ~ ~ R .~ ~ ~ ~ . y ~' ~ ~ ~ ' ; ~' =4 4- ~ ;^ ~ ~ ~ · O ~ I ~ ~ ~ O ~u ~ ~ i ~ ~ 3 ~ s ~ ~ ~= ~¢~m am zmE-o ~ E-Z~ ca ca C) C) . ~ ·= R ~ ~ 9 ~ ~ · > _ j R ~ ~R _ V, ~
73 ca ~i ~ R At, ~ R ~ r ~ ~ ~ a E ~ ~ ~ ~ ~ , a E E ' ~ 'a & ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ . ~ ~ t, ~0 ~OR o ~ R = . ~ ~ ~ o ~ o ~ ~ R 5 ~ ~ ~ R ~ ~ ~ R ~ 3 o ~ To ~ ~ ~ ~ ~ ~ e~ ~ ~ o ~ ~ ~ ° ~ ~ ° O ~En Z ~ Z v, ~ ~ Z ~ ~ ~ ~ V ¢ ~ ~ V ~ ~ ~ ~ ¢ Z ;^ ~ .n ~ C) ~ ~;^ ca 4= R O ~ D C ° E a ~ a a ~@ En . . =b ca o .~ 7 .. V O V,
74 BUREAU OF TRANSPORTATION STATISTICS: PRIORITIES FOR THE FUTURE tical indicators of the transportation system that are relevant to policy and public concerns that it publishes on its own behalf as a statistical agency. (Some indica- tors e.g., trends in airline safety may serve the purposes of both BTS and another modal administration and could be developed and published jointly with the appropriate administration.) BTS's vision of a comprehensive transportation data system should inform its choice of priority areas for indicators, along with input from the other agencies in USDOT and transportation constituencies out- side USDOT. Because of BTS's responsibility to improve transportation data for cross-modal, system-wide analyses, the statistical indicators it decides to develop and publish should feature cross-modal concepts and concerns.3 What Indicators and in What Form? The challenge for BTS is to identify important aspects of the national trans- portation system for which it is possible to develop meaningful and reliable indi- cators. Transportation is largely local, yet it has national effects. For example, the functioning of the highway, rail, and air systems at a major transportation hub like Chicago affects not only the local economy and well-being, but also the national economy and international trade. The difficulty is to develop indicators that have national meaning when appropriate data may be hard to obtain and to interpret. The Transportation Research Board report, Data for Decisions (National Research Council, 1992a), identified several key areas for which it would be useful to have national indicators but for which data are currently difficult to compare across transportation modes: safety; access to services by such groups as elderly, disabled, low-income, and rural populations; and the efficiency and quality of service provided by the transportation system. BTS has covered some of these topics in its Transportation Statistics Annual Reports (TSARs), as well as other topics. The 1996 TSAR includes chapters on passenger travel and the move- ment of freight, with tables on the physical condition of highways, runways and aircraft, and other transportation facilities; the role of transportation in the economy; safety; energy use; and transportation and the environment. Although there is much material in these analyses that could support the development of key indicators, there are also many hurdles to overcome. In the important area of safety, the 1996 TSAR notes some of the conceptual and measurement problems for developing meaningful trend indicators. One con 3In this regard, Data for Decisions (National Research Council, 1992a:32-37) recommends that indicators be developed for types of markets rather than for transportation modes. As an example, cross-modal indicators of travel delays might be developed for intercity markets, such as an indicator that looks specifically at weak links between modes (e.g., highway or rail connections to airports). Such an indicator could help policy makers identify a fuller range of options for improvement of transportation infrastructure than is likely to emerge from analysis of indicators that pertain to particu- lar transportation modes.
ENSURING RELEVANCE 75 ceptual issue is the need to relate trends in injuries and fatalities to measures of risk exposure. The more that people use the transportation system, the more they are exposed to the risk of accidents, so that such measures as fatalities or injuries per number of hours of operation or per unit distance of travel are needed to adjust the raw data appropriately. The appropriate measure of risk exposure may differ across transportation modes, which in turn can make it difficult to compare trends across modes. The 1996 TSAR further notes deficiencies and inconsistencies in the report- ing of accidents across transportation modes and governmental jurisdictions, particularly for crashes that involve property damage only and for crashes that involve injuries but not fatalities. Information is also inadequate with which to assess the role of environmental conditions (e.g., weather, lighting) and other contributing factors (e.g., human fatigue) in causing accidents. There is clearly much to be done to develop consistent and useful indicators in transportation safety, as well as other areas. BTS will need to work closely with statistical and analysis units in the other USDOT modal administrations, with states and metropolitan planning organizations, and with the transportation community at large to identify priority areas for indicators and appropriate data and methods for developing useful time series. As a way to proceed, we suggest that BTS look first to build on a few of the data series that are produced by other USDOT modal administrations and con- sider how it can add value to them and what new series should be developed to fill existing gaps. BTS should also consider methods to integrate already existing data from BTS, other USDOT modal administrations, and other federal statistical agencies to develop key indicators. With regard to the form of presenting indicators, we suggest that BTS estab- lish as a goal regular publication at first annually and moving to a more fre- quent schedule as feasible and desirable of a document containing 10 to 20 items that are relevant to policy issues in the transportation field. The publication could be in the form of a freestanding chartbook with back-up statistical informa- tion and short analyses of the implications of the material presented. (It could also be a supplement to another publication.~4 The focus of the chartbook should reflect key policy concerns, perhaps chang- ing as policy issues do. For example, BTS might plan a few series to explain the effects of increased trade on commodity flow changes, mode of shipment changes, and differences occurring because of changing trade relationships with the North American Free Trade Agreement, the European Community, and the Far East. Similarly, it might plan a series to track the effects on commodity transport of such industry changes as the pending railroad consolidation on the East Coast. 4Such a chartbook (or supplement) would complement rather than replace the annual National Transportation Statistics report; the latter publication brings together for reference purposes many more tables than would appear in a chartbook but without accompanying analyses.
76 BUREAU OF TRANSPORTATION STATISTICS: PRIORITIES FOR THE FUTURE BTS might also consider using data produced by the Bureau of Labor Statistics on employment and combining them with data from the American Travel Survey and the Nationwide Personal Transportation Survey on travel from place of resi- dence to place of work, including the modes used by workers, their costs, and availability. Another set of indicators, as noted previously, might deal with safety in the air, on the highways, and other modes. Another approach might be to select one of the themes used in the TSARs and develop one or more indicators to inform the public about progress for example, changes in transportation pro- ductivity or changes in the relationship between transportation and the environ- ment. (See the section below on "Analysis Programs" for a discussion of how a chartbook of indicators could relate to the TSARs and how the latter could use- fully be reconfigured.) COORDINATION OF DATA COLLECTION AND FILLING GAPS It is important for a statistical agency to coordinate data collection in its area to the extent feasible. Coordination is necessary to make the most cost-effective use of scarce resources to provide relevant, high-quality information for such purposes as developing appropriate statistical indicators and directly serving the information needs of policy makers and other users. (Relatedly, a statistical agency should establish regular sources of input from data users, producers, and methodologists about priority information needs and methods to supply them- see the section below on "Identifying User Needs.") Effective mechanisms for coordination (and input) are required to identify: . areas of overlap in data collection for which it may be possible and desir- able to reduce duplication and associated costs and burdens on respondents and thereby free up resources for other needed data; · areas for which no data system currently provides relevant measures and for which it may be possible to fill gaps; . linkages among data systems that may increase their relevance and ana- lytical power; and · innovations in data collection and analysis methods that may improve the quality of measures across data systems. There are at least three domains for coordination of data collection in the transportation field: coordination within USDOT; coordination between USDOT and other federal statistical and program agencies; and coordination between USDOT and such key data providers and users as states and metropolitan plan- ning organizations. We recommend that BTS take a major step to facilitate data coordination among the modal administrations in USDOT through a department- wide statistical budget.
ENSURING RELEVANCE 77 A USDOT Statistical Budget USDOT supports a large number of data collection and analysis programs, with significant statistical activities in almost every modal administration (see Appendix B). To the existing programs, BTS has added new data collection systems on intermodal flows of passengers and freight. We believe it would help USDOT evaluate and improve the relevance and cost-effectiveness of its large array of statistical activities to have BTS prepare each year a consolidated statistical budget for the department (see recommendation 7 at the end of the chapter). BTS could follow the example of the Statistical Policy Division in the U.S. Office of Management and Budget, which brought together information on agencies' proposed fiscal 1998 statistical budgets across the entire federal gov- ernment for purposes of program review and decision making among competing priorities.5 For USDOT, BTS should compile budget information from all of the modal administrations about their statistical programs, including supporting justifica- tion. BTS could organize this material in several ways for example, by subject area as well as by modal administration and agency. It should add commentary as appropriate for example, noting relationships among programs in different modal administrations or pointing out user needs that no USDOT data collection program currently addresses. BTS would not determine the budget allocations for any other modal admin- istration. Rather, the consolidated statistical budget would be available for the secretary's use in making final proposed budget allocations to transmit to OMB. It would help clarify for the secretary what the individual modal administrations see as priorities for data collection and analysis. At the same time, it would help the secretary determine how well the agencies' priorities accord with department- wide needs and whether some reallocation of resources among data programs within a modal administration would enable the department to be more cost- effective in providing relevant data for policy purposes and to serve other impor- tant needs of the transportation community. To ensure that the preparation of a USDOT statistical budget becomes institutionalized and integrated into the department's decision making, the reauthorization of BTS should directly assign to BTS the responsibility for compiling the statistical budget each year. We repeat that the USDOT statistical budget would be compiled and anno- tated by BTS but that BTS would not make budget decisions for any other modal administration. Also, the statistical budget would not include all USDOT data programs. Many data collection systems in USDOT provide modal administra 5The Statistical Policy Division some years ago regularly produced cross-cutting statistical budgets as part of the preparation of the president's budget submission to Congress. The practice then lapsed and was just resumed this year. (Routinely, the division produces a cross-cutting description of federal statistical activities after the budget preparation is completed i.e., to document rather than to inform decision making see, e.g., Executive Office of the President, 1997b.)
78 BUREAU OF TRANSPORTATION STATISTICS: PRIORITIES FOR THE FUTURE lions with information for program management and regulation and have few statistical uses. Examples are the large number of operational databases of the Federal Aviation Administration (e.g., the Aircraft Registration System and Manufacturing Inspection Management Information System see National Re- search Council, 1992a:111-114~. The budgets for operating such data systems would not be included in the USDOT statistical budget, except for that portion that may be devoted to statistical analysis of the data for public use.6 Practically speaking, the USDOT statistical budget would include the bud- gets of the major statistical units in the modal administrations (e.g., the Safety Data Services Division in the Federal Aviation Administration see Appendix B), plus other programs that are not lodged within a separate statistical unit but that the modal administration identifies as having an important statistical compo- nent. Indeed, the preparation of the USDOT statistical budget may identify areas in which it would be helpful to a modal administration and for transportation policy analysis, planning, and research more generally to develop the statistical applications of an operational database. As BTS enhances its statistical capabili- ties and achieves excellence in its own operations, it should be increasingly able to offer technical assistance to the other modal administrations in this regard. Other Coordination Activities There are other coordination activities that BTS should consider working into its implementation plan, as available resources and the demands of other priorities permit. For example, BTS could undertake periodic reviews of existing transportation data systems to determine how well they meet the requirements for development of indicators on specific topics and, more generally, how well they contribute to BTS's vision of a comprehensive transportation data system. Such reviews may identify data gaps that are important to fill. They may also identify opportunities for linking or integrating data systems to achieve such goals as making the combined data relevant for a broader range of analyses, improving data quality by such means as standardizing definitions for key variables, and reducing costs. An obvious first priority for a cross-system data review would be for BTS to look at sources of data on intermodal transportation, including its two flagship intermodal surveys the Commodity Flow Survey and the American Travel Sur 6The Statistical Policy Division in the U.S. Office of Management and Budget faces a similar issue of defining which data programs to include in the cross-cutting federal statistical budget: the criterion used is that statistical programs of $500,000 or more in annual expenditures are to be included. As an example, the budget includes the Statistics of Income program in the Internal Revenue Service, which produces statistical publications and data files from tax return data, suitably processed to protect the confidentiality of the information for tax filing units, but the budget does not include the vastly larger costs of the Internal Revenue Service to enter the data from tax returns, calculate taxes owed and refunds due, and monitor compliance.
ENSURING RELEVANCE 79 vey and other possibly relevant data sources. In Appendix F. we briefly review sources of data on household travel and develop some ideas about data linkage opportunities and remaining data gaps (see also Bureau of Transportation Statis- tics, 1993c, 1995:96-102~. Conducting cross-system data reviews, as well as carrying out other kinds of coordination activities, will require that BTS involve appropriate agencies through working groups, interagency committees, and the like. In some instances, it will be necessary to involve not only one or more agencies in USDOT, but also out- side agencies for example, other federal statistical and program agencies. Experience has demonstrated the difficulty of achieving effective interagency collaboration, particularly when the agencies involved are from different depart- ments or levels of government. The history of a short-lived federal interagency transportation statistics coordinating committee that was in existence in the early 1990s illustrates the problem. The committee initially attracted a large atten- dance to exchange information; however, no action agenda was developed, atten- dance fell off, and the committee became moribund. This is a common pattern with interagency groups, as participants are pulled back to the agendas of their own agencies and the activities of the interagency group become largely ones of show and tell. Generally, an effective interagency group requires that agencies be involved because they want to be, believe they can accomplish more on the topic together than apart, have an action agenda, contribute people or funding to the extent possible, and have the support of their agency heads. These characteristics sug- gest that it generally makes more sense to establish interagency groups on an as- needed basis with a specific set of issues and agenda in mind than to set up an umbrella committee. An example of such a special-purpose committee in which BTS plays an active role is the Federal Geographic Data Committee, which is working to standardize geographic information system (GIS) capabilities for the federal government as a whole. It may be that other special-purpose interagency committees will be useful to establish in the future (e.g., an interagency commit- tee on the development and appropriate application of data for monitoring air quality and other environmental effects of the transportation system). IDENTIFYING USER NEEDS It is important for a statistical agency to obtain regular input not only on the usefulness of its current products and services (e.g., through customer surveys), but also on unmet data needs and priorities for data, indicators, analyses, and improved concepts and measures that are relevant to users' concerns. The agency must assess and interpret the input it receives users are not always the best judges of appropriate or feasible data constructs or measures; also, they will gen- erally want more than it is possible to provide within budget constraints. None- theless, user input is clearly central to the development by a statistical agency of
80 BUREAU OF TRANSPORTATION STATISTICS: PRIORITIES FOR THE FUTURE its vision of the important information needs in its area and the characteristics of a comprehensive data system to serve those needs. There are many constituencies for transportation data, including federal sta- tistical and program agencies inside and outside USDOT, congressional agen- cies, state and local agencies, private-sector organizations, academic researchers, and the public. Input from federal agencies comes (or will come) from such activities as developing quality standards for transportation data, constructing transportation indicators, and reviewing data systems. In addition, BTS obtains input from the Advisory Council on Transportation Statistics (ACTS) (mandated in the 1991 ISTEA), which meets twice a year to consider priorities for BTS's growth and development. Although the ACTS provides a range of public- and private-sector user perspectives, its membership is small (6 people). BTS also sponsors six standing committees of the Transportation Research Board, which bring together researchers and other users to exchange information about data needs and applications in several areas (see Chapter 2~. However, the transporta- tion data community is so large and diverse that regular communication with many more users in state and local organizations, the private sector, and academia will be needed for BTS to develop and refine its vision and implementation plan for improving transportation data. States and metropolitan planning organizations (MPOs) are particularly im- portant constituencies for BTS to work with because of the federally based struc- ture of planning, investment, and associated data collection and analysis for the U.S. transportation system. States and MPOs play a vital role in developing and implementing transportation policy and in making decisions about investments in transportation infrastructure that have important consequences for the cost-effec- tiveness of the transportation system as a whole. They also provide many key transportation data sets and, in turn, use transportation data for a wide range of purposes. A decision by Congress to devolve yet more responsibilities for trans- portation policy planning and implementation could further strengthen the role of states and MPOs. BTS does not at present operate data collection systems that require working directly with states or MPOs to obtain data; such systems (e.g., the Highway Performance Monitoring System) are lodged with other USDOT modal adminis- trations.7 However, as the lead statistical agency for the department, BTS should develop regular channels of communication with these two important constituen- cies and in the past year it has begun to do so. We recommend as a priority effort that BTS continue with its plans for obtaining regular input from states and MPOs and, relatedly, its plans for technical assistance to help states and MPOs make more effective use of transportation data (see recommendation 8 at the end of the chapter). 7See Ruddick (1996) for a comparative descriptive analysis of nine federal-state data collection systems, including the Highway Performance Monitoring System and the General Highway Statistics Program in USDOT.
ENSURING RELEVANCE 8 Outreach to States and MPOs BTS began over a year ago an active outreach program of meetings with state transportation officials, which were carried out in collaboration with the Office of Highway Information Management in the Federal Highway Adminis- tration and the American Association of State Highway and Transportation Offi- cials. It then expanded these efforts to include MPOs (working through the Asso- ciation of Metropolitan Planning Organizations). A conference held in spring 1997 brought together state and local officials with staff of BTS, the Federal Highway Administration, and the Federal Transit Administration to discuss pri- ority data needs, the appropriate role of each level of government in transporta- tion data collection and dissemination, the implications of technological advances (e.g., intelligent transportation systems) for data collection and dissemination, and the kinds of technical assistance that could help states and MPOs make more effective use of national transportation data sets. We urge that the conference be followed up by considering the most effec- tive communication channels to establish for regular, two-way interaction of BTS and other USDOT modal administrations with states and MPOs. Such interac- tion will be vitally important for BTS to carry out its mission in developing trans- portation indicators and filling key data gaps in its vision of a comprehensive transportation system that is relevant to user information needs. Technical Assistance As an outgrowth of its rounds of meetings with state transportation officials, BTS has begun to conceptualize ways to provide technical assistance to states and MPOs in obtaining, collecting, and analyzing transportation data. Technical assistance can be a draining activity for a small statistical agency, particularly if it involves one-on-one assistance on particular problems of individual organiza- tions. However, it is possible to structure a technical assistance program so that such products as user's guides and application software are developed that have broad utility for many organizations. In light of its mandate for intermodal data, we suggest that BTS focus its technical assistance activities on developing tools for states and MPOs for intermodal analysis, using data from BTS's two surveys, the Commodity Flow Survey (CFS) and the American Travel Survey (ATS), and other relevant sources. The 1991 ISTEA increased the planning requirements for states and MPOs, in- cluding that they consider system-wide issues instead of focusing narrowly on particular transportation modes. The ATS and the CFS, alone and linked with other information, will provide rich data sets for cross-modal analyses of trans- portation flows within and across states and metropolitan areas. The CFS also provides data that could be useful to states in planning future economic develop
82 BUREAU OF TRANSPORTATION STATISTICS: PRIORITIES FOR THE FUTURE ment (e.g., locating or further developing an airport that could be a hub for long- distance shipments of specific kinds of products). Technical assistance in using the ATS, the CFS, and related data could take such forms as user's guides that highlight state and local applications of the data, special analysis software, and innovative methods of data analysis. Assistance could be offered in a variety of formats and venues, such as access via the Internet, continuing education classes, conferences, and, occasionally, on-site work on a particular project. Analytic tools and techniques could be developed in some cases directly by BTS, or by working with one or a few states, or through con- tracts with universities or other organizations. The experience of other federal statistical agencies suggests that funding one or a few states to develop data processing and analysis tools has the advantage that other states may be more receptive to using a state-developed product (see Ruddick, 1996~. If BTS uses contractors for its technical assistance activities, it is important that some BTS analysis staff also be involved, particularly in the development of analysis tools that exploit the information value of the ATS and the CFS. BTS staff need hands-on experience in using the ATS and CFS data for a variety of analysis needs in order not only to help states, MPOs, and others use the data more effectively, but also to set priorities for improving the relevance and quality of the data for the future. ANALYSIS PROGRAMS Data analysis is an important component of the work of a statistical agency- not only analysis of quality measures and issues related to methods, but also analysis of substantive topics. Statistical agencies should not be advocates for particular policies, but they should engage in research that sheds light on the effects of alternative policies and that illuminates trends and relationships in policy-relevant areas. Careful analyses in substantive areas that explain what the data show and qualify findings with information about the quality and appropri- ateness of the data for particular uses are very helpful for users. Such analyses are also critical to the statistical agency itself to help it understand the data in its area, determine how to keep the data relevant for policy and other purposes, and continually refine its vision of a comprehensive transportation data system to serve user information needs. (See Bonnen, 1997, for a discussion of the analysis roles of statistical agencies.) Developing a substantive research program can be difficult for a statistical agency. Such research is often a target for cutbacks when budgets are tight in favor of preserving resources for data collection. Also, analytical researchers and statisticians and methodologists on the staff may not always work together effec- tively because of differing expertise and perspectives. Statistical agencies need to address these challenges in order to have an active in-house research program that benefits the agency and its users.
ENSURING RELEVANCE 83 Encouraging Substantive Research At BTS, the director has emphasized the importance of substantive research on transportation issues. The Transportation Statistics Annual Reports feature each year a special analytical section on a particular topic of policy concern, in addition to providing updated assessments of the state of transportation. BTS also regularly hosts seminars and conferences on research topics, and it recently inaugurated a new twice-yearly, peer-reviewed Journal of Transportation and Statistics that will feature research articles. We support BTS's research initiatives and encourage an expansion of them as resources and the demands of other priorities permit. In particular, we encour- age research by BTS on the substantive uses of data from the CFS and the ATS that can help policy makers understand the problems and opportunities for cost- effective intermodal transportation of people and goods. At the same time, because research is labor-intensive and time-consuming, we urge BTS to assess how it is carrying out its research activities and whether there are more cost-effective ways to approach them. One way for BTS to aug- ment its in-house research capabilities would be for it to announce special re- search initiatives in a request for proposals aimed at university faculty involved in transportation studies. Looking at the demands on its own staff, we encourage BTS to assess the contribution of the TSARs to the agency's analysis functions. The Role of the TSARs BTS is mandated by the 1991 ISTEA to produce a Transportation Statistics Annual Report, and the TSARs produced to date have contained useful data and analyses that were not previously available to transportation planners and ana- lysts. However, the TSAR may not be the best format with which to provide transportation data analyses to the user community or, relatedly, to provide a set of widely followed national transportation indicators. BTS is still a small agency, and the preparation of each year's TSAR absorbs substantial time and energy of BTS's in-house and contractor staff. Yet a thick annual report of textual chapters, even with many tables and charts, does not seem well suited to serve the information needs of policy makers and other users. They are all too likely to lose sight of the forest for the trees and to find such a publication too difficult to use, either for locating a key statistic or for under- standing the relevance of trends and relationships in the data for particular policy issues and concerns. In contrast, such publications as the monthly Survey of Current Business, published by the Bureau of Economic Analysis, the Monthly Labor Review, pub- lished by the Bureau of Labor Statistics, and the quarterly Social Security Bulle- tin regularly include standard, easily locatable tables that update key statistics. In addition, they include articles on selected topics that amplify the material in the
84 BUREAU OF TRANSPORTATION STATISTICS: PRIORITIES FOR THE FUTURE tables. These articles may be substantive (e.g., analyzing particular trends) or related to methods (e.g., analyzing measurement problems for a particular vari- able). A similar monthly publication would not be feasible for BTS at its present stage of development and is likely not needed in any case, given that transporta- tion indicators tend not to show pronounced movements over short periods of time. However, a format that provides regularly updated, standardized tables and charts of key indicators together with topical articles could be more useful and easier to produce than the TSARs. We suggest that BTS consider alternative formats to the TSARs and that, if an alternative format seems workable, that it seek authority to adopt that format in place of the required annual report. One alternative would involve the chartbook that we earlier suggested BTS publish, together with BTS's new twice-yearly Journal of Transportation and Statistics. Under this alternative, the chartbook would include 10 to 20 key statistical indicators with accompanying brief commentary and notes on methods; it would first appear annually but, as resources permit, should be published more fre- quently. It would be a publication that users look forward to receiving in order to follow key trends. BTS's new journal would publish not only peer-reviewed research and methods articles from BTS staff, other USDOT staff, and outside researchers, but also articles on the state of transportation containing the kind of analytical material that currently appears in the TSARs perhaps in a special sec- tion covering selected topics in each issue. (As examples, an article on cross- modal trends in safety might appear every December, and an article on energy impacts of transportation might appear every June.) Our expectation is that re- ports on particular aspects of the transportation system (e.g., safety, access, con- dition of the infrastructure) that are presented in the form of journal articles will be more accessible to users than the current TSAR format and less burdensome on the BTS staff to develop.8 Together, the chartbook and the journal would fulfill the mandate in the 1991 ISTEA for BTS to provide information about the transportation system in an annual report. (In this chapter and the preceding one, we recommend several changes in BTS's roster of publications. Table 4-2 maps BTS's current publica- tions to those that we recommend.) An alternative format that would accomplish the same goal would be to fold both the chartbook and related analytical articles into the new journal as a regularly appearing supplement. Whether this alterna- tive is preferable to a separate chartbook depends on how often it appears useful to publish key indicators and whether the desired publication schedule could be accommodated by the journal. Under either alternative (folding both tables and analyses or just analyses into the journal), a goal for the longer term should be to The articles in the special section of the journal should be reviewed and held to high standards, but the review process should be managed by BTS, given its mandate to produce regular reports on the transportation system, and not by an outside editorial board.
85 a' cq o son a' . . o c) c) o .= a' cq c) · - ~ cq .= vO o .~ cq EM o a' Ad ¢ Em ca to 4= v, v .~ c) ~ ~ ~ ~ l ' ~ ^ 1 l at i ) , ~ ~' ~ [ ' |, ~ ' ~ ~ ca ~o ~·> o ~ ~ o to ~ ~ o if .= ~ ~ o ~ ~ ~o ~ c ~ ~ I ~ ~ .^ ~.^ ~ ~ .^ ~3 %~ 3 ~ ~ ~o I , I l ,, , , . . .
86 BUREAU OF TRANSPORTATION STATISTICS: PRIORITIES FOR THE FUTURE publish the journal on a quarterly instead of a twice-yearly basis in order to ac- commodate the additional material. Whatever publication format is adopted (one of those suggested or another), we repeat that it is important for BTS to find the most cost-effective ways by which to conduct and report the results of substantive research with its data. Such research is an essential component of a statistical agency's mission to organize, interpret, and communicate data so that the data become information that is rel- evant for policy needs and other purposes. RECOMMENDATIONS Vision and Plan (5) BTS should develop a long-term strategy for implementing fully all of the areas in its mandate in order to evolve as a statistical agency that ensures the relevance, as well as the quality, of transportation data. The implementation plan should set priorities within the context of a vision of a comprehensive system of transportation data. National Transportation Indicators (6) BTS should develop key national statistical indicators for the trans- portation system for example, multimodal series in the areas of safety, travel patterns, and the condition of the transportation infrastructure in consultation with the statistical and analysis units in the other USDOT modal administrations and the transportation community. USDOT Statistical Budget (7) In the reauthorization of BTS, Congress should require BTS to com- pile, analyze, and provide to the secretary of transportation a department- wide statistical program budget for the secretary's use in making decisions during the budget process. Building Relationships with States and Metropolitan Planning Organizations (8) BTS should regularly meet with representatives from states and met- ropolitan planning organizations to help determine priorities for key national statistical indicators, other data, analyses, products and services, and im- provements in data concepts and measurements. BTS should also provide technical advice to states and metropolitan planning organizations to help them make more effective use of BTS and other transportation data.
