National Academies Press: OpenBook

The Bureau of Transportation Statistics: Priorities for the Future (1997)

Chapter: 3 FOCUSING ON DATA QUALITY

« Previous: 2 HISTORY AND ASSESSMENT OF BTS
Suggested Citation:"3 FOCUSING ON DATA QUALITY." National Research Council. 1997. The Bureau of Transportation Statistics: Priorities for the Future. Washington, DC: The National Academies Press. doi: 10.17226/5809.
×
Page 31
Suggested Citation:"3 FOCUSING ON DATA QUALITY." National Research Council. 1997. The Bureau of Transportation Statistics: Priorities for the Future. Washington, DC: The National Academies Press. doi: 10.17226/5809.
×
Page 32
Suggested Citation:"3 FOCUSING ON DATA QUALITY." National Research Council. 1997. The Bureau of Transportation Statistics: Priorities for the Future. Washington, DC: The National Academies Press. doi: 10.17226/5809.
×
Page 33
Suggested Citation:"3 FOCUSING ON DATA QUALITY." National Research Council. 1997. The Bureau of Transportation Statistics: Priorities for the Future. Washington, DC: The National Academies Press. doi: 10.17226/5809.
×
Page 34
Suggested Citation:"3 FOCUSING ON DATA QUALITY." National Research Council. 1997. The Bureau of Transportation Statistics: Priorities for the Future. Washington, DC: The National Academies Press. doi: 10.17226/5809.
×
Page 35
Suggested Citation:"3 FOCUSING ON DATA QUALITY." National Research Council. 1997. The Bureau of Transportation Statistics: Priorities for the Future. Washington, DC: The National Academies Press. doi: 10.17226/5809.
×
Page 36
Suggested Citation:"3 FOCUSING ON DATA QUALITY." National Research Council. 1997. The Bureau of Transportation Statistics: Priorities for the Future. Washington, DC: The National Academies Press. doi: 10.17226/5809.
×
Page 37
Suggested Citation:"3 FOCUSING ON DATA QUALITY." National Research Council. 1997. The Bureau of Transportation Statistics: Priorities for the Future. Washington, DC: The National Academies Press. doi: 10.17226/5809.
×
Page 38
Suggested Citation:"3 FOCUSING ON DATA QUALITY." National Research Council. 1997. The Bureau of Transportation Statistics: Priorities for the Future. Washington, DC: The National Academies Press. doi: 10.17226/5809.
×
Page 39
Suggested Citation:"3 FOCUSING ON DATA QUALITY." National Research Council. 1997. The Bureau of Transportation Statistics: Priorities for the Future. Washington, DC: The National Academies Press. doi: 10.17226/5809.
×
Page 40
Suggested Citation:"3 FOCUSING ON DATA QUALITY." National Research Council. 1997. The Bureau of Transportation Statistics: Priorities for the Future. Washington, DC: The National Academies Press. doi: 10.17226/5809.
×
Page 41
Suggested Citation:"3 FOCUSING ON DATA QUALITY." National Research Council. 1997. The Bureau of Transportation Statistics: Priorities for the Future. Washington, DC: The National Academies Press. doi: 10.17226/5809.
×
Page 42
Suggested Citation:"3 FOCUSING ON DATA QUALITY." National Research Council. 1997. The Bureau of Transportation Statistics: Priorities for the Future. Washington, DC: The National Academies Press. doi: 10.17226/5809.
×
Page 43
Suggested Citation:"3 FOCUSING ON DATA QUALITY." National Research Council. 1997. The Bureau of Transportation Statistics: Priorities for the Future. Washington, DC: The National Academies Press. doi: 10.17226/5809.
×
Page 44
Suggested Citation:"3 FOCUSING ON DATA QUALITY." National Research Council. 1997. The Bureau of Transportation Statistics: Priorities for the Future. Washington, DC: The National Academies Press. doi: 10.17226/5809.
×
Page 45
Suggested Citation:"3 FOCUSING ON DATA QUALITY." National Research Council. 1997. The Bureau of Transportation Statistics: Priorities for the Future. Washington, DC: The National Academies Press. doi: 10.17226/5809.
×
Page 46
Suggested Citation:"3 FOCUSING ON DATA QUALITY." National Research Council. 1997. The Bureau of Transportation Statistics: Priorities for the Future. Washington, DC: The National Academies Press. doi: 10.17226/5809.
×
Page 47
Suggested Citation:"3 FOCUSING ON DATA QUALITY." National Research Council. 1997. The Bureau of Transportation Statistics: Priorities for the Future. Washington, DC: The National Academies Press. doi: 10.17226/5809.
×
Page 48
Suggested Citation:"3 FOCUSING ON DATA QUALITY." National Research Council. 1997. The Bureau of Transportation Statistics: Priorities for the Future. Washington, DC: The National Academies Press. doi: 10.17226/5809.
×
Page 49
Suggested Citation:"3 FOCUSING ON DATA QUALITY." National Research Council. 1997. The Bureau of Transportation Statistics: Priorities for the Future. Washington, DC: The National Academies Press. doi: 10.17226/5809.
×
Page 50
Suggested Citation:"3 FOCUSING ON DATA QUALITY." National Research Council. 1997. The Bureau of Transportation Statistics: Priorities for the Future. Washington, DC: The National Academies Press. doi: 10.17226/5809.
×
Page 51
Suggested Citation:"3 FOCUSING ON DATA QUALITY." National Research Council. 1997. The Bureau of Transportation Statistics: Priorities for the Future. Washington, DC: The National Academies Press. doi: 10.17226/5809.
×
Page 52
Suggested Citation:"3 FOCUSING ON DATA QUALITY." National Research Council. 1997. The Bureau of Transportation Statistics: Priorities for the Future. Washington, DC: The National Academies Press. doi: 10.17226/5809.
×
Page 53
Suggested Citation:"3 FOCUSING ON DATA QUALITY." National Research Council. 1997. The Bureau of Transportation Statistics: Priorities for the Future. Washington, DC: The National Academies Press. doi: 10.17226/5809.
×
Page 54
Suggested Citation:"3 FOCUSING ON DATA QUALITY." National Research Council. 1997. The Bureau of Transportation Statistics: Priorities for the Future. Washington, DC: The National Academies Press. doi: 10.17226/5809.
×
Page 55
Suggested Citation:"3 FOCUSING ON DATA QUALITY." National Research Council. 1997. The Bureau of Transportation Statistics: Priorities for the Future. Washington, DC: The National Academies Press. doi: 10.17226/5809.
×
Page 56
Suggested Citation:"3 FOCUSING ON DATA QUALITY." National Research Council. 1997. The Bureau of Transportation Statistics: Priorities for the Future. Washington, DC: The National Academies Press. doi: 10.17226/5809.
×
Page 57
Suggested Citation:"3 FOCUSING ON DATA QUALITY." National Research Council. 1997. The Bureau of Transportation Statistics: Priorities for the Future. Washington, DC: The National Academies Press. doi: 10.17226/5809.
×
Page 58
Suggested Citation:"3 FOCUSING ON DATA QUALITY." National Research Council. 1997. The Bureau of Transportation Statistics: Priorities for the Future. Washington, DC: The National Academies Press. doi: 10.17226/5809.
×
Page 59
Suggested Citation:"3 FOCUSING ON DATA QUALITY." National Research Council. 1997. The Bureau of Transportation Statistics: Priorities for the Future. Washington, DC: The National Academies Press. doi: 10.17226/5809.
×
Page 60
Suggested Citation:"3 FOCUSING ON DATA QUALITY." National Research Council. 1997. The Bureau of Transportation Statistics: Priorities for the Future. Washington, DC: The National Academies Press. doi: 10.17226/5809.
×
Page 61
Suggested Citation:"3 FOCUSING ON DATA QUALITY." National Research Council. 1997. The Bureau of Transportation Statistics: Priorities for the Future. Washington, DC: The National Academies Press. doi: 10.17226/5809.
×
Page 62
Suggested Citation:"3 FOCUSING ON DATA QUALITY." National Research Council. 1997. The Bureau of Transportation Statistics: Priorities for the Future. Washington, DC: The National Academies Press. doi: 10.17226/5809.
×
Page 63

Below is the uncorrected machine-read text of this chapter, intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text of each book. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.

Focusing on Data Quality A statistical agency must be more than a data compilation and dissemination agency. It must create useful information, which means that it must focus on documenting, evaluating, and improving the quality and relevance of the data within its subject area. Without such a focus, resources are likely to be wasted- both by the statistical agency in compiling and disseminating data that are of poor quality or not relevant to analysis needs and by policy makers, planners, and researchers who are left to work with deficient or inappropriate data. To ensure data quality and relevance for its users while working to minimize costs and burden on data providers, a statistical agency must also keep pace with advances in data collection and statistical and analytical methods and techniques. The Bureau of Transportation Statistics (BTS) should begin now to devote more of its attention to data documentation, evaluation, and improvement, even if it means slowing down its efforts to be a one-stop-shopping source for users of every available transportation data set. Many state transportation officials and other users from whom we heard are looking to BTS to help them sort through the mass of available information to distinguish those data that are better and more appropriate for their needs from data that are of lesser quality and usefulness. BTS needs not only to help users in this regard, but also to work to improve the available base of information for addressing important transportation policy is- sues and research questions. The 1991 Intermodal Surface Transportation Effi- ciency Act (ISTEA) recognized the need for a statistical agency with a broad mandate to provide leadership for transportation data improvement by stipulating that BTS issue guidelines to ensure that the information from the U.S. Depart- ment of Transportation (USDOT) is accurate, reliable, relevant, and in a form that permits systematic analysis. 31

32 BUREAU OF TRANSPORTATION STATISTICS: PRIORITIES FOR THE FUTURE "Quality" and "relevance" are multidimensional attributes of data, each of which requires attention by a statistical agency.) In this chapter, we define what we mean by "quality," which encompasses the comparability, accuracy, and vari- ability of the data from a measurement system. BTS needs to focus immediately on data quality to qualify its initial emphasis on making as many kinds of data as widely available as possible. To make such a focus possible, BTS needs to strengthen the statistical and technical capabilities of its staff. The remainder of this chapter first discusses staffing requirements for BTS and then considers pri- ority activities for BTS in the data quality area. Such activities include the devel- opment of quality standards for USDOT and improvements in the documentation of available data, both to assist data users and to provide the basis for continuing evaluation and improvement of transportation data systems. The committee's primary recommendations appear at the end of the chapter. In Chapter 4, we define what we mean by "relevance," which encompasses the appropriateness of concepts, definitions, and measurements, the level of sub- ject and geographic detail, and the timeliness of data from a measurement system. Chapter 4 discusses activities, some of immediate priority and others that repre- sent longer-term goals, for BTS to ensure the relevance of transportation data for policy making and other purposes. DIMENSIONS OF QUALITY "Data quality" concerns the effects of how measurement systems are de- signed and conducted.2 Dimensions of data quality include: · Comparability across data systems and time (e.g., for cross-modal com- parisons), which involves not only consistency of definitions, but also consis- tency, or at least similarity, among design features and data collection and pro- cessing procedures. As an example, the comparability of two data systems may be affected by differences in the method of data collection (such as personal interview, telephone interview, self-report, abstracting information from admin- istrative reports, and obtaining data from such recording or sensing equipment as highway sensors). (See Box 3-1 for examples of lack of comparability of data for a single transportation mode and for cross-modal analysis.) iThe two attributes should not be viewed as totally distinct. Indeed, relevance may be said to encompass quality in that relevance means broadly the usefulness of a data set for an application, and data of poor quality are hardly useful even if they provide relevant information in other respects. However, the requirements (e.g., staff skills) for addressing data quality as distinct from the other substantive dimensions of relevance differ, so that it makes sense to discuss data quality and relevance as separate attributes. 2The discussion draws on the thinking among statistical agencies in the United States and else- where. See, for example, Australian Bureau of Statistics (1990); Bureau of Economic Analysis (1995); Statistics Canada (1992); and Statistics Sweden (1994).

FOCUSING ON DATA QUALITY 33

34 BUREAU OF TRANSPORTATION STATISTICS: PRIORITIES FOR THE FUTURE . Accuracy or bias, which in general concerns how well a set of estimates approximates the true values of the phenomena under study and specifically con cerns errors that are due to systematic mismeasurement.3 Sources of systematic bias may include: differential coverage of groups in the population being studied (e.g., research has documented undercoverage of minorities in the decen nial census and household surveys and of smaller establishments in surveys of businesses and farms); differential nonresponse by reporting units (e.g., lower response rates to surveys for working families who are harder to find at home, underreporting of accidents that involve minor injuries or property damage); missing or erroneous reporting of specific items that is linked with other characteristics (e.g., higher missing data rates in surveys for in come and assets on the part of people with high incomes); biases in imputation and other data editing procedures that attempt to correct for missing data and other reporting errors. Variability, which includes instability in estimates from the use of a sample and from other sources. For example, variability may stem from differ- ences in how stringently enumerators or other data collectors apply specified procedures or from the variable application of editing and imputation procedures. · Extent of revisions in time series, due to supplementation of preliminary data with later reports and other factors (see Young, 1996~. . Statistical agencies have developed tools to measure some of the above as- pects of data quality. Effective application of these tools requires that there be a technical staff engaged in ongoing methodological work to measure data quality and to develop design features for data systems that can provide users with evi- dence about quality. (See discussion below in the section on "Staffing.") A focus on documenting, evaluating, and improving data quality along the above dimensions is a central mission of a statistical agency. Such a focus is particularly important for BTS because data collection is so widely dispersed in the transportation field, making it difficult for users to assess the comparability, accuracy, and variability of data programs across the various transportation modes and a variety of public and private data sources. We did not ourselves conduct a review of the quality of transportation data programs, and hence we cannot say whether they have serious data quality prob- lems. However, no set of data is without error, and every data program has quality problems to a greater or lesser degree. What is important for a statistical 3A more technical definition is that estimates have low bias if they tend to be equal to the true values on average when the sampling process (or other data collection procedure) is repeated many times.

FOCUSING ON DATA QUALITY 35 agency like BTS is to document the errors and other problems in data programs in its area and to assess the extent to which they compromise the use of the data for their intended purposes. On the basis of documentation and evaluation, the agency should identify priority data programs for improvement, taking account of the need for the data and the costs and feasibility of improvement, and should implement a phased effort to effect improvements as resources are available. All that is not to say that data documentation, evaluation, and improvement efforts are not being carried out appropriately and well for particular transporta- tion data programs, such as particular surveys and administrative reporting sys- tems. However, data improvement activities for a particular data program do not ensure that the data will be comparable or of comparable quality when used with data from other programs, which is likely to be necessary to perform analyses across transportation modes or across time. Indeed, the Transportation Research Board report, Data for Decisions (National Research Council, 1992a), empha- sized the problems with cross-modal comparisons. BTS has highlighted some issues of cross-modal comparability in its Trans- portation Statistics Annual Reports (see Box 3-1~. However, our review of its programs and data products and services indicates that BTS to date has not fo- cused sufficient attention on helping users understand the problems of available transportation data nor on developing a systematic program to evaluate and im- prove the quality of transportation data (see sections below on "Documentation" and "Data Evaluation and Improvement;" see also Appendixes D and E). Such a program will require not only that BTS address data programs that it operates directly, but also that it work collaboratively with statistical units in other modal administrations in USDOT and with other data providers to ensure a focus on the quality of transportation data. Given limited budgets, it is rarely possible to improve the quality of a data system on all dimensions nor to improve both quality and relevance to the same extent: statistical agencies are commonly faced with making choices or trade- offs among them. For example, without increasing total costs, it may be difficult to increase the sample size of a survey in order to reduce sampling variability and at the same time devote efforts to reduce undercoverage of people or establish- ments. Similarly, it may be difficult to increase sample size and at the same time maintain the desired frequency of data collection (an aspect of relevance). An important function of a statistical agency is to make these trade-offs on the basis of the best assessment possible of currently available data and the likely payoffs to investments in them. Looking to the future, the increasing capabilities of computer-assisted data collection and the combined uses of administrative and survey data may make it feasible in some instances to achieve simultaneous cost and burden reductions and improvements in data quality and relevance. Statisti- cal agencies need to keep abreast of these developments and integrate them into their work. A prerequisite for BTS to undertake these activities is that it build strong statistical and analytical capabilities in its staff.

36 BUREAU OF TRANSPORTATION STATISTICS: PRIORITIES FOR THE FUTURE STAFFING The 1991 ISTEA envisions BTS as a statistical agency for USDOT that pro- vides leadership to the department in such areas as developing data quality stan- dards, working with the other modal administrations and the states to develop indicators of the transportation system, coordinating collection of transportation data by USDOT with other federal agencies, and improving the quality and rel- evance of transportation data for cross-modal, system-wide analysis. To take on these leadership roles, and in particular to enhance the department's focus on data quality, BTS must have adequate technical and analytical expertise on its staff. Such expertise is essential for BTS to document, evaluate, and improve its own data systems and data products. In turn, excellence in its own operations is a prerequisite for the agency to acquire the stature and moral authority that are necessary for BTS to become credible in a leadership role for the department as a whole. At present, BTSis a small agency; it lacks the depth and breadth of statistical and methodological expertise on its staff to coordinate a comprehensive program for USDOT of documenting, evaluating, and improving the department's data or, more generally, to provide statistical advice to other units.4 A key element in BTS's future evolution as a statistical agency will be its ability to develop the necessary capabilities on its staff. BTS's progress in this regard will become even more important to the extent that budget pressures on the other modal ad- ministrations in USDOT constrain their ability to maintain statistical and analyti- cal expertise in their agencies. Current and Planned Staffing The current BTS staff (37 people as of fall 1996, of which 16 were in the Office of Airline Information) have expertise and experience in a wide range of fields. A number of staff have backgrounds in transportation research, geographic information systems, data technology, and other relevant fields. However, few staff have extensive expertise in statistical methods (e.g., sampling, estimation, survey research and evaluation). Plans to fill 23 authorized vacancies include two mid-level positions for a mathematical statistician and a survey statistician. (These positions were recently filled.) 4As an example of an area in which BTS could provide technical assistance if it had sufficient staff capability, the Federal Aviation Administration was recently faced with the question of whether it could rank the safety records of individual air carriers. This question involves such statistical issues as the appropriate choice of safety indicators (e.g., number of accidents with fatalities per million aircraft-miles flown, number of accidents with fatalities per 100,000 takeoffs, number of fatalities per million passenger-miles flown) and whether differences among those indicators for individual airlines are meaningful (i.e., relate to factors that are under the airlines' control, such as age and maintenance of equipment, versus such factors as weather patterns at the airports used most by an airline).

FOCUSING ON DATA QUALITY 37 An earlier staffing plan, which envisioned a total of about 75 staff after fill- ing vacancies, included a position of senior adviser for statistical policy in the BTS director's office. BTS was directed by the Office of the Secretary to reduce its authorized full-time-equivalent (FTE) staff from 75 to 60 people. In revising its staffing plan, BTS chose to drop the senior statistical adviser position, which was slotted for a GS-15 level (the top grade below the Senior Executive Service), and to share statistical oversight responsibilities among the director, deputy di- rector, and two associate directors. (The director is a presidential appointee; the deputy director and associate directors are Senior Executive Service positions.) The reason given for this decision, which BTS hopes to reverse if it is authorized to have more FTE staff, is that it is difficult to justify positions at the GS-15 level, particularly under strictures from the current administration, as part of its Rein- venting Government initiative, to reduce the ratio of senior supervisors to other staff. Building a Strong Statistical Staff In the panel's view, it is essential for BTS to implement a staffing plan that gives much higher priority to building expertise in statistical methods and related quantitative fields than is provided in the current staffing plan. Needed areas of skill include statistical sampling, statistical design, cognitive foundations of sur- vey measurement, advanced data collection methods, editing, imputation for miss- ing data, and statistical estimation from complex sample surveys. At present, BTS's statistical staff capabilities are augmented by Census Bureau staff who work on the Commodity Flow Survey and the American Travel Survey. How- ever, there is no substitute for sufficient in-house staff with the necessary exper- tise if BTS is to achieve excellence in its own operations and if it is to be able to exercise statistical leadership for the department as a whole.5 BTS should repro- gram a portion of the available vacancies to emphasize statistical and related skills and should move expeditiously to fill those vacancies. To underscore the importance of a strong in-house statistical staff for BTS and to provide a focal point for BTS's work to evaluate and improve the quality of transportation data, the panel believes that BTS should be authorized by the department to appoint an associate director for statistical methods and research at the Senior Executive Service level (see recommendation 2 at the end of the chap- ter). The senior level is justified given the importance for a statistical agency of 5Many statistical agencies, like BTS, use contractor staff for a variety of purposes, including data collection and processing, programming support for analytical work, conference arrangements, and publication preparation. We did not consider in detail the appropriate mix of in-house and contractor staff for a statistical agency many factors enter into the choice of mix, including costs, constraints on full-time-equivalent staff, and past agency experience. However, we stress that a statistical agency must have sufficient in-house statistical and technical capability to carry out key functions and prop- erly direct the work of contractors.

38 BUREAU OF TRANSPORTATION STATISTICS: PRIORITIES FOR THE FUTURE keeping abreast of and applying advanced statistical methods and techniques to such functions as data evaluation and improvement. Although the titles vary, other major statistical agencies, such as the Census Bureau, the National Center for Education Statistics, and the National Center for Health Statistics, have simi- lar positions. The BTS associate director for statistical methods and research should have extensive expertise in such areas as statistical estimation and survey research methods. BTS should authorize the associate director to build a statistical staff that plays a leadership role for BTS in developing data quality standards, design- ing and implementing evaluation studies of BTS data systems, and conducting research on improved methods of data collection, processing, and estimation. The BTS statistical staff would take the lead in working with statistical units in the other USDOT modal administrations to develop standards and priorities for data documentation, evaluation, and improvement of the department's data sys- tems. The BTS statistical staff would also provide technical assistance to the other modal administrations as appropriate. The associate director for statistical methods and research and BTS as a whole could benefit from outside statistical advice on a regular basis. As required by the 1991 ISTEA, BTS currently has an Advisory Council on Transportation Sta- tistics, which meets twice a year to discuss BTS's programs and review new initiatives. This group has a strong user orientation and focuses on issues regard- ing the kinds of transportation data that are needed for important policy purposes. A separate advisory group that focuses on issues of statistical methods and stan- dards would also be very useful. As some other statistical agencies have done (e.g., the Bureau of Justice Sta- tistics, the Census Bureau, the Energy Information Administration), BTS could ask the American Statistical Association to establish a working group of experts to meet regularly with its statistical staff on technical matters. The members of such a group should have expertise in such areas as sampling and survey design, advanced data collection methods, weighting and imputation methods for miss- ing and erroneous data, and statistical estimation from complex sample surveys. As BTS develops closer working relationships with the other modal administra- tions, many of which have large amounts of data collected from administrative reporting systems in addition to sample surveys, a statistical advisory group should also include experts in the design and statistical applications of adminis- trative records. Such a group could assist BTS to evaluate alternative designs and data collection, processing, and analysis procedures for surveys and other data col- lection programs and to establish priorities for statistical research and evaluation. Continuing Staff Development Building and maintaining strong statistical and technical staff capabilities requires an agency's continuing attention. BTS's top management should give

FOCUSING ON DATA QUALITY 39 priority to identifying opportunities for staff development and to encouraging staff to take full advantage of them. Among the kinds of activities that can foster the development of technical skills are attendance at advanced courses, presen- tations at professional association meetings, and publication in professional jour- nals. BTS already has some useful vehicles for staff development in place. Its new peer-reviewed publication, the Journal of Transportation and Statistics, should prove to be a valuable means by which to stimulate methodological re- search and analysis on the part of staff. Also, its regular seminar series, which brings leading transportation researchers from the United States and abroad to present analytical results and discuss important issues for transportation policy, is an important means of enhancing professional knowledge and skills. This pro- gram should be continued and expanded to include relevant issues of statistical methods and approach. Some statistical agencies have specified goals for the performance of tech- nical staff that relate to keeping current in the technical developments of their field. These goals can be achieved by participation in relevant graduate courses at local universities, attendance at continuing education short courses, or atten- dance at other seminars that are relevant to the field. There are several active programs in the Washington, D.C., area that provide opportunities for profes- sional development of these kinds. By placing explicit direction in performance plans for continuous improvement of technical skills, the agency can make ex- plicit its commitment to this goal. Some other ways to enhance professional capabilities are more costly and hence may be appropriate for BTS to consider only when it is somewhat larger and has more resources. For example, several statistical agencies have visiting fellows programs that are administered through the American Statistical Associa- tion, in which distinguished statisticians and other researchers come to the agency for a specified time period to work on topics of mutual interest. The visiting fellows gain insights into the practical operational problems of a statistical agency, and the agency staff benefit from working closely with leading researchers. Such a program involves significant budget commitments and can also take time to become established. It may be difficult for a small agency such as BTS to accom- plish, but the concept is worth investigating for possible implementation at a future date. Similarly, statistical agencies sometimes provide their staff with opportuni- ties to work at other agencies or organizations for periods of 6 to 18 months in areas that will benefit the home agency. When BTS is larger, it could consider occasionally detailing one or two people to another statistical unit within USDOT, to another federal statistical agency, or to another organization with statistical expertise, as a way for staff to gain valuable experiences and insights. Similarly, BTS could sponsor staff from other USDOT modal administrations or other fed- eral statistical agencies to work at BTS. Exchanges of staff between BTS and

40 BUREAU OF TRANSPORTATION STATISTICS: PRIORITIES FOR THE FUTURE other statistical units within USDOT could be particularly valuable in building cooperative relationships, cross-modal perspectives, and a strong commitment to data quality within the department. QUALITY STANDARDS For documenting, evaluating, and improving data quality, it is very helpful for a statistical agency to develop explicit notions of appropriate standards for collecting, processing, and publishing the data. BTS has been working with other USDOT modal administrations to develop improved means of data dissemina- tion, such as more user-friendly CD-ROM formats, which is a step toward facili- tating systematic data analysis. BTS has not yet begun to work with the other modal administrations to develop guidelines for data quality throughout USDOT, as it is mandated to do by the 1991 ISTEA, nor to standardize key concepts, definitions, and procedures to the extent feasible and appropriate in order to fa- cilitate cross-modal analysis. In some people's interpretations, BTS is constrained from moving in this direction by the provision in the 1991 ISTEA that nothing in the legislation shall be construed "~1) to authorize the Bureau to require any other department or agency to collect data; or (2) to reduce the authority of any other officer of the Department of Transportation to collect and disseminate data independently." However, our view is that this provision does not contradict the mandate for BTS to develop guidelines for data quality for USDOT in collaboration with statistical units in the other modal administrations. Indeed, we urge that the reauthorization of BTS strengthen its role by requir- ing it to develop data quality standards, consistent with good statistical practice, that are binding throughout USDOT and available for use by transportation agen- cies outside USDOT and for reference by the public (see recommendation 3 at the end of the chapter). In so doing, Congress will both underscore the importance of focusing on the quality of transportation data and clarify BTS's responsibility to move forward in this area. BTS should develop data quality standards for the department with the coop- eration and input of the other statistical units in USDOT obtained through a de- partment-wide standards committee that is chaired by the BTS director. Coop- erative efforts are essential, so that the other units can come to see the benefits to their users and buy into the process and so that BTS can carry out its leadership function in this area as a facilitator and not as a regulator or enforcer. The stan- dards committee should be mandated in the reauthorization of BTS. The reauthorization should also require that BTS every 2 years prepare a report to the Congress that describes progress during the previous 2 years to set standards and that identifies improvements in data quality by BTS and other USDOT statistical units and in the provision of information about quality to data users. We recommend the biennial report primarily as a tool to promote a focus

FOCUSING ON DATA QUALITY 41 on data quality; it could also usefully describe major steps to improve the rel- evance of transportation data in terms of timeliness, subject matter and geographic detail, appropriateness of concepts and definitions, and the initiation of new data programs or the consolidation or elimination of data programs in order to satisfy users' priority needs more cost-effectively. We specify a biennial rather than annual report so that there is time for progress to be made and for the report to be a substantive document and not simply a time-consuming exercise in fulfilling a requirement. Another way to ensure substance is for each report to identify selected data programs or subject areas in which quality improvements will be sought on a priority basis and to highlight those areas in the next report, commenting more generally on other areas. The biennial report that we recommend is not to be confused with the Trans- portation Statistics Annual Reports that are mandated by the 1991 ISTEA. Those reports have regularly included a section on the state of transportation statistics, but those sections have been general in nature and do not meet the need we see for reporting improvements on specific quality dimensions for specific transporta- tion data programs or sets of related programs. We discuss the role of the Trans- portation Statistics Annual Reports in providing needed time series indicators and analyses of transportation trends in Chapter 4, where we suggest that there may be more cost-effective ways of providing these kinds of information than the current prescribed format. We recognize that BTS is still a new, small agency within the U.S. Depart- ment of Transportation with a challenging array of responsibilities. Also, as dis- cussed earlier, BTS currently lacks the staff resources and the necessary technical capabilities and expertise with which to develop its statistical functions as fully as its data compilation and dissemination functions. Hence, it will not be an easy task for BTS to assume responsibility for leading a process to develop quality standards for USDOT as a whole. However, we believe strongly that BTS must evolve to be the statistical agency for USDOT that is envisioned in the 1991 ISTEA, which means that it must begin to take on a leadership role in several areas. The need for leadership to sort out higher- from lower-quality data and to identify priorities for new and improved data is clear from reviews of transportation data needs (e.g., National Research Council, 1992a). Such reviews invariably cite the large volume of data available from public and private sources but the lack of comparable data that provide useful information for analyses of important transportation issues, par- ticularly those that require a cross-modal or system-wide perspective. We recognize that progress in such areas as developing quality standards cannot happen overnight. Nonetheless, the work must begin, and the agency that was established to be the major statistical unit for the department as a whole must be given the authority and motivation to move forward collaboratively with the other statistical units in USDOT.

42 BUREAU OF TRANSPORTATION STATISTICS: PRIORITIES FOR THE FUTURE Types of Standards The term "quality standards" can take on several different meanings, as dis- cussed below. Statistical agencies may find it useful to develop standards that reflect more than one of these interpretations. Also, standards will usually apply to a range of activities, including data system design and development, data col- lection, data processing, and publication. Standards as consistent definitions and protocols In this interpretation, stan- dards setting involves the development of consistent definitions of key concepts and variables in order to permit comparisons and statistical aggregation. Ex- amples are standard industrial and occupational classifications for the reporting of economic data, standard accounting concepts and fiscal years for the reporting of financial data by governmental units and business enterprises, and base years for indices. The importance of work on standard definitions for transportation concepts to permit cross-modal analysis is clear. Indeed, to date, this is the single area of standards setting that BTS has considered for its future agenda, although there are other equally important areas. BTS has taken the very first step in this area (in its publication, Transportation Expressions) by documenting the various definitions used in transportation data systems for such concepts as "semitrailer" and "fatal- ity." It has also addressed in general terms some of the problems for data use caused by the lack of common definitions in some areas (in its Transportation Statistics Annual Reports see Box 3-1 for examples). However, much more needs to be done to evaluate for users the implications of different definitions for cross-modal analyses and then to work to standardize key definitions.6 Standards as definitions of minimum acceptable quality In this interpretation, standards serve as performance criteria for data collection and publication. For example, for a household or business survey, a statistical agency may set a stan- dard for a minimally acceptable final response rate from the sampled units, such as 75 or 85 percent, and set aside funds to be used for additional follow-up efforts if the initial data collection efforts fall short of obtaining the specified standard.7 Many statistical agencies have minimum publication standards for the reporting of survey estimates: for example, differences across time or population groups will not be reported in summaries of findings if they fall below specified criteria 6Striving for comparability of key concepts and definitions must be undertaken carefully. In some instances, comparability may not be feasible, except by moving to a least common denominator in which importance nuances are blurred or lost. 7For example, one agency's standards manual (Energy Information Administration, no date) speci- fies a minimum final response rate of 75 percent of eligible respondents, covering 85 percent of anticipated aggregates (e.g., total sales volume for regions). Determining an appropriate response rate standard also requires defining the term (who is an "eligible" respondent, whether the calculation is made using weights, etc.).

FOCUSING ON DATA QUALITY 43 for statistical significance; estimates will not be published if they are based on fewer than a specified number of reporting units. Another minimum acceptable quality standard may involve time between completion of data collection and publication: for example, the Bureau of Labor Statistics and the Bureau of the Census commit to completing data collection and publication of the monthly un- employment statistics within a few weeks of the reference week for the estimates. The use of minimum acceptable quality standards is desirable when there is strong evidence linking the standard to the utility of the data. For example, the survey research literature provides substantial evidence that survey non- respondents are likely to differ from respondents in important ways for which editing and imputation are not likely to compensate (see, e.g., Jabine, King, and Petroni, 1990~. Hence, there is justification for establishing a high standard for a minimally acceptable final response rate to a survey in order to minimize bias from nonresponse. As another example, the suppression of publication of esti- mates that do not meet minimum precision thresholds simplifies use of statistical publications. Readers are assured that all estimates presented meet a specified level of reliability. Many USDOT data systems are based on administrative records that repre- sent censuses of the relevant reporting units and not surveys, so that minimum publication standards involving statistical confidence levels or sample sizes are not applicable. (Examples are the Fatal Accident Reporting System of the Na- tional Highway Traffic Safety Administration, the National Bridge Inventory of the Federal Highway Administration, various administrative databases of the Fed- eral Aviation Administration, and operational and financial data on certificated U.S. air carriers of the Office of Airline Information in BTS.~8 However, there can be reporting problems in administrative data systems (e.g., failure to report selected items or to provide any information at all, errors in reporting due to data transmission problems or the use of nonstandard definitions) that may, in some instances, merit the development of a minimum acceptable standard below which data will not be released. Standards as protocols to reveal indicators of the quality of published statistical information In this use of standards, there is a commitment to identify key indi- cators of data quality and to publish them as a matter of standard practice in order to inform data users about limitations and problems in the data. (Agencies with this type of standard may or may not also establish minimum acceptable quality Some data programs in USDOT represent samples of administrative records for which statistical sampling considerations apply (e.g., the Carload Waybill sample of information provided by Class I freight railroads for a 1 percent sample of rail waybills, which the Federal Railroad Administration uses to analyze traffic patterns and competitiveness issues; the General Estimates System of the Na- tional Highway Traffic Safety Administration, which contains information on a sample of police- reported traffic crashes; and the Passenger Origin and Destination Survey of the Office of Airline Information in BTS, which contains information from a 10 percent sample of airline tickets).

44 BUREAU OF TRANSPORTATION STATISTICS: PRIORITIES FOR THE FUTURE standards.) Thus, for surveys, agencies may commit to publishing such quality indicators as sampling variability, response rates, missing data rates, response variance indicators, and comparisons to other similar data series. For administra- tive records-based data systems, agencies may commit to publishing such quality indicators as missing data rates and to describe differences in reporting practices across reporting units (e.g., differences in fiscal years for state or local govern- ment financial reports of highway revenues and expenditures). Standards as methods of quality improvement In this interpretation, agencies use a set of quality indicators, such as those developed for publication (e.g., re- sponse rates, item nonreporting rates), as the basis for setting and tracking data improvement goals. For example, an agency might set a goal of reducing non- response rates by a specified amount for key survey items by experimenting with questionnaire design and question wording. As another example, an agency might set a goal that, over a specified time period, all reporting units for an administra- tive records data system, such as state and local governments, will convert to common definitions of key concepts or to common practices for data reporting. Standards as hortatory statements of practice In this interpretation, agencies issue guidelines or statements of best practice on dimensions of quality (e.g., timeliness, low variability) and seek to nurture aspirations to those practices. However, they do not attempt to enforce minimum acceptable standards. Standards-Setting Practices The utility of standards is that they are tools to achieve data quality; all of the alternative kinds of standards described above can play a role in achieving high- quality data. A new statistical agency faces unusual problems in setting standards and striving for quality. It may easily fail if it merely adopts the practices of mature agencies. BTS has yet to develop a culture that places prime importance on the con- tinuous improvement of data quality. The agency can, however, shape its culture in that direction. In an agency that is attempting to build a culture of commitment to quality improvement, the construction of formal written standards for the pub- lication of estimates and for minimal acceptable data quality can act as a catalyst to communicate to wide audiences the importance of data quality to the mission of the agency. Written standards can thus serve both to define an internal spirit in this direction and to define the image of the organization to the larger world. Established statistical agencies vary in the types of standards they have de- veloped and in how they achieve compliance with quality standards (see U.S. Department of Education, 1988, which reviews the practices of the Bureau of Justice Statistics, the Bureau of Labor Statistics, the Census Bureau, the Energy Information Administration, and the National Center for Health Statistics in the areas of standards setting, quality control, and tabulation and publication review).

FOCUSING ON DATA QUALITY 45 In many long-standing statistical agencies, there are no written standards; the agency believes that the existing organizational culture enforces adherence to a high level of professionalism in carrying out data collection and analysis pro- grams. Newer statistical agencies that contract for data collection with outside organizations more often have written standards (see, e.g., Energy Information Administration, no date; for other examples of written standards, see Flemming, 1992; Freedman et al., 1987; Sirken et al., 1992~. In some agencies, there are units with review authority for quality standards: these units must review tabula- tions and analyses before they are released, with the possibility that the work must be redone if minimum acceptable standards are not met. There are advantages and disadvantages to each of these practices of achiev- ing high standards of data quality. On one hand, written standards can be heavy- handed and make it difficult for agencies to experiment with new methods for data collection, processing, and publication. On the other hand, the absence of written standards means that agencies must have very well-developed systems for training, mentoring, and evaluating their own and contractor staff. In the early years of a statistical agency, it may be necessary to construct written stan- dards in order to develop, at a later stage, an organizational culture that inherently promotes data quality and relevance. Considerations for BTS BTS will need to think through appropriate uses and meanings of quality standards. For its own use, we suggest that BTS develop minimum acceptable quality standards for data from its survey and other data collection programs, commit to publishing specific quality indicators and other kinds of documenta- tion for those data, and plan to use these indicators to guide continuing efforts toward data improvement. For some programs, it may be the case that not enough is known to publish certain kinds of quality indicators (e.g., indicators of various reporting errors). In those instances, it will be important to identify priority areas for evaluation studies that can provide input for more complete documentation and suggest subsequent work to improve data quality. At the same time that BTS is developing its own quality standards, it should be working with other statistical units in USDOT as recommended earlier to de- velop quality standards for the department as a whole. It will clearly be important to work on standardizing definitions and other aspects of data systems for the department, to the extent feasible and appropriate, that can facilitate cross-modal and system-wide analyses of transportation data sets on a comparable basis. Such work will be challenging and will require identification of priority areas to ad- dress, given the large number of transportation issue areas and data systems. There may also be some minimum acceptable quality standards that are ap- propriate to develop for the department, such as pretesting requirements for new survey instruments and reporting forms. However, we do not suggest focusing

46 BUREAU OF TRANSPORTATION STATISTICS: PRIORITIES FOR THE FUTURE on the development of minimum acceptable quality standards, both because of the wide range of transportation data programs and because of the importance of nurturing collaborative and not adversarial relationships of BTS with the other modal administrations. What seems to us feasible and desirable to develop is a set of quality indica- tors and other information that BTS and all of the USDOT modal administrations commit to publish about their data in statistical reports and documentation of data sets that is, not standards for the data themselves but publication standards that inform users about data limitations and, over time, serve to guide the develop- ment of improved data. The minimum acceptable set of quality indicators in reports and documentation will vary by type of data system and type of report. For example, estimates of sampling variability are essential to provide for esti- mates that derive from a survey, but they do not apply for estimates from a census (although there may be other sources of variation that should be documented). Also, it will generally be appropriate to publish fewer quality indicators in brief summaries or abstracts of data systems than in full-blown reports that present detailed tables and analyses or in documentation of data sets; however, even the briefest summary or abstract should include basic quality indicators (see section on "Documenting Data Quality" below; see also Flemming, 19921. In order to carry out a strengthened mandate to establish binding data quality standards for USDOT (whether publication standards, consistent definitions, or minimum acceptable quality standards), BTS in collaboration with the depart- ment-wide standards committee recommended earlier will need to develop ex- plicit written standards in most instances. However, we caution against rigidify- ing standards or setting up an office within BTS that is viewed as having a police function. BTS and the other statistical units in USDOT should work together to develop standards and periodically review and revise them to keep the standards relevant to new methodology for data collection, processing, and publication and to changing transportation data needs. BTS and the other statistical units should also collaborate to prepare the biennial report recommended earlier that describes progress in documenting and improving the quality of transportation data in USDOT. Such a report can be much more than an exercise in meeting a legisla- tive requirement. If well done, it can identify priority areas for data improvement and generally contribute to an ongoing quality assurance function for the depart- ment's data systems. DOCUMENTING DATA QUALITY Setting standards is an important and challenging function of a statistical agency but, to be useful, the standards must be applied in documenting, evaluat- ing, and improving the quality of the data in the agency's subject area. Statistical agencies face daunting tasks in these areas. The question for BTS is where to begin.

FOCUSING ON DATA QUALITY 47 To date, BTS has concentrated on letting users know about the vast array of existing transportation data sources and making it convenient for them to obtain data through its World Wide Web site and other modes of dissemination. With this goal in mind, BTS has not attempted any type of screening or gatekeeping of the data it assembles for redistribution, nor has it made an effort to distinguish the quality or usefulness of particular data sources. It has also not yet begun to evaluate available data systematically nor to lay out a program of improvement of key transportation data sets. The result is that users now have access to a large volume of information of varying quality with no roadmap to assist them in un- derstanding the limitations and appropriate uses of particular data sets. We believe that BTS should now begin to focus more on data quality than on quantity. It should place high priority on the development of more complete information for users about the methods of data collection, error measurement, definitional comparability across data sets, and other dimensions of quality of the transportation data that it makes available (see recommendation 4 at the end of the chapter). Because of the importance of cross-modal analysis and because of BTS's mandate in this area, information provided to users should include how data for one transportation mode relate (or do not relate) to data for other modes. Such efforts at documentation will identify aspects of data systems about which little is known and for which evaluation is needed and help set priorities for evalu- ation studies that can ultimately lead to initiatives for data improvement. Simply to expand the available documentation will require setting priorities and making choices. BTS should begin by ensuring that documentary materials for the data systems that BTS itself sponsors are complete and meet high stan- dards, as discussed above. It should then identify topic areas that are of particular policy importance and work with relevant agencies inside and outside USDOT to develop the most appropriate documentation. (For data systems for which good documentation already exists, BTS should highlight the appropriate references on its web site and in its directory of transportation data sources and other relevant publications.) Below we discuss documentation concerns for BTS's own data systems and then give examples of improvements that BTS could make in the short term to its key publications and web site to help users understand the limita- tions and uses of other available data sets. Documentation of BTS Data BTS currently sponsors two major surveys about intermodal transport of people and goods the Commodity Flow Survey (CFS) and the American Travel Survey (ATS). The Census Bureau cosponsors and collects the CFS data as part of the economic census program conducted every 5 years; it also collects the ATS data under contract from BTS. (Current plans are to conduct the ATS every 5 years as well.) The Census Bureau has released a series of reports from the CFS (see, e.g.,

48 BUREAU OF TRANSPORTATION STATISTICS: PRIORITIES FOR THE FUTURE Bureau of the Census, 1996a), which include information about the data, reflect- ing the Census Bureau's long-standing practice and standards in this area. Each report provides a description of the survey and the data collection and estimation methods used, definitions of variables, assessments of comparability with previ- ous surveys and data reliability (including estimates of sampling variability), and a copy of the questionnaire. Although extensive, the documentation does not answer some important questions: for example, there is no information on nonresponse rates by shippers to the survey. Also, the documentation does not address analytical uses of the data that are appropriate given what is known about the data quality. BTS could usefully develop examples of appropriate applications of the CFS data for system-wide transportation policy analysis, including examples of analy- sis of trends over time from comparisons with earlier rounds of the CFS, to the extent feasible. Materials that guide the states in appropriate use of the CFS data could also be very helpful (see discussion of technical assistance to the states in Chapter 4~. Developing such materials will require that BTS staff themselves become expert users of the CFS data, which, in turn, is one of the best ways for a statistical agency to evaluate the quality of a data set and to determine needed improvements in both data and documentation. Assuming that future rounds of the CFS continue to be cosponsored with the Census Bureau, BTS should become an active partner in planning and reviewing the accompanying informational materials. Data from the 1995 ATS are not yet available. However, in contrast to the CFS, the ATS reports that will be released shortly will be BTS publications and not Census Bureau publications. BTS should give careful attention to the type and extent of documentation that is provided with the reports and with computer- readable data products from the ATS. Important information to include is a dis- cussion of comparability of the 1995 ATS and the 1995 and earlier rounds of the Nationwide Personal Transportation Survey (NPTS) and of how the two data sets can be used together for analysis purposes. (The 1995 ATS covered trips of 75 or more miles by a sample of 80,000 households over the course of a year; the 1995 NPTS covered a day's worth of trips together with longer trips over a two-week period for a considerably smaller sample of 22,000 households.) Microdata will be available from the ATS; microdata are also available from the NPTS.9 Complex microdata products require extensive documentation so that users can analyze the data with full understanding of the meaning of the variables and structure of the data file. Such documentation should include not only a codebook, which provides essential information on locations and codes of 9The ATS and NPTS microdata files protect the confidentiality of responses from individual people and households by several methods, such as coding place of residence to broad geographic areas. Protecting the confidentiality of business respondents is more difficult (e.g., because of significant variation in such characteristics as size); hence, microdata files are not available from the CFS.

FOCUSING ON DATA QUALITY 49 variables, but also a user's guide, which typically includes information on the survey design, the structure of the microdata set (e.g., if there are multiple types of records), limitations of the data and cautions for analysts, detailed definitions of variables, how to construct estimates of sampling error, comparability with related data sets, and the like (see, e.g., Bureau of the Census, 1991, 1992~. Finally, it is important to document the results of evaluation studies of com- plex, ongoing data collection systems, such as the ATS and CFS, in a way that highlights their implications for appropriate use of the data and that identifies areas for future improvement. One approach is to develop and periodically up- date a quality profile, which brings together all that is known about the sources and extent of error nonsampling error as well as sampling error that may af- fect the estimates from a survey or other data system (see, e.g., Energy Informa- tion Administration, 1996; Jabine, 1994; Jabine, King, and Petroni, 1990~. Another approach is to put out a methods bulletin every 2-3 years with chap- ters on all of the data collection programs in an agency, reviewing for each the survey design, collection and processing methods, and whatever is known about the error and quality of the estimates (see, e.g., Bureau of Labor Statistics, 1992~. A methods bulletin is readily updated individual chapters can be expanded as more is learned about particular data programs. The Bureau of Economic Analy- sis has begun to develop a methods bulletin for its estimates with a series of articles in the Survey of Current Business that will be combined into a single document. In addition to developing quality profiles or chapters in a methods bulletin series for the ATS and CFS, BTS should immediately begin to use its planned Journal of Transportation and Statistics as an outlet for publishing methodologi- cal papers about its surveys (subject to peer review). BTS should also encourage staff of statistical units in other USDOT modal administrations to publish meth- odological papers about their own data systems in the journal and should investi- gate the possibility of joint articles with staff from other statistical agencies on issues of mutual interest. Documentation of Other Data BTS's work with the other USDOT modal administrations to develop stan- dards for data documentation will ultimately lead to more consistent and com- plete information for users about the quality of the department's data systems and how they can be used for cross-modal, system-wide analyses. At this stage, BTS must necessarily accept the documentation that other agencies provide for data that they furnish to BTS to disseminate in statistical compendia, CD-ROMs, and via the BTS web site. Yet there are modest steps that BTS can take now to emphasize for transportation data producers and users the importance of focusing on data quality.

50 BUREAU OF TRANSPORTATION STATISTICS: PRIORITIES FOR THE FUTURE Directory of Transportation Data Sources BTS's directory is a helpful basic reference document for users to learn about available transportation data. The number of entries has approximately doubled from the first edition in December 1993 to the 1996 edition (see Table 2-1), and the directory now covers data sets from USDOT agencies, other federal agencies, the United Nations, state governments, private organizations, Canada, and Mexico. The information provided for each data set (report, CD-ROM, data tape) includes: title; mode of transportation; brief abstract; source of data; geographic coverage; time span, when first developed, update frequency, last update; file attributes if applicable (e.g., number of records); significant features or limitations; corresponding print source; sponsoring and performing organizational); and availability and contact for additional information. Several additions would enhance the ability of the directory to focus users on data quality. For computer-readable data sets, BTS should add references to available documentation. For surveys, the abstract should provide not only the sample size, but also the response rate, both of which are important and easily conveyed indicators of quality. BTS should also increase the number of entries for which significant features or limitations are provided (many entries lack any information under this heading) and consider how to provide information on the suitability (or lack of suitability) of a data system for cross-modal analysis. Fi- nally, in addition to a title index and an index by transportation mode, it would help users who want to find relevant data on a particular topic, such as safety, for the directory to include a subject index. National Transportation Statistics Compendium The annual National Transportation Statistics (NTS) reports are intended to serve the same reference function for transportation as the annual Statistical Ab- stract of the United States does for a wide range of subject areas that is, to bring together a large number of data series in a single, regularly updated volume. The NTS reports provide historical trend data for all of the transportation modes on performance, safety, costs, energy use, and other topics, compiled from USDOT agencies and other sources. The 1996 volume includes 134 tables and 42 charts. (As in the Statistical Abstract of the United States, there is no analytical text in the NTS reports.)

FOCUSING ON DATA QUALITY 51 However, the usefulness of the NTS reports is compromised by the lack of detailed explanatory notes, including those that would indicate significant changes in definitions across time and the implications of those changes for data compara- bility. Also lacking are explanatory materials that would help users understand the extent to which it is appropriate to compare data series on particular topics across transportation modes. The Statistical Abstract of the United States, which includes many topics besides transportation, provides information on sampling and nonsampling errors for major data sources that is not found in the NTS reports, along with more extensive table notes for transportation data series than are found in the NTS reports. As an example, the Statistical Abstract of the United States (Bureau of the Census, 1996b:614) indicates the changes in the definition of Class I railroads since 1950 that were adopted by the Interstate Commerce Commission for regu- latory purposes. The NTS provides the current definition, which is that a Class I railroad is one that has $250 million or more in operating revenues in 1992 dol- lars. However, it does not indicate historical changes: in 1950, Class I railroads were those with $4.5 million in operating revenues (1992 dollars); the threshold was revised 6 times in real dollar terms between 1950 and 1982. The 1996 NTS shows a pronounced decline in the number of Class I railroads from 1960 to 1994 and in the numbers of freight cars, employees, and miles of track owned by Class I railroads. Some portion of this decline is undoubtedly real due to consolida- tion of rail companies, loss of business to trucking companies, and other factors. However, some portion of the decline may be an artifact of the definitional changes. Finally, the NTS reports include numerous charts and graphs, many of which are useful in identifying important trends, but some of the charts need to be re- thought in order to satisfy principles of good graphic design (see Cleveland, 1985, 1993; Tufte, 1983~. Furthermore, reducing their number could free up space for material that explains and interprets key data series. In Appendix D, we use the section in the 1996 NTS on airline safety to illustrate some of the kinds of changes that BTS should plan to make over the next few years, topic by topic, to improve the usefulness of the NTS volumes to help users understand the quality of the data and their appropriateness for cross-modal analysis.~° Data Products BTS has released numerous data sets on CD-ROM, most of which were ob- tained from other agencies. One example is a CD-ROM of historical data from the National Highway Traffic Safety Administration's Fatal Accident Reporting System (FARS) and General Estimates System (GES). (FARS provides a census i°The 1997 NTS reflects improvements in tables that anticipate some of the comments in Appendix D; it has no charts or graphs.

52 BUREAU OF TRANSPORTATION STATISTICS: PRIORITIES FOR THE FUTURE of traffic crashes that involve fatalities, and GES provides a probability sample of all police-reported traffic crashes.) Another example is a CD-ROM of data from the 1983 and 1990 Nationwide Personal Transportation Survey, sponsored by the Federal Highway Administration. BTS's Transportation Data Sampler CD- ROMs include selected reports, aggregate data sets, and microdata sets from a variety of sources. Documentation from the source agency is provided for data sets (e.g., the two NPTS files); however, it is not always clear which files contain documentation. For example, documentation for the Census Bureau's 1992 Truck Inventory and Use Survey on Transportation Data Sampler-3 is split among sev- eral files and not clearly identified for the user. Indeed, although the sampler has a brief description of each subdirectory that corresponds to a particular data sys- tem, it does not briefly describe each file within a subdirectory. The user has to hunt to find particular data sets and documentation. Printed material that accompanies each CD-ROM should stress the impor- tance of reviewing the documentation before accessing and using a data set. Also, either that material or an introductory document on the CD-ROM itself should provide a clear index, with brief annotations, to all of the files on the CD-ROM. In particular, the description should note whether the file is documentation or data and, if the latter, whether the file contains microdata for individual reporting units (households, accidents, establishments, etc.) or whether the data are aggre- gated in some manner. Microdata are more useful than aggregate data for de- tailed analyses and research, but they can be more difficult to use. Finally, BTS should begin a program of reviewing documentation that is provided for data sets to determine if it contains minimum essential information and, if not, hold up data release until the needed information is added. For this purpose, BTS should be able to draw on the many existing examples of documen- tation standards (see, e.g., Flemming, 1992; Sirken et al., 1992) to develop a working set of minimum acceptable standards in advance of the final set of stan- dards that is developed for the department as a whole. Web Site The BTS web site (see Figure 2-1) is a vast cornucopia of material, includ- ing: descriptions of BTS data products and services; data from selected reports and files from BTS and other sources; reports, reference documents, and many other publications from a wide range of sources (in the National Transportation Library portion of the site see Figure 2-2~; and links to other agencies, includ- ing the USDOT modal administrations, other federal agencies, and private orga- nizations with some connection to the transportation field. The amount of mate- rial that is accessible through the site is impressive. However, we have several concerns (discussed below) about the site's ability to help users locate high-qual- ity data and understand their uses and limitations. Adding to our concern is the

FOCUSING ON DATA QUALITY 53 impression that the site is being built piecemeal with little thought given to an overall structure that reflects a data user perspective. National Transportation Library (NTL) By far the largest part of the BTS web site is the NTL, which has received praise from reference librarians in the trans- portation field and is well organized to help users find documents on a particular topic. Yet the NTL contains few data and little information about data that is available elsewhere. For example, the safety portion of the site has almost 200 entnes, few of which provide relevant data the documents provided include, among others, reports of the U.S. General Accounting Office on aviation safety issues, the marine safety manual of the U.S. Coast Guard, recommended emer- gency preparedness guidelines for rail transit systems, and bicycle helmet laws by state. These and similar documents may serve a useful reference purpose. How- ever, we question the wisdom of devoting resources and staff attention to expand the NTL if that means fewer resources with which to improve the BTS web site as a guide to users about available data. There is also a problem that the quality of the documents may vary widely, and BTS has no way to control quality. Search capabilities for data The BTS web site can be confusing for the user who wants to find high-quality data (as opposed to reference documents) on a particular cross-modal topic, such as safety. The two sections of the site that provide data or descriptions of data are "Products" and "Databases." (Some BTS products also appear under "BTS Programs." The "Beefing Room" has a "Statis- tics" section, but it is limited to data on airline operations from the Office of Airline Information.) The BTS "Products" section has a subject index in addition to an index by transportation mode; it is also searchable by the user (the search engine actually searches the entire site). However, the entries in the subject index are very gen- eral for example, the NTS reports are listed as sources of safety data without further elaboration (see Figure 3-1~.~3 A keyword search on "safety data" brings ~ iThe BTS web site is updated frequently. Since we first began looking at the site in 1996, BTS has not only added new content, but also improved the organization of the site in several respects; how- ever, more needs to be done. Our comments are based on the site as of April-June 1997. i2The NTL page carries a general disclaimer that inclusion of a document in the NTL "does not necessarily constitute endorsement" by BTS or USDOT (see Figure 2-2). BTS has also recently begun the use of automated software to check for documents in the NTL that are inaccessible because the site of the originating organization has been taken off the web or for another reason. The links for such documents will then be removed or corrected. (These kinds of problems can happen frequently: a review by our panel of the safety section of the NTL prior to the installation of regular checks found that almost two-fifths of the documents were inaccessible because the host server could not be found, the document could not be found on the host server, or the document did not contain information.) i3That the full NTS reports are not yet available on the BTS web site is surprising, given that the site provides the complete text of statistical reports from other modal administrations (e.g., 1995 Highway Statistics from the Federal Highway Administration). Recently, the 1996 Transportation Statistics An- nual Report was made directly accessible through the BTS web site, as were the tables in the 1997 NTS.

54 BUREAU OF TRANSPORTATION STATISTICS: PRIORITIES FOR THE FUTURE ....................................................................................................... - . Subject Listing . .............................................................................................................................................................................................................. Please check News & Updates to see a listing of products as they become available online. Safety Directory of Transportation Data Sources 1996 - You can now add or adjust a Data Source for inclusion in the 1997 edition. National Transportation Statistics (NTS) 1993 National Transportation Statistics (NTS) 1997 - NTS Tables available for download in MS Excel 5.0 format. State and Metropolitan Analysis for Regional Transportation (SMART) - Available through the National Transportation Library on the BTS Homepage. Traffic Safety Data: PARS and GES Transportation Data Sampler-3 Transportation in the United States: A Review Transportation Safety Transportation Statistics Annual Report (TSAR) 1994 Transportation Statistics Annual Report (TSAR) 1995 Transportation Statistics Annual Report (TSAR) 1996 - Available in PDF format. Transportation Statistics: In Brief [Alphabetical Listing] [Media Listing] [Mode Listing] [BTS Products Page] [BTS Services] [Order Form] Feedback? Questions? commentsff~bts.gov FIGURE 3-1 Portion of subject index for "Products" section of BTS web site. up some but not all of the entries under "safety" in the "Products" subject index together with new entries. The entries are annotated; however, the annotations in many instances are not informative about the content of the item listed (e.g., see the listing for "homepage.rtf" in Figure 3-2~. The "Databases" section of the site brings up the National Transportation Data Archive, which contains statistical reports, data sets, and descriptions of data sets, including entries from the Federal Aviation Administration, Federal

FOCUSING ON DATA QUALITY excite for web servers found documents about: safety data 68% about.htm Summary: Welcome to the Office of System Safety's Homepage. Who Are We? 68% index.html 55 Summary: Fatal Accident Reporting System Database. The Traffic Safety Data set was developed by the Bureau of Transportation Statistics and the National Highway Traffic Safety Administration's (NHTSA) National Center for Statistics and Analysis (NCSA) to make traffic safety data easily accessible and widely available. 68% BTS Products - Traffic Safety Data: 1988-1993 Summary: What is it? The Traffic Safety Data: 1988-1993 was produced by the Bureau of Transportation Statistics to make traffic safety data easily accessible and widely available. 66% SMART-SAFETY@BTS.GOVdocFrame.html Summary: *NEW* Emergency Medical Services (EMS) Public Information, Education and Relations (PIER) - National Standard Curriculum (PDF file). *NEW* Public Education Video Clips [Served by Federal Railway Administration]. 66% SMART-SURVEY@BTS.GOVdocFrame.html Summary: *NEW* Sample Transportation Surveys. *NEW* Survey of Motor Carriers in the Rochester Transportation Management Area. 64% BTS Products - Traffic Safety Summary: What is it? The Traffic Safety CD-ROM contains 2 years of statistics from the National Highway Traffic Safety Administration's (NHTSA) Fatal Accident Reporting System (PARS), 1975-1994 and 7 years of statistics from the General Estimates System (GES), 1988-1994 in ASCII format along with its associated technical documentation. 64% homepage.rtf Summa~y:{/rtf 1 /ansi/deff4/deflang 1 033{/fonttbl{/f4/froman/fcharsetO/fprq2TimesNewRoman ;} } {/colortbl ;/redO/g reenO/bl ueO ;/redO/g reenO/bl ue255;/redO/g reen255/bl ue255 ;/redO/ FIGURE 3-2 Results of searching BTS web site for "Safety Data."

56 BUREAU OF TRANSPORTATION STATISTICS: PRIORITIES FOR THE FUTURE green255/blueO;/red255/greenO/blue255;/red255/greenO/blueO;/red255/green255/blueO;/ red255/g reen255/bl ue255 ;/redO/g reenO/bl ue 128 ;/redO/g reen 128/bl ue 128 ;/redO/g reen 128/ blueO;/red128/greenO/blue128;/red128/greenO/blue/snextONormal;} {/*/cs10 /additive Default ParagraphFont;} } {/info{/title About the Aviation Safety Office} {/author FeliceBrunner} {/ operatorFeliceBrunner} {/creatim/yr1996/mo3/dy27/hr14/min41 } {/revtim/yr1996/mo3/dy28/hr11 /min 12} {/printim/ yr1996/mo3/dy28/hr11 /min29} {/version5} {/edmins131 } {/nofpagesO} {/nofwordsO} {/ nofcharsO} {/vern49213} } /widowctrl/ftn bj/aenddoc/noextrasprl/prcolbl//fetO/sectd/l i nexO/ endnhere{/*/pnseclvl1/pnucrm/pnstart1/pnindent720/pnhang{/pntxta.} } {/*/pnseclvl2/pnucltr/ pnstart1 /pnind 64% ts91395k.html Summary: "Drive Smart" Nights at Central Pennsylvania Speedways PENNSYLVANIA PROBLEM IDENTIFICATION Observation of those who frequent professional auto racing events shows that racing fans are more likely than other motorists to drive faster than the speed limit and drive after drinking and are less likely to wear a safety belt. In an effort to communicate directly with this high-risk segment of the driving public, Pennsylvania's Center for Highway Safety Program collaborated with the South Central Pennsylvania Highway Safety Program to establish special safe driving promotions at local speedways throughout 13 counties in Central Pennsylvania. 64% ts91395h.html Summary: 100% Platinum Pacesetter Safety Belt Honor Roll MARYLAND PROBLEM IDEN- TIFICATION During the past several years, Maryland law enforcement agencies received extensive state and national recognition for their promotion of safety belt use. Prior goals and programs established by concerned highway safety groups in Maryland helped move communities towards increased safety belt use rates. 64% SMART-PUBLIC@BTS.GOVdocFrame.html Summary: *NEW* Public Involvement Procedures for New Hampshire Transportation Im- provement Projects.1992 Transportation & Air Quality Planning Guidelines. 64% ts91395i.html Summary: Cornhusker Highway Community/Corridor Traffic Safety Project NEBRASKA PROBLEM IDENTIFICATION Highway 6, also known as Cornhusker Highway, in Lincoln, Nebraska has a high rate of traffic crashes. The road is a high speed arterial with an average daily traffic flow of 32,000 vehicles and a multitude of access points. 64% The National Transportation Safety Section Summary: Safety. Take part in our new Communications Center! 64% Finding the DOT Records You Want Summary: Guide To Finding The DOT Records You Want. FIGURE 3-2 Continued

FOCUSING ON DATA QUALITY ~ 64% BTS Products - Transportation Statistics: In Brief 57 Summary: Transportation Statistics: In Brief. Transportation Statistics: In Brief is a pocket pamphlet designed to highlight two years of transportation data, 1 980 and 1 994. 62% Major Customers Summary: Major Customers. The Office of Airline Information provides the airline financial, traffic and economic data systems that are the critical foundation of DOT's regulatory, advo- cacy and policy decision-making processes. 62% ts91395g.html Summary: Accident Location Analysis System IOWA PROBLEM IDENTIFICATION The Bureau of Transportation Safety at the Iowa Department of Transportation (IDOT) maintains a database of traffic records sent in by investigating officers as well as drivers involved in crashes occurring on public road systems in Iowa. All crashes that result in a fatality, a personal injury or at least $500 property damage are included in the database. 62% tab9-2.txt Summary: TABLE 9.2 AIRLINES (Air Carriers Operating under 14 CFR 121) Accidents, Fatalities, and Rates (Preliminary Data) 1993 Scheduled Nonscheduled Accidents Total 23 0 Fatal 1 0 Fatalities 1 0 Aircraft Hours Flown (000) 1 11,900 624 Departures (000) 1 7,732 312 Accident Rate Per 100,000 Hours Flown Total 0.19 0.00 Fatal 0.01 0.00 Accident Rate Per 100,000 Departures Total 0.30 0.00 Fatal 0.01 0.00 1 Exposure data estimate source: Research and Special Programs Administration and FAA Source: National Transportation Safety ... 62% tab9-2.txt Summary: TABLE 9.2 AIRLINES (Air Carriers Operating under 14 CFR 121) Accidents, Fatalities, and Rates (Preliminary Data) 1993 Scheduled Nonscheduled Accidents Total 23 0 Fatal 1 0 Fatalities 1 0 Aircraft Hours Flown (000) 1 11,900 624 Departures (000) 1 7,732 312 Accident Rate Per 100,000 Hours Flown Total 0.19 0.00 Fatal 0.01 0.00 Accident Rate Per 100,000 Departures Total 0.30 0.00 Fatal 0.01 0.00 1 Exposure data estimate source: Research and Special Programs Administration and FAA Source: National Transportation Safety ... 62% tab9-3.txt Summary: TABLE 9.3 AIRLINES (Air Carriers Operating under 14 CFR 121) Fatal Acci- dents, Fatalities (Preliminary Data) 1993 Location Operator Date Serv. Aircraft Fatalities Total Reported Type On- Type of Total Pass- Crew Others board Accident engers SCHED- ULED SERVICE Chicago, IL Simmons 4/4 Psgr ATR 1 0 0 1 48 Ground Airlines 42-300 crewmember dba: struck American by Eagle propeller NONSCHEDULED SERVICE None Source: National Transportation Safety ... FIGURE 3-2 Continued

58 BUREAU OF TRANSPORTATION STATISTICS: PRIORITIES FOR THE FUTURE 62% tab9-3.txt Summary: TABLE 9.3 AIRLINES (Air Carriers Operating under 14 CFR 121) Fatal Acci- dents, Fatalities (Preliminary Data) 1993 Location Operator Date Serv. Aircraft Fatalities Total Reported Type On- Type of Total Pass- Crew Others board Accident engers SCHED- ULED SERVICE Chicago, IL Simmons 4/4 Psgr ATR 1 0 0 1 48 Ground Airlines 42-300 crewmember dba: struck American by Eagle propeller NONSCHEDULED SERVICE None Source: National Transportation Safety ... [Results by Excite] FIGURE 3-2 Continued Highway Administration, National Highway Traffic Safety Administration, and the Federal Transit Administration, as well as BTS. At present, the archive con- tains 11 listings, of which some are descriptions of data products rather than data (see Figure 3-3~. To help the user locate additional data and information, the BTS site pro- vides links to the web sites of the other USDOT modal administrations (these links are to the main administrations and not to their statistical units). The BTS site also has links to many other organizations (commercial, private, government, nonprofit, libraries) with some relation to transportation. The user can search any and all of these sites for data; however, their design does not always facilitate such a search. The BTS site itself provides no guidance for users in their search of other sites. Such guidance could take the form of putting the Directory of Transportation Data Sources on the BTS site, making its contents searchable by keyword, and, when applicable, adding links to other web sites to obtain more information or to see the actual data. Alternatively, such guidance could be pro- vided through short essays that inform the user of major data series in particular cross-modal topic areas and where to find them. Data documentation The BTS web site gives no evidence of the application of consistent standards for the information provided about the quality and limita- tions of available data. The brief descriptions that are provided in the "Products" section for BTS CD-ROM products vary in content and usefulness (see Appendix E). Each data set listed in the National Transportation Data Archive (see Figure 3-3) has a contents page that links to the following headings: Detailed Descrip- tion, Reports and Products, Searchable Database (operational as yet for only some of the data sets), Questions and Comments, Methods and Limitations, Future Plans, Applications, and Related Topics. This selection of headings appears po- tentially very useful; however, to date, there is limited or no information provided for such key headings as Methods and Limitations for many of the data sets in the National Data Archive. Several of the data sets in the archive reproduce publica

FOCUSING ON DATA QUALITY 59 FIGURE 3-3 Contents of National Transportation Data Archive in "Databases" section of BTS web site. lions from other USDOT modal administrations (e.g., highway statistics); the documentation that is provided in these publications about data collection meth- ods and data quality varies according to the practices of the originating agencies. At this stage of its development, it may not be feasible for BTS to standard- ize the documentation for all of the data sets it makes accessible on its web site from the other modal administrations (or other sources), although such standard- ization should be a goal of the work to develop department-wide quality stan- dards. However, BTS can and should move quickly to standardize documenta- tion for its own data sets and also to standardize and improve the descriptions of its CD-ROM and other products. It should also include on the site a prominent statement to advise users about the importance of understanding the meaning and limitations of available data sets before attempting to work with them. Providing

60 BUREAU OF TRANSPORTATION STATISTICS: PRIORITIES FOR THE FUTURE contact names whenever possible for users to learn not only about the content and scope of a data set, but also about its uses and limitations, would be helpful. DATA EVALUATION AND IMPROVEMENT Documentation of data quality and relevance is essential for users; however, such documentation is necessarily limited to the extent that a data set has not been evaluated on a range of dimensions. Major responsibilities of statistical agencies are to evaluate their data and, on the basis of such evaluations, to inform users of current limitations of the data and to develop ongoing programs to improve data quality (see National Research Council, 1992b, and Appendix C). To support continuing data improvement, agencies need to conduct statistical research on data collection, processing, and estimation methods and also substantive research on the issues for which the data are compiled (see discussion in Chapter 4~. BTS should plan and begin to implement systematic programs of evaluation and im- provement of key transportation data sets. BTS Surveys As a first priority, BTS should review the evaluations it has completed or has under way for the two major intermodal surveys that it sponsors to determine what further evaluations are needed and what the evaluation results imply for appropriate use of the data and for future design decisions. These two surveys- the Commodity Flow Survey and the American Travel Survey are the largest component of the BTS budget, accounting for over one-third of BTS expendi- tures over fiscal years 1994 to 1997, most of which represents funds transferred to the Census Bureau for data collection and processing. Some evaluations that the Census Bureau, working with BTS, has completed of the 1995 ATS include assessments of"recallbias" and"time-in-samplebias."~4 Recall bias is inferred when respondents report a behavior more frequently for a period closer to the interview than for a period that is farther away in time. Time- in-sample bias is inferred when respondents change their behavior or reporting of their behavior over successive interviews. Both types of bias are important to evaluate for the ATS because it consisted of four interviews with the same house- holds, each interview covering a 3-month reporting period. Additional types of evaluation studies that would be important to undertake for both the ATS and CFS include comparisons of the characteristics of survey respondents with nonrespondents (including the implications of differences for the accuracy of key survey estimates) and comparisons of selected survey esti i4Two of the BTS staff are sworn census agents, so that they can work with confidential microdata at the Census Bureau for evaluation purposes.

FOCUSING ON DATA QUALITY 61 mates with estimates from other data sources. For example, trip reporting behav- ior could be compared for the 1995 ATS with the 1995 NETS and the 1995 Con- sumer Expenditure Survey (CEX). Aggregate comparisons between two data systems must be made with care to allow for differences in definitions and data collection and processing procedures, but the identification of discrepancies can lead to further research to determine reasons and suggest ways to improve one or both data systems. A comparative evaluation study that could be particularly useful for the ATS concerns transportation costs. Because of a belief that households are poor re- porters of costs, the ATS questionnaire does not ask about trip costs. BTS ex- pects that the U.S. Travel Data Center, a private organization, will develop model- based estimates of long-distance trip costs on the basis of trip characteristics. When such estimates are developed, it would be useful to compare them with trip cost information from the CEX. For subsequent rounds of the CFS and the ATS, BTS should consider addi- tional research and evaluation both prior to and as part of the surveys. For ex- ample, cognitive research techniques could be used to evaluate and improve the ATS questionnaire. Given the importance of information on travel costs, it could be worthwhile to embed an experiment within the ATS in which trip costs are obtained from a subsample of respondents and the completeness of their report- ing is evaluated against other sources. The results of evaluation studies should be used, together with assessments of the usefulness of CFS and ATS data by BTS staff and other analysts, to guide periodic reevaluations of the overall design of the two surveys. At present, the plans for the two surveys are to continue the historical pattern of conducting them at 5-year intervals with large sample sizes. (The 1995 ATS sample includes 80,000 households, the largest sample size of any U.S. national household sur- vey.) The large sample sizes in the 5-year design are intended to support needed subnational geographic analysis of interarea travel flows, but the cost is that up- dates are available only at relatively long intervals. An alternative design for the two surveys would be to have continuing small samples that provide national estimates on, say, an annual basis and to augment those samples periodically to obtain more detailed interarea data. Yet another design would be to have small national samples with added samples each year for specific areas that would "roll" across the country in some fashion. The rolling sample design would be helpful in the congressional budget process, in that it would smooth out peaks and valleys in required funding levels. However, the subnational estimates it provides could be difficult to interpret because the infor- mation for each year's area sample would necessarily pertain to transportation by residents or shippers within the specified areas and not also to movements of people or goods into those areas from nonsample areas. Careful consideration of transportation analysis needs and of the costs of alternative designs will be required to determine an optimal strategy for how

62 BUREAU OF TRANSPORTATION STATISTICS: PRIORITIES FOR THE FUTURE often the surveys are fielded and the corresponding sample sizes and designs. That strategy may turn out to be the current design of large surveys at 5-year intervals; however, that design should be chosen on the basis of research, evalua- tion, and user input and not just continued from the past. (Evaluation results should also inform other design choices, such as length of recall period and ques- tionnaire content and wording.~5 Finally, an assessment of the design of the ATS, and perhaps of the CFS as well, should take into account other similar surveys and the possibilities for coordinating or integrating their designs (see Appendix F). Other Data Once BTS gains experience and expertise in evaluating its own data systems and a reputation for excellence in this regard, then it will be in a position to advise other USDOT statistical units about evaluation and improvement of their data systems, particularly from the perspective of improving the usefulness of the data for cross-modal, system-wide analyses of transportation issues. Such a role is in keeping with the establishment of BTS in the 1991 ISTEA as the statistical agency with a broad mandate to improve transportation data within the department. BTS can begin immediately to work with the other USDOT modal adminis- trations to identify additional information on data quality and limitations that should be added to the descriptions on the BTS web site and in BTS compendia and reference publications. (This work will naturally be part of BTS's strength- ened mandate to develop data quality standards for USDOT.) Cooperative ef- forts with other modal administrations to undertake more extensive documenta- tion and to refine existing evaluation and improvement programs (or to launch new programs) for their data systems will require a carefully planned and staged approach. Work toward that end should be guided by a vision of transportation data needs within which to identify priority areas for attention in the short, me- dium, and long term (see Chapter 4) and by the data quality standards that are developed by BTS in cooperation with the other modal administrations. RECOMMENDATIONS Staffing (2) BTS should be authorized to appoint an associate director for statis i5Another design choice is whether to include a longitudinal component, in which data are obtained from the same reporting units over time, as was done to a limited extent in the 1995 ATS (households were interviewed four times over 1 year). Longitudinal data permit analysis of complex behavior patterns but can require significant resources and pose such problems as attrition (sample units drop- ping out of the survey) and time-in-sample bias.

FOCUSING ON DATA QUALITY 63 tical methods and research at the Senior Executive Service level to provide leadership in improving the quality of transportation statistics. BTS should give priority to hiring highly qualified staff with expertise in statistical meth- ods. Quality Standards (3) In the reauthorization of BTS, the Congress should strengthen cur- rent law by assigning responsibility to BTS to establish data quality stan- dards, consistent with good statistical practice, that are binding throughout USDOT and available for use by transportation agencies outside USDOT and for reference by the public. The reauthorization should also: · require the secretary of transportation to appoint a departmental standards committee, chaired by the BTS director and with representatives from the USDOT statistical units, to work with BTS in developing depart- ment-wide data quality standards and · require BTS to prepare every 2 years a report to the Congress that identifies improvements achieved in data quality by BTS and the statistical units in the other USDOT modal administrations and in the provision of information about quality to data users. Data Documentation (4) BTS should improve the documentation of the transportation data it makes available so that users can readily assess their quality, including accu- racy, variability, and comparability across transportation modes and over time.

Next: 4 ENSURING RELEVANCE »
The Bureau of Transportation Statistics: Priorities for the Future Get This Book
×
Buy Paperback | $48.00
MyNAP members save 10% online.
Login or Register to save!
Download Free PDF

How can the Bureau of Transportation Statistics (BTS), the nation's newest federal statistical agency, contribute to the work of the U.S. Department of Transportation (USDOT)? What is the appropriate role for such an agency as a part of a major department?

BTS was authorized in 1991 by the Intermodal Surface Transportation Efficiency Act (ISTEA) in recognition of the need for more and better data for transportation officials at local, state, and federal levels. While the USDOT has many long-standing data collection programs for particular transportation modes (highways, rail, air, etc.), it has never had a statistical agency with a mandate to improve the quality and relevance of transportation data for important system-wide, cross-modal analyses of the nation's transportation system.

This book examines how BTS can provide statistical leadership for USDOT, define and maintain quality standards for transportation data, and improve data documentation. It considers BTS's role in developing national transportation indicators, coordinating data collection throughout USDOT, filling gaps, identifying user needs, and developing analysis programs for transportation data.

Anyone concerned with having high-quality, relevant transportation indicators and other data available for policy planning, evaluation, and research will be interested in this book, as will students of effective government.

  1. ×

    Welcome to OpenBook!

    You're looking at OpenBook, NAP.edu's online reading room since 1999. Based on feedback from you, our users, we've made some improvements that make it easier than ever to read thousands of publications on our website.

    Do you want to take a quick tour of the OpenBook's features?

    No Thanks Take a Tour »
  2. ×

    Show this book's table of contents, where you can jump to any chapter by name.

    « Back Next »
  3. ×

    ...or use these buttons to go back to the previous chapter or skip to the next one.

    « Back Next »
  4. ×

    Jump up to the previous page or down to the next one. Also, you can type in a page number and press Enter to go directly to that page in the book.

    « Back Next »
  5. ×

    To search the entire text of this book, type in your search term here and press Enter.

    « Back Next »
  6. ×

    Share a link to this book page on your preferred social network or via email.

    « Back Next »
  7. ×

    View our suggested citation for this chapter.

    « Back Next »
  8. ×

    Ready to take your reading offline? Click here to buy this book in print or download it as a free PDF, if available.

    « Back Next »
Stay Connected!