Click for next page ( 32


The National Academies | 500 Fifth St. N.W. | Washington, D.C. 20001
Copyright © National Academy of Sciences. All rights reserved.
Terms of Use and Privacy Statement



Below are the first 10 and last 10 pages of uncorrected machine-read text (when available) of this chapter, followed by the top 30 algorithmically extracted key phrases from the chapter as a whole.
Intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text on the opening pages of each chapter. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.

Do not use for reproduction, copying, pasting, or reading; exclusively for search engines.

OCR for page 31
Focusing on Data Quality A statistical agency must be more than a data compilation and dissemination agency. It must create useful information, which means that it must focus on documenting, evaluating, and improving the quality and relevance of the data within its subject area. Without such a focus, resources are likely to be wasted- both by the statistical agency in compiling and disseminating data that are of poor quality or not relevant to analysis needs and by policy makers, planners, and researchers who are left to work with deficient or inappropriate data. To ensure data quality and relevance for its users while working to minimize costs and burden on data providers, a statistical agency must also keep pace with advances in data collection and statistical and analytical methods and techniques. The Bureau of Transportation Statistics (BTS) should begin now to devote more of its attention to data documentation, evaluation, and improvement, even if it means slowing down its efforts to be a one-stop-shopping source for users of every available transportation data set. Many state transportation officials and other users from whom we heard are looking to BTS to help them sort through the mass of available information to distinguish those data that are better and more appropriate for their needs from data that are of lesser quality and usefulness. BTS needs not only to help users in this regard, but also to work to improve the available base of information for addressing important transportation policy is- sues and research questions. The 1991 Intermodal Surface Transportation Effi- ciency Act (ISTEA) recognized the need for a statistical agency with a broad mandate to provide leadership for transportation data improvement by stipulating that BTS issue guidelines to ensure that the information from the U.S. Depart- ment of Transportation (USDOT) is accurate, reliable, relevant, and in a form that permits systematic analysis. 31

OCR for page 31
32 BUREAU OF TRANSPORTATION STATISTICS: PRIORITIES FOR THE FUTURE "Quality" and "relevance" are multidimensional attributes of data, each of which requires attention by a statistical agency.) In this chapter, we define what we mean by "quality," which encompasses the comparability, accuracy, and vari- ability of the data from a measurement system. BTS needs to focus immediately on data quality to qualify its initial emphasis on making as many kinds of data as widely available as possible. To make such a focus possible, BTS needs to strengthen the statistical and technical capabilities of its staff. The remainder of this chapter first discusses staffing requirements for BTS and then considers pri- ority activities for BTS in the data quality area. Such activities include the devel- opment of quality standards for USDOT and improvements in the documentation of available data, both to assist data users and to provide the basis for continuing evaluation and improvement of transportation data systems. The committee's primary recommendations appear at the end of the chapter. In Chapter 4, we define what we mean by "relevance," which encompasses the appropriateness of concepts, definitions, and measurements, the level of sub- ject and geographic detail, and the timeliness of data from a measurement system. Chapter 4 discusses activities, some of immediate priority and others that repre- sent longer-term goals, for BTS to ensure the relevance of transportation data for policy making and other purposes. DIMENSIONS OF QUALITY "Data quality" concerns the effects of how measurement systems are de- signed and conducted.2 Dimensions of data quality include: Comparability across data systems and time (e.g., for cross-modal com- parisons), which involves not only consistency of definitions, but also consis- tency, or at least similarity, among design features and data collection and pro- cessing procedures. As an example, the comparability of two data systems may be affected by differences in the method of data collection (such as personal interview, telephone interview, self-report, abstracting information from admin- istrative reports, and obtaining data from such recording or sensing equipment as highway sensors). (See Box 3-1 for examples of lack of comparability of data for a single transportation mode and for cross-modal analysis.) iThe two attributes should not be viewed as totally distinct. Indeed, relevance may be said to encompass quality in that relevance means broadly the usefulness of a data set for an application, and data of poor quality are hardly useful even if they provide relevant information in other respects. However, the requirements (e.g., staff skills) for addressing data quality as distinct from the other substantive dimensions of relevance differ, so that it makes sense to discuss data quality and relevance as separate attributes. 2The discussion draws on the thinking among statistical agencies in the United States and else- where. See, for example, Australian Bureau of Statistics (1990); Bureau of Economic Analysis (1995); Statistics Canada (1992); and Statistics Sweden (1994).

OCR for page 31
FOCUSING ON DATA QUALITY 33

OCR for page 31
34 BUREAU OF TRANSPORTATION STATISTICS: PRIORITIES FOR THE FUTURE . Accuracy or bias, which in general concerns how well a set of estimates approximates the true values of the phenomena under study and specifically con cerns errors that are due to systematic mismeasurement.3 Sources of systematic bias may include: differential coverage of groups in the population being studied (e.g., research has documented undercoverage of minorities in the decen nial census and household surveys and of smaller establishments in surveys of businesses and farms); differential nonresponse by reporting units (e.g., lower response rates to surveys for working families who are harder to find at home, underreporting of accidents that involve minor injuries or property damage); missing or erroneous reporting of specific items that is linked with other characteristics (e.g., higher missing data rates in surveys for in come and assets on the part of people with high incomes); biases in imputation and other data editing procedures that attempt to correct for missing data and other reporting errors. Variability, which includes instability in estimates from the use of a sample and from other sources. For example, variability may stem from differ- ences in how stringently enumerators or other data collectors apply specified procedures or from the variable application of editing and imputation procedures. Extent of revisions in time series, due to supplementation of preliminary data with later reports and other factors (see Young, 1996~. . Statistical agencies have developed tools to measure some of the above as- pects of data quality. Effective application of these tools requires that there be a technical staff engaged in ongoing methodological work to measure data quality and to develop design features for data systems that can provide users with evi- dence about quality. (See discussion below in the section on "Staffing.") A focus on documenting, evaluating, and improving data quality along the above dimensions is a central mission of a statistical agency. Such a focus is particularly important for BTS because data collection is so widely dispersed in the transportation field, making it difficult for users to assess the comparability, accuracy, and variability of data programs across the various transportation modes and a variety of public and private data sources. We did not ourselves conduct a review of the quality of transportation data programs, and hence we cannot say whether they have serious data quality prob- lems. However, no set of data is without error, and every data program has quality problems to a greater or lesser degree. What is important for a statistical 3A more technical definition is that estimates have low bias if they tend to be equal to the true values on average when the sampling process (or other data collection procedure) is repeated many times.

OCR for page 31
FOCUSING ON DATA QUALITY 35 agency like BTS is to document the errors and other problems in data programs in its area and to assess the extent to which they compromise the use of the data for their intended purposes. On the basis of documentation and evaluation, the agency should identify priority data programs for improvement, taking account of the need for the data and the costs and feasibility of improvement, and should implement a phased effort to effect improvements as resources are available. All that is not to say that data documentation, evaluation, and improvement efforts are not being carried out appropriately and well for particular transporta- tion data programs, such as particular surveys and administrative reporting sys- tems. However, data improvement activities for a particular data program do not ensure that the data will be comparable or of comparable quality when used with data from other programs, which is likely to be necessary to perform analyses across transportation modes or across time. Indeed, the Transportation Research Board report, Data for Decisions (National Research Council, 1992a), empha- sized the problems with cross-modal comparisons. BTS has highlighted some issues of cross-modal comparability in its Trans- portation Statistics Annual Reports (see Box 3-1~. However, our review of its programs and data products and services indicates that BTS to date has not fo- cused sufficient attention on helping users understand the problems of available transportation data nor on developing a systematic program to evaluate and im- prove the quality of transportation data (see sections below on "Documentation" and "Data Evaluation and Improvement;" see also Appendixes D and E). Such a program will require not only that BTS address data programs that it operates directly, but also that it work collaboratively with statistical units in other modal administrations in USDOT and with other data providers to ensure a focus on the quality of transportation data. Given limited budgets, it is rarely possible to improve the quality of a data system on all dimensions nor to improve both quality and relevance to the same extent: statistical agencies are commonly faced with making choices or trade- offs among them. For example, without increasing total costs, it may be difficult to increase the sample size of a survey in order to reduce sampling variability and at the same time devote efforts to reduce undercoverage of people or establish- ments. Similarly, it may be difficult to increase sample size and at the same time maintain the desired frequency of data collection (an aspect of relevance). An important function of a statistical agency is to make these trade-offs on the basis of the best assessment possible of currently available data and the likely payoffs to investments in them. Looking to the future, the increasing capabilities of computer-assisted data collection and the combined uses of administrative and survey data may make it feasible in some instances to achieve simultaneous cost and burden reductions and improvements in data quality and relevance. Statisti- cal agencies need to keep abreast of these developments and integrate them into their work. A prerequisite for BTS to undertake these activities is that it build strong statistical and analytical capabilities in its staff.

OCR for page 31
36 BUREAU OF TRANSPORTATION STATISTICS: PRIORITIES FOR THE FUTURE STAFFING The 1991 ISTEA envisions BTS as a statistical agency for USDOT that pro- vides leadership to the department in such areas as developing data quality stan- dards, working with the other modal administrations and the states to develop indicators of the transportation system, coordinating collection of transportation data by USDOT with other federal agencies, and improving the quality and rel- evance of transportation data for cross-modal, system-wide analysis. To take on these leadership roles, and in particular to enhance the department's focus on data quality, BTS must have adequate technical and analytical expertise on its staff. Such expertise is essential for BTS to document, evaluate, and improve its own data systems and data products. In turn, excellence in its own operations is a prerequisite for the agency to acquire the stature and moral authority that are necessary for BTS to become credible in a leadership role for the department as a whole. At present, BTSis a small agency; it lacks the depth and breadth of statistical and methodological expertise on its staff to coordinate a comprehensive program for USDOT of documenting, evaluating, and improving the department's data or, more generally, to provide statistical advice to other units.4 A key element in BTS's future evolution as a statistical agency will be its ability to develop the necessary capabilities on its staff. BTS's progress in this regard will become even more important to the extent that budget pressures on the other modal ad- ministrations in USDOT constrain their ability to maintain statistical and analyti- cal expertise in their agencies. Current and Planned Staffing The current BTS staff (37 people as of fall 1996, of which 16 were in the Office of Airline Information) have expertise and experience in a wide range of fields. A number of staff have backgrounds in transportation research, geographic information systems, data technology, and other relevant fields. However, few staff have extensive expertise in statistical methods (e.g., sampling, estimation, survey research and evaluation). Plans to fill 23 authorized vacancies include two mid-level positions for a mathematical statistician and a survey statistician. (These positions were recently filled.) 4As an example of an area in which BTS could provide technical assistance if it had sufficient staff capability, the Federal Aviation Administration was recently faced with the question of whether it could rank the safety records of individual air carriers. This question involves such statistical issues as the appropriate choice of safety indicators (e.g., number of accidents with fatalities per million aircraft-miles flown, number of accidents with fatalities per 100,000 takeoffs, number of fatalities per million passenger-miles flown) and whether differences among those indicators for individual airlines are meaningful (i.e., relate to factors that are under the airlines' control, such as age and maintenance of equipment, versus such factors as weather patterns at the airports used most by an airline).

OCR for page 31
FOCUSING ON DATA QUALITY 37 An earlier staffing plan, which envisioned a total of about 75 staff after fill- ing vacancies, included a position of senior adviser for statistical policy in the BTS director's office. BTS was directed by the Office of the Secretary to reduce its authorized full-time-equivalent (FTE) staff from 75 to 60 people. In revising its staffing plan, BTS chose to drop the senior statistical adviser position, which was slotted for a GS-15 level (the top grade below the Senior Executive Service), and to share statistical oversight responsibilities among the director, deputy di- rector, and two associate directors. (The director is a presidential appointee; the deputy director and associate directors are Senior Executive Service positions.) The reason given for this decision, which BTS hopes to reverse if it is authorized to have more FTE staff, is that it is difficult to justify positions at the GS-15 level, particularly under strictures from the current administration, as part of its Rein- venting Government initiative, to reduce the ratio of senior supervisors to other staff. Building a Strong Statistical Staff In the panel's view, it is essential for BTS to implement a staffing plan that gives much higher priority to building expertise in statistical methods and related quantitative fields than is provided in the current staffing plan. Needed areas of skill include statistical sampling, statistical design, cognitive foundations of sur- vey measurement, advanced data collection methods, editing, imputation for miss- ing data, and statistical estimation from complex sample surveys. At present, BTS's statistical staff capabilities are augmented by Census Bureau staff who work on the Commodity Flow Survey and the American Travel Survey. How- ever, there is no substitute for sufficient in-house staff with the necessary exper- tise if BTS is to achieve excellence in its own operations and if it is to be able to exercise statistical leadership for the department as a whole.5 BTS should repro- gram a portion of the available vacancies to emphasize statistical and related skills and should move expeditiously to fill those vacancies. To underscore the importance of a strong in-house statistical staff for BTS and to provide a focal point for BTS's work to evaluate and improve the quality of transportation data, the panel believes that BTS should be authorized by the department to appoint an associate director for statistical methods and research at the Senior Executive Service level (see recommendation 2 at the end of the chap- ter). The senior level is justified given the importance for a statistical agency of 5Many statistical agencies, like BTS, use contractor staff for a variety of purposes, including data collection and processing, programming support for analytical work, conference arrangements, and publication preparation. We did not consider in detail the appropriate mix of in-house and contractor staff for a statistical agency many factors enter into the choice of mix, including costs, constraints on full-time-equivalent staff, and past agency experience. However, we stress that a statistical agency must have sufficient in-house statistical and technical capability to carry out key functions and prop- erly direct the work of contractors.

OCR for page 31
38 BUREAU OF TRANSPORTATION STATISTICS: PRIORITIES FOR THE FUTURE keeping abreast of and applying advanced statistical methods and techniques to such functions as data evaluation and improvement. Although the titles vary, other major statistical agencies, such as the Census Bureau, the National Center for Education Statistics, and the National Center for Health Statistics, have simi- lar positions. The BTS associate director for statistical methods and research should have extensive expertise in such areas as statistical estimation and survey research methods. BTS should authorize the associate director to build a statistical staff that plays a leadership role for BTS in developing data quality standards, design- ing and implementing evaluation studies of BTS data systems, and conducting research on improved methods of data collection, processing, and estimation. The BTS statistical staff would take the lead in working with statistical units in the other USDOT modal administrations to develop standards and priorities for data documentation, evaluation, and improvement of the department's data sys- tems. The BTS statistical staff would also provide technical assistance to the other modal administrations as appropriate. The associate director for statistical methods and research and BTS as a whole could benefit from outside statistical advice on a regular basis. As required by the 1991 ISTEA, BTS currently has an Advisory Council on Transportation Sta- tistics, which meets twice a year to discuss BTS's programs and review new initiatives. This group has a strong user orientation and focuses on issues regard- ing the kinds of transportation data that are needed for important policy purposes. A separate advisory group that focuses on issues of statistical methods and stan- dards would also be very useful. As some other statistical agencies have done (e.g., the Bureau of Justice Sta- tistics, the Census Bureau, the Energy Information Administration), BTS could ask the American Statistical Association to establish a working group of experts to meet regularly with its statistical staff on technical matters. The members of such a group should have expertise in such areas as sampling and survey design, advanced data collection methods, weighting and imputation methods for miss- ing and erroneous data, and statistical estimation from complex sample surveys. As BTS develops closer working relationships with the other modal administra- tions, many of which have large amounts of data collected from administrative reporting systems in addition to sample surveys, a statistical advisory group should also include experts in the design and statistical applications of adminis- trative records. Such a group could assist BTS to evaluate alternative designs and data collection, processing, and analysis procedures for surveys and other data col- lection programs and to establish priorities for statistical research and evaluation. Continuing Staff Development Building and maintaining strong statistical and technical staff capabilities requires an agency's continuing attention. BTS's top management should give

OCR for page 31
FOCUSING ON DATA QUALITY 39 priority to identifying opportunities for staff development and to encouraging staff to take full advantage of them. Among the kinds of activities that can foster the development of technical skills are attendance at advanced courses, presen- tations at professional association meetings, and publication in professional jour- nals. BTS already has some useful vehicles for staff development in place. Its new peer-reviewed publication, the Journal of Transportation and Statistics, should prove to be a valuable means by which to stimulate methodological re- search and analysis on the part of staff. Also, its regular seminar series, which brings leading transportation researchers from the United States and abroad to present analytical results and discuss important issues for transportation policy, is an important means of enhancing professional knowledge and skills. This pro- gram should be continued and expanded to include relevant issues of statistical methods and approach. Some statistical agencies have specified goals for the performance of tech- nical staff that relate to keeping current in the technical developments of their field. These goals can be achieved by participation in relevant graduate courses at local universities, attendance at continuing education short courses, or atten- dance at other seminars that are relevant to the field. There are several active programs in the Washington, D.C., area that provide opportunities for profes- sional development of these kinds. By placing explicit direction in performance plans for continuous improvement of technical skills, the agency can make ex- plicit its commitment to this goal. Some other ways to enhance professional capabilities are more costly and hence may be appropriate for BTS to consider only when it is somewhat larger and has more resources. For example, several statistical agencies have visiting fellows programs that are administered through the American Statistical Associa- tion, in which distinguished statisticians and other researchers come to the agency for a specified time period to work on topics of mutual interest. The visiting fellows gain insights into the practical operational problems of a statistical agency, and the agency staff benefit from working closely with leading researchers. Such a program involves significant budget commitments and can also take time to become established. It may be difficult for a small agency such as BTS to accom- plish, but the concept is worth investigating for possible implementation at a future date. Similarly, statistical agencies sometimes provide their staff with opportuni- ties to work at other agencies or organizations for periods of 6 to 18 months in areas that will benefit the home agency. When BTS is larger, it could consider occasionally detailing one or two people to another statistical unit within USDOT, to another federal statistical agency, or to another organization with statistical expertise, as a way for staff to gain valuable experiences and insights. Similarly, BTS could sponsor staff from other USDOT modal administrations or other fed- eral statistical agencies to work at BTS. Exchanges of staff between BTS and

OCR for page 31
40 BUREAU OF TRANSPORTATION STATISTICS: PRIORITIES FOR THE FUTURE other statistical units within USDOT could be particularly valuable in building cooperative relationships, cross-modal perspectives, and a strong commitment to data quality within the department. QUALITY STANDARDS For documenting, evaluating, and improving data quality, it is very helpful for a statistical agency to develop explicit notions of appropriate standards for collecting, processing, and publishing the data. BTS has been working with other USDOT modal administrations to develop improved means of data dissemina- tion, such as more user-friendly CD-ROM formats, which is a step toward facili- tating systematic data analysis. BTS has not yet begun to work with the other modal administrations to develop guidelines for data quality throughout USDOT, as it is mandated to do by the 1991 ISTEA, nor to standardize key concepts, definitions, and procedures to the extent feasible and appropriate in order to fa- cilitate cross-modal analysis. In some people's interpretations, BTS is constrained from moving in this direction by the provision in the 1991 ISTEA that nothing in the legislation shall be construed "~1) to authorize the Bureau to require any other department or agency to collect data; or (2) to reduce the authority of any other officer of the Department of Transportation to collect and disseminate data independently." However, our view is that this provision does not contradict the mandate for BTS to develop guidelines for data quality for USDOT in collaboration with statistical units in the other modal administrations. Indeed, we urge that the reauthorization of BTS strengthen its role by requir- ing it to develop data quality standards, consistent with good statistical practice, that are binding throughout USDOT and available for use by transportation agen- cies outside USDOT and for reference by the public (see recommendation 3 at the end of the chapter). In so doing, Congress will both underscore the importance of focusing on the quality of transportation data and clarify BTS's responsibility to move forward in this area. BTS should develop data quality standards for the department with the coop- eration and input of the other statistical units in USDOT obtained through a de- partment-wide standards committee that is chaired by the BTS director. Coop- erative efforts are essential, so that the other units can come to see the benefits to their users and buy into the process and so that BTS can carry out its leadership function in this area as a facilitator and not as a regulator or enforcer. The stan- dards committee should be mandated in the reauthorization of BTS. The reauthorization should also require that BTS every 2 years prepare a report to the Congress that describes progress during the previous 2 years to set standards and that identifies improvements in data quality by BTS and other USDOT statistical units and in the provision of information about quality to data users. We recommend the biennial report primarily as a tool to promote a focus

OCR for page 31
FOCUSING ON DATA QUALITY 41 on data quality; it could also usefully describe major steps to improve the rel- evance of transportation data in terms of timeliness, subject matter and geographic detail, appropriateness of concepts and definitions, and the initiation of new data programs or the consolidation or elimination of data programs in order to satisfy users' priority needs more cost-effectively. We specify a biennial rather than annual report so that there is time for progress to be made and for the report to be a substantive document and not simply a time-consuming exercise in fulfilling a requirement. Another way to ensure substance is for each report to identify selected data programs or subject areas in which quality improvements will be sought on a priority basis and to highlight those areas in the next report, commenting more generally on other areas. The biennial report that we recommend is not to be confused with the Trans- portation Statistics Annual Reports that are mandated by the 1991 ISTEA. Those reports have regularly included a section on the state of transportation statistics, but those sections have been general in nature and do not meet the need we see for reporting improvements on specific quality dimensions for specific transporta- tion data programs or sets of related programs. We discuss the role of the Trans- portation Statistics Annual Reports in providing needed time series indicators and analyses of transportation trends in Chapter 4, where we suggest that there may be more cost-effective ways of providing these kinds of information than the current prescribed format. We recognize that BTS is still a new, small agency within the U.S. Depart- ment of Transportation with a challenging array of responsibilities. Also, as dis- cussed earlier, BTS currently lacks the staff resources and the necessary technical capabilities and expertise with which to develop its statistical functions as fully as its data compilation and dissemination functions. Hence, it will not be an easy task for BTS to assume responsibility for leading a process to develop quality standards for USDOT as a whole. However, we believe strongly that BTS must evolve to be the statistical agency for USDOT that is envisioned in the 1991 ISTEA, which means that it must begin to take on a leadership role in several areas. The need for leadership to sort out higher- from lower-quality data and to identify priorities for new and improved data is clear from reviews of transportation data needs (e.g., National Research Council, 1992a). Such reviews invariably cite the large volume of data available from public and private sources but the lack of comparable data that provide useful information for analyses of important transportation issues, par- ticularly those that require a cross-modal or system-wide perspective. We recognize that progress in such areas as developing quality standards cannot happen overnight. Nonetheless, the work must begin, and the agency that was established to be the major statistical unit for the department as a whole must be given the authority and motivation to move forward collaboratively with the other statistical units in USDOT.

OCR for page 31
FOCUSING ON DATA QUALITY 53 impression that the site is being built piecemeal with little thought given to an overall structure that reflects a data user perspective. National Transportation Library (NTL) By far the largest part of the BTS web site is the NTL, which has received praise from reference librarians in the trans- portation field and is well organized to help users find documents on a particular topic. Yet the NTL contains few data and little information about data that is available elsewhere. For example, the safety portion of the site has almost 200 entnes, few of which provide relevant data the documents provided include, among others, reports of the U.S. General Accounting Office on aviation safety issues, the marine safety manual of the U.S. Coast Guard, recommended emer- gency preparedness guidelines for rail transit systems, and bicycle helmet laws by state. These and similar documents may serve a useful reference purpose. How- ever, we question the wisdom of devoting resources and staff attention to expand the NTL if that means fewer resources with which to improve the BTS web site as a guide to users about available data. There is also a problem that the quality of the documents may vary widely, and BTS has no way to control quality. Search capabilities for data The BTS web site can be confusing for the user who wants to find high-quality data (as opposed to reference documents) on a particular cross-modal topic, such as safety. The two sections of the site that provide data or descriptions of data are "Products" and "Databases." (Some BTS products also appear under "BTS Programs." The "Beefing Room" has a "Statis- tics" section, but it is limited to data on airline operations from the Office of Airline Information.) The BTS "Products" section has a subject index in addition to an index by transportation mode; it is also searchable by the user (the search engine actually searches the entire site). However, the entries in the subject index are very gen- eral for example, the NTS reports are listed as sources of safety data without further elaboration (see Figure 3-1~.~3 A keyword search on "safety data" brings ~ iThe BTS web site is updated frequently. Since we first began looking at the site in 1996, BTS has not only added new content, but also improved the organization of the site in several respects; how- ever, more needs to be done. Our comments are based on the site as of April-June 1997. i2The NTL page carries a general disclaimer that inclusion of a document in the NTL "does not necessarily constitute endorsement" by BTS or USDOT (see Figure 2-2). BTS has also recently begun the use of automated software to check for documents in the NTL that are inaccessible because the site of the originating organization has been taken off the web or for another reason. The links for such documents will then be removed or corrected. (These kinds of problems can happen frequently: a review by our panel of the safety section of the NTL prior to the installation of regular checks found that almost two-fifths of the documents were inaccessible because the host server could not be found, the document could not be found on the host server, or the document did not contain information.) i3That the full NTS reports are not yet available on the BTS web site is surprising, given that the site provides the complete text of statistical reports from other modal administrations (e.g., 1995 Highway Statistics from the Federal Highway Administration). Recently, the 1996 Transportation Statistics An- nual Report was made directly accessible through the BTS web site, as were the tables in the 1997 NTS.

OCR for page 31
54 BUREAU OF TRANSPORTATION STATISTICS: PRIORITIES FOR THE FUTURE ....................................................................................................... - . Subject Listing . .............................................................................................................................................................................................................. Please check News & Updates to see a listing of products as they become available online. Safety Directory of Transportation Data Sources 1996 - You can now add or adjust a Data Source for inclusion in the 1997 edition. National Transportation Statistics (NTS) 1993 National Transportation Statistics (NTS) 1997 - NTS Tables available for download in MS Excel 5.0 format. State and Metropolitan Analysis for Regional Transportation (SMART) - Available through the National Transportation Library on the BTS Homepage. Traffic Safety Data: PARS and GES Transportation Data Sampler-3 Transportation in the United States: A Review Transportation Safety Transportation Statistics Annual Report (TSAR) 1994 Transportation Statistics Annual Report (TSAR) 1995 Transportation Statistics Annual Report (TSAR) 1996 - Available in PDF format. Transportation Statistics: In Brief [Alphabetical Listing] [Media Listing] [Mode Listing] [BTS Products Page] [BTS Services] [Order Form] Feedback? Questions? commentsff~bts.gov FIGURE 3-1 Portion of subject index for "Products" section of BTS web site. up some but not all of the entries under "safety" in the "Products" subject index together with new entries. The entries are annotated; however, the annotations in many instances are not informative about the content of the item listed (e.g., see the listing for "homepage.rtf" in Figure 3-2~. The "Databases" section of the site brings up the National Transportation Data Archive, which contains statistical reports, data sets, and descriptions of data sets, including entries from the Federal Aviation Administration, Federal

OCR for page 31
FOCUSING ON DATA QUALITY excite for web servers found documents about: safety data 68% about.htm Summary: Welcome to the Office of System Safety's Homepage. Who Are We? 68% index.html 55 Summary: Fatal Accident Reporting System Database. The Traffic Safety Data set was developed by the Bureau of Transportation Statistics and the National Highway Traffic Safety Administration's (NHTSA) National Center for Statistics and Analysis (NCSA) to make traffic safety data easily accessible and widely available. 68% BTS Products - Traffic Safety Data: 1988-1993 Summary: What is it? The Traffic Safety Data: 1988-1993 was produced by the Bureau of Transportation Statistics to make traffic safety data easily accessible and widely available. 66% SMART-SAFETY@BTS.GOVdocFrame.html Summary: *NEW* Emergency Medical Services (EMS) Public Information, Education and Relations (PIER) - National Standard Curriculum (PDF file). *NEW* Public Education Video Clips [Served by Federal Railway Administration]. 66% SMART-SURVEY@BTS.GOVdocFrame.html Summary: *NEW* Sample Transportation Surveys. *NEW* Survey of Motor Carriers in the Rochester Transportation Management Area. 64% BTS Products - Traffic Safety Summary: What is it? The Traffic Safety CD-ROM contains 2 years of statistics from the National Highway Traffic Safety Administration's (NHTSA) Fatal Accident Reporting System (PARS), 1975-1994 and 7 years of statistics from the General Estimates System (GES), 1988-1994 in ASCII format along with its associated technical documentation. 64% homepage.rtf Summa~y:{/rtf 1 /ansi/deff4/deflang 1 033{/fonttbl{/f4/froman/fcharsetO/fprq2TimesNewRoman ;} } {/colortbl ;/redO/g reenO/bl ueO ;/redO/g reenO/bl ue255;/redO/g reen255/bl ue255 ;/redO/ FIGURE 3-2 Results of searching BTS web site for "Safety Data."

OCR for page 31
56 BUREAU OF TRANSPORTATION STATISTICS: PRIORITIES FOR THE FUTURE green255/blueO;/red255/greenO/blue255;/red255/greenO/blueO;/red255/green255/blueO;/ red255/g reen255/bl ue255 ;/redO/g reenO/bl ue 128 ;/redO/g reen 128/bl ue 128 ;/redO/g reen 128/ blueO;/red128/greenO/blue128;/red128/greenO/blue/snextONormal;} {/*/cs10 /additive Default ParagraphFont;} } {/info{/title About the Aviation Safety Office} {/author FeliceBrunner} {/ operatorFeliceBrunner} {/creatim/yr1996/mo3/dy27/hr14/min41 } {/revtim/yr1996/mo3/dy28/hr11 /min 12} {/printim/ yr1996/mo3/dy28/hr11 /min29} {/version5} {/edmins131 } {/nofpagesO} {/nofwordsO} {/ nofcharsO} {/vern49213} } /widowctrl/ftn bj/aenddoc/noextrasprl/prcolbl//fetO/sectd/l i nexO/ endnhere{/*/pnseclvl1/pnucrm/pnstart1/pnindent720/pnhang{/pntxta.} } {/*/pnseclvl2/pnucltr/ pnstart1 /pnind 64% ts91395k.html Summary: "Drive Smart" Nights at Central Pennsylvania Speedways PENNSYLVANIA PROBLEM IDENTIFICATION Observation of those who frequent professional auto racing events shows that racing fans are more likely than other motorists to drive faster than the speed limit and drive after drinking and are less likely to wear a safety belt. In an effort to communicate directly with this high-risk segment of the driving public, Pennsylvania's Center for Highway Safety Program collaborated with the South Central Pennsylvania Highway Safety Program to establish special safe driving promotions at local speedways throughout 13 counties in Central Pennsylvania. 64% ts91395h.html Summary: 100% Platinum Pacesetter Safety Belt Honor Roll MARYLAND PROBLEM IDEN- TIFICATION During the past several years, Maryland law enforcement agencies received extensive state and national recognition for their promotion of safety belt use. Prior goals and programs established by concerned highway safety groups in Maryland helped move communities towards increased safety belt use rates. 64% SMART-PUBLIC@BTS.GOVdocFrame.html Summary: *NEW* Public Involvement Procedures for New Hampshire Transportation Im- provement Projects.1992 Transportation & Air Quality Planning Guidelines. 64% ts91395i.html Summary: Cornhusker Highway Community/Corridor Traffic Safety Project NEBRASKA PROBLEM IDENTIFICATION Highway 6, also known as Cornhusker Highway, in Lincoln, Nebraska has a high rate of traffic crashes. The road is a high speed arterial with an average daily traffic flow of 32,000 vehicles and a multitude of access points. 64% The National Transportation Safety Section Summary: Safety. Take part in our new Communications Center! 64% Finding the DOT Records You Want Summary: Guide To Finding The DOT Records You Want. FIGURE 3-2 Continued

OCR for page 31
FOCUSING ON DATA QUALITY ~ 64% BTS Products - Transportation Statistics: In Brief 57 Summary: Transportation Statistics: In Brief. Transportation Statistics: In Brief is a pocket pamphlet designed to highlight two years of transportation data, 1 980 and 1 994. 62% Major Customers Summary: Major Customers. The Office of Airline Information provides the airline financial, traffic and economic data systems that are the critical foundation of DOT's regulatory, advo- cacy and policy decision-making processes. 62% ts91395g.html Summary: Accident Location Analysis System IOWA PROBLEM IDENTIFICATION The Bureau of Transportation Safety at the Iowa Department of Transportation (IDOT) maintains a database of traffic records sent in by investigating officers as well as drivers involved in crashes occurring on public road systems in Iowa. All crashes that result in a fatality, a personal injury or at least $500 property damage are included in the database. 62% tab9-2.txt Summary: TABLE 9.2 AIRLINES (Air Carriers Operating under 14 CFR 121) Accidents, Fatalities, and Rates (Preliminary Data) 1993 Scheduled Nonscheduled Accidents Total 23 0 Fatal 1 0 Fatalities 1 0 Aircraft Hours Flown (000) 1 11,900 624 Departures (000) 1 7,732 312 Accident Rate Per 100,000 Hours Flown Total 0.19 0.00 Fatal 0.01 0.00 Accident Rate Per 100,000 Departures Total 0.30 0.00 Fatal 0.01 0.00 1 Exposure data estimate source: Research and Special Programs Administration and FAA Source: National Transportation Safety ... 62% tab9-2.txt Summary: TABLE 9.2 AIRLINES (Air Carriers Operating under 14 CFR 121) Accidents, Fatalities, and Rates (Preliminary Data) 1993 Scheduled Nonscheduled Accidents Total 23 0 Fatal 1 0 Fatalities 1 0 Aircraft Hours Flown (000) 1 11,900 624 Departures (000) 1 7,732 312 Accident Rate Per 100,000 Hours Flown Total 0.19 0.00 Fatal 0.01 0.00 Accident Rate Per 100,000 Departures Total 0.30 0.00 Fatal 0.01 0.00 1 Exposure data estimate source: Research and Special Programs Administration and FAA Source: National Transportation Safety ... 62% tab9-3.txt Summary: TABLE 9.3 AIRLINES (Air Carriers Operating under 14 CFR 121) Fatal Acci- dents, Fatalities (Preliminary Data) 1993 Location Operator Date Serv. Aircraft Fatalities Total Reported Type On- Type of Total Pass- Crew Others board Accident engers SCHED- ULED SERVICE Chicago, IL Simmons 4/4 Psgr ATR 1 0 0 1 48 Ground Airlines 42-300 crewmember dba: struck American by Eagle propeller NONSCHEDULED SERVICE None Source: National Transportation Safety ... FIGURE 3-2 Continued

OCR for page 31
58 BUREAU OF TRANSPORTATION STATISTICS: PRIORITIES FOR THE FUTURE 62% tab9-3.txt Summary: TABLE 9.3 AIRLINES (Air Carriers Operating under 14 CFR 121) Fatal Acci- dents, Fatalities (Preliminary Data) 1993 Location Operator Date Serv. Aircraft Fatalities Total Reported Type On- Type of Total Pass- Crew Others board Accident engers SCHED- ULED SERVICE Chicago, IL Simmons 4/4 Psgr ATR 1 0 0 1 48 Ground Airlines 42-300 crewmember dba: struck American by Eagle propeller NONSCHEDULED SERVICE None Source: National Transportation Safety ... [Results by Excite] FIGURE 3-2 Continued Highway Administration, National Highway Traffic Safety Administration, and the Federal Transit Administration, as well as BTS. At present, the archive con- tains 11 listings, of which some are descriptions of data products rather than data (see Figure 3-3~. To help the user locate additional data and information, the BTS site pro- vides links to the web sites of the other USDOT modal administrations (these links are to the main administrations and not to their statistical units). The BTS site also has links to many other organizations (commercial, private, government, nonprofit, libraries) with some relation to transportation. The user can search any and all of these sites for data; however, their design does not always facilitate such a search. The BTS site itself provides no guidance for users in their search of other sites. Such guidance could take the form of putting the Directory of Transportation Data Sources on the BTS site, making its contents searchable by keyword, and, when applicable, adding links to other web sites to obtain more information or to see the actual data. Alternatively, such guidance could be pro- vided through short essays that inform the user of major data series in particular cross-modal topic areas and where to find them. Data documentation The BTS web site gives no evidence of the application of consistent standards for the information provided about the quality and limita- tions of available data. The brief descriptions that are provided in the "Products" section for BTS CD-ROM products vary in content and usefulness (see Appendix E). Each data set listed in the National Transportation Data Archive (see Figure 3-3) has a contents page that links to the following headings: Detailed Descrip- tion, Reports and Products, Searchable Database (operational as yet for only some of the data sets), Questions and Comments, Methods and Limitations, Future Plans, Applications, and Related Topics. This selection of headings appears po- tentially very useful; however, to date, there is limited or no information provided for such key headings as Methods and Limitations for many of the data sets in the National Data Archive. Several of the data sets in the archive reproduce publica

OCR for page 31
FOCUSING ON DATA QUALITY 59 FIGURE 3-3 Contents of National Transportation Data Archive in "Databases" section of BTS web site. lions from other USDOT modal administrations (e.g., highway statistics); the documentation that is provided in these publications about data collection meth- ods and data quality varies according to the practices of the originating agencies. At this stage of its development, it may not be feasible for BTS to standard- ize the documentation for all of the data sets it makes accessible on its web site from the other modal administrations (or other sources), although such standard- ization should be a goal of the work to develop department-wide quality stan- dards. However, BTS can and should move quickly to standardize documenta- tion for its own data sets and also to standardize and improve the descriptions of its CD-ROM and other products. It should also include on the site a prominent statement to advise users about the importance of understanding the meaning and limitations of available data sets before attempting to work with them. Providing

OCR for page 31
60 BUREAU OF TRANSPORTATION STATISTICS: PRIORITIES FOR THE FUTURE contact names whenever possible for users to learn not only about the content and scope of a data set, but also about its uses and limitations, would be helpful. DATA EVALUATION AND IMPROVEMENT Documentation of data quality and relevance is essential for users; however, such documentation is necessarily limited to the extent that a data set has not been evaluated on a range of dimensions. Major responsibilities of statistical agencies are to evaluate their data and, on the basis of such evaluations, to inform users of current limitations of the data and to develop ongoing programs to improve data quality (see National Research Council, 1992b, and Appendix C). To support continuing data improvement, agencies need to conduct statistical research on data collection, processing, and estimation methods and also substantive research on the issues for which the data are compiled (see discussion in Chapter 4~. BTS should plan and begin to implement systematic programs of evaluation and im- provement of key transportation data sets. BTS Surveys As a first priority, BTS should review the evaluations it has completed or has under way for the two major intermodal surveys that it sponsors to determine what further evaluations are needed and what the evaluation results imply for appropriate use of the data and for future design decisions. These two surveys- the Commodity Flow Survey and the American Travel Survey are the largest component of the BTS budget, accounting for over one-third of BTS expendi- tures over fiscal years 1994 to 1997, most of which represents funds transferred to the Census Bureau for data collection and processing. Some evaluations that the Census Bureau, working with BTS, has completed of the 1995 ATS include assessments of"recallbias" and"time-in-samplebias."~4 Recall bias is inferred when respondents report a behavior more frequently for a period closer to the interview than for a period that is farther away in time. Time- in-sample bias is inferred when respondents change their behavior or reporting of their behavior over successive interviews. Both types of bias are important to evaluate for the ATS because it consisted of four interviews with the same house- holds, each interview covering a 3-month reporting period. Additional types of evaluation studies that would be important to undertake for both the ATS and CFS include comparisons of the characteristics of survey respondents with nonrespondents (including the implications of differences for the accuracy of key survey estimates) and comparisons of selected survey esti i4Two of the BTS staff are sworn census agents, so that they can work with confidential microdata at the Census Bureau for evaluation purposes.

OCR for page 31
FOCUSING ON DATA QUALITY 61 mates with estimates from other data sources. For example, trip reporting behav- ior could be compared for the 1995 ATS with the 1995 NETS and the 1995 Con- sumer Expenditure Survey (CEX). Aggregate comparisons between two data systems must be made with care to allow for differences in definitions and data collection and processing procedures, but the identification of discrepancies can lead to further research to determine reasons and suggest ways to improve one or both data systems. A comparative evaluation study that could be particularly useful for the ATS concerns transportation costs. Because of a belief that households are poor re- porters of costs, the ATS questionnaire does not ask about trip costs. BTS ex- pects that the U.S. Travel Data Center, a private organization, will develop model- based estimates of long-distance trip costs on the basis of trip characteristics. When such estimates are developed, it would be useful to compare them with trip cost information from the CEX. For subsequent rounds of the CFS and the ATS, BTS should consider addi- tional research and evaluation both prior to and as part of the surveys. For ex- ample, cognitive research techniques could be used to evaluate and improve the ATS questionnaire. Given the importance of information on travel costs, it could be worthwhile to embed an experiment within the ATS in which trip costs are obtained from a subsample of respondents and the completeness of their report- ing is evaluated against other sources. The results of evaluation studies should be used, together with assessments of the usefulness of CFS and ATS data by BTS staff and other analysts, to guide periodic reevaluations of the overall design of the two surveys. At present, the plans for the two surveys are to continue the historical pattern of conducting them at 5-year intervals with large sample sizes. (The 1995 ATS sample includes 80,000 households, the largest sample size of any U.S. national household sur- vey.) The large sample sizes in the 5-year design are intended to support needed subnational geographic analysis of interarea travel flows, but the cost is that up- dates are available only at relatively long intervals. An alternative design for the two surveys would be to have continuing small samples that provide national estimates on, say, an annual basis and to augment those samples periodically to obtain more detailed interarea data. Yet another design would be to have small national samples with added samples each year for specific areas that would "roll" across the country in some fashion. The rolling sample design would be helpful in the congressional budget process, in that it would smooth out peaks and valleys in required funding levels. However, the subnational estimates it provides could be difficult to interpret because the infor- mation for each year's area sample would necessarily pertain to transportation by residents or shippers within the specified areas and not also to movements of people or goods into those areas from nonsample areas. Careful consideration of transportation analysis needs and of the costs of alternative designs will be required to determine an optimal strategy for how

OCR for page 31
62 BUREAU OF TRANSPORTATION STATISTICS: PRIORITIES FOR THE FUTURE often the surveys are fielded and the corresponding sample sizes and designs. That strategy may turn out to be the current design of large surveys at 5-year intervals; however, that design should be chosen on the basis of research, evalua- tion, and user input and not just continued from the past. (Evaluation results should also inform other design choices, such as length of recall period and ques- tionnaire content and wording.~5 Finally, an assessment of the design of the ATS, and perhaps of the CFS as well, should take into account other similar surveys and the possibilities for coordinating or integrating their designs (see Appendix F). Other Data Once BTS gains experience and expertise in evaluating its own data systems and a reputation for excellence in this regard, then it will be in a position to advise other USDOT statistical units about evaluation and improvement of their data systems, particularly from the perspective of improving the usefulness of the data for cross-modal, system-wide analyses of transportation issues. Such a role is in keeping with the establishment of BTS in the 1991 ISTEA as the statistical agency with a broad mandate to improve transportation data within the department. BTS can begin immediately to work with the other USDOT modal adminis- trations to identify additional information on data quality and limitations that should be added to the descriptions on the BTS web site and in BTS compendia and reference publications. (This work will naturally be part of BTS's strength- ened mandate to develop data quality standards for USDOT.) Cooperative ef- forts with other modal administrations to undertake more extensive documenta- tion and to refine existing evaluation and improvement programs (or to launch new programs) for their data systems will require a carefully planned and staged approach. Work toward that end should be guided by a vision of transportation data needs within which to identify priority areas for attention in the short, me- dium, and long term (see Chapter 4) and by the data quality standards that are developed by BTS in cooperation with the other modal administrations. RECOMMENDATIONS Staffing (2) BTS should be authorized to appoint an associate director for statis i5Another design choice is whether to include a longitudinal component, in which data are obtained from the same reporting units over time, as was done to a limited extent in the 1995 ATS (households were interviewed four times over 1 year). Longitudinal data permit analysis of complex behavior patterns but can require significant resources and pose such problems as attrition (sample units drop- ping out of the survey) and time-in-sample bias.

OCR for page 31
FOCUSING ON DATA QUALITY 63 tical methods and research at the Senior Executive Service level to provide leadership in improving the quality of transportation statistics. BTS should give priority to hiring highly qualified staff with expertise in statistical meth- ods. Quality Standards (3) In the reauthorization of BTS, the Congress should strengthen cur- rent law by assigning responsibility to BTS to establish data quality stan- dards, consistent with good statistical practice, that are binding throughout USDOT and available for use by transportation agencies outside USDOT and for reference by the public. The reauthorization should also: require the secretary of transportation to appoint a departmental standards committee, chaired by the BTS director and with representatives from the USDOT statistical units, to work with BTS in developing depart- ment-wide data quality standards and require BTS to prepare every 2 years a report to the Congress that identifies improvements achieved in data quality by BTS and the statistical units in the other USDOT modal administrations and in the provision of information about quality to data users. Data Documentation (4) BTS should improve the documentation of the transportation data it makes available so that users can readily assess their quality, including accu- racy, variability, and comparability across transportation modes and over time.