|
Items in cart [0] |
Questions? Call 888-624-8373 |
| Copyright © 2009. National Academy of Sciences. All rights reserved. Terms of Use and Privacy Statement |
Below are the first 10 and last 10 pages of uncorrected machine-read text (when available) of this chapter, followed by the top 30 algorithmically extracted key phrases from the chapter as a whole.
Intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text on the opening pages of each chapter.
Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.
Do not use for reproduction, copying, pasting, or reading; exclusively for search engines.
OCR for page 31
Focusing on Data Quality
A statistical agency must be more than a data compilation and dissemination
agency. It must create useful information, which means that it must focus on
documenting, evaluating, and improving the quality and relevance of the data
within its subject area. Without such a focus, resources are likely to be wasted-
both by the statistical agency in compiling and disseminating data that are of poor
quality or not relevant to analysis needs and by policy makers, planners, and
researchers who are left to work with deficient or inappropriate data. To ensure
data quality and relevance for its users while working to minimize costs and
burden on data providers, a statistical agency must also keep pace with advances
in data collection and statistical and analytical methods and techniques.
The Bureau of Transportation Statistics (BTS) should begin now to devote
more of its attention to data documentation, evaluation, and improvement, even if
it means slowing down its efforts to be a one-stop-shopping source for users of
every available transportation data set. Many state transportation officials and
other users from whom we heard are looking to BTS to help them sort through the
mass of available information to distinguish those data that are better and more
appropriate for their needs from data that are of lesser quality and usefulness.
BTS needs not only to help users in this regard, but also to work to improve the
available base of information for addressing important transportation policy is-
sues and research questions. The 1991 Intermodal Surface Transportation Effi-
ciency Act (ISTEA) recognized the need for a statistical agency with a broad
mandate to provide leadership for transportation data improvement by stipulating
that BTS issue guidelines to ensure that the information from the U.S. Depart-
ment of Transportation (USDOT) is accurate, reliable, relevant, and in a form that
permits systematic analysis.
31
OCR for page 32
32 BUREAU OF TRANSPORTATION STATISTICS: PRIORITIES FOR THE FUTURE
"Quality" and "relevance" are multidimensional attributes of data, each of
which requires attention by a statistical agency.) In this chapter, we define what
we mean by "quality," which encompasses the comparability, accuracy, and vari-
ability of the data from a measurement system. BTS needs to focus immediately
on data quality to qualify its initial emphasis on making as many kinds of data as
widely available as possible. To make such a focus possible, BTS needs to
strengthen the statistical and technical capabilities of its staff. The remainder of
this chapter first discusses staffing requirements for BTS and then considers pri-
ority activities for BTS in the data quality area. Such activities include the devel-
opment of quality standards for USDOT and improvements in the documentation
of available data, both to assist data users and to provide the basis for continuing
evaluation and improvement of transportation data systems. The committee's
primary recommendations appear at the end of the chapter.
In Chapter 4, we define what we mean by "relevance," which encompasses
the appropriateness of concepts, definitions, and measurements, the level of sub-
ject and geographic detail, and the timeliness of data from a measurement system.
Chapter 4 discusses activities, some of immediate priority and others that repre-
sent longer-term goals, for BTS to ensure the relevance of transportation data for
policy making and other purposes.
DIMENSIONS OF QUALITY
"Data quality" concerns the effects of how measurement systems are de-
signed and conducted.2 Dimensions of data quality include:
· Comparability across data systems and time (e.g., for cross-modal com-
parisons), which involves not only consistency of definitions, but also consis-
tency, or at least similarity, among design features and data collection and pro-
cessing procedures. As an example, the comparability of two data systems may
be affected by differences in the method of data collection (such as personal
interview, telephone interview, self-report, abstracting information from admin-
istrative reports, and obtaining data from such recording or sensing equipment as
highway sensors). (See Box 3-1 for examples of lack of comparability of data for
a single transportation mode and for cross-modal analysis.)
iThe two attributes should not be viewed as totally distinct. Indeed, relevance may be said to
encompass quality in that relevance means broadly the usefulness of a data set for an application, and
data of poor quality are hardly useful even if they provide relevant information in other respects.
However, the requirements (e.g., staff skills) for addressing data quality as distinct from the other
substantive dimensions of relevance differ, so that it makes sense to discuss data quality and relevance
as separate attributes.
2The discussion draws on the thinking among statistical agencies in the United States and else-
where. See, for example, Australian Bureau of Statistics (1990); Bureau of Economic Analysis (1995);
Statistics Canada (1992); and Statistics Sweden (1994).
OCR for page 33
FOCUSING ON DATA QUALITY
33
OCR for page 34
34 BUREAU OF TRANSPORTATION STATISTICS: PRIORITIES FOR THE FUTURE
.
Accuracy or bias, which in general concerns how well a set of estimates
approximates the true values of the phenomena under study and specifically con
cerns errors that are due to systematic mismeasurement.3 Sources of systematic
bias may include:
differential coverage of groups in the population being studied (e.g.,
research has documented undercoverage of minorities in the decen
nial census and household surveys and of smaller establishments in
surveys of businesses and farms);
differential nonresponse by reporting units (e.g., lower response rates
to surveys for working families who are harder to find at home,
underreporting of accidents that involve minor injuries or property
damage);
missing or erroneous reporting of specific items that is linked with
other characteristics (e.g., higher missing data rates in surveys for in
come and assets on the part of people with high incomes);
biases in imputation and other data editing procedures that attempt to
correct for missing data and other reporting errors.
Variability, which includes instability in estimates from the use of a
sample and from other sources. For example, variability may stem from differ-
ences in how stringently enumerators or other data collectors apply specified
procedures or from the variable application of editing and imputation procedures.
· Extent of revisions in time series, due to supplementation of preliminary
data with later reports and other factors (see Young, 1996~.
.
Statistical agencies have developed tools to measure some of the above as-
pects of data quality. Effective application of these tools requires that there be a
technical staff engaged in ongoing methodological work to measure data quality
and to develop design features for data systems that can provide users with evi-
dence about quality. (See discussion below in the section on "Staffing.")
A focus on documenting, evaluating, and improving data quality along the
above dimensions is a central mission of a statistical agency. Such a focus is
particularly important for BTS because data collection is so widely dispersed in
the transportation field, making it difficult for users to assess the comparability,
accuracy, and variability of data programs across the various transportation modes
and a variety of public and private data sources.
We did not ourselves conduct a review of the quality of transportation data
programs, and hence we cannot say whether they have serious data quality prob-
lems. However, no set of data is without error, and every data program has
quality problems to a greater or lesser degree. What is important for a statistical
3A more technical definition is that estimates have low bias if they tend to be equal to the true
values on average when the sampling process (or other data collection procedure) is repeated many
times.
OCR for page 35
FOCUSING ON DATA QUALITY
35
agency like BTS is to document the errors and other problems in data programs in
its area and to assess the extent to which they compromise the use of the data for
their intended purposes. On the basis of documentation and evaluation, the
agency should identify priority data programs for improvement, taking account
of the need for the data and the costs and feasibility of improvement, and should
implement a phased effort to effect improvements as resources are available.
All that is not to say that data documentation, evaluation, and improvement
efforts are not being carried out appropriately and well for particular transporta-
tion data programs, such as particular surveys and administrative reporting sys-
tems. However, data improvement activities for a particular data program do not
ensure that the data will be comparable or of comparable quality when used with
data from other programs, which is likely to be necessary to perform analyses
across transportation modes or across time. Indeed, the Transportation Research
Board report, Data for Decisions (National Research Council, 1992a), empha-
sized the problems with cross-modal comparisons.
BTS has highlighted some issues of cross-modal comparability in its Trans-
portation Statistics Annual Reports (see Box 3-1~. However, our review of its
programs and data products and services indicates that BTS to date has not fo-
cused sufficient attention on helping users understand the problems of available
transportation data nor on developing a systematic program to evaluate and im-
prove the quality of transportation data (see sections below on "Documentation"
and "Data Evaluation and Improvement;" see also Appendixes D and E). Such a
program will require not only that BTS address data programs that it operates
directly, but also that it work collaboratively with statistical units in other modal
administrations in USDOT and with other data providers to ensure a focus on the
quality of transportation data.
Given limited budgets, it is rarely possible to improve the quality of a data
system on all dimensions nor to improve both quality and relevance to the same
extent: statistical agencies are commonly faced with making choices or trade-
offs among them. For example, without increasing total costs, it may be difficult
to increase the sample size of a survey in order to reduce sampling variability and
at the same time devote efforts to reduce undercoverage of people or establish-
ments. Similarly, it may be difficult to increase sample size and at the same time
maintain the desired frequency of data collection (an aspect of relevance). An
important function of a statistical agency is to make these trade-offs on the basis
of the best assessment possible of currently available data and the likely payoffs
to investments in them. Looking to the future, the increasing capabilities of
computer-assisted data collection and the combined uses of administrative and
survey data may make it feasible in some instances to achieve simultaneous cost
and burden reductions and improvements in data quality and relevance. Statisti-
cal agencies need to keep abreast of these developments and integrate them into
their work. A prerequisite for BTS to undertake these activities is that it build
strong statistical and analytical capabilities in its staff.
OCR for page 36
36 BUREAU OF TRANSPORTATION STATISTICS: PRIORITIES FOR THE FUTURE
STAFFING
The 1991 ISTEA envisions BTS as a statistical agency for USDOT that pro-
vides leadership to the department in such areas as developing data quality stan-
dards, working with the other modal administrations and the states to develop
indicators of the transportation system, coordinating collection of transportation
data by USDOT with other federal agencies, and improving the quality and rel-
evance of transportation data for cross-modal, system-wide analysis. To take on
these leadership roles, and in particular to enhance the department's focus on data
quality, BTS must have adequate technical and analytical expertise on its staff.
Such expertise is essential for BTS to document, evaluate, and improve its own
data systems and data products. In turn, excellence in its own operations is a
prerequisite for the agency to acquire the stature and moral authority that are
necessary for BTS to become credible in a leadership role for the department as a
whole.
At present, BTSis a small agency; it lacks the depth and breadth of statistical
and methodological expertise on its staff to coordinate a comprehensive program
for USDOT of documenting, evaluating, and improving the department's data or,
more generally, to provide statistical advice to other units.4 A key element in
BTS's future evolution as a statistical agency will be its ability to develop the
necessary capabilities on its staff. BTS's progress in this regard will become
even more important to the extent that budget pressures on the other modal ad-
ministrations in USDOT constrain their ability to maintain statistical and analyti-
cal expertise in their agencies.
Current and Planned Staffing
The current BTS staff (37 people as of fall 1996, of which 16 were in the
Office of Airline Information) have expertise and experience in a wide range of
fields. A number of staff have backgrounds in transportation research, geographic
information systems, data technology, and other relevant fields. However, few
staff have extensive expertise in statistical methods (e.g., sampling, estimation,
survey research and evaluation). Plans to fill 23 authorized vacancies include
two mid-level positions for a mathematical statistician and a survey statistician.
(These positions were recently filled.)
4As an example of an area in which BTS could provide technical assistance if it had sufficient staff
capability, the Federal Aviation Administration was recently faced with the question of whether it
could rank the safety records of individual air carriers. This question involves such statistical issues
as the appropriate choice of safety indicators (e.g., number of accidents with fatalities per million
aircraft-miles flown, number of accidents with fatalities per 100,000 takeoffs, number of fatalities per
million passenger-miles flown) and whether differences among those indicators for individual airlines
are meaningful (i.e., relate to factors that are under the airlines' control, such as age and maintenance
of equipment, versus such factors as weather patterns at the airports used most by an airline).
OCR for page 37
FOCUSING ON DATA QUALITY
37
An earlier staffing plan, which envisioned a total of about 75 staff after fill-
ing vacancies, included a position of senior adviser for statistical policy in the
BTS director's office. BTS was directed by the Office of the Secretary to reduce
its authorized full-time-equivalent (FTE) staff from 75 to 60 people. In revising
its staffing plan, BTS chose to drop the senior statistical adviser position, which
was slotted for a GS-15 level (the top grade below the Senior Executive Service),
and to share statistical oversight responsibilities among the director, deputy di-
rector, and two associate directors. (The director is a presidential appointee; the
deputy director and associate directors are Senior Executive Service positions.)
The reason given for this decision, which BTS hopes to reverse if it is authorized
to have more FTE staff, is that it is difficult to justify positions at the GS-15 level,
particularly under strictures from the current administration, as part of its Rein-
venting Government initiative, to reduce the ratio of senior supervisors to other
staff.
Building a Strong Statistical Staff
In the panel's view, it is essential for BTS to implement a staffing plan that
gives much higher priority to building expertise in statistical methods and related
quantitative fields than is provided in the current staffing plan. Needed areas of
skill include statistical sampling, statistical design, cognitive foundations of sur-
vey measurement, advanced data collection methods, editing, imputation for miss-
ing data, and statistical estimation from complex sample surveys. At present,
BTS's statistical staff capabilities are augmented by Census Bureau staff who
work on the Commodity Flow Survey and the American Travel Survey. How-
ever, there is no substitute for sufficient in-house staff with the necessary exper-
tise if BTS is to achieve excellence in its own operations and if it is to be able to
exercise statistical leadership for the department as a whole.5 BTS should repro-
gram a portion of the available vacancies to emphasize statistical and related
skills and should move expeditiously to fill those vacancies.
To underscore the importance of a strong in-house statistical staff for BTS
and to provide a focal point for BTS's work to evaluate and improve the quality
of transportation data, the panel believes that BTS should be authorized by the
department to appoint an associate director for statistical methods and research at
the Senior Executive Service level (see recommendation 2 at the end of the chap-
ter). The senior level is justified given the importance for a statistical agency of
5Many statistical agencies, like BTS, use contractor staff for a variety of purposes, including data
collection and processing, programming support for analytical work, conference arrangements, and
publication preparation. We did not consider in detail the appropriate mix of in-house and contractor
staff for a statistical agency many factors enter into the choice of mix, including costs, constraints
on full-time-equivalent staff, and past agency experience. However, we stress that a statistical agency
must have sufficient in-house statistical and technical capability to carry out key functions and prop-
erly direct the work of contractors.
OCR for page 38
38 BUREAU OF TRANSPORTATION STATISTICS: PRIORITIES FOR THE FUTURE
keeping abreast of and applying advanced statistical methods and techniques to
such functions as data evaluation and improvement. Although the titles vary,
other major statistical agencies, such as the Census Bureau, the National Center
for Education Statistics, and the National Center for Health Statistics, have simi-
lar positions.
The BTS associate director for statistical methods and research should have
extensive expertise in such areas as statistical estimation and survey research
methods. BTS should authorize the associate director to build a statistical staff
that plays a leadership role for BTS in developing data quality standards, design-
ing and implementing evaluation studies of BTS data systems, and conducting
research on improved methods of data collection, processing, and estimation.
The BTS statistical staff would take the lead in working with statistical units in
the other USDOT modal administrations to develop standards and priorities for
data documentation, evaluation, and improvement of the department's data sys-
tems. The BTS statistical staff would also provide technical assistance to the
other modal administrations as appropriate.
The associate director for statistical methods and research and BTS as a whole
could benefit from outside statistical advice on a regular basis. As required by
the 1991 ISTEA, BTS currently has an Advisory Council on Transportation Sta-
tistics, which meets twice a year to discuss BTS's programs and review new
initiatives. This group has a strong user orientation and focuses on issues regard-
ing the kinds of transportation data that are needed for important policy purposes.
A separate advisory group that focuses on issues of statistical methods and stan-
dards would also be very useful.
As some other statistical agencies have done (e.g., the Bureau of Justice Sta-
tistics, the Census Bureau, the Energy Information Administration), BTS could
ask the American Statistical Association to establish a working group of experts
to meet regularly with its statistical staff on technical matters. The members of
such a group should have expertise in such areas as sampling and survey design,
advanced data collection methods, weighting and imputation methods for miss-
ing and erroneous data, and statistical estimation from complex sample surveys.
As BTS develops closer working relationships with the other modal administra-
tions, many of which have large amounts of data collected from administrative
reporting systems in addition to sample surveys, a statistical advisory group
should also include experts in the design and statistical applications of adminis-
trative records. Such a group could assist BTS to evaluate alternative designs and
data collection, processing, and analysis procedures for surveys and other data col-
lection programs and to establish priorities for statistical research and evaluation.
Continuing Staff Development
Building and maintaining strong statistical and technical staff capabilities
requires an agency's continuing attention. BTS's top management should give
OCR for page 39
FOCUSING ON DATA QUALITY
39
priority to identifying opportunities for staff development and to encouraging
staff to take full advantage of them. Among the kinds of activities that can foster
the development of technical skills are attendance at advanced courses, presen-
tations at professional association meetings, and publication in professional jour-
nals.
BTS already has some useful vehicles for staff development in place. Its
new peer-reviewed publication, the Journal of Transportation and Statistics,
should prove to be a valuable means by which to stimulate methodological re-
search and analysis on the part of staff. Also, its regular seminar series, which
brings leading transportation researchers from the United States and abroad to
present analytical results and discuss important issues for transportation policy, is
an important means of enhancing professional knowledge and skills. This pro-
gram should be continued and expanded to include relevant issues of statistical
methods and approach.
Some statistical agencies have specified goals for the performance of tech-
nical staff that relate to keeping current in the technical developments of their
field. These goals can be achieved by participation in relevant graduate courses
at local universities, attendance at continuing education short courses, or atten-
dance at other seminars that are relevant to the field. There are several active
programs in the Washington, D.C., area that provide opportunities for profes-
sional development of these kinds. By placing explicit direction in performance
plans for continuous improvement of technical skills, the agency can make ex-
plicit its commitment to this goal.
Some other ways to enhance professional capabilities are more costly and
hence may be appropriate for BTS to consider only when it is somewhat larger
and has more resources. For example, several statistical agencies have visiting
fellows programs that are administered through the American Statistical Associa-
tion, in which distinguished statisticians and other researchers come to the agency
for a specified time period to work on topics of mutual interest. The visiting
fellows gain insights into the practical operational problems of a statistical agency,
and the agency staff benefit from working closely with leading researchers. Such
a program involves significant budget commitments and can also take time to
become established. It may be difficult for a small agency such as BTS to accom-
plish, but the concept is worth investigating for possible implementation at a
future date.
Similarly, statistical agencies sometimes provide their staff with opportuni-
ties to work at other agencies or organizations for periods of 6 to 18 months in
areas that will benefit the home agency. When BTS is larger, it could consider
occasionally detailing one or two people to another statistical unit within USDOT,
to another federal statistical agency, or to another organization with statistical
expertise, as a way for staff to gain valuable experiences and insights. Similarly,
BTS could sponsor staff from other USDOT modal administrations or other fed-
eral statistical agencies to work at BTS. Exchanges of staff between BTS and
OCR for page 40
40 BUREAU OF TRANSPORTATION STATISTICS: PRIORITIES FOR THE FUTURE
other statistical units within USDOT could be particularly valuable in building
cooperative relationships, cross-modal perspectives, and a strong commitment to
data quality within the department.
QUALITY STANDARDS
For documenting, evaluating, and improving data quality, it is very helpful
for a statistical agency to develop explicit notions of appropriate standards for
collecting, processing, and publishing the data. BTS has been working with other
USDOT modal administrations to develop improved means of data dissemina-
tion, such as more user-friendly CD-ROM formats, which is a step toward facili-
tating systematic data analysis. BTS has not yet begun to work with the other
modal administrations to develop guidelines for data quality throughout USDOT,
as it is mandated to do by the 1991 ISTEA, nor to standardize key concepts,
definitions, and procedures to the extent feasible and appropriate in order to fa-
cilitate cross-modal analysis.
In some people's interpretations, BTS is constrained from moving in this
direction by the provision in the 1991 ISTEA that nothing in the legislation shall
be construed "~1) to authorize the Bureau to require any other department or
agency to collect data; or (2) to reduce the authority of any other officer of the
Department of Transportation to collect and disseminate data independently."
However, our view is that this provision does not contradict the mandate for BTS
to develop guidelines for data quality for USDOT in collaboration with statistical
units in the other modal administrations.
Indeed, we urge that the reauthorization of BTS strengthen its role by requir-
ing it to develop data quality standards, consistent with good statistical practice,
that are binding throughout USDOT and available for use by transportation agen-
cies outside USDOT and for reference by the public (see recommendation 3 at the
end of the chapter). In so doing, Congress will both underscore the importance of
focusing on the quality of transportation data and clarify BTS's responsibility to
move forward in this area.
BTS should develop data quality standards for the department with the coop-
eration and input of the other statistical units in USDOT obtained through a de-
partment-wide standards committee that is chaired by the BTS director. Coop-
erative efforts are essential, so that the other units can come to see the benefits to
their users and buy into the process and so that BTS can carry out its leadership
function in this area as a facilitator and not as a regulator or enforcer. The stan-
dards committee should be mandated in the reauthorization of BTS.
The reauthorization should also require that BTS every 2 years prepare a
report to the Congress that describes progress during the previous 2 years to set
standards and that identifies improvements in data quality by BTS and other
USDOT statistical units and in the provision of information about quality to data
users. We recommend the biennial report primarily as a tool to promote a focus
OCR for page 41
FOCUSING ON DATA QUALITY
41
on data quality; it could also usefully describe major steps to improve the rel-
evance of transportation data in terms of timeliness, subject matter and geographic
detail, appropriateness of concepts and definitions, and the initiation of new data
programs or the consolidation or elimination of data programs in order to satisfy
users' priority needs more cost-effectively.
We specify a biennial rather than annual report so that there is time for
progress to be made and for the report to be a substantive document and not
simply a time-consuming exercise in fulfilling a requirement. Another way to
ensure substance is for each report to identify selected data programs or subject
areas in which quality improvements will be sought on a priority basis and to
highlight those areas in the next report, commenting more generally on other
areas.
The biennial report that we recommend is not to be confused with the Trans-
portation Statistics Annual Reports that are mandated by the 1991 ISTEA. Those
reports have regularly included a section on the state of transportation statistics,
but those sections have been general in nature and do not meet the need we see for
reporting improvements on specific quality dimensions for specific transporta-
tion data programs or sets of related programs. We discuss the role of the Trans-
portation Statistics Annual Reports in providing needed time series indicators
and analyses of transportation trends in Chapter 4, where we suggest that there
may be more cost-effective ways of providing these kinds of information than the
current prescribed format.
We recognize that BTS is still a new, small agency within the U.S. Depart-
ment of Transportation with a challenging array of responsibilities. Also, as dis-
cussed earlier, BTS currently lacks the staff resources and the necessary technical
capabilities and expertise with which to develop its statistical functions as fully as
its data compilation and dissemination functions. Hence, it will not be an easy
task for BTS to assume responsibility for leading a process to develop quality
standards for USDOT as a whole.
However, we believe strongly that BTS must evolve to be the statistical
agency for USDOT that is envisioned in the 1991 ISTEA, which means that it
must begin to take on a leadership role in several areas. The need for leadership
to sort out higher- from lower-quality data and to identify priorities for new and
improved data is clear from reviews of transportation data needs (e.g., National
Research Council, 1992a). Such reviews invariably cite the large volume of data
available from public and private sources but the lack of comparable data that
provide useful information for analyses of important transportation issues, par-
ticularly those that require a cross-modal or system-wide perspective.
We recognize that progress in such areas as developing quality standards
cannot happen overnight. Nonetheless, the work must begin, and the agency that
was established to be the major statistical unit for the department as a whole must
be given the authority and motivation to move forward collaboratively with the
other statistical units in USDOT.
OCR for page 53
FOCUSING ON DATA QUALITY
53
impression that the site is being built piecemeal with little thought given to an
overall structure that reflects a data user perspective.
National Transportation Library (NTL) By far the largest part of the BTS web
site is the NTL, which has received praise from reference librarians in the trans-
portation field and is well organized to help users find documents on a particular
topic. Yet the NTL contains few data and little information about data that is
available elsewhere. For example, the safety portion of the site has almost 200
entnes, few of which provide relevant data the documents provided include,
among others, reports of the U.S. General Accounting Office on aviation safety
issues, the marine safety manual of the U.S. Coast Guard, recommended emer-
gency preparedness guidelines for rail transit systems, and bicycle helmet laws
by state.
These and similar documents may serve a useful reference purpose. How-
ever, we question the wisdom of devoting resources and staff attention to expand
the NTL if that means fewer resources with which to improve the BTS web site as
a guide to users about available data. There is also a problem that the quality of
the documents may vary widely, and BTS has no way to control quality.
Search capabilities for data The BTS web site can be confusing for the user
who wants to find high-quality data (as opposed to reference documents) on a
particular cross-modal topic, such as safety. The two sections of the site that
provide data or descriptions of data are "Products" and "Databases." (Some BTS
products also appear under "BTS Programs." The "Beefing Room" has a "Statis-
tics" section, but it is limited to data on airline operations from the Office of
Airline Information.)
The BTS "Products" section has a subject index in addition to an index by
transportation mode; it is also searchable by the user (the search engine actually
searches the entire site). However, the entries in the subject index are very gen-
eral for example, the NTS reports are listed as sources of safety data without
further elaboration (see Figure 3-1~.~3 A keyword search on "safety data" brings
~ iThe BTS web site is updated frequently. Since we first began looking at the site in 1996, BTS has
not only added new content, but also improved the organization of the site in several respects; how-
ever, more needs to be done. Our comments are based on the site as of April-June 1997.
i2The NTL page carries a general disclaimer that inclusion of a document in the NTL "does not
necessarily constitute endorsement" by BTS or USDOT (see Figure 2-2). BTS has also recently
begun the use of automated software to check for documents in the NTL that are inaccessible because
the site of the originating organization has been taken off the web or for another reason. The links for
such documents will then be removed or corrected. (These kinds of problems can happen frequently:
a review by our panel of the safety section of the NTL prior to the installation of regular checks found
that almost two-fifths of the documents were inaccessible because the host server could not be found,
the document could not be found on the host server, or the document did not contain information.)
i3That the full NTS reports are not yet available on the BTS web site is surprising, given that the site
provides the complete text of statistical reports from other modal administrations (e.g., 1995 Highway
Statistics from the Federal Highway Administration). Recently, the 1996 Transportation Statistics An-
nual Report was made directly accessible through the BTS web site, as were the tables in the 1997 NTS.
OCR for page 54
54 BUREAU OF TRANSPORTATION STATISTICS: PRIORITIES FOR THE FUTURE
....................................................................................................... - .
Subject Listing
. ..............................................................................................................................................................................................................
Please check News & Updates to see a listing of products as they become available online.
Safety
Directory of Transportation Data Sources 1996 - You can now add or adjust a Data
Source for inclusion in the 1997 edition.
National Transportation Statistics (NTS) 1993
National Transportation Statistics (NTS) 1997 - NTS Tables available for download in
MS Excel 5.0 format.
State and Metropolitan Analysis for Regional Transportation (SMART) - Available
through the National Transportation Library on the BTS Homepage.
Traffic Safety Data: PARS and GES
Transportation Data Sampler-3
Transportation in the United States: A Review
Transportation Safety
Transportation Statistics Annual Report (TSAR) 1994
Transportation Statistics Annual Report (TSAR) 1995
Transportation Statistics Annual Report (TSAR) 1996 - Available in PDF format.
Transportation Statistics: In Brief
[Alphabetical Listing] [Media Listing] [Mode Listing]
[BTS Products Page] [BTS Services] [Order Form]
Feedback? Questions? commentsff~bts.gov
FIGURE 3-1 Portion of subject index for "Products" section of BTS web site.
up some but not all of the entries under "safety" in the "Products" subject index
together with new entries. The entries are annotated; however, the annotations in
many instances are not informative about the content of the item listed (e.g., see
the listing for "homepage.rtf" in Figure 3-2~.
The "Databases" section of the site brings up the National Transportation
Data Archive, which contains statistical reports, data sets, and descriptions of
data sets, including entries from the Federal Aviation Administration, Federal
OCR for page 55
FOCUSING ON DATA QUALITY
excite for web servers found documents about: safety data
68% about.htm
Summary: Welcome to the Office of System Safety's Homepage. Who Are We?
68% index.html
55
Summary: Fatal Accident Reporting System Database. The Traffic Safety Data set was
developed by the Bureau of Transportation Statistics and the National Highway Traffic
Safety Administration's (NHTSA) National Center for Statistics and Analysis (NCSA) to make
traffic safety data easily accessible and widely available.
68% BTS Products - Traffic Safety Data: 1988-1993
Summary: What is it? The Traffic Safety Data: 1988-1993 was produced by the Bureau of
Transportation Statistics to make traffic safety data easily accessible and widely available.
66% SMART-SAFETY@BTS.GOVdocFrame.html
Summary: *NEW* Emergency Medical Services (EMS) Public Information, Education and
Relations (PIER) - National Standard Curriculum (PDF file). *NEW* Public Education Video
Clips [Served by Federal Railway Administration].
66% SMART-SURVEY@BTS.GOVdocFrame.html
Summary: *NEW* Sample Transportation Surveys. *NEW* Survey of Motor Carriers in the
Rochester Transportation Management Area.
64% BTS Products - Traffic Safety
Summary: What is it? The Traffic Safety CD-ROM contains 2 years of statistics from the
National Highway Traffic Safety Administration's (NHTSA) Fatal Accident Reporting System
(PARS), 1975-1994 and 7 years of statistics from the General Estimates System (GES),
1988-1994 in ASCII format along with its associated technical documentation.
64% homepage.rtf
Summa~y:{/rtf 1 /ansi/deff4/deflang 1 033{/fonttbl{/f4/froman/fcharsetO/fprq2TimesNewRoman ;}
} {/colortbl ;/redO/g reenO/bl ueO ;/redO/g reenO/bl ue255;/redO/g reen255/bl ue255 ;/redO/
FIGURE 3-2 Results of searching BTS web site for "Safety Data."
OCR for page 56
56 BUREAU OF TRANSPORTATION STATISTICS: PRIORITIES FOR THE FUTURE
green255/blueO;/red255/greenO/blue255;/red255/greenO/blueO;/red255/green255/blueO;/
red255/g reen255/bl ue255 ;/redO/g reenO/bl ue 128 ;/redO/g reen 128/bl ue 128 ;/redO/g reen 128/
blueO;/red128/greenO/blue128;/red128/greenO/blue/snextONormal;} {/*/cs10 /additive Default
ParagraphFont;} } {/info{/title About the Aviation Safety Office} {/author FeliceBrunner} {/
operatorFeliceBrunner}
{/creatim/yr1996/mo3/dy27/hr14/min41 } {/revtim/yr1996/mo3/dy28/hr11 /min 12} {/printim/
yr1996/mo3/dy28/hr11 /min29} {/version5} {/edmins131 } {/nofpagesO} {/nofwordsO} {/
nofcharsO} {/vern49213} } /widowctrl/ftn bj/aenddoc/noextrasprl/prcolbl//fetO/sectd/l i nexO/
endnhere{/*/pnseclvl1/pnucrm/pnstart1/pnindent720/pnhang{/pntxta.} } {/*/pnseclvl2/pnucltr/
pnstart1 /pnind
64% ts91395k.html
Summary: "Drive Smart" Nights at Central Pennsylvania Speedways PENNSYLVANIA
PROBLEM IDENTIFICATION Observation of those who frequent professional auto racing
events shows that racing fans are more likely than other motorists to drive faster than the
speed limit and drive after drinking and are less likely to wear a safety belt. In an effort to
communicate directly with this high-risk segment of the driving public, Pennsylvania's Center
for Highway Safety Program collaborated with the South Central Pennsylvania Highway
Safety Program to establish special safe driving promotions at local speedways throughout
13 counties in Central Pennsylvania.
64% ts91395h.html
Summary: 100% Platinum Pacesetter Safety Belt Honor Roll MARYLAND PROBLEM IDEN-
TIFICATION During the past several years, Maryland law enforcement agencies received
extensive state and national recognition for their promotion of safety belt use. Prior goals
and programs established by concerned highway safety groups in Maryland helped move
communities towards increased safety belt use rates.
64% SMART-PUBLIC@BTS.GOVdocFrame.html
Summary: *NEW* Public Involvement Procedures for New Hampshire Transportation Im-
provement Projects.1992 Transportation & Air Quality Planning Guidelines.
64% ts91395i.html
Summary: Cornhusker Highway Community/Corridor Traffic Safety Project NEBRASKA
PROBLEM IDENTIFICATION Highway 6, also known as Cornhusker Highway, in Lincoln,
Nebraska has a high rate of traffic crashes. The road is a high speed arterial with an average
daily traffic flow of 32,000 vehicles and a multitude of access points.
64% The National Transportation Safety Section
Summary: Safety. Take part in our new Communications Center!
64% Finding the DOT Records You Want
Summary: Guide To Finding The DOT Records You Want.
FIGURE 3-2 Continued
OCR for page 57
FOCUSING ON DATA QUALITY
~ 64% BTS Products - Transportation Statistics: In Brief
57
Summary: Transportation Statistics: In Brief. Transportation Statistics: In Brief is a pocket
pamphlet designed to highlight two years of transportation data, 1 980 and 1 994.
62% Major Customers
Summary: Major Customers. The Office of Airline Information provides the airline financial,
traffic and economic data systems that are the critical foundation of DOT's regulatory, advo-
cacy and policy decision-making processes.
62% ts91395g.html
Summary: Accident Location Analysis System IOWA PROBLEM IDENTIFICATION The
Bureau of Transportation Safety at the Iowa Department of Transportation (IDOT) maintains
a database of traffic records sent in by investigating officers as well as drivers involved in
crashes occurring on public road systems in Iowa. All crashes that result in a fatality, a
personal injury or at least $500 property damage are included in the database.
62% tab9-2.txt
Summary: TABLE 9.2 AIRLINES (Air Carriers Operating under 14 CFR 121) Accidents,
Fatalities, and Rates (Preliminary Data) 1993 Scheduled Nonscheduled Accidents Total 23
0 Fatal 1 0 Fatalities 1 0 Aircraft Hours Flown (000) 1 11,900 624 Departures (000) 1 7,732
312 Accident Rate Per 100,000 Hours Flown Total 0.19 0.00 Fatal 0.01 0.00 Accident Rate
Per 100,000 Departures Total 0.30 0.00 Fatal 0.01 0.00 1 Exposure data estimate source:
Research and Special Programs Administration and FAA Source: National Transportation
Safety ...
62% tab9-2.txt
Summary: TABLE 9.2 AIRLINES (Air Carriers Operating under 14 CFR 121) Accidents,
Fatalities, and Rates (Preliminary Data) 1993 Scheduled Nonscheduled Accidents Total 23
0 Fatal 1 0 Fatalities 1 0 Aircraft Hours Flown (000) 1 11,900 624 Departures (000) 1 7,732
312 Accident Rate Per 100,000 Hours Flown Total 0.19 0.00 Fatal 0.01 0.00 Accident Rate
Per 100,000 Departures Total 0.30 0.00 Fatal 0.01 0.00 1 Exposure data estimate source:
Research and Special Programs Administration and FAA Source: National Transportation
Safety ...
62% tab9-3.txt
Summary: TABLE 9.3 AIRLINES (Air Carriers Operating under 14 CFR 121) Fatal Acci-
dents, Fatalities (Preliminary Data) 1993 Location Operator Date Serv. Aircraft Fatalities
Total Reported Type On- Type of Total Pass- Crew Others board Accident engers SCHED-
ULED SERVICE Chicago, IL Simmons 4/4 Psgr ATR 1 0 0 1 48 Ground Airlines 42-300
crewmember dba: struck American by Eagle propeller NONSCHEDULED SERVICE None
Source: National Transportation Safety ...
FIGURE 3-2 Continued
OCR for page 58
58 BUREAU OF TRANSPORTATION STATISTICS: PRIORITIES FOR THE FUTURE
62% tab9-3.txt
Summary: TABLE 9.3 AIRLINES (Air Carriers Operating under 14 CFR 121) Fatal Acci-
dents, Fatalities (Preliminary Data) 1993 Location Operator Date Serv. Aircraft Fatalities
Total Reported Type On- Type of Total Pass- Crew Others board Accident engers SCHED-
ULED SERVICE Chicago, IL Simmons 4/4 Psgr ATR 1 0 0 1 48 Ground Airlines 42-300
crewmember dba: struck American by Eagle propeller NONSCHEDULED SERVICE None
Source: National Transportation Safety ...
[Results by Excite]
FIGURE 3-2 Continued
Highway Administration, National Highway Traffic Safety Administration, and
the Federal Transit Administration, as well as BTS. At present, the archive con-
tains 11 listings, of which some are descriptions of data products rather than data
(see Figure 3-3~.
To help the user locate additional data and information, the BTS site pro-
vides links to the web sites of the other USDOT modal administrations (these
links are to the main administrations and not to their statistical units). The BTS
site also has links to many other organizations (commercial, private, government,
nonprofit, libraries) with some relation to transportation. The user can search any
and all of these sites for data; however, their design does not always facilitate
such a search. The BTS site itself provides no guidance for users in their search
of other sites. Such guidance could take the form of putting the Directory of
Transportation Data Sources on the BTS site, making its contents searchable by
keyword, and, when applicable, adding links to other web sites to obtain more
information or to see the actual data. Alternatively, such guidance could be pro-
vided through short essays that inform the user of major data series in particular
cross-modal topic areas and where to find them.
Data documentation The BTS web site gives no evidence of the application of
consistent standards for the information provided about the quality and limita-
tions of available data. The brief descriptions that are provided in the "Products"
section for BTS CD-ROM products vary in content and usefulness (see Appendix
E). Each data set listed in the National Transportation Data Archive (see Figure
3-3) has a contents page that links to the following headings: Detailed Descrip-
tion, Reports and Products, Searchable Database (operational as yet for only some
of the data sets), Questions and Comments, Methods and Limitations, Future
Plans, Applications, and Related Topics. This selection of headings appears po-
tentially very useful; however, to date, there is limited or no information provided
for such key headings as Methods and Limitations for many of the data sets in the
National Data Archive. Several of the data sets in the archive reproduce publica
OCR for page 59
FOCUSING ON DATA QUALITY
59
FIGURE 3-3 Contents of National Transportation Data Archive in "Databases" section
of BTS web site.
lions from other USDOT modal administrations (e.g., highway statistics); the
documentation that is provided in these publications about data collection meth-
ods and data quality varies according to the practices of the originating agencies.
At this stage of its development, it may not be feasible for BTS to standard-
ize the documentation for all of the data sets it makes accessible on its web site
from the other modal administrations (or other sources), although such standard-
ization should be a goal of the work to develop department-wide quality stan-
dards. However, BTS can and should move quickly to standardize documenta-
tion for its own data sets and also to standardize and improve the descriptions of
its CD-ROM and other products. It should also include on the site a prominent
statement to advise users about the importance of understanding the meaning and
limitations of available data sets before attempting to work with them. Providing
OCR for page 60
60 BUREAU OF TRANSPORTATION STATISTICS: PRIORITIES FOR THE FUTURE
contact names whenever possible for users to learn not only about the content and
scope of a data set, but also about its uses and limitations, would be helpful.
DATA EVALUATION AND IMPROVEMENT
Documentation of data quality and relevance is essential for users; however,
such documentation is necessarily limited to the extent that a data set has not been
evaluated on a range of dimensions. Major responsibilities of statistical agencies
are to evaluate their data and, on the basis of such evaluations, to inform users of
current limitations of the data and to develop ongoing programs to improve data
quality (see National Research Council, 1992b, and Appendix C). To support
continuing data improvement, agencies need to conduct statistical research on
data collection, processing, and estimation methods and also substantive research
on the issues for which the data are compiled (see discussion in Chapter 4~. BTS
should plan and begin to implement systematic programs of evaluation and im-
provement of key transportation data sets.
BTS Surveys
As a first priority, BTS should review the evaluations it has completed or has
under way for the two major intermodal surveys that it sponsors to determine
what further evaluations are needed and what the evaluation results imply for
appropriate use of the data and for future design decisions. These two surveys-
the Commodity Flow Survey and the American Travel Survey are the largest
component of the BTS budget, accounting for over one-third of BTS expendi-
tures over fiscal years 1994 to 1997, most of which represents funds transferred
to the Census Bureau for data collection and processing.
Some evaluations that the Census Bureau, working with BTS, has completed
of the 1995 ATS include assessments of"recallbias" and"time-in-samplebias."~4
Recall bias is inferred when respondents report a behavior more frequently for a
period closer to the interview than for a period that is farther away in time. Time-
in-sample bias is inferred when respondents change their behavior or reporting of
their behavior over successive interviews. Both types of bias are important to
evaluate for the ATS because it consisted of four interviews with the same house-
holds, each interview covering a 3-month reporting period.
Additional types of evaluation studies that would be important to undertake
for both the ATS and CFS include comparisons of the characteristics of survey
respondents with nonrespondents (including the implications of differences for
the accuracy of key survey estimates) and comparisons of selected survey esti
i4Two of the BTS staff are sworn census agents, so that they can work with confidential microdata
at the Census Bureau for evaluation purposes.
OCR for page 61
FOCUSING ON DATA QUALITY
61
mates with estimates from other data sources. For example, trip reporting behav-
ior could be compared for the 1995 ATS with the 1995 NETS and the 1995 Con-
sumer Expenditure Survey (CEX). Aggregate comparisons between two data
systems must be made with care to allow for differences in definitions and data
collection and processing procedures, but the identification of discrepancies can
lead to further research to determine reasons and suggest ways to improve one or
both data systems.
A comparative evaluation study that could be particularly useful for the ATS
concerns transportation costs. Because of a belief that households are poor re-
porters of costs, the ATS questionnaire does not ask about trip costs. BTS ex-
pects that the U.S. Travel Data Center, a private organization, will develop model-
based estimates of long-distance trip costs on the basis of trip characteristics.
When such estimates are developed, it would be useful to compare them with trip
cost information from the CEX.
For subsequent rounds of the CFS and the ATS, BTS should consider addi-
tional research and evaluation both prior to and as part of the surveys. For ex-
ample, cognitive research techniques could be used to evaluate and improve the
ATS questionnaire. Given the importance of information on travel costs, it could
be worthwhile to embed an experiment within the ATS in which trip costs are
obtained from a subsample of respondents and the completeness of their report-
ing is evaluated against other sources.
The results of evaluation studies should be used, together with assessments
of the usefulness of CFS and ATS data by BTS staff and other analysts, to guide
periodic reevaluations of the overall design of the two surveys. At present, the
plans for the two surveys are to continue the historical pattern of conducting them
at 5-year intervals with large sample sizes. (The 1995 ATS sample includes
80,000 households, the largest sample size of any U.S. national household sur-
vey.) The large sample sizes in the 5-year design are intended to support needed
subnational geographic analysis of interarea travel flows, but the cost is that up-
dates are available only at relatively long intervals.
An alternative design for the two surveys would be to have continuing small
samples that provide national estimates on, say, an annual basis and to augment
those samples periodically to obtain more detailed interarea data. Yet another
design would be to have small national samples with added samples each year for
specific areas that would "roll" across the country in some fashion. The rolling
sample design would be helpful in the congressional budget process, in that it
would smooth out peaks and valleys in required funding levels. However, the
subnational estimates it provides could be difficult to interpret because the infor-
mation for each year's area sample would necessarily pertain to transportation by
residents or shippers within the specified areas and not also to movements of
people or goods into those areas from nonsample areas.
Careful consideration of transportation analysis needs and of the costs of
alternative designs will be required to determine an optimal strategy for how
OCR for page 62
62 BUREAU OF TRANSPORTATION STATISTICS: PRIORITIES FOR THE FUTURE
often the surveys are fielded and the corresponding sample sizes and designs.
That strategy may turn out to be the current design of large surveys at 5-year
intervals; however, that design should be chosen on the basis of research, evalua-
tion, and user input and not just continued from the past. (Evaluation results
should also inform other design choices, such as length of recall period and ques-
tionnaire content and wording.~5 Finally, an assessment of the design of the
ATS, and perhaps of the CFS as well, should take into account other similar
surveys and the possibilities for coordinating or integrating their designs (see
Appendix F).
Other Data
Once BTS gains experience and expertise in evaluating its own data systems
and a reputation for excellence in this regard, then it will be in a position to advise
other USDOT statistical units about evaluation and improvement of their data
systems, particularly from the perspective of improving the usefulness of the data
for cross-modal, system-wide analyses of transportation issues. Such a role is in
keeping with the establishment of BTS in the 1991 ISTEA as the statistical agency
with a broad mandate to improve transportation data within the department.
BTS can begin immediately to work with the other USDOT modal adminis-
trations to identify additional information on data quality and limitations that
should be added to the descriptions on the BTS web site and in BTS compendia
and reference publications. (This work will naturally be part of BTS's strength-
ened mandate to develop data quality standards for USDOT.) Cooperative ef-
forts with other modal administrations to undertake more extensive documenta-
tion and to refine existing evaluation and improvement programs (or to launch
new programs) for their data systems will require a carefully planned and staged
approach. Work toward that end should be guided by a vision of transportation
data needs within which to identify priority areas for attention in the short, me-
dium, and long term (see Chapter 4) and by the data quality standards that are
developed by BTS in cooperation with the other modal administrations.
RECOMMENDATIONS
Staffing
(2) BTS should be authorized to appoint an associate director for statis
i5Another design choice is whether to include a longitudinal component, in which data are obtained
from the same reporting units over time, as was done to a limited extent in the 1995 ATS (households
were interviewed four times over 1 year). Longitudinal data permit analysis of complex behavior
patterns but can require significant resources and pose such problems as attrition (sample units drop-
ping out of the survey) and time-in-sample bias.
OCR for page 63
FOCUSING ON DATA QUALITY
63
tical methods and research at the Senior Executive Service level to provide
leadership in improving the quality of transportation statistics. BTS should
give priority to hiring highly qualified staff with expertise in statistical meth-
ods.
Quality Standards
(3) In the reauthorization of BTS, the Congress should strengthen cur-
rent law by assigning responsibility to BTS to establish data quality stan-
dards, consistent with good statistical practice, that are binding throughout
USDOT and available for use by transportation agencies outside USDOT
and for reference by the public. The reauthorization should also:
· require the secretary of transportation to appoint a departmental
standards committee, chaired by the BTS director and with representatives
from the USDOT statistical units, to work with BTS in developing depart-
ment-wide data quality standards and
· require BTS to prepare every 2 years a report to the Congress that
identifies improvements achieved in data quality by BTS and the statistical
units in the other USDOT modal administrations and in the provision of
information about quality to data users.
Data Documentation
(4) BTS should improve the documentation of the transportation data it
makes available so that users can readily assess their quality, including accu-
racy, variability, and comparability across transportation modes and over
time.
Representative terms from entire chapter:
transportation data
