Summary
Commercial motor vehicle (CMV) companies are responsible for moving freight and passengers in a safe manner over the nation’s highways. To help ensure a high standard of safety, the Federal Motor Carrier Safety Administration (FMCSA), with its mission “to reduce crashes, injuries and fatalities involving large trucks and buses,” makes use of the Compliance, Safety, and Accountability Program, and, in particular, the Safety Measurement System (SMS), to rank CMV carriers by the degree to which they operate safely. This ranking is a function of the frequency of different groups of violations assessed during (mainly) roadside inspections, or is a function of the frequency of crashes that a carrier has, or both over the most recent 2-year period. SMS is used as a prioritization tool that allows the agency to target for interventions motor carriers that are operating unsafely and therefore are likely to be at higher risk for future crashes.
SMS partitions 899 possible violations that can arise from roadside inspections into six groups and forms a metric for each of these groups that are weighted frequencies of violations. SMS augments these six metrics with another metric, crash frequency (again weighted). These seven metrics are referred to as Behavior Analysis and Safety Improvement Categories (BASICs). The noncrash BASICs make use of groups of violations that are associated with similar types of unsafe practices: Unsafe Driving, Hours-of-Service Compliance, Vehicle Maintenance, Controlled Substances/Alcohol, Hazardous Materials Compliance, and Driver Fitness. The seventh BASIC is referred to as Crash Indicator. For each carrier with enough inspections, violations, and crashes to meet FMCSA’s data
sufficiency standards, these seven measures are computed, and carriers are ranked within the peer groups to see which have the greatest frequency of crashes or violations. FMCSA has set thresholds within each of these groups, and carriers that rank above these thresholds are subject to a range of interventions, which include warning letters, on-site investigations, and more serious actions such as fines and suspension of operations.
Some stakeholders and outside reviewers, among others, have criticized SMS for, among other things: (1) making use of highly variable assessments, (2) not accounting for crashes where the motor carrier is not at fault, (3) including carriers that have very different tasks in the same peer groups, (4) using measures that are sensitive to effects from one or more individual states, (5) using measures that are not predictive of a carrier’s future crash frequency, or (6) using measures that are not reflective of a carrier’s efforts to improve its safety performance over time.
Given the stakes involved, it is important to examine SMS to see whether these and other criticisms are valid, and to examine the performance of this system to see whether improvements can be made. The need for this review of SMS was written into the Fixing America’s Surface Transportation (FAST) Act of 2015, in which it is recommended that FMCSA fund a study by the National Academies of Sciences, Engineering, and Medicine to evaluate SMS. Two units in the National Academies—the Committee on National Statistics in collaboration with the Transportation Research Board—began work in March 2016, convening the Panel on the Review of the Compliance, Safety, and Accountability Program of the Federal Motor Carrier Safety Administration for this congressionally mandated study. The panel was charged with analyzing the ability of SMS measures to discriminate between low- and high-risk carriers, assess the public usage of SMS, review the data and methodology used to calculate the measures, and provide advice on additional data collection and safety assessment methodologies. The panel met in June 2016 and three additional times prior to issuing this report. These meetings, reinforced by additional research and analysis, resulted in six recommendations by the panel, which are presented in this Summary and explained in more detail throughout the full report.
The FMCSA has as its mission to prevent commercial motor vehicle–related injuries and fatalities. SMS is a prioritization tool that allows the agency to identify motor carriers with safety compliance problems for intervention by FMCSA. The agency’s Motor Carrier Management Information System (MCMIS), which contains data from commercial motor vehicle crashes, carrier registrations, commercial motor vehicle inspections, and inspection violations, is used for input into SMS. The state-based inspection system that feeds into MCMIS, using inspectors trained
by the Commercial Vehicle Safety Alliance (CVSA), inspects more than 3 million commercial motor vehicles every year in the United States in order to determine whether truck and bus carriers are operating in violation of safety regulations. FMCSA deserves considerable credit for making use of these data in an attempt to discriminate between safe and unsafe motor carriers.
ASSESSMENT OF THE CURRENT SAFETY MEASUREMENT SYSTEM
Multiple factors contribute to crashes, many of which are not present in MCMIS. Given that, and the relative rarity of crashes for small carriers, the panel agrees that development of a crash prediction model based on carrier-level behavior using MCMIS data is not a productive way to approach the problem of discrimination between safe and unsafe commercial motor carriers. With SMS, FMCSA has instead adopted a sensible, related approach based on prevention rather than prediction. That is, SMS has the objective of identifying carriers that give too little priority to practices indicative of safety performance. By intervening with those carriers, the hope is to encourage them to modify their behavior and, by so doing, reduce future crashes. We believe that the general approach taken by SMS is sound, and shares much with similar programs in other areas of transportation safety. Further, we have examined, to the extent possible, the various issues that have been raised in criticism of SMS. We have found, for the most part, that the current SMS implementation is defensible as being fair and not overtly biased against various types of carriers, to the extent that data on MCMIS can be used for this purpose.
However, we believe some features of SMS implementation can be improved upon, and some of the details of the implementation are ad hoc and not fully supported by empirical studies. Many of these details of implementation would be easily addressed if the algorithm currently used were replaced by a statistical model that is natural to this sort of discrimination problem. Therefore, we reached the following conclusion:
CONCLUSION: The Safety Measurement System (SMS) is structured in a reasonable way, and its method of identifying motor carriers for alert status is defensible. However, much of what is now done is ad hoc and based on subject-matter expertise that has not been sufficiently empirically validated. This argues for the Federal Motor Carrier Safety Administration adopting a more statistically principled approach that can include the expert opinion that is implicit in SMS in a natural way.
A MORE NATURAL STATISTICAL MODEL
The general approach taken by SMS can be shown to be related to item response theory (IRT) models that have been successfully applied in several similar contexts. These types of models accumulate zero-one responses to “tests,” using the results to differentiate between “test takers” that have different latent traits. These are models that describe the relation between where an individual falls on the continuum of a given construct, such as depression, and the probability that he or she will give a particular response to a scale item designed to measure that construct. In IRT, such a construct is called a latent trait, because that trait is assumed to underlie and directly influence responses to items on the scale designed to measure that trait. Examples include assessment of elementary and high school teachers, and determination of hospital rankings. In these and other areas of application, IRT models have been shown to be very effective. Given various theoretical advantages, we recommend the following:
RECOMMENDATION: The Federal Motor Carrier Safety Administration (FMCSA) should develop the suggested item response theory (IRT) model over the next 2 years. If it is then demonstrated to perform well in identifying motor carriers for alerts, FMCSA should use it to replace the Safety Measurement System (SMS) in a manner akin to the way SMS replaced SafeStat. Specifically, IRT models would have the following specific advantages over SMS:
- Instead of severity weights being based on expert opinion or dated empirical information, the item discrimination parameters are estimated based on a combination of current observed data and expert opinion, and ultimately on data alone.
- IRT models can enhance the transparency of the evaluation system.
- They support the direct estimation of variability of scores and ranks.
- They can account for the probability of being selected for inspection.
- They can provide a basis with which to evaluate how data insufficiency could impact safety ratings of carriers.
- They can provide a basis to more rigorously evaluate the structure of the current Behavior Analysis and Safety Improvement Categories (BASICs), including which violations go into which BASIC.
- They can provide for a natural way to examine the issue of further stratification.
- They can provide for the possibility that safety is inherently multidimensional, which could inform how many BASICs are needed in the SMS model.
- They can take account of time and thereby inform about the proper time weights in SMS.
- They can allow for the addition of new safety measures as they become available, without having to start from scratch.
- They can produce ranking ranges (by sampling from the posterior distribution of theta) to better understand overlap in the measures (i.e., uncertainty).
- They can adapt to changes in safety over time.
DATA IMPROVEMENTS
We considered data improvements in two respects. First, there are possible improvements to the variables collected in MCMIS that would not require major changes in what is currently done. Second, there are variables that would benefit SMS that would need alternative sources for their collection.
Improvement of MCMIS Data
The two most important areas in which improvements could be made to the information that MCMIS collects are in exposure data and crash data. While updates are required for data on vehicle miles traveled (VMT) and the average number of power units (APU) every 2 years, the impact of flawed or out-of-date VMT and APU data on SMS percentile ranks may not be fully appreciated by the carriers. Therefore, increased efforts are needed to acquire better data. Also, a sizable fraction of crash data is missing, and these data are collected in a nonstandard manner across states. Further, much of the information provided by police reports is not represented on MCMIS. Therefore, we make the following recommendation:
RECOMMENDATION: The Federal Motor Carrier Safety Administration (FMCSA) should continue to collaborate with states and other agencies to improve the quality of Motor Carrier Management Information System (MCMIS) data in support of the Safety Measurement System (SMS). Two specific data elements require immediate attention: carrier exposure and crash data. The current exposure data are missing with high frequency, and data that are collected are likely of unsatisfactory quality. Further, to improve the
exposure data collected involves collecting not only higher-quality vehicle miles traveled data but also this information by state and by month. This will enable SMS to (partially) accommodate existing heterogeneity in the environments where carriers travel. Crash data are also missing too often. Also, there is information available from police reports currently not represented on MCMIS that could be helpful in understanding the contributing factors in a crash. Such information could help to validate the assumptions linking violations to crash frequency. To address these issues, FMCSA should support the states in collecting more complete crash data, and in universal adoption of the Model Minimum Uniform Crash Criteria, as well as developing and supplying the code needed to automatically extract the data needed for the MCMIS crash file.
Improvement through Additional Variables and Possible New Sources
The information available on MCMIS is limited in terms of the ability to determine the factors that contributed to a crash. As a previous National Academies of Sciences, Engineering, and Medicine panel (2016) argued, we believe that it is reasonable to think of the causes of CMV crashes grouped into four categories, due to: (1) characteristics of the driver, (2) characteristics of the vehicle, (3) the driving environment, and (4) practices and procedures of the carrier. MCMIS is relatively good at capturing many characteristics of the vehicle and some aspects of the driver and the environment. However, it is incomplete regarding carrier operations. Since SMS is founded on the belief that a substantial fraction of crashes are due at least in part to carrier operations—and it is those operations that SMS is attempting to modify—it is clearly important to consider how to gain knowledge of carrier-related factors when making improvements to SMS. Further, in any study of which factors contributed to crashes, omission of any important (confounding) factors impairs the analysis.
Therefore, it would be extremely useful to know the carrier’s cargo, the driver turnover rate, and the level of driver compensation. Then, should FMCSA issue an intervention to a carrier, it would be informative to see what aspects of carrier operations were modified to try to address the intervention. Toward this end, we suggest that FMCSA look into how the following carrier characteristics might be collected externally to MCMIS:
- Information on turnover rate: This information could be very predictive of a company’s treatment of its employees, which
-
could be related to safety operations. In addition, a low turnover rate is likely associated with employment of drivers with longer tenures and hence greater experience.
- Information on type of cargo carried: Since current questions on type of business are producing lower-quality information, it might be preferable to ask a carrier about their typical cargos. The response to that question is nearly the same as type of business and might be easier to answer. (This could certainly be collected through the MCS-150.)
- Information on compensation level and method: It is known that drivers who are better compensated, and those not compensated as a function of miles traveled, have fewer crashes.
- Better information on exposure: We believe that state tax information is a possible source of high-quality VMT data, and therefore, we suggest that FMCSA interact with state taxing authorities to see whether an interagency agreement can be struck to share this information in support of SMS. In addition, at the end of 2017, electronic on-board recorders will be required for most carriers, and results for all carriers could be reported to FMCSA. Having the number of vehicle miles traveled at the end of the year would be extremely easy to produce and would be definitive.
We, therefore, make the following recommendation:
RECOMMENDATION: The Federal Motor Carrier Safety Administration (FMCSA) should investigate ways of collecting data that will likely benefit the recommended methodology for safety assessment. This includes data on carrier characteristics—such as information on driver turnover rate, type of cargo, method and level of compensation, and better information on exposure. This additional data collection will likely require additional funds for research and development of the data collection instrument, and greater collaboration between FMCSA and the states as to how to undertake this new data collection effort so that it is standardized across the states. Protection and use of carrier-specific data must be addressed as well.
TRANSPARENCY, REPRODUCIBILITY, AND PUBLIC DISCLOSURE OF SAFETY RANKINGS
SMS percentile ranks have very important implications for CMV carriers. Hence, it would be useful if the CMV community were able to reproduce the SMS measures and percentile ranks, that researchers have
easier access to the SMS algorithm and the MCMIS database, that FMCSA better communicate with researchers about how SMS functions, and that carriers be able to know the implications of recent violations and crashes on their measures and percentile ranks. Therefore, we make the following recommendation:
RECOMMENDATION: The Federal Motor Carrier Safety Administration (FMCSA) should structure a user-friendly version of the Motor Carrier Management Information System data file used as input to the Safety Measurement System (SMS) without any personally identifiable information to facilitate its use by external parties, such as researchers, and by carriers. In addition, FMCSA should make user-friendly computer code used to compute SMS elements available to individuals in accordance with reproducibility and transparency guidelines.
The panel was asked to comment on whether SMS percentile ranks should be made public. We are unable to recommend to FMCSA whether to make all SMS percentile ranks public. An understanding of the consequences of public consumption of the information requires a formal evaluation, possibly designed using randomization or controlled release of specific components of the SMS percentiles. In particular, what is needed to know is the current operating characteristics of SMS. That is, given that SMS can assess whether a carrier was one that should or should not have received an intervention given its safety behavior, what is the false positive and false negative rate of SMS? Given this, we make the following recommendation:
RECOMMENDATION: The Federal Motor Carrier Safety Administration should undertake a study to better understand the statistical operating characteristics of the percentile ranks to support decisions regarding the usability of public scores.
SMS percentile ranks are a relative metric, and so a motor carrier’s efforts toward improving its safety performance will not be reflected in the percentile ranks if other carriers in its peer group have improved even more. On the other hand, a relative score has the advantage that if FMCSA sets an absolute standard of performance, since the entire industry is getting progressively safer, the standard will at some point become irrelevant. Having a relative metric permits FMCSA to keep pressing for better performance. Further, FMCSA operates on a fixed budget, and how it functions is consistent with a relative measure. Since there are advantages of both relative and absolute measures, we believe that FMCSA
should strongly consider use of a two-dimensional measure that takes into consideration both the SMS score and the percentile rank, using some objective formula, to decide on which carriers will receive interventions.
RECOMMENDATION: Given that there are good reasons for both an absolute and a relative metric on safety performance, the Federal Motor Carrier Safety Administration should decide on the carriers that receive Safety Measurement System (SMS) alerts using both the SMS percentile ranks and the SMS measures, and the percentile ranks should be computed both conditionally within safety event groups and over all motor carriers.
