Appendix C

Item Response Theory Example Using Motor Carrier Management Information System Data

Jacob Spertus

The following provides a brief worked example of an item response theory (IRT) model as discussed in Chapters 3, 4, and 5 applied to actual Safety Measurement System data. It uses a very small proportion of the full data, purposefully selected for quality and carrier size. It is not intended to be a comprehensive, unbiased analysis but merely a proof of concept. A full implementation would look roughly similar, but with many more carriers and violations included.

This appendix is included to show that the proposed model is relatively easy to implement in the Motor Carrier Management Information System. However, this model is in many ways a toy model. It is only applied to 865 carriers (not 200,000) and only unsafe driving violations that appeared for 5 or more carriers are included; there are none of the features, including multidimensionality (in fact, only one BASIC is used); and the priors have not been updated over time. Therefore, the quality of the fit should not be seen as an indication of the quality of the fit of the full model as represented in Chapters 4 and 5.

Suppose for each carrier i ∈ {1,…,C} we have a measure of exposure E_i, a count of eligible inspections n^k_i, and a count of violations y^k_i, where k ∈ {1,…,K} indicates the type of violation. Note that inspections are superscripted with k because the type of inspection determines which violations can be recorded. We propose the following model:

The parameter λ in equation (C-1) estimates the rate of inspections per vehicle miles traveled (VMT) across the population. p_ik in equation (C-2) represents the probability of carrier i receiving violation k at a given inspection. It is modeled as a logistic function of the prevalence (or difficulty) parameter β_k, which reflects the marginal prevalence of violation k in the data, and discrimination parameter α_k, which reflects the association of violation k and latent safety. Thus for a given violation k a higher value of β_k indicates that it is observed more frequently, and a higher value of α_k indicates it is more associated with safety or danger.

To implement this model on a small scale, we use SMS data from 2014 to 2015, selecting a subset of the carrier population and including only unsafe driving violations. The carrier subpopulation are those carriers with 100 or more average power units (APUs), more than 80,000 VMT per APU reported, and less than 200,000 VMT per APU reported (medium to high utilization carriers according to SMS methodology). This leaves only 865 carriers. Finally in order to set a lower bound on sparsity, we drop violations that occur in less than 5 carriers. After this processing, 35 different types of unsafe driving violations (features) remain for 865 carriers (observations).

We used the ‘rstan’ interface to Stan for Hamiltonian Monte Carlo to fit the fully Bayesian models described above (Stan Development Team 2016). In the first step, a simple Poisson model (equation (C-1) is fit. For exposure E_i, we take 100,000 VMT. λ is specified with a non-informative prior and thus gives, essentially, the average inspections per 100,000 VMT. The observed mean in the population is 2.014, while the posterior mean (95% CI) of λ is 1.806 (1.802, 1.811)

The second step uses equations (C-2) and (C-3), and priors (C-4), (C-5), and (C-6), to fit a binomial IRT model. The prior on θ_i is selected to enforce identifiability of the model. The log-normal prior on α_k, which has support only over the positive real numbers, encodes the assumption that α_k is strictly positive. We provide default choices for the priors on β_k and

α_k by matching the specification in the ‘edstan’ R package for IRT models (Furr, 2017). Note that substantive knowledge about the effect of certain violations on safety could be included via the prior (C-6). The ‘trials’ for this IRT model are inspections eligible for unsafe driving violations, that is, driver inspections (level 1, 2, 3, or 6). A brief caveat: unsafe driving violations (e.g., speeding) often precipitate an inspection, so inspections might not be the best denominator for these violations. VMT might be a better exposure variable, requiring a Poisson model with VMT as offset.

Figure C-1 shows the characteristic curves for θ. The probability of receiving a given violation at an inspection is plotted against θ. Thus more unsafe carriers (with higher θ) receive more violations over all, and certain violations with especially high frequency.

The top 20 most discriminating violations (highest α) are shown in Table C-1.

Figure C-2 plots unsafe driving violations per inspection against θ. There is a very close mapping between these two metrics, such that θ more or less returns the number of unsafe driving violations per inspection.

Figure C-3 plots crashes per 100,000 VMT against θ. There is a positive relationship but also very high variance about the least squares line (low R²). The Spearman correlation between crashes per 100000 VMT and θs from the binomial IRT is 0.23.

TABLE C-1 α, β, and Prevalence for the 20 Violations with the Highest α, Value Generated from Binomial Item Response Theory

NOTE: CMV, commercial motor vehicle.

REFERENCES