Naturalistic Driving Study: Alcohol Sensor Performance (2015)

Below is the uncorrected machine-read text of this chapter, intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text of each book. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.

11 Findings and Applications Algorithm Considerations and Development Previous research efforts for the alcohol sensor and an exam- ination of trips known to have imbibed and unimbibed alco- hol presence created the framework for the development of an alcohol-detection algorithm using alcohol sensor data. These considerations and examples are detailed below along with a description of their translation into the final non- proprietary algorithm used in this research. Critical Considerations for Algorithm Development As shown in Figure 3.1, alcohol sensor readings are almost always defined by certain characteristics. In particular, the figure shows an unimpaired trip free from the presence of unimbibed alcohol; it also shows the warm-up period and “shadow” that characterize the alcohol sensor output. Almost all readings, even in the presence of alcohol, have a warm-up period and shadow. The shadow is seen in the figure as a set of sensor readings that mirror the primary readings and is typically 15 mV below the primary sensor output. Figure 3.1 also demonstrates a warm-up period before the alcohol sensor readings stabilize, which can range from a couple of seconds to a couple of minutes. In this and other graphs showing alcohol sensor readings, the y-axis represents alcohol sensor readings in mV. The x-axis represents time in milli seconds (ms). In the unimpaired trip represented by this graph, the baseline reading can also be observed when no alco- hol is present. These baseline readings are most often above 4,000 mV. The presence of alcohol causes the sensor reading to drop. The sensor was also sensitive to a variety of unimbibed alco- hol and other substances. These included windshield wiper fluid, hand sanitizer, chewing gum, fast food, cologne, per- fume, cigarettes, aerosol spray, glass cleaner, mouthwash, and other substances containing alcohol or an alcohol base. Many of these substances have very similar effects on the sensor. An example of sensor readings under various unimbibed sub- stances can be seen in Figure 3.2. Again, the y-axis represents mV, and the x-axis represents time (ms). This figure shows that sensor readings typically drop sharply at the introduc- tion of unimbibed substances and have a slow, often gradual, return to baseline. Fast food differs slightly, with a slow and less pronounced drop in sensor values. Additional investiga- tion was not warranted: it is unknown what components of the fast food excited the sensor or the breadth of food types that may have had an influence. Additionally, several positive cases of impairment were identified. These included clear visual evidence based on driver performance and video, drivers verbally reporting alcohol involvement using the critical incident button, video footage of police arresting a driver for driving under the influ- ence (DUI), or visual confirmation of alcoholic beverages in the vehicle. These trips were useful in developing algorithms that could detect alcohol-impaired drivers. Figure 3.3 shows an example of a trip with impaired pas- sengers. During this trip, several intoxicated individuals got into the back seat of a vehicle while at a stop sign. The indi- viduals had plastic cups with what was later reported verbally by the driver to be alcohol. In the associated video, the driver, who had not been drinking, pressed the critical incident button once the passengers exited to report that they smelled strongly of alcohol. The figure shows a dip in sensor readings when the intoxicated passengers entered the vehicle followed by a rise to baseline when they exited the vehicle, showing how sen- sor readings typically drop under the influence of intoxicated individuals. From these confirmed alcohol-imbibed trips, it was evident the alcohol sensor had the potential to detect the presence of imbibed alcohol regardless of the number of individuals in the vehicle who consumed alcohol or their position within the C h A p t e r 3

12 vehicle. The readings appeared to indicate a stronger sensor response with a decrease in readings when the individual who consumed alcohol was in the driver or passenger seat rather than in the back seat. However, a similar pattern of sensor read- ings was observed outside the magnitude of the drop in sensor readings based on a front-seat versus back-seat individual hav- ing a positive BrAC. From these cases there did not appear to be a difference in sensor response whether the individual who consumed alcohol was in the driver seat or the passenger seat. A scenario in which the driver with a positive BrAC was the sole occupant of a vehicle would produce sensor readings similar to a scenario in which the front-seat passenger was intoxicated and the driver was sober. These observations were consistent with the alcohol sensor setup. As a result of the sen- sor’s location near the rearview mirror, it was more exposed to the breath of the front-seat passenger and driver rather than back-seat passengers. Algorithm Development The information from the graphs in Figure 3.3 was used to create an alcohol-detection algorithm using the alcohol sen- sor to detect impaired individuals within a vehicle and to dif- ferentiate imbibed from unimbibed sources of alcohol. To remove bias from the performance metrics of the final algo- rithm, none of the trips used for development of the algo- rithm were included in any of the evaluation data sets. Across the files, it was observed that both the slope and absolute value of alcohol sensor readings were critical to understand- ing alcohol presence within a vehicle. Researchers considered the following while developing the alcohol algorithm, which should be used as guidance in future development: • A moving average allowed the sensor shadow to be elimi- nated by removing all points that deviated strongly from the average. • The absolute value of sensor readings was shown to relate to the presence of imbibed alcohol. While the strength of this effect can potentially vary from sensor to sensor, a threshold of 3,965 mV was established for the alcohol-detection algo- rithm. Any trip with an average below this level after con- trolling for unimbibed alcohol was classified as containing imbibed alcohol. • A quick change, or steep slope, in alcohol sensor readings was most often the result of unimbibed alcohol and should be considered in reducing false positives. • There appeared to be a large amount of variance in whether or not a particular trip had a warm-up period and the length of that period. For the purposes of this alcohol-detection algorithm, a warm-up period of 50 seconds was set. This helped ensure that short trips were not thrown out while removing some of the anomalous readings that could occur at the beginning of trip files. Mechanical Breather Boozooka Validation Before assessing the accuracy of the alcohol-detection algo- rithm on the experimental gold standard data set, the Boozooka needed to be validated for representativeness and accuracy. This was accomplished by comparing alcohol sensor readings Figure 3.1. Alcohol sensor shadow effect and example of warm-up period. M illi vo lts (m V) Milliseconds (ms)

13 Figure 3.2. Effect of various types of unimbibed alcohol on alcohol sensor. Windshield Wiper Fluid Hand Sanitizer Perfume Fast Food Cigarette Smoke

14 in the presence of the Boozooka with the controlled trips with intoxicated individuals. As shown in Figure 3.4, values for the first alcohol sensor (top) and second alcohol sensor (bottom) changed consistently as a function of the Boozooka’s activity. This operational control indicated the Boozooka worked at a fundamental level. Indeed, the sensor readings changed in a way that was consistent with other known impaired trips and were highly correlated. Direct comparisons between trips with an impaired passen- ger and the Boozooka were also conducted. These tests matched the BrAC levels of the individual to the Boozooka for direct comparisons. Figure 3.5 shows average alcohol sensor readings at various BrAC categories broken down by Boozooka and human testing. The pattern of sensor readings was consistent for both conditions—voltage decreased as BrAC increased. However, the average value of sensor readings was substantially lower for the Boozooka condition than for human testing. This was likely the result of setting the Boozooka to “breathe” 10 liters per minute, the average rate for an individual of medium-to-large build. The researcher used for human testing was substantially smaller in weight and likely also breathed at a much lower rate, resulting in lower volumes of alcohol entering the cabin at a similar BrAC. The consistent trend of sensor read- ings created confidence in the Boozooka as a reasonable proxy for an intoxicated individual. Albeit, the discrepancy served as a reminder that many factors can influence the volume of alcohol that enters a cabin. Future research could expand the Boozooka testing by looking at the influence of breathing rate across a larger spectrum of representative values. Mechanical Breather Gold Standard Data results The dual sensor setup (i.e., two head units each with an alcohol sensor installed in the same vehicle) provided a unique oppor- tunity to investigate how two sensors operated under virtually identical environmental conditions. Across all trips, the Pearson correlation between Alcohol Sensor 1 (AS1) and Alcohol Sensor 2 (AS2) was .989, p < 0.01. While this did not necessarily dem- onstrate that both sensors worked perfectly at assessing alcohol presence, it did indicate that the sensors behaved similarly across various levels of alcohol presence and trips. It should be noted that correlations are a measure of similarity of rank order. Thus, this strong correlation between sensors did not necessar- ily mean they provided identical readings but rather that their readings varied to similar degrees across time periods. The gold standard data set also allowed for the examination of whether or not BrAC differences could be detected using the SHRP 2 alcohol sensor. Pearson correlations between AS1 and BrAC were -.526, p < 0.01, and between AS2 and BrAC were -.534, p < 0.01. This indicated that differences in BrAC could potentially be detected within a given sensor. Figure 3.6 shows the average readings for each alcohol sensor across all trips categorized by ranges of BrAC; it also M illi vo lts (m V) Milliseconds (ms) Figure 3.3. Alcohol-involved trip.

15 Bo o zo o ka O ff Bo o zo o ka O n M illi vo lts (m V) Milliseconds (ms) M illi vo lts (m V) Figure 3.4. Change in sensor based on Boozooka on/off status (functional control). 3500 3550 3600 3650 3700 3750 3800 3850 3900 3950 4000 0.01–0.05 0.05–0.10 0.10–0.15 0.15+ Al co ho l S en so r R ea di ng (m V) Breath Alcohol Concentration Range (g/dL) Human Boozooka Figure 3.5. Alcohol sensor readings as a function of BrAC for Boozooka and human testing. 3300 3400 3500 3600 3700 3800 3900 4000 4100 Control 0.01–0.05 0.05–0.10 0.10–0.15 0.15+ Al co ho l S en so r R ea di ng (m V) Breath Alcohol Concentration Range (g/dL) Alcohol Sensor #1 Alcohol Sensor #2 Figure 3.6. Alcohol sensor readings as a function of BrAC by sensor.

16 shows that alcohol sensor readings dropped for both sensors as BrAC increased. However, although the absolute value of the alcohol sensor readings for AS1 and AS2 were similar under no alcohol presence, AS1 had a steeper decrease in sensor readings with increased BrAC. Thus, while sensor readings were sensitive to BrAC, the sensitivity appeared to differ across sensors. Without standardizing sensor read- ings or knowing the calibration for a given sensor, BrAC estimates from sensor values are likely not possible. Addi- tional research with several sensors would be necessary to confirm this finding. In addition to providing a better basic understanding of the alcohol sensors, the gold standard data set also helped evaluate an algorithm to detect in-vehicle alcohol presence by using a confusion matrix based on the principles of Sig- nal Detection Theory. In this confusion matrix, each trip was classified according to whether alcohol was actually pres- ent within a vehicle and the algorithm’s estimate of alcohol presence. For example, if alcohol was present in the vehicle but the algorithm estimated that no alcohol was present, then that instance would be classified as a false negative. A sample confusion matrix indicating four possible outcomes is displayed in Figure 3.7. Results for the alcohol-detection algorithm against the gold standard data set are presented in Figure 3.8. As shown in the figure, the sensitivity of the algorithm was 96.6%. The specificity of the algorithm was 100%. Overall, only three of the 100 trips were incorrectly categorized by the alcohol- detection algorithm. These trips were at low BrACs, and sensor readings were close to the threshold for detection. The results for the gold standard data set indicate that the algorithm performed well at differentiating trips based on alcohol presence. Naturalistic test Data results The naturalistic test data set contained 659 trips and was designed to be overrepresented with cases in which the alcohol sensor indicated a positive reading. This allowed researchers to examine the ability of the alcohol sensor to differentiate imbibed from unimbibed alcohol. Out of these 659 trips, unimbibed alcohol was found in 290 trips. While the percentage of trips with unimbibed alco- hol was inflated because of how trips were select, the relative frequency of various unimbibed substances in relation to each other was likely to be representative of the broader SHRP 2 database. Of the 290 trips, 58% (n = 167) contained windshield wiper fluid. This was followed by hand sanitizer (n = 26; 9%), cologne/perfume (n = 24; 8%), multiple sub- stances (n = 18; 6%), cigarettes/other drugs (n = 36; 13%), fast food (n = 12; 4%), glass cleaner (n = 4; 1%), and chewing gum (n = 3; 1%). Since this data set oversampled positive sensor readings, it was most useful for evaluating the alcohol-detection algo- rithm’s accuracy at differentiating between these positive readings. Table 3.1 is a contingency table depicting the accu- racy of the algorithm at differentiating all possible combi- nations of alcohol presence. The true state is shown in the columns of the table, and the algorithm response is shown in the rows. A chi-square test on the contingency table was sta- tistically significant, c2(9) = 365.4, p < 0.01. This result indi- cated that the alcohol-detection algorithm performed better than chance at estimating and differentiating alcohol pres- ence within a vehicle on this data set. For the aforementioned reasons, the number of false positives was inflated in this data set. However, the chi-square test shows that the alcohol- detection algorithm was moderately accurate at differentiat- ing alcohol presence. While the alcohol-detection algorithm was 91.7% accurate when there was imbibed alcohol, it was only 22.6% accurate at predicting unimbibed alcohol alone and 35.5% accurate at predicting the presence of both imbibed and unimbibed alcohol. Even though the number of false positives was inflated in this data set, it was possible to explore the alcohol algo- rithm’s accuracy at detecting alcohol presence in a confu- sion matrix. Though this bias toward alcohol-positive cases should be considered when interpreting the results, a con- fusion matrix was created by classifying trips as imbibed alcohol present or no imbibed alcohol present. This was Figure 3.7. Confusion matrix. True State Alcohol present No alcohol present Algorithm Response Alcohol present True positive False positive No alcohol present False negative True negative Figure 3.8. Confusion matrix for gold standard data set. True State Alcohol present No alcohol present Algorithm Response Alcohol present 85 0 No alcohol present 3 12

17 done by categorizing trips coded as “imbibed alcohol only” and “both imbibed and unimbibed alcohol present” in an “alcohol present” classification. Similarly, trips that were coded as “unimbibed alcohol only” or “neither imbibed nor unimbibed alcohol present” were categorized as “no alco- hol present.” The results of the confusion matrix are depicted in Figure 3.9. The sensitivity was 93.7% and the specificity was 36.9%. Again, this range was likely the result of the inflated number of false positives, and the algorithm likely performed worse than it would on the entire SHRP 2 database. As mentioned, a subset of the naturalistic test data set was extracted to form the impaired data set. The ability of the alco- hol sensor algorithm to differentiate the type of alcohol pres- ence within the vehicle was evaluated using a chi-square test on the impaired data set. The chi-square contingency table is shown in Table 3.2, with rows representing the algorithm response and columns representing the estimated true state. The overall chi-square test was significant, c2(9) = 259.95, p < 0.01, indicating the alcohol sensor algorithm performed better than chance at estimating alcohol presence. It should be noted that the accuracy of the alcohol-detection algorithm led to many cells in the contingency table having fewer than five observations, which can bias chi-square values. The table shows that the alcohol-detection algorithm was particularly accurate at estimating no alcohol presence and impaired driver or passenger trips. Figure 3.9. Confusion matrix for alcohol-detection algorithm against naturalistic test data set. True State Imbibed alcohol present No imbibed alcohol present Algorithm Response Imbibed alcohol present 194 285 No alcohol present 13 167 Table 3.1. Contingency Table Showing Algorithm Accuracy on the Full Naturalistic Test Data Set Algorithm Response True State Total Percent Accurate No Alcohol Present Imbibed Alcohol Unimbibed Alcohol Both Types of Alcohol No Alcohol Present 101 1 1 0 103 98.1% Imbibed Alcohol 99 133 107 33 372 35.7% Unimbibed Alcohol 12 5 53 7 77 68.8% Both Types of Alcohol 5 6 74 22 107 20.5% Total 217 145 235 62 659 Percent Accurate 46.5% 91.7% 22.6% 35.5% Table 3.2. Contingency Table Showing Algorithm Accuracy on the Impaired Data Set Algorithm Response True State Total Percent Accurate No Alcohol Present Imbibed Alcohol Unimbibed Alcohol Both Types of Alcohol No Alcohol Present 96 1 0 0 97 99.0% Imbibed Alcohol 0 54 0 16 70 77.1% Unimbibed Alcohol 0 3 2 3 8 25.0% Both Types of Alcohol 0 3 0 10 13 76.9% Total 96 61 2 29 188 Percent Accurate 100.0% 88.5% 100.0% 34.5%

18 The confusion matrix for the impaired data set is shown in Figure 3.10. The sensitivity of the algorithm against this data set was 92% and the specificity was 100%. This indicated the alcohol-detection algorithm performed extremely well when trips were classified with greater certainty of imbibed alcohol involvement. In particular, the algorithm was not prone to false negatives, suggesting that alcohol-detection performance increased as the observable signs of intoxication increased.Figure 3.10. Confusion matrix for impaired data set. True State Alcohol present No alcohol present Algorithm Response Alcohol present 83 0 No alcohol present 7 98