Naturalistic Driving Study: Technical Coordination and Quality Control (2014)

Below is the uncorrected machine-read text of this chapter, intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text of each book. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.

16 C h a p t e r 1 Background In 2005, Congress approved the second Strategic Highway Research Program (SHRP 2) to define a program of research into roadway safety and congestion. To support that program effort, it was decided that a major aspect of the Safety area of SHRP 2 would be to conduct a large-scale naturalistic driv- ing study (NDS) that would provide unique insights into the way people drive, what else they are doing while driving, and what situations and activities precede particular crash-related event types. The goal of this unprecedented data collection effort was to create a rich data resource for researchers, regu- lators, advocates, students, and other interested parties all over the world to analyze and address many of the key trans- portation safety research questions for at least a generation to come. In 2007, the SHRP 2 Study Design project (S05) was under- taken to plan all aspects of the SHRP 2 NDS. Its outcome was a comprehensive study plan for the SHRP 2 NDS, ranging from the development of research questions (which guided all sub- sequent activities) to the definition of the onboard data col- lection system, sampling targets, and driver assessment plans. It further addressed how the study would be managed and administered across the distributed data collection sites. Qual- ity assurance was considered for all phases of the study, includ- ing assessing and assuring the quality of all collected data. The outcomes of this planning effort are documented in the Study Design final report, Design of the In-Vehicle Driving Behavior and Crash Risk Study (Antin et al. 2011). On the heels of the Study Design project, the Virginia Tech Transportation Institute (VTTI), the Project S06 Coordination Contractor, began implementing the framework established in Antin et al. (2011) to conduct the ambitious research task of collecting, archiving, reducing, and beginning to analyze this extensive database of driving and related data. The scope of this study ranks among the largest of any driving-based research study conducted in the world to date, facilitated by recent technological advances in collecting, storing, com- pressing, and mining data. This final report for Project S06 provides a summary of the oversight and coordination efforts for the SHRP 2 Naturalis- tic Driving Study (NDS) with special emphasis on the techni- cal coordination and quality control. One goal of this report was to capture the key elements of the study so that it can be replicated (or improved on) if a consortium of individuals so desires in the future. Another goal was to provide sufficient details such that all researchers who desire to analyze the data will have sufficient understanding of how the data were col- lected to make appropriate analytical and scientific decisions during the course of their data mining/analysis efforts. Objectives of the technical Coordination and Quality Control project The Coordination Contractor was tasked with the technical coordination and quality control of the SHRP 2 NDS. The objectives of this effort entailed the following: • Human subjects protection, including design of the IRB consent forms and application for the Certificate of Confi- dentiality and design of the participant enrollment protocols; • Development of the driver assessment protocols; • Design and development of all custom installation, align- ment, and calibration hardware; • Design and development of all custom software to support bench testing, installation, shakedown testing, maintenance, and deinstallation; • Design and development of all custom hardware and soft- ware in support of the vehicle-based data collection; • Design of the data management system including collec- tion, encryption, staging, transmission, security, storage, and quality assurance; and • Design of the inventory management software and protocols. Background and Objectives

17 DAS Data The DAS collected time series sensor data from a suite of inte- grated sensors and cameras as listed below. Note that each stream of data was collected at its own native frequency, and these details can be found in the Time Series Data Dictionary in the SHRP 2 Data Access website (https://insight.shrp2nds.us). • Multiple video views: 44 Forward roadway, driver’s face and upper torso, driver interactions with wheel and center stack, and rear and right of the vehicle; • Periodic still photo of cabin (permanently blurred); • Machine-vision-based applications: 44 Head pose monitor, and 44 Lane tracker; • Accelerometers (x, y, and z axes); • Rate sensors (x, y, and z axes); • GPS: 44 Latitude, longitude, elevation, time, velocity; • Forward radar: 44 X and Y positions, and 44 Xdot and Ydot velocities; • Illuminance sensor; • Passive cabin alcohol presence sensor; • Incident pushbutton: 44 Marks data and opens a 30-s audio recording channel; • Turn signals; and • Vehicle network data (as available; examples listed below): 44 Accelerator, 44 Brake pedal activation, 44 Antilock braking system (ABS), 44 Gear position, 44 Steering wheel angle, 44 Speed, 44 Horn, 44 Seat belt information, and 44 Airbag deployment. Demographics A variety of basic demographics data were collected from each participant during the screening and enrollment pro- cesses. Examples include birth date, gender, marital status, and profession. Additional information was gathered regard- ing other members of the household, vehicle counts, and licensure information. Driver Assessments Participants were evaluated along several dimensions of func- tional ability relevant for driving safety and risk. In this way, In addition to these objectives, the overarching task was to coordinate and oversee the participant- and vehicle-based operations at each of the six data collection sites. The desired results included the collection of valid participant, vehicle, driving, and crash-related data collected via interviews and questionnaires as well as from continuous onboard sensors and video cameras. The collected data had to be stored securely in a manner that protected the rights and privacy of the more than 3,000 participants enrolled in the study. This study involved a wide range of activities, including the coincident design and purchase of the onboard data acquisi- tion system (DAS) equipment (actually procured under Proj- ect S12A, DAS Procurement). The major efforts fell into six key task categories described below. Human Subjects Protection Efforts to coordinate the human subjects protection and IRB requirements of multiple IRBs began a full 2 years before the first NDS study vehicle was installed. Throughout the SHRP 2 NDS, the task of protecting participants and their data has remained among the most critical. The study sponsor and all contractors were fully committed to protecting the data and identity of study volunteers and to treating them in a manner that complies fully with the requirements of the Virginia Tech and National Academy of Sciences (NAS) IRB policies and pro- cedures. Further, these policies were drawn directly from the Code of Federal Regulations, Title 45 Public Welfare, Depart- ment of Health and Human Services, Part 46, Protection of Human Subjects (45 CFR 46), widely regarded as the gold stan- dard of guiding principles in human subjects research. In addition, issuance of a Certificate of Confidentiality was initiated through the National Institutes of Health (NIH). The Certificate of Confidentiality served to protect participants from forced disclosure requests originating in the legal arena, such as subpoenas, that might seek identifying information or data to be used against them in a court of law. This certificate protects the identity of the participant, and thus the identify- ing research information and data collected during the term of the Certificate of Confidentiality; however, the identity of con- sented participants is intended to be protected in perpetuity. Data Categories To fulfill the goals of this study, several categories of data were collected, each encompassing from several to many individual data items. These categories are described below at a high level; details on each category and individual data element can be found in the Data Dictionaries located on the SHRP 2 Data Access website (https://insight.shrp2nds.us). As of December 20, 2013, registration was required to access the site; there is no cost associated with the registration.

18 intended to serve as very gross metrics of upper- and lower- body strength and capability. Upon exiting the study, participants were given the oppor- tunity to express their feelings about their experiences during the study by completing an Exit Survey (Appendix X). Vehicle Information In addition to make, model, year, and vehicle category (i.e., car, van, sport utility vehicle, or truck), the presence of integrated onboard technologies was recorded, including communica- tions features such as Bluetooth cell phone and OnStar. Post Hoc Crash Investigations One of the primary motivations for this study was that it would afford the opportunity to record and observe actual crash- related events (i.e., crashes and near crashes) from multiple perspectives, including the participant’s behaviors leading up to the event, and with high temporal resolution kinematics information synchronized with the video images. In addition to all of these DAS-based crash data, site con- tractors also performed more traditional post hoc crash ana- lyses for some of the most salient crashes. In most cases, site contractors were informed by the participant that a crash had taken place, though the timing of such notifications relative to the crash event varied widely. A rubric or guide was created by the Coordination Contractor to help site contractors determine for each crash of which they were aware whether or not to conduct a post hoc investigation, and, if so, whether to conduct a Level 1 or Level 2 investigation. The rubric incorporated researchers can correlate any one or more scores on these metrics of functional ability to the safety-related or other out- comes observed in the naturalistic driving record. The entire suite of assessments was designed to be precisely replicable across the six data collection sites and to be conducted within a 2-hour window, coinciding with vehicle installation. Ques- tionnaires could be filled in at the installation site or later, as per the participant’s needs; paper versions were also avail- able if requested by a participant. Assessment protocols are detailed in Appendix A. Questionnaires were used to assess sleep, health, Attention Deficit Hyperactivity Disorder (ADHD), sensation seeking, risk perception and behaviors, and driving knowledge and behaviors (Table 1.1). A complete listing of every item on every questionnaire can be found in the Data Dictionaries (https://insight.shrp2nds.us). Other assessments included a wide range of vision tests conducted using an Optec 6500P Vision Analyzer (Table 1.2) and metrics of cognitive function- ing (Table 1.3). Finally, Table 1.4 lists the tests of physical ability Table 1.1. Assessment Questionnaires Administered Name Description Sleep Questionnaire A questionnaire designed to determine the participant’s sleeping patterns, habits, and level of fatigue (Appendix B). Perception of Risk Survey and Frequency of Risky Behavior Questionnaires A questionnaire designed to gauge the participant’s perception of dangerous or unsafe driving behaviors or scenarios and a questionnaire designed to gauge the frequency and a participant’s willing- ness to engage in dangerous, unsafe, or risky behaviors (Appendix C). Barkley’s ADHD Quick Screen A short, clinical ADHD screening assess- ment. This screening instrument opera- tionalizes ADHD symptoms in terms of specific behaviors (Appendix D). Sensation Seeking Scale A survey comprising questions to gauge the degree to which the participant engages in sensation seeking behavior. The test measures the participant’s sensory stimu- lation preferences (Appendix E). Driving Knowledge A test of knowledge of driving laws and appropriate behaviors (Appendix F). Medical Conditions and Medications Survey and Exit Survey Questionnaires designed to obtain partici- pants’ self-reported medical history. The questions focus on the identification of conditions that could affect driving per- formance and safety (Appendix G). Modified Manchester Driver Behavior A self-reported driver behavior survey. The participant is asked to indicate how often he/she commits each described error (accidental) or violation (deliberate) (Appendix H). Table 1.2. Vision Tests Near/Far Visual Ability Monocular/ Binocular Day/ Night Glare/ No Glare Far Acuity Binocular Day Near Acuity Binocular Day Contrast sensitivity Monocular Day Contrast sensitivity Monocular Night Glare Contrast sensitivity Monocular Night No Glare Depth perception Binocular Color perception Binocular Peripheral vision Monocular

19 Traffic Safety Administration’s National Motor Vehicle Crash Causation Survey (NMVCCS). These interview questions were delivered electronically with branching functionality incorpo- rated. Only the relevant portions were visible to the person submitting information. A Level 2 post hoc crash investigation included everything involved in a Level 1 analysis; in addition, it involved visit- ing the site of the crash to document it photographically and to construct a crash site diagram using specialized software. This same software package was also used to collect all of the Level 1 or 2 post hoc crash data for a crash in a single PDF file. Note that active crash scenes (i.e., with vehicles still at final rest and emergency personnel on site) were never visited as a part of any SHRP 2 post hoc crash investigation. DAS Management The management of the DAS involved procurement and acceptance testing, as well as distribution and warranty/ repair efforts throughout the course of the study. This entailed ensuring that the equipment was purchased and inventoried in a manner compliant with federal regulations and contrac- tual obligations. Further, it required ensuring that equipment was built to rigid performance specifications and repaired as needed. Acceptance testing was performed at both the component level and in the context of a full system before being factors related to crash severity (i.e., injuries, speed, and prop- erty damage), roadway factors, driver age, and driver condi- tion (e.g., drowsy, emotional, focused). A Level 1 post hoc crash investigation involved the collection of as much data about a crash as could be accomplished with- out having an investigator visit the actual site of the crash. Such data included, as available, a police accident report, redacted to remove any personally identifying information; publicly available images of the site of the crash (e.g., from Google Earth); photos of the participant’s post-crash vehicle, obtained during retrieval of the data drive; and a phone interview of the participant. Items in the phone interview (Appendix I) were modeled after data items collected in the National Highway Table 1.3. Cognitive Assessments Assessment Type Description Clock drawing test Dementia screen The participant is presented with pencil and paper; on the paper is a circle and nothing else. The participant is asked to draw numbers in the circle to make the circle look like the face of a clock and then draw the hands of the clock to read “10 after 11.” Conners’ Continuous Performance Test Version 5 (CPT II) Executive function The CPT II is a task-oriented computerized assessment of attention disorders and neurological functioning. Results indicate the likelihood that an individual has an attention disorder. Visualizing missing information— Motor-Free Visual Perception Test Visual-cognitive Participants are shown a reference image and four similar but incomplete figures. Participants are instructed to indicate which incomplete figure could be com- pleted to duplicate the target figure; only one of the incomplete figures can be completed in such a way as to form an exact duplicate of the target figure. Visual information processing speed— Useful Field of View (UFOV) Visual-cognitive Participants are briefly presented one of two very similar target stimuli (truck or car icon that differed only slightly) in the center of the display. Simultaneously, a second target icon—the same as the central target—is presented in one of eight possible peripheral locations at varying eccentricities in a 35-degree region around the central visual field. Participants have to identify both what the cen- tral target is and the location of the peripheral target. The presentation duration of the stimulus display is dynamically varied up or down until the participant reaches a 75% correct response accuracy. Presentation time is recorded. Trail making (Parts A and B) Visual-cognitive In Part A, participants use a touch screen to connect in order (i.e., 1-2- . . . n) a series of randomly arranged numbers, then in Part B they connect a series of randomly arranged numbers and letters in alternating progressing sequences (i.e., 1-A-2-B-3 . . . n). Time-to-completion of the entire series is recorded. Table 1.4. Physical Ability Metrics Assessment Body Area Description Grip strength Upper body Left- and right-hand grip strength measurement is conducted using the Jamar Hydraulic Hand Dynamometer. Rapid pace walk Lower body Participant is asked to walk as quickly as possible, without tripping or falling, 10 feet, then the same 10 feet back to the starting point.

20 Coordination Contractor checked and validated the presence and quality of nonsensor/nondriving data. These included demographics, driver functional assessments, vehicle features, and post hoc crash investigations data. Outliers were identi- fied, and validation was requested from the site contractors or data were flagged as anomalous. Reporting A required component of the Coordination Contractor’s role included reporting on an array of project activities. A wide variety of reports was generated by the Coordination Con- tractor on weekly, monthly, or quarterly bases; many were produced on an ad hoc basis. These reports provided crucial information on the current and projected status of key study metrics, which helped study leaders guide the progress of the study toward its ultimate goals. Early in the study, reports focused on administrative efforts and the procurement and provision of equipment. Once inven- tory supplies met demand, ad hoc reports refocused on data quality, sample design, and the volume of data being collected and processed. Project Management Overall guidance of the SHRP 2 Safety program was the respon- sibility of the Safety Technical Coordinating Committee (TCC) composed of volunteer experts. All aspects of the SHRP 2 Safety program were reviewed and approved by the Safety TCC, and the committee met biannually to review progress and approve any program modifications. The Safety TCC was assisted by Expert Task Groups that developed requests for proposals released for use as a field-ready system. In addition, constrained DAS resources had to be continually redistributed across the site contractor, manufacturer, and Coordination Contractor sites in a manner that maintained sufficient working stock for each planned installation and maintenance. System Integration Due to the complex interaction among the DAS, related soft- ware, and the SHRP 2 NDS database, a series of integration efforts were prepared and monitored throughout the study to assess all aspects of data collection, including participant- based information and data, vehicle-based information and data, and DAS functionality. These integration efforts required that security protocols be implemented in such a way that, if data or information were lost, no personally identifying information would be compromised. Vehicle-based data were encrypted to prevent the unintended release of such infor- mation. System integration efforts were also employed to ensure the healthy functioning of the DAS, relying on a series of routinely and remotely administered automated “health” checks to monitor DAS performance. Supporting Activities for the Site Contractor Efforts The objective of supporting activities for the site contractors was to provide sufficient operational infrastructure and equip- ment so that high-quality data could be collected in a consis- tent manner across the six different data collection sites, each operated by a different contractor (though the Erie County, New York, site contractor did oversee operations at both the New York and Florida sites at a high level). To this end, pro- curement of all study-related hardware and software was con- ducted by the Coordination Contractor who provided it to the site contractors. This effort included the development or pro- curement of vision and grip strength testers; over 45 laptop and desktop computers; custom-built DAS installation, calibration, and alignment tools; and custom software. Quality Control and Oversight An essential requirement for the conduct of the SHRP 2 NDS included elements of quality control and structured oversight. To meet this contract deliverable, the Coordination Contrac- tor routinely tested data elements from participant-, vehicle-, and DAS-based perspectives, applying both manual and auto- mated processes to sampled data files. A sampling of vehicle metric variable quality is shown in Table 1.5. In addition to automated and manual quality checks of DAS-generated sensor and video data on an ongoing basis, the Table 1.5. Quality Assessment of Select Vehicle Metrics (7/15/2013) Data Item Good Quality (%) NETWORK SPEED 97.35 ACCELERATOR Position 97.65 TURN SIGNAL Status 94.29 BRAKE PEDAL 96.15 Usable FACE video (i.e., >80% per vehicle) 99.08 Usable FORWARD video (i.e., >80% for car) 97.61 Usable REAR video (i.e., >80% per vehicle) 94.07 Usable LAP video (i.e., >80% per vehicle) 99.26 Usable IMU (acceleration, x-axis) 99.27 Usable GPS (speed only) 96.06

21 The preparation phase describes those activities that had to be accomplished before the start of data collection from both a studywide perspective and in terms of the readiness of each individual site. The data collection phase describes the period that began on October 26, 2010, when the first vehicle was installed by the site contractor in Erie County, New York. How- ever, preparation activities continued well into the early months of 2011, when the last data collection site began installing vehi- cles. The data management and processing phase describes the data collected and what processes were used to collect, secure, and protect the data and to assess their quality. The cell phone records chapter of this document describes the stand-alone project managed by the Coordination Con- tractor with cooperation from the site contractors. This task provided a valuable resource for researchers to answer research questions related to the participants’ use of cell phones while driving. Figure 1.1 illustrates the areas of NDS study importance. (RFPs), proposals, and recommended contractors and provided expert guidance on many issues such as data access policies and procedures. SHRP 2 staff were responsible for communicating all committee directions to the Coordination Contractor and monitoring contract activities. Visualization of project and report Layout This report maps the key objectives described previously to the generalized project phases bulleted below. Then, summaries of outcomes, lessons learned, and future research implications follow. • Preparation; • Data collection; • Data management and processing; and • Cell phone records (stand-alone task). A0 S06 Technical Coordina on and Oversight A0 Quality Assurance A0 Par cipant Sample and Protec ons A0 Equipment A0 S07 Site Prepara ons A0 Data Management A0 Administra ve Tools A0 Inventory A0 Site Inves ga ons A0 Maintenance A0 IRB and CoC A0 Sample A0 Recruitment A0 Vehicle Data A0 Assessment Data A0 Protec ons A0 Inges on A0 Processing A0 Purchasing A0 Training A0 Provisioning A0 So ware A0 Wiki A0 Mee ngs A0 Reports A0 Website A0 S07 Weekly A0 S06 Standup A0 Data Types A0 Changes/Upgrades, i.e. So ware (U&T) and hardware (Legacy) A0 Data Access A0 DAS Equipment A0 Assessment Equipment A0 Recruits A0 Telemetry Plans A0 Par cipant Informa on A0 System Integra on Figure 1.1. Visualization of project.