Naturalistic Driving Study: Technical Coordination and Quality Control (2014)

Below is the uncorrected machine-read text of this chapter, intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text of each book. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.

40 C h a p t e r 3 human Subjects protections The data collection phase involved a greater than anticipated level of effort related to human subjects protection and other IRB-related activities. Continuing IRB activities ranged from coordination and synchronization of multiple independent IRBs, coordination of amendment requests, assessment and reporting of adverse events, and addressing the IRB and ethical aspects of participant requests for data (typically to demonstrate to authorities or insurance adjustors that she or he was not at fault in a particular incident). Each site (and the study as a whole) also underwent an annual IRB review (sometimes known as continuing review and approval); these applications for continuing approval often required that sum- mary data be gathered as of a certain date (e.g., demographic information about participants). There was IRB-related activ- ity every month of the 38 months of data collection, and for many months of preparation before data collection. Coordination of Multiple IRBs Once the study was under way, coordination activities were primarily concerned with the timing of amendment requests and consistency reviews conducted for any amendment that resulted in a change to the consent, assent, and parental per- mission forms. The timing of amendment requests initially followed a VT→NAS→ data collection sites serial pattern of approvals (i.e., first Virginia Tech IRB approval was sought and secured, then NAS IRB approval was sought and secured; finally, all sites not formally relying on the Virginia Tech IRB submit- ted the amendment to their respective IRBs in parallel with one another). Later amendments followed a VT→NAS and data collection sites pattern (i.e., first Virginia Tech IRB approval was sought and secured, then the amendment was submitted in par- allel to the NAS IRB as well as the IRBs of all data collection sites not relying on the Virginia Tech IRB). This latter method was found to be more efficient given that both the Virginia Tech and NAS IRBs occasionally presented amendments for full board review, and the IRBs in each case met monthly at most. Reviews of site contractor amendments affecting the con- sent, assent, and parental permission forms were conducted by Coordination Contractor personnel. Such reviews served to keep forms and protocols across the data collection sites as con- sistent as possible, as permitted by the respective IRBs involved. Amendment Overview As reported in Chapter 2, six amendments were required before the first participants were enrolled in the study. Once the study was under way, operational logistics and alterations in sample design strategies resulted in an additional 12 amendments (about one every 3 months, on average). A few of the amend- ments were site specific (e.g., a special recruiting method that only applied to one or two sites), while most were intended to be applied studywide (e.g., allowing for varying periods of participation beyond the initially envisioned 1- or 2-year peri- ods). Appendix S includes a brief description of each of the amendments submitted and approved during the data collec- tion phase of the study. Adverse Events In general, as part of the IRB approval and continuing oversight process, PIs must agree to identify and report unanticipated problems and adverse events to the governing IRB(s). In the course of this project there were eight adverse events reported to the various IRBs, all in the less-than-serious, unanticipated-problem category. The typical reporting process involved the site personnel reporting the incident to the Coor- dination Contractor using the RT system. Coordination Con- tractor staff would note the event (usually within hours) and circulate it among the Coordination Contractor and SHRP 2 staff (including the VTTI IRB coordinator). Coordination Contractor personnel would gather more information as Data Collection Phase

41 Industry-Driven Changes The primary change that occurred due to shifts in techno- logy was the data drive interface. When the design process started, Parallel Advanced Technology Attachment (PATA) was the standard interface used in hard drives. By 2010, Serial Advanced Technology Attachment (SATA) had replaced PATA as the standard, which was reflected in both the price and the availability of solid-state drives with the PATA interface. Because the physical interface is incompatible between the two standards, the NextGen main unit had to have a design modification to allow for the inclusion of a daughterboard to accommodate the new interface and extend the functionality of the existing motherboard to include the change. System Performance As a result of early testing that coincided with the procure- ment phase, both PCBs in the NextGen main unit required an additional design revision to improve the performance and robustness of the system. The final design of one of the two boards was released and introduced early in the produc- tion cycle. There were complaints that because the system was radiat- ing electromagnetic interference (EMI), it might be affecting some vehicles’ onboard systems, which themselves relied on radio frequency (RF) to function [e.g., the tire pressure monitoring system (TPMS)]. Due to such concerns, a small filter board was designed that attenuated the frequency cor- responding to the TPMS carrier frequency. This board was added to a subset of the NextGen main units so that site con- tractors could install them on vehicles perceived to have such problems. It should be noted that TPMS systems routinely malfunction with no DAS installed, and it was never defini- tively demonstrated that the RF signature emitted by the DAS was actually interfering with any vehicle’s functionality. Study Needs To help meet the recruiting requirements of the study, the Coordination Contractor provided engineering support to accommodate a larger pool of vehicle types. The additional changes required the design and procurement of new plastics, cables, and the addition of a derivative design in the manu- facturing process. Two examples—one mechanical and one that involved hardware and software—are provided. • Mechanical upgrade. Additional parts to work with the radar mounts were designed to allow installers to have a greater range of adjustment of the angle of the radar to accommodate a wider variety of vehicles (i.e., by fitting a greater range of front bumper designs). required for IRB reporting purposes, then develop an action plan to prevent future occurrences (if applicable). Once it was determined that an adverse event had occurred, an Adverse Event Report would be drafted and circulated among the Coordination Contractor and site staff, as appro- priate, for refinement in terms of technical accuracy. Within five business days of becoming aware of the event, the Coor- dinating Contractor would submit the final Adverse Event Report to the VT IRB, as was the Coordination Contractor’s obligation. Once the VT IRB had responded (usually within two business days), the report and its resolution would be provided to the NAS IRB. At the same time, the affected site (if not relying on the VT IRB) would submit reports to its own IRB to meet its reporting deadlines. Those issues determined to be adverse events in this study ranged from possible radio- frequency interference to stolen DASs to complaints from a nonparticipant about the study vehicle recording video infor- mation on his property to a potential breach of privacy. Participant Video Requests During the Study Design project, great care was taken to ensure the security of participant data, most notably through the acquisition of the Certificate of Confidentiality. On 33 occasions, participants asked to view or obtain their driving data, and, while not a contractually mandated aspect of the study, site and Coordination Contractors made prudent decisions to support such requests when the seriousness of the incident in question—from financial, legal, or ethical perspectives—warranted such attention. Most often, these requests were made so that participants could demonstrate that they were not at fault in the case of a crash. Other requests were made out of curiosity about the participant’s own driving habits. In 17 cases, concerns were alleviated without the review of video. In five cases, video was unavailable. In the 11 remain- ing cases, video was shared with the site contractor and/or the consented participant in a secure fashion. Videos were shown by Coordination Contractor personnel via WebEx to a room at a site contractor’s facility. The site contractor’s role was to pro- vide for a secure video screening environment, making certain that only appropriate individuals (e.g., family member, attorney or counselor, or police officer) were in attendance and that no recordings were made. No video was ever released to anyone. DaS Design Updates and procurement The DAS system continued to go through design revisions during the procurement phase and even into the production phase. These changes were necessitated by three major factors: industrywide changes in technology, system performance, and needs of the study.

42 antennas and rear cameras (the fragility of which necessitated frequent replacement), were also undertaken by the Coordi- nation Contractor. Coordination Contractor Facilitation of Site Contractor activities The Coordination Contractor performed multiple functions in support of study operations throughout the data collection phase of the project, employing a highly developed protocol for providing equipment to site contractors in support of instal- lation, maintenance, and deinstallation activities. As necessary, redistribution of study equipment was conducted to maximize overall study installation and maintenance efficiency. Provision of DAS Kits From the inception of the data collection period, the Coor- dination Contractor gathered information regarding shelf inventory and site contractors’ planned installation activities on a weekly basis. As the study grew increasingly complex, collected information also elicited more detailed information concerning planned maintenance activities, deinstallations, and types of equipment configurations (e.g., prime or legacy networks) to be installed in vehicles. This information was collated into an inventory planning spreadsheet, which trans- formed these inputs into a detailed set of expected near-term hardware required to fulfill all needs at each site. Summary of S07 Inventory Needs The inventory goal for DAS equipment was for each site to have enough equipment to support 3 to 6 weeks of installa- tion and maintenance activities. To establish a basis for DAS supplies, it was decided that a site’s inventory status would be based on scheduled activities over the upcoming 3-week period, actual activities undertaken over the previous 3-week period, and on-the-shelf inventory. Each site was then classi- fied as falling into one of the following status categories: • Shortage: Site has less than 3 weeks’ supply. • Sufficient: Site has 3–6 weeks’ supply. • Surplus: Site has more than 6 weeks’ supply. Based on each site’s status, Coordination Contractor staff could readily allocate on-hand resources to the neediest sites or call for equipment redistribution across sites to ensure the greatest overall study throughput and efficiency. Introducing an additional layer of challenge to the Coordination Con- tractor’s equipment hardware provisioning task was its lack of access to real-time information pertaining to scheduled • Hardware/software upgrade. To expand the pool of vehi- cles, it was decided to include older vehicles that conform to the legacy OBD-II network protocol (i.e., essentially accommodating vehicle model years 1996–2008). This required a modification to the existing network box, a new cable design, and a change to the database. In addition, this necessitated a modification to the installation software to ensure that correct network cables were installed to avoid the possibility of negatively affecting the vehicle’s network. Software Upgrades Shakedown Software In early 2011, study leadership recognized the need to cap- ture and rectify any installation problems before the newly equipped vehicle left the installation facility. As a result, a “shakedown” process was implemented. The shakedown activ- ity was performed by hardware technicians immediately following vehicle installation. During the activity, hardware technicians performed a short, predefined test drive to col- lect baseline data and, using shakedown software, confirm that all video cameras and primary sensors were functioning properly. This brief process ensured quality data were being captured from the initial installation. Usage and Tracking Software In summer 2012, the Coordination Contractor released a software package to the DAS which populated the trip data- base with trip-specific information, including time-stamped begin and end GPS coordinates. In over a dozen cases, pro- spective vehicle coordinates were identified that enabled the location of participants/vehicles that had been nonresponsive to repeated site contractor servicing requests. In a number of cases, the participant had either moved or sold the vehicle without notifying the site contractor, or the vehicle had been repossessed. Supplemental Procurement of Components The Coordination Contractor initially procured 2,085 units of each of the major components of the DAS. Additional sup- ply was procured in the cases of radars and solid-state drives. More radars were procured because that was the most vulner- able component when the front of a subject vehicle struck another car or object. More data drives were procured so that supplies would be available to install in a new vehicle while others were still in the process of having their data extracted. Consequently, 2,155 radars and 2,212 solid-state drives were included in the original purchase. Additionally, supplemen- tal requisition of parts not intended for reuse, such as GPRS

43 Parts to Be Used for DAS Maintenance Ideally, maintenance activities (i.e., replacing components that had been determined to have malfunctioned in some manner) were to be supported by a site contractor’s spare parts inven- tory, preserving complete kits for use in installation activities. The parts to be used for maintenance, listed in Table 3.2, were included in the equipment supply estimation tool so that Coor- dination Contractor personnel could swiftly identify quantities of specific component types that needed to be sent to a particu- lar location. In this way, planned maintenance appointments could be conducted in the most efficient fashion. activities at data collection sites. Information was procured from the data collection sites on a weekly basis, but that infor- mation could conceivably prove out-of-date even as soon as close of business on the day it was received due to cancella- tions, reschedulings, and the like. Table 3.1, an example taken from one of the regularly generated site performance summa- ries, illustrates the discrepancy between planned activities and actual rates of installation and deinstallation. This incongruity required the Coordination Contractor not only to consider a site’s intended slate of activities but also to examine historical data regarding actual completed activities relative to planned activities in devising equipment-provisioning strategies. Table 3.1. Example Site Performance Summary Variable Buffalo Tampa Seattle Durham Bloomington State College Total Vehicles on road 438 443 423 303 158 147 1,912 June–July installations 11 14 6 4 10 7 52 Scheduled June–July installations 22 15 21 8 1 3 70 Percentage of scheduled June–July installations performed 50% 93% 29% 50% 1,000% 233% 74% June–July deinstallations 13 20 13 6 3 6 61 Scheduled June–July deinstallations 12 5 10 6 1 5 39 Percentage of June–July deinstallations performed 108% 400% 130% 100% 300% 120% 156% Full kits on shelf 11 18 17 10 10 10 76 Partial kits on shelf 4 0 8 0 0 0 12 Kits en route to site 0 0 0 0 0 0 0 Total kits at site 453 461 448 313 168 157 2,000 Percentage of allocated inventory at site 100% 102% 106% 101% 108% 101% 96% Table 3.2. Parts to Be Used for Maintenance Site Maintenance Items Next 3 Weeks Surplus/Shortage After Maintenance Activities of Next 3 Weeks SSDs NextGens Head Units Radar Interface Boxes Buffalo 4 18.6 20.8 25.2 28.2 Tampa 11 30.4 -3.8 0.8 0.8 Seattle 15 28 -2 18 15 Durham 0 39 16 4 5 Bloomington 0 17 6 7 8 State College 0 39 3 9 10 Total 30 172 40 64 67

44 The surplus figures (indicated by positive numbers) and shortage figures (denoted by negative numbers) were automati- cally generated using an algorithm grounded in the following assumptions based on previous experience in the study: • 80% of maintenance visits require a NextGen; • 60% of maintenance visits require a data drive; • 20% of maintenance visits require a head unit; and • 20% of maintenance visits require an RIB. Outstanding maintenance tickets were reviewed periodi- cally and the algorithm adjusted accordingly. Kit Builder Recognizing the NextGen to be the primary component of the DAS, the Coordination Contractor defined a partial kit as a NextGen lacking accompanying parts to support a com- plete installation. For example, if a site contractor returned an inventory assessment like the one in Table 3.3, the Coordina- tion Contractor would conclude that the site had 31 full kits and 2 partial kits on the shelf based on the algorithm employed by the tool to calculate kit quantities (that is, 25 full kits, as listed, plus a sufficient number of individual components to comprise 6 additional full kits, with 2 partial kits left over). The kit builder tab, pictured in Table 3.4, afforded an effi- cient means for Coordination Contractor staff managing Table 3.3. Sample Site Contractor Inventory Assessment Component Quantity Good Quantity in Need of Repair/Service Full kit 25 0 NextGen 8 0 Head unit 6 0 RIB 6 0 Radar 6 0 Network box 6 0 Legacy network box 15 0 Rear camera 15 0 Antenna 15 0 OBD-II cable 15 0 Legacy OBD-II cable 15 0 RIB power cable 7 0 Network box power cable 7 0 NextGen power cable 7 0 RIB/radar cable 7 0 Network box/DAS cable 7 0 Head unit/DAS cable 7 0 SSD 7 0 Table 3.4. Kit Builder Component Buffalo Tampa Seattle Durham Bloomington State College NextGen 26 16 10 0 10 3 Head unit 0 2 8 6 -4 6 RIB 3 8 -4 5 0 7 Radar -4 -1 -6 5 -3 3 Network box 4 12 3 17 1 8 Legacy network box -16 -9 -3 2 -9 0 Rear camera 7 2 -10 8 1 16 Antenna -3 9 30 21 13 23 OBD cable 3 24 36 45 11 13 Legacy OBD cable -17 -7 -1 7 -2 7 RIB power cable 1 15 -9 5 2 5 Network box power cable 1 13 -10 2 0 3 NextGen power cable 1 12 -8 2 2 6 RIB/radar cable 1 16 -7 1 2 4 Network box/DAS cable 1 4 -7 0 -3 2 Head unit/DAS cable 1 11 -8 1 1 2 SSD -16 4 30 8 15 27

45 was not published, the findings of the Battelle research team (led by Christian Richard in 2008) suggested that such an approach would be feasible, at least for older drivers. How- ever, contingency plans were also implemented. As a result, a three-tiered approach to recruitment was employed. The Tier 1 approach, random cold-calling, was begun in the Erie County, New York, area approximately 1 month in advance of the hoped-for first vehicle installation. This process was con- tinued across all sites, during which time the efficiency of the process was continually monitored. It was agreed in early 2011 that the recruitment efficiency rate for this approach was too low in terms of success rate per attempt (i.e., in the order of <2%) and overall cost per recruit. Further, a major efficiency- limiting factor identified by VT CSR management was that many of the individuals called did not have an eligible vehicle. Therefore, it was decided in July 2011 to move to the Tier 2 approach, focused random calling: phone calls were still randomly placed but only to those households believed to possess eligible vehicles. This was accomplished by procure- ment of data from Polk and other similar organizations as well as receipt of customer lists from certain OEMs. The effi- ciency of the Tier 2 approach was also continually monitored; it was used from July 2011 through April 2012, at which point its efficiency was also determined to be insufficient. Site-Based Recruiting Tactics On January 19, 2011, the site contractors were authorized to begin pursuing local recruitment using more traditional meth- ods in tandem with VT CSR activities. They employed a wide range of approaches including those listed below. In addition to maintaining a list of these approaches and publishing it on the wiki, the Coordination Contractor facilitated discussion of these ideas in the context of a weekly conference call with the site contractors. These coordinating efforts on the part of the Coordination Contractor permitted the best and most innova- tive approaches devised by the site contractors to be efficiently shared with the others and deployed as appropriate. • Flyers: 44 Distributed at student centers on campus, 44 Distributed at departments of motor vehicles, 44 Distributed at private schools, 44 Tailored to specific demographic groups, and 44 Distributed on cars; • Facebook, Twitter, and LinkedIn; • Facebook-targeted ads; • Banner ad on traffic website; • State department of transportation websites; • Craigslist; • Local news spot; • Television advertisement; equipment provision for multiple locations to readily assess the studywide inventory situation and quickly formulate a strategy for maximizing the number of complete kits, and consequently, for maximizing the number of installations performed and the amount of data collected. The numbers in the table were automatically generated based on encoding of the partial kit parameters into an Excel spreadsheet. Nega- tive values indicate the designated quantity of a component should be sent to a site to complete a kit. The kit builder pictured in Table 3.4 was featured in the June 20, 2012, site contractor supply estimation report; it was used to construct a plan that ultimately led to the completion of 52 kits at three project sites from previously nonintegrated parts, providing support for 52 additional vehicle installations. Equipment Redistribution Plans The complex and varied nature of daily operational activities at six data collection sites necessitated frequent and thought- ful equipment redistribution exercises to provide adequate support for installation and maintenance activities and main- tain appropriate inventory levels studywide. These efforts were planned using the site contractor supply estimation tool described in the previous section and implemented keeping in mind the primary study goals of maximizing data collec- tion efforts and maintaining the health of the vehicle fleet. The design of each required a commitment to balancing the needs of individual sites with those of the larger study. Cumu- latively, these endeavors resulted in 314 vehicle installations and 46 maintenance activities that might not have otherwise been possible to accomplish. Technical Support The Coordination Contractor established a separate trusted domain (shrp2nds) and provisioned SHRP 2 NDS users under that domain to share designated network resources at the Coor- dination Contractor. This arrangement allowed the Coordina- tion Contractor’s technical support staff to access shrp2nds domain resources (e.g., desktops, laptops, site servers). The Coordination Contractor also worked with network engineers at each of the site contractor locations to facilitate the flow of data to/from the Coordination Contractor. This effort was critical to establish networks that could support the bandwidth required to move the amounts of data gener- ated at each data collection site. recruiting processes In the course of the study design phase, Battelle researchers conducted a pilot study to assess the efficacy of the cold-calling approach to recruitment. Though the outcome of this exercise

46 three discrete hardware installation configurations: prime, subprime, legacy, and basic as described in greater detail below. Prime. The Coordination Contractor, through negotiations with various OEMs independent of this project, pro- cured onboard diagnostic parameter IDs (OBD-II PIDs), thus enabling the acquisition of network variables. These included Controller Area Network (CAN) speed, turn sig- nal, brake light, and headlight information. Acquisition of these variables facilitated the collection of the most robust possible data set. Hardware technicians used a standard network box and OBD-II cable connected to the Data Link Connector (DLC) of the vehicle to communicate with the vehicle network. The initial study design limited the eligible vehicle list to these vehicles, later termed prime vehicles. Subprime. These vehicles were generally manufactured in 2009 or later and not included on the initial eligible vehicle list, as the OBD-II PIDs had not been acquired for these vehicles. These vehicles only yielded information on vehi- cle speed, throttle position, and ambient air temperature. Subprime vehicles were installed with a standard DAS con- figuration, identical to that used in prime vehicles. Legacy. These vehicles, like their subprime counterparts, pro- vided a reduced set of network data in comparison to that generated by the prime vehicles. Legacy vehicles were man- ufactured before 2009 and employed an older communi- cations network for onboard diagnostics. The data set for legacy vehicles was limited to speed, throttle position, and ambient air temperature. Legacy vehicles required specially modified legacy network boxes and cables. Basic. These vehicles were manufactured before 1996 and did not have modern vehicle networks compatible with the SHRP 2 network interface box. Basic vehicles were installed with the standard network box, omitting any net- work cable; and no vehicle network data were acquired. The Coordination Contractor provided site contractors with considerable guidance concerning vehicle classification and installation. Table 3.6 summarizes the salient features of the four SHRP 2 NDS vehicle classes, listed in descending order of data richness. While the Coordination Contractor was able to make an initial judgment as to whether a vehi- cle of a particular year, make, and model would be eligible for inclusion, site contractors were instructed to bench test potential legacy vehicles to confirm availability of speed and accelerator position variables. The expansion of the vehicle fleet increased the intricacy of the equipment provisioning process. As Table 3.6 shows, legacy vehicles required a different installation configuration from vehicles in the other classes. This additional layer of complexity required Coordination Contractor staff to broaden their focus to consider not only the number of kits required to support installations at data collection sites, but also the specific types of kits required at each location. • Posters in high schools; • Contacts with local school boards; • Tour of facility for high school science classes; • Parent Teacher Associations and band boosters; • Community driver education school; • Community traffic safety team; • Senior centers; • Focus groups for teens and older people; • Booths at local events and fairs; • Local Virginia Tech alumni chapter e-mail; • University magazine; • Civic organizations; • PowerPoint presentations; • Radio advertisements; • Radio station events; • Professional marketers; • Newspapers: 44 Classified ads, 44 Nonclassified ads, and 44 University papers; • Mass e-mail; • Local movie banner advertisements; • Cable news feed; • Instrumented vehicle at events; • Collegiate summer orientation; • Vehicle wrap or magnet; • Additional letter sent to follow up on stale leads; and • Postcards. Recruitment was tracked in terms of recruitment medium: 1. Calls out. VT CSR personnel made random or focused cold calls, or called individuals on an OEM-provided list. 2. Calls in. The participant called a toll-free number after having seen an ad or flyer. 3. Web-based screening tool (WBST). VT CSR staff developed this tool around February 2011. The WBST facilitated the standard screening process but required no human screener, which proved to be extremely efficient. The total number of individuals who expressed an interest in participating and who were successfully screened is shown in Table 3.5. Network Data Classification Difficulty recruiting younger and older drivers compelled the Coordination Contractor, in cooperation with SHRP 2 pro- gram managers, to expand the vehicle fleet to include a broader range of vehicles for which incomplete vehicle network infor- mation was available. The enlargement of the vehicle fleet necessitated the creation of four distinct vehicle classes with

47 leaders realized that recruitment at the ends of the spectrum was lacking, the Safety TCC was consulted in April 2012 and they agreed to discontinue strict adherence to the experimental sample design shown in Table 2.4. Instead, a set of goals or guiding principles was established to guide further recruitment strategy and installation priori- ties. These goals focused on allocating recruitment, inventory, and human resources in a manner aimed at increasing the number and time in study of participants in the following age groups: 16–17, 18–20, 21–25, and 76+, with the higher-risk males being prioritized over females. In addition, whenever Goal-Based Recruitment Strategy Drivers at the ends of the age spectrum were among the most interesting participants from risk and behavioral perspec- tives. However, as recruitment progressed, early summaries showed that recruits roughly mirrored the driving population at large in terms of age distribution, which meant that recruit- ment efforts were not sufficiently effective at oversampling the youngest and oldest groups of drivers. Figure 3.1 shows the distribution of the age of participants as of April 2012 plotted against that of drivers in the United States for 2011. Once study Table 3.5. Recruitment Summary by Method, Age Group, and Site Method Age Group (years) State College Bloomington Durham Seattle Tampa Buffalo Total Calls Out 16–17 1 0 3 1 1 0 6 18–20 1 5 6 4 4 9 29 21–25 13 10 13 11 17 19 83 26–35 47 45 93 56 86 94 421 36–50 138 109 284 191 222 283 1,227 51–65 208 168 284 243 297 370 1,570 66–75 86 68 142 190 213 191 890 76+ 37 29 42 91 111 88 398 Subtotal 531 434 867 787 951 1,054 4,624 Calls in 16–17 3 4 23 0 19 5 54 18–20 11 15 42 0 68 17 153 21–25 6 9 44 5 75 19 158 26–35 7 13 49 10 41 32 152 36–50 9 17 78 21 97 62 284 51–65 15 16 81 15 148 76 351 66–75 8 5 72 7 108 48 248 76+ 4 9 95 2 49 37 196 Subtotal 63 88 484 60 605 296 1,596 WBST 16–17 27 44 172 255 282 125 905 18–20 92 80 188 398 840 413 2,011 21–25 142 140 290 671 650 584 2,477 26–35 138 61 339 517 503 508 2,066 36–50 127 75 427 532 662 396 2,219 51–65 125 56 220 414 456 269 1,540 66–75 42 24 100 102 214 70 552 76+ 27 15 84 117 68 76 387 Subtotal 720 495 1,820 3,006 3,675 2,441 12,157 Total 1,314 1,017 3,171 3,853 5,231 3,791 18,377 Note: WBST = web-based screening tool.

48 desired, but there were virtually no recruits satisfying both cri- teria. In other cases, the goals were conceptually at odds. An example of this is the notion of allocating DAS kit resources to the most highly desired recruits, while at the same time not wanting to let a DAS kit remain unused in inventory. In all such cases, the Coordination and site contractors used best judgment to make recruiting decisions in the best interest of the goals of the study. Secondary Drivers As noted above, during the course of an extended period of participation, it was quite likely that individuals other than the consented participant might drive the vehicle. That pre- sented both a significant problem as well as a tremendous opportunity. The significant problem was that when an unconsented individual drove an instrumented vehicle, data, including potentially identifying face video were being continuously recorded during that trip. Of course, this recording of data and video occurred automatically, with no known way to prevent or stop it. This was a serious issue because any trip could end in a serious at-fault crash or otherwise include the recording of embarrassing, compromising, or illegal behav- iors. All IRBs providing approvals for the study vetted this issue, and the solution agreed to by all was that it was per- mitted to record such data (as it was essentially impossible to prevent this from taking place). However, once definitively determined to have been collected from an unconsented driver, these data ultimately had to be expunged, without ever being analyzed or otherwise used in any research protocol or related activity. However, there was also a tremendous opportunity. If one or more individuals other than the primary participant possible, site contractors were asked to prioritize the instal- lations of vehicle network classifications in the following order: (1) prime, (2) subprime/legacy, and (3) basic. Further, no additional drivers of Toyota Priuses were permitted to be enrolled in the study, as it was determined that this particu- lar make/model was already overrepresented in the sample. Finally, an overarching goal was to collect as much data as possible; in other words, a DAS kit collecting data from a vehicle (any driver, any vehicle) was preferable to that same DAS kit sitting on a shelf. Of course, it was not typically an easy task to prioritize recruits with this approach as some of the goals, while con- ceptually compatible, were often practically at odds. An exam- ple is that both young drivers and prime vehicles were highly Table 3.6. SHRP 2 NDS Vehicle Classes Class Definition Vehicle Count Prime Vehicles included on the original eligible vehicle list for which the Coordination Contractor procured PIDs 1,717 Subprime Vehicles manufactured primarily after 2009 for which Coordina- tion Contractor was not able to procure PIDs but was able to obtain information through the CAN communication protocol, an industry standard after 2009 488 Legacy Vehicles manufactured between 1996 and 2008, employing an older network for vehicle communications 736 Basic Vehicles manufactured before 1996 421 Figure 3.1. Distribution of age of participants as of April 2012 versus that of U.S. drivers for 2011.

49 Pressing the incident button created a marker in the data stream, making it possible for an analyst to find that specific portion of the data record and capture a reference image from the secondary driver’s face video. Figure 3.2 shows the number of known secondary drivers by age group across the sites. Note that even if a secondary driver was consented, and thus provided data that can be analyzed, a reference image was still needed to positively identify trips by that individual as being a consented driver. Such images are still being collected. Note that the number of “not known” is far larger than any other age-group-gender combination. That is because secondary drivers typically never came to the site contractor facility and often provided incomplete informa- tion on their intake forms. The subset of those consented drivers who provided a usable reference image as of January 14, 2014, is shown in Fig- ure 3.3. Additional reference images may still surface through a variety of processes. Multiple Primary Participants Initially, only one primary participant per household was per- mitted. However, in a concerted attempt to recruit more teens and other young drivers into the study, it was later agreed that if a household included a young driver (typically a child of the first primary driver) whose primary vehicle was the first primary driver’s vehicle, then those young individuals were also eligible to be consented and to be treated, essentially, in every respect as primary participants, including having all their assessment data collected. In August 2011, the Coordi- nation Contractor implemented a protocol for the inclusion of these drivers in the study. Figure 3.4 shows the number of households with two primary drivers enrolled in the study across the six data collection sites. were expected to drive the vehicle on a fairly regular basis (e.g., the participant’s close family members), then these indi- viduals might generate interesting and/or useful data. Thus, up to three additional individuals for a given participant were given the opportunity to enroll in the study as secondary drivers. These were individuals other than the primary participant who could be expected to regularly drive the instrumented vehicle, and who granted informed consent to use their driv- ing data over a particular period. In addition, these second- ary drivers were asked to fill in two surveys (Demographics and Medical Conditions and Medications) and to provide a face image of some kind. Each was compensated $25 for participating. Collecting a face image from each consented participant was crucial because it served as a reference image for the post hoc driver identification process, in which every trip file was evaluated manually by trained analysts to deter- mine whether or not the driver was a consented participant. This process of collecting a reference image was relatively simple for most primary participants, because the system’s in-vehicle cameras took a facial snapshot reference image of the driver in the driver’s seat as a part of the DAS instal- lation process. This image would ostensibly serve as a high- quality reference image as it could be expected to be very similar to actual trip file images seen by the driver identi- fication analysts. However, there was no similar procedure available for secondary drivers, as these individuals consented by mail and never actually visited the site contractor’s site. Thus, secondary drivers were asked to provide a reference image by sending a hardcopy or electronic photo or by pressing the incident button on the head unit while prepar- ing to drive the vehicle. After pressing, the secondary par- ticipant was asked to state, “My name is [first name only spoken here], and I am a secondary driver in the study.” Total = 398 participants Figure 3.2. Consented secondary drivers by age-gender group across site irrespective of reference image.

50 then it might be a good idea to investigate whether that component had previously been installed in another vehicle and whether or not it demonstrated that same problem in the previous installation. To this end, a database viewer, the Component Information Tracker, was designed. The initial intent was for Coordination Contractor staff to respond to hotline-generated issues during after-hours calls. Later, the tracker became a heavily relied on software tool that stream- lined the diagnosis of all vehicle and component issues. The Coordination Contractor used this software program to pro- vide a simple view into the database that succinctly pre- sented logged activity related to all components, including the history of a component’s location, from creation at the manufacturer through installation in one or more vehicles, inventory exercise updates, deinstallations, and shipping and receiving activities. Figure 3.5 shows an example of a component history detail using the Component Informa- tion Tracker. Component Shipping Tool In large part because the study inventory was in a continual state of flux across the six data collection sites, the Coordi- nation Contractor site, and the contract manufacturer site, the need for more standardized protocols for shipping and inventory management processes was identified. The result- ing software, Component Shipping Tool, allowed S07 sites to associate components with a shipping box via barcode scan- ning protocols so that shipments could be tracked at box and component levels. This midstudy improvement allowed for better component location information within the database and significantly eased the amount of time needed to inven- tory components as they were returned from site contractors during the deinstallation phase of the project. administrative tools and processes Software Additions and Improvements Throughout the course of the SHRP 2 NDS, several improve- ments were identified that could positively influence the effi- ciency with which the study processes could be managed. Of particular benefit, inventory, telemetry plans for the DAS, and participant management tools were addressed in midstudy cycles, which complemented existing management tools. Component Information Tracker In late 2010, the need to be able to assess the history of a component’s installations in vehicles and known problems during particular time periods was recognized as offering the potential for use in troubleshooting. For example, if a prob- lem was found with a component that affected data quality, Total = 209 participants Figure 3.3. Consented secondary drivers by age-gender group across site with reference image. Total = 40 households Figure 3.4. Number of households with multiple primary drivers, by site.

51 future guidance. In addition, the Coordination Contractor also conducted weekly operations team meetings with site contractors and TRB staff to discuss the issues faced by the site contractors and to address communications, supplies, outstanding work items, resources, or efficiency-enhancing solutions. Further, the Coordination Contractor conducted quarterly meetings involving their technical resources and the hands-on technicians working with each of the site contrac- tors. At these meetings, detailed technical issues were discussed and solutions offered. Figure 3.6 illustrates the Coordination Contractor’s communications model. Weekly Operations Team Conference Call As discussed in Chapter 2’s description of preparations for the conduct of the study, the weekly operations team confer- ence call was designed to provide continuing training and to elicit feedback concerning study progress at remote sites. This call proved to be an essential component of the collaborative relationship between Coordination Contractor and site con- tractors. Site contractors were encouraged to contribute to the setting of the weekly agenda by submitting topics of concern via the RT system. Topics addressed included IRB policy and its practical implications, issues of recruitment and participant Mission Control System Enhancements During the study, a number of design improvements to MCS were identified, which can be generally characterized as im provements to display views and cross-referenced infor- mation that linked participants and vehicles and compo- nents. Further improvements added functional capabilities for uploading additional types of documents and linked par- ticipants to additional primary and secondary participants. While many of these served to improve efficiency of site con- tractor activities, others improved the efficiency of back-end algorithms. Intra-Study Communications Continual communication among all stakeholders was crucial to the success of this project. The Coordination Contractor provided weekly status updates on all aspects of the project to TRB staff at a standing meeting. This weekly meeting also provided a routine avenue for the Coordination Contractor and TRB staff to discuss the most pressing issues, and for TRB staff to contribute to problem solving. VTTI senior leader- ship provided similar updates to TRB’s Technical Coordinat- ing and Oversight Committees during their biannual project oversight meetings to make critical decisions and provide Figure 3.5. Example of component-level information via the Component Information Tracker.

52 staff, Coordination Contractor staff and site contractor staff, and Coordination Contractor staff and DAS technicians. Maintaining a regular schedule of conference calls along with the SHRP 2 wiki provided a continual flow of information, and sustained a high level of flexibility in managing critical issues. TRB and Other Ad Hoc Presentations Summer Safety Symposium One key communications medium was the SHRP 2 Summer Safety Symposium, which served to communicate SHRP 2 activities to the broader research community and other inter- ested parties. The Coordination Contractor coordinated the 3rd Annual SHRP 2 Safety Symposium in the summer of 2008. In addition to the standard introductions and overviews provided by SHRP 2 staff, the agenda for this symposium included a presentation and discussion of the SHRP 2 NDS study design. Further, updates and presentations were made by representatives from European and Canadian naturalistic driving studies and other SHRP 2 Safety contractors. Future SHRP 2 NDS project plans were also communicated with the larger research community. A keynote presentation was provided by Dr. Christine Branche, then Acting Director of management, and matters related to vehicle installation and maintenance. Not only did it provide an opportunity for dissemination of vital study information and policy to site contractor PIs, it afforded the group a forum for discussion of systemic issues faced by the site contractors. All of these aspects contributed to decision making by the Coordination Contractor in conjunction with study sponsors. Weekly Conference Call Weekly conference calls, including SHRP 2 staff and Coordi- nation Contractor personnel, afforded an occasion to monitor and reflect on study status (as represented by a robust suite of reports) and guide the study through every issue encountered. Often, these calls served as decision points for critical opera- tional issues. Decisions on many critical and time-sensitive operational issues were made quickly and verbally rather than using a lengthy process of written proposals, review, amend- ments, and sign-offs. This provided the necessary flexibility to keep the whole project moving forward as quickly and effec- tively as possible. Strategic issues were referred to the relevant SHRP 2 committee. Successful conduct of a research program on the scale of the SHRP 2 NDS required clear, consistent, and constant com- munication between SHRP 2 and Coordination Contractor Figure 3.6. Communications model.

53 4. Serve as a resource for senior management in sharing study progress with external stakeholders and the wider research community. Recruitment Reports Recruitment reports were prepared by the VT CSR and dis- seminated weekly by the Coordination Contractor to site contractor PIs to provide insight regarding the success (or lack thereof) of various recruitment strategies, and to afford the PI an understanding of the size and age group/gender makeup of the latest slate of prospective recruits. The time period between a prospective recruit’s first communication of interest in the study and a site contractor’s receipt of his or her contact information was at times prolonged by eligibility screening and database processing. Providing the recruitment reports was therefore invaluable to site contractors in helping them anticipate future installation opportunities and allocating resources appropriately to attract additional recruits in accor- dance with the study sampling scheme or guidelines. Sometimes, knowing why a potential recruit did not ini- tially qualify proved to be an impetus to advance the study. For example, in the months preceding the expansion of the eligi- ble vehicle list to include subprime, legacy, and basic vehicles, a number of prospective participants were rejected because their vehicles were not yet considered eligible for installation. See, for example, the graph taken from the April 28, 2011, recruitment report pictured in Figure 3.7. The majority of recruits initially rejected were disqualified due to vehicle eli- gibility issues. Reports like these helped study administrators identify impediments to full recruitment in all segments of the sample and spurred them toward remediation solutions, such as the decision to include vehicles previously excluded from the study fleet. This is illustrative of the way in which reports served not only a descriptive but also a prescriptive purpose. Operations Metrics An operations metrics report was prepared weekly and cir- culated among Coordination Contractor staff to present a glimpse into current study operations. The report encom- passed all aspects of the study, including data quality, data ingestion progress, vehicle communications status, counts of participants and vehicles installed in the past week, solid- state drive status, outstanding maintenance items, inventory counts, and statistics pertaining to fleet issues of particular interest. The counts dictated the work activities of Coor- dination Contractor personnel each week; and the associ- ated spreadsheets provided a road map for the completion of that work, identifying vehicles with communications issues or video quality problems, or in need of a solid-state drive the National Institute for Occupational Safety and Health (NIOSH). The relevance of this presentation was especially well received given that the missions of the SHRP 2 NDS and NIOSH are so closely related—that is, to generate new knowledge about safety (and health) and transfer that knowl- edge into practice globally. In subsequent years, the Coordination Contractor actively participated in the 4th, 5th, 6th, 7th, and 8th Annual SHRP 2 Summer Safety Symposia, each time updating status regard- ing the planning, implementation, data collection, and data sharing for the SHRP 2 NDS. This participation was of para- mount importance given the overarching importance to SHRP 2 and the National Academy of Sciences of gather- ing and disseminating to the nation objective information related to precrashes, crashes, and exposure on the nation’s highways. Biannual SHRP 2 Committee Meetings The Coordination Contractor staff provided technical input and presentations biannually to members of the SHRP 2 Tech- nical Coordinating and Oversight Committees. General information about the NDS, which included study strategies as well as statistical assessments and metrics, was provided to inform the committees charged with technical and admin- istrative oversight for the study. Strategic decisions made by the Safety TCC included revisions to the recruiting process to speed up enrollment (such as increasing the incentive) and the recommendation to extend the data collection period to achieve the overall participant-years exposure goal. All bud- get requests associated with these decisions were referred to the Oversight Committee for approval. Reports In addition to fulfilling the contractual obligation to pro- vide comprehensive monthly and quarterly reports detailing study status and accomplishments, a variety of other reports were prepared on a regular basis by Coordination Contrac- tor staff for a number of purposes. These included providing accountability to the project sponsor, guiding oversight of site contractor activities, assessing issues of specific concern, and serving as an engine to drive daily operations of Oversight Committee and site contractor staff. These reports provided crucial insights into study status and progress along a variety of dimensions, serving as key decision-making tools. They were designed with four objectives in mind: 1. Measure progress toward study goals; 2. Serve as a study management tool; 3. Guide protocol refinements and policy decisions in the interest of advancement of study goals; and

54 trip files taken from the ingested data pool. This segment of the report was of particular importance to Coordination Contrac- tor staff. A glimpse into the quality of video and sensor data allowed for insight into the quality of collected data and provided a proactive approach to vehicle maintenance; files verified as bad could be correlated with specific vehicles, and appropriate remedies could be applied. Data Volume Reports One metric of the total quantity of data collected was the “vehicle-time period” (e.g., vehicle-month or vehicle-year). swap. Access to a complete list of vehicles affected by a spe- cific issue, such as faulty illumination of TPMS indicators, facilitated creation of maintenance request tickets in the RT software. This weekly snapshot of the state of the study was frequently upgraded to include additional data or data repre- sentations as requested by SHRP 2 staff or as necessitated by current issues. It allowed the Coordination Contractor and SHRP 2 staff to maintain a high level of scrutiny concerning all aspects of the SHRP 2 NDS. Table 3.7 presents a snapshot of the vehicle metrics portion of the operations metrics report. The counts in the “No. Verified Good,” “No. Verified Bad,” and “Not Verified” columns refer to Figure 3.7. Reasons for recruit ineligibility in Florida, April 2011. (“Complete Agreed” means passed initial screening.) Table 3.7. Vehicle Metrics Portion of Operations Metrics Report Data Item September 25, 2012 No. Verified Good No. Verified Bad Not Verified No. Verified Good/ Total Verified Network speed reported 1,325 36 97.35% Accelerator position reported 541 13 97.65% Turn signal status reported 528 32 94.29% Brake pedal reported 525 21 96.15% Usable face video (i.e., >80% per vehicle) 1,506 3 272 99.80% Usable forward video (i.e., >80% for car) 1,496 2 283 99.87% Usable rear video (i.e., >80% per vehicle) 1,351 154 276 89.77% Usable lap video (i.e., >80% per vehicle) 1,405 104 272 93.11% Usable inertial measurement unit (acceleration, x-axis) 1,489 11 99.27% Usable GPS (speed only) 1,098 45 96.06%

55 assumptions about study term completion and extensions as did projections with regard to vehicle-months. Site Priority Summary The site priority summary, presented monthly to Coordi- nation Contractor management and shared with SHRP 2 staff, provided a comprehensive assessment of the condition of the SHRP 2 NDS vehicle fleet and participant pool. An example report is included in Table 3.8. The insight afforded by the site priority summary assisted Coordination Con- tractor administrators in monitoring the effectiveness of the issue prioritization process described in the next section. Site Priorities Process As the study expanded to include close to 2,000 vehicles on the road, the challenge of ensuring the full functionality of installed DASs in such a sizable fleet necessitated a systematic, centralized approach to setting maintenance priorities. The Coordination Contractor assigned each maintenance item a priority level of 1, 2, or 3. Priority levels were defined as delin- eated in Table 3.9. Priority levels were assigned based on the following crite- ria, listed here in decreasing priority order: • Communication status. Vehicles not communicating within the past 30 days were flagged for maintenance. • SSD decrementing. Discernible change in the fullness of the SSD was a strong indication as to whether the data acquisi- tion system was collecting data as intended. • Delivery of Advanced Health Checks. Vehicles failing to deliver an Advanced Health Check within the past 45 days were flagged for maintenance. • Excessive communication or history thereof. Indication of potential power issues with the DAS were cause for concern. The Coordination Contractor disseminated site priority spreadsheets listing all priority maintenance tickets to site contractors on a monthly basis. Quality Metrics The quality metrics report, submitted weekly by the Coor- dination Contractor to SHRP 2, provided an overview of study operations. These included vehicles on the road rela- tive to study targets, sample cell population, vehicle instal- lations by vehicle type, data ingestion and processing, data quality, status of components in the repair process, crash counts by category, percentage of trips in which a consented participant had been confirmed as the driver, and counts According to this concept, each participant contributes one vehicle-month of data for each full month of participation in the study and one vehicle-year of data for each full year in the study, regardless of the actual number of driving miles or hours represented during that time period. It may be argued that driving distance and time are more accurate ways to determine overall data quantity, but these were also more dif- ficult to accurately capture or estimate during the conduct of the study. The total number of vehicle-months collected, on the other hand, could much more readily be calculated at any given point in time for any desired subset of the data (e.g., by age group, gender, and/or data collection site). In developing the vehicle-months/miles report and pro- jections calculator, Coordination Contractors were seeking a tool that would (1) provide a metric for study progress toward the sample design goal of 3,900 DAS-years and (2) make it possible to project data acquisition for individual sample cells based on the allocation of future installations at both the indi- vidual data collection site and overall study levels. Generating the vehicle-months portion of the report involved calculating the number of vehicle-months collected by each vehicle in the study and projecting how many additional vehicle-months each participant was expected to accumulate. The projection was based on a set of guidelines and assumptions about how long each participant was expected to remain in the study and how future DAS resources would be allocated. Projected vehicle-months resulting from extensions and from reinstallations were calculated separately, affording study administrators the flexibility to make choices about future study policy that would yield optimal data acquisition for every sample cell of the study. Figure 3.8 is taken from the vehicle-months report dated August 31, 2012, and was used by SHRP 2 program managers and the Coordination Con- tractor to assign reinstallation targets for older and younger sample cells such that progress toward study goals was optimized. The concept of vehicle-miles was defined in a similar fashion as that of vehicle-months described above. Every mile driven by a participant represented a single vehicle-mile (regard- less of duration or location of the trip). Total vehicle-miles acquired for in-study reporting purposes were calculated on a vehicle-by-vehicle basis factoring in data accumulation in megabytes (MB) since the last SSD swap. The calculation assumed data collection at a rate of 10 MB/min and an aver- age speed of 30 mph. A participant’s driving frequency since the most recent swap of the SSD was assumed to be represen- tative of his or her driving frequency over the course of his or her tenure in the study. Calculation of projected vehicle-miles was based on the notion that the ratio of accumulated vehicle- miles to accumulated vehicle-months would be equal to the ratio of projected vehicle-miles to projected vehicle-months. In this way, vehicle-miles projections rested on the same set of

56 Figure 3.8. Vehicle-months report, August 31, 2012.

57 Table 3.8. Site Priority Summary, April 9, 2013 Issue Buffalo Tampa Seattle Durham Bloomington State College Total SHRP 2 Fleet Snapshot Vehicles on road 450 455 437 304 151 146 1,943 Communicating 446 454 434 302 150 145 1,931 Percentage of installed vehicles communicating 99.11% 99.78% 99.31% 99.34% 99.34% 99.32% 99.00% SSDs recording data 445 453 437 304 149 146 1,934 Percentage of installed vehicles recording data 98.89% 99.56% 100.00% 100.00% 98.68% 100.00% 99.54% Major Fleet Issues Not communicating 4 1 3 2 1 1 12 SSDs not decrementing/not recording data 5 2 0 0 2 0 9 Failing to deliver Advanced Health Checks 1 1 3 1 1 1 8 Communicating in excess 0 0 0 0 0 0 0 History of excessive communication requiring attention 1 15 6 7 3 5 37 SSDs >90% full 3 11 0 0 1 2 17 Percentage of vehicle fleet with SSDs exceeding 90% 0.67% 2.42% 0.00% 0.00% 0.66% 1.37% 0.87% S07 Site Progress Priority 1 issues identified in previous report (2-22-13) 24 23 5 7 11 14 84 Priority 1 issues completed from previous report (2-22-13) 21 20 5 6 11 14 77 Percentage of Priority 1 issues completed from previous report (2-22-13) 87.50% 86.96% 100.00% 85.71% 100.00% 100.00% 91.67% Priority 1 issues remaining from previous report 3 3 0 1 0 0 7 Action taken (e.g., participant called) 3 3 0 0 0 0 6 No action noted 0 0 0 1 0 0 1 Table 3.9. Maintenance Priority Levels Priority Level Definition Examples 1 Data loss is imminent or participant is potentially exposed to danger; a visit to the vehicle is required within a week. • SSD fullness exceeding 90% • Failure to communicate in more than 30 days • Failure to deliver Advanced Health Checks in more than 45 days • Dangling head unit • Front or face camera issues 2 Pertaining to data quality–related processes; a visit to the vehicle is required within 2 weeks. • SSD fullness exceeding 70% • Non-DAS-related assessment needs to be repeated for currently installed participant 3 Not pertaining to a current installation or participant; a visit to the vehicle is required but may be delayed until a higher priority issue exists concurrently. • Rear camera misalignment • NextGen anomalies not threatening acquisition of data

58 Reports Summary Reports, both internal and external, played a vital role in the successful conduct of the SHRP 2 NDS. They proved invalu- able, both as quantitative assessments of all aspects of the study as well as engines for generating the daily and weekly workflow of Coordination Contractor personnel and site contractor staff alike. Inventory Exercises In conjunction with inventory requirements established in the Project S12A DAS Procurement contract, the Coordina- tion Contractor was required to perform routine inventory exercises to demonstrate that the DAS components were being adequately tracked throughout the course of the study. To this end, all inventoried components were labeled with a barcode, and a custom software tool placed them in the data- base when scanned. Four inventory exercises were conducted during the course of the project. The first one was conducted in fall 2012; the exercise was repeated in spring 2013 and fall 2013. At the conclusion of the SHRP 2 NDS and following the deinstallation of all study vehicles, a final inventory exercise of SSDs reaching capacity at each data collection site. As the data collection period drew to a close, the focus turned from installation progress to the number of deinstallations accomplished at each site relative to targets designed to ensure that all study equipment was removed from subject vehicles by November 30, 2013. An example of the chart chroni- cling this progress is pictured in Figure 3.9. All of these inputs allowed Coordination Contractor management and study administrators to formulate sound study policies that maximized data collection and maintained a thriving vehicle fleet. Ad Hoc Reports The Coordination Contractor generated several periodic reports on weekly, monthly, or quarterly bases. In addition, many were produced on an ad hoc basis. These reports pro- vided crucial information on the current and projected status of key progress metrics, which helped study leaders guide the research effort toward its ultimate goals. Reports of note included TPMS/RF interference assess- ment, secondary driver Cell Phone Records Study eligibility assessment, and secondary driver data needs evaluation. Figure 3.9. Quality metrics showing vehicles on road relative to nominal target for each site (i.e., 450 for Buffalo, Florida, and Seattle; 300 for Durham; and 150 for Bloomington and State College).

59 temporarily missing or misplaced until the site contractor reported information that changed the status to Stolen or Decommissioned. Additionally, components could have been unavailable for use for several other reasons due to theft/loss, damage, or needing repair. To date, • 25 components have been stolen or documented as perma- nently lost (0.19%); • 79 components have been decommissioned due to irrepa- rable damage (0.60%); and • 152 components remain out of service due to needed repairs (1.15%). was conducted. The outcomes of the fall 2013 inventory exercise are summarized in Table 3.10. The final inventory accounting will be conducted after delivery of this report and submitted later. Detailed instructions were provided to the site contractors in advance of each inventory exercise. Following the exercise, components that were not located were rendered unusable in the database, thus making it impossible to install that com- ponent in a vehicle or otherwise use it until it was properly scanned and its status corrected. As components were found after the inventory, a manual adjustment to the database reset the functionality so that the components could be used again. Note that items labeled Missing were presumed to be Table 3.10. Inventory Exercise Summary Factor Fall 2012 Spring 2013 Fall 2013 Final Accounting Total purchased 12,902 12,902 13,152 13,152 Quantity missing 62 38 75 TBD Percent missing 0.48% 0.29% 0.57% TBD