Naturalistic Driving Study: Technical Coordination and Quality Control (2014)

Below is the uncorrected machine-read text of this chapter, intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text of each book. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.

79 C h a p t e r 7 human Subjects protections One of the key lessons learned from this project was that the human subjects protection and IRB-related activities took far more time, effort, and coordination than was initially antici- pated by some. In addition, in terms of total duration, this effort commenced several years before the first vehicle went on the road, involved continual attention throughout the data collection period, and will continue for as long as the data are being accessed and analyzed. Coordinating the activities, including conducting con- sistency checks for each amendment and informed consent form, across the six sites and six actively involved IRBs proved a daunting effort. (At least 98 unique consent forms were approved and being used by the end of the study, with the number possibly reaching several hundred considering all site-based variations.) Lengthy IRB Approval Processes Large, complex, multisite studies require sufficient time to plan and prepare (not just for IRB tasks, but at every stage). Before any significant IRB tasks can be conducted, all sites and institutions must be known and under contract. Once this happens, a meeting should be held at the earliest pos- sible opportunity with all involved IRBs in attendance. All institutions and IRBs should then work together to prepare acceptable documents and procedures before any official sub- missions. The simplest and most expedient approach is one in which all relevant IRBs and institutions designate a single IRB as their officially designated IRB of Record using Letters of Authorization, preferably with the IRB of Record having substantial experience with this sort of research effort. How- ever, in reality it is likely that many will choose not to cede control in this way, and they certainly should not be forced or coerced to do so. The makeup and leadership of an IRB will change periodically, and this may affect the approval process even if all IRBs had seemed to be in agreement. Finally, not all IRBs meet monthly, and significant time lags are com- mon when multiple institutions are involved. The main les- sons learned were not to underestimate the effort and time required when developing the timeline and budget for a proj- ect of this magnitude. Flexibility for Future Use of Data Wide availability of the resulting data was given high priority from the beginning of SHRP 2 NDS discussions, while protec- tion of participant confidentiality was also an important con- sideration. The participating IRBs cooperated in making sure that the research protocols and consent documents adequately protected participants while also allowing for flexible use of the data within those constraints. Nonidentifying and deiden- tified data are allowed to be widely shared, in some cases even becoming available on a data sharing website, while identify- ing data must be stored securely and viewed or analyzed via secure means (such as a secure data enclave). These levels of access and potential means of sharing data after the study was complete had to be designed and acted on long before the first participant enrolled in the study. This was a lesson learned from previous naturalistic driving studies such as the 100-Car Study (Dingus et al. 2006) in which the investigators originally underestimated the long-term interest in the data and did not allow for the same degree of flexibility. General Flexibility During the Study A similar lesson learned concerns general study flexibility. As can be seen from the amendments discussed in Chapters 2 and 3, study details were somewhat fluid, particularly in terms of compensation and length of enrollment. All the IRBs involved in the approval process reviewed the protocol with a full board review (the most stringent level of review) due to the presence of minors in the study. Most IRBs convene Lessons Learned

80 Coordination Contractor did observe roaming charges for the DAS modems, providing corroborating evidence that some participants had traveled across the border. Secondary Driver Consent The primary obstacle in successful management of par- ticipation of secondary drivers was the lack of appreciable contact between Coordination or site contractors and these individuals. Most often, these drivers were approached about involvement in the study by the primary driver handing them a packet of informed consent and related materials. This posed a formidable challenge in obtaining all the documen- tation required, namely a signed consent form and a refer- ence image, because the degree of relationship between these individuals and study personnel was limited at best. It is important to note that the minimum and sufficient docu- mentation required to be considered a secondary participant was a properly signed and dated consent form. However, when reviewing and analyzing actual data, a reference image for the secondary driver is also necessary, because this image definitively links the consented secondary participant with his/her trips in the data record. Without a reference image, there is no way to determine who the driver is; so these oth- erwise valid trips have to be excluded from the database until such time, if ever, that a reference image can be found or procured. Of the 398 consented secondary drivers, 209 pro- vided a reference image. An attempt to facilitate procurement of a greater number of secondary driver reference images was made by allowing secondary drivers to self-identify using the incident button (i.e., the secondary participant was asked to press the incident button, look toward the head unit, and say something close to “My name is [first name only] and I am a secondary participant”). Amendments late in the data collec- tion period eased this burden somewhat, with direct contact permitted between site contractors and secondary drivers and incentives established for primary drivers who were able to provide a reference image for their secondary drivers. These efforts produced only modest results; however, current IRB protocols allow follow-up contact with secondary drivers who have completed their enrollment in the study. Participant Video/Data Requests Throughout the study, several participants asked to view or obtain a copy of their data, generally in reference to an isolated event such as a crash or other incident. While ser- vicing such requests was not directly a part of the project, both site contractor and Coordination Contractor person- nel felt obliged and were happy to attempt to help when the seriousness of the situation warranted (i.e., in legal, financial, or ethical terms). once a month; others typically meet less often. Therefore, it was important to keep the amendment processes flowing as swiftly as possible to allow maximum study flexibility. Never- theless, one important lesson learned was that a large, long- term, multisite study with numerous IRBs is something like a large ship at sea—once the iceberg is spotted, it takes a while to turn the ship. It is possible, but it takes a while. Compensation Schemes Recruitment was initially very difficult and inefficient for a variety of reasons, as previously discussed. As one aspect of addressing this area of concern, it was agreed early on to increase compensation from $300 per year to $500 per year in an attempt to attract more participants, especially younger ones, for whom level of compensation seemed to be higher on the list of motivating factors. Eventually, recruitment efficiency and numbers improved as strategies and tactics changed; how- ever, it is difficult to attribute the reason for such improve- ments solely to the increased compensation. It did seem to all stakeholders that compensation has to be substantial to attract and fairly compensate participants for a study of this extent and nature—especially if the study hopes to attract younger participants, including teens. Study Challenges Border Crossing As noted in the informed consent form, one of the risks that participants may have been exposed to was the risk of being detained or arrested or having their vehicle impounded if they drove in areas where cameras were not permitted, such as inter- national border crossings, military or intelligence locations, or manufacturing plants. The Erie County, New York, site and the Seattle, Washington, site were located in close proximity to the Canadian border, and both sites reported that it is com- monplace for many residents in those areas to visit Canada for business or leisure purposes. During the study design phase of the project, SHRP 2 staff and the Coordination Contrac- tor worked together to communicate information about the study to both the U.S. Department of Homeland Security and Canadian border officials in an effort to alleviate their con- cerns about what images were captured by the DAS and to minimize any risk to participants who might travel between the two countries during the course of their participation in the study. However, no formal agreements could be reached with these government agencies, and as a result, individuals who admitted to routinely crossing the U.S.–Canadian bor- der were deemed ineligible. It should be noted that the site contractors later learned that some participants did cross the border, perhaps repeatedly, and without incident. The

81 possible flaws and weakness that could have been engineered out of the system; instead, work-arounds had to be applied. Launch of New Product and CM Moving a product from prototype to production has its own set of challenges. Small design changes, which are often required to allow for effective manufacturing in a production environment, may not be apparent in one-off production. Launching a new CM relationship also requires additional time for both parties to learn to work efficiently with each other. Each company has its unique set of strengths and weak- nesses and its standard operating procedures that have been implemented to aid in the production process. Learning to work within these structures while at the same time launch- ing a new product added complexity to this project since there was no prior knowledge of business processes or the manu- facturing process for this specific product. Consequently, working with a new CM on a new product added significant effort and time. Start of Installations Ideally, at the start of the installation phase at the site contrac- tor facilities, the manufacturing process and product would have had a few months to work out the issues mentioned above, allowing a minimum amount of final inventory to be in stock. Since this was not possible, the demands placed on the supply chain simply had to be managed. Toward that end, concessions were made to try to ensure systems were avail- able to support the installation schedule (which had its own constraints). This amplified some of the inherent weakness in both the design cycle and the CM and its processes, with little time to address root causes. Some of the concessions made to try to mitigate the effect on the study as a whole included using prototype parts to compensate for design delays as well as significant delays in the delivery of parts from vendors. Older board revisions were released for production to backfill demand while new revisions were completed. In addition, VTTI personnel spent significant time on site at the CM to work through issues in real time to try to reduce the learning cycle and to ensure that the project was suitably prioritized by the CM. This last point was probably the key to getting kits delivered in time and with sufficient quality to support project needs and constraints. Once the manufacturing supply caught up with the demand, oversight of the CM was primarily done remotely through weekly status calls, supplemented by visits to the manufac- turing facility every 4 to 6 weeks. During the manufacturing phase of the study, it was determined at different times that the CM’s level of effort being applied to the manufacturing and the repair of the product was less than desired. On two Protocols were established such that the relevant video clips could be shown to the participant and others autho- rized by the participant (e.g., lawyers, police officers, insur- ance adjusters, family members) without releasing the video to anyone. First, the data drive in question had to be sent to the Coordination Contractor or its data extracted as per usual (though this sometimes took longer). Then, the drive’s data had to be processed and the video reviewed. Sometimes, the Coordination Contractor’s notes on the video would be relayed to the participant via the site contractor and this suf- ficed; no further actions were requested or required. Other times, when the participant still wanted to review the video, a secure WebEx meeting was established with the site contractor managing the other login. Further, it was the responsibility of the site contractor to make certain that only those individuals who were supposed to view the video were present and that no one was recording the information in any way. Video requests were generally rejected when the request originated with a stated intent to prosecute other parties. Throughout the study, over 30 requests for video data were received. The effort involved in this “good citizen” activity took much more time than originally anticipated, as each request took up to several hours to isolate the data within the database, review the data, and share the data with the partici- pant one or more times depending on the nature of the event. equipment Issues There were three significant interdependent milestones per- taining to the equipment in the study, and thus the biggest challenge was timing. These milestones included completing the DAS design, establishing a new CM partnership featuring a new product line, and starting installations. There were sub- stantial difficulties with successfully performing all three at once. The following is a discussion of each of the three issues. Design The DAS was a significant development in the evolution of technology for the collection of naturalistic driving data. Though the design was conducted independently of this contract, it also involved a departure from the way that VTTI had developed equipment in the past. Rather than designing it entirely in-house, an outside design firm was contracted to design and lay out the main unit. This proved more diffi- cult than expected, as first one design firm and then a second were unable to fulfill the needs of the DAS design. Ideally, the design would have been complete when the RFP was released for the CM selection. This would have allowed the production prototype phase of the CM selection to be used as beta test- ing of the system and the new CM. This, in turn, would have allowed additional testing time with the systems to identify

82 • Natural disaster. This was the least likely event to affect the project; however, the tsunami that hit the coast of Japan in March 2011 severely damaged the plant that makes the standard connectors used on the PCBs in the DAS. That was the only supplier, and overnight the supply chain was broken with no means to repair it quickly or find another source. The standard lead time was pushed out by 8+ weeks for each of the above cases. By using brokered stock or leveraging our relationship with suppliers and vendors to obtain limited stock, impacts to the project were mitigated to the greatest extent possible. In the case of the connectors, the study team was able to redistribute current inventory to meet immediate demands while waiting for the supply chain to be restored to its normal capacity. Design Adjustments As discussed previously, a number of design modifications occurred during the manufacturing and installation period. This was to be expected given the timing of the completion of the design cycle, the launch of a new CM relationship, and the start of DAS installations on a scale which had never been done before. For all of these, the more experience staff have with the product (the DAS and the large scale of the study) and the process (manufacturing), the fewer changes are typi- cally required. Since all three had aspects that were new, it was expected that changes would be needed. Repair Challenges The primary challenges with developing a consistent method for processing repairs efficiently at the CM fell into three main categories: • Leverage. When the contract started, the study had sig- nificant leverage due to the size of the order; it was ben- eficial to the CM to complete the order from a financial perspective. However, in the case of warranty work, there is no payment; and with repair work being performed at a fixed, prenegotiated level, the potential for return is sig- nificantly less than that for new contracts. Consequently, as the project moved from new production to support mode, the CM’s priorities naturally shifted to new manufacturing contracts. Thus, additional negotiations were required to get the CM to clear the remaining RMA backlog. • Limited resources. This problem affected the repair process in two unique ways. First, because of the total duration of the production run, warranty work was being requested before the completion of production. Since there were a limited number of test stations and a limited number of occasions, the CM chose to reallocate resources away from SHRP 2 NDS activities. At one point, there was a significant turnover in personnel, including the production manager. The next run of boards had a yield rate of approximately 20%, when previously it had been greater than 95%. On another occasion, at the conclusion of a large contract, a significant lay- off occurred, which reduced the number of people who were trained and available to work on the DAS. This required VTTI to spend additional time at the CM to retrain new person- nel. This was particularly detrimental to the return merchan- dise authorization (RMA) process as a working knowledge of the design is necessary for effective troubleshooting of problems. Component Availability Component availability became a problem when the supply chain was disrupted. This factor also had a major impact on the manufacturing process and kit delivery. One effect that was already mentioned was the shift in the industry standard from the PATA to SATA interface, which created a supply shortage of data drives and necessitated a design change to the system. Board-level components are used as building blocks by all elec- tronics designs. Each component manufacturer has its own unique product offerings that, once designed into a system, can only be supplied by that manufacturer. All the companies that use that particular part are dependent on the supply from the manufacturer. If anything happens to disrupt this supply chain, it can affect production schedules. Following are four examples of things that disrupted the supply chain: • Demand fluctuations. On two separate occasions, a com- ponent for which there was a large supply of stock on the market had its supply depleted in less than a month. One of those instances was precipitated by Apple’s announcement of the release of the next iPhone model, which happened to use (and utterly consume) one of the same components as the DAS. • Business strategy. Component manufacturers occasion- ally reduce production levels to free up resources for other products or to decrease supply to drive up price. One of the key suppliers did this during the early phase of production. • Plant emergencies. Like board-level components, availabil- ity of raw materials can also affect supply chain. Early dur- ing the initial release, the one plant that supplied the raw material used in some of the plastic parts was destroyed in a fire. This particular material was used heavily in the automotive industry. As such, the raw material supply to the project vendor was cut to a fraction of the normal sup- ply, as the plant had to distribute its limited supply across all of its customers. This affected both delivery times and quantities available to support the SHRP 2 NDS.

83 throughout the course of the project from study design through the coordination effort. Table 7.2 summarizes the per- centage of staff that worked to accomplish various roles during the study. It should be noted that for most who contributed to the SHRP 2 effort, only a portion of their time was required. However, for a handful of individuals, their full-time devotion to the project was required to administer the study logistics and reporting. This administrative function served as a hub role through which most other functions were coordinated. personnel at the CM working on the product, the choice had to be made between applying resources to making new product or repairing existing product. Since new produc- tion almost always nets more systems per hour than repair, it was decided to let the resources be applied to new pro- duction until that was completed. This decision, however, created a larger backlog of repair items, which took signifi- cant time to clear. • High turnover. In the past 3 years, American Computer Development, Inc. (ACDI) has had at least three RMA man- agers and floor managers in addition to two known layoffs that affected personnel working directly on the DAS. Each time management changed, a change in procedure had to be accommodated. Each time floor personnel changed, training on the product was required, which caused delays in the repair process since the knowledge base was lost with the people that had been involved in production. For all these delays caused by business practices outside the project’s control, VTTI made significant efforts to support the manufacturing process and leverage the business relationship, past and future, to try to mitigate the impact on the study. DAS Supply and Inventory A major challenge facing the Coordination Contractor in providing requisite equipment was the lack of real-time information regarding the site contractor schedules. As dis- cussed in Chapter 3, a weekly installation schedule request was sent to the site contractor to elicit information regarding planned activities. However, these activities were dynamic in nature. A schedule could conceivably look very different mere hours after the reply was sent, and the site contractor staff had more important priorities, such as installing vehicles and managing participants. Table 7.1 illustrates the discrepancy between planned and actual installation and deinstallation activities during March 2013. This month marked the height of the full installation period, a time during which the value of effective allocation of equipment was at a premium. The Coordination Contractor would have benefitted greatly from a real-time mechanism for viewing changes to the schedule at each site as they occurred. Site-Based Facilitation Staffing the Study The Coordination Contractor provided highly complex tech- nical and logistic support to the site contractors. In addi- tion, the Coordination Contractor provided data collection equipment, data management tools, and data storage systems. Over 100 different Coordination Contractor staff worked Table 7.1. Scheduled Versus Actual Activities, March 2013 Site Scheduled Performed Percentage of Scheduled Actually Performed Installations Buffalo 19 16 84% Tampa 21 19 90% Seattle 25 17 68% Durham 22 12 54% Bloomington 0 1 NA State College 8 10 125% Deinstallations Buffalo 7 16 228% Tampa 8 19 237% Seattle 4 17 425% Durham 2 12 600% Bloomington 0 1 NA State College 3 10 333% Table 7.2. Full-Time Equivalent (FTE) Personnel Assigned to Various Roles Role/Function FTE Staff Validation of consented driver images 44 DAS-focused maintenance, troubleshooting, training with installation focus; software upgrades for vehicles 27 Quality control of DAS-based and participant-based variables 20 Maintenance and technical support to computer operations, database 16 Purchase, monitoring, and supply of DAS equipment inventory 7 Administration of study logistics and reporting 6 Study oversight and IRB 4 Total 124

84 The need for an efficient workflow system to manage the processing and tracking of the huge volume of files generated during this project should not be underestimated. Workflow agents can be thought of as simple robots that specialize in a single task, with the workflow providing oversight of the status of each discreet step. The workflow engine is as critical as any other component of the entire architecture. participant Management Effectiveness of Recruiting Methods Recruiting Teens and Older Drivers As early as 2011, an analysis of the recruited cohort reflected deficiencies in attracting both younger and older drivers to the study. By August 2011, the Coordination Contractor had put a protocol in place to include teen children of primary drivers as additional primary drivers even though they shared a household with an existing primary driver. This strategy enjoyed limited success, as Figure 7.1 portrays. Ultimately, the biggest obstacle to success with these groups was the exclusive nature of the initial eligible vehi- cle list. Younger and older drivers alike were less likely to drive these newer vehicles. The Coordination Contractor addressed this issue by expanding its vehicle fleet to include older vehicles, conceding that the data set for the younger and older segments of the study population would be less robust due to impoverished or nonexistent vehicle network configurations. Figure 7.2 depicts the composition of the group of vehicles driven by participants ages 16–25 and 76 and older. Figure 7.3 shows the number of installations among younger and older drivers throughout the course of the study. Note the increase in installations among all four sample cells in this cohort with the introduction of legacy and basic vehi- cles to the vehicle fleet. Installations increased 166% for these Technical Support The Coordination Contractor established a separate trusted domain (shrp2nds) and provisioned SHRP 2 users under that domain to share designated network resources at the Coordina- tion Contractor’s facility. This arrangement allowed the Coor- dination Contractor technical support staff to access shrp2nds domain resources (e.g., desktops, laptops, site servers) without having to establish a separate set of domain credentials just to support SHRP 2 issues. The Coordination Contractor also worked with network engineers at each of the site contractor locations to facilitate network traffic in support of the SHRP 2 study. This effort was critical to establish communication on the high-speed research networks used to send collected driving data from the site contractors to the Coordination Contractor. S06 implemented Request Tracker (RT) as a ticketing sys- tem to track (and work) any SHRP 2 support issues that arose. Currently, the separate facilities that serve as data reten- tion facilities are all located on the campus of Virginia Tech. Ideally, they should be geographically distant from one another in case a large-scale disaster (e.g., flood, hurricane, tornado, earthquake) was to strike in the general vicinity of the campus. This was not the result of poor planning but rather due to budget constraints imposed on S06. The initial infrastructure design was a cooperative process between the Coordination Contractor system administrators and the university’s system administrators. Implementation of the new environment followed with both teams sharing installation and management responsibilities, although the university retained root control until the Coordination Con- tractor’s technical resources gained sufficient experience to assume full control. While the architecture contemplated sig- nificant investment in file server storage and database stor- age, the estimates turned out to be too low: file server storage doubled in capacity and database storage tripled over the course of the data collection phase. The shortfall was primar- ily the result of the study duration being extended beyond its initially planned period. The Coordination Contractor’s technical staff had significant cumulative experience with networking, hardware, software, and database architecture, implementation, configuration, maintenance, and trouble- shooting; but those skills were focused within a few key indi- viduals. Over the course of this project (and as a matter of necessity), widespread cross-pollination of the expertise was required to support this effort from an information technol- ogy perspective. Staff headcount was five full-time equiva- lents, although the resources working on the project at any one time changed to focus on different aspects of the proj- ect, as needed. The personnel involved comprised system architects, network engineers, data center engineers, system administrators, developers, and statisticians. Figure 7.1. Additional primary participants, ages 16–20, contributing at least 4 months of data, by site.

85 was not fully anticipated how no-shows, especially among the younger participants, would affect overall study progress. When considering the impact of participants who fail to show up for an appointment, it is important to note that it costs virtually as much in terms of staff resources for a non replaced no-show as for an actual installation. Thus, no-shows were a significant drain on study resources. Future studies should probably factor some proportion of no-shows into the overall schedule, especially if a more substantial proportion of younger participants are targeted. Alternatively, or in conjunction with building in such allow- ances, future planners of similar studies should consider including incentives for participants to show up at their scheduled time slots. In this study, one incentive was the offer of additional compensation for participants who showed up on time for their initially scheduled visit, which produced modest results. Other options explored included having recruits “on call” for installation (i.e., those who agreed to be called on short notice to fill scheduling voids). All sites provided Saturday hours to accommodate schedules, as many recruits worked or attended school during the week. groups between August and December 2011. Of all older and younger drivers in the study, 93% were instrumented after the decision to include legacy and basic vehicles. To a lesser extent, both method of recruitment and mode of enrollment varied by age group. Similarly, the mode of com- munication employed by recruits in expressing their interest in the study differed by age group, as shown in Figure 7.4. The overwhelming majority of participants age 35 and under accessed the study via the web-based screener, while par- ticipants ages 36–75 favored contact through the call center. Clearly, future studies will need to maintain a strong online presence if there is the goal to attract younger recruits. Participant Issues No-Shows When planning the study, each site was thoughtfully provi- sioned and staffed to fulfill the desired level of throughput of installations and related activities each day. However, it Figure 7.4. Breakdown of mode of expression of interest in participation, by age group.Figure 7.2. Vehicles driven by younger and older drivers contributing at least 4 months of data, by class. Figure 7.3. Installations among younger and older drivers with addition of legacy and basic vehicles, over time.

86 was that the vehicle was sold or repossessed without the site contractor’s knowledge. Supporting Data Analyses During Data Collection Concurrent with the data ingestion phase, the initial data analysis efforts (Project S08) were also under way. Although the S08 analyses added some visibility to the data set and its potential applications, supporting these analyses while actively engaging in data collection and processing activities proved very difficult. These difficulties existed particularly when exposure had to be assessed and when sampling criteria had to be developed and applied. In both cases, priorities in the processing queue may have introduced some unwanted biases, which will not be fully understood until the data set becomes static. Furthermore, satisfying the needs of analysis projects in some cases reduced the efficiency with which data could be processed (e.g., when particular events were searched for within trips whose processing was not complete). Any similar efforts in the future should consider avoiding supporting data analysis efforts until a static data set has been developed, or consider processing approaches that are sequential in nature (although these are much less efficient). Missing Participants During the conduct of the SHRP 2 NDS, there were a few occasions when a participant was characterized as “miss- ing.” This condition generally occurred when a participant was not responsive to the site contractor or when his or her vehicle’s DAS modem suddenly, with no obvious tech- nical reason, stopped communicating. When these condi- tions were noted, participants who were identified as missing were sought by both site contractors and the Coordination Contractor in a variety of ways. Tracking software was often used to review recent GPS coordinates to determine if the participant was routinely parking his or her vehicle at a par- ticular address. As needed, other news and social media were reviewed to assess whether a participant was believed to be deceased. As additional information was exchanged between the site and Coordination Contractors, study personnel were often able to attempt contact after normal business hours or travel to remote locations to retrieve or service the DAS. Ten participants were reported to the Coordination Contractor as missing. In many cases, participants were located; however, at study end, four participants remained missing, with little to no hope of recovering the equipment and data. The most common reason for a participant/vehicle pairing to go missing