National Academies Press: OpenBook

Site-Based Video System Design and Development (2012)

Chapter: Chapter 11 - Performance Evaluation

« Previous: Chapter 10 - Trajectory Refinement and Estimation of Motion Variables
Page 68
Suggested Citation:"Chapter 11 - Performance Evaluation." National Academies of Sciences, Engineering, and Medicine. 2012. Site-Based Video System Design and Development. Washington, DC: The National Academies Press. doi: 10.17226/22836.
×
Page 68
Page 69
Suggested Citation:"Chapter 11 - Performance Evaluation." National Academies of Sciences, Engineering, and Medicine. 2012. Site-Based Video System Design and Development. Washington, DC: The National Academies Press. doi: 10.17226/22836.
×
Page 69
Page 70
Suggested Citation:"Chapter 11 - Performance Evaluation." National Academies of Sciences, Engineering, and Medicine. 2012. Site-Based Video System Design and Development. Washington, DC: The National Academies Press. doi: 10.17226/22836.
×
Page 70
Page 71
Suggested Citation:"Chapter 11 - Performance Evaluation." National Academies of Sciences, Engineering, and Medicine. 2012. Site-Based Video System Design and Development. Washington, DC: The National Academies Press. doi: 10.17226/22836.
×
Page 71
Page 72
Suggested Citation:"Chapter 11 - Performance Evaluation." National Academies of Sciences, Engineering, and Medicine. 2012. Site-Based Video System Design and Development. Washington, DC: The National Academies Press. doi: 10.17226/22836.
×
Page 72
Page 73
Suggested Citation:"Chapter 11 - Performance Evaluation." National Academies of Sciences, Engineering, and Medicine. 2012. Site-Based Video System Design and Development. Washington, DC: The National Academies Press. doi: 10.17226/22836.
×
Page 73
Page 74
Suggested Citation:"Chapter 11 - Performance Evaluation." National Academies of Sciences, Engineering, and Medicine. 2012. Site-Based Video System Design and Development. Washington, DC: The National Academies Press. doi: 10.17226/22836.
×
Page 74

Below is the uncorrected machine-read text of this chapter, intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text of each book. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.

68 C h a p t e r 1 1 The challenges of tracking vehicles using video imaging have been described and methods for overcoming those challenges explored. Now results of the prototype Site Observer are pre- sented, and its feasible use as a fully automated motion and trajectory sensing device for robust use in traffic situations is considered. From a hardware standpoint, the system per- formed flawlessly during the latter part of a Michigan winter, even with a semi-permanent installation; camera and com- puter hardware were located at the intersection for several months. Not only were there no failures, the real-time system did not generate any dropped or missing image frames, and there were no times when the system froze or otherwise cor- rupted or lost source data. There is no doubt that the system runs and generates data, so the question reduces to whether it produces useful data, and if so, how complete and accurate are the data. This question is addressed in the following. It is important to consider current performance in the context of a prototype for which additional refinement and improvement are possible and probably necessary. The key point to demonstrate is that the system operation—extracting features from individual cameras and combining them with- out further access to video images—provides valid data for safety analysis (for example, in conflict metric analysis). It is not expected that the system will resolve every single vehicle with the best possible precision; in its current functional form, some trajectories generated may be corrupted; an example is seen in Figure 11.4. Such corrupted trajectories are rare, and although a precise error rate is not objectively known, based on manual video review, it appears less than 1% of cases are prone to these kinds of problems. Such cases are easily screened out (for example, the KF automatically computes error levels) but are retained so the baseline performance of the system is not disguised. Although further analysis on the completeness of the data is possible, it is already clear that this is not an issue for the low traffic densities seen in the current study; minimal data loss means there is minimal bias in any conflict analysis. The system was run for approximately 30 h during February and March 2010, at the installation site in Ann Arbor (see Chapter 7). During this time 17,593 vehicles passed through the intersection, with a breakdown by type shown in Figure 11.1. For comparison, Figure 11.2 shows corresponding numbers from a weekday 24-h period, in March 2010; these data were provided by Washtenaw County Road Commission, and obtained from the existing traffic signal system. The Site Observer was run over five separate days, capturing data for approximately 6 h in each case. A further 30-min run was added to capture data using instrumented vehicles. The two figures show similar trends, although the Site Observer data have greater mean counts per hour, as expected, because the system was run only during daytime hours. Provided sufficient clusters are detected, the vehicles can be localized and tracked. Outgoing vehicles generally seem more consistent and well populated with clusters than do incoming vehicles, and it is possible that faster moving vehi- cles tend to have shorter incoming cluster tracks than do slower ones. This is tested with a sample of 200 trajectories (see Figure 11.3); all are through vehicles traveling north to south or south to north. Here the blue crosses correspond to incoming vehicles, and red circles are for outgoing vehicles. Both cases are shown because the road geometry is not sym- metric; the ground rises to the north and falls to the south. It is clear that some very long tracks are seen in the outgoing vehicles, as great as 250 m, which is well beyond expectations (and nothing beyond 150 m should be deemed reliable). Inter- estingly, although the upper plot shows some speed depen- dence (the highest speed incoming tracks tend to be shorter), overall the only clear trend is for outgoing cluster tracks to be longer than incoming ones. For vehicles arriving from the north, there is a sharp cutoff at approximately 80 m, the rea- son for which is unknown. Moving now onto more specific measures of performance, the team’s focus is estimating (1) the accuracy of positioning and (2) the accuracy of derived motion variables. Performance Evaluation

69 NE ES SW WN 0 500 1000 1500 2000 2500 3000 3500 4000 4500 5000 Left Turns NW EN SE WS 0 500 1000 1500 2000 2500 3000 3500 4000 4500 5000 Right Turns NS EW SN WE 0 500 1000 1500 2000 2500 3000 3500 4000 4500 5000 Straight Figure 11.1. Traffic volumes during the 30-h study period (NE: enter from the north, exit toward the east, and so forth; 17,593 vehicle trajectories were recorded). NE ES SW WN 0 500 1000 1500 2000 2500 3000 3500 4000 Left Turns NW EN SE WS 0 500 1000 1500 2000 2500 3000 3500 4000 Right Turns NS EW SN WE 0 500 1000 1500 2000 2500 3000 3500 4000 Straight Figure 11.2. Traffic volumes during a comparable 24-h study period (from Traffic Signal Camera System, courtesy of Washtenaw County Road Commission; 11,622 vehicle movements were detected). critically on camera coverage near the center of the intersec- tion and that subject to this, the position errors are no worse than approximately 0.5 m within the intersection, increasing to approximately 1 m at distances of 100 m. A way to confirm these estimates and make them more precise is to look at trajec- tory dispersion. Figure 11.4 illustrates the relevant type of data available from the Site Observer; centroid plots are made after criteria were applied for intersection entry and exit leg choice. Apart positioning accuracy Unfortunately, a full and objective reference data set is not available for validation of vehicle positioning. Even the instru- mented vehicles have limited precision in their positioning, with GPS position guaranteed only within a few meters. It is possible to use manual digitization of images to provide this reference, but this is tedious and would tell little beyond what has been seen informally: that position accuracy depends

70 from the one corrupt trajectory, the vehicle movements appear highly consistent. It undoubtedly is the case that some of the lateral dispersion seen is attributable to real-world variations in vehicle path, but some of the dispersion is caused by positioning errors. The two cannot be separated, but the overall result puts lower limits on the system accu- racy. The dispersion appears greatest for left-turning vehicles (with only two cameras covering this movement) and possi- bly at distances further from the intersection. Thus, lateral dispersion using samples of vehicles is ana- lyzed. Five hundred trajectories of north-to-south vehicles were sampled and their lateral dispersion evaluated at dis- tances of 50 and 25 m, both to the north and to the south of the intersection. Results are shown in Figure 11.5 and 6 8 10 12 14 16 18 20 22 24 0 50 100 150 200 250 speed at intersection center (m/s) fu rth er d ist an ce o f c lu st er tr ac kin g Incoming (South) Outgoing (North) 6 8 10 12 14 16 18 20 22 24 0 50 100 150 200 250 speed at intersection center (m/s) fu rth er d ist an ce o f c lu st er tr ac kin g Incoming (North) Outgoing (South) Figure 11.3. Lengths of cluster tracks: through vehicles, north and south directions.

71 Table 11.1. The mean values are biased to negative values, but this likely is because only an approximate value was used for the x-coordinate of the lane center, or it could be that drivers tend to follow the right lane marker more when approaching the intersection (additional analysis is easily possible; the power of this type of data set is that with addi- tional effort, any number of detailed investigations are made feasible). The team’s focus is on the dispersion values, which are very constant indeed—out to 50 m, there appears to be no disadvantage caused by pixilation errors. From this, it seems that the RMS errors are slightly higher than those sug- gested in Chapter 5. The ideal requirements for 20-cm RMS accuracy for the prototype system have not been met; rather, these results indicate values approximately twice that of the target, with RMS errors of approximately 40 cm. Although slightly disappointing, these are first generation results from M et er s Meters Meters Meters M et er s M et er s Figure 11.4. Samples of detected trajectories in selected directions. The central plot shows a single corrupted trajectory. Less than 1% of trajectories are this type and can be removed by screening. -4 -2 0 2 4 0 50 100 150 200 dispersion for Y = 50 -4 -2 0 2 4 0 100 200 300 dispersion for Y = 25 -4 -2 0 2 4 0 100 200 300 dispersion for Y = 25 Lane center offset (m) -4 -2 0 2 4 0 50 100 150 200 dispersion for Y = 50 Lane center offset (m) Figure 11.5. Frequency distributions for lateral dispersion within the lane for through vehicles. North-to-south movement: upper left, 50 m north; upper right, 25 m north; lower left, 25 m south; and lower right, 50 m south.

72 performance of motion estimation from the KF. The results, shown below, are reasonably accurate, except around the ini- tial points. The reason for this in not definitely known but seems likely to be associated with increased variation in clus- ter positions when the vehicle is first detected, approaching the intersection from distance. After this initial deviation, both velocity and acceleration values match the vehicle quite well, within approximately 1 ms-1 and 1 ms-2, respectively. In Figure 11.7, the vehicle is approaching at a speed that will exacerbate the effect of early cluster detection, whereas for the left-turning vehicles in Figure 11.8 the challenge is more one of near-zero velocity (as well as reduced camera cover- age); initial values are not badly compromised, and although the prototype system, and there is every reason to expect improvement as various aspects are optimized (see Chapter 13 for additional discussion of this topic). Speed and acceleration evaluation For speed and acceleration, independent reference informa- tion does exist, albeit for a small number of cases. A number of passes were made through the intersection using instru- mented vehicles. The onboard sensors measure a large num- ber of variables, including speed, longitudinal acceleration, lateral acceleration, and yaw rate. Four particular (mild) con- flict events were staged, two events with a legal right turn on red ahead of and into the path of the other vehicle going straight, and two events with a permissive left turn across path (LTAP/OD) in front of an oncoming through vehicle. Figure 11.6 shows the various paths. The main interest here is not the conflict metrics (although these do show up in the next chapter) but on checking the Table 11.1. Mean and Standard Deviations of Lateral Offset Distance North from Intersection Center Mean Offset (m) Standard Deviation (m) +50 -0.11 0.63 +25 -0.42 0.69 -25 -0.38 0.62 -50 -0.54 0.55 M et er s Meters Figure 11.6. Simulated conflict paths using instrumented vehicles. Case 1 Case 2 Figure 11.7. Speed and acceleration estimates for through vehicles, right turn into path. Blue: video measurement; red dashes: vehicle measurement.

73 Case 1 Case 2 Figure 11.8. Speed and acceleration estimates for slow-turning vehicles, right turn into path. Blue: video measurement; red dashes: vehicle measurement. accelerations are not especially well determined as the turn- ing vehicle first starts to accelerate, the velocity estimates are stable and match well. For the cases with left turn across path, the through vehicle again maintains its speed. In fact, for the case shown here (Figure 11.9), a period of acceleration is captured well by the camera system. The velocity and accelerations are not perfect, but they give usable quantitative information, sufficient for calculating time-to-collision values and acceleration or iden- tifying braking events. In Figures 11.10 and 11.11, the KF results for the turning vehicle can be seen; the yaw rate (turn- ing rate) is captured with a high degree of accuracy, and use- ful information on speed and acceleration is provided. Overall, accepting that there are imperfections because video images, rather than installed sensors, are being used, the Site Observer clearly achieves a highly useful degree of motion capture. The velocity estimates are used in Chapter 12 when TTC distribu- tions are found. Figure 11.9. LTAP/OD conflict—through vehicle. Blue: video based; red dashes: vehicle based.

74 Figure 11.10. LTAP/OD conflict—turning vehicle. Blue: video based; red dashes: vehicle based. Figure 11.11. LTAP/OD conflict—yaw rate and lateral acceleration estimates for the turning vehicle. Blue: video based; red dashes: vehicle based.

Next: Chapter 12 - Conflict Analysis »
Site-Based Video System Design and Development Get This Book
×
 Site-Based Video System Design and Development
MyNAP members save 10% online.
Login or Register to save!
Download Free PDF

TRB’s second Strategic Highway Research Program (SHRP 2) Report S2-S09-RW-1: Site-Based Video System Design and Development documents the development of a Site Observer, a prototype system capable of capturing vehicle movements through intersections by using a site-based video imaging system.

The Site Observer system for viewing crashes and near crashes as well as a basis for developing objective measures of intersection conflicts. In addition, the system can be used to collect before-and-after data when design or operational changes are made at intersections. Furthermore, it yields detailed and searchable data that can help determine exposure measures.

This report is available in electronic format only.

READ FREE ONLINE

  1. ×

    Welcome to OpenBook!

    You're looking at OpenBook, NAP.edu's online reading room since 1999. Based on feedback from you, our users, we've made some improvements that make it easier than ever to read thousands of publications on our website.

    Do you want to take a quick tour of the OpenBook's features?

    No Thanks Take a Tour »
  2. ×

    Show this book's table of contents, where you can jump to any chapter by name.

    « Back Next »
  3. ×

    ...or use these buttons to go back to the previous chapter or skip to the next one.

    « Back Next »
  4. ×

    Jump up to the previous page or down to the next one. Also, you can type in a page number and press Enter to go directly to that page in the book.

    « Back Next »
  5. ×

    To search the entire text of this book, type in your search term here and press Enter.

    « Back Next »
  6. ×

    Share a link to this book page on your preferred social network or via email.

    « Back Next »
  7. ×

    View our suggested citation for this chapter.

    « Back Next »
  8. ×

    Ready to take your reading offline? Click here to buy this book in print or download it as a free PDF, if available.

    « Back Next »
Stay Connected!