Below is the uncorrected machine-read text of this chapter, intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text of each book. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.
54 The following information summarizes SHRP 2 S04A Task 3 activities, with the objective of further evaluating the capa- bilities of the mobile data collection vendors who partici- pated in the SHRP 2 S03 Roadway Measurement System Evaluation (Rodeo) project. This was accomplished through documenting each vendorâs ability to collect roadway geo- metric data and some selected inventory data features on two of the six Rodeo sites (6S and 7N). Task milestones included the following: ⢠May 12, 2010: Initial instructions and data format to vendors. ⢠May 24, 2010: Final instructions and data format to vendors. ⢠June 1, 2010: Deadline for vendor questions regarding data submittal. ⢠June 15, 2010: Final data submittal from vendors. ⢠July 27, 2010: Presentation of results to Expert Task Group (ETG). B.1 Data Collection Seven vendors participated by completing three data collec- tion runs for each of the two roadway segments identified as 6S and 7N. The seven data features collected for each roadway segment included roadway section (location/length), hori- zontal curve, grade, cross slope, lane width, paved shoulder width, and regulatory signs. B.2 Evaluation B.2.1 Precision Versus Accuracy Evaluation Each vendor submitted its data to the SHRP 2 S04A contractor for evaluation in terms of accuracy and precision (Figure B.1). A determination of both accuracy and precision was made. For features involving location: ⢠Precision was evaluated through the following: 44 Determining the geometric mean for the three sets of coordinate pairs (projected) (i.e., average of X, aver- age of Y). 44 Computing the Euclidian distance between each coordi- nate pair and the geometric mean coordinates. ⢠Accuracy was evaluated through the following: 44 Computing the Euclidian distance between each coordi- nate pair and the ground truth coordinate pair. Ground truth data sets were acquired from an independent surveyor for this project. For features involving grade, cross slope, and width: ⢠Precision was evaluated through the following: 44 Computing the average value of three repetitions. 44 Computing the difference between each repetition and the average value. ⢠Accuracy was evaluated through the following: 44 Computing the difference between each repetition and the ground truth. Figure B.2 presents how accuracy and precision were approached in the evaluation. Accuracy compares measure- ment to ground truth. Precision, however, focuses on repeat- ability, specifically the relationship among consecutive measurements by one vendor at the same location. High accuracy and precision are most desirable. Both aspects were critical to selecting the right vendor and technology for the project. Figure B.3 gives an example of precision and accuracy results for one of the seven features, roadway section. Here the assessment for the location of the beginning and ending points of sections is given; a similar assessment was done for A p p E n D i x B Technology Evaluation
55 section length. The data in the figure shows the precision and accuracy of reporting the right begin and end location for each section. Vendors 1, 3, and 5 reported the most precise and accurate information. Figure B.4 presents the results of horizontal curve assess- ment. Vendors were evaluated for locations of point of cur- vature (PC) and point of tangency (PT), curve radius, and curve length. The figure shows the difference between vendor data and the ground truth by showing the numbers of curves in each range (less than 3 ft, between 3 and 25 ft, between 25 and 50 ft, and more than 50 ft). B.2.2 Use of GIS for Evaluation Previously collected surveyor data were provided to the research team for use in the vendor evaluation. These data were provided in multiple formats and coordinate systems. The coordinate system of each of the features was deter- mined, and a single, common coordinate system was selected for evaluation purposes. Data sets, both computer-aided design (CAD) and tabular, were converted to ArcGIS for- mat and projected onto the common coordinate system. Where necessary, appropriate attributes, such as sign type or curve radius, were added to the representative GIS-based features. Feature geometry (e.g., length and begin/end coor- dinates) was computed and added as attributes to each feature. The survey-provided grade and cross-slope data had to be addressed in a different manner. Specifically, these data were provided in tabular format, with their locations conveyed using project stationing. Therefore, station-based linear ref- erencing systems (LRS) were created for each project site 5.1 5.2 5.3 5.4 5.5 5.6 5.7 5.8 5.9 6 6.1 1.2 1.25 1.3 1.35 1.4 1.45 âXâ = Computed Average 5.1 5.2 5.3 5.4 5.5 5.6 5.7 5.8 5.9 6 6.1 1.2 1.25 1.3 1.35 1.4 1.45 âGâ = Ground Truth G Figure B.1. Calculating precision and accuracy. High accuracy, but low precision. High precision, but low accuracy. Figure B.2. Accuracy versus precision.
Figure B.3. Section begin and end points, accuracy and precision by vendor. Figure B.4. Horizontal curve evaluation summary. 56
57 Figure B.5. Example comparison of vendor data with ground truth data. (6S and 7N) in ArcGIS. A grade and cross-slope linear event table was created, and dynamic segmentation was used to create GIS-based features for each round. This process auto- matically associated the appropriate grade and cross-slope attributes with each feature. Figure B.5 presents an example where curve length, curve radius, and segment grade from the vendor are compared to the ground truth data using centerline shape files. B.3 Results The vendor evaluation results were presented to the ETG at a meeting in Washington, D.C. The ETG members scored each vendor based on whether they were positive, neutral, or nega- tive for each data element and on the importance level of that data element. Possible scores ranged from -18 (worst perfor- mance) to +18 (best performance). The ETG made the rec- ommendation to prequalify three of the seven vendors based on the data presented, minimum accuracy requirements, and data item importance level. The S04A team, working with SHRP 2, conducted technical briefings with the disqualified vendors. Based on the evaluation, a minimum accuracy table (Table B.1) was developed and incorporated into the S04B Table B.1. Required Minimum Accuraciesa Data Element Accuracy Requirement Curvature length 50 ft Curvature radius 50 ft PC 25 ft PT 25 ft Grade (+ or -) 1.0% Cross slope/superelevation 1.0% Lane width 1 ft Paved shoulder width 1 ft Inventory features (signs) location 7 ft a Initial minimum accuracy requirements that were changed. request for proposals and qualifications (RFP&Q) that the three prequalified vendors were provided. Table B.2 presents the importance levels for each data element investigated in the vendor evaluation. The evaluation results are summarized in Table B.3. Defi- nitions for each performance category are provided.
58 Table B.2. Data Importance Levels Data Element Importance Level Curvature length High Curvature radius High PC High PT High Grade (+ or -) High Lane width Moderate Paved shoulder width Moderate Inventory features (signs) location Low Sign type (correct MUTCD) Low Table B.3. Vendor Evaluation Results Roadway Feature Number of Vendors by Performance Positive Neutral Negative Curve: PC/PT location 2 3 2 Curve: Length 3 2 2 Curve: Radius 4 1 2 Roadway: Longitudinal grade 2 3 2 Roadway: Cross slope 0 5 2 Roadway: Lane width 6 1 0 Roadway: Paved shoulder width 5 1 1 Sign: Location 4 3 0 Sign: Message (MUTCD only) 3 3 1 Note: Positive (score of 1): The majority of the data met the minimum accuracy requirement. Neutral (score of 0): Roughly half of the data met the minimum accuracy requirement. Negative (score of -1): The majority of the data did not meet the minimum accuracy requirement.
