National Academies Press: OpenBook

Quality Management of Pavement Condition Data Collection (2009)

Chapter: Chapter Four - Quality Management Practices

« Previous: Chapter Three - Data Quality Management Concepts
Page 30
Suggested Citation:"Chapter Four - Quality Management Practices." National Academies of Sciences, Engineering, and Medicine. 2009. Quality Management of Pavement Condition Data Collection. Washington, DC: The National Academies Press. doi: 10.17226/14325.
×
Page 30
Page 31
Suggested Citation:"Chapter Four - Quality Management Practices." National Academies of Sciences, Engineering, and Medicine. 2009. Quality Management of Pavement Condition Data Collection. Washington, DC: The National Academies Press. doi: 10.17226/14325.
×
Page 31
Page 32
Suggested Citation:"Chapter Four - Quality Management Practices." National Academies of Sciences, Engineering, and Medicine. 2009. Quality Management of Pavement Condition Data Collection. Washington, DC: The National Academies Press. doi: 10.17226/14325.
×
Page 32
Page 33
Suggested Citation:"Chapter Four - Quality Management Practices." National Academies of Sciences, Engineering, and Medicine. 2009. Quality Management of Pavement Condition Data Collection. Washington, DC: The National Academies Press. doi: 10.17226/14325.
×
Page 33
Page 34
Suggested Citation:"Chapter Four - Quality Management Practices." National Academies of Sciences, Engineering, and Medicine. 2009. Quality Management of Pavement Condition Data Collection. Washington, DC: The National Academies Press. doi: 10.17226/14325.
×
Page 34
Page 35
Suggested Citation:"Chapter Four - Quality Management Practices." National Academies of Sciences, Engineering, and Medicine. 2009. Quality Management of Pavement Condition Data Collection. Washington, DC: The National Academies Press. doi: 10.17226/14325.
×
Page 35
Page 36
Suggested Citation:"Chapter Four - Quality Management Practices." National Academies of Sciences, Engineering, and Medicine. 2009. Quality Management of Pavement Condition Data Collection. Washington, DC: The National Academies Press. doi: 10.17226/14325.
×
Page 36
Page 37
Suggested Citation:"Chapter Four - Quality Management Practices." National Academies of Sciences, Engineering, and Medicine. 2009. Quality Management of Pavement Condition Data Collection. Washington, DC: The National Academies Press. doi: 10.17226/14325.
×
Page 37
Page 38
Suggested Citation:"Chapter Four - Quality Management Practices." National Academies of Sciences, Engineering, and Medicine. 2009. Quality Management of Pavement Condition Data Collection. Washington, DC: The National Academies Press. doi: 10.17226/14325.
×
Page 38
Page 39
Suggested Citation:"Chapter Four - Quality Management Practices." National Academies of Sciences, Engineering, and Medicine. 2009. Quality Management of Pavement Condition Data Collection. Washington, DC: The National Academies Press. doi: 10.17226/14325.
×
Page 39
Page 40
Suggested Citation:"Chapter Four - Quality Management Practices." National Academies of Sciences, Engineering, and Medicine. 2009. Quality Management of Pavement Condition Data Collection. Washington, DC: The National Academies Press. doi: 10.17226/14325.
×
Page 40
Page 41
Suggested Citation:"Chapter Four - Quality Management Practices." National Academies of Sciences, Engineering, and Medicine. 2009. Quality Management of Pavement Condition Data Collection. Washington, DC: The National Academies Press. doi: 10.17226/14325.
×
Page 41
Page 42
Suggested Citation:"Chapter Four - Quality Management Practices." National Academies of Sciences, Engineering, and Medicine. 2009. Quality Management of Pavement Condition Data Collection. Washington, DC: The National Academies Press. doi: 10.17226/14325.
×
Page 42

Below is the uncorrected machine-read text of this chapter, intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text of each book. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.

30 This chapter focuses on the specific quality management principles and techniques currently being followed by trans- portation agencies for pavement condition data collection. It discusses the various approaches and tools used for quality control, quality acceptance, and independent assurance. The creation and implementation of a comprehensive data collection quality management program is a very important step to ensure that quality data are collected. However, as reported in the previous sections, a significant percentage of highway agencies (approximately half ) still do not have such procedures in place. With respect to how these plans are developed, the survey showed that most agencies pre- pare their own quality acceptance plans, whereas the quality control plans are developed either by the agency, the data collection service provider, or as a collaborative effort. Only one agency reported having used a third-party contractor for developing the pavement data collection quality manage- ment plan. The following sections discuss the main techniques and procedures used for quality control, acceptance, and inde- pendent assurance. As discussed in chapter three, the dis- tinction between quality control and acceptance activities depends on how the activities are incorporated into the man- agement plan, rather than the activities themselves. Many of the tools used for quality control and quality assurance are the same. To avoid duplication, this chapter discussed the various activities and tools organized in the most common configuration. QUALITY CONTROL Common quality control activities include personnel training and certification, equipment calibration, certification and ver- ification, production quality data verification, and on-vehicle and office data checks. These are covered in detail in the following sections after a brief discussion on pavement con- dition data variability and quality control planning for control- ling this variability. Quality Control and Variability The purpose of quality control is to quantify the variability in the process, maintain it within acceptable limits, identify the source of variability that can be controlled, and take the neces- sary production adjustments to minimize the “controllable” variability. It is also important that the quality control process detect problems soon, before large quantities of data have to be re-collected. The sources of variability for the various pavement condition indicators were discussed in detail in chapter three. Contents of a Quality Control Plan Based on the examples reviewed, it becomes apparent that a comprehensive quality control plan typically includes the following elements: • Clear delineation of the responsibilities; • Documented (and available) manuals and procedures; • Training of survey personnel; • Equipment calibration, certification, and inspection procedures; • Equipment and/or process quality verification procedures (e.g., testing of control sections) before starting and during production testing; and • Checks for data reasonableness, consistency, and com- pleteness. The survey revealed that a large percentage of the respon- dents (64%) have a formal data collection quality control plan or require the data collection service providers to develop such a plan (Figure 14). It is noted that some of the agencies that checked the “other” option indicated they have procedures for only some of the pavement condition indicators or that such a plan was under development. Approximately half of the data collection quality control plans were developed by the service provider collecting the data. Although it was not asked specifically, it is hypothesized that this corresponds to all or most of the services being contracted. It is also interesting to note that a shorter survey sent to pavement data collection service providers showed that all the service providers have a formal data collection quality control plan; however, none of them provided a copy and several indicated that the plan was project-specific or proprietary. This last result is an indication of the impor- tance data collection companies place on the quality control procedures. CHAPTER FOUR QUALITY MANAGEMENT PRACTICES

31 All the data collection service providers responded that they use calibration of equipment and/or analysis criteria before the data collection, and software routines that check if the data are within the expected ranges for missing road segments or data elements, and for inconsistencies in the data. At least half of the service providers also indicated using the other techniques. An illustrative example of a pavement data collection quality control plan is presented in Figure 15. The figure presents examples of the activities that are typically conducted before, during, and after the production process, rather than a comprehensive list of all available tools and processes. For example, the Maryland State Highway Administration (MDSHA) is one of the agencies that reported having a formal quality control process for its in-house pavement data col- lection activities. The agency developed a detailed quality management program when transitioning from windshield distress data collection to an automated system to measure smoothness, rutting, and cracking (66). In the case of the automated crack detection, the quality control plan includes (1) review of completeness, (2) review of section-level data, and (3) review of data management. The first step simply ver- ifies that all fields are processed. The section-level review is conducted by examining approximately 50% of the sections. The operator also verifies that the data have been saved and inputs a subjective evaluation of the crack detection process (good, fair, and poor). Further details on this approach are provided in a case study presented in chapter five. Personnel Training and Certification It is very important that the personnel operating the equipment or conducting the visual surveys are continuously trained. This is particularly critical given the current environment in the transportation profession, where the work force is highly mobile, and technicians and engineers change positions and employers quite frequently. Training is even more impor- tant for the distress surveys because the classification of the distresses is somehow subjective. Personnel Training Personnel training is necessary for obtaining repeatable and reproducible pavement condition data. It affects manual, semi- automated, and automated practices. Adequate training helps improve the consistency and accuracy of visual surveys and proper operation of the equipment using automated or semi- automated procedures. According to the survey of practice, pavement evaluation personnel are trained mostly through on-the-job training from experienced staff and in-house training programs. The pavement condition data collection is primarily composed of experienced technicians (69% have more than 6 years and 26% more than 10 years of experience). They hold associate degrees (44%) or high-school diplomas (39%), with only a small per- centage having bachelor’s and graduate degrees. The pavement data collection staff for the pavement data collection service providers appears to be a little less experienced (57% have more than 6 years and 14% more than 10 years) and have a 64.3% 19.6% 5.4% 10.7% 0% 20% 40% 60% 80% Yes No Not Sure No Response 7.1% 1.8% 23.2% 32.1% 0% 20% 40% 60% Other Prepared by independent third party Developed by Agency Prepared by data collection contractor FIGURE 14 Percentage of highway agencies having a formal quality control plan. From the available quality management tools and meth- ods, the most common methods/tools used for quality control are the following (in order of decreasing fre- quency, the percentage of agencies citing each method/ tool is provided within brackets): 1. Calibration of equipment and/or analysis criteria before the data collection [94%], 2. Testing of known control segments before data collection [94%], 3. Periodic testing of known control segments during production [81%], 4. Software routines that check if the data are within the expected ranges [57%], and 5. Software routines that check for missing road seg- ments or data elements [55%].

higher average level of education, with 43% having graduate degrees (MS/PhD). It can be noted that the sample of pavement data collection service providers that responded to the survey was much smaller than the number of highway agencies. Only one service provider requires certification for the pavement evaluation staff. Personnel Certification Practices Only a small percentage of highway agencies (15%) currently require “certification” of the pavement distress raters for the agency, service provider, or both. However, most agencies indicated that they only use experienced personnel to rate pavements and that they undergo extensive training before collecting data. The accreditation workshops developed by LTPP provide a good example of a certification practice. These workshops were intended for experienced technicians who had completed high school, had experience with data collection, had received formal training on the Distress Identification Manual, and had experience in assisting an accredited rater in distress data collection or data interpretation for both asphalt and concrete pavement. The accreditation includes a written test and a two-part film-interpretation examination. The results are compared with reference values provided by experienced raters. To remain accredited, it is important that a rater regularly perform a minimum number of interpretations per year (67, 68). Ponniah et al. (69) present another example of a rater certification program. The paper presents the results of one 32 of the workshops organized by the Ontario Ministry of Trans- portation every two years to certify pavement raters. As part of the workshop, the raters from five regions evaluated a cir- cuit of nine sections and the results were statistically analyzed. Reference values for each section were obtained by a panel of four experts. The analysis compared the accuracy and precision of the raters, established province-wide, within-region, and between-region variability, investigated the effect of reducing the number of severity and density levels, and identified the distress types particularly hard to evaluate. The investigation showed that there were significant differences among raters, but no regional bias. The study also found that reducing the number of severity and density levels would help to reduce the variability in the pavement condition index. Equipment and Method Calibration, Certification, and Verification The verification that the equipment is functioning according to expectations and that the collection and analysis methods are being followed is key for ensuring the quality of the collected data. This is typically done before the initiation of the data collection activities and periodically after that. Equipment or process verification and validation is typically assessed by determining their repeatability and reproducibility (16). Repeatability is the variation in measurements taken by a single piece of equipment on the same road segment(s) and under the same conditions over a short period of time. It is generally evaluated based on the standard deviation of repeated values from different measurements. Repeatability measures DURING PRO DUCTIO N B EFO RE PRO DUCTIO N A FTER PRO DUCTIO N FIGURE 15 Example of quality control plan.

33 the ability of the equipment/technology/raters to produce the same values on repeated measurements. Reproducibil- ity is the ability of a technology or equipment to accurately reproduce or replicate measurements. The reproducibility of pavement condition measurements is typically measured as the standard deviation of measurements taken with different equipment or using different technologies. It is a measure of how well two different devices/methods/raters are able to measure the same pavement condition value on the same road segment(s). Equipment Calibration Equipment calibration is critical for the collection of accurate pavement data, especially for sensor-based measurements. Calibration is a systematic process to validate the data collec- tion methodology and/or equipment by comparing the mea- surements with a standard reference value or ground truth that is considered correct. Adjustment to the equipment or tech- nique may be required to match the “correct” measurement. The LTPP pioneered the development of detailed and well-documented calibration procedures for pavement con- dition measurement equipment. For example, the LTPP defined a clear distress rating methodology and established rigor- ous calibration procedures for the high-quality photographic images used for distress identification and quantification (67). Equipment calibration and harmonization of the measurements is the subject of several current national and international research efforts. The basic principles are discussed later in this section. Additional sources of information about equip- ment calibration and verification can be found in the litera- ture references provided. Distress For pavement distress, calibration is usually done by evaluating control sites where the pavement condition is closely monitored by a group of experts (70). These experts determine the condition of the control site, usually through careful evaluation and consensus ratings before equipment calibration, or in the case of manual data collection, personnel training. The expert ratings are considered the reference ratings of the control site. Statistical confidence intervals are often calculated to determine the requirements for equipment and personnel criteria calibration (18). For example, a 95% con- fidence interval with respect to the reference rating has been used for evaluating distress data collection equipment (71). Calibration of rutting measurements is typically conducted at the profiler calibration centers, such as the one at the Texas Transportation Institute (72). The Iowa DOT uses eight control sites (four asphalt con- crete and four PCC sections) for the initial verification of the data collection equipment and methodology. The service provider tests these sections before starting the production data collection. The sections have a variety of distress conditions and serve as a sample of the state and local roads in the state. The reference distress measurements are determined by expe- rienced staff and the DOT equipment is used to collect ride and rutting information. The service provider measures the site three times and the data are compared with the benchmark data collected by the Iowa DOT. The final data delivery require- ments are set based on this comparison. The control sites are also measured monthly by the service provider during produc- tion or whenever there is a change in equipment or subsystems on that same equipment (73). Smoothness Proper calibration of smoothness equipment requires an accurate and repeatable reference measurement. Early approaches to determine this reference value included the use of rod and level to determine the actual road profile. Another common method for determining the reference value is to perform a continuous, close-looped test in accordance with ASTM E950 using a static inclinometer. Slow-speed profile measuring devices, often called “walking” profilers, have also been developed. Standard methods for evaluating profiler accu- racy are provided in ASTM E950 Measuring the Longitudinal Profile of Traveled Surfaces with an Accelerometer Established Inertial Profiling Reference and AASHTO PP49 (74). These methods are generally considered appropriate for network- level pavement data collection (see Table 2); however, recent research has suggested that the method used in ASTM E950 does not ensure that two calibrated profilers can measure the same value of IRI within an acceptable tolerance for construction quality control (a project-level function) (75). Class Longitudinal Sample (LS) (mm) Vertical Measurement Resolution (VR) (mm) Precision Requirement (SD, mm) Bias Requirement (mm) Class 1 LS 25 VR 0.1 0.38 1.25 Class 2 25 < LS 150 0.1 < VR 0.2 0.76 2.50 Class 3 150 < LS 300 0.2 < VR 0.5 2.5 6.25 Class 4 LS > 300 VR > 0.5 — — Source: ASTM E950, Measuring the Longitudinal Profile of Traveled Surfaces with an Accelerometer Established Inertial Profiling Reference. TABLE 2 TOLERANCES FOR THE VARIOUS TYPES OF PROFILERS ACCORDING TO ASTM E950

Additional information on calibration of longitudinal and transverse profiles can be obtained from the Transportation Pooled Fund Project 5(063), Improving the Quality of Profiler Measurement (55); University of Michigan Road Roughness Home Page (76), PIARC evenness harmonization studies (56), and the Road Profiler User Group meetings and equipment comparison studies (77). The aforementioned pooled fund (55) objectives are to assist states with the implementation of AASHTO provisional standards; establish a level of integrity to the measurements; deliver sample procurement specifica- tions, maintenance guidelines, and profile analysis software; establish criteria for verification centers and assist with the development of these centers; develop and deploy a traceable verification process; and provide technical review of related software. This research program is currently working on select- ing equipment and technologies for measuring the reference profile (ground truth) and devising a practical approach for pro- filer calibration and certification, which probably will involve the establishment of regional certification centers. Structural Capacity For structural capacity (FWD) equip- ment, this phase typically includes sending the equipment for calibration and/or certification to a regional calibration center. These centers are operated by the equipment manufacturers and/or independent agencies; typically, university research and engineering centers. Regional FWD calibration centers were first established in four states (Colorado, Texas, Minnesota, and Pennsylvania) for supporting the LTPP data collection. The hardware and software in these centers have recently been updated through a pooled-fund study (78, 79). A new protocol for testing the load cell and deflection sensors has been pre- pared; this protocol determines gain factors or dynamic cali- bration factors that are entered into the FWD software as multipliers (79). It is important that the mass and drop height (load) levels produce loads within ±10% of three pre-selected target loads and that the sensor gain factors agree within a standard deviation of 0.003. The protocol also includes a pro- cedure for conducting field-based relative calibration using a stand provided by the FWD manufacturer. The Transportation Pooled Fund Project 5(039) FWD Cali- bration Center and Operational Improvements (78,79) and the LTPP Manual for Falling Weight Deflectometer Measure- ments (80) provide additional information on FWD calibration. Friction Properties Friction measuring equipment is also sent for calibration and/or certification at regional calibration centers. The regional calibration sites for the locked-wheel testers include the East Coast use of the Field Test and Eval- uation Center for Eastern States in East Liberty, Ohio (81) and the Central and Western Field Test Center in College Station, Texas (82). The principal features of the friction calibration centers include water calibration and evaluation, force mea- surement calibration, and dynamic correlation. Additional information on calibration and harmonization of friction properties measurements can be found in the PIARC 34 experiments to compare and harmonize texture and skid resistance measurements (83), the Transportation Pooled Fund Project 5(141) Pavement Surface Properties Consortium (84), and the AASHTO Guide for Pavement Friction (57). Equipment and Method Certification There is an increasing trend toward establishing formal certi- fication procedures for the pavement condition data collection equipment and methods. These procedures typically require the use of a “certification center” that verifies the correct functioning of the various components of the equipment and the training and skills of the operators and provides an official certificate of compliance with a specific standard. Certification typically implies that the equipment and/or operator have successfully passed formal verification testing. The granted certificate attests that the measurements meet some minimum accuracy and precision requirements. Examples include the Texas DOT inertial profiler operator and equipment certification program (72) and the LTPP dis- tress rater accreditation program (67,68). In the Texas DOT procedure, the profiler operators are required to pass written and practical tests to be certified to receive an operator iden- tification card that specifies the type or brand of inertial pro- filer they are certified to operate. The equipment certification includes the collection of profile data on two test sections in accordance with Test Method Tex-1001-S (85) in 3 hours. The results are evaluated for repeatability and accuracy and a profiler must meet all of the requirements to pass certification and receive a decal that is placed on the profiler as evidence of certification. Equipment and Method Verification Verification tests are periodic checks that control the accuracy of pavement condition data collection by examining the data and/or comparing it with known reference measurements. It is important that the data collection process is verified both before and during the data collection process. This is necessary to ensure that the actual pavement condition measurements meet the quality requirements and are adequate for support- ing the decision processes that will be using the data. In addition, this process could verify that the measurements are consistent with the historical records to ensure year-to- year consistency. Data Verification Procedures During the data collection process, a variety of methods are available for ensuring the continued collection of satisfactory quality data. The purpose of verifying collected data in real time or near real time is to avoid production of large quantities of unsatisfactory data. This benefits both the data collection teams and pavement managers by allowing quick detection

35 and correction of errors and minimizing the delays and costs associated with poor data quality collection. For this reason, it is recommended that the data be verified frequently during production. For pavement distress data collection, the verifi- cation of the distress ratings can be done for individual distress quantities, individual distress indices, multiple distress indices, or overall condition indices. Quality management techniques that can be used to ver- ify the data collection process include periodic retesting of “control” pavement segments, oversampling, and reanalyz- ing or resurveying a sample of the sections measured by an independent evaluator. If the verification process identifies deficiencies, the equipment will be checked and possibly recalibrated if automated or semi-automated procedures are used, or the raters’ criteria will be normalized so that all rating work remains within the acceptable limits of variation. This may include additional rater training (45). All of the substan- dard data need to be reevaluated and the corrective actions be recorded and documented to support long-term quality improvement goals. Testing of Control Sections The testing of control pavement segments is used to determine: (1) the accuracy of the procedure if the results are compared with those obtained against reference measurements deter- mined using the best available practical technique for that particular pavement condition indicator; or (2) its repeata- bility and reproducibility if the results are compared with results obtained with the same equipment or method. The locations of these segments can be known or “blind” for data collection teams. Typically, the testing of known con- trol sections is used for quality control and testing of blind controls for quality acceptance. In both blind and known segment testing a second team of raters reevaluates a segment of pavement for comparison testing. If the data collection team’s ratings are outside the established reevaluation team’s confidence interval, the equipment and/or procedure is re- checked and the data collected since the last satisfactory evaluation is either closely examined for accuracy or rejected entirely (86). Oversampling Another method of data verification during data collection is oversampling. In this procedure the data collection team samples the same segment(s) multiple times. This is similar to verification of blind or known pavement segments, because data from the retest are used for comparison purposes (87). However, this method of verification is generally considered less rigorous than verification testing by another team of raters, because if the source of error is systematic it most likely will not be detected by retesting undertaken by the same data collection team and equipment. This method of quality man- agement is considered effective for assessing random errors because it is unlikely that the same random error will occur multiple times over the same pavement segment. If more than one evaluation team or type of equipment is available, cross- measurements by different teams or equipment could be used to overcome this limitation. Sampling and Independent Reanalyzing or Resurveying Another data verification method consists of reanalyzing or resurveying a sample of the sections measured by an indepen- dent evaluator. The analysis of the sampled data is typically different depending on the level of pavement management considered. Network-Level Data Checks These checks often include statistical testing of the differences between the mean values (of the parameter being evaluated) for the quality control or acceptance samples and the production surveys for the same sections. The analyses typically include paired t-tests to assess the potential bias of the collected data and provides an indication of whether the pavement condition is consistently under- or overestimated as a result of the automated data collection process. The differences between individual mea- surements from the verification sample and the production field survey are computed for each sampling unit and the mean difference is tested against the null value using a pre-selected level of confidence (typically 95%). A two-sided t-test is used to determine if there is a significant difference. If there is a significant difference, then a one-sided t-test can be used to estimate how far off the mean difference was by evaluating the achievable tolerance levels (12). Project-Level Data Checks If the collected data are to be used for making project-level decisions, the mean comparison may not be applicable because some individual differences can exceed the acceptable range at the project level owing to limitations and the production data collection and processing technology (88). Statistical tests based on individual measure- ment rating may be more appropriate in these cases. These tests involve selecting a sample from a dataset, rating each individual observation within this sample using established pass–fail criteria for minimum acceptable quality, and con- cluding whether the whole dataset satisfies criteria for mini- mum acceptable quality based on the number of “failed” observations in the sample (88). Comparison of Manual and Automated Distress Surveys There have been several studies to verify AASHTO provisional protocol PP-44-01, which offers interesting approaches for comparing the results of different pavement condition col- lection methods; for example, manual versus automated surveys. These analyses provide good examples of tools and

methods that can be used to compare the production and con- trol measurements. Groeger et al. (89) used the cumulative cracking length to compare two automated cracking detection procedures for a network, including approximately 2,000 data points. The results of the automated process were then compared with the average of three experienced evaluators that classified the section using a five-level condition scale (very good, good, fair, poor, and very poor). The data were evaluated as a function of the percentage of points that fall within one, two, and three deviations in the five-level scale. For example, if a pavement was classified as poor by one method and very poor by another, the deviation is one. The study found that the automated pro- cedure produced good results for longitudinal and transverse cracks, with 94% of the data falling within one deviation of the visual assessment. Raman et al. (90) used statistical analysis to compare the severity and extent of the transverse crack by various proce- dures. The researchers used analysis of variance in the cases where data were normally distributed and nonparametric test (Kruskar–Wallis) in the remaining cases. Statistical compar- ison of sample and full-section image data showed that a 5% sampling rate was enough to evaluate transverse cracks with the precision desired for network-level pavement management in Kansas. Wang et al. (91) compared the use of an automated cracking survey system with manual evaluations in Arkansas using the provisional AASHTO protocol. The evaluators reviewed and analyzed 5% of the images for each comparison section. The study found some differences between the manual and auto- mated process, especially for Level 1 and 3 cracks; however, it also suggested that these discrepancies may be the result of the low repeatability of the manual surveys. The Ontario Ministry of Transportation compared auto- mated and semi-automated pavement distress collection tech- niques from three service providers with in-house manual surveys (92). The study included sections with surface-treated, hot-mix asphalt, composite, and PCC pavement structures. An overall pavement condition index, the distress manifestation index, was used for the comparisons. The investigation con- cluded that, in general, automated results are comparable with manual surveys. However, the authors emphasized the need of supplementing the automated collection with manual surveys, especially for project-level analysis, because some of the pave- ment distresses were difficult to identify with the automated methods. Determination of Sample Size One important element of the quality control process is the determination of how big a sample must be to have an accept- 36 able degree of confidence that the sample is representative of the entire process and sufficient to verify the required accuracy in the measurements. For pavement data collection, the per- centage of data that is checked in the quality control process typically ranges between 2% and 10%. However, it may be noted that the sample size is also dependent on the scope of the quality control task in hand. For example, computer-based checks can be applied to all the data, whereas cross-testing of control sections in general is often limited to a small sample of the network. The pavement data collection service providers surveyed indicated that they typically review 2% to 5% of the data (29%), 6% to 10% (29%), or more than 10% (42%) as part of their regular quality control practices. The selection of the number of segments to verify for qual- ity control (and/or quality acceptance) purposes is often set at a “rational” number based on previous experience. How- ever, there are a series of statistical techniques that allow the calculation of the required sample size based on the desired accuracy and degree of risk that the agency is willing to take. Procedures similar to the one developed by the National Parks Service (88) and that are discussed in the quality acceptance section can be used for determining the most appropriate qual- ity control sample. Software Data Checks Many agencies use software routines that check the data for inconsistencies for both quality control and acceptance, although these checks are slightly more prevalent for qual- ity acceptance than for quality control. There is some vari- ation in verification methods used for quality control: 55% of the agencies surveyed perform checks for detecting miss- ing segments or data elements, 57% check for ratings that are out of expected ranges, and 38% use statistical analysis to check for data inconsistencies. The checks may include on-vehicle data checks, data and video checks when the data are received in the office, condition rating data checks, and/or final database checks after it has been entered into the relevant pavement/asset management databases. On-vehicle data checks are conducted in real time as the data are being collected and/or periodi- cally (e.g., at the end of the day). Real-time checks typically include visual displays of certain data that alert crews if anything is malfunctioning and/or data that are out of range. Periodic diagnostics/data checks are typically scheduled at fixed intervals during breaks of the data collection to verify the correct functioning of the equipment. These diagnosis checks are important to avoid collecting large amount of deficient data. Final database checks are conducted to verify that data have been formatted properly and all the different data have been entered in the final database. These later checks include tests for completeness and format, time-history com- parisons, plots on GIS, etc.

37 The use of software to check for data inconsistencies can provide a noticeable improvement with respect to the accuracy of the data, identify areas for data collection improvements, and standardize data formats. These improvements might allow for better data analysis and time-history updates. QUALITY ACCEPTANCE Quality acceptance activities include all procedures used for acceptance testing of both the pavement condition data that are collected by the agency and those that are collected by a service provider. These tests validate that the data meet the established requirements before they are used to support pave- ment management decisions. Quality management techniques commonly used for this purpose include testing of control and verification sites, sampling and re-rating, complete data- base checks, GIS-based quality acceptance checks, and time- history comparisons. Quality Acceptance Plan Figure 16 summarizes the percentage of agencies that indicated that they have a formal pavement condition data quality acceptance plan. Two additional agencies indicated that they are working on developing such a plan. Quality acceptance processes typically require that a sam- ple or all of the data are checked to determine if some of the data may need to be corrected or resurveyed. Although listed by some of the agencies as quality acceptance activities, it is the opinion of the authors that the first two (calibration and testing of control sections before data collection) are more correctly classified as quality control activities. The qual- ity acceptance procedure, however, could verify that these procedures were conducted as specified and that the required tolerances were met. In addition, the procedures typically include testing of known or blind control sections, automatic check on all the data, detailed checks on a sample of the col- lected data comparisons with data from previous data collec- tion campaigns. In the case of data collection contracts, quality acceptance is often also linked to payments. The most common methods/tools used for quality accep- tance are the following (in order of decreasing fre- quency; the percentage of agencies citing each method/ tool is provided within brackets): 1. Calibration of equipment and/or analysis criteria before the data collection [80%], 2. Testing of known control segments before data collection [73%], 3. Periodic testing of known control segments during production [71%], 4. Software routines that check if the data are within the expected ranges [71%], 5. Software routines that check for missing road seg- ments or data elements [61%], 6. Statistical/software routines that check for incon- sistencies in the data [50%], and 7. Comparison with existing time-series data [50%]. 48.2% 37.5% 7.1% 7.1% 0% 20% 40% 60% Yes No Not Sure No Response 8.9% 7.1% 32.1% 0% 20% 40% 60% Other Prepared by Independent third party Developed by agency FIGURE 16 Percentage of highway agencies having a formal quality acceptance plan. The percentages of the data that are typically reviewed by the agencies are shown in Figure 17. It may be noted that the survey did not differentiate between general checks for com- pleteness and reasonableness and detailed checks for accu- racy, repeatability, and reproducibility. An example of a detailed quality acceptance plan is pre- sented elsewhere (93). The New Mexico DOT checks the quality of the pavement condition data collected by a service providers (universities) for consistency, completeness, and reasonableness. The agency checks that all values fall within acceptable data ranges and that the distress types and severities are reasonable. The agency also randomly selects sites and conducts data checks on both blind and known locations. These checks include comparing results with previous years’ data to identify locations where large changes occurred. If there had been areas where large changes occurred, the data have to be checked for reasonableness and consistency (93). Similar procedures are followed by agencies that collect data in-house. MDSHA uses quality acceptance checks to val-

idate that the quality control process was conducted properly. The quality acceptance is done by a quality assurance auditor, who is not the operator. This quality auditor checks the data management spreadsheet to verify that the data are complete, verifies that all data have been saved and backed-up, and re-checks a random sample of 10% of the data collected (66). Control and Verification Site Testing Approximately three-fourths of the agencies use known or blind control sites as part of the quality acceptance procedure. Blind sites are often used for distress rating comparisons. Although the terms control and verification sites are often used interchangeably, there is a practical difference: • Control sites are those in which the reference measure- ments have been determined and can be used to deter- mine both accuracy and repeatability of data. The con- 38 trol sites are measured using the reference procedure; for example, manual visual survey for distress data. • Verification sites are used to determine continued repeatability and/or reproducibility. They are measured periodically by the same equipment/crew or by differ- ent devices/crews. The first case is typically called over- sampling and allows determining the repeatability. The second is referred as cross-testing and determines the reproducibility. The percentage of data collected that has to be corrected/ resurveyed as a result of deficiencies identified by the quality acceptance process is very similar for in-house and contracted data collection. Approximately two-thirds of the respondents (64% for in-house and 67% for contracted) indicated that their staff or the service providers need to correct less than 2% of the data collected. Most of the others (30% for in-house and 33% for contracted) reported having to correct between 2% and 5% of the data. Establishing Acceptance Criteria A key aspect of the quality acceptance procedure is the definition of what constitutes “acceptable” data. Agencies need to define the criteria to determine how much variation is allowable between the reference value (or ground truth) and actual data measurement. The criteria are usually different for the various pavement condition indicators (e.g., smoothness and distress data), and include limits for accuracy and repeata- bility. It is important that the criteria reflect the actual capa- bilities of the available technology and that the service provider and agency agree on the acceptance criteria. For example, the tolerance for IRI might be ±5%. Table 3 presents, as an example, the criteria originally defined for the Pennsylvania DOT for quality acceptance. Additional examples of tolerances used by DOTs are compared in Table 4. < 2% 9.8% 2 to 5% 24.4% 6 to 10% 17.1% > 10% 34.1% None 14.6% Question: If you have a pavement data collection quality assurance plan, what percentage of the data collected do you typically review in this plan? FIGURE 17 Percentage of data reviewed for quality assurance. Reported Value Initial Criteria Percent Within Limits (PWL) Recommended Action if Criteria Not Met IRI ±25% 95% Reject deliverable Individual Distress Severity Combination ±30% 90% Feedback on potential bias or drift in ratings. Retrain on definitions. Total Fatigue Cracking ±20% 90% Reject deliverable Total Non-Fatigue Cracking ±20% 90% Reject deliverable Total Joint Spalling ±20% 90% Reject deliverable Transverse Cracking, Jointed Plain Concrete ±20% 90% Reject deliverable Location Reference— Segment/Offset Correct Segment All Return deliverable for correction Location Reference— Segment Begin ±10 ft 95% Return deliverable for correction and systems check Panoramic Images Legible signs 80% Report problem. Reject subsequent deliverables. Source: Ganesan et al. (94). TABLE 3 INITIAL PAVEMENT CONDITION ACCEPTANCE CRITERIA FOR THE PENNSYLVANIA DOT

39 Sampling for Quality Acceptance Testing Another important aspect is the determination of how large of a sample is needed for determining and verifying the accuracy and repeatability of the data collection procedure. For exam- ple, for automated or semi-automated distress data collection it is common to extract a sample of the collected pictures and review the ratings for accuracy. A 5% sample is common for this purpose; however, statistical approaches can be used to determine the required sample size. Larger samples may be required for research-quality data; for example, the LTPP distress data collection protocol requires that 10% of each lot is checked for distress mismatches, questionable severity levels, or errors in the test section or survey date. Selezneva et al. (88) presents a series of promising sampling approaches that have been evaluated for quality assurance of the data collected for the National Park Service PMS. The data are surveyed by a service provider that collects pavement and right-of-way images, rutting, smoothness, road horizontal and vertical alignment, and GPS coordinates. Surface-cracking data are determined from the images using an automated crack detection system that detects the type, severity, and amount of cracking within a 0.01-mile section. The data are then aggre- gated in indexes for individual surface distresses, a compos- ite distress index [the surface condition rating (SCR) and an overall pavement condition rating (PCR)]. The quality control and acceptance checks required for field data collection are documented in the Road Inventory Program—National Park Service Quality Assurance Manual (95). The checks include equipment checks, diagnostics of data collection and processing hardware and software, and a verification survey of a sample of selected parks by a review panel. Collected distress data are subject to a two-step data quality control and acceptance process in which the service provider first applies a series of internal quality control checks and the FHWA then conducts quality acceptance checks. The quality acceptance checks verify that the collected data are rated in accordance with the approved methodology for dis- tress identification and distress severity ratings by manually rating selected pavement distress images provided by the ser- vice provider’s automated crack detection system. The size of the sample requiring these later checks is determined using the following procedures. Sample Size Determination for Assessing Network-Level Accuracy This section discusses available tools to determine how large a sample is needed to verify that the accuracy of the measure- ments is within a specified range (e.g., ±10%) with a certain degree of confidence. Statistical testing of the mean differences is typically conducted using two- and one-sided t-tests. The size of the sample required for conducting meaningful comparisons is a function of the following statistical parameters: 1. Test significance (alpha) used as threshold for statistical significance (e.g., an alpha of 0.05 is typically selected to achieve a 95% level of confidence); 2. Desired precision or maximum acceptable difference; 3. Variability in the population, determined as the stan- dard deviations of the computed differences (unknown at the time of analysis); and 4. Test power, or probability of correctly rejecting the null hypothesis (no difference between surveys) when it is false, which is typically computed by using a parame- ter beta defined as a probability of not detecting a dif- ference when the difference exists (e.g., to achieve test power of 90%, beta would be limited to 10%). The sample size is selected by balancing accuracy and cost. Although a larger sample allows for identifying a finer mean difference as statistically significant, the cost and effort of obtaining the sample and processing and analyzing the data may offset the benefit of the added precision of the results. A pilot application of the methodology for one park showed that a tolerance level of five SCR points would require 103 to 112 sample sections, which was considered relatively high because of the high cost for the field survey. If the tolerance level for the difference in mean SCR values between manual and automated surveys is relaxed to 10 SCR points, it would require 26 to 28 sample units, which was considered more reasonable (88). This type of precision of ±10% points of a 100 point index is also used by other agencies. Sample Size Determination for Conformity Testing To compare a sample of individual measurements against the reference and determine what percentage of the data collected Virginia British Columbia Condition Indicator Range Criteria Accuracy Repeatability Smoothness (IRI) 10% of Class I 0.1 m/km Rut Depth ±3 mm ±3 mm Pavement Condition Index (surface distress) ±10 95% ±1 PDI [scale 1 to 10] ±1 SD of PDI for five runs PDI = Pavement Distress Index; SD = standard deviation. TABLE 4 EXAMPLE OF TOLERANCE FOR VARIOUS PAVEMENT INDICATORS

fail to meet the quality acceptance criteria (e.g., ±20%), Selezneva et al. (88) recommend the use of the lot acceptance sampling plan methodology. The decisions about acceptance of production data are based on counting the number of un- acceptable quality observations in a random sample of obser- vations from a set. To determine the required sample size for a known targeted maximum number of unacceptable observations in the sample, the analyst defines the following statistical parameters: 1. Acceptable quality level or the percentage of automat- ically collected data points that are expected to differ from the reference value assessment as a result of lim- itations of production technology; 2. Lot tolerance percent defective (LTPD) or percentage of automatically collected data points differing from the reference value assessment of the same data points that would make a dataset unacceptable by the specifications; 3. Type I Error (service provider’s risk) or probability of rejecting a dataset that has a defect level equal to the acceptable quality level; and 4. Type II Error (client’s risk) or probability of accepting a dataset with a defect level equal to the lot tolerance percent defective. For large datasets, where the sample size is less than 10% of the total number of observations, n0, that must pass the acceptance criteria without a single unacceptable observation can be computed using Eq. (1), and the number n that must pass the criteria with no more than one unacceptable obser- vation using Eq. (2): where R = reliability of the data production procedures expressed as a fraction; the percentage of the observations not passing the pass–fail criteria because of limitations of the production methodology is equal to 100(1 − R); and C = confidence level expressed as a fraction (for a 95% level of confidence, C = 0.95). Other Statistical Analyses The Cohen’s weighted Kappa Statistic has also been proposed as a measure of agreement between raters, which evaluates the probability of agreement beyond chance. This method allows for the use of weights between disagreements, so that less important disagreements have less of an effect than more important disagreements and more weight can be given to R n R R Cn n( ) + −( )( ) = −−1 1 21 ( ) n C R0 1 1= −( ) ( ) ln ln ( ) 40 those distresses that have the most effect on PMS decisions. For example, inconsistencies among distress severities are weighted less than disagreements among distress identification. This type of analysis has been shown to be more effective in identifying data inconsistencies than traditional methods, but assignment of weights and benchmarks needs to be carefully done (86). Complete Database Checks Typical quality acceptance procedures also include automatic checks on all the collected data to determine (1) if the data are within the expected ranges, (2) if there are any missing road segments or data elements, and/or (3) conduct simple statis- tical analysis and/or check to find possible inconsistencies in the data. This can be done as the data are being submitted (e.g., weekly) or after the entire product has been submitted. It is recommended that at least some of these checks be con- ducted frequently to identify possible issues as soon as pos- sible and avoid collecting large quantities of bad data. These checks are similar to the ones discussed for quality control but are conducted as a second check by the owner agency in the case of contracted data collection or by an inde- pendent quality acceptance auditor (internal or external) in the case of in-house data collection. Examples include checks to verify essential “general” information included in the con- dition database, sensor checks to flag out-of-range values for different indicators, and distress checks to verify that the dis- tresses identified match the surface type (e.g., fatigue cracking on asphalt pavements). An example of a well-developed set of software checks has been presented by Wolters et al. (52). This publication describes an application developed for Oklahoma DOT for checking the pavement data quality. The program conducts four types of checks: preliminary, sensor, distress, and spe- cial. All of the data checks can be organized into reports to identify inconsistencies and areas where data collection pro- tocol may need to be modified (52). The Iowa DOT quality acceptance process includes checks to verify that the measurements are between the expected minimum and maximum values, and to identify segments with missing roughness, rutting, or distress data, and sections in which all distress values are zero on continuous segments. The quality acceptance procedures also compare pavement condi- tion index values from year to year. To improve communi- cations with the service providers and reduce the amount of rejected data, some of these steps have also been incorporated into the service provider’s quality control process (73). Use of Geographic Information Systems in Pavement Condition Data Quality Management One particular technique that is gaining acceptance is the use of GIS-based checks for quality acceptance. The visualization

41 and spatial analysis tools available in GIS can be very useful for detecting missing sections, inconsistencies in the location of some sections, and unexpected changes in pavement con- dition. ODOT, for example, has recently begun using GIS for complementing the agency’s quality acceptance procedures. Zhang and Smadi (73) present another example of the use of GIS to support the data collection quality management. Time-History Comparisons Approximately half of the responding agencies conduct com- parisons of the pavement data collection with existing time- series data to identify unexpected changes in the condition that may be an indication of data collection problems. Larson et al. (96) presents some interesting approaches for comparing time-history pavement condition data. MDSHA also conducts a comparison of the current percentage of pavement sections in acceptable condition with those obtained over the past 5 years to highlight potential data collection problems; this approach is discussed in detail in chapter five. The Florida DOT requires that the Ride Rating (100-point scale) be within plus or minus eight points of the previous year’s survey. It is important that the data collection crew rerun the section if a rating falls outside this range. The second run must not vary by more than plus or minus one ride rating point from the first run. If the second measurement differs by more than plus or minus one rating point, then additional runs are required (97). The LTPP distress data quality control protocols require that the rater analyze the images, compare them with the closest available survey (before or after) in a side-by-side plot, and resolve any differences before the distress maps are shipped to the quality acceptance contractor. The contractor checks 10% of each lot, and the data undergoes a higher-order quality acceptance, which includes time-series comparison and information management checks. The time-series checks plot the distress versus time with a 3-standard deviation error band (computed based on an average coefficient of variation obtained from variability studies), preservation treatments, and linear trend lines. A software tool, called Distress Viewer and Analysis has been developed by LTPP to assist in this process. The comparisons allow for identifying missed main- tenance treatments or errors in the distresses identified (67). Quality Acceptance of Contracted Data Collection This section covers some quality assurance issues that are applicable only to agencies that have outsourced the data collection services. An important point is the use of the qual- ity control data as part of the quality assurance process. For example, an agency may chose to use the service provider quality control data on the control segments for quality assur- ance purposes, and only validate a fraction of these data dur- ing the quality acceptance checks. According to the survey (Figure 18), only approximately one-third (30%) of the agen- cies that contract the data collection services use the service provider’s quality control results as part of the quality assur- ance process. A significant number responded that they were not using these data (43%) or were not sure (27%). More than two-thirds (69%) of the agencies that have out- sourced at least part of the data collection indicated that it was a positive step, with only one (3%) responding that it was not (Figure 19). The remaining agencies (28%) were not sure. Furthermore, 89% of the agencies are satisfied with the perfor- mance of the data collection service provider(s) most recently used or currently being used. However, this satisfaction is not universal; two agencies responded that they were not satisfied with the performance of their service providers. INDEPENDENT VERIFICATION It is surprising that only a very small number of agencies use third-party verification of data as a quality management tool; 4% for quality control and 12% for quality assurance. In the case of the Virginia DOT (VDOT), the use of an independent consultant to review the data led to a substantial reduction Yes 30% No 43% Not Sure 27% Question: Do you use the contractor quality control data as part of the quality assurance process? FIGURE 18 Percentage of respondents that use service provider quality control data for quality acceptance. Yes 69% No 3% Not Sure 28% Question: Overall, would you consider the outsourcing of pavement data collection a positive step in your pavement management practices? FIGURE 19 Degree of satisfaction with the outsourcing of pavement data collection.

in the number of pavement segments identified as requiring treatment and a large reduction in the estimated budgetary requirements (51). Although this is only a single case, it indi- cates that third-party verification of data may be a cost-effective quality management tool. The techniques and approaches used for the verification are typically very similar to those applied for the quality acceptance. A sample (e.g., 5% to 10%) of the data is typically subjected to the independent verification. DATA REJECTION In general, the amount of data that has to be corrected because of errors detected during the quality assurance process is rela- tively small. More than half of the agencies (52%) reported that less than 2% of the data need to be corrected or resubmitted by the service provider, and 39% reported having to correct 2% to 5% and only 8% (two agencies) reported having to reevaluate or correct 6% or more. This result is consistent with the responses indicated that, in general, outsourcing of the data collection with appropriate quality acceptance procedures has been beneficial for pavement management practices. Although there does not appear to be a clear connection between network size and amount of rejected data, states with larger networks generally reported a data rejection rate of less than 2%, whereas agencies with smaller networks showed more variation. Nearly all agencies reported a data rejection rate of less than 5%. Agencies with a formalized quality man- agement plan appear to reject less data. This is expected because a formalized quality management plan would clarify data acceptance procedures to all parties and data collec- tion teams would most likely follow the quality control pro- cedures and would not submit data that would not meet the known standards. SUMMARY This chapter covered the main quality control, quality accep- tance, and independent assurance principles and techniques currently being followed by transportation agencies for pave- ment condition data collection. As discussed in the previous 42 chapter, the distinction between quality control and accep- tance activities depends on how the activities are incorpo- rated into the management plan, rather than the activities themselves. The purpose of the quality control plan is to quantify the variability in the process, maintain it within acceptable limits, identify the source of variability that can be controlled, and take the necessary production adjustments to minimize the “controllable” variability. A large percentage of the respon- dents (64%) have a formal data collection quality control plan or require the service provider to develop such a plan. A comprehensive quality control plan typically includes clear delineation of the responsibilities, documented manuals and procedures, personnel training, equipment and/or process cal- ibration, certification and inspection, verification procedures before starting and during production testing (e.g., using con- trol sites), and checks for data reasonableness, consistency, and completeness. Quality acceptance activities include all procedures used for acceptance testing of both the pavement condition data that are collected by the agency and those that are collected by a service provider. These tests validate that the data meet the established requirements before they are used to sup- port pavement management decisions. Approximately half of the agencies that responded to this question have a formal pavement condition data quality acceptance plan. Quality management techniques commonly used for this purpose include testing of control and verification sites, sampling and re-rating, complete database checks, GIS-based quality acceptance checks, and time-history comparisons. Impor- tant aspects for the testing of control and verification sites include establishing the acceptance criteria and the size of the sample required. In some cases, agencies also use an independent verifica- tion by a third party to resurvey or reevaluate a sample of the data. The techniques and approaches used for the independent verification are typically similar to those applied for the quality acceptance.

Next: Chapter Five - Case Studies »
Quality Management of Pavement Condition Data Collection Get This Book
×
MyNAP members save 10% online.
Login or Register to save!
Download Free PDF

TRB’s National Cooperative Highway Research Program (NCHRP) Synthesis 401: Quality Management of Pavement Condition Data Collection explores the quality management practices being employed by public highway agencies for automated, semi-automated, and manual pavement data collection and delivery.

  1. ×

    Welcome to OpenBook!

    You're looking at OpenBook, NAP.edu's online reading room since 1999. Based on feedback from you, our users, we've made some improvements that make it easier than ever to read thousands of publications on our website.

    Do you want to take a quick tour of the OpenBook's features?

    No Thanks Take a Tour »
  2. ×

    Show this book's table of contents, where you can jump to any chapter by name.

    « Back Next »
  3. ×

    ...or use these buttons to go back to the previous chapter or skip to the next one.

    « Back Next »
  4. ×

    Jump up to the previous page or down to the next one. Also, you can type in a page number and press Enter to go directly to that page in the book.

    « Back Next »
  5. ×

    To search the entire text of this book, type in your search term here and press Enter.

    « Back Next »
  6. ×

    Share a link to this book page on your preferred social network or via email.

    « Back Next »
  7. ×

    View our suggested citation for this chapter.

    « Back Next »
  8. ×

    Ready to take your reading offline? Click here to buy this book in print or download it as a free PDF, if available.

    « Back Next »
Stay Connected!