Click for next page ( 31


The National Academies | 500 Fifth St. N.W. | Washington, D.C. 20001
Copyright © National Academy of Sciences. All rights reserved.
Terms of Use and Privacy Statement



Below are the first 10 and last 10 pages of uncorrected machine-read text (when available) of this chapter, followed by the top 30 algorithmically extracted key phrases from the chapter as a whole.
Intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text on the opening pages of each chapter. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.

Do not use for reproduction, copying, pasting, or reading; exclusively for search engines.

OCR for page 30
30 CHAPTER FOUR QUALITY MANAGEMENT PRACTICES This chapter focuses on the specific quality management source of variability that can be controlled, and take the neces- principles and techniques currently being followed by trans- sary production adjustments to minimize the "controllable" portation agencies for pavement condition data collection. It variability. It is also important that the quality control process discusses the various approaches and tools used for quality detect problems soon, before large quantities of data have control, quality acceptance, and independent assurance. to be re-collected. The sources of variability for the various pavement condition indicators were discussed in detail in The creation and implementation of a comprehensive data chapter three. collection quality management program is a very important step to ensure that quality data are collected. However, as reported in the previous sections, a significant percentage Contents of a Quality Control Plan of highway agencies (approximately half ) still do not have such procedures in place. With respect to how these plans Based on the examples reviewed, it becomes apparent that a are developed, the survey showed that most agencies pre- comprehensive quality control plan typically includes the pare their own quality acceptance plans, whereas the quality following elements: control plans are developed either by the agency, the data collection service provider, or as a collaborative effort. Only Clear delineation of the responsibilities; one agency reported having used a third-party contractor for Documented (and available) manuals and procedures; developing the pavement data collection quality manage- Training of survey personnel; ment plan. Equipment calibration, certification, and inspection procedures; The following sections discuss the main techniques and Equipment and/or process quality verification procedures procedures used for quality control, acceptance, and inde- (e.g., testing of control sections) before starting and pendent assurance. As discussed in chapter three, the dis- during production testing; and tinction between quality control and acceptance activities Checks for data reasonableness, consistency, and com- depends on how the activities are incorporated into the man- pleteness. agement plan, rather than the activities themselves. Many of the tools used for quality control and quality assurance are the same. To avoid duplication, this chapter discussed The survey revealed that a large percentage of the respon- the various activities and tools organized in the most common dents (64%) have a formal data collection quality control plan configuration. or require the data collection service providers to develop such a plan (Figure 14). It is noted that some of the agencies that checked the "other" option indicated they have procedures QUALITY CONTROL for only some of the pavement condition indicators or that such a plan was under development. Common quality control activities include personnel training and certification, equipment calibration, certification and ver- Approximately half of the data collection quality control ification, production quality data verification, and on-vehicle plans were developed by the service provider collecting the and office data checks. These are covered in detail in the data. Although it was not asked specifically, it is hypothesized following sections after a brief discussion on pavement con- that this corresponds to all or most of the services being dition data variability and quality control planning for control- contracted. It is also interesting to note that a shorter survey ling this variability. sent to pavement data collection service providers showed that all the service providers have a formal data collection quality control plan; however, none of them provided a Quality Control and Variability copy and several indicated that the plan was project-specific or proprietary. This last result is an indication of the impor- The purpose of quality control is to quantify the variability in tance data collection companies place on the quality control the process, maintain it within acceptable limits, identify the procedures.

OCR for page 30
31 Prepared by data No Response 10.7% 32.1% collection contractor Not Sure 5.4% Developed by Agency 23.2% Prepared by independent No 19.6% 1.8% third party Yes 64.3% Other 7.1% 0% 20% 40% 60% 80% 0% 20% 40% 60% FIGURE 14 Percentage of highway agencies having a formal quality control plan. (1) review of completeness, (2) review of section-level data, From the available quality management tools and meth- and (3) review of data management. The first step simply ver- ods, the most common methods/tools used for quality ifies that all fields are processed. The section-level review is control are the following (in order of decreasing fre- conducted by examining approximately 50% of the sections. quency, the percentage of agencies citing each method/ The operator also verifies that the data have been saved and tool is provided within brackets): inputs a subjective evaluation of the crack detection process (good, fair, and poor). Further details on this approach are 1. Calibration of equipment and/or analysis criteria provided in a case study presented in chapter five. before the data collection [94%], 2. Testing of known control segments before data collection [94%], 3. Periodic testing of known control segments during Personnel Training and Certification production [81%], 4. Software routines that check if the data are within It is very important that the personnel operating the equipment the expected ranges [57%], and or conducting the visual surveys are continuously trained. 5. Software routines that check for missing road seg- This is particularly critical given the current environment in ments or data elements [55%]. the transportation profession, where the work force is highly mobile, and technicians and engineers change positions and employers quite frequently. Training is even more impor- tant for the distress surveys because the classification of the distresses is somehow subjective. All the data collection service providers responded that they use calibration of equipment and/or analysis criteria before the data collection, and software routines that check if the data Personnel Training are within the expected ranges for missing road segments or data elements, and for inconsistencies in the data. At least Personnel training is necessary for obtaining repeatable and half of the service providers also indicated using the other reproducible pavement condition data. It affects manual, semi- techniques. automated, and automated practices. Adequate training helps improve the consistency and accuracy of visual surveys and An illustrative example of a pavement data collection proper operation of the equipment using automated or semi- quality control plan is presented in Figure 15. The figure automated procedures. presents examples of the activities that are typically conducted before, during, and after the production process, rather than According to the survey of practice, pavement evaluation a comprehensive list of all available tools and processes. personnel are trained mostly through on-the-job training from experienced staff and in-house training programs. The For example, the Maryland State Highway Administration pavement condition data collection is primarily composed of (MDSHA) is one of the agencies that reported having a formal experienced technicians (69% have more than 6 years and 26% quality control process for its in-house pavement data col- more than 10 years of experience). They hold associate degrees lection activities. The agency developed a detailed quality (44%) or high-school diplomas (39%), with only a small per- management program when transitioning from windshield centage having bachelor's and graduate degrees. The pavement distress data collection to an automated system to measure data collection staff for the pavement data collection service smoothness, rutting, and cracking (66). In the case of the providers appears to be a little less experienced (57% have automated crack detection, the quality control plan includes more than 6 years and 14% more than 10 years) and have a

OCR for page 30
32 PRODUCTION BEFORE PRODUCTION DURING PRODUCTION AFTER FIGURE 15 Example of quality control plan. higher average level of education, with 43% having graduate of the workshops organized by the Ontario Ministry of Trans- degrees (MS/PhD). It can be noted that the sample of pavement portation every two years to certify pavement raters. As part data collection service providers that responded to the survey of the workshop, the raters from five regions evaluated a cir- was much smaller than the number of highway agencies. Only cuit of nine sections and the results were statistically analyzed. one service provider requires certification for the pavement Reference values for each section were obtained by a panel of evaluation staff. four experts. The analysis compared the accuracy and precision of the raters, established province-wide, within-region, and between-region variability, investigated the effect of reducing Personnel Certification Practices the number of severity and density levels, and identified the Only a small percentage of highway agencies (15%) currently distress types particularly hard to evaluate. The investigation require "certification" of the pavement distress raters for the showed that there were significant differences among raters, agency, service provider, or both. However, most agencies but no regional bias. The study also found that reducing the indicated that they only use experienced personnel to rate number of severity and density levels would help to reduce pavements and that they undergo extensive training before the variability in the pavement condition index. collecting data. The accreditation workshops developed by LTPP provide a good example of a certification practice. These Equipment and Method Calibration, workshops were intended for experienced technicians who had Certification, and Verification completed high school, had experience with data collection, had received formal training on the Distress Identification The verification that the equipment is functioning according Manual, and had experience in assisting an accredited rater to expectations and that the collection and analysis methods are in distress data collection or data interpretation for both being followed is key for ensuring the quality of the collected asphalt and concrete pavement. The accreditation includes a data. This is typically done before the initiation of the data written test and a two-part film-interpretation examination. collection activities and periodically after that. Equipment The results are compared with reference values provided by or process verification and validation is typically assessed experienced raters. To remain accredited, it is important that by determining their repeatability and reproducibility (16). a rater regularly perform a minimum number of interpretations Repeatability is the variation in measurements taken by a per year (67, 68). single piece of equipment on the same road segment(s) and under the same conditions over a short period of time. It is Ponniah et al. (69) present another example of a rater generally evaluated based on the standard deviation of repeated certification program. The paper presents the results of one values from different measurements. Repeatability measures

OCR for page 30
33 the ability of the equipment/technology/raters to produce calculated to determine the requirements for equipment and the same values on repeated measurements. Reproducibil- personnel criteria calibration (18). For example, a 95% con- ity is the ability of a technology or equipment to accurately fidence interval with respect to the reference rating has been reproduce or replicate measurements. The reproducibility of used for evaluating distress data collection equipment (71). pavement condition measurements is typically measured as Calibration of rutting measurements is typically conducted the standard deviation of measurements taken with different at the profiler calibration centers, such as the one at the equipment or using different technologies. It is a measure Texas Transportation Institute (72). of how well two different devices/methods/raters are able to measure the same pavement condition value on the same The Iowa DOT uses eight control sites (four asphalt con- road segment(s). crete and four PCC sections) for the initial verification of the data collection equipment and methodology. The service provider tests these sections before starting the production data Equipment Calibration collection. The sections have a variety of distress conditions Equipment calibration is critical for the collection of accurate and serve as a sample of the state and local roads in the state. pavement data, especially for sensor-based measurements. The reference distress measurements are determined by expe- Calibration is a systematic process to validate the data collec- rienced staff and the DOT equipment is used to collect ride and tion methodology and/or equipment by comparing the mea- rutting information. The service provider measures the site surements with a standard reference value or ground truth three times and the data are compared with the benchmark data that is considered correct. Adjustment to the equipment or tech- collected by the Iowa DOT. The final data delivery require- nique may be required to match the "correct" measurement. ments are set based on this comparison. The control sites are also measured monthly by the service provider during produc- The LTPP pioneered the development of detailed and tion or whenever there is a change in equipment or subsystems well-documented calibration procedures for pavement con- on that same equipment (73). dition measurement equipment. For example, the LTPP defined a clear distress rating methodology and established rigor- Smoothness Proper calibration of smoothness equipment ous calibration procedures for the high-quality photographic requires an accurate and repeatable reference measurement. images used for distress identification and quantification (67). Early approaches to determine this reference value included Equipment calibration and harmonization of the measurements the use of rod and level to determine the actual road profile. is the subject of several current national and international Another common method for determining the reference value research efforts. The basic principles are discussed later in is to perform a continuous, close-looped test in accordance with this section. Additional sources of information about equip- ASTM E950 using a static inclinometer. Slow-speed profile ment calibration and verification can be found in the litera- measuring devices, often called "walking" profilers, have also ture references provided. been developed. Standard methods for evaluating profiler accu- racy are provided in ASTM E950 Measuring the Longitudinal Distress For pavement distress, calibration is usually done Profile of Traveled Surfaces with an Accelerometer Established by evaluating control sites where the pavement condition is Inertial Profiling Reference and AASHTO PP49 (74). These closely monitored by a group of experts (70). These experts methods are generally considered appropriate for network- determine the condition of the control site, usually through level pavement data collection (see Table 2); however, recent careful evaluation and consensus ratings before equipment research has suggested that the method used in ASTM E950 calibration, or in the case of manual data collection, personnel does not ensure that two calibrated profilers can measure training. The expert ratings are considered the reference ratings the same value of IRI within an acceptable tolerance for of the control site. Statistical confidence intervals are often construction quality control (a project-level function) (75). TABLE 2 TOLERANCES FOR THE VARIOUS TYPES OF PROFILERS ACCORDING TO ASTM E950 Longitudinal Vertical Precision Bias Class Sample (LS) Measurement Requirement Requirement (mm) Resolution (VR) (SD, mm) (mm) (mm) Class 1 LS 25 VR 0.1 0.38 1.25 Class 2 25 < LS 150 0.1 < VR 0.2 0.76 2.50 Class 3 150 < LS 300 0.2 < VR 0.5 2.5 6.25 Class 4 LS > 300 VR > 0.5 -- -- Source: ASTM E950, Measuring the Longitudinal Profile of Traveled Surfaces with an Accelerometer Established Inertial Profiling Reference.

OCR for page 30
34 Additional information on calibration of longitudinal and experiments to compare and harmonize texture and skid transverse profiles can be obtained from the Transportation resistance measurements (83), the Transportation Pooled Fund Pooled Fund Project 5(063), Improving the Quality of Profiler Project 5(141) Pavement Surface Properties Consortium (84), Measurement (55); University of Michigan Road Roughness and the AASHTO Guide for Pavement Friction (57). Home Page (76), PIARC evenness harmonization studies (56), and the Road Profiler User Group meetings and equipment comparison studies (77). The aforementioned pooled fund Equipment and Method Certification (55) objectives are to assist states with the implementation of AASHTO provisional standards; establish a level of integrity There is an increasing trend toward establishing formal certi- to the measurements; deliver sample procurement specifica- fication procedures for the pavement condition data collection tions, maintenance guidelines, and profile analysis software; equipment and methods. These procedures typically require establish criteria for verification centers and assist with the the use of a "certification center" that verifies the correct development of these centers; develop and deploy a traceable functioning of the various components of the equipment and verification process; and provide technical review of related the training and skills of the operators and provides an official software. This research program is currently working on select- certificate of compliance with a specific standard. Certification ing equipment and technologies for measuring the reference typically implies that the equipment and/or operator have profile (ground truth) and devising a practical approach for pro- successfully passed formal verification testing. The granted filer calibration and certification, which probably will involve certificate attests that the measurements meet some minimum the establishment of regional certification centers. accuracy and precision requirements. Structural Capacity For structural capacity (FWD) equip- Examples include the Texas DOT inertial profiler operator ment, this phase typically includes sending the equipment for and equipment certification program (72) and the LTPP dis- calibration and/or certification to a regional calibration center. tress rater accreditation program (67,68). In the Texas DOT These centers are operated by the equipment manufacturers procedure, the profiler operators are required to pass written and/or independent agencies; typically, university research and and practical tests to be certified to receive an operator iden- engineering centers. Regional FWD calibration centers were tification card that specifies the type or brand of inertial pro- first established in four states (Colorado, Texas, Minnesota, filer they are certified to operate. The equipment certification and Pennsylvania) for supporting the LTPP data collection. includes the collection of profile data on two test sections The hardware and software in these centers have recently been in accordance with Test Method Tex-1001-S (85) in 3 hours. updated through a pooled-fund study (78, 79). A new protocol The results are evaluated for repeatability and accuracy and for testing the load cell and deflection sensors has been pre- a profiler must meet all of the requirements to pass certification pared; this protocol determines gain factors or dynamic cali- and receive a decal that is placed on the profiler as evidence bration factors that are entered into the FWD software as of certification. multipliers (79). It is important that the mass and drop height (load) levels produce loads within 10% of three pre-selected Equipment and Method Verification target loads and that the sensor gain factors agree within a standard deviation of 0.003. The protocol also includes a pro- Verification tests are periodic checks that control the accuracy cedure for conducting field-based relative calibration using a of pavement condition data collection by examining the data stand provided by the FWD manufacturer. and/or comparing it with known reference measurements. It is important that the data collection process is verified both The Transportation Pooled Fund Project 5(039) FWD Cali- before and during the data collection process. This is necessary bration Center and Operational Improvements (78,79) and to ensure that the actual pavement condition measurements the LTPP Manual for Falling Weight Deflectometer Measure- meet the quality requirements and are adequate for support- ments (80) provide additional information on FWD calibration. ing the decision processes that will be using the data. In addition, this process could verify that the measurements Friction Properties Friction measuring equipment is also are consistent with the historical records to ensure year-to- sent for calibration and/or certification at regional calibration year consistency. centers. The regional calibration sites for the locked-wheel testers include the East Coast use of the Field Test and Eval- uation Center for Eastern States in East Liberty, Ohio (81) and Data Verification Procedures the Central and Western Field Test Center in College Station, Texas (82). The principal features of the friction calibration During the data collection process, a variety of methods are centers include water calibration and evaluation, force mea- available for ensuring the continued collection of satisfactory surement calibration, and dynamic correlation. quality data. The purpose of verifying collected data in real time or near real time is to avoid production of large quantities Additional information on calibration and harmonization of unsatisfactory data. This benefits both the data collection of friction properties measurements can be found in the PIARC teams and pavement managers by allowing quick detection

OCR for page 30
35 and correction of errors and minimizing the delays and costs because it is unlikely that the same random error will occur associated with poor data quality collection. For this reason, multiple times over the same pavement segment. If more than it is recommended that the data be verified frequently during one evaluation team or type of equipment is available, cross- production. For pavement distress data collection, the verifi- measurements by different teams or equipment could be used cation of the distress ratings can be done for individual distress to overcome this limitation. quantities, individual distress indices, multiple distress indices, or overall condition indices. Sampling and Independent Quality management techniques that can be used to ver- Reanalyzing or Resurveying ify the data collection process include periodic retesting of "control" pavement segments, oversampling, and reanalyz- Another data verification method consists of reanalyzing or ing or resurveying a sample of the sections measured by an resurveying a sample of the sections measured by an indepen- independent evaluator. If the verification process identifies dent evaluator. The analysis of the sampled data is typically deficiencies, the equipment will be checked and possibly different depending on the level of pavement management recalibrated if automated or semi-automated procedures are considered. used, or the raters' criteria will be normalized so that all rating work remains within the acceptable limits of variation. This Network-Level Data Checks These checks often include may include additional rater training (45). All of the substan- statistical testing of the differences between the mean values dard data need to be reevaluated and the corrective actions (of the parameter being evaluated) for the quality control be recorded and documented to support long-term quality or acceptance samples and the production surveys for the improvement goals. same sections. The analyses typically include paired t-tests to assess the potential bias of the collected data and provides an indication of whether the pavement condition is consistently Testing of Control Sections under- or overestimated as a result of the automated data collection process. The differences between individual mea- The testing of control pavement segments is used to determine: surements from the verification sample and the production field (1) the accuracy of the procedure if the results are compared survey are computed for each sampling unit and the mean with those obtained against reference measurements deter- difference is tested against the null value using a pre-selected mined using the best available practical technique for that level of confidence (typically 95%). A two-sided t-test is used particular pavement condition indicator; or (2) its repeata- to determine if there is a significant difference. If there is a bility and reproducibility if the results are compared with results obtained with the same equipment or method. The significant difference, then a one-sided t-test can be used to locations of these segments can be known or "blind" for estimate how far off the mean difference was by evaluating data collection teams. Typically, the testing of known con- the achievable tolerance levels (12). trol sections is used for quality control and testing of blind controls for quality acceptance. In both blind and known Project-Level Data Checks If the collected data are to be segment testing a second team of raters reevaluates a segment used for making project-level decisions, the mean comparison of pavement for comparison testing. If the data collection may not be applicable because some individual differences team's ratings are outside the established reevaluation team's can exceed the acceptable range at the project level owing to confidence interval, the equipment and/or procedure is re- limitations and the production data collection and processing checked and the data collected since the last satisfactory technology (88). Statistical tests based on individual measure- evaluation is either closely examined for accuracy or rejected ment rating may be more appropriate in these cases. These entirely (86). tests involve selecting a sample from a dataset, rating each individual observation within this sample using established passfail criteria for minimum acceptable quality, and con- Oversampling cluding whether the whole dataset satisfies criteria for mini- mum acceptable quality based on the number of "failed" Another method of data verification during data collection observations in the sample (88). is oversampling. In this procedure the data collection team samples the same segment(s) multiple times. This is similar to verification of blind or known pavement segments, because Comparison of Manual and Automated data from the retest are used for comparison purposes (87). Distress Surveys However, this method of verification is generally considered less rigorous than verification testing by another team of raters, There have been several studies to verify AASHTO provisional because if the source of error is systematic it most likely will protocol PP-44-01, which offers interesting approaches for not be detected by retesting undertaken by the same data comparing the results of different pavement condition col- collection team and equipment. This method of quality man- lection methods; for example, manual versus automated agement is considered effective for assessing random errors surveys. These analyses provide good examples of tools and

OCR for page 30
36 methods that can be used to compare the production and con- able degree of confidence that the sample is representative of trol measurements. the entire process and sufficient to verify the required accuracy in the measurements. For pavement data collection, the per- Groeger et al. (89) used the cumulative cracking length to centage of data that is checked in the quality control process compare two automated cracking detection procedures for typically ranges between 2% and 10%. However, it may be a network, including approximately 2,000 data points. The noted that the sample size is also dependent on the scope of results of the automated process were then compared with the quality control task in hand. For example, computer-based the average of three experienced evaluators that classified the checks can be applied to all the data, whereas cross-testing of section using a five-level condition scale (very good, good, fair, control sections in general is often limited to a small sample of poor, and very poor). The data were evaluated as a function the network. The pavement data collection service providers of the percentage of points that fall within one, two, and three surveyed indicated that they typically review 2% to 5% of the deviations in the five-level scale. For example, if a pavement data (29%), 6% to 10% (29%), or more than 10% (42%) as was classified as poor by one method and very poor by another, part of their regular quality control practices. the deviation is one. The study found that the automated pro- cedure produced good results for longitudinal and transverse The selection of the number of segments to verify for qual- cracks, with 94% of the data falling within one deviation of ity control (and/or quality acceptance) purposes is often set the visual assessment. at a "rational" number based on previous experience. How- ever, there are a series of statistical techniques that allow the Raman et al. (90) used statistical analysis to compare the calculation of the required sample size based on the desired severity and extent of the transverse crack by various proce- accuracy and degree of risk that the agency is willing to take. dures. The researchers used analysis of variance in the cases Procedures similar to the one developed by the National Parks where data were normally distributed and nonparametric test Service (88) and that are discussed in the quality acceptance (KruskarWallis) in the remaining cases. Statistical compar- section can be used for determining the most appropriate qual- ison of sample and full-section image data showed that a 5% ity control sample. sampling rate was enough to evaluate transverse cracks with the precision desired for network-level pavement management Software Data Checks in Kansas. Many agencies use software routines that check the data Wang et al. (91) compared the use of an automated cracking for inconsistencies for both quality control and acceptance, survey system with manual evaluations in Arkansas using the although these checks are slightly more prevalent for qual- provisional AASHTO protocol. The evaluators reviewed and ity acceptance than for quality control. There is some vari- analyzed 5% of the images for each comparison section. The ation in verification methods used for quality control: 55% study found some differences between the manual and auto- of the agencies surveyed perform checks for detecting miss- mated process, especially for Level 1 and 3 cracks; however, ing segments or data elements, 57% check for ratings that are it also suggested that these discrepancies may be the result of out of expected ranges, and 38% use statistical analysis to the low repeatability of the manual surveys. check for data inconsistencies. The Ontario Ministry of Transportation compared auto- The checks may include on-vehicle data checks, data mated and semi-automated pavement distress collection tech- and video checks when the data are received in the office, niques from three service providers with in-house manual condition rating data checks, and/or final database checks surveys (92). The study included sections with surface-treated, after it has been entered into the relevant pavement/asset hot-mix asphalt, composite, and PCC pavement structures. An management databases. On-vehicle data checks are conducted overall pavement condition index, the distress manifestation in real time as the data are being collected and/or periodi- index, was used for the comparisons. The investigation con- cally (e.g., at the end of the day). Real-time checks typically cluded that, in general, automated results are comparable with include visual displays of certain data that alert crews if manual surveys. However, the authors emphasized the need of anything is malfunctioning and/or data that are out of range. supplementing the automated collection with manual surveys, Periodic diagnostics/data checks are typically scheduled at especially for project-level analysis, because some of the pave- fixed intervals during breaks of the data collection to verify ment distresses were difficult to identify with the automated the correct functioning of the equipment. These diagnosis methods. checks are important to avoid collecting large amount of deficient data. Final database checks are conducted to verify Determination of Sample Size that data have been formatted properly and all the different data have been entered in the final database. These later checks One important element of the quality control process is the include tests for completeness and format, time-history com- determination of how big a sample must be to have an accept- parisons, plots on GIS, etc.