Cover Image

Not for Sale



View/Hide Left Panel
Click for next page ( 55


The National Academies | 500 Fifth St. N.W. | Washington, D.C. 20001
Copyright © National Academy of Sciences. All rights reserved.
Terms of Use and Privacy Statement



Below are the first 10 and last 10 pages of uncorrected machine-read text (when available) of this chapter, followed by the top 30 algorithmically extracted key phrases from the chapter as a whole.
Intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text on the opening pages of each chapter. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.

Do not use for reproduction, copying, pasting, or reading; exclusively for search engines.

OCR for page 54
Chapter 3: Measurement The third set of customers consists of those impacted by the operation of roads. Prominent among this group are adjacent property owners who can be impacted by what economists call externalities--costs or benefits experienced by others than the producers or consumers of a product. Pollution or changes in property value because of the use and activities occurring on the road or in the right-of-way are examples of externalities. Property owners adjacent to roads and who experience externalities are among those who pay for the roads, particularly in cities and counties where property taxes are a major source of road funding. Economic value to maintenance customers can be conveniently grouped into the following types: Avoided user costs, Avoidable life-cycle costs, and Avoided external costs. Customers are willing to pay to avoid user costs, life-cycle costs, and external costs; hence, the willingness to pay is also an important measure of economic value. Appendix D includes a discussion of how to calculate life-cycle costs, user costs, and willingness to pay. COMMONLY RECOGNIZED MEASURES A prerequisite for benchmarking of any type, including customer-driven benchmarking, is that benchmarking participants agree on the measures that will be used. This is true regardless of whether all the benchmarking participants are within your organization or whether you benchmark with other organizations. Therefore, one of the early tasks in benchmarking is to begin to establish a foundation for agreed-upon measures. There are several ways to tackle this prerequisite. First, if you are planning to do benchmarking only within your organization you can establish your own agreed-upon measures. Second, if you are benchmarking with other organizations, you can begin the process of identifying your partners, establishing what you plan to benchmark, and gaining agreement on the measures you will 56

OCR for page 54
use. Third, whether you are doing internal or external benchmarking, you can determine whether there is a pre-existing set of commonly recognized customer-driven measures for benchmarking maintenance activities. Importance and Adopted Measures The issue of widely agreed-upon measures for benchmarking and other purposes is of such importance that the AASHTO Subcommittee on Maintenance and FHWA sponsored the National Workshop on Commonly Recognized Measures for Maintenance in June 2000 in Scottsdale, Arizona. At the workshop, states agreed to an initial set of "commonly recognized measures" that reflect the outcomes and satisfaction that customers experience from the delivery of maintenance products and services. Table 1 summarizes the measures that were adopted by the states at the workshop. In only a few cases was a recommended measure fully defined. In most instances, the workshop participants adopted a type of measure with the expectation that the definition, units of measure, and other aspects of a measurement protocol would be established in the future. A view was expressed that it is not necessary to be overly specific in the workshop. It was sufficient for workshop participants to identify areas where there is general agreement that commonly recognized measures exist, particularly ones that relate directly or indirectly to the customer. The adopted commonly recognized measures exist side-by-side with other performance measures that many states have already developed and generally use for maintenance management and asset management. However, over time it is anticipated that an increasing number of states, cities, counties, turnpike authorities, and contractors will apply commonly recognized measures for an increasing number of purposes. The common measures are useful for customer-driven benchmarking, customer-driven asset management systems, performance-based contracting, and public reporting of maintenance performance. Commonly recognized measures create efficiencies in data collection, measurement systems, and management systems. 57

OCR for page 54
Chapter 3: Measurement Table 1. Commonly Recognized Measures Adopted by Consensus (continued on next page) 58

OCR for page 54
Table 1. (Continued) Key Issues in Adopting Agreed-Upon Measures When adopting benchmarking measures, there are a number of key issues to consider: Desirable attributes of the measurement scale; Types of measures to avoid; Selection of appropriate units; Segment length; Repeatability, reliability, and accuracy; and Protocols. 59

OCR for page 54
Chapter 3: Measurement Type of Measurement Scale Just as you would select an appropriate tool to pound a nail into wood, it is critical to select measures for benchmarking that have the appropriate attributes. The measures need to support objective, repeatable measurement--in some cases, with desirable precision and statistical confidence. To do so, the measures generally need to have a continuous measurement scale, be expressed in units with appropriate resolution, apply to standard lengths or parts of roadway geometry, and be taken under a standard and rigorous protocol. There may also be a need for acceptance testing of data using random sampling. Continuous Measures The Virginia DOT contracted for the It is strongly recommended that wherever possible you apply collection of roadway measures with a continuous scale. A continuous scale extends inventory feature and indefinitely from a starting point, and the units of condition data. The measurement can be divided into equal, arbitrarily small contract specified that the intervals. Examples of continuous scales are as follows: data must be accurate Extent of bridge deck distress measured in terms of within plus or minus 5 percentage of the deck area affected, percent, with 95 percent accuracy. The contractor Roughness measured according to the International had to agree to Roughness Index (IRI), acceptance testing to Shoulder edge drop-off measured in inches or centimeters ensure that the data was and arbitrarily small fractions thereof, of the accuracy and statistical confidence Retroreflectivity of signs measured as candelas per foot- specified in the contract. meter square foot, Mean response time to fix a problem, and Mean time between failures. By using a continuous scale, you remove the subjectivity and difficulty of having to define the meaning of scale intervals other than the units of measurement. The results of a measurement can be of any magnitude from very small to very large. Measurement systems that use continuous, well- established scales are more likely to be repeatable, and there is a basis for establishing the statistical quality of measures to any degree of accuracy and statistical confidence. 60

OCR for page 54
Discrete or Continuous Interval Scale If you cannot select a continuous scale, the next best type of scale is a discrete scale with constant intervals between steps, otherwise known as a continuous interval scale. Examples of such an interval scale are 1, 2, 3, 4, 5 and 1, 2, 3, 4, 5, 6, 7, 8, 9, 10. A discrete interval scale has most of the desirable attributes of a continuous scale. However, one must define what each step in the scale means, and this can be fraught with subjectivity and technical challenges. The following customer rating scale attempts to maintain equal distances between each step of the scale: 1 = Very dissatisfied; 2 = Somewhat dissatisfied; 3 = Satisfied; 4 = Somewhat more than satisfied; and 5 = Very satisfied. A similar type of scale might also be a letter grade--for example, "A, B, C, D, and E." This type of scale has the same strengths and weaknesses of the continuous interval scale if it is equivalent to "1, 2, 3, 4, 5" or some other similar equally spaced discrete scale. Occasionally, a letter scale has a leap in it--for example, A, B, C, D, and F. Usually this type of scale implies that measurement will occur in constant steps up to a point, and thereafter the only measurement of concern is failure. You are likely to encounter a measurement system that involves probabilistic condition states. The measurement scale is likely to be a discrete scale such as 1, 2, 3, 4, and 5. Probabilistic condition states are used to identify the probability that a maintainable element, such as bridge deck or pavement surface, will deteriorate from one condition state to another. The distances between steps on the scale are not necessarily even, but are defined by alternative actions that may be considered for maintaining a maintenance element in a particular condition state. 61

OCR for page 54
Chapter 3: Measurement Binary Measures Binary measures take on just two values such as "0 or 1" or "yes or no." The project team recommends avoiding binary scales because they have much less resolution than do a continuous or continuous interval scale. Establishing the definition of each value is likely to be much more subjective. In some cases, using a binary scale is the only logical choice. Examples are whether a traffic signal is working, a sign is up or knocked down, or a drainage structure is blocked. Types of Measures to Avoid Do not choose targets, objectives, or goals for benchmarking measures. Frequently people confuse these points on the measurement scale with the measurement scale itself. A target, goal, or objective may have such importance (for example, a performance target agreed upon by a Chief Administrative Officer and the legislature) that managers may think of little else besides whether the target or goal is being met. The measurement process of benchmarking is not about targets, objectives, or goals; it is about measuring performance along some scale to discern best performers so that benchmarking partners can explore what work methods and business processes lie behind best performances and can adopt or improve upon best practices. You should also avoid choosing measures that represent thresholds for actions, such as minimum tolerable conditions (e.g., a warrant to replace a traffic signal). An important exception is a probabilistic condition state that has alternative actions associated with it. Selection of Appropriate Units Not only can a measurement scale be too coarse to differentiate performance, but choosing inappropriate units can have the same effect. You may decide to measure litter count per unit of elapsed distance along the roadside. If you select as your measure litter count per mile, you will get one result; if you select litter count per tenth of mile, you will get another; and if you select litter count per foot or inch, the quantity of litter you 62

OCR for page 54
observe may always be close to zero, which is not very useful for benchmarking. This example reveals how important it is to select units that will provide enough resolution to measure the performances of different organization units. Agreement on Road Segment Length and Geometric Measurements In reaching an agreement with your benchmarking partners regarding what measures to use, you may find it necessary to define an agreed-upon segment length or other standard measurement procedures pertaining to roadway geometrics so that everyone takes the same measurement in the same manner. Suppose you are measuring guardrail condition. Do you measure the percent of total guardrail damaged over a 1-mile distance, over a tenth of a mile, or over a kilometer? Suppose you are measuring the presence of a type of noxious weed. How will you define the area over which you will take measurements? Perhaps you might agree with your partners that a measure of roadside vegetation management will be sight distance at intersections. How will you define the procedure for measuring sight distance? Do you determine, for example, how many feet from the corner along one side of the intersection you can see a car at an equal distance along the other side of the intersection? In general, you will need to reach prior agreement on how to define segment length, area, and other geometric procedures for different types of measurements. Repeatability, Reliability, and Accuracy of Measurement Any measure selected for customer-driven benchmarking needs to be repeatable and reliable. Repeatable means that different people who apply the measure and take a measurement under the same circumstances obtain the same or nearly the same result. To obtain repeatability usually requires training. Each person who takes a measure requires instruction on how to do it. If equipment is involved such as a profilometer or a friction meter, it will need to be calibrated and recalibrated from time to time to ensure repeatability. 63

OCR for page 54
Chapter 3: Measurement The measure needs to be reliable. The equipment used to take the measure should not break down easily. Excessive measurement deviations should not occur due to normal changes in weather, normal wear and tear, or a switch in the personnel who are taking the measurement. The measure needs to be accurate, and it is desirable to specify its accuracy. In other words, if you or others take repeated measurements, you should get the same result within some range. This range is often referred to as the "accuracy" or "precision" of the measure and is expressed as "plus or minus" some percentage deviation from the mean score (e.g., plus or minus 5 percent). The accuracy or precision is a random variable; the measurements will occur within the accuracy with some statistical confidence level--for example, 95 percent of the time. Indeed you should specify what accuracy and statistical confidence you expect of your measurements. Measurement Protocols In general, it is a good idea to develop formal protocols for measurement. Protocols exist for taking many different kinds of measurement--for example, the IRI and rutting. A good protocol should set the purpose, scope, measurement, data recording procedure, and quality assurance and should document references. An outline of a protocol for edge drop-off (taken from the proceedings of the National Workshop on Commonly Recognized Measures) might consist of the following: 1. Purpose. The edge drop-off protocol defines a standard method for estimating and summarizing edge drop-off. The purpose is to produce consistent estimates of edge drop-off. 2. Scope. Applies to estimating edge drop-off on any pavement surface, but does not provide specifications for equipment. Any equipment capable of taking the measurement is acceptable for the protocol. Safety issues in applying the protocol are the responsibility of the organization taking the measurement. 3. Measurement. Each agency should designate the lane(s), shoulders, and direction(s) of travel to be surveyed based on sound engineering principles and management needs. Edge drop-off is an elevation difference between the 64

OCR for page 54
paved travel lane and shoulder, between a paved shoulder and an unpaved shoulder, or both. Measurements are made longitudinally at maximum intervals of 15 meters (50 feet). 4. Data Recording. Data collection sections should be of a constant length within some prescribed range as determined by the agency. Sample intervals within each data collection section should be of uniform length. Minimum sample section lengths are 30 meters (100 feet). There are five edge drop-off condition levels defined as a function of the length of the edge drop-off and the elevation difference (for example, the edge drop-off condition levels used by the Texas DOT). The minimum data recorded should consist of section identification, the length of the data collection section, the date of collection, and the rating for the section. 5. Quality Assurance. Each agency should develop a quality assurance plan that addresses personnel certification training, accuracy of equipment (including calibration), daily quality control procedures, and periodic and ongoing quality control. 6. Reference Documents. A list should be provided of references associated with the measurement protocol.3 Data Availability, Quality, and Costs Some types of measures depend on making a calculation--for example, a measure of response to customer demand for control of ice and snow is the ratio of the time it takes to restore pavement to bare condition from the onset of a snowstorm relative to the duration of the snowstorm. To calculate this ratio, you need to track how long it takes from the start of each storm to the point in time when snow removal crews have removed all the snow from the roadway. You also need to calculate how long the snowstorm is. Neither of these numbers is trivial to determine. You have to define when a storm starts. Does this mean the storm begins when precipitation starts, or when snow starts to stick to the roads? Do you measure where snow sticks to the road in one standard place or along every section of road and then take an average of the time the snow starts to stick? Similar 3 Proceedings, National Workshop on Commonly Recognized Measures. 65