Below is the uncorrected machine-read text of this chapter, intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text of each book. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.
6 "valuating Inspection and Maintenance Programs: Methods for Estimating Emissions Reductions Inspection and maintenance (~/M) program evaluation helps address whether the investment of human and capital resources required for I/M programs is beneficial. Although a judgment about whether I/M is beneficial is beyond the scope of this report, the committee charge does include a call for identifying evaluation criteria end methods. We will describe a set of criteria and discuss methods of evaluation using these criteria in this chapter and the next. Ideally, the primary criterion should tee the effect ofanI/M program on air quality and the associated effects on human health and welfare. However, it is very difficult to separate the relatively modest emissions impacts of I/M programs from other policies designed to reduce emissions and from other anthropogenic and natural changes that influence air quality. Because ofthis difficulty, attempts to assess the impacts of I/M programs on ambient air emis- sions have been difficulteven for carbon monoxide (CO), a pollutant that is generated almost solely by light-duty vehicles (I DVs) (Scherrer and Kittelson 1994; ENVIRON 1998). Thus, the criteria and methods of evaluation discussed in this chapter focus on the reduction in emissions brought about by I/M programs. Difficulties in estimating emissions reductions arise because vehicle emissions are variable over time and with driving method, emission tests themselves are variable and imperfect, and perhaps most important, the behavior of motorists, technicians, 146
Methods for Estimating Emissions Reductions 147 and even state enforcement authorities play a key role in the emissions reduc- tions achieved. This chapter focuses on defining the emissions reductions to be measured, describes some ofthe obstacles to measuring them, and outlines possible methods for measuring them. Other discussions on emissions reduc- tions are found in Chapter 3, which describes prior studies that have estimated emissions-reduction benefits from I/M programs, and in Chapter 5, which describes how emissions reductions are estimated in MOB TEE. A full evaluation of I/M requires that the criteria be defined more broadly thanjust the reduction in emissions. At a minimum, additional criteria include cost and cost-effectiveness of program designs, enforcement requirements, and such factors as public acceptance and political feasibility. These additional criteria are discussed in Chapter 7. METHODS FOR MEASURING EMISSIONS REDUCTIONS There are several inherent difficulties in evaluating the emissions reduc- tions from an I/M program. One is defining the baseline, the condition against which the I/M program is compared. Attempting to discern the benefits by comparing an area with an I/M program with a reference area (either an area with a reference I/M program or a non-I/M fleet) is confounded by differences between the area and its reference in climate, socioeconomic conditions, and other characteristics. Additionally, vehicle technologies are also continuing to improve, so the emissions benefits of a program depend on when they are being measured. Sorting out vehicle repairs or scrappage that occurs because of an I/M program versus what would occur even in its absence is also diffi- cult. Finally, there are numerous statistical issues associated with evaluating I/M, some of which are summarized in Appendix C. An I/M program has the potential to reduce emissions in a number of ways. Motorists might be persuaded to better maintain their vehicles as a result ofthe program. Emissions might tee reduced as a result of repairs made in anticipation of an I/M inspection (referred to as pre-inspection repairs) or as a result of failing the inspection test. Finally, some vehicles may be scrapped or sold outside the I/M area because, given the age or condition of the vehicle, the owner did not think the repair was worth the cost. We summarize these sources of emissions reduction resulting from an I/M program in Table 6-] . It is important to contrast these sources of emissions reductions with the
148 Evaluating Vehicle Emissions I/M Programs TABLE 6-l Sources of Emissions Reduction from I/M Type of Emissions Reduction 1. Improved maintenance,which leads to lower emissions Data Requirements 2. Repairs to emissions equipment made before an emissions test in anticipation of the test Roadside tests or remote sensing 3. (a) Repair of a vehicle's emissions In-program test data, comprehensive remote- systems as a result of failing a test sensing data, or roadside-pullover data (b) Length of time repairs last for a In-program test data, change-of-ownership vehicle repaired as a result of fail- test data, comprehensive remote-sensing ing an I/M test data, or roadside-pullover data 4. Early scrapping or transfer of high- In-program test data together with remote- emitting vehicles outside the I/M sensing data or vehicle-registration data region (fleet effects) MOBILE modeling approach to I/M described in Chapter 5, which attributes most emissions benefits to the instantaneous repair of failed vehicles. We emphasize et the outact that the components of emissions reductions arising from an I/M program, as described in Table 6-l, are very difficult to estimate. Human behavior and lack of complete evidence confound the esti- mation of emissions reductions at every turn. For example, emissions reduc- tions as measured by in-program data on individual vehicles might not repre- sent real emissions reductions on some vehicles but might result from partial repair or even retesting with no repair. Repair in anticipation of the I/M pro- gram might represent real and long-lasting emissions reduction from some vehicles but would not be accounted for if only I/M test observations were used for evaluation. Also important is the amount of emissions-reduction- related repairs that would be done without any I/M program. Because of these issues, the approaches for evaluation described in this chapter include data needs and methods that attempt to account for all the factors that influence emissions reductions technical, behavioral, and others. All the categories of emissions reductions listed in Table 6-1 must be evaluated to determine the effectiveness of a program. We first discuss sources of data for measuring emissions changes from I/M.
Methods for Estimating Emissions Reductions ~ 49 Data to Estimate I/M Emissions Reductions I/M programs have the potential to reduce tailpipe emissions of nitrogen oxides (NOX) andCO end both tailpipe end evaporative (includingliquidleaks) emissions of hydrocarbons (HC). Tailpipe emissions are easier to measure but depend on many factors related to the condition and operation ofthe vehicle; measurements can come from in-program (test results from centralized or decentralized I/M programs) or on-road (remote sensing, roadside pullovers) sources. Attributes of each are discussedbelow. Non-tailpipe HC emissions are very difficult and expensive to measure; they require special equipment, invasivetestmethods,andiongtesttimes. As aresult,although many tailpipe emissions data are available, there are no evaporative emissions measurements that directly measure I/M effectiveness to reduce all sources of non-tailpipe emissions. As discussed in Chapter I, evaporative emissions represent a significant but unquantified source of overall vehicle HC emissions. Data from I/M Programs Tailpipe data from I/M programs (in-program data) can come from the program itself or from separate tests run for the purpose of evaluating the program. Inspection lane data cover tailpipe, visual, and some functional tests, such as a test of the gas cap. Data can be from idle tests, steady-state loaded- mode tests (e.g., the acceleration simulation mode (ASM) test), or transient loaded-mode tests (e.g., the IM240~. Chapter 3 contains descriptions ofthese tests. Idle tests measure concentrations of CO and HC; steady-state loaded- mode tests measure concentrations of CO, HC, andNOx; andtransientIoaded- mode tests measure mass emissions of CO, HC, and NOX. Data can also be gathered from visual and functional tests. Using in-program data is appealing because this information can be collected at little or no extra cost. Because ofthe very large amount of data collected as part of ongoing I/M programs, detailed analysis can reveal information about the vehicle fleet and the I/M program. All tested vehicles are identified, and they can be followed Some examples of the separate in-program tests include running two consecutive tests on vehicles or, for a program that uses a fast-pass system, running full IM240 tests for the purpose of gathering unambiguous in-program data.
~ 50 Evaluating Vehicle Emissions I/M Programs from cycle to cycle, giving estimates of emissions deterioration rates and repair effectiveness. These data also can be used to develop vehicle profiles (dis- cussed in Chapter 4) and information about the performance of test stations. Several factors might contribute to underestimating actual emissions reduc- tions using in-program data. Several sources of emissions reduction listed in Table 6- ~ cannot be measured with in-program data, such as the effect of an I/M program on improving vehicle maintenance before the I/M test or on emissions reduction gained by causing vehicles to leave the area or be scrapped early. Measuring the impact of I/M programs on these parameters is made difficult by the inherent turnover of the fleet and maintenance that would occur in the absence of a program. Conversely, there are reasons why in-program data might overestimate actual emissions reductions. In-program data do not include emissions from vehicles that avoid testing. Avoidance can result in exaggerated emissions reductions in a number of ways. Owners might not bring their vehicles to be tested at all. That is a problem if it is assumed that all vehicles are tested and that failing vehicles receive repairs yielding some average emissions reduction. Owners might collude with technicians running the test program to falsify the emissions level of a failing vehicle so that it passes. Or owners might tempo- rariTy fix a vehicle to pass the test without fully repairing it. If vehicles are prepared for the test, they are not typical of vehicles on the road. in this case, again, in-program data overstates emissions reductions if it is assumed that all I/M repairs last some average length of time. Finally, because ofthe statistical problem referred to as "regression to the mean," emissions of a failing vehicle likely will be Tower on retesting, even in the absence of repairs.2 Since only vehicles that fail are retested, this group has higher then average emissions. Thus, even in the absence of repairs, their emissions would tend to move closer to the mean ofthe fleet upon retesting. Not accounting for this phenomenon would tend to overstate actual emissions reductions from in-program data. There are other more general problems with in-program data. The gold- standard tests to measure tailpipe emissions are the Federal Test Procedure Regression to the mean is a statistical phenomenon where the initial scores of a selected group within a normal distribution will tend to move toward the population mean in a follow-up test. Although the movement of an individual score cannot be predicted based on this phenomenon, the group average will likely move toward the population mean during follow-up tests.
Methods for Estimating Emissions Reductions 151 (FTP) and the supplemental FTP (SFTP) dynamometer tests. These tests are used to certify that emissions from new vehicles do not exceed federal emis- sions standards. However, these tests are far too costly end lime-consuming to measure a large sample of vehicles. Most I/M tests suffer from a lack of consistent preconditioning and an inability to represent all the driving modes represented in the FTP and SFTP.4 Environmental conditions also have been shown to affect emissions test results (Anderson and Wilkes 1998; EPA 2000b). Additionally, vehicle emissions vary from test to test, especially for many high-emitting vehicles (Knepper et al. ~ 993; Bishop and Stedman ~ 996; Coninx 2000~. Many factors can contribute to variability in repeated emissions tests of the same vehicle. These factors, which are also summarized in Wenze! et al. (2000), include the presence of intermittent failures of emissions- contro] system components (such as a malfunctioning oxygen sensor) or fluctu- ations in back-to-back emissions test results because of differences in the measurement equipment, calibrations, or test personnel (e.g., different driving styles in tracking a target speed-time trace on a dynamometer). Some test results, such as those for idle and ASM tests, also must be converted from concentration measurements to mass emission rates. The correlation ofthe idle test with the FTP is illustratedin Figure 6-l for the same car tested in the same laboratory (Haskew et al. ~ 987~. It includes 604 obser- vations on mode] year ~ 98 ~ and ~ 982 vehicles and is plotted on a logarithmic scale to reduce data scatter. Data from Roadside Testing Exhaust emissions test data can also be obtained from vehicles subject to roadside tests. Under such a program, vehicles are randomly purled over and given an emissions test similar to the vehicle inspection test. Visual and func- tional tests can also be done to determine tampering rates (see the discussion 3A typical estimate for an FTP test is $800 to $1,000 per test for a vehicle delivered to the laboratory. The additional charge for an SFTP is $750. 4The FTP and SFTP have extensive protocols regarding fuel specifications and environmental conditions. Their driving cycles include driving modes not included in transient loaded-mode I/M tests, including cold-start and high accelerations. They are also done under laboratory conditions, with better-calibrated instruments and better- trained technicians.
152 Evaluating Vehicle Emissions I/M Programs 604 Observations 60. . ~0 'IFTP }IC AV6- 0.77 ~7 ,.; $+ + ~ . A+ ~ A+ . . ~ 1 . . . ~ ~ Be ~0 100 it' I~ ~ ~ - ~ ~ 1000 FIGURE 6-1 EPA emissions-factor data from 1981-1982 industry closed-Loop cars. The correlation of the idle concentration test with the FTP test is illustrated in the plot. The horizontal scale is the idle test concentration measurement in parts per million (ppm); the vertical scale gives the FTP mass test result for the same vehicle tested in the same laboratory. Logarithmic scales are used to keep the data on the plot. Source: Haskew et al. 1987. Reprinted by permission from SAE paper 871 103; copyright 1987, Society of Automotive Engineers, Warrendale, PA. ofthe results from the U.S. Environmental Protection Agency (EPA) national tampering survey in Chapter 3), and to check components ofthe evaporative emissions-control system. Roadside testing is fairly expensive because it re- quires portable testing equipment, technicians to do the testing, and officers to pull over vehicles. California's roadside test program measured emissions from about ~ 0,000 vehicles per year from ~ 997 to the present. EPA' s national tam- pering surveys, conducted from the late ~ 970s through ~ 992, measured about 7,500 vehicles per year between lL985 and 1992. Because ofthe costs and time required to perform roadside testing and for political reasons, emissions on far fewer vehicles are measured in this manner than are measured in the inspection lanes. This implies that samples need to be carefully defined to avoid selection bias and to obtain a representative sample ofthe fleet. However, roadside testing represents an independent source of in-
Methods for Estimating Emissions Reductions 153 use evaluation data. Many of the shortcomings of in-program data, such as motorists avoiding testing or making repairs solely to pass the test, are not present in roadside pullover data. Roadside testing coupled with a vehicle's I/M history can demonstrate the effectiveness of an I/M program or changes to a program, the duration of repairs, the occurrence of pretest repairs, the level of compliance with the program, and whether there is a need to increase enforcement. Data from Remote Sensing Remote sensing is a third source of tailpipe emissions data (see Chapter 4 for a Ascription of remote sensing). Remote sensing allows a large number of in-use vehicles to be measured. As with roadside testing, remote sensing can help assess the effectiveness of an I/M program or changes to the pro- gram, the duration of repairs, the occurrence of pretest repairs, the level of compliance with the program, and whether there is a need to increase enforce- ment. Remote-sensing measures emissions for about one-half second of Unving for each vehicle. Care to select proper measurement sites is needed for remote-sensing evaluations of I/M programs so that cold-start and off- cycle emissions are avoided. Other issues discussed in Chapter 4 relate to quality control and quality assurance, site selection, and coverage. Measure- ments at a single site represent only one sample and one range of operating conditions. The correlation of results from the vehicles measured by remote sensing to vehicles and operating conditions (such as the load on the vehicle) of the entire fleet or to results measured From a standard test procedure re- mains problematic.S As with any measurement that estimates only concentra- tion of pollutants, there is also the issue of converting measurements from concentration units to mass emissions. Remote sensing provides the only way to estimate pretest repairs effec- sThe vehicle specific load (VSP) distribution (percentage of vehicles at different loads) can be calculated for a fleet of vehicles measured by remote sensing. In an IM240 test, a single vehicle is subjected to different loads for different amounts oftime. The load distribution in the IM240 will be different in almost all cases than the load distribution of a fleet of vehicles measured in a remote-sensing campaign. Jimenez et al. (1999) have recommended that the load distribution in the remote-sensing fleet be adjusted to that of the IM240 test by weighting the remote-sensing measurements by load so that the weighted load distribution is the same as that in the IM240 test.
154 Evaluating Vehicle Em issior~s I/M Programs lively. Extensive remote sensing can identify trends in emissions by mode] year before the I/M test to determine whether owners are taking actions to reduce emissions before theirtests. As with roadside testing, remote sensing offers a way to determine the extent of noncompliance with an I/M program, including whether vehicles that fait I/M tests are still driven in the I/M area. Remote sensing can also determine the percentage of vehicles exempt from testing but Diving in the I/M area, together with an estimate oftheir contnbu- tion to overall vehicle emissions. Data on Evaporative Emissions The evaporative emissions reductions possible from periodic inspection and repair programs are difficult, if no/impossible, to estimate. One needs to know the frequencies and impacts of evaporative system failures, the ability to detect failure, and the ability of the service industry to make effective and durable repairs. The only current data with which to estimate evaporative emissions from I/M tests on pre-on-board diagnostic (OBD) vehicles are limited to visual observations and gas-cap and iill-line pressure tests. There is no method for directly measuring evaporative emissions short of subjecting vehicles to a laboratory SHED (sealed housing for evaporative determination) test.6 Duration of Vehicle Repairs Issues The length oftime that repairs remain effective is a central issue for evalu- ating I/M program benefits. If emissions reductions from vehicle repairs last on average 2 years, then emissions reductions from I/M are double what they would tee if reductions lasted only ~ year. There has been some analysis ofthis issue, but it has certainly not been resolved, and further work must be done. The evidence to date is mixed. A number of studies find that at least some vehicles have reductions that last a very short time. Lawson ~ ~ 993) observed that many high-emitting vehicles stopped in random roadside pullovers had either passed their emissions test in the previous 90 days or went on to pass their emissions test in the following 90 days. This finding suggested that some 6The SHED test involves placing the vehicle in a sealed enclosure and monitoring HC concentrations over time.
Methods for Estimating Emissions Reductions 155 repairs were of short duration done simply to pass the test. The IMRC study (2000) found from change-of-ownership data that about 20% of vehicles that failed and later passed would have failed again immediately after the test, and McCTintock ~ ~ 999a) found in Colorado using remote-sensing data that a sign~fi- cant amount ofthe emission reduction estimated from lane data was lost within ~ month of the final IM240 test. Wenze} (in press) found in Arizona that half of on-road benefits disappear after ~ year and that 40°/O ofthe vehicles that fad] in one I/M cycle, fait again in the next cycle. Rajan ~ ~ 996) found in a Califor- nia study that on average vehicle emissions ofthe repaired vehicles returned to their original level after 2 years. There is also evidence that some emission reductions lastformuchiongerperiods. Again, the IMRC study (2000) found that the 80°/O of vehicles that appeared to receive lasting repairs had emissions reductions for NOX that extended at least a year, and for HC, deterioration in those benefits began after 9 months. This study was not able to look beyond a year after repair. in summary, there appears to be a distribution of repair duration of the vehicles that do have emissions reductions as a result of I/M. That distribution ranges from a matter of days to several years or beyond. There needs to be more study of the factors that influence how long repairs last, such as how different I/M program configurations might affect repair durability. Methods for Estimating EM Tailpipe Emissions Reflections The data sources described above can be used to estimate vehicle tailpipe emissions for model-year-specific emissions by a number of possible methods. We summarize three methods and describe the advantages and disadvantages of each. The reader is referred to Chapter 3, which discusses previous evalu- ations using some of the methods described below. Reference Method This method compares the emissions by model year measured by the test program with those of a reference program. The reference program may be a null program (non-~/M) or a benchmark I/M program. The EPA has recom- mended comparison with a benchmark I/M program. Before companug vehi- cle fleets in the test and reference program, they are adjusted for model year
156 Evaluating Vehicle Emissions I/M Programs and, if possible, odometer reading. Other factors influence fleet emissions besides I/M. These parameters include vehicle types, vehicle models, socio- economic characteristics of vehicle owners, altitude, climate, and fuel. Adjust- ments between the two fleets need to be made for these factors in order to estimate the effect ofthe I/M program. The effects ofthese other factors may be seen by comparing fleets from different areas having the same I/M pro- gram status. EPA released guidance on using the reference method for AM evaluation where the Arizona I/M program is used as the benchmark (Sierra Research ~ 997; EPA ~ 9986~. Critiques of this method are discussed in the following section. An advantage ofthe reference method is thatit canbe applied at any point during the program' s lifetime as opposed to only when there is an incremental change in the program (as with the step method of evaluation described be- low). It can account for the effects of multiple I/M cycles, pre-inspection maintenance and repairs, and deterioration of emissions repairs. If remote sensing is used to collect the data, deterioration of emissions for vehicles not inspected in the I/M program can also be estimated. However, it is difficult to find two areas that are similar in all respects except for their I/M programs. To the extent that the test area and the baseline non-~/M area are different outside of I/M program status and those differences have an impact on vehicle emissions, the difference must be fully accounted for in the analysis. Unexplained differences would be incorrectly attributable to the I/M program. Critiques of the EPA-Approved Reference Method The EPA guidance on the use of a benchmark program in Phoenix was critiqued by Wenzel and Sawyer (1998), and by Rothman (1998~. The EPA guidance document, the Sierra Research report describing this method, end the critiques may be found at the EPA web site (ht~://www.epa.gov/oms/epg/ progeval.htm). Comments in the critiques were not incorporated into the guid- ance document. Wenze] and Sawyer's concerns included possible sample bias in recruitment options; errors in the conversion of concentrations to mass emissions rates; and the fact that the method of model-year stratification, in which vehicles are grouped by mode! year, does not accurately group vehicles according to the technologies used in a vehicle's fuel delivery and computer control systems. Rothman's concerns included recruitment bias end use ofthe
Methods for Estimating Emissions Reductions 157 MOBILE model to get the evaporative emissions from a pressure test. The author recommenced complementary use of remote-sensing data from several independent sites. This guidance for I/M evaluations of state programs also was discussed in the I/M Program Evaluation Panel at the ~ 6th Annual Mobile Sources/Clean Air Conference, September 2000, sponsored by the National Center for Vehi- cle Emissions and Safety at Colorado State University. Richard Joy, of Sierra Research, stated that previous applications ofthe EPA Guidance on I/M pro- gram evaluation were not valid because vehicles in the benchmark pro gram in Phoenix were not uniformly preconditioned. He said that Sierra Research had looked at the following alternative evaluation methods before coming up with this evaluation method and rejected them: · Using the MOBILE model. · Comparing I/M test results before and after repair. · Comparing emissions test results from I/M to non-~/M areas. · Comparing air quality before and after I/M implementation. Comparison to a benchmark I/M program, with measurements based on adequately preconditioned (uniformly warmed up) vehicles, is recommended by Joy as the best way to evaluate other I/M programs. Potential problems to the benchmark evaluation method included the following: · Accounting for seasonal, fuel, temporal, and geographical differences (to be corrected by applying MOBILE correction factors for these parame- ters). · Loss of benefits due to program evasion and test fraud. · Bias introduced in recruiting vehicles. Application ofthis evaluation method requires emissions tests on a random sample of vehicles. According to Joy, the best method of recruiting vehicles would be by random roadside testing. However, some states do not allow roadside testing, and many others might not have the political will. Alternative recruitment methods alihave shortcomings. Random selectionafterI/Mtest- ing does not capture vehicles avoiding testing. Recruiting vehicles by mail introduces selection bias, even when vehicle owners are threatenedwithregis- tration denial for noncompliance. During the panel discussion at the conference, David Arnlin, ofthe CaTifor-
158 Evaluating Vehicle Emissions I/M Programs nia Bureau of Automotive Repair, nosed that the California evaluation of emis- sions reductions using roadside pullover data (described in Chapter 3 and in CARB 2000b) showed different emissions reductions in Sacramento and Los Angeles. Depending on which city was used as the benchmark, the other city's I/M program could have been considered to be exemplary or insufficient. Socioeconomic differences, vehicle mode] differences, and other factors might be responsible for the differences in vehicle emissions reductions. Such fac- tors are not accounted for or estimated in the current EPA guidance for evalu- ating I/M using the EPA-approved method. Step Method When an I/M program is initiated or significantly mode led, at some point about half the vehicles will fall under the new program and half will still be tested under the old program. in the step method, emissions of a random sample of vehicles tested under the new program are compared with those of vehicles yet to be tested to determine the effect ofthe change.7 The method was applied to roadside pullover emissions data by Lawson (1993) and the California Air Resources Board (CARB 2000b) to evaluate different versions of the Smog Check program. The CARB evaluation is described in Chapter 3 . Using emissions data collected with remote sensing, Stedman et al. ~~ 997) also used the step method to estimate the incremental effect ofthe enhanced Colorado I/M program. A critical advantage of the step method is that the tested and untested cohorts come from the same vehicle fleet.8 Thus, there is no need to correct for differences in climate, fuels, or socioeconomic factors required in applica- tion of the reference method. Further, the method possibly can be able to detect an impact of the program on fleet composition. Motorists might re- 7It should be noted, however, that the application of the reference method to an ongoing benchmark I/M program, the step method, or the comprehensive methods is complicated by the possible residual effects of prior I/M cycles, which might affect the emissions reductions occurring during subsequent cycles. tin the case of a step change in a biennial program where, for example, all even model-year vehicles have gone through the test and all odd model-year vehicles have not (or vice versa), corrections must be made for subfleet differences in vehicle age, mileage accumulation, or emissions-control technologies.
Methods for Estimating Emissions Reductions 159 . register their vehicles outside the I/M area to avoid being subject to the pro- gram, as was observed by Stedman et al. (1997, 1998) and McClintock (l 999b). The step method would tee able to observe that there are more such vehicles in the tested cohort than in the untested group. The step method can be applied to estimate the impact of a change to an I/M program or to estimate the deterioration in vehicle emissions from one test cycle to the next. Comprehensive Method In this method, vehicles are split into groups according to their test results: initial pass, faiVpass, faiVwaiver, and faiVno-pass (see Table 6-2~. Average vehicle emissions by test group are followed over time using remote sensing (Wenze] 1999a; IMRC 2000) or using change of ownership I/M test data (IMRC 2000~. Periodic test-cycle-to-test-cycle in-program data can also be used to estimate initial emissions reductions and repair deterioration for fail/pass cars. To get estimates of repair deterioration with better resolution (months rasher then annual or biennia!) from in-program data, initial test data from inspections made at intermediate times are necessary, such as emissions tests that can occur with change of ownership. The level and change of emis- sions over time give information about the emissions reduction seen as a result of the test and pretest repairs together with the emissions deterioration be- tween one test and the next. The comprehensive method can be used over a number of cycles. As described in Chapter 3, this method was used by the California Inspection and Maintenance Review Committee (IMRC 2000) to help estimate the emissions reduction for the Smog Check II program. A simplified version ofthis method involves calculating emissions reduc- tions for vehicles that fait and are repaired. Initial inspection data for each failing vehicle are compared with observations of emissions at final retest (whether the vehicle passed or not). Ando et al. (2000) used this method to evaluate the emissions reductions from the Arizona program. It has the advan- tage of being relatively simple because it requires only data collected by the I/M program. It also follows individual vehicles instead of examining changes in average emissions by model years. However, it does not account for all sources of emissions reductions listed in Table 6-1 . It is likely to overestimate emissions reductions due to I/M because of the regression to the mean prob- lem and because it does not provide any estimate of the amount of cheating (fixing to pass the test). It will understate emissions reductions because it does
160 Evaluating Vehicle Emissions I/M Programs TABLE 6-2 Vehicle Categones with Respect to I/M Status Pass Fail · Passing emissions test, and low on-road emissions · Passing emissions test, but high on-road emissions · Passing within X months · Obtaining a waiver after repairs over some minimum expenditure · No final pass within X months Legal: (1) Scrappeda (2) Sold outside the I/M region and no longer driven in it Illegal: (1) Dnven in the region, but without passing the I/M test (2) Sold outside the region but still driven in it (In some places this is against I/M rules.) aThere is some natural scrappage rate, which is about 5% per year in California for vehicles 10 years old. not provide an assessment of how much legitimate repair is occurring in antici- pation ofthe I/M test. Although the use in some states of fast-pass or fast-faiT algorithms may make analyzing test data more difficult, Ando et al. (1999) suggested one method for estimating full IM240 emissions from partial test results. Methods for Estimating Emissions Reductions From Induced Fleet Change In some cases, the methods descnbed above will provide estimates of emissions changes resulting from both vehicle repair and changes in the make- up of the fleet. The step method can evaluate both of these effects. Other methods have provided only estimates of emissions reductions by model year. To aggregate these estimates to emissions reductions from the fleet, estimates of any changes in the fleet makeup resulting from the I/M program must be included. Because ofthe I/M program, some motorists might decide it is either too expensive or too onerous to get their vehicles through the inspection pro- cess. The latter group of vehicles can be of several types. If a vehicle fails and receives some repair, it might qualify for a waiver in the region. In other cases, motorists might decide to scrap a vehicle earlier than they otherwise
Methods for Estimating Emissions Reductions 161 would, instead oftryingto get through the inspection process; or they might sell the vehicle in another market that is not subject to the same I/M requirements.9 Table 6-2 summarizes all the possibilities for the fates of failing vehicles. Induced Scrapping Motorists might fail the test, obtain an estimate of repair costs, and elect to scrap the vehicle instead of repairing it. Even if the potential repair bill exceeds the waiver amount, motorists might decide to scrap rather than face future repair cost uncertainty (most waivers are one time only). To evaluate this effect, we must compare the underlying "natural rate" of scrappage of vehicles with the scrappage rate with the I/M program. If motorists do scrap their vehicles early because of I/M, the resulting emissions reductions depend on what the vehicles' remaining lifetimes would have been without I/M and the difference in emissions between the scrapped vehicles and the replacement vehicles or alternative transportation modes. If a scrapped vehicle is marginal rather than a high emitter, the emissions reduc- tion induced is not large. Although there is reasonably good statistical information about the ex- pected remaining lifetimes of vehicles of different vintages, very little is Mown about the remaining lifetimes of vehicles having trouble passing the I/M test. These vehicles are more likely to be in worse overall condition and have Tower economic value compared with vehicles of similar age that do not have trouble passing the I/M test. It is clear that their expected remaining lifetimes would be lower with an I/M program, but how much lower? Alberini et al. (1996) found that vehicles scrapped under the voluntary Delaware Vehicle Retire- ment Program were more polluting than the average older vehicle and had about half the expected remaining lifetime of the average older vehicle, or about ~ .7 years. Dill (2000) reported that vehicles scrapped under a CARB buy-back program and under the Bay Area Air Quality Management Distnct' s vehicle buy-beck program had a lifetime expectancy of about 3 years. These vehicles also had higher emissions than other vehicles ofthe same model year. Finally, replacement~ansportation for scrapped vehicles is complicatedby the fact that the purchase of another vehicle, whether new or used, starts a 9Many regions have no I/M requirements because air-quality measurements do not exceed state or federal standards.
~ 62 Evaluating Vehicle Emissions I/M Programs series oftransactions through the market that are virtually impossible to iden- tify. The usual assumption is that scrapped vehicles are replaced by vehicles that represent the average of all vehicles in the fleet. Despite these difficulties, simplifying assumptions canbe made to estimate fleet effects from I/M. The induced scrappage rate canbe determined jointly with the induced relocation rate (see below) in the following way. From I/M program data, the number of vehicles or share ofthe inspected fleet that fails end never passes can be calculated. Some fraction ofthese vehicles will still be driven in the region, either with lapsed registrations or through some other illegal means. An estimate ofthis number can be determined by remote sens- ing or automatic license-plate readers together with I/M lane data (e.g., see Colorado Air Quality Control Commission ~ 999 and TMRC 2000~. Because these vehicles have not been repaired and might still be driven in the area, they should not tee included in any estimate of emissions reductions. The remaining vehicles can be considered either scrapped or relocated to another region; in either case, emissions in the region fall. In a biennial I/M program, a reason- able assumption is that the vehicles scrapped because of the program are scrapped 2 years earlier than they otherwise would (see discussion of the Alberini et al. ~ ~ 996) and Dill (2000) studies above) and that the emissions of the replacement vehicles are the same as the fleet average. Further research is needed to better understand how well such assumptions reflect reality. Induced Relocation Motorists or dealers might sell vehicles outside the region rather than pay large repair bills. These vehicles can be treated just like the scrapped vehicles described above forthe purposes ofthis analysis, if itcanbe demonstrated that they do indeed remain outside the program area. SUMMARY OF RECOMMENDATIONS FOR EVALUATION OF EMISSIONS REDUCTIONS On the basis of our review of methods for evaluation here in Chapter 6 end results of previous evaluations of emissions reductions in Chapter 3, the committee has a number of findings and recommendations. First, we summa- rize some key areas of uncertainty that future evaluations and studies need to address.
Methods for Estimating Emissions Reductions ~ 63 · Little is known about the durability of repairs in I/M programs. There is some evidence that a share of repaired vehicles go back to their failing emission levels within a few months, but other vehicles retain low emis- sions for longer periods (Rajan 1996; ENVIRON 1998; McClintock 1999a; Wenzel ~999a; IMRC2000; RegionalAir Quality Council2000~. Understand- ing how long repairs last is critical because the effectiveness of repairs has a large impact on the total emissions reductions achieved by an I/M program and the required frequency of retesting faiVpass vehicles. · A related issue is that of vehicles with widely variable emissions (Knepper et al. ~ 993; Bishop and Stedman ~ 996; Coninx 2000~. Are there vehicles with intermittent problems that produce emissions levels that vary between high and low for no apparent reason, so that they fait an initial test, pass the next without repair, and then appear again as high emitters on the road a short time later? How many vehicles pass a retest simplybecause their emissions were Tow that time rather than because they had received effective repairs? How many ofthese vehicles are there, and what are the implications for traditional I/M testing? · There is evidence that there are significant numbers of vehicles whose emissions decrease in the weeks before their I/M test (IMRC 2000~. These vehicles were earlier referred to as having had pretest repairs. The number of such vehicles and the extent and duration of their repairs need to be in- cluded in the evaluation of an I/M program. Additionally, it is necessary to consider how many of these vehicles would have received repairs without an I/M program. · Many vehicles that fail in I/M tests never get repaired to a passing level. These vehicles need to be tracked to determine whether they are scrapped or still driven in the I/M region. · The benefit of I/M programs in reducing non-tailpipe HC emissions is unlmo~vn. The potential for benefits from reducing evaporative emissions and liquid gasoline leaks should be evaluated. In addition to shedding light on these questions, evaluations must quantify all emissions reductions attributable to I/M programs. Based on the issues discussed in Chapters 3 and 5 related to modeling, it is clear that the MOBILE i°Here we are referring to vehicles that obtain repairs after having initially failed the I/M test. We distinguished between these fail/pass cars and the fail/pass cars that do not get repairs but rely on improved preconditioning or the variable nature of vehicle emissions to pass an emissions test after initially failing.
~ 64 Evaluating Vehicle Emissions I/M Programs and EMFAC models should not be used as evaluation tools. in the past, both have greatly overestimated the benefits of vehicle emissions-control programs, such as I/M programs. Evaluations of emissions benefits must be based on I/M-test and on-road vehicle measurements associated with the program being evaluated. Ideally, emissions benefits would be estimated with multiple sources o f e m i s s i o n s ~ a t a a n ~ m u ~ t i p ~ e e v a ~ u a t i o n m e t h o ~ s . E v a ~ u a t i o n s w o u ~ ~ b e p e r - formed periodically at the same location and compared with results from other program evaluations. Such a comprehensive evaluation would quantify all sources of emissions reductions listed in Table 6-~. The committee also recognizes that comprehensive evaluations can require a commitment of money and time not available to all state programs. The committee recommends that EPA ensure that at least some comprehensive evaluations are done that address the full array of emissions impacts incorpo- ratingmultiple data sources and evaluation methods. The committee realizes that there mustbe incentives for some states to do such comprehensive evalu- ations on a long-term basis, possibly by spreading the costs across all states. Some states, such as California and Colorado, already have incentives in their state requirements for evaluating I/M programs. As described in Chapter 3, these states have already performed multiple evaluations by a variety of meth- ods. Any state undertaking such a comprehensive evaluation also needs to have a well-established procedure for collecting and analyzing vehicle emissions data from in-program and on-road sources. The guidance for data collection and evaluation should be peer reviewed, and comments gathered during the review should be addressed. The committee believes that selecting three to five states with different program types and from different regions in the country would provide a suffi- cientrange offull evaluations. Besides providing for an estimate ofthe emis- sions reductions, comprehensive evaluations could tee usedby all programs to improve forecasts of I/M benefits from the MOBILE and EMFAC models and to assess the potential emissions impacts from changes in program design. Full evaluations could also be used to help less comprehensive program evaluations estimate allpotential sources of emissions impacts due to I/M. Some ofthe evaluations should tee conducted independently ofthe agencies that menage the I/M program or have a role in setting I/M policy and all evaluations should be peerreviewedbyindependentscientists,economists,andstatisticians. Results of evaluations should tee made public so that all states canbeneiit from whet is learned. EPA also should pursue publishing some aspects of these evalua-
Methods for Estimating Emissions Reductions ~ 65 lions inprofessiona~jou~nals so that they may tee further reviewed and dissemi- nated. States Conducting Comprehensive Evaluations Comprehensive evaluations must account for all the emissions reductions in Table 6-~: emissions impacts from pretest repairs; the initial emissions re- ductions from repairs and the length oftime repairs last; the amount of fraud in the program (including vehicles that fraudulently register outside an I/M area but still operate within the area); and the number of vehicles that are scrapped or sold because ofthe I/M program. One difiFicultyis the need to account for repairs that would have been done without an I/M program. There is no per- fect method for evaluating all emissions impacts from I/M the reference, step, and comprehensive methods all have their own inherent limitations. The ideal evaluation is a reference method comparing an I/M area with a non-~/M area, which would take into account normal repair (from the non-~/M area), pre-inspection repair, test fraud, and I/M benefit from repair (from the I/M area). However, there are a number of challenges to this approach, as dis- cussed earlier in this chapter. The most appropriate evaluation method will vary with the type of I/M program in place and the availability of comparison sites; the best method must include a non-~/M fleet for baseline comparisons. The most appropriate method might also depend on the timing of changes in the I/M program. However, evaluation methods and data sources are comple- mentary. Each adds information to reduce the uncertainty ofthe estimate and to better understand the effectiveness of the I/M program. For a comprehensive evaluation, in-program data canbe used to determine how many vehicles take the test. Program populations can be compared with actual vehicle registrations to evaluate how many vehicles are actually partici- pating in the program. These data can also show the number of vehicles that pass the test after initially failing in each I/M cycle. However, a key issue for determining emissions reductions is to know how many ofthese vehicles were repaired and how long repairs last. Because repairs could be temporary or simply made for "passing the test," remote sensing and random roadside pull- overs can be used to assess how well the repaired vehicles are staying re- paired. Roadside testing and in-program testing are ways to discover tamper- ing with emissions-control components. Remote-sensing or roadside-pullover data can be used to estimate the percentage of emissions reduction due to pre-
~ 66 Evaluating Vehicle Emissions I/M Programs inspection repairs. However, these must be distinguished from repairs that would have occurred without an I/M program. Program data can also identify how many vehicles fait the test and never pass. License-plate data collected as part of a remote-sensing database are necessary to determine how many ofthose vehicles are still driven in the region and to indicate whether changes to the makeup of the vehicle fleet are due to an I/M program. States Conducting Shortened Evaluations For the states that cannot do a comprehensive evaluation, a shortened evaluation should tee developed. The short evaluation should do the following: Not be based on the MOBILE model. · Include all components of emissions reduction (from Table 6-~. · Use the best evidence from data collected in full evaluations for the value of urdmown emissions-reduction components. · Use some on-road data from the local area if possible. in a shortened evaluation, in-program data, registration data, and any local on-road data that can be collected will be used along with evidence from more thorough evaluations to estimate the emissions-reduction components in Table 6- ~ . To the extent data are not available to estimate some aspect of emissions reduction from a program' s own data sources, data and assumptions based on the best comprehensive evaluations from other sources must be used. Infor- mation and assumptions should be updated over time as more evidence be- comes available. The use of assumptions based on evaluations of otherpro- grams for certain key variables is a reasonable approach if there are not great differences among programs in these variables. For example, if the amount of pretest repair is reasonably consistent across states that have measured this parameter, then use of the average amount of such repair is a good proxy in areas that do not measure it directly. If large variations are found in aspects of I/M performance among states conducting thorough evaluations, then this shortened evaluation method would have to be reevaluated. EPA will need to develop guidance for a shortened evaluation method. A review committee should be established to advise EPA in the selection of shortened evaluation methods and in selection of what information can be drawn from comprehensive evaluations to inform the shortened evaluation.
Methods for Estimating Emissions Reductions ~ 67 Assumptions used in the shortened evaluation can then tee continually improved over time as more evidence becomes available. The shortened evaluation method must be linked with results from compre- hensive evaluations. The shortened evaluation method should not go forward unless comprehensive evaluations are also being performed. Any shortened evaluation method needs to be validated. States conducting comprehensive evaluations oftheirI/M programs also should do a shortened evaluation. EPA will review the differences between the results ofthe comprehensive evalua- tion and the shortened evaluation and will modify the shortened evaluation method so that its results are more similar to the comprehensive evaluation" Validation of the shortened evaluation should be done at least once every 3 years. One way of structuring the shortened method would involve the following steps: · In-program data are collected over a test cycle and an estimate is made of emissions reductions for all failing vehicles: initial test results minus final test results (even for vehicles that still do notpass). In some cases, these results need to be adjusted to a standard emissions test (e.g., IM240 and FTP) to estimate the emissions benefit using correlation equations.' ~ · Adjustments are made to account for regression to the mean in test data. The magnitude of the adjustment can be made based on other more comprehensive evaluations. · The adjusted in-program data are aggregated across all failing vehicles and an assumption is made about the expected average length of repair dura- tion, not including fraud. (This assumption about repair duration would be taken from on-road evidence from comprehensive evaluations.) · A further adjustment to the results would be made based on an esti- mate of fraudulent emissions reductions or emissions reductions that are made only for the period of the test. This estimate could be based on the audits of the testing stations run by the state, remote-sensing or roadside-survey data if any are available for the particular region, or some average estimate of fraud in other programs. Sphere are inherent difficulties with correlating other test results with an IM240 test that evaluates emissions over only 30-240 seconds of a 2-year period and only when the vehicle is on its "best behavior" to 'pass the test." However, such correla- tions must be done to estimate mass emissions reductions.
68 Evaluating Vehicle Emissions I/M Programs · Further adjustment should be made to reflect pretest repairs and re- pairs that would take place without an I/M program. This adjustment could be based on estimates from other more comprehensive evaluations. ~ Retirement or relocation of vehicles as a result of I/M can be esti- mated based on the evidence from more comprehensive studies and from local vehicle-registration data and remote-sensing data. Performance Indicators In addition, both types of program evaluations should compile performance indicators. Although they do not directly estimate emissions reductions, perfor- mance indicators are relatively easy to measure, supplement the evaluations described above, end provide relatively concise indicators of a program 's suc- cess. EPA, and the states themselves, should use these performance indicators to rate states' I/M programs and to help direct improvements nationally. These performance indicators could include the following: · An estimate ofthe total number of vehicles driven in the I/M region, the share of those vehicles that are eligible for inspection, and the share of those that actually are inspected. · Failure rates by mode} year at the program cutpoints. · Estimates ofthe average emissions of passing vehicles and average emissions of failing vehicles. Share of failing vehicles that actually get repaired to below program cutpoints and their average emission rates before and after repair. Share of failing vehicles that do not ever pass the I/M test, their aver- age emissions, and an estimate ofthe number ofthose still driven in the area. The rate of repeat failures from one I/M cycle to the next. Estimates of the actual number of high emitters on the road. These indicators could also be checked against assumptions used in model- ing to make the models more realistic and improve the forecast of emissions reductions for the state implementation plan. Over time, alternative programs to reduce in-use vehicle emissions might be developed by states. Systems that rely exclusively on remote sensing or OBD may be used as technology improves in the future. These programs would also need to be evaluated. Some ofthe issues associated with evaluat- ing these emerging testing technologies are discussed in the following chapter.
