Click for next page ( 136


The National Academies | 500 Fifth St. N.W. | Washington, D.C. 20001
Copyright © National Academy of Sciences. All rights reserved.
Terms of Use and Privacy Statement



Below are the first 10 and last 10 pages of uncorrected machine-read text (when available) of this chapter, followed by the top 30 algorithmically extracted key phrases from the chapter as a whole.
Intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text on the opening pages of each chapter. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.

Do not use for reproduction, copying, pasting, or reading; exclusively for search engines.

OCR for page 135
4 Mode' Uncertainly and Evaluation MOBILE IS A TOOL for estimating current and forecasting future mobile- source emissions, including quantifying the effects of control measures. These results form key elements of many air-quality regulatory compli- ance programs and directly affect transportation planning and the selec- tion of control strategies. Thus, there is a need for a high degree of accu- racy from MOBILE. One of the specific charges to this committee is to assess, to the extent practical, the overall accuracy of the current version of the MOBILE model in predicting atmospheric emissions. Such information is derived from model evaluation. However, the necessary information to quantitatively evaluate the current or upcoming versions of MOBILE with great confi- dence does not exist. This lack of information is one of the most serious concerns with MOBILE and its use. Assessing accuracy in the context of MOBILE involves model evalua- tion, typically by comparing real-world emissions with model predictions. It also involves considering uncertainty and bias that arise *om the wide variety of observations, assumptions, and mathematical relationships that underlie MOBILE model algorithms. This chapter discusses uncertainty and model evaluation, and reviews previous studies on these topics. iThis report uses model evaluation in reference to assessing the ability of a model to accurately represent the real world, for example by being able to estimate the emissions from mobile sources with little error. This differs from model validation that refers to assessing the correctness of the form of the model. MOBILE is based 735

OCR for page 135
7 36 MODELING MOB/[E-SOURCE EMISSIONS DEFINITION OF TERMS Accuracy, Uncertainty, and Bias Accuracy in MOBILE refers to its ability to correctly estimate the true value of emissions. Bias is the tendency for estimates to be consistently higher or lower than the true values. Uncertainty is the variability (or scatter) in MOBILE's prediction about the actual emissions. MOBILE is accurate, for example, if it predicts the correct emissions factors for the fleet of on-road vehicles, and if it predicts the actual changes that result from mobile-source emissions control programs such as inspection and maintenance (I/M). (MOBILE, as typically applied, provides only point estimates without any statistical confidence intervals around those esti- mates). MOBILE predictions can be assessed by comparing accurate in- use measurements of vehicle emissions and air quality to model predic- tions. This is termed model evaluation and is the subject of the second half of this chapter. Uncertainty and bias in MOBILE arise from many sources, primarily from the data used to construct the model, and from errors in analyses and assumptions leading to model formulations (discussed further below). Un- certainty and bias in MOBILE are difficult to assess because of the com- plexity of the model, the uncertainty in the underlying emissions data and model formulation, the uncertainty in the input data, and the difficulty in obtaining accurate measures of real-world emissions (e.g., from analyzing ambient data). Uncertainty, as used here, should not be confused with repeatability. If MOBILE is run several times with the same set of inputs, the model will always generate exactly the same output. The model is repeatable simply because it has no stochastic or random component; this does not by any means imply that the model is accurate. Figure 4-1 shows the difference between bias and uncertainty. The top box in Figure 4-1 shows a case in which the model provides accurate re- sults. The bottom box shows estimates that are uncertain (in that they are scattered about a one-to-one correspondence) but unbiased. The mid- dle box shows estimates that are biased; on average the predictions are below the actual emissions. For MOBILE's many uses, it is important that the model's estimates be accurate with a high degree of certainty and on statistical analyses of data, and the form of MOBILE is a set of statistical rela- tionships. Thus, model validation would correspond to determining whether the statistical relationships were derived in a valid manner and implemented correctly. Model evaluation determines whether those relationships provide accurate results.

OCR for page 135
M ODES UNCERTAINTY AND EVALUATION 7 3 7 0.50 0.40 - ~e 0.30 o - LO E 0.20 0.10 0.00 0.50 - 0.40- ._ c a ._ In ._ - - ~ 0.10- / /. 0.00 - 0.00 0.30 - 0.20 - Accurate / - / / ,,' / . . . .. . . . .. ... / ,. ./, , . . 1 0.00 0.10 0.20 0.30 0.40 0.50 Predicted Emissions (g/mi) Biased 7 - ~ ... , . ~ 1 0.10 0.20 0.30 0.40 0.50 Predicted Emissions (g/mi) Uncertain 0.50 0.40- E 0.30- o ._ u' E 0.20 I as 0.10 0.00 - / 0.00 0.10 0.20 0.30 0.40 0.50 Pre d icte d Eon is s ions (g/m i) Series1 Series1 Series1 FIGURE 4-1 Representation of bias and uncertainty (hypothetical data).

OCR for page 135
~ 38 M ODEL/NG MOB!LE-SOURCE EMISSIONS a low bias. If not, transportation and air-quality planners could be led to implement costly, unnecessary control programs. Sensitivity Model sensitivity refers to the variation in model output in response to changes in model inputs such as average speed, ambient temperature, fuel volatility, and I/M program parameters. It is important that air-quality planners understand the effects of changes in these model inputs. The U.S. Environmental Protection Agency (EPA) has provided sensitivity analyses for some earlier versions of MOBILE (see discussion below), but not extensive analyses for MOBILES. A comprehensive sensitivity analy- ses should be performed for all model inputs and provided as part of user guides for all future versions of MOBILE. TYPES AND SOURCES OF UNCERTAINTY AND ERROR Many kinds of uncertainty plague the emissions estimates provided by MOBILE. A major fraction of those uncertainties arise from limitations in the scientific and technical basis of MOBILE as well as the data inputs, including sampling and measurement errors. As described above, the pri- mary source of uncertainty in MOBILE output is in the underlying emis- sions data used to generate the model formulations. There can also be limitations in MOBILE's structure. Input uncertainties, including data on vehicle characteristics and usage, are propagated through the model and contribute substantially to uncertainty. Another source of uncertainty that should be analyzed is the true variability in the system being mod- eled. It is clear that it is not practical to eliminate all uncertainty from mobile-source emissions models. MOBILE is a statistically based model, and its accuracy depends on valid and comprehensive samples as the foun- dation of the statistical relationships within. Some uncertainty comes from the random variation in the relatively small samples. No sample short of 100% (a complete census) can be large enough to completely elimi- nate randomness from affecting parameter error, but in all practical cir- cumstances, the influence of random variation on an estimate of a parame- ter decreases as the sample size gets larger. Even if 100% of the on-road vehicles were tested, uncertainty would still arise from the incompleteness and unrepresentativeness of the tests and errors in the testing procedure. Analysts must also consider true variability and its impact on uncer- tainty. Within defined fleets, manufacturers do not produce exactly the same kinds of vehicles. Vehicles from a single manufacturer are not iden-

OCR for page 135
MODEL UNCERTAINTY AND EVA[UAT!ON 7 39 tical. Also, emissions from any one vehicle vary from time to time and place to place in ways that are correlated with age, mode of driving, and many other things. Fleet characteristics vary spatially, as do topography and driving habits. The contribution of this true variability to the overall uncertainty of MOBILE is unknown, but could be partially characterized in an advanced mobile-source emissions model if the data (both input and model formulation) were available. Uncertainties further arise from limitations inherent in the model's structure. For example, no fleet of vehicles is well characterized. Road- way networks and associated driving patterns are not perfectly repre- sented. MOBILE does not capture all the factors leading to emissions (particularly high emissions, such as those induced by steep road grades). These and other model limitations discussed in Chapter 3 influence uncer- tainty. Following are descriptions of the general types of uncertainties that arise in the MOBILE model. Several examples of each type are provided, for both MOBILES and MOBILES. For additional information, see the study by Wenzel et al. (In press), which discusses emissions variability in more detail, and also describes other issues that complicate the statistical analysis of vehicle-emissions test data. Nonrepresentative Vehicle Samples MOBILE algorithms and emissions factors are based largely on test data from in-use vehicles that are solicited through the mail or by recruit- ment at I/M test stations. Typically, the owner is provided with a small payment and a rental vehicle until testing is completed. Free vehicle re- pairs are sometimes provided as an incentive. Response rates for such recruitment efforts are very low, typically less than 25% and sometimes as low as 5%. Recruited vehicles have serious bias issues because high emitters and tampered vehicles as well as expensive luxury vehicles are less likely to be voluntarily submitted for testing. As discussed in Chapter 3, very high emitting vehicles are a relatively small fraction of the on-road vehicle fleet, but they contribute a very large fraction of total vehicle emissions. Emis- sions from high-emitting vehicles are much more variable than emissions from normal emitters (Knepper et al. 1993), and thus require a large sam- pling fraction to obtain reasonably accurate estimates of their emissions (and to estimate the effects of control programs for their emissions). It is thus critically important that such vehicles be appropriately represented in emissions testing programs. If emissions from high emitters are not properly characterized, then MOBILE emissions factors can be seriously biased.

OCR for page 135
740 MODELING MOBILE-SOURCE EMISSIONS EPA recognizes that underrepresentation of high-emitting vehicles in the Federal Test Procedure (FTP) and IM240 databases underlying MOBlLE5's basic emissions rates is a serious shortcoming that produces biased Jow) emissions estimates. For MOBILES, EPA is proposing to ad- just the basic emissions rates based on data from the Dayton, Ohio, IM240 program. Although this is a step in the right direction, there are still bi- ases in the Dayton IM240 data. One type of bias arises from noncompli- ance with the program - the database does not include emissions from ve- hicles that are registered in the I/M area but do not obtain the required inspection (and repair if needed). Remote sensing studies have shown that these noncomplying vehicles have higher than average emissions (e.g., Stedman et al. 1997; Stedman et al. 1998; Wenzel 1999~. In fact, the I/M program compliance rate is one of the inputs to MOBILE, and the model assumes that noncomplying vehicles have emissions about twice as high as complying vehicles. A second problem is that when the Dayton program was first implemented, there is strong evidence that owners of vehicles that had been registered in the I/M area changed their registration to sur- rounding counties that were not subject to the Dayton I/M program ~IcClintock 1999~. Again, these vehicles are more likely to be higher emitting vehicles. Although EPA recognizes the inherent biases in their testing data, use of the Dayton IM240 data to adjust for those biases still likely results in biased emissions estimates. Some of the MOBILE algorithms and correction factors are based on very small samples, which may be nonrepresentative of the population. An example for MOBILES is the test data used to derive air-conditioning correction factors for light-duty vehicles (LDVs). As described in Chapter 3, the effects of full air-conditioning operation on vehicle emissions were determined from a sample (of 37 vehicles) 23 passenger cars and 14 light-duty trucks, al] from model years 1990 to 1996. Only five of the vehi- cles were high emitters for at least one pollutant. Similarly, the activity data for air-conditioning operation were obtained from a fleet of only 20 vehicles operating in Phoenix, Arizona, from August to October 1994. And, although the actual air-conditioning operation depends on the torque generated by the compressor, no data were available for this variable. Variability in Vehicle Emissions Vehicle emissions are highly variable, for a variety of reasons. Two ve- hicles of the same make and manufacturer, model year, technology, and accumulated mileage can have very different emissions measured on the same test or drive cycle. Such variation can be caused by factors such as how the vehicle has been driven and maintained, prior tampering with emissions control system components, and repeated excessive driving

OCR for page 135
MODEL UNCERTAINTY AND EVALUATION 7 4 7 loads. Studies have also shown relationships between socioeconomic fac- tors and vehicle emissions, with vehicles in lower than median income households exhibiting higher than average emissions (Singer and Harley 2000). Vehicles of the same age and technology also exhibit differences in emis- sions because of manufacturing and emissions-control design differences. Analyses of I/M data have shown that specific vehicle models have much lower or much higher emissions than average (Wenzel 1997~. In general, vehicles with higher emissions exhibit much more test-to-test variability than lower-emitting vehicles (Bishop and Stedman 1996~. Many factors can contribute to variability in repeated emissions tests of the same vehicle. For example, failures of some emissions control system components (such as a partially degraded catalyst) can be intermittent and therefore result in higher emissions some of the time. Other sources of differences in the vehicle emissions test data underlying the MOBILE model arise from the testing process. For example, back-to-back emissions tests will vary because of differences in the measurement equipment, cali- brations, and personnel (e.g., driving styles in tracking a target speed-time trace on a dynamometer). Those and other factors create uncertainties in the statistical models fitted to the data that are the basis of MOBILE emissions factors. An ex- ample of the enormous scatter in the emissions test data underlying MOBILE is shown in Figure 4-2. The figure shows the test data used to estimate NOX emissions for Tier 1 LDVs in MOBILE6. The solid line in the middle of the figure corresponds to 2 times the FTP standard; vehicles with emissions above this level are considered high emitters. Partly be- cause of the lack of sufficient data, and partly because EPA assumes that high emitters are "broken" vehicles whose emissions are always high no matter how old the vehicles are, emissions for the high emitters are mod- eled as a constant (the upper dashed line). This emissions estimate is then adjusted with the Dayton IM240 data as described above. Below the solid line, the vehicles are considered to be normal emitters. For these vehicles, the basic emissions rate is modeled as a linear function of accumulated mileage (the lower dashed line). Clearly there is large uncertainty in the emissions data and consequently in the basic emissions rates for normal- and high-emitting vehicles estimated from these data.2 These vehicle-to- vehicle differences are critical for some uses of MOBILE, but the scatter 2EPA has updated their original analysis of these data for Tier 1 NOx emissions using additional data sets. The additional data reduces the confidence limits (al- though not the scatter) for the normal emission regression equation. However, there is still significant uncertainty in the estimated mean value for high emitters.

OCR for page 135
742 - U] ._ I: o . . ~ . 1 ~ 1 ~ , 1;: 1 ,o .!2 1 ~ 1 I: 1 ~ , ~ 1 ~ , ~ 1 At is ! it 1 . . . . . . ~ , ~ Cal O ~ 10 ~ ~ O ~ ~ ~ ~~ ~ O 2 1 ~ ~ 1 l ~ ~ 41 t ~ 1 ~ i l . . ~ , . . t . $: LU It it l. ; ~ t=.- . .' . ~ :~N .~ L lo N EN 004 it_ ._' A A o . - A _' A C) . _~ ~5 O _ _ do lo lo lo ._ o of ~ . O O IwI6 ~ N O O O O CO o y Fl a C~ a a) ~D a) Ce C~ CQ ~Q ~ O .. :m ~ ~ O a, v C) - .o a 1

OCR for page 135
M ODEL UNCERTAINTY AND EVALUATION 7 43 tends to get smoothed out in aggregated data, such as estimates of area- wide burdens. What does not get smoothed out is consistent bias, such as systematic under-estimation of the emissions from high-emitters. Incorrect Model Formulation The emissions factors, emissions-factor adjustments, and estimates of emission control program effects in MOBILES and its predecessors are estimated from statistical analyses of available test data. The statistical models are chosen using both engineering considerations (to represent the physical process) and statistical considerations (such as selecting the model that produces the best statistical correlation). Uncertainties can arise from incorrect or inappropriate engineering and statistical models. Examples of incorrect physical-engineering models include the following: Neither MOBILES nor MOBILES includes road-grade effects, as these are not incorporated into the test data underlying the models. Road- grade influences vehicle emissions; higher emissions occur under vehicle load conditions such as steep road grades. Light-duty truck emissions are sometimes estimated from automobile test data, because of the lack of sufficient data for trucks. Although some light-duty trucks are used (and may emit like) passenger vehicles, other light-duty trucks are used regularly as working vehicles and frequently carry heavy loads; emissions from these working trucks are likely to be higher than automobiles of the same model and age. In MOBILE, the lack of sufficient data to estimate the effects of high emitters is sometimes filled in by assuming that high emitters behave as normal emitters do. In EPA's multistep adjustment of basic emissions rates for high emit- ters using Dayton IM240 data, one step is to estimate full IM240 emis- sions from fast-pass IM240 data (see Glossary for definition of fast-pass). This is done using second-by-second IM240 data from Wisconsin, so simu- lated fast-pass emissions can be compared to full IM240 emissions (EPA l999g). The regression equation developed by EPA has the time (in sec- onds) of the fast-pass as one of the predictors of the full IM240 emissions; full IM240 emissions are assumed to be linearly related to the logarithm of fast-pass emissions. Given the variation in the speed-time trace of the IM240 data, there is no physical reason why fast-pass time should be lin- early related to full IM240 (or to the log of full IM240) emissions. How- ever, the coefficient for the fast-pass time is statistically significant in the regressions for all three pollutants (CO, VOCs, and NOX). This appears to be because newer vehicles are much more likely to pass early in the test than older vehicles (Pollack et al. 1999a).

OCR for page 135
744 MODELING MOBILE-SOURCE EM`SS/ONS Inaccurate physical-engineering models can also arise from the failure to obtain correct data in a test program. Examples of that include the follow- ~ng: MOBILES speed-correction factors (SCFs) were determined from emissions of vehicles driving over a series of test driving cycles. The driv- ing cycles were characterized only by average speed, although, as dis- cussed in Chapter 3, there are many other factors in a driving pattern that also affect vehicle emissions. The SCFs used in MOBILE6 are facility-spe- cific. For each facility, driving cycles were developed from analyses of real- world (instrumented vehicles) driving-pattern data. Although there is more aggressive driving in these cycles, and they are facility-specific, the cycles are still characterized for a given facility, by a single parameter- average speed. The development of modal (or second-by-second) emissions models (discussed in Chapter 5) will go a long way towards resolving these Issues. The FTP cycle and the facility-specific speed correction factor cycles of MOBILE6 represent samples of a universe of driver and vehicle behav- ior. That universe may have low- frequency events with extremely high emissions. If these low-frequency high-emission events are not properly represented in the cycles used for MOBILE, an accurate picture of exhaust emissions will not be obtained. Some of the data critical to estimation of air-conditioning effects on emissions were not available for use in MOBILE6. These include the ef- fect of vehicle speed, the time the compressor is on, the validity of the heat index as a measure of air-conditioner use, car occupant behavior in air- conditioner use, or the effect of the actual compressor torque. Examples of incorrect statistical models include the following: In some analyses, the intercept of the statistical model is forced through zero, to match physical processes; this represents a trade-off be- tween the correct statistical model and the correct physical model. A1- though such model alteration makes sense from an engineering point of view, it introduces bias into the resulting statistical model. Two examples of this practice in MOBILE6 are the following: The determination of hot-running emissions from FTP test data; the test data are first transformed using logarithms, then the intercept is forced to be zero (EPA l999k). MOBILE6 uses Dayton IM240 data to adjust FTP test data to ac- count for the absence of high emitters in the FTP sample. The exhaust-

OCR for page 135
M ODEE UNCERTAINTY AND EVALUATION 7 45 emissions rate adjustment is treated as an additive function of mileage, with zero increase in the adjustment at zero mileage (EPA l999g). Logarithmic transformations are frequently applied to emissions data for fitting statistical models to bring the emissions data closer to the nor- mal (Gausian) distribution, for which most statistical methods are devel- oped. The transformed data may still be non-normal, and the translation of results back into the original scale may be hard to interpret. In addition, the log transformation has the effect of reducing the influ- ence of high values, which may be of greatest concern. Distributions of vehicle emissions typically show the majority of the emissions from normal emitters at relatively low levels, and a small fraction of the emissions at very high levels. These distributions with long right tails are much closer to lognormal than normal, although other distributions such as gamma have been fit as well (Zhang et al. 1994~. Another reason that logarithmic models are commonly fit is that variability in vehicle emissions is higher at higher emissions levels; logarithmic transformations stabilize the vari- ance. However, logarithmic transformations are usually inappropriate in analyses of data sets with both normal and high emitters. With Togarith- mic transformations, the effects of high emitters are minimized relative to normal emitters, whereas in the atmosphere the effects of high emitters are, in fact, much greater than normal emitters. In summary, logarithmic transformations offer some convenience in analysis and the opportunity to use simple statistical models, but at a cost of introducing potentially seri- ous errors. Figure 4-3 shows an example from MOBILES of an inappropriate use of a logarithmic model. The data in the figure show hydrocarbon (HC) emis- sions from model-year 1993 fuel-injected cars from a data set of Wisconsin IM240 second-by-second emissions; these data were used as part of the multistep process to adjust basic emissions rates using Dayton IM240 data (EPA l999m). The Dayton data contain many fast-pass emissions tests, and the Wisconsin data were used to develop regression equations to pre- dict fast-pass emissions (FHC on the horizontal axis) to full IM240 emis- sions (HC on the vertical axis). The graph on the left is a scatterplot of the full IM240 HC emissions against the fast-pass HC emissions; the top and right sides of the figure provide histograms of each of these variables indi- vidually. Clearly these emissions measurements are not normally distrib- uted. The right graph shows the same data, but with a logarithmic trans- formation applied to both the full IM240 (LHCnz on the vertical axis) and fast-pass HC (LFHC on the horizontal axis) measurements. Both the scat- terplot and the marginal histograms show that the logarithmic transfor- mation is an over-transformation it diminishes the effects of the high-

OCR for page 135
756 MODEL/NO MOB/LE-SOURCE EM!SS/ONS grams (Wenzel 1999~. The study included 4 million readings of 1.2 million vehicles. The study compared the MOBl:LE and IM240s estimate of the emission reductions resulting from an I/M program versus estimates made from the remote-sensing data for vehicles subject to IM240 and remote sensing tests. The results, shown in Table 4-2, indicate emissions reduc- tions from I/M programs calculated from IM240 data are less than the re- ductions predicted by MOBIILE. Using remote sensing to estimate emis- sions reductions from I/M produces the smallest estimated emissions re- ductions. The discrepancies, particularly between results from IM240 and remote sensing data are likely due to about 33% of vehicles failing I/M testing (high emitters) do not return for retest (disappear), and 60% of those that fail continue to operate in the airshed after 6 months; and repaired vehicles have a high deterioration rate. ROADSIDE INSPECTION Roadside pullover studies in which a random sample of vehicles are pulled off the road and subject to a loaded-mode emissions tests could per- haps offers the best direct measure of the overall real-world fleet emis- sions rates if a sufficient sample size is collected. Because on-road vehicles are selected at random and emissions are measured under actual real- world conditions, it does not have most of the disadvantages of IM240 test- ing and remote sensing previously mentioned. TABLE 4-2 Comparison of Emission Reduction Estimates for an I/M Program Based on MOBILE, IM240, and Remote Sensing Data Emissions Reduction Method CO VOC NOX MOBILES Prediction 16.2% 16.9% 16.7% Arizona IM240 Data Analysis 14.5% 14.0% 7.1% Arizona Remote-Sensing Data 7% 11% Not measured Analysis Source: Wenzel 1999. 4A loaded-mode test is one that puts a car through a simulated driving cycle on a dynamometer.

OCR for page 135
M ODES UNCERTAINTY AND EVALUATION 7 57 The California Bureau of Automotive Repair (BAR) conducted random road-side pullover inspections of over 27,000 vehicles in 1997-1999. The objectives of the study are to help characterize fleet emissions and estab- lish a baseline for evaluating California's I/M program effectiveness. BAR recognized the potential inaccuracies of directly using I/M program data to assess real-world effectiveness of I/M. BAR is also conducting a similar series of tests after implementation of an enhanced I/M program in Cali- fornia to accurately assess the effectiveness of this program. When the results and analysis of this test program become available, they might pro- vide the best insight into the accuracy of emissions models such as MOBILE. AMBIENT AIR-QUALITY MONITORING AND MODELING Using MOBILE-generated emissions data in airshed modeling and com- paring the results with measured air-quaTity data offers yet another ap- proach to testing the accuracy of MOBILE emissions predictions. An ex- tensive ozone-modeling and emissions-inventory study comparison was made for the South Coast Air Basin in the Los Angeles area in 1987 (Chico et al 1993; Harley et al 1993b; Wagner and Wheeler 1993~. This study used California's EMFAC mobile-source emissions-inventory model. EMFAC has produced lower estimated emissions than previous versions of MOBILE, but the emerging MOBILES is expected to compare relatively closely with previous versions of EMFAC. Thus, the California study has relevance to evaluating the accuracy of MOBILE. This study found that airshed modeling substantially underpredicted ozone levels. Additionally, it was determined that when the on-road mobile-source VOC emissions, predicted by EMFAC, were multiplied by a factor of 2.5 and the airshed model rerun, the airshed model predictions of ozone closely matched ambi- ent measurements. Figure 4-6 shows the comparison between ambient observations and the airshed model run for both levels of VOC emissions. The South Coast study also compared CO to NOX ratios and VOC to NOX ratios derived from the EMFAC-airshed model predictions with ambient measurements in the Sherman Way tunnel in Van Nuys. Table 4-3 shows that the two measured values are similar, further indicating that the emissions model underpredicts CO and VOC emissions from motor vehi- cles by about a factor of 2. A more recent Desert Research Institute (DRI) study of ambient air- quality measurements versus emissions model estimates for Los Angeles (Zielinska et al 1999) indicates a deviation in the VOC to NOX ratios simi- lar to the findings in thel987 South Coast study. The results of this study found that the EMFAC model underestimates the VOC to NOX ratio of

OCR for page 135
7 58 M ODE[ING M OBILE-SOURCE EMISSIONS 40 20 _ E Cat 10 of o . ,, , ,., , ,., ,.,.,,,, , ,,,,,I,,,,, ,,., - 00 06 12 18 00 Ob 12 18 00 06 12 1B Aug 26 Aug 27 Aug 28 ,, , .,, ,,,,, ,,,, , , , ., ~ ~ I ~ ~ ~ ~ I .~'rr''' l, I, ,,,, i I, I, I . ., . ~ . . ~ CLAR + OBSERVED BASE SENSITIVITY + ~1 Jar t I ~ ) I ) ~ I+ / 1 ~ 1 + J I Hi. i+ L ,+ LIT ~ , , 5 ~ , ,' ~ AL / ~ _ I A ~ _ FIGURE 4-6 Comparison of airshed model predictions of diurnal ozone con- centrations with observations of ambient ozone concentrations for two dif- ferent VOC emissions levels. Model-predicted VOC emissions are multi- plied by a factor of 2.5 in the sensitivity case to improve the fit to the am- bient ozone concentration profile. Observations are at Claremont (CLAR), California. Source: Fujita 1999. emissions by a factor of 2.15 compared with ambient data collected in Los Angeles in 1995. MOBILE6 will possibly make these ambient and emis- sions inventory discrepancies even worse. This conclusion is based on emissions data in EPA's paper (1998k) on possible regulatory action with respect to Tier 2 vehicle emissions and gasoline sulfur standards, in which EPA made adjustments to MOBILE5b to simulate the expected MOBILE6. MOBILE6 was unavailable to the committee to confirm if this conclusion is correct. The South Coast and DRI studies are thus consistent with other inde- pendent techniques (IM240 and remote sensing) in demonstrating that EMFAC (and, likely MOBILE as well) underestimates VOC emissions and, in particular, that it results in a significantly inaccurate estimate of the VOC to NOX emissions ratio of the real-world fleet.

OCR for page 135
M ODES UNCERTAINTY AND EVALUATION 7 59 TABLE 4-3 Ratio of Measured to Modeled Emissions, Using EMFAC7E Model Results, for Overall Ambient Measurements and Tunnel Studies R1_ I ~ ]~ LC 1 LNOX measured 1.5 LC NOX voc _ Modeled NOX Measured 2.5 CIVIC] LNOX UMOdeJed 2.1 2.2 Source: Fujita 1999. TUNNEL STUDIES The analysis of air samples in highway tunnels has been used for sev- eral years as a means of measuring vehicle emissions and testing model predictions (e.g., Pierson et al. 1990~. Tunnel studies capture the emis- sions from a large number (typically thousands) of in-use vehicles, thus providing measurements of the fleet's average emissions. Generally the vehicles are operating in a hot-stabilized cruise mode with average speeds from around 25 to 70 miles per hour (mph). Some tunnels have a signifi- cant grade. A report published by the Coordinating Research Council (CRC) (Gertler et al. 1997) summarizes results from a 1995 study in five different tunnels in Boston, New York City, Phoenix, and Los Angeles as wed as *om several previous urban tunnel studies. The CRC tunnel study results are summarized in Table 4-4, where the average emissions factors for CO, VOCs, and NOx are given, as well as the ratios of several emissions fac- tors. The sampled fleets were largely light-duty gasoline fueled vehicles. The data for the Fort McHenry and Tuscarora tunnels were obtained for a mixture of light-duty and heavy-duty vehicles and were analyzed to ex- tract the light-duty component reported in the table. The third column of the table shows a range of about 2-fold in the aver- age speed of the vehicles in the different tunnels. Emissions of CO, VOCs, and NOX differed substantially among the tunnels, typically about a factor

OCR for page 135
760 A ho CQ A ~ a) ~ ox- Cal o m ~ V e o o a) . - ~ V ~ a) ED o AL > ~ tD Cal o V2 Ct I En Cal o V_~` V ~ V X ~ so ox I, Cal CO o ~ o Cal o ~ ~ ~ ~ ~ ~ ~4 ~ ~ o o o o o ~ ~ o o o o o o o ~ ~ o o Ct ~ Cal Go ~ ~ ~ o ~ -co ~ Do ~ 00 Dad ~ ~ ~ ~ ~ . . . . . . . . o o o o o ~ o o o ~ ~ ~ 0 Rae ~ cat 0 0 ~ co c~ co ~ o C~ ~ 00 ~ CO ~ ~ ~ C~ L~ ~ O C~ ~ C~ O ~ ~ ~ 00 ~7 O O ~ ~ ~ 00 C~ 0 00 00 ~ C9 ~ ~ CO . . O O O O ~ ~ O O O C~ ~ ~ ~ ~ ~ O O 1 - t~ ~ ~ C~ O O 00 O CO . . . . ~ 00 CD . . . . . c~ L~ ~ ~ ~D ~ ~ ~ ~ C~ 5- L~ ~ ~ ~ L~ ~ d4 ~ ~ Ct ~ ~ ~ ~ ~ 00 a a) ~n u2 ~ ~ ~ ~ ~ > ~, ~ ~ ~ O ~ O a, ~ - ~ ~ ~ 0 ~ u2 ~ ~ V ~ V ~ E~ bD C~ ,. 5- C~ Ct 5- C) 5- Ct C? q) ~Q E~ ._, C~ C~ C~ :^ ,Q o - , CO .G d o 5- Ct a 3 ~ ~o ~ ~, U) {~5 >. C~ U) 5- J3 ~ ._, - Ct s~ s~ . . C) o U:

OCR for page 135
M ODEL UNCERTAINTY AND EVALUATION 7 6 7 of 4. These differences can be accounted for by the differences in the ages and modes of operation of the vehicles being tested. However, there is much less variation in the ratios of some of the pollutants, notably the VOC to NOx and CO to CO2 ratios. The CRC report discusses some incon- sistencies between these data and both the MOBILE and California's EMFAC emissions models. Although tunnel data provide only a snapshot of vehicle emissions, they can be valuable and should continue to be used in testing the accuracy of MOBILE and examining the effects of fuels, operating mode, and fleet composition. However, vehicle operation in tunnels tends to significantly deviate from average real-worId conditions. Tunnel traffic tends to have higher speeds, less stop-and-go driving, and less loaded-mode operation than average real-worId urban conditions. CHEMICAL-MASS BALANCE Chemical-mass balance (CMB) is another approach for evaluating emis- sions model estimates with ambient observations. CMB uses a sophisti- cated set of chemical fingerprints derived from speciation of source emis- sions, which are apportioned mathematically to ambient air samples. An oxidant assessment study in southeast Texas (Fujita et al. 1995) included extensive comparison CMB estimates versus MOBILE-estimated VOC and NOx emissions. This study distinguished contributions from liquid gasoline and gasoline vapor from vehicle exhaust in the ambient atmospheric measurements. The findings of this study and another CMB study (Korc et al. 1995) coun- ter arguments that it is evaporative emissions that account for MOBILE's underprediction of VOCs. The major conclusions of these studies are The sum of ambient liquid gasoline, gasoline vapors, industrial and compressed natural gas contributions agrees reasonably well with and val- idates the corresponding MOBILE emissions inventory estimates. The discrepancies between CMB ambient- and emissions-derived VOC to NOx ratios and ambient- and emissions-derived acetylene (a major tracer of fingerprint of motor vehicle exhaust) at the Clinton site suggest that the absolute amount of on-road mobile-source exhaust VOC emissions were substantially underestimated by MOBILE. The average ratio of CMB-derived ambient VOC emissions from mo- bile sources compared with those estimated from MOBILE at the Clinton site was 2.3. The latest state-of-the-art CMB study was conducted in the Denver area

OCR for page 135
7 62 M ODELING M OBI[E-SOURCE EMISSIONS in 1996 and 1997 under the Northern Front Range Air Quality Study (NFRAQS). The study was directed toward evaluating particulate matter (PM) emissions and sources; it indicated that PARTS greatly underesti- mates the contribution of gasoline vehicles compared with diesel vehicles. Tables 3-10 and 3-11 show this underestimation of emissions rates for LDVs by PARTS. The study also found a very large contribution of start emissions and high emitters to the total emissions from gasoline vehicles. Table 4-5 provides a summary of pertinent NFRAQS results. Underesti- mating emissions from starts and high emitters may be a significant cause of MOBILE's underprediction of other emissions (CO and VOCs) found in various studies. Watson et al. (in press) contains a recent summary of CMB studies. These studies tend to show that the relative contributions of mobile source VOC emissions to the total inventory determined by CMB are two to three times higher than those estimated using mobile source emissions factor models such as MOBILE and EMFAC. FUEL-BASED APPROACH TO EMISSIONS ANALYSIS Another approach to evaluating mobile-source emissions estimates from MOBILE is to compare MOBILE's results with those estimated through a fuel-based approach. The remote-sensing and tunnel-study methods de- scribed above measure exhaust emissions under operating conditions, but, unlike a dynamometer test, they do not measure the emissions on a gram per mile basis. The remote-sensing and tunnel studies measure the emis- sions concentrations of VOCs, CO, and NOx relative to the concentration of the combustion product carbon dioxide (COD. Carbon dioxide is the major carbon-containing product of fuel combustion, and thus the CO2 emission provides a measure of the amount of fuel burned. For improved accuracy, a correction is applied for the carbon in the VOCs and CO emissions. The determination of the concentrations of VOCs, CO, and NOx relative to CO2 provide measurements of these emissions relative to the amount of fuel consumed. These fuel-based emissions factors, measured in grams per gallon (g/gal), vary much less with changes in the vehicle mode of opera- tion than do the travel-based (g/mi) emissions factors (Singer and Harley 1996; Singer et al. 1999~. This might give the fuel-based emissions method an advantage over the travel-based method because the latter might not accurately represent the variations in the driving cycles of urban areas. Because the fuel-based emissions approach is less sensitive to the details of the vehicles' operation (speed and acceleration), this method is less sus- ceptible to inaccuracies derived from the MOBILE model's failure to repre- sent realistic urban vehicle operation.

OCR for page 135
MODEL UNCERTAINTY AND EVALUATION 7 63 TABLE 4-5 Comparison of CMB and PARTS Particulate Emissions from NFRAQS Study Vehicle Category Cold Starts CMB PARTS 32.5% 3/4 Non-Smokers 7.5% 29.0% High PM Emitter 31.3% 3/4 Diesel 28.8% 71.0% - Source: Fujita et al. 1998; Watson et al. 1998. The fuel-based emissions inventory is a concept developed from mea- surements of vehicle emissions on a gram per gallon of fuel. If the emis- sions amounts in terms of g/gal for a representative on-road fleet are known, then the fleet-wide emissions rate using the fuel consumption rate in gallons per unit of time can be calculated. Fuel consumption is deter- mined from fuel sales records. Knowledge of the fleet composition and the fuel economy of the different types of vehicles is also required to estimate the emissions inventory. As described in previous sections, the MOBILE model employs a travel- based method to develop emissions inventories. This requires knowledge of the emissions levels (g/mi) for different modes of driving or for represen- tative driving trips, vehicle activity or use, and the fleet composition. Both the travel-based and fuel-based methods require information on the in-use fleet composition, ambient temperature, and other factors that affect emis- sions and vary with geographical region. Singer and Harley (1996) developed a fuel-based CO inventory for the South Coast Air Basin in California and compared the results to Califor- nia's MVEl7F model. The fuel-based CO inventory estimate was of a fac- tor 2.2 times larger for cars and 2.6 times larger for trucks than the travel- based model. In a second study used more than 60,000 remote-sensing measurements made at 38 sites in Los Angeles between May and October 1997, to develop CO and VOC inventories using the fuel-based method (Singer and Harley 2000~. Their estimates for the on-road, fleet-stabilized exhaust emissions from cars and light- and medium-duty trucks are larger than the California MVEl7G model by factors of 2.4 ~ 0.2 for CO and 3.5 0.6 for VOCs. Similar tests of the MOBILE model emissions inventories should be made to test the model's consistency and accuracy. The fuel- based approach to emissions inventories is a promising method that can be used to reduce the uncertainties in emissions predictions.

OCR for page 135
64 MODEL/NO MOB/[E-SOURCE EMISSIONS SUMMARY OF FINDINGS AND RECOMMENDATIONS In this chapter, quantitative estimates of MOBILE's overall accuracy and uncertainty were not provided. The data are not available to do so with any confidence. Some studies have identified significant discrepancies in MOBILE predictions, but a complete assessment is not possible. Spe- cific findings and recommendations follow. Findings 1. EPA has done very little to use existing independent techniques to test the overall ability of MOBILE to accurately estimate real-world emis- sions of the overall on-road fleet. Others have applied many different tech- niques that test the accuracy of MOBILE. Most studies have found that MOBILES and earlier versions are substantially underpredicting VOC emissions (approximately by a factor of 2) and, to some extent, underpre- dicting CO and the PM contributions of gasoline-powered vehicles to the overall emissions. This is in contrast to NOx emissions, which appear to be more accurately estimated. 2. At present there is an inadequate understanding and quantification of the sources of uncertainties in MOBILE. These uncertainties arise from small and nonrepresentative emissions data, statistical analyses of these data, and assumptions that underlie and define the MOBILE model's aigo- rithms and predictions. Quantification of uncertainties is critical for un- derstanding the weaknesses in the model, and identifying the most critical needs for further emissions test data. 3. A critical unanswered question at the heart of issues related to MO- BILE's uncertainty and evaluation is how accurate the model needs to be to serve its various uses described in Chapter 2. It appears that EPA has not determined a desirable level of accuracy for MOBILE. It is clear that EPA, working with the user community, should determine the level of ac- curacy needed, and plan accordingly. For example, if it was determined that the current version of MOBILE is accurate enough to fulfill all of its roles (which it is not), little further work in that direction is necessary. On the other hand, if a significant improvement in model accuracy is de- manded, then considerable work is suggested, and possibly a new design if the current approach is too limited (as is concluded here). There will not be a single answer, as various applications demand different levels of accu- racy. Designing future emissions models should take required accuracy for different applications into account. 4. EPA has provided only limited sensitivity analyses for earlier ver- sions of MOBILE. Although the primary source of uncertainty in MOBILE

OCR for page 135
M ODEE UNCERTAINTY AND EVALUATION 7 65 output is in the underlying emissions data and data analyses used to gen- erate the model formulations, uncertainty in model inputs can also create uncertainty in model outputs. It is important that air-quality planners understand the sensitivity of model outputs to uncertainty in model inputs such as average speed, ambient temperature, fuel volatility, and I/M pro- gram parameters. 5. A dominant cause of MOBILE's underprediction of real-world emis- sions appears to be related to the driving-cycle testing protocol (because the FTP-based driving cycle is not representative of current driving pat- terns) and associated adjustment factors and default values forming the basis for MOBILE. In the real world, slower speeds, heavier-Ioaded oper- ating conditions associated with more congested stop-and-go driving condi- tions, and a more dominant role of cold starts, aD of which produce en- riched engine-operating conditions, appear to be the key factors in explain- ing the discrepancy of the real-world driving cycle compared with the aver- age driving cycle reflected by MOBILE. This observation indicates the es- sential need to move toward a true modal modeling approach. 6. Other important factors that appear to be significant sources of error in MOBILE are recruitment bias in vehicles that are FTP-tested and small databases for sensitive parameters. 7. Although corrections have been made to many parts of the emerging MOBILES model to improve its accuracy, it appears the end result might deviate even more than past versions of the model with respect to the VOC to NOx ratios. This will create, among other things, greater uncertainty in ozone modeling. Recommendations 1. EPA should assess the levels of accuracy needed to fulfill its regula- tory responsibilities and required for specific applications of the MOBILE model. EPA should compare the needed accuracy to the accuracy of the MOBILE model, and identify specific elements of the model that contrib- ute most to its inaccuracy. EPA should use the results of such an assess- ment to help guide the development of the next generation of models that would have improved accuracy in critical model components. 2. Enhanced model evaluation studies should begin immediately and continue throughout the long-term evolution and development of mobile- source emissions models. These studies need to be conducted to reduce gaps between model-predicted emissions and the resulting air quality, and also to reduce gaps between model-predicted emissions reductions from control programs, such as vehicle I/M programs, and those that actually occur in implementation. The evaluation should include (but not be lim-

OCR for page 135
7 66 M ODELING M OBI1E-SOURCE EMISSIONS ited to) field observations, including tunnel studies; remote-sensing mea- surements; source-receptor modeling; roadside pullovers; and air-quality monitoring and modeling; vehicle emissions testing data from vehicle I/M programs; and other vehicle emissions tests. Evaluation studies should be done with oversight and guidance from an independent body, including technical experts, and should be undertaken in tandem with the sensitiv- ity and uncertainty studies suggested in the next recommendations. 3. Rigorous sensitivity analyses should be performed for all model in- puts and provided as part of user guides for MOBILES and all future ver- sions of MOBILE. From these sensitivity analyses, EPA should provide guidance to transportation and air-quality planners on the most critical model inputs affecting model results. 4. EPA, along with other agencies and industries, should undertake the necessary measures to conduct quantitative uncertainty analyses of the mobile-source emissions models in the modeling toolkit (discussed in Chapter 6), especially the MOBILE model. Future versions of the MOBILE model and other models in the toolkit should be developed to fa- cilitate uncertainty analyses. Results of the uncertainty analyses should be used to guide research plans for obtaining additional test data that would increase the accuracy of the model.