National Academies Press: OpenBook
« Previous: Chapter 3 - Data Collection and Processing
Page 34
Suggested Citation:"Chapter 4 - Before/After Studies." National Academies of Sciences, Engineering, and Medicine. 2008. Cost-Effective Performance Measures for Travel Time Delay, Variation, and Reliability. Washington, DC: The National Academies Press. doi: 10.17226/14167.
×
Page 34
Page 35
Suggested Citation:"Chapter 4 - Before/After Studies." National Academies of Sciences, Engineering, and Medicine. 2008. Cost-Effective Performance Measures for Travel Time Delay, Variation, and Reliability. Washington, DC: The National Academies Press. doi: 10.17226/14167.
×
Page 35
Page 36
Suggested Citation:"Chapter 4 - Before/After Studies." National Academies of Sciences, Engineering, and Medicine. 2008. Cost-Effective Performance Measures for Travel Time Delay, Variation, and Reliability. Washington, DC: The National Academies Press. doi: 10.17226/14167.
×
Page 36

Below is the uncorrected machine-read text of this chapter, intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text of each book. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.

34 4.1 Introduction This chapter provides guidance on the evaluation of the ef- fectiveness of improvements designed to reduce travel time, delay, and variability. Once an improvement is implemented it is generally desirable to assess its effectiveness in meeting stated objectives or delivering promised benefits. This helps the agency understand the effectiveness of the components of its program and to better design new programs in the future. A common method for evaluating the effectiveness of im- provements in the field is the before/after study, which measures system performance before and after implementa- tion of a specific improvement. A variation on the before/after methodology is the use of estimated data to conduct a hypo- thetical “what if” or “with and without project” type of analysis to support planning decisions about future investments. The concept is the same: to isolate the impact of the proposed proj- ect or action, and to apply valid statistical tests to determine whether the change is significant and can in fact be attributed to the project being evaluated. The major task of a before/after study is to distinguish between random results and actual differences between the before and after conditions. The ability to distinguish ac- tual differences from random results hinges on the ability of the analyst to gather a sufficient sample size. Data-rich agencies (those with continuous surveillance technology in place on some of their facilities) will be able to distinguish smaller actual differences from random variations in results simply because of their ability to gather more data. Data-poor agencies will have to go to greater expense to gather the before and after data and will generally be able to distinguish only larger actual differences from random results. This chapter describes a standard statistical method, called hypothesis testing, for determining if the results of the before and after study could have resulted from luck (i.e., random variation in observed results). If the result could have been the result of luck, that does not necessarily mean there is no real difference. It just could mean the analyst was unable to gather enough data to be able to tell the difference between luck and actual effects. The statistical test described in this chapter is limited to de- termining whether the before mean value of the performance measure (e.g., a travel time or delay-based measure) is signif- icantly different from the after mean. The test cannot be used for determining the significance of changes in the variance of travel time or delay, which is necessary to determine whether an improvement in the BI or other similar measure of relia- bility is statistically significant. To determine whether the before and after standard deviations (or variances) are signif- icantly different, the reader should consider applying Lev- ene’s test for equality of variances. Information on Levene’s test and alternate tests on the equality of variances, as well as example applications, can be found in the National Insti- tute of Standards and Technology Engineering Statistics Handbook, Section 1.3.5.10, viewable and downloadable at http://www.itl.nist.gov/div898/handbook/eda/section3/eda3 5a.htm. Barring application of such sophisticated statistical tests, the analyst would rely on professional experience, familiar- ity with the specific situation, and confidence in the data col- lection methods to make a professional judgment whether a change in the BI resulted from the specified capital or oper- ational improvement, as opposed to random variation or luck. Another approach is to track trends in the BI over time and compare to changes in total delay over the same period. Improvements in reliability, as evidenced by a lower BI, can occur irrespective of increases in average total delay. Opera- tional strategies such as freeway service patrols, for example, can contribute to a reduction in the magnitude of the few worst occurrences of delay, and thus have a bigger impact on reliability than on average delay. C H A P T E R 4 Before/After Studies

4.2 Common Pitfalls of Before/After Studies The validity of any conclusions drawn from a before/after study hinges on the validity of the assumption that all other conditions (except for the improvement itself) are identical when both the before and after measurements are made. It is generally impossible to achieve perfect validity in the real world since so many conditions change over time. A common problem for before/after studies is traffic demand grows over time. Ideally the before study is done just before the start of construction of the improvement and the after study is performed as soon after the improvement is completed and opened to traffic. However, it takes time for people to adapt to a new improvement, thus it is not a good idea to gather after data within the first few days or weeks after a project is opened. One must find a compromise point in time when most travel- ers are thought to have adapted to the new project and the least amount of elapsed time since the before study was completed. There also are several potential additional (often unknown) differences between the before and after conditions that can affect the results, usually without the investigator’s knowledge. Examples include changes in gasoline prices, highway im- provements elsewhere in the region, accidents on the day of the study at other facilities in the region, etc. Another common problem of before/after studies is that you may not be surveying all of the travelers impacted by the project in your before and after studies. New travelers may show up on the facility who were not there before it was con- structed. Odds are that the travel times of these new travelers on the facility were not captured in your before study of the facility, unless other routes serving the same trip patterns also were studied. So you may be underestimating the benefits to the public of the improvement if you only consider the net change in travel times on the facility itself. These are weaknesses of any before/after study the analyst must seek to minimize, but can never completely eliminate. Another common pitfall, but one the analyst can avoid, is obtaining insufficient numbers of before measurements. The number of before measurements of travel time or delay, and the variance among the measurements will determine the ulti- mate sensitivity of the before/after test. The analyst should consult Section 3.3 and use the methods there to determine an adequate number of measurements for the before condition. 4.3 Selection of Performance Measures for Before/After Studies The analyst needs to select the set of performance measures that will be used to determine if the improvement has resulted in the desired improvement in system performance. In choos- ing between the use of travel time or delay as a performance measure, the analyst should recognize that travel time incor- porates a component (free-flow travel time) that is often large and relatively insensitive to most facility improvements. This makes travel time a difficult measure to use for the detection of performance improvements, particularly over a large area or multiple segments or facilities. Delay is a much more sensitive measure for detecting performance improvements. However, delay is more volatile, requiring more measurements in order to determine its average within an acceptable confidence interval. So there is an explicit tradeoff in terms of level of confidence one may have in the results of the before/after analysis and the cost or resource requirements of that analysis. 4.4 Determining if Conditions Are Significantly Better It is tempting to measure the mean travel time (or delay) before the improvement and the mean travel time after the improvements and decide that conditions are better based solely on a comparison of the two means. However, since you do not measure travel times every hour of every day of the year, it could have been the result of plain luck, not an actual difference. Statistical hypothesis testing is used to determine if your results could have been due to luck and not the improvement. Hypothesis testing determines if the analyst has performed an adequate number of measurements for the before and after conditions to truly tell if the improvement was effective at the analyst’s desired level of confidence. The test begins with the specification of a null hypothe- sis that you hopefully will be able to reject: “The measured difference in mean travel time for the before and after conditions occurred by random chance. There really is no significant difference in the mean travel time between the before and after conditions.” A statistic is computed for a selected level of confidence, and if the difference between the two means is less than that statistic, then the null hypothesis is accepted and it is concluded that there is insufficient evidence to prove that the after condition is better than the before condition. The analyst can accept this outcome, or alternatively, either make more measure- ments of travel time for each condition (to improve the sensitivity of the test) or relax standards (confidence level) for rejecting the null hypothesis. The specification of the problem is: Null hypothesis: H x y0 0: μ μ− = 35

against where μx = The mean travel time (or mean observation of some other relevant measure, such as delay) for alternative x (before); and μy = The mean for alternative y (after). This is a two-sided t test with the following optimal rejec- tion region for a given alpha (acceptable Type I error). where The absolute value of the difference in the mean re- sults for alternative x (before) and alternative y (after); sp = The pooled standard deviation; t = The Student’s t distribution for a level of confidence of (1 − alpha) and (n + m − 2) degrees of freedom; n = Sample size for alternative x (before); and m = Sample size for alternative y (after). where sp = Pooled standard deviation; sx = Standard deviation of results for alternative x (before); sy = Standard deviation of results for alternative y (after); s n s m s m n p x y2 2 21 1 2 = −( ) + −( ) − −( ) x y− = x y t s n m n m p− > ⋅ +− + −( / );( )1 2 2 1 1 α H x y1 0: μ μ− ≠ n = Sample size for alternative x (before); and m = Sample size for alternative y (after). The probability of mistakenly accepting the null hypothe- sis is alpha (alpha is usually set to 5 percent to get a 95 percent confidence level test). This is Type I error. There also is the chance of mistakenly rejecting the null hy- pothesis. This is called Type II error and it varies with the dif- ference between the sample means, their standard deviation, and the sample size. (Analysts should consult standard statisti- cal textbooks for tables on the Type II errors associated with dif- ferent confidence intervals and sample sizes.) 4.5 What to Do If the Null Hypothesis Cannot Be Rejected If the null hypothesis of no significant difference in the mean results for the before and after conditions cannot be re- jected, the analyst has the following options: 1. Change to a more sensitive performance measure. For ex- ample, delay will tend to be much more sensitive to im- provements than travel time because a large component of travel time is often the free-flow travel time, which is un- affected by most improvements. 2. Increase the number of measurements made for the after condition in the hopes that the variance of the mean will decrease sufficiently to reject the null hypothesis. 3. Reduce the confidence level from 95 percent to a lower level where the before and after conditions are significantly dif- ferent, and report the lower confidence level in the results. 4. Accept results that the improvement did not significantly improve conditions. 36

Next: Chapter 5 - Identification of Deficiencies »
Cost-Effective Performance Measures for Travel Time Delay, Variation, and Reliability Get This Book
×
MyNAP members save 10% online.
Login or Register to save!
Download Free PDF

TRB's National Cooperative Highway Research Program (NCHRP) Report 618: Cost-Effective Performance Measures for Travel Time Delay, Variation, and Reliability explores a framework and methods to predict, measure, and report travel time, delay, and reliability from a customer-oriented perspective.

  1. ×

    Welcome to OpenBook!

    You're looking at OpenBook, NAP.edu's online reading room since 1999. Based on feedback from you, our users, we've made some improvements that make it easier than ever to read thousands of publications on our website.

    Do you want to take a quick tour of the OpenBook's features?

    No Thanks Take a Tour »
  2. ×

    Show this book's table of contents, where you can jump to any chapter by name.

    « Back Next »
  3. ×

    ...or use these buttons to go back to the previous chapter or skip to the next one.

    « Back Next »
  4. ×

    Jump up to the previous page or down to the next one. Also, you can type in a page number and press Enter to go directly to that page in the book.

    « Back Next »
  5. ×

    To search the entire text of this book, type in your search term here and press Enter.

    « Back Next »
  6. ×

    Share a link to this book page on your preferred social network or via email.

    « Back Next »
  7. ×

    View our suggested citation for this chapter.

    « Back Next »
  8. ×

    Ready to take your reading offline? Click here to buy this book in print or download it as a free PDF, if available.

    « Back Next »
Stay Connected!