Click for next page ( 35


The National Academies | 500 Fifth St. N.W. | Washington, D.C. 20001
Copyright © National Academy of Sciences. All rights reserved.
Terms of Use and Privacy Statement



Below are the first 10 and last 10 pages of uncorrected machine-read text (when available) of this chapter, followed by the top 30 algorithmically extracted key phrases from the chapter as a whole.
Intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text on the opening pages of each chapter. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.

Do not use for reproduction, copying, pasting, or reading; exclusively for search engines.

OCR for page 34
34 CHAPTER 4 Before/After Studies 4.1 Introduction variation in observed results). If the result could have been the result of luck, that does not necessarily mean there is no This chapter provides guidance on the evaluation of the ef- real difference. It just could mean the analyst was unable to fectiveness of improvements designed to reduce travel time, gather enough data to be able to tell the difference between delay, and variability. Once an improvement is implemented it luck and actual effects. is generally desirable to assess its effectiveness in meeting stated The statistical test described in this chapter is limited to de- objectives or delivering promised benefits. This helps the termining whether the before mean value of the performance agency understand the effectiveness of the components of its measure (e.g., a travel time or delay-based measure) is signif- program and to better design new programs in the future. icantly different from the after mean. The test cannot be used A common method for evaluating the effectiveness of im- for determining the significance of changes in the variance of provements in the field is the before/after study, which travel time or delay, which is necessary to determine whether measures system performance before and after implementa- an improvement in the BI or other similar measure of relia- tion of a specific improvement. A variation on the before/after bility is statistically significant. To determine whether the methodology is the use of estimated data to conduct a hypo- before and after standard deviations (or variances) are signif- thetical "what if" or "with and without project" type of analysis icantly different, the reader should consider applying Lev- to support planning decisions about future investments. The ene's test for equality of variances. Information on Levene's concept is the same: to isolate the impact of the proposed proj- test and alternate tests on the equality of variances, as well ect or action, and to apply valid statistical tests to determine as example applications, can be found in the National Insti- whether the change is significant and can in fact be attributed tute of Standards and Technology Engineering Statistics to the project being evaluated. Handbook, Section 1.3.5.10, viewable and downloadable at The major task of a before/after study is to distinguish http://www.itl.nist.gov/div898/handbook/eda/section3/eda3 between random results and actual differences between the 5a.htm. before and after conditions. The ability to distinguish ac- Barring application of such sophisticated statistical tests, tual differences from random results hinges on the ability the analyst would rely on professional experience, familiar- of the analyst to gather a sufficient sample size. Data-rich ity with the specific situation, and confidence in the data col- agencies (those with continuous surveillance technology in lection methods to make a professional judgment whether a place on some of their facilities) will be able to distinguish change in the BI resulted from the specified capital or oper- smaller actual differences from random variations in ational improvement, as opposed to random variation or results simply because of their ability to gather more data. luck. Another approach is to track trends in the BI over time Data-poor agencies will have to go to greater expense to and compare to changes in total delay over the same period. gather the before and after data and will generally be able Improvements in reliability, as evidenced by a lower BI, can to distinguish only larger actual differences from random occur irrespective of increases in average total delay. Opera- results. tional strategies such as freeway service patrols, for example, This chapter describes a standard statistical method, called can contribute to a reduction in the magnitude of the few hypothesis testing, for determining if the results of the before worst occurrences of delay, and thus have a bigger impact on and after study could have resulted from luck (i.e., random reliability than on average delay.