National Academies Press: OpenBook

Traffic Forecasting Accuracy Assessment Research (2019)

Chapter: Part I: Guidance Document

« Previous: Front Matter
Page 1
Suggested Citation:"Part I: Guidance Document." National Academies of Sciences, Engineering, and Medicine. 2019. Traffic Forecasting Accuracy Assessment Research. Washington, DC: The National Academies Press. doi: 10.17226/25637.
×
Page 1
Page 2
Suggested Citation:"Part I: Guidance Document." National Academies of Sciences, Engineering, and Medicine. 2019. Traffic Forecasting Accuracy Assessment Research. Washington, DC: The National Academies Press. doi: 10.17226/25637.
×
Page 2
Page 3
Suggested Citation:"Part I: Guidance Document." National Academies of Sciences, Engineering, and Medicine. 2019. Traffic Forecasting Accuracy Assessment Research. Washington, DC: The National Academies Press. doi: 10.17226/25637.
×
Page 3
Page 4
Suggested Citation:"Part I: Guidance Document." National Academies of Sciences, Engineering, and Medicine. 2019. Traffic Forecasting Accuracy Assessment Research. Washington, DC: The National Academies Press. doi: 10.17226/25637.
×
Page 4
Page 5
Suggested Citation:"Part I: Guidance Document." National Academies of Sciences, Engineering, and Medicine. 2019. Traffic Forecasting Accuracy Assessment Research. Washington, DC: The National Academies Press. doi: 10.17226/25637.
×
Page 5
Page 6
Suggested Citation:"Part I: Guidance Document." National Academies of Sciences, Engineering, and Medicine. 2019. Traffic Forecasting Accuracy Assessment Research. Washington, DC: The National Academies Press. doi: 10.17226/25637.
×
Page 6
Page 7
Suggested Citation:"Part I: Guidance Document." National Academies of Sciences, Engineering, and Medicine. 2019. Traffic Forecasting Accuracy Assessment Research. Washington, DC: The National Academies Press. doi: 10.17226/25637.
×
Page 7
Page 8
Suggested Citation:"Part I: Guidance Document." National Academies of Sciences, Engineering, and Medicine. 2019. Traffic Forecasting Accuracy Assessment Research. Washington, DC: The National Academies Press. doi: 10.17226/25637.
×
Page 8
Page 9
Suggested Citation:"Part I: Guidance Document." National Academies of Sciences, Engineering, and Medicine. 2019. Traffic Forecasting Accuracy Assessment Research. Washington, DC: The National Academies Press. doi: 10.17226/25637.
×
Page 9
Page 10
Suggested Citation:"Part I: Guidance Document." National Academies of Sciences, Engineering, and Medicine. 2019. Traffic Forecasting Accuracy Assessment Research. Washington, DC: The National Academies Press. doi: 10.17226/25637.
×
Page 10
Page 11
Suggested Citation:"Part I: Guidance Document." National Academies of Sciences, Engineering, and Medicine. 2019. Traffic Forecasting Accuracy Assessment Research. Washington, DC: The National Academies Press. doi: 10.17226/25637.
×
Page 11
Page 12
Suggested Citation:"Part I: Guidance Document." National Academies of Sciences, Engineering, and Medicine. 2019. Traffic Forecasting Accuracy Assessment Research. Washington, DC: The National Academies Press. doi: 10.17226/25637.
×
Page 12
Page 13
Suggested Citation:"Part I: Guidance Document." National Academies of Sciences, Engineering, and Medicine. 2019. Traffic Forecasting Accuracy Assessment Research. Washington, DC: The National Academies Press. doi: 10.17226/25637.
×
Page 13
Page 14
Suggested Citation:"Part I: Guidance Document." National Academies of Sciences, Engineering, and Medicine. 2019. Traffic Forecasting Accuracy Assessment Research. Washington, DC: The National Academies Press. doi: 10.17226/25637.
×
Page 14
Page 15
Suggested Citation:"Part I: Guidance Document." National Academies of Sciences, Engineering, and Medicine. 2019. Traffic Forecasting Accuracy Assessment Research. Washington, DC: The National Academies Press. doi: 10.17226/25637.
×
Page 15
Page 16
Suggested Citation:"Part I: Guidance Document." National Academies of Sciences, Engineering, and Medicine. 2019. Traffic Forecasting Accuracy Assessment Research. Washington, DC: The National Academies Press. doi: 10.17226/25637.
×
Page 16
Page 17
Suggested Citation:"Part I: Guidance Document." National Academies of Sciences, Engineering, and Medicine. 2019. Traffic Forecasting Accuracy Assessment Research. Washington, DC: The National Academies Press. doi: 10.17226/25637.
×
Page 17
Page 18
Suggested Citation:"Part I: Guidance Document." National Academies of Sciences, Engineering, and Medicine. 2019. Traffic Forecasting Accuracy Assessment Research. Washington, DC: The National Academies Press. doi: 10.17226/25637.
×
Page 18
Page 19
Suggested Citation:"Part I: Guidance Document." National Academies of Sciences, Engineering, and Medicine. 2019. Traffic Forecasting Accuracy Assessment Research. Washington, DC: The National Academies Press. doi: 10.17226/25637.
×
Page 19
Page 20
Suggested Citation:"Part I: Guidance Document." National Academies of Sciences, Engineering, and Medicine. 2019. Traffic Forecasting Accuracy Assessment Research. Washington, DC: The National Academies Press. doi: 10.17226/25637.
×
Page 20
Page 21
Suggested Citation:"Part I: Guidance Document." National Academies of Sciences, Engineering, and Medicine. 2019. Traffic Forecasting Accuracy Assessment Research. Washington, DC: The National Academies Press. doi: 10.17226/25637.
×
Page 21
Page 22
Suggested Citation:"Part I: Guidance Document." National Academies of Sciences, Engineering, and Medicine. 2019. Traffic Forecasting Accuracy Assessment Research. Washington, DC: The National Academies Press. doi: 10.17226/25637.
×
Page 22
Page 23
Suggested Citation:"Part I: Guidance Document." National Academies of Sciences, Engineering, and Medicine. 2019. Traffic Forecasting Accuracy Assessment Research. Washington, DC: The National Academies Press. doi: 10.17226/25637.
×
Page 23
Page 24
Suggested Citation:"Part I: Guidance Document." National Academies of Sciences, Engineering, and Medicine. 2019. Traffic Forecasting Accuracy Assessment Research. Washington, DC: The National Academies Press. doi: 10.17226/25637.
×
Page 24
Page 25
Suggested Citation:"Part I: Guidance Document." National Academies of Sciences, Engineering, and Medicine. 2019. Traffic Forecasting Accuracy Assessment Research. Washington, DC: The National Academies Press. doi: 10.17226/25637.
×
Page 25
Page 26
Suggested Citation:"Part I: Guidance Document." National Academies of Sciences, Engineering, and Medicine. 2019. Traffic Forecasting Accuracy Assessment Research. Washington, DC: The National Academies Press. doi: 10.17226/25637.
×
Page 26
Page 27
Suggested Citation:"Part I: Guidance Document." National Academies of Sciences, Engineering, and Medicine. 2019. Traffic Forecasting Accuracy Assessment Research. Washington, DC: The National Academies Press. doi: 10.17226/25637.
×
Page 27
Page 28
Suggested Citation:"Part I: Guidance Document." National Academies of Sciences, Engineering, and Medicine. 2019. Traffic Forecasting Accuracy Assessment Research. Washington, DC: The National Academies Press. doi: 10.17226/25637.
×
Page 28
Page 29
Suggested Citation:"Part I: Guidance Document." National Academies of Sciences, Engineering, and Medicine. 2019. Traffic Forecasting Accuracy Assessment Research. Washington, DC: The National Academies Press. doi: 10.17226/25637.
×
Page 29
Page 30
Suggested Citation:"Part I: Guidance Document." National Academies of Sciences, Engineering, and Medicine. 2019. Traffic Forecasting Accuracy Assessment Research. Washington, DC: The National Academies Press. doi: 10.17226/25637.
×
Page 30
Page 31
Suggested Citation:"Part I: Guidance Document." National Academies of Sciences, Engineering, and Medicine. 2019. Traffic Forecasting Accuracy Assessment Research. Washington, DC: The National Academies Press. doi: 10.17226/25637.
×
Page 31
Page 32
Suggested Citation:"Part I: Guidance Document." National Academies of Sciences, Engineering, and Medicine. 2019. Traffic Forecasting Accuracy Assessment Research. Washington, DC: The National Academies Press. doi: 10.17226/25637.
×
Page 32
Page 33
Suggested Citation:"Part I: Guidance Document." National Academies of Sciences, Engineering, and Medicine. 2019. Traffic Forecasting Accuracy Assessment Research. Washington, DC: The National Academies Press. doi: 10.17226/25637.
×
Page 33
Page 34
Suggested Citation:"Part I: Guidance Document." National Academies of Sciences, Engineering, and Medicine. 2019. Traffic Forecasting Accuracy Assessment Research. Washington, DC: The National Academies Press. doi: 10.17226/25637.
×
Page 34
Page 35
Suggested Citation:"Part I: Guidance Document." National Academies of Sciences, Engineering, and Medicine. 2019. Traffic Forecasting Accuracy Assessment Research. Washington, DC: The National Academies Press. doi: 10.17226/25637.
×
Page 35
Page 36
Suggested Citation:"Part I: Guidance Document." National Academies of Sciences, Engineering, and Medicine. 2019. Traffic Forecasting Accuracy Assessment Research. Washington, DC: The National Academies Press. doi: 10.17226/25637.
×
Page 36
Page 37
Suggested Citation:"Part I: Guidance Document." National Academies of Sciences, Engineering, and Medicine. 2019. Traffic Forecasting Accuracy Assessment Research. Washington, DC: The National Academies Press. doi: 10.17226/25637.
×
Page 37
Page 38
Suggested Citation:"Part I: Guidance Document." National Academies of Sciences, Engineering, and Medicine. 2019. Traffic Forecasting Accuracy Assessment Research. Washington, DC: The National Academies Press. doi: 10.17226/25637.
×
Page 38
Page 39
Suggested Citation:"Part I: Guidance Document." National Academies of Sciences, Engineering, and Medicine. 2019. Traffic Forecasting Accuracy Assessment Research. Washington, DC: The National Academies Press. doi: 10.17226/25637.
×
Page 39
Page 40
Suggested Citation:"Part I: Guidance Document." National Academies of Sciences, Engineering, and Medicine. 2019. Traffic Forecasting Accuracy Assessment Research. Washington, DC: The National Academies Press. doi: 10.17226/25637.
×
Page 40
Page 41
Suggested Citation:"Part I: Guidance Document." National Academies of Sciences, Engineering, and Medicine. 2019. Traffic Forecasting Accuracy Assessment Research. Washington, DC: The National Academies Press. doi: 10.17226/25637.
×
Page 41
Page 42
Suggested Citation:"Part I: Guidance Document." National Academies of Sciences, Engineering, and Medicine. 2019. Traffic Forecasting Accuracy Assessment Research. Washington, DC: The National Academies Press. doi: 10.17226/25637.
×
Page 42
Page 43
Suggested Citation:"Part I: Guidance Document." National Academies of Sciences, Engineering, and Medicine. 2019. Traffic Forecasting Accuracy Assessment Research. Washington, DC: The National Academies Press. doi: 10.17226/25637.
×
Page 43
Page 44
Suggested Citation:"Part I: Guidance Document." National Academies of Sciences, Engineering, and Medicine. 2019. Traffic Forecasting Accuracy Assessment Research. Washington, DC: The National Academies Press. doi: 10.17226/25637.
×
Page 44
Page 45
Suggested Citation:"Part I: Guidance Document." National Academies of Sciences, Engineering, and Medicine. 2019. Traffic Forecasting Accuracy Assessment Research. Washington, DC: The National Academies Press. doi: 10.17226/25637.
×
Page 45
Page 46
Suggested Citation:"Part I: Guidance Document." National Academies of Sciences, Engineering, and Medicine. 2019. Traffic Forecasting Accuracy Assessment Research. Washington, DC: The National Academies Press. doi: 10.17226/25637.
×
Page 46
Page 47
Suggested Citation:"Part I: Guidance Document." National Academies of Sciences, Engineering, and Medicine. 2019. Traffic Forecasting Accuracy Assessment Research. Washington, DC: The National Academies Press. doi: 10.17226/25637.
×
Page 47
Page 48
Suggested Citation:"Part I: Guidance Document." National Academies of Sciences, Engineering, and Medicine. 2019. Traffic Forecasting Accuracy Assessment Research. Washington, DC: The National Academies Press. doi: 10.17226/25637.
×
Page 48
Page 49
Suggested Citation:"Part I: Guidance Document." National Academies of Sciences, Engineering, and Medicine. 2019. Traffic Forecasting Accuracy Assessment Research. Washington, DC: The National Academies Press. doi: 10.17226/25637.
×
Page 49
Page 50
Suggested Citation:"Part I: Guidance Document." National Academies of Sciences, Engineering, and Medicine. 2019. Traffic Forecasting Accuracy Assessment Research. Washington, DC: The National Academies Press. doi: 10.17226/25637.
×
Page 50
Page 51
Suggested Citation:"Part I: Guidance Document." National Academies of Sciences, Engineering, and Medicine. 2019. Traffic Forecasting Accuracy Assessment Research. Washington, DC: The National Academies Press. doi: 10.17226/25637.
×
Page 51
Page 52
Suggested Citation:"Part I: Guidance Document." National Academies of Sciences, Engineering, and Medicine. 2019. Traffic Forecasting Accuracy Assessment Research. Washington, DC: The National Academies Press. doi: 10.17226/25637.
×
Page 52
Page 53
Suggested Citation:"Part I: Guidance Document." National Academies of Sciences, Engineering, and Medicine. 2019. Traffic Forecasting Accuracy Assessment Research. Washington, DC: The National Academies Press. doi: 10.17226/25637.
×
Page 53
Page 54
Suggested Citation:"Part I: Guidance Document." National Academies of Sciences, Engineering, and Medicine. 2019. Traffic Forecasting Accuracy Assessment Research. Washington, DC: The National Academies Press. doi: 10.17226/25637.
×
Page 54
Page 55
Suggested Citation:"Part I: Guidance Document." National Academies of Sciences, Engineering, and Medicine. 2019. Traffic Forecasting Accuracy Assessment Research. Washington, DC: The National Academies Press. doi: 10.17226/25637.
×
Page 55
Page 56
Suggested Citation:"Part I: Guidance Document." National Academies of Sciences, Engineering, and Medicine. 2019. Traffic Forecasting Accuracy Assessment Research. Washington, DC: The National Academies Press. doi: 10.17226/25637.
×
Page 56
Page 57
Suggested Citation:"Part I: Guidance Document." National Academies of Sciences, Engineering, and Medicine. 2019. Traffic Forecasting Accuracy Assessment Research. Washington, DC: The National Academies Press. doi: 10.17226/25637.
×
Page 57
Page 58
Suggested Citation:"Part I: Guidance Document." National Academies of Sciences, Engineering, and Medicine. 2019. Traffic Forecasting Accuracy Assessment Research. Washington, DC: The National Academies Press. doi: 10.17226/25637.
×
Page 58
Page 59
Suggested Citation:"Part I: Guidance Document." National Academies of Sciences, Engineering, and Medicine. 2019. Traffic Forecasting Accuracy Assessment Research. Washington, DC: The National Academies Press. doi: 10.17226/25637.
×
Page 59
Page 60
Suggested Citation:"Part I: Guidance Document." National Academies of Sciences, Engineering, and Medicine. 2019. Traffic Forecasting Accuracy Assessment Research. Washington, DC: The National Academies Press. doi: 10.17226/25637.
×
Page 60
Page 61
Suggested Citation:"Part I: Guidance Document." National Academies of Sciences, Engineering, and Medicine. 2019. Traffic Forecasting Accuracy Assessment Research. Washington, DC: The National Academies Press. doi: 10.17226/25637.
×
Page 61
Page 62
Suggested Citation:"Part I: Guidance Document." National Academies of Sciences, Engineering, and Medicine. 2019. Traffic Forecasting Accuracy Assessment Research. Washington, DC: The National Academies Press. doi: 10.17226/25637.
×
Page 62
Page 63
Suggested Citation:"Part I: Guidance Document." National Academies of Sciences, Engineering, and Medicine. 2019. Traffic Forecasting Accuracy Assessment Research. Washington, DC: The National Academies Press. doi: 10.17226/25637.
×
Page 63
Page 64
Suggested Citation:"Part I: Guidance Document." National Academies of Sciences, Engineering, and Medicine. 2019. Traffic Forecasting Accuracy Assessment Research. Washington, DC: The National Academies Press. doi: 10.17226/25637.
×
Page 64
Page 65
Suggested Citation:"Part I: Guidance Document." National Academies of Sciences, Engineering, and Medicine. 2019. Traffic Forecasting Accuracy Assessment Research. Washington, DC: The National Academies Press. doi: 10.17226/25637.
×
Page 65
Page 66
Suggested Citation:"Part I: Guidance Document." National Academies of Sciences, Engineering, and Medicine. 2019. Traffic Forecasting Accuracy Assessment Research. Washington, DC: The National Academies Press. doi: 10.17226/25637.
×
Page 66
Page 67
Suggested Citation:"Part I: Guidance Document." National Academies of Sciences, Engineering, and Medicine. 2019. Traffic Forecasting Accuracy Assessment Research. Washington, DC: The National Academies Press. doi: 10.17226/25637.
×
Page 67
Page 68
Suggested Citation:"Part I: Guidance Document." National Academies of Sciences, Engineering, and Medicine. 2019. Traffic Forecasting Accuracy Assessment Research. Washington, DC: The National Academies Press. doi: 10.17226/25637.
×
Page 68
Page 69
Suggested Citation:"Part I: Guidance Document." National Academies of Sciences, Engineering, and Medicine. 2019. Traffic Forecasting Accuracy Assessment Research. Washington, DC: The National Academies Press. doi: 10.17226/25637.
×
Page 69
Page 70
Suggested Citation:"Part I: Guidance Document." National Academies of Sciences, Engineering, and Medicine. 2019. Traffic Forecasting Accuracy Assessment Research. Washington, DC: The National Academies Press. doi: 10.17226/25637.
×
Page 70
Page 71
Suggested Citation:"Part I: Guidance Document." National Academies of Sciences, Engineering, and Medicine. 2019. Traffic Forecasting Accuracy Assessment Research. Washington, DC: The National Academies Press. doi: 10.17226/25637.
×
Page 71
Page 72
Suggested Citation:"Part I: Guidance Document." National Academies of Sciences, Engineering, and Medicine. 2019. Traffic Forecasting Accuracy Assessment Research. Washington, DC: The National Academies Press. doi: 10.17226/25637.
×
Page 72
Page 73
Suggested Citation:"Part I: Guidance Document." National Academies of Sciences, Engineering, and Medicine. 2019. Traffic Forecasting Accuracy Assessment Research. Washington, DC: The National Academies Press. doi: 10.17226/25637.
×
Page 73
Page 74
Suggested Citation:"Part I: Guidance Document." National Academies of Sciences, Engineering, and Medicine. 2019. Traffic Forecasting Accuracy Assessment Research. Washington, DC: The National Academies Press. doi: 10.17226/25637.
×
Page 74
Page 75
Suggested Citation:"Part I: Guidance Document." National Academies of Sciences, Engineering, and Medicine. 2019. Traffic Forecasting Accuracy Assessment Research. Washington, DC: The National Academies Press. doi: 10.17226/25637.
×
Page 75
Page 76
Suggested Citation:"Part I: Guidance Document." National Academies of Sciences, Engineering, and Medicine. 2019. Traffic Forecasting Accuracy Assessment Research. Washington, DC: The National Academies Press. doi: 10.17226/25637.
×
Page 76
Page 77
Suggested Citation:"Part I: Guidance Document." National Academies of Sciences, Engineering, and Medicine. 2019. Traffic Forecasting Accuracy Assessment Research. Washington, DC: The National Academies Press. doi: 10.17226/25637.
×
Page 77
Page 78
Suggested Citation:"Part I: Guidance Document." National Academies of Sciences, Engineering, and Medicine. 2019. Traffic Forecasting Accuracy Assessment Research. Washington, DC: The National Academies Press. doi: 10.17226/25637.
×
Page 78
Page 79
Suggested Citation:"Part I: Guidance Document." National Academies of Sciences, Engineering, and Medicine. 2019. Traffic Forecasting Accuracy Assessment Research. Washington, DC: The National Academies Press. doi: 10.17226/25637.
×
Page 79
Page 80
Suggested Citation:"Part I: Guidance Document." National Academies of Sciences, Engineering, and Medicine. 2019. Traffic Forecasting Accuracy Assessment Research. Washington, DC: The National Academies Press. doi: 10.17226/25637.
×
Page 80
Page 81
Suggested Citation:"Part I: Guidance Document." National Academies of Sciences, Engineering, and Medicine. 2019. Traffic Forecasting Accuracy Assessment Research. Washington, DC: The National Academies Press. doi: 10.17226/25637.
×
Page 81
Page 82
Suggested Citation:"Part I: Guidance Document." National Academies of Sciences, Engineering, and Medicine. 2019. Traffic Forecasting Accuracy Assessment Research. Washington, DC: The National Academies Press. doi: 10.17226/25637.
×
Page 82
Page 83
Suggested Citation:"Part I: Guidance Document." National Academies of Sciences, Engineering, and Medicine. 2019. Traffic Forecasting Accuracy Assessment Research. Washington, DC: The National Academies Press. doi: 10.17226/25637.
×
Page 83
Page 84
Suggested Citation:"Part I: Guidance Document." National Academies of Sciences, Engineering, and Medicine. 2019. Traffic Forecasting Accuracy Assessment Research. Washington, DC: The National Academies Press. doi: 10.17226/25637.
×
Page 84
Page 85
Suggested Citation:"Part I: Guidance Document." National Academies of Sciences, Engineering, and Medicine. 2019. Traffic Forecasting Accuracy Assessment Research. Washington, DC: The National Academies Press. doi: 10.17226/25637.
×
Page 85
Page 86
Suggested Citation:"Part I: Guidance Document." National Academies of Sciences, Engineering, and Medicine. 2019. Traffic Forecasting Accuracy Assessment Research. Washington, DC: The National Academies Press. doi: 10.17226/25637.
×
Page 86
Page 87
Suggested Citation:"Part I: Guidance Document." National Academies of Sciences, Engineering, and Medicine. 2019. Traffic Forecasting Accuracy Assessment Research. Washington, DC: The National Academies Press. doi: 10.17226/25637.
×
Page 87
Page 88
Suggested Citation:"Part I: Guidance Document." National Academies of Sciences, Engineering, and Medicine. 2019. Traffic Forecasting Accuracy Assessment Research. Washington, DC: The National Academies Press. doi: 10.17226/25637.
×
Page 88
Page 89
Suggested Citation:"Part I: Guidance Document." National Academies of Sciences, Engineering, and Medicine. 2019. Traffic Forecasting Accuracy Assessment Research. Washington, DC: The National Academies Press. doi: 10.17226/25637.
×
Page 89
Page 90
Suggested Citation:"Part I: Guidance Document." National Academies of Sciences, Engineering, and Medicine. 2019. Traffic Forecasting Accuracy Assessment Research. Washington, DC: The National Academies Press. doi: 10.17226/25637.
×
Page 90
Page 91
Suggested Citation:"Part I: Guidance Document." National Academies of Sciences, Engineering, and Medicine. 2019. Traffic Forecasting Accuracy Assessment Research. Washington, DC: The National Academies Press. doi: 10.17226/25637.
×
Page 91
Page 92
Suggested Citation:"Part I: Guidance Document." National Academies of Sciences, Engineering, and Medicine. 2019. Traffic Forecasting Accuracy Assessment Research. Washington, DC: The National Academies Press. doi: 10.17226/25637.
×
Page 92
Page 93
Suggested Citation:"Part I: Guidance Document." National Academies of Sciences, Engineering, and Medicine. 2019. Traffic Forecasting Accuracy Assessment Research. Washington, DC: The National Academies Press. doi: 10.17226/25637.
×
Page 93
Page 94
Suggested Citation:"Part I: Guidance Document." National Academies of Sciences, Engineering, and Medicine. 2019. Traffic Forecasting Accuracy Assessment Research. Washington, DC: The National Academies Press. doi: 10.17226/25637.
×
Page 94
Page 95
Suggested Citation:"Part I: Guidance Document." National Academies of Sciences, Engineering, and Medicine. 2019. Traffic Forecasting Accuracy Assessment Research. Washington, DC: The National Academies Press. doi: 10.17226/25637.
×
Page 95
Page 96
Suggested Citation:"Part I: Guidance Document." National Academies of Sciences, Engineering, and Medicine. 2019. Traffic Forecasting Accuracy Assessment Research. Washington, DC: The National Academies Press. doi: 10.17226/25637.
×
Page 96
Page 97
Suggested Citation:"Part I: Guidance Document." National Academies of Sciences, Engineering, and Medicine. 2019. Traffic Forecasting Accuracy Assessment Research. Washington, DC: The National Academies Press. doi: 10.17226/25637.
×
Page 97
Page 98
Suggested Citation:"Part I: Guidance Document." National Academies of Sciences, Engineering, and Medicine. 2019. Traffic Forecasting Accuracy Assessment Research. Washington, DC: The National Academies Press. doi: 10.17226/25637.
×
Page 98
Page 99
Suggested Citation:"Part I: Guidance Document." National Academies of Sciences, Engineering, and Medicine. 2019. Traffic Forecasting Accuracy Assessment Research. Washington, DC: The National Academies Press. doi: 10.17226/25637.
×
Page 99
Page 100
Suggested Citation:"Part I: Guidance Document." National Academies of Sciences, Engineering, and Medicine. 2019. Traffic Forecasting Accuracy Assessment Research. Washington, DC: The National Academies Press. doi: 10.17226/25637.
×
Page 100
Page 101
Suggested Citation:"Part I: Guidance Document." National Academies of Sciences, Engineering, and Medicine. 2019. Traffic Forecasting Accuracy Assessment Research. Washington, DC: The National Academies Press. doi: 10.17226/25637.
×
Page 101
Page 102
Suggested Citation:"Part I: Guidance Document." National Academies of Sciences, Engineering, and Medicine. 2019. Traffic Forecasting Accuracy Assessment Research. Washington, DC: The National Academies Press. doi: 10.17226/25637.
×
Page 102
Page 103
Suggested Citation:"Part I: Guidance Document." National Academies of Sciences, Engineering, and Medicine. 2019. Traffic Forecasting Accuracy Assessment Research. Washington, DC: The National Academies Press. doi: 10.17226/25637.
×
Page 103
Page 104
Suggested Citation:"Part I: Guidance Document." National Academies of Sciences, Engineering, and Medicine. 2019. Traffic Forecasting Accuracy Assessment Research. Washington, DC: The National Academies Press. doi: 10.17226/25637.
×
Page 104
Page 105
Suggested Citation:"Part I: Guidance Document." National Academies of Sciences, Engineering, and Medicine. 2019. Traffic Forecasting Accuracy Assessment Research. Washington, DC: The National Academies Press. doi: 10.17226/25637.
×
Page 105
Page 106
Suggested Citation:"Part I: Guidance Document." National Academies of Sciences, Engineering, and Medicine. 2019. Traffic Forecasting Accuracy Assessment Research. Washington, DC: The National Academies Press. doi: 10.17226/25637.
×
Page 106
Page 107
Suggested Citation:"Part I: Guidance Document." National Academies of Sciences, Engineering, and Medicine. 2019. Traffic Forecasting Accuracy Assessment Research. Washington, DC: The National Academies Press. doi: 10.17226/25637.
×
Page 107
Page 108
Suggested Citation:"Part I: Guidance Document." National Academies of Sciences, Engineering, and Medicine. 2019. Traffic Forecasting Accuracy Assessment Research. Washington, DC: The National Academies Press. doi: 10.17226/25637.
×
Page 108
Page 109
Suggested Citation:"Part I: Guidance Document." National Academies of Sciences, Engineering, and Medicine. 2019. Traffic Forecasting Accuracy Assessment Research. Washington, DC: The National Academies Press. doi: 10.17226/25637.
×
Page 109
Page 110
Suggested Citation:"Part I: Guidance Document." National Academies of Sciences, Engineering, and Medicine. 2019. Traffic Forecasting Accuracy Assessment Research. Washington, DC: The National Academies Press. doi: 10.17226/25637.
×
Page 110
Page 111
Suggested Citation:"Part I: Guidance Document." National Academies of Sciences, Engineering, and Medicine. 2019. Traffic Forecasting Accuracy Assessment Research. Washington, DC: The National Academies Press. doi: 10.17226/25637.
×
Page 111
Page 112
Suggested Citation:"Part I: Guidance Document." National Academies of Sciences, Engineering, and Medicine. 2019. Traffic Forecasting Accuracy Assessment Research. Washington, DC: The National Academies Press. doi: 10.17226/25637.
×
Page 112
Page 113
Suggested Citation:"Part I: Guidance Document." National Academies of Sciences, Engineering, and Medicine. 2019. Traffic Forecasting Accuracy Assessment Research. Washington, DC: The National Academies Press. doi: 10.17226/25637.
×
Page 113
Page 114
Suggested Citation:"Part I: Guidance Document." National Academies of Sciences, Engineering, and Medicine. 2019. Traffic Forecasting Accuracy Assessment Research. Washington, DC: The National Academies Press. doi: 10.17226/25637.
×
Page 114
Page 115
Suggested Citation:"Part I: Guidance Document." National Academies of Sciences, Engineering, and Medicine. 2019. Traffic Forecasting Accuracy Assessment Research. Washington, DC: The National Academies Press. doi: 10.17226/25637.
×
Page 115

Below is the uncorrected machine-read text of this chapter, intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text of each book. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.

NCHRP Research Report 934 Traffic Forecasting Accuracy Assessment Research Part I: Guidance Document I-1

Traffic Forecasting Accuracy Assessment Research Guidance Document I-2 Part I Contents Sumary ....................................................................................................................................................... I-1 Introduction ............................................................................................................................ I-20  Purpose of the Guidance Document ............................................................................................... I-20  Notes on Forecast Accuracy, Reliability and Utility .............................................................. I-21  Research Summary ........................................................................................................................ I-23  Background ............................................................................................................................. I-23  Research Approach ................................................................................................................. I-24  Large N Analysis: Data and Methodology .............................................................................. I-26  Large N Analysis: Results ....................................................................................................... I-30  Deep Dives: Objectives, Data, Cases and Methodologies ...................................................... I-33  Deep Dives: Results and Interpretations ................................................................................. I-35  Conclusions and Recommendations .............................................................................................. I-35  Lessons Learned ...................................................................................................................... I-35  Summary of Recommendations .............................................................................................. I-37  Reasons to Implement These Recommendations .................................................................... I-41  Using Measured Accuracy to Communicate Uncertainty ...................................................... I-45  Quantifying Uncertainty ................................................................................................................ I-45  Introduction to Quantile Regression .............................................................................................. I-47  Default versus Local Quantile Regression ..................................................................................... I-48  Applying Quantile Regression Methods ........................................................................................ I-49  Archiving Traffic Forecasts and Associated Data .................................................................. I-54  Archiving Levels ............................................................................................................................ I-54  Bronze Archiving Level .......................................................................................................... I-55  Silver Archiving Level ............................................................................................................ I-56  Gold Archiving Level ............................................................................................................. I-57  Forecast Archive and Information System ..................................................................................... I-57  Development and Design Features ......................................................................................... I-57  Data Specification ................................................................................................................... I-62  Data Storage ............................................................................................................................ I-65  Reporting Accuracy Results ................................................................................................... I-68 

Traffic Forecasting Accuracy Assessment Research Guidance Document I-3 Segment and Project Level Observations ...................................................................................... I-68  Summary Report ............................................................................................................................ I-69  Updating Quantile Regression Models .......................................................................................... I-72  Using the Forecast Archive and Information System .................................................................... I-73  Deep Dives ..................................................................................................................................... I-74  Improving Traffic Forecasting Methods ................................................................................ I-77  Using Deep Dives to Guide Model Improvements ........................................................................ I-77  Project-Level Testing and Validation ............................................................................................ I-78  Large-N Analysis for Method Selection ........................................................................................ I-79  Implementation and Future Research ..................................................................................... I-82  References ................................................................................................................................................ I-84  Appendix A: Electronic Resources .......................................................................................................... I-87  Appendix B: Traffic Forecast Preservation Annotated Outline (Silver Standard) ................................... I-89  Appendix C: Deep Dive Annotated Outline ............................................................................................ I-93  Appendix D: Forecast Card Data Assumptions ..................................................................................... I-103  Appendix E: Implementation Plan ......................................................................................................... I-110 

Traffic Forecasting Accuracy Assessment Research Guidance Document I-4 List of Figures Figure 1: Distribution of Percent Difference from Forecast (Project Level) .............................................. 30  Figure 2: Percent Difference from Forecast as a function of Forecast Volume (Project Level) ................. 31  Figure 3: Expected Ranges of Actual Traffic (Base Model) ....................................................................... 32  Figure 4: Chart of Sample Forecasting Accuracy Data .............................................................................. 46  Figure 5: Example of Quantile Regression Results .................................................................................... 47  Figure 6: Demonstration #1 Quantile Regression Lines ............................................................................. 52  Figure 7: Demonstration #2 Quantile Regression Lines ............................................................................. 53  Figure 8: Forecast Card Data Schema ......................................................................................................... 64  Figure 9: Forecast Card Illustration for an Example Project ...................................................................... 65  Figure 10: Distribution of Percent Difference from Forecast ..................................................................... 70  Figure 11: Distribution of Percent Difference from Forecast by Functional Class (Segment Level Analysis) ..................................................................................................................................................... 71  Figure 12: Example of Adjusting Forecasts using Elasticity Computations ............................................... 76  Figure 13: Example Pre- and Post-Opening Traffic Forecast Comparison (Source: Atkins 2017) ............ 83  List of Tables Table 1: Actual Volume Estimates Given 40,000 ADT Forecast (Example) ............................................. 49  Table 2: Variables Used in Study Team Quantile Regression Analysis ..................................................... 50  Table 3: Example of Quantile Regression Variables and Coefficients ....................................................... 51  Table 4: Demonstration #1 Applying Quantile Regression ........................................................................ 51  Table 5: Demonstration #2 Applying Quantile Regression ........................................................................ 52  Table 6: Common Data Fields in Forecast Accuracy Sources .................................................................... 59  Table 7: Forecast Inaccuracy by Functional Class (Segment Level Analysis) ........................................... 71 

Traffic Forecasting Accuracy Assessment Research Summary I-5 Summary 1. Introduction Traffic forecasts are projections of future traffic conditions on existing or proposed roadways. Traffic forecasts are used to inform important decisions about transportation projects, including the selection of which projects to build and certain design elements of those projects. It is in the public interest to ensure that those decisions are based on the most accurate and objective possible forecasts. However, we must also recognize that forecasts will always be influenced by some factors that are unexpected, unpredictable, and difficult to anticipate. Therefore, it is prudent to quantify the expected inaccuracy of traffic forecasts and consider that uncertainty in making decisions. Together, more accurate traffic forecasts and a better understanding of the uncertainty around traffic forecasts can lead to a more efficient allocation of resources and build public confidence in the agencies that produce those forecasts. While forecast accuracy has long been a topic of interest to scholars and to critics of transportation planners and policy makers, the evidence on forecast accuracy remains limited. Those scholars and critics have offered possible reasons for forecast inaccuracy, including poor data on which forecasts are based, incorrect assumptions about future conditions, limitations of the forecasting methods used, and political motivations that sometimes that cause people to distort forecasts intentionally. Much of the existing work considers toll roads, rail transit, or mega-projects, and often that work is more speculative about possible causes of inaccuracy than definitive about actual causes. Only a small set of empirical studies examine non-toll traffic forecast accuracy in the United States. According to David Hartgen (2013), “the greatest knowledge gap in US travel demand modeling is the unknown accuracy of US urban road traffic forecasts”. There are reasons that there have been few studies of the topic. Assembling data to study forecast accuracy can be cumbersome. It involves sorting through forecast documents and files created years earlier, and requires collecting data on the project after (and ideally before) it opens. In their review of forecast accuracy studies, Nicolaisen and Driscoll (2014) state, “The lack of availability for necessary data items is a general problem and probably the biggest limitation to advances in the field.” “The greatest knowledge gap in US travel demand modeling is the unknown accuracy of US urban road traffic forecasts.” - David Hartgen, 2013

Traffic Forecasting Accuracy Assessment Research Summary I-6 This research aims to fill that gap, focusing specifically on project-level traffic forecasts of public roads in the US. It assembles the largest known database of traffic forecast accuracy. It reports empirical evidence on the accuracy of these forecasts and factors related to accuracy. It goes on to consider a series of case studies aimed at providing a better understanding of the sources of forecast inaccuracy. Together, these provide empirical evidence about the accuracy of past traffic forecasts. Most past studies have been analytical rather than prescriptive. They report what was observed but offer little advice to planners, engineers, or policy makers as to how they may improve forecasting practice. In contrast, this study makes specific recommendations by which state departments of transportation and others, like metropolitan planning organizations, can improve the accuracy of traffic forecasts going forward. Other fields have demonstrated the effectiveness of reviews that have led to the adoption of improved forecasting practice, including the National Oceanic and Atmospheric Administration (NOAA) which adopted a highly successful Hurricane Forecasting Improvement Program. While attentive to insights and explanations in the assessment of past forecasts, this study emphasizes improving practice. It addresses what agencies can do to improve the accuracy of traffic forecasts, and how to consider the uncertainty inherent in forecasts when making decisions about the transportation system. 2. Research approach A review of the literature revealed that most prior studies of traffic forecasting accuracy had adopted either of two types of studies to assess the accuracy of the forecasts that they studied. The two approaches are complementary, and we employed both in this study. The first approach relies on gathering a large sample of forecasts for which data were collected and the forecasts were made sufficiently long ago that the horizon year of the forecasts has come. This makes it possible to compare the forecasts of traffic with measured traffic flows on the facilities for which the forecasts were made. With a large sample of such forecasts, we use statistical analysis to examine correlations between forecast accuracy and data inputs, facility types, methods used to conduct the forecasts, and factors exogenous to the forecasts that influenced their accuracy. This analysis was based on a large sample of cases and will be referred to throughout this report as the “large N analysis.” The large N analysis required compiling a database of forecast and counted traffic for 1,291 projects from six states and four European countries. These projects are composed of 3,912 individual segments. Segments include sections of roadway between major intersections, opposing directions of a freeway, or ramps in an interchange. The database includes forecast and actual traffic volumes, as well as information such as the type of improvement, the facility type, the forecast method, and the project location. It is the largest known database for assessing traffic forecast accuracy, and allowed development of distributions of forecast errors, analysis of relationships between measured traffic volumes and the forecast traffic volumes and a variety of potentially descriptive variables. “The lack of availability for necessary data items is a general problem and probably the biggest limitation to advances in the field.” - Nicolaisen and Driscoll, 2014

Traffic Forecasting Accuracy Assessment Research Summary I-7 The second type of study identified in the literature review consisted of case studies of particular facilities in which forecasts were made at some date in the past, the projects were planned in detail and built, and resulting traffic flows were observed. Most were case studies of a single project or of a small number of projects, using customized data collection including review of historical documents, before and after surveys of travelers, and interviews of those who participated in project decision making. For example, the Federal Transit Administration conducts such before and after analyses of patronage and cost forecasts for major capital investments in public transit. The depth of the analysis may lead researchers to identify sources of forecast errors—such as errors in inputs, incorrect assumptions, model specification, and changes in the project definition. It is, however, difficult to generalize from particular case studies. This study included six case studies and they are referred to throughout this report as “deep dives”. Large N studies and deep dives complement one another by shining different lights on the same problem. The former addresses the question, “How accurate are traffic forecasts?” while the latter addresses the question, “What are the sources of forecast error?” A third question addressed in this research is, “How can we improve forecasting practice?” For that, we considered the lessons learned from this research and reviewed those lessons with traffic forecasting practitioners. Figure 1 shows how these three questions relate to the methods and outputs of this research. Figure 1: Research questions and methods Question: How accurate are traffic forecasts? • Method: Statistical analysis of actual vs forecast traffic for a large sample of projects after they open, known as "large N analysis". • Output: Distribution of expected traffic volume as a function of forecast volume. Question: What are the sources of forecast error? • Method: "Deep dives" into forecasts of six substantial projects after they open. • Output: Estimated effect of known errors, and remaining unknown error. Question: How can we improve forecasting practice? • Method: Derive lessons from this research and review with practitioners. • Output: Recommendations for how to learn from past traffic forecasts.

Traffic Forecasting Accuracy Assessment Research Summary I-8 This research is intended to help agencies like state DOTs and MPOs, improve their future forecasts. We focus not only on analysis of past projects, but also on establishing a process of continual improvement. The proposed process is informed by the lessons learned conducting the empirical analysis and recognizes the capacity of and challenges facing organizations that collect the data, calibrate and operate the models, and report findings in highly politicized environments. To be sure that the recommendations are useful in practice, the team made every effort to learn from the agencies that had produced the forecasts, and drew from our own experience as practicing traffic forecasters. Whenever possible, the team tried to replicate what agency forecasters had done, including running the travel demand models when possible. We conducted a workshop with practitioners from state DOTs and MPOs to present findings. The resulting recommendations focus on the process of collecting data about forecasts, learning from comparisons to actual outcomes, and using the insight gained both to improve future forecasts and to understand the uncertainty around those forecasts. 3. How accurate are traffic forecasts? The most important question to decision makers about forecast accuracy may be: Given a forecast, what range of likely outcomes should we expect? The large N analysis aimed to answer that question by comparing traffic forecasts made at the time projects were planned with flows measured after the projects were completed. The comparisons in this analysis are for Average Daily Traffic (ADT) in the first post-opening year for which traffic counts are available. We report the accuracy of a project as the percentage difference from forecast (PDFF) for project i: PDFFi= Counted - Forecast VolumeForecast Volume *100% Negative values indicate outcomes lower than forecast, and positive values indicate outcomes higher than forecast. This formulation is appealing because it expresses the error as a function of the forecast, which always is known earlier than traffic counts. The distribution of the percent difference from forecast when measured this way over the dataset was used to portray the systematic performance of traffic forecasts. On average, the counted traffic volume is about 6% lower than forecast, showing some bias. This can be observed in Figure 2, in which the distribution is heavier on the negative side, with a mean of -5.7% and a median of - 7.5%. The mean of the absolute percent difference from forecast, which is a measure of the spread, is 17.3%. Nine in 10 project outcomes are within the range of -38% of the forecast to +37% of the forecast. Bias: On average, the counted traffic volume is about 6% lower than forecast. Spread: On average, the counted traffic is about 17% different from forecast.

Traffic Forecasting Accuracy Assessment Research Summary I-9 Figure 2: Distribution of Percent Difference from Forecast (Project Level) Figure 3 presents the percent difference from forecast as a function of forecasted volumes, and shows that percentage errors decrease as traffic volumes increase. Figure 3: Percent Difference from Forecast as a function of Forecast Volume (Project Level) 𝑃𝐷𝐹𝐹 𝐴𝑐𝑡𝑢𝑎𝑙 ForecastForecast ∗ 100 𝑃𝐷𝐹𝐹 𝐴𝑐𝑡𝑢𝑎𝑙 ForecastForecast ∗ 100 Traffic forecasts are more accurate, in percentage terms, for higher volume roads.

Traffic Forecasting Accuracy Assessment Research Summary I-10 Quantile regression was used to explore the uncertainties inherent in forecasting traffic. Quantile regression is similar to standard regression, but rather than estimating a line of best fit through the center of a cloud of points, it estimates the lines along the edges, corresponding to specific percentiles. Using the data for projects included in this study, we developed several quantile regression models of the actual traffic as a function of the forecast traffic for the 5th, 20th, 50th (median), 80th and 95th percentiles. Figure 4 shows the simplest of these models, with the percentiles representing the uncertainty in outcomes expected around a forecast. Additional models were estimated in this research to test the effects of various descriptive variables. Figure 4: Expected Ranges of Actual Traffic (Base Model) Plots such as Figure 4 provide a means of estimating the range of uncertainty around a forecast at the time the forecast is made. The lines in the graph depict various percentile values and can be interpreted as the range of measured traffic over a forecast volume. For example, it can be expected that 95% of all projects with the forecasted ADT of 30,000 will have counted traffic below 46,578. Only 5% of the projects will experience traffic less than 17,898. Not considering other variables, this range (45,578 to 17,898 for forecast volume of 30,000) includes 90% of the projects. A number of observations emerge from the Large-N analysis. These observations and the supporting data are described in the accompanying technical report and are summarized here: Perfect Forecast 5th Percentile Median 95th Percentile 20th Percentile 80th … 0 10000 20000 30000 40000 50000 60000 0 10000 20000 30000 40000 50000 60000 Ex pe ct ed  A DT Forecast ADT The quantile regression models presented in this research provide a means of estimating the range of uncertainty around a forecast.

Traffic Forecasting Accuracy Assessment Research Summary I-11 1. Traffic forecasts show a modest bias, with measured ADT about 6% lower than forecast ADT. The mean percent difference from forecast is -5.6% at a project level and the median is -7.5%. The difference between the mean and median values occurs because the distribution is asymmetric—counted traffic is more likely to be lower than forecast values, but there is a long right-hand tail of the distribution indicating that a small number of projects experienced traffic much higher than forecast. 2. Traffic forecasts show a significant spread, with a mean absolute percent difference from forecast of 25% at the segment level and 17% at a project level. Some 90% of segment forecasts fall within the range -45% to +66%, and 90% of project level forecasts fall within the range of -38% to +37%. 3. Traffic forecasts are more accurate for higher volume roads. This can be observed in Table 2. This result echoes the maximum desirable deviation guidance from NCHRP 765: Analytical Travel Forecasting Approaches for Project-Level Planning and Design, where there are tighter targets for calibrating a travel model for higher volume links. 4. Traffic forecasts are more accurate for higher functional classes, over and above the volume effect described above. Our quantile regression results show narrower forecast windows for freeways than for arterials, and for arterials than for collectors and locals. The counted volumes on lower-class roads are more likely to be lower than the forecasts. These results may be due to limitations of zone size and network detail. 5. The unemployment rate in the opening year is an important determinant of forecast accuracy. Traffic occurs where there is economic activity and unemployment rates reflect that. For each percentage point of increase (such as from 5% to 6% ) in the unemployment rate in the project’s opening year, the median estimated traffic decreases by 3%. For example, consider two roads, each with the same forecast, but one scheduled to open in 2005 with an unemployment rate of 4.5% and one scheduled to open in 2010 with an unemployment rate of 9.5%. We would expect the opening year ADT to be 15% lower for the project that opens in 2010 ((9.5-4.5) * 0.03). 6. Forecasts implicitly assume that economic conditions present in the year the forecast is made will perpetuate. A high unemployment rate in the year the forecast is produced is more likely to result in ADT in the horizon year that is higher than the forecast, while a low unemployment rate in the year the forecast is produced would have the opposite effect. 7. Traffic forecasts become less accurate as the forecast horizon increases, but the result is asymmetric, with actual ADT more likely to be higher than forecast as the forecast horizon increases. The forecast horizon is the length of time into the future for which forecasts are prepared, which we measure as the number of years between when the forecast is made, and the project opens. The quantile regression results show that the median, 80th percentile and 95th percentile estimates increase with an increase in this variable, but that the 5th and 20th percentile estimates either stay flat or increase by a smaller amount. 8. Regional travel models produce more accurate forecasts than traffic count trends. The mean absolute percent difference from forecast for regional travel models is 16.9% compared to 22.2% for traffic count trends. In addition, the quantile regression models show that using a travel model narrows the uncertainty window. Regional travel models produce more accurate forecasts than traffic count trends.

Traffic Forecasting Accuracy Assessment Research Summary I-12 9. Some agencies have made more accurate forecasts than others. The best agencies (with more than a handful of projects) have a mean absolute percentage difference from forecast (MAPDFF) of 13.7%, compared to 32% for the worst. A portion of these differences are significant in the quantile regression models. 10. Traffic forecasts have improved over time. This can be observed both in our assessment of the year the forecast was produced and in the opening year. Forecasts for projects that opened in the 1990s were especially poor, exhibiting mean volumes 15% higher than forecast, with a MAPDFF of 28.1%. The quantile regression models for forecasting show that while older forecasts do not show a significant bias relative to newer forecasts, they do have a broader uncertainty window, although this result may be confounded by types of projects recorded since 2000 in the database, which tend to be more routine. 11. We find that 95% of forecasts reviewed are “accurate to within half of a lane”. We find that for 1% of cases, the actual traffic is higher than forecast and additional lanes would be needed to maintain the forecast level-of-service. Conversely, for 4% of cases, actual traffic is lower than forecast, and the same level-of-service could be maintained with fewer lanes. While all these observations are based on a large sample of projects, it is not a random sample of all highway projects. This limits our ability to generalize from this analysis. The years in which projects in the database opened to traffic range from 1970 to 2017, with about 90% of the projects opening to traffic in 2003 or later. Earlier projects were more likely to be major infrastructure capital investment projects and more recent ones were more often routine resurfacing projects on existing roadways. If a forecaster has an interest in a specific type of project, there is value in repeating this analysis using a sample of projects more similar to the type for which the forecast is to be made. The accompanying Guidance Document describes how to do this for an agency’s own forecasts, and the data used in this research are being made publicly available to support future research, as described in the Guidance Document. 4. What are the sources of forecast error? The statistical analysis described above provided useful indications of the magnitudes of errors and suggested factors predictive of more or less accurate forecasts, but it is limited in its ability to determine the causes of forecast error. To better understand these causes, we conducted six case studies, or “deep dives” of traffic forecasts in different states for highways having rather different contexts and forecast results. The sample of projects chosen for deep dives is listed in Table 1. The cases included a new bridge, the expansion and extension of an arterial on the fringe of an urban area, a major new expressway built as a toll road, the rebuilding and expansion of an urban freeway, and a state highway bypass around a small town. We find that 95% of forecasts reviewed are “accurate to within half of a lane”.

Traffic Forecasting Accuracy Assessment Research Summary I-13 Table 1: Projects selected Project Name Brief Description Eastown Road Extension Project, Lima, Ohio Widened a 2.5-mile segment of the arterial from 2 lanes to 5 lanes and extended the arterial an additional mile Indian River Bridge, Palm City, Florida This 0.6 mile long bridge with four travel lanes in total. runs along CR 714 (Martin Highway), connecting with the Indian River Street and goes across the St. Lucie River. Central Artery Tunnel, Boston, Massachusetts Reconstruction of Interstate Highway 93 (I-93) in downtown Boston, the extension of I-90 to Logan International Airport, the construction of two new bridges over the Charles River, six interchanges and the Rose Kennedy Greenway in the space vacated by the previous elevated I-93 Central Artery in Boston, Massachusetts. Cynthiana Bypass, Cynthiana, Kentucky A 2-lane, state highway bypass project, to the west of the City from the southern terminus where US 62S and US27S meet. South Bay Expressway, San Diego, California A 9.2-mile tolled highway segment of SR 125 in eastern San Diego, CA. This project was funded as a Public Private Partnership (P3). It opened in 2007 and the P3 filed for bankruptcy in 2010. US 41 (later renamed as I-41), Brown County, Wisconsin A project of capacity addition, reconstruction of nine interchanges, constructing 24 roundabouts, adding collector- distributer lanes, and building two system interchanges located in Brown County, Wisconsin. For each deep dive, the project team identified factors that could contribute to forecast inaccuracy and quantified the contribution of each where it was possible to do so. For example, population and regional employment forecasts were used to inform forecasts of traffic growth, and the population had not grown as had been forecast, or an economic downturn had caused shortfalls in expected job growth. With the benefit of hindsight, we now know the population and employment in the project opening year, and were able to calculate how the forecast traffic volume would change if it were based instead on the true population and employment values. In some cases, the project team had access to the data and models that had been used to create the original traffic forecasts, and we were able to re-run those models with corrected inputs. In other cases, we relied on published elasticities to adjust the traffic volumes forecasts. Table 2 shows a summary of that analysis for the five projects for which it could be conducted. The first column shows the original PDFF, and the middle column shows the remaining PDFF after adjusting for the factor listed and all factors above. The analysis was limited based on the available data, models and documentation. For example, if a forecast report did not document fuel price assumptions, it was not possible to adjust for errors in fuel price assumptions. The last column shows the remaining PDFF after all adjustments and represents the error that remains for unknown reasons, which could include limitations in the forecast method, inaccurate assumptions that we could not test, or other factors.

Traffic Forecasting Accuracy Assessment Research Summary I-14 Table 2: Known sources of forecast inaccuracy for deep dives Project Original Percent Difference from Forecast Remaining percent difference from forecast  after adjusting for errors in: Remaining percent  difference from  forecast after all  adjustments Eastown Road Extension Project, Lima, Ohio -43% Employment -39% -28% Population/Household -38% Car Ownership -37% Fuel Price/Efficiency -34% Travel Time/Speed -28% Indian River Bridge, Palm City, Florida -60% Employment -59% -56% Population -61% Fuel Price -56% Central Artery Tunnel, Boston, Massachusetts -16% Employment -10% -10% Population -14% Fuel Price -10% Cynthiana Bypass, Cynthiana, Kentucky -27% Employment -25% -8% Population -25% External Trips Only 7% US 41 (later renamed as I-41), Brown County, Wisconsin -5% Population -4% -6% Fuel Price -6% Southbay Expressway, San Diego, California Revenue less than projected, leading to bankruptcy of P3. Socio Economic Growth The available documentation did not allow the effect of these factors on traffic volume to be quantified. Border crossing Toll rates Several observations can be made based on these deep dives: 1. The reasons for forecast inaccuracy are diverse. While the above points list some of the factors that contribute to forecast inaccuracy, it is clear from our limited sample that the reasons for inaccuracies are diverse—external forecasts, travel speeds, population and employment forecasts, and short-term variations from a long-term trend have all be identified as contributing factors in one or more of the Deep Dives. 2. The forecasts for all six projects considered show “optimism bias”. For each project, the observed traffic is less than forecast, and for all except US 41, correcting for the factors listed reduces the difference between the forecast and observed traffic. 3. Employment, population and fuel price forecasts frequently contribute to forecast inaccuracy: Adjustments to the forecasts using elasticities and model re-runs confirmed that significant errors in opening year forecasts employment, fuel price and travel speed had a major role in the over-estimation of traffic volumes. In addition, we observe that macro- economic conditions in the opening year influence forecast accuracy, particularly for projects which opened during or after an economic downturn.

Traffic Forecasting Accuracy Assessment Research Summary I-15 4. External traffic and travel speed assumptions also affect traffic forecasts: For the Bypass extension project in Cynthiana, the estimated growth rate for external trips was the largest source of forecast error. Travel speed was an important factor for the Eastown Road Extension because inaccurate speeds led to too much diversion from competing roads. Because of the limited number of projects examined and the diversity of reasons for forecast inaccuracy, it may be difficult to generalize our findings, and it makes it hard to identify a simple way of improving forecasts. We cannot determine why any optimism bias occurs, nor can we determine why it remains after adjusting for known errors—the best we can do from these case studies is to observe its presence. Nonetheless, the findings can help forecasters anticipate the likely sources of problems, such as through traffic for a bypass project (Cynthiana Bypass), traffic diverted from a parallel facility (Eastown Road Extension), or cross-border traffic (Southbay Expressway)—and give extra scrutiny in forecasting to those issues. Those cases that had access to the original traffic forecasting model runs and data were the most insightful, and provide a future opportunity to study the effects of forecasting methods and data in greater depth. 5. How can we improve traffic forecasting practice? One of the most important and overarching conclusions of this study is that agencies should take far more seriously the analysis of their past forecasting errors so that they can learn from the cumulative record. Forecasts are essential elements in the creation of effective highway plans and project designs, and because forecasts are always subject to error, agencies should document their forecasts and revisit them in order to identify assumptions and other factors that lead to errors. As illustrated by our quantile regression of the data in our large N analysis, systematically tracking forecast accuracy provides insight into the range of uncertainty surrounding traffic forecasts. Especially for complex and expensive capital investment projects, the most efficacious type of forecasting could well involve the development of ranges of future traffic. Instead of dismissing forecasts as inherently subject to error, agencies could make forecasts more useful and more believable by the public if they embrace the uncertainty as an element of all forecasting. Building on this conclusion, the authors of this study offer the following four recommendations for improving traffic forecasting practice. These recommendations are directed to technical staff at state DOTs, MPOs, and other similar organizations. Recommendation 1: Use a range of forecasts to communicate uncertainty Consistent with past research, our results show a distribution of experienced traffic volumes around the forecast volumes. These distributions provide a basic understanding of the uncertainty in outcomes surrounding forecasts. A goal of forecasting is to both to minimize the bias in this distribution, and to reduce the variance such that the forecasts more closely align with counted traffic, but we cannot realistically expect to achieve perfection. Instead of perfection, the goal should be to While certain known factors, such as population, employment and fuel price forecasts are important contributors, the reasons for forecast inaccuracy are diverse and do not lend themselves to easy solutions.

Traffic Forecasting Accuracy Assessment Research Summary I-16 achieve forecasts that are “good enough” to make an informed decision about a project. One definition of “good enough” is that the forecast is close enough to the actual outcomes that the decision would remain the same if the decision had been made with perfect knowledge. In order to evaluate whether a forecast is sufficient to inform the decision at hand, we recommend that forecasters explicitly acknowledge the uncertainty inherent in forecasting by reporting a range of forecasts. If a traffic count at the low or high end of the range would not change the decision, then the sponsors can safely proceed with little worry about the risk of an inaccurate forecast. If actual future traffic at the low or high end of the range would change the decision, that should be considered a warning flag. Further study may be warranted to better understand the risks involved, or decision makers may choose to instead select an alternative with lower risk. The quantile regression models described in this report provide a means of estimating the range of uncertainty around traffic forecast. The accompanying guidance document describes in more detail how to use these models. Recommendation 2: Systematically archive traffic forecasts and collect observed data before and after the project opens As discussed in Recommendation 1, there is value in understanding the historic accuracy of forecasts in part because it provides an empirical means of communicating the uncertainty in outcomes surrounding a forecast. This recommendation is predicated on having the data to support such analyses. To continue to achieve these benefits, we recommend that agencies responsible for traffic forecasts systematically archive traffic forecasts and collect data on outcomes after the project opens. Because it is much more difficult to assemble the data afterwards, we recommend that agencies archive their forecasts at the time they are made. We found that we could learn more from projects where we had more information available, but recognize that it requires effort to collect it. We recommend three tiers of archiving, with each building upon the previous. These seek to balance the importance of the project with the effort involved in compiling the data:  Bronze. The first level, termed “Bronze”, records the basic information of the forecast and as well as basic details about the type of project and the method of forecasting. After the project opens, the measured traffic should be added. Bronze level archiving is recommended for all project-level traffic forecasts.  Silver. The “Silver” level involves documenting specific details about the project and assumptions about the forecast. It is recommended for large projects and those that represent new or innovative solutions, as well as a sample of typical projects to monitor the accuracy of projects that comprise the largest number of forecasts.  Gold. The “Gold” level, focuses on making the traffic forecast reproducible after project opening. This provides the information needed to more clearly identify the sources of forecasting error. The Gold level is recommended for unique projects, innovative projects that have not been previously forecasted, and a sample of typical projects.

Traffic Forecasting Accuracy Assessment Research Summary I-17 The guidance document provides specific recommendations regarding what to archive and how to do so efficiently, for each of the three tiers. Recommendation 3: Periodically report the accuracy of forecasts relative to observed data We recommend that agencies responsible for producing traffic forecasts periodically report the accuracy of their forecasts relative to outcomes measured when the roads are in service. Doing so will accomplish several things. First, it reveals any bias in traffic forecasts, such as the observation in this research that observed traffic is, on average, 6% lower than forecast. Even if we cannot attribute that bias to a particular source, understanding its presence and magnitude provides more information to the decision making process. Second, it will provide the empirical information necessary to estimate the uncertainty surrounding their traffic forecasts, as described in Recommendation 1. Third, for agencies with a history of producing accurate forecasts, it provides an opportunity to demonstrate their good work and show that they perform better than their peers. Those agencies that produce more accurate forecasts would be justified in using a narrower range when estimating the uncertainty around future forecasts. The guidance document discusses the recommended content of forecast accuracy reports, and specific methods by which to report forecast accuracy. Recommendation 4: Consider the results of past accuracy assessments in improving traffic forecasting methods We are not aware of efforts to consider how well travel models perform in forecasting as a means to improve the next generation of travel models. That should change. Therefore, we recommend that when agencies set out to improve their traffic forecasting methods or to update their travel demand models, they consider the results of past forecast accuracy assessments in doing so. This may be done in several ways:  If Deep Dives reveal specific sources of error in past forecasts, those sources should be given extra scrutiny when developing new methods. Conversely, if Deep Dives reveal that a particular process is not a major source of error, then additional resources need not be allocated to further refining that process.  Data collected on counted traffic volumes (Recommendation 2) can be used as a benchmark against which to test a new travel model. Rather than focusing the validation on the model’s fit against base-year data, this would test whether the new model is able to replicate the change that occurs when a new project opens. This tests a model in the way it will be used, and a much more rigorous means of testing.  To the extent that Large-N analyses can be used to demonstrate better accuracy for one method over another, that information should inform the selection of methods for future use. We were not able to demonstrate such differences in this research, due largely to challenges in isolating the effect of the method on accuracy versus the type of project and other factors. A more rigorous research design would control for these factors by testing multiple methods for the same project, or by more carefully recording the details of all projects so they can be more fully considered in the analysis.

Traffic Forecasting Accuracy Assessment Research Summary I-18 The guidance document describes the specific ways in which forecast accuracy data can be used to improve traffic forecasting methods. We recommend that these approaches be integrated into travel model development and improvement projects. 6. Why should transportation agencies implement these recommendations? While conscientious forecasters strive for objectivity, this does not necessarily ensure that their forecasts are accurate, nor does it ensure that their forecasts are viewed as credible in the eyes of decision makers or the public. Both are important and related. An inaccurate forecast may lead to a sub-optimal decision for a project, but it may also undermine the trust in forecasts made for other projects. To meet this challenge, we recommend a strategy of deliberate transparency, as outlined in the above recommendations. If the agency can build a track record of accurate forecasts, it provides evidence with which to build trust in their abilities and establish the credibility of future forecasts. Reporting inaccurate results demonstrates a willingness to learn and improve in the same way that scientists may report data that contradicts their previous predictions. In addition, acknowledging the uncertainty inherent in forecasting and reporting a range is a way for the forecasting agency to protect its own credibility. For example, a 15% difference from a single point forecast may be criticized as inaccurate, but if that same forecast were reported with a range of +/- 20%, it may instead be considered accurate because it is within the reported range. Several transportation agencies, as listed in the acknowledgments, provided the data to make this research possible. All participating agencies have agreed to allow their data to be publicly shared, and we are making it available for future researchers who may wish to continue this work as more projects open. The agencies that shared data for this study are a model of transparency and should be celebrated for their efforts to learn from past forecasts and engage in a process of continued improvement. 7. Further reading This executive summary is accompanied by a two volume report. Part I: Guidance Document provides more detailed guidance on implementing the recommendations for improving traffic forecasting practice. Chapter 1 provides an overview of the The agencies that shared data for this study are a model of transparency and should be celebrated for their efforts to learn from past forecasts and engage in a process of continued improvement. Acknowledging the uncertainty inherent in forecasting and reporting a range is a way for the forecasting agency to protect its own credibility.

Traffic Forecasting Accuracy Assessment Research Summary I-19 research as a whole, and additional detail on the recommendations. Chapters 2 through 5 correspond to each of the four recommendations, providing details on how to implement them. The appendices provide resources to facilitate their implementation. There are several electronic resources, including references to a spreadsheet implementation of the quantile regression models, software that can be used to track forecast accuracy in accordance with the bronze standard, and the data used in this research. Part II: Technical Report presents the methods of analysis and the results upon which the guidance is based. The appendices include a literature review and technical details about both the large N analysis and the deep dives.

I-20 Introduction Purpose of the Guidance Document Accurate traffic forecasts for highway planning and design help ensure that public dollars are spent wisely. Forecasts inform discussions about whether, when, how, and where to invest public resources to manage traffic flow, widen and remodel existing facilities, and where to locate, align, and how to size new ones. It is therefore in the interest of transportation planners and policy makers to base such decisions on the most accurate possible traffic forecasts. However, we must also recognize that no forecasts will be perfectly accurate. It is prudent to quantify the expected inaccuracy around traffic forecasts and consider that uncertainty in making decisions. Together, more accurate traffic forecasts and a better understanding of the uncertainty around traffic forecasts can lead to a more efficient allocation of resources and build public confidence in the agencies that produce those forecasts. This project, sponsored by the National Cooperative Highway Research Program (NCHRP), sought to develop a process and methods by which to analyze and improve the accuracy, reliability, and utility of project-level traffic forecasts. For purposes of this study, the terms accuracy and reliability addressed how well the forecasting procedures estimated what actually occurred; while utility encompasses how well a particular projected outcome informs a decision; and project level is meant to include a single project or a bundle of closely related projects. An important aspect of this objective is that it aims not only to analyze the accuracy of existing traffic forecasts, but also to develop a process for continuing to do so. In light of this dual objective, the product of this research is presented in two parts: 1. Part 1: Guidance Document. This document provides guidance to Metropolitan Planning Organizations (MPOs), state Departments of Transportation (DOTs), and others to help them improve the accuracy, reliability, and utility of traffic forecasts as applied to transportation planning, design, and operation. 2. Part 2: Technical Report. The accompanying technical report presents the detailed results of the analysis upon which this guidance is based. While Part 2 presents the results of what was done already, Part 1 provides the instructions for how to apply those findings to improve traffic forecasts, and how to implement a process of ongoing improvement. Specifically, the guidance describes how to use the measured accuracy of past traffic forecasts to estimate the uncertainty around new traffic forecasts. It goes on to describe tools that engineers and planners can use to archive forecasts for use in the future and to track the accuracy of those forecasts. Finally, it describes how to apply the findings of forecast accuracy evaluations to improve the traffic models used to generate those forecasts.

Traffic Forecasting Accuracy Assessment Research Guidance Document I-21 Chapter 1 provides a summary of the research supporting this guidance document and an overview of the four key recommendations resulting from this research (Section 1.3.2). After reading Chapter 1, users should understand the basis for the recommendations, what the key recommendations are, and why they might wish to implement those recommendations. Chapters 2 through 5 are written as a “how-to” guide, focusing on the more detailed steps necessary to implement those recommendations. Each chapter corresponds to one recommendation and is organized around the time at which a task would be undertaken. The additional chapters include:  Chapter 2 describes how to use measured forecast accuracy to communicate uncertainty around new forecasts. This would be done by the person preparing a traffic forecast at the time the forecast is made.  Chapter 3 describes ways to archive traffic forecasts. An important finding of this report is that it is easier to analyze forecast accuracy if the forecasts are archived at the time the forecast is made, rather than trying to dig them up after-the-fact. Therefore, a person at the organization responsible for the forecasts would set up the archival structure, and the forecaster would use that structure.  Chapter 4 discusses methods to report forecast accuracy. This activity would be conducted using the archived forecasts (Chapter 3) and accompanying observed data, and would be used to inform estimates of the uncertainty around new forecasts (Chapter 2), as well to improve traffic forecasting models (Chapter 5).  Chapter 5 describes ways to use forecast accuracy data to improve enhanced or new models. Such activities would be engaged in based on the measured accuracy of those models.  Chapter 6 discusses the reasons to implement these recommendations and directions for future research. This report is accompanied by an electronic appendix of ready-to-use tools discussed in the chapters. Most of the products presented are geared towards highway traffic forecasts but can be modified with modest efforts for other modes. The expected audience is engineers and planners who are involved in generating traffic forecasts, with an accompanying executive summary intended for decision makers and planners who may be involved as consumers of traffic forecasts and wish to have a broad understanding of their typical accuracy. This guidebook will help agencies “jump start” forecast accuracy archiving and analysis. It provides information to help forecasters communicate the importance of instituting internal archiving procedures and introduces ready-to-use tools found in the electronic appendix. Notes on Forecast Accuracy, Reliability and Utility The objective of this study is to develop a process to analyze and improve the accuracy, reliability and utility of project-level traffic forecasts. For the purpose of this project, the three terms can be defined as: • Accuracy is the measure of how well the forecast estimates actual project outcomes,

Traffic Forecasting Accuracy Assessment Research Guidance Document I-22 • Reliability likelihood that a process applied to multiple, similar projects will generate a forecast similarly accurate for all such projects, and • Utility is how well the projected outcome informs a decision. Accuracy includes, but is not necessarily limited to, forecast versus actual traffic volumes. Because accuracy is dependent on knowing the actual outcome, it can only be measured after a project opens. Reliability, while sometimes used interchangeably with accuracy in common language, is actually distinct in scientific terms. In a scientific context, it is the likelihood that someone repeating the experiment will obtain the same result. Thus, it is a comparison of multiple forecasts to each other or of multiple observations to each other. In contrast to accuracy, forecast reliability can be evaluated at any point after the forecast is made, and does not require waiting for the project to open. One important place reliability shows up in this project is in the use of traffic counts. Traffic counts collected at the same location on different days generally have some variation. If counts on different days are close in value, they are reliable and if there is a large variation, they are unreliable. The best outcome we can hope for is that the accuracy of the forecast exceeds the reliability of the traffic counts. The reliability of forecasts can be considered as well. We might expect that a reliable forecast is one where a different analyst making a different set of reasonable assumptions, or using a slightly different version of the model inputs would get nearly the same result. In this way, reliability is closely related to sensitivity analysis or risk and uncertainty analysis where the analyst tests a range of assumptions to see whether the projected outcome changes by a large or small amount. Utility has to do with the value of the information provided. Consider, for example, developing a traffic forecast with a straight-line projection from traffic count growth versus with a four-step travel model. Either one could be accurate and reliable, but if a key question faced by decision makes is how much diversion there is from parallel roads, then the straight-line forecast has no utility for answering the question. The same issue can arise for a model that has fixed time-of-day factors if the project requires the consideration of peak spreading. In this way, the utility of a forecast is inherently dependent on the questions being asked. In fact, one of the key justifications offered for the development of advanced models is that they are sensitive to a broader range of policy questions (Donnelly et al. 2010). During the course of our research, we chose to focus our efforts largely on the question of accuracy as an important unanswered question that we could effectively address. That focus is reflected in the content of this guidance document and the technical report. We acknowledge that reliability and utility are also important in traffic forecasting. In fact, several of the recommendations related to forecast accuracy will also promote greater reliability in forecasting. Specifically, the recommendations related to archiving forecasts and documenting important assumptions promote a level of transparency that will make it easier to replicate forecasts and test their reliability, even before the project opens. Forecast utility is an important consideration that may be related to accuracy. For example, a forecast may be so inaccurate that it is not useful. However, we leave it largely to the judgment of the forecaster and to other researchers to consider how to enhance the utility of traffic forecasts as it relates to the questions they answer.

Traffic Forecasting Accuracy Assessment Research Guidance Document I-23 Research Summary This section provides an overview of the studies research approach and findings. Further detail is included in Part 2: Technical Report. Background Scholars and critics of transportation planners and policy makers have for several decades focused international attention on forecast accuracy, documenting observed inaccuracy levels to decision makers and the public. They have also addressed reasons for forecast inaccuracy, including poor data on which forecasts are based, inappropriate assumptions about future conditions, the use of overly simplistic forecasting procedures, and political motivations that sometimes distort forecasts intentionally. A review of the literature reveals that the length of the forecast horizon, the nature of the facility for which forecasts are made, stability or volatility in population growth and economic factors, all influence forecast accuracy in predicable ways and that in addition forecasts will always be influenced by some factors which are unexpected, unpredictable, and therefore difficult to anticipate. But most past studies have been analytical rather than prescriptive. They report what was observed but offer little advice to planners, engineers, or policy makers as to how they may improve forecasting practice by transportation agencies and their consultants. Forecast accuracy is easy to understand in general terms, as the extent to which a forecast of traffic volumes at one point in time estimated for a date in the future matches the traffic that is counted when that future date arrives. But the issue becomes far more complicated when addressed in appropriate depth. To conduct meaningful analysis, it is necessary to make many assumptions and to adopt conventions for which judgements are needed for which there are no obvious standards of correctness or precision. Traffic volumes can be forecast for short or long periods of time, varying between peak fifteen-minute periods or entire days, including or not including weekends along with weekdays, and differentiating or aggregating forecasts across seasons of the year. Volumes can be forecast for each direction of traffic flow on a highway or for both directions combined. Flows can be forecast by short segments of roadways or averaged over long expanses. Just as traffic volumes can be expressed in different ways, accuracy also can be expressed using various metrics. Absolute differences between forecast and measured traffic volumes can result in rather different estimates of accuracy depending upon how they are aggregated across road segments and time periods, or whether errors are expressed as percentages of forecast flows or of measured traffic volumes. Extreme values, or outliers, affect assessments of accuracy and can be included in comparisons if it is believed they are valid measurements, or excluded from forecast accuracy assessment if, in the judgement of the analysts, they result from measurement or recording errors. This study is intended to fill gaps in the literature by proposing tools and techniques by which state departments of transportation and others, like metropolitan planning organizations, can improve the accuracy of their traffic forecasts by adopting new practices based on what others have done with some success. Other fields have demonstrated the effectiveness of reviews that have led to the adoption of improved forecasting practice, including the National Oceanic and Atmospheric Administration (NOAA) which adopted a highly successful Hurricane Forecasting Improvement Program. Informed by a growing literature critiquing transportation forecasts and emulating the

Traffic Forecasting Accuracy Assessment Research Guidance Document I-24 success of other fields, this study addresses ways transportation agencies can improve documentation and assessment of traffic forecasting experience to improve future applications. While being attentive to insights and explanations of forecast accuracy, the emphasis here is on practice. It addresses what agencies can do to improve the accuracy of their traffic forecasts. Assumptions and compromises were necessarily made about the many sources of error and after considering differences of opinion about best practices for comparing accuracy and measuring traffic flows, but the objective was always to improve the applicability of knowledge to agency practice. Research Approach To meet the study objective, we: (1) analyzed traffic forecasting accuracy and usefulness using information from various sources including state departments of transportation (DOTs), metropolitan planning organizations (MPOs), counties, and other transportation agencies actively involved in forecasting travel demand in competitive modes; (2) assessed transportation agency experience with respect to accuracy of various forecasting approaches; (3) identified methods for improving flexibility and adaptability of available forecasting techniques to changing assumptions and input data; (4) considered alternative ways of incorporating risk and uncertainty into forecasts; and (5) identified potential methods that can help the traffic forecasting industry improve forecasting usefulness and accuracy while improving their ability to communicate and explain these forecasts to affected communities. A review of the literature revealed that most prior studies of traffic forecasting accuracy had adopted either of two types of studies to assess the accuracy of the forecasts that they studied. The two approaches are complementary, and it was decided to employ both in this study. The first approach relies on gathering a large sample of forecasts for which data were collected and the forecasts were made sufficiently long ago that the horizon year of the forecasts has come, making it possible to compare the forecasts of traffic with measured traffic flows on the facilities for which the forecasts were made. With a large enough sample of such forecasts, statistical analysis can be used to examine correlations between forecast accuracy and data inputs, facility types. methods used to conduct the forecasts, and factors exogenous to the forecasts that influenced their accuracy. Because this analysis involved a large sample of cases, it will be referred to throughout this report as the “large N analysis.” The second type of study identified in the literature review consisted of case studies of particular facilities in which forecasts were made at some date in the past, the projects were planned in detail and built, and resulting traffic flows were observed. Most often, case studies have been performed of a single project or of a very small number of projects, using customized data collection including review of historical documents, before and after surveys of travelers, and interviews of those who participated in project decision making, The depth of the analysis may lead researchers to identify with confidence sources of forecast errors—such as errors in inputs, incorrect assumptions, model specification, changes in the project definition, and others. The Federal Transit Administration conducts such before and after analyses of patronage and cost forecasts for major capital investments in public transit, but there are of course many fewer such investments nationally than there are highway construction projects, and a much lower proportion of forecasts for highway than transit projects have

Traffic Forecasting Accuracy Assessment Research Guidance Document I-25 been studied in this manner. The advantage of detailed case studies is that they allow a complex set of issues to be thoroughly investigated and their interactions explored. They reveal the importance of assumptions made by modelers in relation to available data and the strengths or weaknesses of particular models that were used. The disadvantage is that it is that it is very difficult or impossible to generalize from particular case studies. This study included six such case studies and they will be referred to throughout this report as “deep dives.” Large N studies and deep dives clearly complement one another by shining different lights on the same problem – the inaccuracy of traffic forecasts. Because taken together the two types of studies provide analysts with insights that add to our overall understanding, this study used both approaches and reached conclusions after comparing findings from each component of the study. The large N analysis required compiling a database of forecast and counted traffic for about 1,300 projects from six states and four European countries. It is the largest known database for assessing traffic forecast accuracy, and allowed development of distributions of forecast errors, analysis of relationships between measured traffic volumes and the forecast traffic volumes as well as a variety of potentially descriptive variables. Where systematic errors were found in the forecasts, we could study whether errors were functions of factors such as the type of project, the time between the forecast and the opening year, and the forecasting methods that had been used. The combination of large N analysis and deep dives was intended to reveal as many as possible of the sources of error in traffic forecasts but in the end some of the error will remain unexplained because it will always be impossible to account for every deviation between forecasts and measured traffic. This results in part from that fact that the data were drawn from numerous dissimilar sources and collected with different levels of precision. But, it also reflects inherent uncertainty in forecasting and ultimately the inability to specify the numerous sources of potential disparities between assumptions and reality. This research is intended to help agencies like state DOTs and MPOs improve their forecasts. Because of this, findings about forecast accuracy and model improvement are not sufficient. They must recognize the capacity of and challenges facing the organizations that collect the data, calibrate and operate the models, and report their findings in highly politicized environments. For these reasons, and others, the findings are aimed at agencies and their processes as much as or more than at technical characteristics of forecasts. We asked transportation agencies about their standard practices and their greatest challenges. Among the most important findings are those about what organizations can do to strengthen forecasts and the guidebook is meant to be used by practitioners for this purpose. To be sure that the recommendations are useful in practice, the team made every effort to learn from the agencies that had produced the forecasts. For example, forecasters consistently identified inadequate data availability as their greatest obstacle to accurate forecasting. Whenever possible, the team tried to replicate what agency forecasters had done, including actually running the travel demand when possible. Conclusions from this analysis led to judgements about the data that should be collected and archived about forecasts and project outcomes that are measured years or decades later, information should be archived from a forecast? Considerations of agency culture, capacity, and practice also were considered when guidelines were developed for future deep dives or in-depth case studies of traffic forecasts.

Traffic Forecasting Accuracy Assessment Research Guidance Document I-26 In the following sections we summarize the methods, data, findings and conclusions from the large N and deep dive analyses. Based on these findings, the guidebook was prepared to enable agencies to build data archives and to develop methods that will enable them to carry out their own assessments of forecast accuracy and hopefully to systematically improve the accuracy and utility of their forecasts in the future. Large N Analysis: Data and Methodology The large N analysis compared traffic forecasts made at the time projects were planned with flows that were measured after the projects were completed. To carry out this analysis, a data base was assembled containing both traffic forecasts made when projects were planned and later traffic count data provided by six states and four European countries. The states that provided data included Florida, Massachusetts (only one project), Michigan, Minnesota, Ohio and Wisconsin, and the four European countries were Denmark, Norway, Sweden and the United Kingdom. Additional data from Virginia and Kentucky were gathered but not included in the database and could be added at a future date if additional resources were to be made available for that purpose. The records were compiled from databases maintained by departments of transportation (DOTs) complemented by project reports, traffic studies and environmental impact statements as well as other databases assembled by researchers for prior studies. The three types of data incorporated include information identifying and describing each project, information about the traffic forecasts, and information about traffic flows that were eventually counted. The database contains unique project identification numbers and each project’s improvement type, facility type, location, length, and for traffic forecasts includes information about who made the forecast, the year each forecast was made, the forecast target year, and whether the forecast was for the opening year, a mid-year in the life of the project, or a distant future year as far as twenty years after opening. Also recorded was the forecasting methodology, and traffic count information such as the dates of the counts and the method by which the counts were done and identified the particular segments where they were done. The forecasted traffic and measured traffic for a project in the target year were compared and several metrics were calculated to ascertain the level of inaccuracy in the traffic forecasts. Forecasts were recorded for elements of projects that were referred to as “segments.” For example, forecasts for an interchange improvement project could include segment-level estimates for each direction of travel on the freeway, for both directions of the crossing arterial, and for each of the ramps. The database includes at least some information about 16,360 segments of 2,348 projects, but a substantial proportion of the projects in the database have not yet opened to traffic and some of the segments for which forecasts were made had no subsequent count data associated with them. Others did not pass quality control checks for inclusion in the statistical analysis. The statistical analysis of the large N data was done for a carefully filtered subset of 1,291 projects comprising 3,912 segments, while the records describing the remaining segments and projects are available for future analysis. The data base is not a random sample of all highway projects, and this limits our ability to generalize from the analysis. The years in which projects in the database opened to traffic ranges from 1970 to 2017, with about 90% of the projects opening to traffic in 2003 or later. While the exact nature and scale of each project is not known in every case, almost half of the entries in the database were design forecasts for repaving projects. Earlier projects were more likely to be major

Traffic Forecasting Accuracy Assessment Research Guidance Document I-27 infrastructure capital investment projects and more recent ones were more often routine resurfacing projects on existing roadways. This arose because some state agencies began tracking all forecasts as a matter of course only within the past ten to fifteen years and, in earlier years, information was retained only for major investments. In addition to the mix of projects in the database, there also were notable differences in the forecasting methods used across agencies. Because the traffic counts were of average daily traffic, comparisons could not be made of peak period traffic, by day of the week, or by season. We evaluated the accuracy of opening year forecasts for the practical reason that the interim and design years have not yet been reached for a large majority of projects included in the data base. When opening year traffic counts were unavailable for some projects, we used the earliest available traffic counts and adjusted year of completion forecasts to compare them with forecast volume. To do this we scaled the forecast to the year of the first post-opening count by linear interpolation so that both data points were for the same year. The European projects were taken from a doctoral dissertation which and had already been scaled to match the count year using a 1.5% annual traffic growth rate. We used this approach for European projects and interpolated between opening and design year for US projects. The obvious way to represent the accuracy of project level traffic forecasts is comparing them with counts of actual traffic after the projects have been in service, but there are rather different ways in which to do so. The literature revealed that previous studies of multiple projects defined errors or differences between forecast and measured traffic in different ways. Some measured error as the predicted volume minus the measured volume; using this metric a positive result is an over-prediction. Others defined error as the measured volume minus the predicted volume; for them a positive value represents under-prediction. A popular metric used to determine the accuracy of traffic forecasts is the “half a lane” criterion. This criterion specifies that the forecast is accurate if the forecast volume varies from measured volume by less than half a lane’s capacity from the constructed facility’s capacity. If the forecast is more than half a lane less than the facility’s capacity, the facility could have been constructed with one fewer lane in each direction. If the forecast was more than half a lane than the facility capacity, the facility needs one additional lane in each direction. Calculating whether a forecast is within a half a lane requires several assumptions, such as the share of the daily traffic that occurs in the peak hour. The error between a forecast and measured traffic also can be expressed as a percentage of the counted traffic or as a percentage of the forecast traffic. An advantage of the former is that the percentage is expressed in terms of a measured or “real” quantity (observed traffic); an advantage of the latter is that when the forecast is made, uncertainty can be expressed in terms of the forecast value since the observed value is of course not known until later. Yet another approach found in some earlier studies would be to evaluate forecast accuracy by comparing the ratio of measured (actual) to forecast traffic. In this study, we measured forecast accuracy as the percent difference from forecast: PDFFi= Counted - Forecast VolumeForecast Volume *100% (1)

Traffic Forecasting Accuracy Assessment Research Guidance Document I-28 In which PDFFi is the percentage difference for project i. Negative values indicate over prediction in that the traffic flow counted on the actual project is lower than the forecast, and positive values indicate under predictions because the measured or actual outcome is higher than the forecast. This is appealing because it expresses the error as a function of the forecast, which always is known earlier than traffic counts. The distribution of the percent difference from forecast when measured this way over the dataset was used to portray the systematic performance of traffic forecasts. When expressing the sizes of errors, some researchers have used the mean percentage error and others have preferred the mean absolute percentage error, disregarding the positive or negative sign associated with the error. Mean absolute percentage error (MAPE) allows better understanding of the size of inaccuracies across projects since positive and negative errors tend to offset one another when calculating the mean percent error. We continue in this tradition. We do this using the percent difference from forecast, referring to this as the mean absolute percent difference from forecast (MAPDFF) instead of MAPE. We measure the size of the errors as the: Mean Absolute Percent Difference from Forecast MAPDFF = 1 n *∑ |PDFFi|ni=1 (2) Where n is the total number of projects. When assessing project forecast accuracy, it is important to distinguish between comparisons by segment or by project. Road projects, as noted earlier, are typically divided into several links or segments within the project boundary. The links can be on different alignments or carrying traffic in different directions. Each project in the database was given a unique identification number, and segments that comprised the projects also were identified. Comparisons of accuracy can thus be made using segments or aggregating them into projects. Accuracy metrics at a segment level are not independent. A project containing multiple segments connected end-to-end likely had traffic for all its segments made at the same time, using the same methods, and employing the same external forecasts of population and economic growth. The errors for the various segments are likely highly correlated—they are likely to be uniformly high or low. Whether we treat these as one combined observation or several independent observations, we would expect the average error to be similar. There would be a difference, however, in the measured t-statistics because the larger sample of segments could suggest significance whereas the smaller sample of projects might not. Segment-level analysis, while it might seem less valid than project comparisons, has other merits because a few measures of inaccuracy are better represented in analysis of segments. When assessing forecast inaccuracy over roadways of different functional classes, segment level results provide better representation than aggregated results over the entire project. In previous studies, some researchers weighted segments within a project by their volumes when arriving at accuracy figures for projects, but a limitation of doing that is that segments have different lengths and thus should probably not be weighted by traffic volumes alone. We have little data on the lengths of links and did not choose to weight by volumes. To the extent possible we reported the distribution of forecast errors at both project and segment levels. For project-level comparisons, we averaged traffic volumes across all segments and measured the error statistics by comparing the average forecast and average measured or actual traffic. The variables with which the volumes were compared in further analysis were also aggregated.

Traffic Forecasting Accuracy Assessment Research Guidance Document I-29 Improvement type, area type and functional class could differ by segment but the most prevalent one among the segments was used to classify the projects. For example, if most of the segments in a project consist of resurfacing and reconstruction and no major improvement, the project is considered to be of that type even if one segment might include work otherwise classified as a major improvement. Forecast methods, unemployment rates and years of forecast and observation were the same for all segments comprising a project and could be used directly for project level analysis. Past research used ordinary least squares (OLS) regression to compare forecast traffic with flows measured once the projects are open to traffic. Such comparisons can be used to identify biases in estimates and are usually done by regressing the actual or counted volumes of traffic as a function of the forecast value using equation: 𝑦 𝛼 𝛽𝑦 𝜀 4 in which 𝑦 is the actual measured traffic for project i, 𝑦 , is the forecast traffic for project i and 𝜀 is a random error term. α and β are estimated terms in the regression. Here α=0 and β=1 implies unbiasedness while higher or lower coefficients imply upward or downward biases in the forecasts. OLS and Random Effect Linear Regression may be used to explain variations in error forecasts as functions of explanatory variables (such as year the project opens or elapsed time since opening). To compare the effects of potential explanatory variables, the model used is: 𝑦 𝛼 𝛽𝑦 𝛾𝑋 𝜀 5 where Xi is a vector of descriptive variables associated with project i, and γ is a vector of estimated model coefficients associated with those descriptive variables. To consider multiplicative effects as opposed to additive effects, we scaled the regressors by the forecast value: 𝑦 𝛼 𝛽𝑦 𝛾𝑋 𝑦 𝜀 6 In such a formulation, γ =0 indicates no effect of that particular term, while positive values would increase the forecast by that amount and negative values would reduce the forecast by that amount. In addition to the estimation of biases, this study examined how the distribution of errors relates to the descriptive variables. For example, it could be the case that forecasts having longer time horizons are unbiased with respect to those having shorter time horizons – they are not systematically higher or lower - but they could have larger errors, as measured by the MAPE. To explore this issue, we extended the regression framework to use quantile regression instead of ordinary least square (OLS) regression. Whereas OLS predicts the mean value, quantile regression predicts the values for specific percentiles in the distribution. We estimated quantile regression models of the measured or counted traffic volumes as a function of the forecast volumes and other descriptive variables. We did so for the 5th percentile, 20th percentile, median, 80th percentile and 95th percentile. The median value provides the expected value, while the 5th or 20th percentiles provide a lower bound on the expected value, and the 80th and 95th percentiles provide upper bounds. This range of models allows

Traffic Forecasting Accuracy Assessment Research Guidance Document I-30 for a comparison of the variability of the forecasts within ranges as well as the more usual and simple estimates of means alone. Large N Analysis: Results Analysis of data from 3,912 segments which comprised 1,291 separate projects is presented in the Technical Report and is graphically summarized in Figures 1 and 2. Figure 1 shows that traffic forecasts more often over-predicted future flows than under-predicted them (i.e. the distribution of errors shown in Figure 1 is heavier on the negative side). The mean of the absolute percent difference from forecast is 17.31% with a standard deviation of 24.93. Figure 2 presents project forecast errors as a function of forecasted volumes, and shows that percentage errors decrease as traffic volumes increase. When expressed as a percentage, an error of a particular size is a smaller percentage of a larger forecast, so this trend is expected, and we also found that the distribution of errors became less dispersed around the mean error as forecast volumes increased. Figure 1: Distribution of Percent Difference from Forecast (Project Level) 𝑃𝐷𝐹𝐹 𝐴𝑐𝑡𝑢𝑎𝑙 ForecastForecast ∗ 100

Traffic Forecasting Accuracy Assessment Research Guidance Document I-31 Figure 2: Percent Difference from Forecast as a function of Forecast Volume (Project Level) As noted above, quantile regression was used to explore the uncertainties inherent in forecasting traffic. It is useful to include that uncertainty as part of the information presented when developing forecasts by showing, as we do in Figure 3, bands within which we can be increasingly certain that the forecasted traffic will lie. Using the data for projects included in this study, we developed several quantile regression models to assess the biases in the forecasts on the variables described in the previous chapter. The models were developed for the 5th, 20th, 50th (median), 80th and 95th percentile values, and of course they demonstrate rather wide ranges of potential actual traffic at a given level of certainly as the mean volume forecasts grow. The lines in the graph depicting various percentile values can be interpreted as the range of actual traffic over a forecast volume. For example, it can be expected that 95% of all projects with the forecasted Average Daily Traffic (ADT) of 30,000 will have actual traffic below 45,578. Only 5% of the projects will experience actual traffic less than 17,898. Not considering other variables, this range (45,578 to 17,898 for forecast volume of 30,000) includes 90% of the projects. 𝑃𝐷𝐹𝐹 𝐴𝑐𝑡𝑢𝑎𝑙 ForecastForecast ∗ 100

Traffic Forecasting Accuracy Assessment Research Guidance Document I-32 Figure 3: Expected Ranges of Actual Traffic (Base Model) We can make a number of observations from the Large-N analysis. These observations and the supporting data are described in the accompanying technical report, but summarized here: 1. Traffic forecasts show a modest bias, with actual ADT about 6% lower than forecast ADT. 2. Traffic forecasts show a significant spread, with a mean absolute percent difference from forecast of 25% at the segment level and 17% at a project level. 3. Traffic forecasts are more accurate for higher volume roads. 4. Traffic forecasts are more accurate for higher functional classes, over and above the volume effect described above. 5. The unemployment rate in the opening year is an important determinant of forecast accuracy. 6. Forecasts appear to implicitly assume that the economic conditions present in the year the forecast is made will perpetuate. 7. Traffic forecasts become less accurate as the forecast horizon increases, but the result is asymmetric, with actual ADT more likely to be higher than forecast as the forecast horizon increases. 8. Regional travel models produce more accurate forecasts than traffic count trends. 9. Some agencies have more accurate forecasts than others. 10. Traffic forecasts have improved over time. 11. We find that 95% of forecasts reviewed are “accurate to within half of a lane.” An important limitation to the above observations is worth noting here. The projects analyzed were selected based on the availability of data, not in a way to create a random or representative sample of all projects. Therefore, there may be selection bias that influences the above observations. Specifically, while the opening year of projects ranges from 1970 to 2017, about 90% of projects open Perfect Forecast 5th Percentile Median 95th Percentile 20th Percentile 80th … 0 10000 20000 30000 40000 50000 60000 0 10000 20000 30000 40000 50000 60000 Ex pe ct ed  A DT Forecast ADT

Traffic Forecasting Accuracy Assessment Research Guidance Document I-33 in year 2003 or later. While we don’t always know the exact nature of a project, inspecting the data reveals that the older projects tend to be major infrastructure projects, while the newer projects are more likely to be more routine work, such as improvements to existing facilities, or sometimes repaving projects. Likewise, the type of project, the methods used, and the specific data recorded are all a function of the agency providing the data. This means that those factors may interact in ways that make it difficult to draw firm conclusions about questions such as the effectiveness of different methods. For this reason, we are making available all of the data underlying this report for future researchers to analyze further. In addition, we encourage users to collect their own data based on projects similar in nature to those they are interested in, as described in the recommendations. Deep Dives: Objectives, Data, Cases and Methodologies The statistical analysis of data for a large number of projects - the large N analysis, provided useful indications of the magnitudes of errors and suggested factors that might in general be causes of forecast errors. But the aggregate analysis that was done on the large sample could not determine with certainty the causes of forecast errors. To the extent that we could do so, we tried to shed more light on factors that help to explain those errors more completely. We conducted six fairly detailed case studies of traffic forecasts in different states for highways having rather different contexts and forecast results. Our ability to conduct “deep dives” was limited by time, money, and, most critically, by the availability of data. While six is a small number, and one of the six was insufficiently completed to allow confident conclusions, the insights gained were substantial and richly complemented what had been learned from the large N analysis. The deep dives investigated projects that were large enough in scope and budget to be meaningful changes in the local transportation network. Projects were also chosen to represent a variety of types of highway improvements. The case studies were chosen in part because the projects under investigation had been completed, were already open to traffic, and there was post-opening data to compare with forecasts. For each case study there was detailed information available about the forecasts that had been conducted as the projects were being planned. Ideally, this included in some cases forecasting model runs that could be studied because they had been archived. Where model runs had not been preserved, detailed reports of forecasts were used, and in some cases indirect sources, including environmental impact reports or other public documents, could be used to arrive at detailed forecasts. The sample of projects chosen for deep dives is listed in Table ES 1. The cases included a new bridge, the expansion and extension of an arterial on the fringe of an urban area, a major new expressway built as a toll road, the rebuilding and expansion of an urban freeway, and a state highway bypass around a small town. Chapter 3 of the Technical Report includes details of each case study project, including the location of each project, the most important design and contextual variables, sources of data collected for each, the methods and models used to make the forecasts, forecasts of traffic, and post-opening descriptions of conditions. The descriptions in Chapter 3 are complemented by Appendix C which provides more detailed technical information for each case study. In part, the number of deep dives was small because it proved to be difficult to find suitable case studies. We did find a few agencies who carefully archived forecasts, but none had been doing so for more than a decade and projects planned more recently were less likely to be completed and open to traffic. Where they still were present, long-time senior staff had kept their own records and

Traffic Forecasting Accuracy Assessment Research Guidance Document I-34 some recalled what was done in particular cases, but staff turnover, retirements, and deaths made it rare that such people were found. Because of limited funding, a large proportion of relatively recent projects consisted of toll roads, making it difficult to identify toll free highways for case studies. Table 1: Projects selected for Deep Dive Analysis Project Name Brief Description Eastown Road Extension Project, Lima, Ohio Widened a 2.5-mile segment of the arterial from 2 lanes to 5 lanes and extended the arterial an additional mile Indian River Bridge, Palm City, Florida This 0.6 mile long bridge with four travel lanes in total. runs along CR 714 (Martin Highway), connecting with the Indian River Street and goes across the St. Lucie River. Central Artery Tunnel, Boston, Massachusetts Reconstruction of Interstate Highway 93 (I-93) in downtown Boston, the extension of I-90 to Logan International Airport, the construction of two new bridges over the Charles River, six interchanges and the Rose Kennedy Greenway in the space vacated by the previous elevated I-93 Central Artery in Boston, Massachusetts. Cynthiana Bypass, Cynthiana, Kentucky A 2-lane, state highway bypass project, to the west of the City from the southern terminus where US 62S and US27S meet. South Bay Expressway, San Diego, California A 9.2-mile tolled highway segment of SR 125 in eastern San Diego, CA. US 41 (later renamed as I-41), Brown County, Wisconsin A project of capacity addition, reconstruction of nine interchanges, constructing 24 roundabouts, adding collector- distributer lanes, and building two system interchanges located in Brown County, Wisconsin. Deep dives enabled the team to explore which elements of forecasts could clearly be identified as sources of inaccuracies. Population forecasts and regional employment forecasts had in some cases been used as the basis of forecasts in traffic growth, and the population had not grown as had been forecast, or an economic downturn had caused shortfalls in expected job growth. Where such departures from expected trends could be identified their influences on traffic forecasts were estimated. If major changes had been made in the scope of a project after the preparation of forecasts - for example changes in roadway alignment, the number of lanes or the elimination or addition of entrances and exits - those could explain differences between forecasts and realized traffic. In some, but not all, cases the data and models that had been used to make the forecasts were available to the team and we were able to recreate forecasts using the population and employment growth that had actually occurred and roadway features that had changed, to determine whether the data input into the forecasts, or the forecasting models had been the sources of the errors. While the team endeavored to attribute errors identified in the case studies to their sources, some of the differences between forecast and measured outcomes of course remained unexplained. Where clear patterns enabled the team to reach conclusions about sources of errors and those were used to formulate advice for future forecasters.

Traffic Forecasting Accuracy Assessment Research Guidance Document I-35 Deep Dives: Results and Interpretations The case studies were consistent with the large N analysis in that forecast traffic was generally higher than traffic that was measured after the projects opened. The deep dives enabled us to explore in greater depth than the large N analysis factors that were likely to have contributed to this overestimation. The case studies demonstrated that the sources of errors in forecasts were the result of the different contexts and conditions of particular projects and that made it difficult to generalize. A few general patterns and some informative though more conclusions can be stated, however. There was a well-documented economic downturn and recovery during the decade preceding this research, and it seemed likely that forecasts had been based on erroneous expectations about underlying growth and price trends that likely would have been projected to continue and then did not. The recession caused a loss of jobs and economic activity that could significantly impact work trips, leisure activities, and consumption patterns which in turn impact traffic flows. The case studies revealed that traffic forecasts overestimated traffic in part because they were based on exogenous projections of future employment (jobs) and population that were too high and on petroleum fuel prices that were too low in comparison with what had actually occurred. Projections based on past trends did not anticipate the dramatic economic perturbations that had occurred. Where modeling software was available and the models were rerun with actual employment and population growth figures and true fuel prices, the forecasts that they produced were much closer to actual, measured traffic flows. Where models could not be rerun because they were not available, the calculation of elasticities (percent changes in traffic as a function of percentage changes in the input variables) led to similar conclusions. In one case, the economic downturn had, in addition to reducing employment, income, and travel, also caused new development at the urban fringe to be delayed long enough that anticipated toll revenue had not been realized. In some cases, errors in traffic forecasts resulted from errors that had been made in exogenous estimates of “external” traffic, meaning traffic that originated or terminated outside the project area. Taken as a given when a bypass was designed, the rate of growth in external traffic departed from the projections to the extent that traffic on the bypass was incorrectly forecast. In another case, traffic projected to be crossing an international border as an input into traffic forecasts for a project, and that exogenous forecast similarly proved to be incorrect in part because of its insensitivity to the economic downturn. Conclusions and Recommendations Based on the research described above, we offer the following conclusions, and accompanying recommendations for improving traffic forecasting practice. Lessons Learned One of the most important and overarching conclusions from this study is that agencies should take far more seriously the analysis of their past forecasting errors to that they can learn from the cumulative record. It is tempting to assert that the future is always uncertain and thus forecasts of the future will always be wrong, but doing that is far too glib. Agencies may believe that they avoid

Traffic Forecasting Accuracy Assessment Research Guidance Document I-36 political embarrassments by not archiving past forecasts and by not addressing the sources of their past forecast errors. This may be easy and low in cost in the short run, but it explicitly prevents them from improving future forecasts and using resources more efficiently in the future. Forecasts are essential elements in the creation of effective highway plans and project designs, and because forecasts are always subject to error it is essential that agencies document their forecasts and revisit them in order to identify assumptions that lead to errors that are compounded over time. Project documentation is often insufficient to allow agencies to evaluate the sources of their forecast errors. In the deep dives, we found that the forecasting accuracy improved after accounting for several exogenous variables like employment rate and population. However, the effect of changes in other potentially important variables could not be ascertained for some of the projects. Improved documentation of the forecast methodology would make such assessments more informative, particularly on the definition of the variables used in the model. Improved documentation can be achieved through standardized approaches to documentation and through archiving past forecasts so that they become accessible to agency staff in the future. Some transportation agencies have started archiving their forecasts in recent years, and we are beginning to see the benefits of that foresight. The data used in the Large N analysis were provided by several state DOTs and several researchers who studied forecasts over time. Some of the efforts involved creating databases of all project-level traffic forecasts in recent years. Many of these are for projects that have not yet opened to traffic. We based this analysis on about 1,300 projects that have opened and are in service, and thousands more are available in the database, waiting to be evaluated after they open. Forecast evaluation was most effective when archived model runs were available. The most successful deep dives, resulting in the most insight into sources of forecast errors, were those for which we had archived model runs and associated data inputs available. This provided deeper understanding of the parameters and methodology used for forecasting traffic and allowed us to directly test the effect of changes. While agencies frequently express skepticism about the usefulness of archived model runs, we were able to successfully run and learn from all three of the archived model runs we were provided. These included models that had been used to make forecasts as far as 15 or more years ago including some that had been developed using software that had since been superseded by several later versions. When forecasting models and input data are systematically archived, it becomes possible to compare the accuracy of different forecasting methods by comparing competing forecasts for the same project. For example, we were in some cases able to compare traffic forecasts that were simple straight-line projections of past trends with forecasts made with a travel model and to show that models generally provide better results than trend extrapolation. Despite this, it was difficult to arrive at meaningful comparisons of the accuracy of forecasts in which different models were employed. This is both because the details of models and their specific features were typically not recorded as part of the data we were given, and because model differences vary among agencies that state they are using the same models. The best way to compare types of models would be to produce competing forecasts for the same set of projects and compare their accuracy. This would be equivalent to a controlled experiment accounting for all relevant factors.

Traffic Forecasting Accuracy Assessment Research Guidance Document I-37 A final observation well illustrated by the quantile regression of the data in our large N analysis, is that, especially for complex and expensive capital investment projects, the most efficacious forecasting could well involve the development of ranges of future traffic. Instead of dismissing forecasts as inherently subject to error, agencies could make forecasts more useful and more believable by the public if they embrace the uncertainty as an element of all forecasting. Summary of Recommendations Following from the above lessons we learned during the course of this project, as well as our own experience across several decades of producing and using traffic forecasts, the authors of this report offer four major recommendations, described below. These recommendations apply to agencies or organizations responsible for creating project-level traffic forecasts. Generally, this is expected to be MPOs or state DOTs, but it could also include their contractors or private firms engaged in generating traffic forecasts either for the public sector or for private investors. Recommendation 1: Use a range of forecasts to communicate uncertainty Consistent with past research, our results show a distribution of actual traffic volumes around the forecast volume. These distributions provide a basic understanding of the uncertainty in outcomes surrounding a forecast. A goal of forecasting is to both to minimize the bias in this distribution, and to reduce the variance such that the forecasts more closely align with actual traffic. While our results show that forecasts have tended to improve over time (subject to confounding with the types of project in more recent forecasts), we cannot ever expect to achieve perfection in the realm of forecasting. Instead of perfection, the goal should be to achieve forecasts that are “good enough” to make an informed decision about a project. One definition of “good enough” is that the forecast is close enough to the actual outcomes that the decision would remain the same if the decision had been made with perfect knowledge. For example, if the forecast is used to make a decision about how many lanes to build on a roadway, the conventional wisdom is that the traffic forecast should be “accurate to within half of a lane.” Our analysis shows that 95% of the projects considered meet this threshold. A corollary definition of “good enough” is that decision makers are willing to accept the consequences of a sub-optimal decision as a trade-off for the ability to move forward with imperfect information. If the consequences of an imperfect decision are low, then fewer resources can be invested in forecasting, whereas more extensive study and more accurate forecasts may be warranted when the consequences are high. This will naturally distinguish between smaller routine projects, and the larger mega-projects, or projects that are otherwise unique. Similarly, the effect of traffic forecasts on decisions depends on the information that is most relevant to the decision. While research focuses specifically on project-level forecasts of the average daily traffic (ADT), the information relevant to projects aimed primarily at improving traffic safety may be different. In order to evaluate whether a forecast is sufficient to inform the decision at hand, we recommend that forecasters explicitly acknowledge the uncertainty inherent in forecasting by reporting a range of forecasts. If an actual outcome at the low or high end of the range would not

Traffic Forecasting Accuracy Assessment Research Guidance Document I-38 change the decision, then the sponsors can safely proceed with little worry about the risk of the project. If an actual outcome at the low or high end of the range would change the decision, that should be considered a warning flag. Further study may be warranted to better understand the risks involved, or decision makers may choose to instead select a project with lower risk. There are multiple mechanisms for considering uncertainty in traffic forecasts. During our outreach efforts to preview this recommendation, we talked to a number of analysts who preferred to consider uncertainty by running travel models multiple times with different inputs, such as low, medium and high employment growth forecasts. This is a good approach, and has the advantage that it allows the travel model to account for the non-linear relationships between traffic volume and congested speed. The limitation of this approach is that the analyst must determine a reasonable range of inputs with which to run the travel model. In this report, we present an alternative method for determining a range of forecasts derived from the historic accuracy of traffic forecasts. The quantile regression models described above provide this mechanism. They are estimated from the actual traffic volume as a function of the forecast traffic volume, and provide a means of predicting the range of expected traffic from a single forecast traffic volume. They are capable of considering the characteristics of the project and the forecast. For example, forecasts with a shorter time horizon may have a narrower range of expected outcomes. The ranges are empirical, meaning that they consider the full set of possible errors that have occurred in the past, rather than leaving it up to the analyst to determine a reasonable range of inputs. This is both an advantage and a disadvantage. It is beneficial because it may implicitly incorporate factors that the analyst may not consider on her own. However, it is limiting if the future looks very different from the past. For example, a risk of forecasts made in 2019 may be the effect of car-sharing, and that risk is not one that has been an issue for projects that are already open. One important advantage to the quantile regression approach is that it is very simple to apply. To obtain a range of ADT forecasts, it is as simple as tracing lines on a chart or inputting values to a spreadsheet. Both are provided along with this report and are derived from the full set of projects considered here. Chapter 2 describes the details of how to apply this method to obtain this range of forecasts. Recommendation 2: Systematically archive traffic forecasts and collect observed data before and after the project opens As discussed in Recommendation 1, there is value in understanding the historic accuracy of forecasts in part because it provides an empirical means of communicating the uncertainty in outcomes surrounding a forecast. This ability is predicated on having the data to support such analyses. This research is only possible due to the foresight of the agencies that started collecting the necessary data and were willing to share it. The data accompanying this report provide a snapshot for forecasts made by a specific set of agencies for a particular set of projects. However, these data will become dated as new projects continue to be planned and to open. In addition, these data are general, and the experience of individual forecasting agencies may vary. For example, consider an agency that has invested a specific set of traffic forecasting methods. That agency may be interested in understanding how well their specific methods perform. Likewise, a particular agency may frequently build a particular type of project, and have an interest in understanding the range of actual outcomes as a function of the forecast traffic for similar projects.

Traffic Forecasting Accuracy Assessment Research Guidance Document I-39 For these reasons, we recommend that agencies responsible for traffic forecasts systematically archive those traffic forecasts and collect data on actual outcomes after the project opens. We recommend that the archiving happen at the time the forecast is made, because we can confirm the experience of other researchers that it is much more difficult to assemble the data afterwards. We recommend that the archiving and the collection of associated data be made systematic, such that it is a normal part of the forecasting process to ensure it happens. We recommend that the process be standardized, such that comparable information be collected for all projects. This standardization will make the data collection easier, because the forecaster does not need to decide what to record and can instead fill in a template. It will also ensure that the data can be more readily compared across projects and across agencies. In our research, we found that we were able to learn more from projects where we had more information available. The basic project information available to the Large-N analysis allowed us to create the overall distributions of forecast accuracy, consider the effect of different factors, and generate the quantile regression models. The more detailed information available to the Deep Dives allowed us to better understand why the forecasts were right or wrong. We also recognize that there is effort involved in this archiving and data collection process, and there is more effort involved in compiling and archiving more detailed project data. Therefore, it is appropriate that the effort expended relate both to the intended use of the data and the importance of the project.  Bronze. The first level, termed “Bronze”, records the basic information of the forecast and the actual traffic volume, as well as basic details about the type of project and the method of forecasting. Bronze level archiving is recommended for all project-level traffic forecasts.  Silver. The second level, termed “Silver”, adds upon the Bronze level to record details and nuances that would otherwise not be captured. It is not recommended for all projects, but for projects that are larger than typical projects or represent new or innovative solutions that do not have a long track record of accurate forecasts. It is recommended to apply the Silver level to a sample of typical projects to monitor the accuracy of projects that comprise the largest number of forecasts. The Silver level consists of documenting specific details about the project and assumptions about the forecast.  Gold. The third level, termed “Gold”, builds upon the Silver level by focusing on making the traffic forecast reproducible after project opening. In this way, the sources of forecasting error can be definitively identified. The Gold level is recommended for unique projects, innovative projects that have not been previously forecasted, and a sample of typical projects. Chapter 3 provides specific recommendations of what to archive and how to do so efficiently, within the context of these three tiers. Recommendation 3: Periodically report the accuracy of forecasts relative to observed data We recommend that agencies responsible for producing traffic forecasts periodically report the accuracy of their forecasts relative to actual outcomes. Doing so will accomplish several things. First, it will provide the empirical information necessary to uncertainty surrounding their traffic forecasts,

Traffic Forecasting Accuracy Assessment Research Guidance Document I-40 as described in Recommendation 1. Second, it will ensure a degree of accountability and transparency in the forecasting process. For accountability, it removes any incentive to “nudge” the forecasts in one direction or another. For transparency, it makes it clear that the agency is acting in good faith to produce the best possible forecasts and has nothing to hide. This may be especially important if the analysis shows the forecasts to be biased in one direction or another, because it provides an opportunity to explain the reasons for that bias, and to account for such bias adjusting the expected range of future forecasts. For agencies with a history of producing accurate forecasts, it provides an opportunity to demonstrate their good work and show that they perform better than their peers. In such situations, those agencies would be justified in using a narrower range when estimating the uncertainty around future forecasts. This reporting would rely on the data systematically collected according to Recommendation 2. It is expected that such a report might include three main components:  Updating the overall distribution of forecast error, with possible exploration of distributions by project type and other dimensions.  Estimating quantile regression models from local data for the purpose of generating more specific ranges of expected outcomes as a function of the forecast.  Specific Deep Dives aimed at understanding sources of forecast error for either typical or important projects. Chapter 4 discusses specific methods by which to report forecast accuracy. In doing so, it provides a structure for users to follow based on the research done in this project, aiming to make the overall process less burdensome. Recommendation 4: Consider the results of past accuracy assessments in improving traffic forecasting methods The traffic forecasts considered in this research were generated using multiple methods, the most common being the use of a travel demand model, and through extrapolating trends in traffic counts. While extrapolating traffic count trends is a simple process, the analyst must still make choices about the details of the method, such as how long of a past trend to consider, and whether the growth should be linear. Travel demand models consider a broader set of factors in forecasting and come in a variety of forms. Often, the methods used to generate forecasts and the details used in those methods are selected based on the judgment of the analyst. They are usually estimated from travel survey data, and calibrated to match observed base-year traffic counts and other base-year data. Sometimes they undergo sensitivity testing to ensure that they respond reasonably to changes in input, and sometimes they are tested in backcasting exercises to ensure they can reasonably replicate past conditions. However, forecasting is distinct from backcasting and from comparisons against base- year conditions because the future is unknown at the time of a forecast. We are not aware of efforts to consider how well travel models perform in forecasting as a means to improve the next generation of travel models. That should change. We recommend that when agencies set out to improve their traffic forecasting methods or to update their travel demand models, they consider the results of past forecast accuracy assessments in doing so. This may take several forms:

Traffic Forecasting Accuracy Assessment Research Guidance Document I-41  If Deep Dives reveal specific sources of error in past forecasts, those sources should be given extra scrutiny in developing new methods. Conversely, if Deep Dives reveal that a particular process is not a major source of error, then additional resources do not need to be allocated to further refining that process.  Data collected on actual project outcomes (Recommendation 2) can be used as a benchmark against which to test a new travel model. Rather than focusing the validation only on the model’s fit against base-year data, this would test whether the new model is able to replicate the change that occurs when a new project opens. This is akin to testing a model in the way it will be used, and a much more rigorous means of testing.  To the extent that Large-N analyses can be used to demonstrate better accuracy for one method over another, that information should inform the selection of methods for future use. We regret that we were not able to demonstrate such differences in this research, due largely to challenges in isolating the effect of the method on accuracy versus the type of project and other factors. A more rigorous research design would control for these factors by testing multiple methods for the same project, or by more carefully recording the details of all projects so they can be more fully considered in the analysis. Chapter 5 describes the specific ways in which forecast accuracy data can be used to improve traffic forecasting methods. We recommend that these approaches be integrated into travel model development and improvement projects. Reasons to Implement These Recommendations Traffic forecasts are used to inform important decisions about transportation projects, including the selection of which projects to build and certain design elements of those projects. Because transportation projects involve trade-offs between the benefits and costs of a projects, it is necessary that such decisions involve a political element to consider those trade-offs. However, engineers and other technical experts have an ethical obligation to ensure that the accounting of benefits and costs is done in an objective manner, allowing the decision makers to focus more on the trade-offs rather than on the quality of the information provided. While contentious forecasters strive for objectivity, this does not necessarily ensure that their forecasts are accurate, nor does it ensure that their forecasts are viewed as credible in the eyes of decision makers or the public. Both are important, and both are related. An inaccurate forecast may lead to a sub-optimal decision for that specific project, but it may also undermine the trust in forecasts made for other projects. If decision makers dismiss a forecast because they do not view it as credible, then the decision-making process may become purely political without the benefit of objective information about the effects of the proposed project. One possible strategy to prevent this outcome is to avoid drawing attention to inaccurate forecasts, and instead ask decision makers and the public to trust the technical expertise of those preparing the forecasts. There are several characteristics of forecasts (Wachs 1990) that enable this strategy: forecasts are technically complex, they require subjective assumptions, and they cannot be verified until the intended action is taken. The first two characteristics make it easy to ask for a reliance on the experts, while the third makes it easy to dismiss a retrospective view as irrelevant because the

Traffic Forecasting Accuracy Assessment Research Guidance Document I-42 decision has already been made. While there is some logic to that strategy, there is also some risk, because if that trust is broken, there is no clear mechanism by which to rebuild it. Even if a forecaster is very objective and careful in their analysis, their credibility could be undermined by another forecaster who is less so. For example, Flyvbjerg (2007) argues that some planners deliberately misrepresent the costs and benefits of large infrastructure projects to increase the likelihood that they will be built. Such deliberate misrepresentation is clearly worse than the forecaster who happens to be wrong because of an unexpected recession at about the time the project opens. How should those forecasters who are careful and objective in their analysis distinguish themselves from those who are less so? This distinction can be made by reversing the above strategy, and instead being deliberately transparent. While transparency does not necessarily ensure that forecasts will be accurate, it does send a clear message that the agency preparing the forecasts has nothing to hide, that any inaccuracies are the result of unexpected outcomes and not deliberate misrepresentation, and that the agency is legitimately interested in learning from those inaccuracies and using them to improve. If the agency can build a track record of accurate forecasts, it provides evidence with which to build trust in their abilities and establish the credibility of future forecasts. These benefits related to building credibility are in addition to the benefits associated with using the information to generate more accurate forecasts. This approach aligns with the broader scientific process, which involves making testable predictions (forecasts) based on theory and assumptions, evaluating whether those predictions are correct, and revising the theory or assumptions if they are not. In addition to this process of prediction and testing, there are community-level components necessary to ensure the validity and credibility of science (Galileo Galilei 1638), including:  Data recording and sharing,  Reproducibility, and  External review. Archiving forecasts and ensuring they are available for external review helps to build this broader validity and credibility in the same way that publishing scientific results in peer-reviewed journals does so for science more broadly. The process ensures the advancement of science, even if individual predictions are later shown to be inaccurate because those predictions provide a learning opportunity. In addition, the recommended approach facilitates the implementation of performance-based planning and programming (PBPP), which is a point of emphasis in federal transportation legislation, Moving Ahead for Progress in the 21st Century (MAP-21). PBPP aims to select transportation projects based on their expected ability to meet stated goals and objectives, then monitor and evaluate whether they are effective in doing so. Monitoring and evaluating the accuracy of traffic forecasting tools is a way of applying PBPP principles to the tools. In addition, monitoring transportation project outcomes enables PBPP methods to be applied at the project level. This can demonstrate value from those projects, and helps forecasters develop professional judgment by providing them with a library of projects similar to what they may currently be forecasting. This would enable what Flyvbjerg refers

Traffic Forecasting Accuracy Assessment Research Guidance Document I-43 to as reference class forecasting (Flyvbjerg et al. 2006), and is especially valuable for new forecasters who do not have a lifetime of their own experience to draw from. Finally, the recommended approach is consistent with the standards for evidence-based decision-making put forth by the International Standards Organization and codified in their ISO 9000 standard (International Standards Organization (ISO) n.d.). ISO 9000 focuses on developing a process for ensuring and monitoring quality, such as the process recommended in this guidebook, and is widely used in the private and public sectors in the United States and Europe. Specific reasons to implement individual recommendations are listed below. Recommendation 1: Use a range of forecasts to communicate uncertainty Reporting a range of forecasts explicitly communicates the risk associated with forecasts, and it is possible that the range results in a different decision, or the introduction of strategies to manage that risk. If the project decision would be the same across the range of forecasts, this adds confidence that the decision is defensible. Acknowledging the uncertainty inherent in forecasting and reporting a range is a way for the forecasting agency to protect its own credibility. For example, consider a forecast that a road will carry 20,000 ADT. When the road opens, the actual traffic is only 16,000 ADT. It is easy to criticize the forecast as inaccurate because there is a -20% difference from the forecast. However, if the original forecast were to report an expected value of 20,000 ADT, with a range of 15,000 to 25,000 ADT, then the actual traffic would be within the range, and this would be considered an accurate forecast. Recommendation 2: Systematically archive traffic forecasts and collect observed data before and after the project opens In addition to enabling the remaining recommendations, systematically archiving forecasts and the associated project outcomes promotes transparency as discussed above. The very fact that details about the forecasts will be transparent encourages a higher level of care and quality in the forecasts. This is especially true when the full model runs are archived according to the Gold Standard. Traffic forecasting models can be complex tools, and it is easy to be careless in their application in terms of managing files and scenarios, setting input parameters, and other details. If the model runs are stored and documented in such a way that another person can reproduce the results, then it forces the original analyst to be meticulous in its application. This also prevents the files from becoming lost or disorganized on someone’s individual computer. Recommendation 3: Periodically report the accuracy of forecasts relative to observed data Updating and reporting forecast accuracy results with local data provides a better indication of the performance of the tools that a specific agency will use. This can document improvement or better than typical accuracy. If an agency has a track-record of accurate forecasts, using this data to update the quantile regression models will allow the ranges considered in Recommendation 1 to be narrower.

Traffic Forecasting Accuracy Assessment Research Guidance Document I-44 Recommendation 4: Consider the results of past accuracy assessments in improving traffic forecasting methods Another reason to review the accuracy of forecasts is to understand the advantages of particular travel models compared with others. Efficient use of resources continues to be an overarching goal. Of every transportation agency. By comparing forecasting methods and models, forecasters can compare cost differences among methods to the differences in forecast accuracy and reasonableness in the real-world. As a result, cheaper forecasting techniques can be used for types of projects that are well understood. Cost savings for more routine projects can allow more resources to be used for more intensive techniques to address more challenging situations.

Traffic Forecasting Accuracy Assessment Research Guidance Document I-45 Using Measured Accuracy to Communicate Uncertainty Forecasting is a challenging activity which always includes an inherent level of uncertainty. Yet forecasters traditionally are asked to provide a point-forecast – usually a single traffic volume (or one set of traffic volumes for an entire project) – devoid of any recognition of uncertainty. As noted in Chapter 1, we recommended that forecasters use a range of forecasts to communicate this uncertainty. Effectively communicating uncertainty to forecast recipients has the following benefits:  Those who apply the forecast can be aware that the forecast may deviate from the primary point-forecast, encouraging them to account for uncertainty in their decisions.  The uncertainty in the inputs and assumptions behind the forecasts are passed along to the customers of the forecast. The inputs and assumptions, such as population and employment growth forecasts, incorporate inherent uncertainty. The travel forecaster typically “absorbs” these uncertainties by providing point-forecasts. This implicitly implies that there is no uncertainty in the forecast. Consequently, any errors in the forecast will be blamed on the forecaster and the travel model.  Counted volumes that fall within the uncertainty range will be deemed accurate, reliving forecasters of the need to defend even minuscule errors from point-forecasts.  General acknowledgement that travel models cannot be accurate, but rather are a tool used to provide general information about project demand. This chapter describes how to use measured forecast accuracy to communicate uncertainty. Quantifying Uncertainty One method to quantify uncertainty is to vary the forecast inputs and assumptions to reflect their uncertainty ranges and re-run the travel demand model with multiple inputs. This process can be repeated many times, so that all primary inputs vary by their (minimum and maximum) extreme values individually and collectively. The result is a distribution of outcomes reflecting the specified range of inputs and assumptions. This method is less pragmatic if the travel model has long running times, project schedules are constrained, or if a simple a trend line extrapolation was used to produce the point-forecast. These situations commonly occur in traffic forecasting, so an alternative method is needed.

Traffic Forecasting Accuracy Assessment Research Guidance Document I-46 Using quantile regression lines derived from archived forecasts offer an alternative method. Quantile regression is like linear regression. Instead of computing the standard errors based on the sample mean, the errors are based on a specified quantile (e.g., 10th percentile, 20th percentile, etc.) of the sample. For example, assume Figure 4 represents a series of project forecasts and their actual values. Each point represents one project. The forecast ADT was predicted several years prior for the project opening year. The actual ADT was measured after the project opened. Consider that our goal is to predict the actual ADT as a function of the forecast ADT. Estimating such a model using standard linear regression, would result in the line drawn through the middle of the cloud of data. If this line has a slope of 1 and an intercept of 0, it would indicate that the forecast ADT is an unbiased estimate of the actual ADT. (The forecast is not always an unbiased estimate of actual ADT—past research has shown that forecasts sometime systematically over- or under-estimate actual demand.) If all of the points fell perfectly on the diagonal, it would indicate that the forecasts are a perfect predictor of the actual outcomes. Quantile regression, by contrast, draws lines at the edges of the cloud. The function of the lines is the same – to create a linear estimate for the cloud of data – but quantile regression can be used to estimate the range within which the estimates lie, rather than the mean or average of the cloud. This provides a means for forecasters to supply upper and lower forecast bounds. Figure 4: Chart of Sample Forecasting Accuracy Data

Traffic Forecasting Accuracy Assessment Research Guidance Document I-47 Introduction to Quantile Regression Quantiles, or percentiles, are breakpoints that divide a frequency distribution into intervals with the specified probability. For example, the 5th percentile (quantile 0.05) is the value for which there is a 0.05 probability of a value drawn randomly from the distribution being lower than the specified value. At the 95th percentile, there is a 0.05 probability of a value drawn randomly from the distribution being higher than the specified value. Therefore, a range of quantiles can be used to express a range of likely outcomes. Given a historical dataset of forecasted and actual traffic volumes one can estimate a model: 𝑦 𝛼 𝛽𝑦 𝜀 where 𝑦 is the actual traffic on project i, 𝑦 is the forecast traffic on project i, and 𝜀 is a random error term. 𝛼 and 𝛽 are estimated terms in the regression. Here α=0 and β=1 implies unbiasedness. Whereas linear regression would estimate a single 𝛼 and single 𝛽, quantile regression instead estimates one 𝛼 for each quantile of interest and one 𝛽 for each quantile of interest. Such a model must be estimated based on historic data—using forecasts that were made in the past for projects that have since opened, such that actual data can be collected. Figure 5 illustrates the results of quantile regressions. The lines are estimated from a sample dataset of project forecasts and actual volumes. The x-axis represents the forecast ADT, and the y- axis represents the actual ADT in the year for which the forecast was made. Figure 5: Example of Quantile Regression Results The chart contains six lines (from top to bottom):

Traffic Forecasting Accuracy Assessment Research Guidance Document I-48 1. The orange line represents estimates of actual volumes based on the lowest 95% of forecast error values; 2. The green line represents estimates of actual volumes based on the lowest 80% of forecast error values; 3. The gray line represents the perfect forecast (i.e., 45-degree line, or when the actual forecast always matches the forecasted ADT); this line is used for reference; 4. The yellow line represents the 50th percentile of forecast errors, also known as the median; 5. The dark blue line represents estimates of actual volumes based on the lowest 20% of forecast error values; and 6. The light blue line represents estimates of actual volumes based on the lowest 5% of forecast error values. In general, quantile regression lines (lines #1, 2, 4, 5 and 6) represent estimated changes in the actual or measured volumes, based on the nth percentile of error values, given changes in forecasted volume. So lower percentile lines – percentiles less than 50% – reflect over-forecasted values more strongly, while higher percentile lines – percentiles greater than 50% – more strongly reflect under- forecasted values. Error ranges, the difference between the 80th and 95th percentile lines and the 20th and 5th percentile lines, generally widen as the forecasted value increases. A point on the quantile regression lines represents an estimate of the actual volume, based on the nth percentile of error values, for a selected forecasted volume. If the yellow line representing the median aligns perfectly with the gray line, it means that the forecasts are unbiased. The lower and higher quantile lines provide the range within which actual ADT falls. For example, 60% of actual outcomes are expected to fall between the 20th and 80th percentile lines, and 90% of actual outcomes are expected to fall between the 5th and 95th percentile lines. If the 5th and 95th percentile lines are closer to the diagonal, it indicates that actual outcomes fall in a narrower range around the forecast. While quantile regression models such as these must be estimated using data from after the project opens, they can be applied to future forecasts before a project opens. In this way, they can be used to provide a range of expected actual values. The wording of such as statement would be, “Based on historic accuracy, if we have a forecast of X, we would expect that 90% of actual outcomes to fall between the range of Y5 and Y95.” Default versus Local Quantile Regression The next section describes how to apply the quantile regression models that were estimated for this research using data from several agencies. These models are best viewed as a default, and should be used in the absence of other data. However, it is important to recognize that the specific types of projects considered by and forecasting methods used by an agency may differ from these defaults. It is therefore preferable to compile local forecast accuracy data as described in Chapter 3, and use those data to estimate local quantile regression models as described in Chapter 4. Doing so is expected to provide a better view of the uncertainty window relevant to a particular agency. If an agency develops a track record of producing accurate forecasts, then the resulting quantile regression models will have a narrower range, supporting more certainty in project decisions.

Traffic Forecasting Accuracy Assessment Research Guidance Document I-49 Applying Quantile Regression Methods By developing quantile regression lines from a historical dataset of project traffic forecasts and measured or actual traffic, these lines provide uncertainty ranges based on past experience. Assume Figure 5 was created by an agency that recently developed quantile regression lines using their historical accuracy database. The agency produced a traffic forecast of 40,000 ADT for a project, and they wish to provide an uncertainty range. Looking at Figure 5, for a 40,000 ADT forecast, the percentile estimates are shown in the following table. Table 1: Actual Volume Estimates Given 40,000 ADT Forecast (Example) Percentile Actual Volume Estimate 5% 25,000 20% 32,000 80% 44,000 95% 60,000 The agency has the option of providing multiple uncertainty ranges, depending on their confidence in the forecast and historical track record on accuracy. A narrower range would involve the 20% and 80% percentile estimates: based on the agency’s accuracy track record, about 60% (6 of 10) of past projects would have produced an actual traffic volume between 32,000 and 44,000. The 60% value is derived by subtracting the lower percentile value from the higher percentile value (80% - 20% = 60%). A wider range would involve the 5% and 95% percentile estimates: based on the agency’s accuracy track record, about 90% (9 of 10) of past projects would have produced an actual traffic volume between 25,000 and 60,000. These ranges can be provided to decision makers, agency colleagues and/or the public in lieu of the 40,000 point-forecast. The use of a narrower or wider band depends on the agency's tolerance for risk. We recommend the 5th and 95th percentiles as a default range. These regression equations can include many variables. The study team assembled three versions of quantile regressions using the traffic database discussed in the accompanying Technical Report:  A simple version that includes only the forecast volume and an intercept as variables;  A forecasting version that includes selected variables; this version is best representative of a practical application because the included variables should be known at the time the forecast is made; and  An inclusive version that includes an extensive set of variables; this version is best used for applied research purposes because it includes variables that may only be known after the project opens. The following table provides a listing of the variables used in each version. This table shows forecasters a range of possible variables for their individual use.

Traffic Forecasting Accuracy Assessment Research Guidance Document I-50 Table 2: Variables Used in Study Team Quantile Regression Analysis Variable Simple Forecasting Inclusive Intercept √ √ √ Forecast volume √ √ √ High forecast volume (if forecast volume 30k+) √ √ Unemployment Rate (for year of forecast) √ √ Unemployment Rate (anticipated for opening year) √ Forecast horizon (years between year of forecast and anticipated opening year) √ √ Forecast horizon is unknown √ New roadway √ √ Project adds capacity √ Unknown improvement type √ Travel model used to produce forecast √ √ Unknown method used to produce forecast √ Forecast was developed by consultant (outside agency) √ Forecast was produced between 1960 and 1990 √ Forecast was produced between 1991 and 2002 √ Forecast was produced between 2003 and 2008 √ Forecast was produced between 2009 and 2012 √ Forecast is for arterial roadway √ √ Forecast is for collector or local roadway √ √ Forecast is for an unknown roadway √ Note: a checkmark (√) denotes the variable is included in the quantile regression equation Other variables can be included by agencies depending on three conditions: (1) they are important to the agency at the time the forecast is communicated to decision makers, (2) the empirical and statistical significance is reviewed prior to inclusion, and (3) there is data to support their analysis. The purpose of the variables is to widen or narrow the quantile estimates based on project characteristics, forecast assumptions and conditions in place at the time the forecast is made. For each project, the forecaster can input the variables to generate the quantile regression lines. Then the forecaster can estimate the upper and lower bounds of the range based on (a) the forecast volume and (b) the percentile line desired for the uncertainty range. Two examples are provided to explain the application. In both examples, an agency has developed five quantile regression equations for 5th, 20th, 50th, 80th and 95th percentiles. The variables and their magnitudes for each of the five quantiles are provided in the following table.

Traffic Forecasting Accuracy Assessment Research Guidance Document I-51 Table 3: Example of Quantile Regression Variables and Coefficients For the first demonstration, assume an agency recently used a travel model to generate a forecast of 50,000 ADT for a new freeway. The opening year will be 10 years from today. The unemployment for the region was 4% at the time of the forecast. After the forecast is finalized, the forecaster enters the following information in the spreadsheet (see Table 4). Table 4: Demonstration #1 Applying Quantile Regression The spreadsheet is the basis for computing the resulting regression lines, producing the chart shown in Figure 6. The agency decides to communicate uncertainty using the 20th and 80th percentile lines as the lower and upper bounds of the range. With the 50,000 ADT forecast, the agency provides the forecast range of 45,000 to 54,000 ADT. Pseudo R‐Squared Coef. t value Coef. t value Coef. t value Coef. t value Coef. t value (Intercept)                           ‐ ‐182.27 ‐1.77 154.578 3.08 255.551 4.67 287.909 3.94 976.786 4.79 AdjustedForecast                        0.70464 15.97 0.73181 36.19 0.89089 45.20 1.02667 44.19 1.25361 23.88 AdjustedForecast_over30k                0.02375 0.57 0.05735 3.05 ‐0.0042 ‐0.22 ‐0.1902 ‐8.30 ‐0.4132 ‐9.89 Scale_UnemploymentRate_YearProduced     ‐0.0058 ‐1.41 0.00487 2.77 0.00164 0.87 0.00693 2.76 0.00999 1.87 Scale_YearForecastProduced_before2010   ‐0.0067 ‐5.64 ‐0.0051 ‐5.23 0.00022 0.27 0.00384 3.91 0.00324 2.36 Scale_DiffYear                          0.00586 2.81 0.00898 6.68 0.00759 5.62 0.0142 8.23 0.0196 10.50 Scale_IT_NewRoad                        0.09326 4.34 0.00948 1.10 ‐0.0081 ‐0.90 ‐0.036 ‐1.93 ‐0.0901 ‐4.29 Scale_FM_TravelModel                    0.06756 3.31 0.0136 1.63 ‐0.0076 ‐0.52 ‐0.0185 ‐1.25 ‐0.1006 ‐7.36 Scale_FC_Arterial                       ‐0.1495 ‐5.24 ‐0.061 ‐4.86 ‐0.0621 ‐5.17 ‐0.084 ‐5.96 ‐0.1163 ‐5.88 Scale_FC_CollectorLocal                 ‐0.2121 ‐4.03 ‐0.1114 ‐4.79 ‐0.1255 ‐5.21 ‐0.2008 ‐5.78 ‐0.3214 ‐2.36 5th Percentile 50th Percentile 95th Percentile 0.475 0.739 0.830 20th Percentile 0.631 80th Percentile 0.804 User can change the yellow cells to see the effect.  Pseudo R‐Squared Coef. t value Coef. t value Coef. t value Coef. t value Coef. t value Values 5th  Percentil e 20th  Percentil e 50th  Percentil e 80th  Percentil e 95th  Percentil e (Intercept)                           ‐ ‐182.27 ‐1.77 154.578 3.08 255.551 4.67 287.909 3.94 976.786 4.79  ‐182  155  256  288  977 AdjustedForecast                        0.70464 15.97 0.73181 36.19 0.89089 45.20 1.02667 44.19 1.25361 23.88  0.705  0.732  0.891  1.027  1.254 AdjustedForecast_over30k                0.02375 0.57 0.05735 3.05 ‐0.0042 ‐0.22 ‐0.1902 ‐8.30 ‐0.4132 ‐9.89 Scale_UnemploymentRate_YearProduced ‐0.0058 ‐1.41 0.00487 2.77 0.00164 0.87 0.00693 2.76 0.00999 1.87  4  ‐0.023  0.019  0.007  0.028  0.040 Scale_YearForecastProduced_before2010 ‐0.0067 ‐5.64 ‐0.0051 ‐5.23 0.00022 0.27 0.00384 3.91 0.00324 2.36 ‐      ‐            ‐            ‐            ‐            ‐            Scale_DiffYear                          0.00586 2.81 0.00898 6.68 0.00759 5.62 0.0142 8.23 0.0196 10.50  10  0.059  0.090  0.076  0.142  0.196 Scale_IT_NewRoad                        0.09326 4.34 0.00948 1.10 ‐0.0081 ‐0.90 ‐0.036 ‐1.93 ‐0.0901 ‐4.29  1  0.093  0.009  ‐0.008  ‐0.036  ‐0.090 Scale_FM_TravelModel                    0.06756 3.31 0.0136 1.63 ‐0.0076 ‐0.52 ‐0.0185 ‐1.25 ‐0.1006 ‐7.36  1  0.068  0.014  ‐0.008  ‐0.018  ‐0.101 Scale_FC_Arterial                       ‐0.1495 ‐5.24 ‐0.061 ‐4.86 ‐0.0621 ‐5.17 ‐0.084 ‐5.96 ‐0.1163 ‐5.88 ‐      ‐            ‐            ‐            ‐            ‐            Scale_FC_CollectorLocal                 ‐0.2121 ‐4.03 ‐0.1114 ‐4.79 ‐0.1255 ‐5.21 ‐0.2008 ‐5.78 ‐0.3214 ‐2.36 ‐      ‐            ‐            ‐            ‐            ‐            5th Percentile 50th Percentile 95th Percentile 0.475 0.739 0.830 Contribution to Equation 20th Percentile 0.631 80th Percentile 0.804

Traffic Forecasting Accuracy Assessment Research Guidance Document I-52 Now assume an agency recently used a travel model to generate a forecast of 15,000 ADT for a widening of a local roadway. The opening year will be 2 years from today. The unemployment for the region was 8% at the time of the forecast. After the forecast is finalized, the forecaster enters the following information in the spreadsheet (see Table 5). The spreadsheet allows one to compute the resulting regression lines, producing the chart shown in Figure 7. The agency decides to communicate uncertainty using the 5th and 95th percentile lines as the lower and upper bounds of the range. With the 15,000 ADT forecast, the agency provides the forecast range of 7,500 to 17,500 ADT with a 90% certainty that the counted traffic will fall between the upper and lower bounds. The 50th percentile should be considered the “most likely” or expected value and may be an adjustment from the forecast. Table 5: Demonstration #2 Applying Quantile Regression User can change the yellow cells to see the effect.  Pseudo R‐Squared Coef. t value Coef. t value Coef. t value Coef. t value Coef. t value Values 5th  Percentil e 20th  Percentil e 50th  Percentil e 80th  Percentil e 95th  Percentil e (Intercept)       ‐ ‐182.27 ‐1.77 154.578 3.08 255.551 4.67 287.909 3.94 976.786 4.79  ‐182  155  256  288  977 AdjustedForecast    0.70464 15.97 0.73181 36.19 0.89089 45.20 1.02667 44.19 1.25361 23.88  0.705  0.732  0.891  1.027  1.254 AdjustedForecast_over30k    0.02375 0.57 0.05735 3.05 ‐0.0042 ‐0.22 ‐0.1902 ‐8.30 ‐0.4132 ‐9.89 Scale_UnemploymentRate_YearProduced    ‐0.0058 ‐1.41 0.00487 2.77 0.00164 0.87 0.00693 2.76 0.00999 1.87  8  ‐0.047  0.039  0.013  0.055  0.080 Scale_YearForecastProduced_before2010   ‐0.0067 ‐5.64 ‐0.0051 ‐5.23 0.00022 0.27 0.00384 3.91 0.00324 2.36 ‐     ‐     ‐       ‐     ‐      ‐     Scale_DiffYear     0.00586 2.81 0.00898 6.68 0.00759 5.62 0.0142 8.23 0.0196 10.50  2  0.012  0.018  0.015  0.028  0.039 Scale_IT_NewRoad     0.09326 4.34 0.00948 1.10 ‐0.0081 ‐0.90 ‐0.036 ‐1.93 ‐0.0901 ‐4.29 ‐     ‐     ‐       ‐     ‐      ‐     Scale_FM_TravelModel    0.06756 3.31 0.0136 1.63 ‐0.0076 ‐0.52 ‐0.0185 ‐1.25 ‐0.1006 ‐7.36 ‐     ‐     ‐       ‐     ‐      ‐     Scale_FC_Arterial    ‐0.1495 ‐5.24 ‐0.061 ‐4.86 ‐0.0621 ‐5.17 ‐0.084 ‐5.96 ‐0.1163 ‐5.88 ‐     ‐     ‐       ‐     ‐      ‐     Scale_FC_CollectorLocal    ‐0.2121 ‐4.03 ‐0.1114 ‐4.79 ‐0.1255 ‐5.21 ‐0.2008 ‐5.78 ‐0.3214 ‐2.36  1  ‐0.212  ‐0.111  ‐0.126  ‐0.201  ‐0.321 5th Percentile 50th Percentile 95th Percentile 0.475 0.739 0.830 Contribution to Equation 20th Percentile 0.631 80th Percentile 0.804 Figure 6: Demonstration #1 Quantile Regression Lines

Traffic Forecasting Accuracy Assessment Research Guidance Document I-53 Figure 7: Demonstration #2 Quantile Regression Lines

Traffic Forecasting Accuracy Assessment Research Guidance Document I-54 Archiving Traffic Forecasts and Associated Data We recommend that agencies responsible for traffic forecasts systematically archive those forecasts and collect observed data on project outcomes both before and after the project opens. In Chapter 1, we introduced three tiers of archiving—Bronze, Silver and Gold--that preserve different amounts of information about forecasts. In this chapter, we provide specific recommendations for what those levels should include and describe a forecast archive and information system to provide a specific structure for that preservation. A strong archival process ensures the necessary details about the forecasts are preserved in a readily-accessible format once the project is opened to traffic. While some agencies are archiving some details of their project forecasts, the NCHRP 08-110 study revealed that archiving procedures were not consistently followed. For example, the NCHRP 08-110 study had to remove over 1,000 projects from the original collection of projects due to incomplete information. This amount reduced the project forecast database by nearly half. Strict archiving procedures would have greatly increased the study’s database and strengthened its findings. For the Bronze archiving level, a goal of this guidebook is to specifically enumerate the information that should be preserved. We have developed a forecast archive and information system that provides a standardized data structure in which this information should be preserved. Other examples of inconsistent archiving processes were revealed during the study’s Deep Dive analysis. The Deep Dive results of one project was severely constrained because of the lack of relevant documentation available. Another project had copious information about the project, including voluminous environmental reports, but actually little relevant to the project forecast. Throughout the study, the research team noted (1) that just knowing where to begin searching for documentation related to project forecasts would dramatically reduce the analysis time and (2) much of the project- related document actually had sparse information about the details of the forecast. The Silver and Gold levels detail the information that should be preserved to support more detailed retrospective analysis of forecasts, such as Deep Dives, for a subset of projects. Archiving forecasts in a consistent manner reduces the time needed to analyze the forecast accuracy and strengthens any findings. Archiving Levels The three archiving levels are:

Traffic Forecasting Accuracy Assessment Research Guidance Document I-55  Bronze. The first level, termed “Bronze”, records the basic information of the forecast and the actual traffic volume, as well as basic details about the type of project and the method of forecasting. Bronze level archiving is recommended for all project-level traffic forecasts.  Silver. The second level, termed “Silver”, adds upon the Bronze level to record details and nuances that would otherwise not be captured. It is not recommended for all projects, but for projects that are larger than typical projects or represent new or innovative solutions that do not have a long track record of accurate forecasts. It is recommended to apply the Silver level to a sample of typical projects to monitor the accuracy of projects that comprise the largest number of forecasts. The Silver level consists of documenting specific details about the project and assumptions about the forecast.  Gold. The third level, termed “Gold”, builds upon the Silver level by focusing on making the traffic forecast reproducible after project opening. In this way, the sources of forecasting error can be definitively identified. The Gold level is recommended for unique projects, innovative projects that have not been previously forecasted, and a sample of typical projects. Below is a description of what each level should involve. Bronze Archiving Level The first level, termed “Bronze”, records the basic information of the forecast and the actual traffic volume. Bronze level archiving is recommended for all project-level traffic forecasts. The information suggested to preserve are: Project information, such as name, city/area location, roadway, project ID used by the agency, a short description of the project, the type of project (widening, resurfacing, etc.), year the project was assumed to be completed and the year it actually was completed, and construction cost. Forecast information, such as the forecast year and type (opening or design year), the year the forecast was produced, the forecast itself, the methodology used to produce the forecast, the version of the model (if used to generate the forecast), and whether post-processing was applied to the raw forecast. The forecasts for individual segments of a project could be recorded individually. Actual traffic count information, such as the traffic count, the year of observation, and the units of the traffic count (e.g., ADT, peak period counts, etc.). It is important that the units of the forecasts and actual traffic count information be in the same units. For example, many travel models output traffic estimates for a “typical weekday”, while traffic counts may be recorded as Average Annual Daily Traffic (AADT). In such cases, the units and any adjustments made for comparison should be clear. Recording this information requires little effort beyond organizing the tracking mechanism. Bronze-level archiving systems are currently in practice in several places, including the Ohio Department of Transportation and Districts 4 and 5 of the Florida Department of Transportation.

Traffic Forecasting Accuracy Assessment Research Guidance Document I-56 An online, open source bronze-level archiving system has been developed for this project. User documentation is incorporated directly into the online tool. This is part of the ForecastCards repository described in Appendix A, with the associated data stored in the accompanying ForecastCardData repository. Section 3.3 provides an introduction to the archiving system developed for this project. Silver Archiving Level The second level, termed “Silver”, adds upon the Bronze level to record details and nuances that would otherwise not be captured. It is not recommended for all projects, but for projects that are larger than typical projects or represent new or innovative solutions that do not have a long track record of accurate forecasts. It is recommended to apply the Silver level to a sample of typical projects to monitor the accuracy of projects that comprise the largest number of forecasts. The Silver level preserves more details about the forecast, the methodology and assumptions. This ensures that a detailed review of the sources of forecasting error can be identified. The information suggested to preserve in this level are: Project Description, which includes the Bronze-level information, descriptions of the project boundaries, a project map, and a description of key travel markets anticipated to use or benefit from the project. (Travel markets are meaningful quantities of trips that traverse from one geographic area to another, sometimes further characterized by trip purpose, time period, line-haul or circulation/distribution movements, or socio-economic criteria.) Description of the Traffic Forecasts, which includes the Bronze-level information, the most recent traffic count used as a basis for the forecast or to validate the travel model, and the uncertainty windows for the project. Forecasting Methodology, which includes a description of the methodology used to develop the forecasts, and a summary of how well the model grasps the existing and expected travel markets and likely to use or benefit most. Assumptions, which documents typical assumptions, such as population, auto fuel prices, auto ownership, and expected changes in technology and/or land use, as well as extra-ordinary assumptions that are specific to the project. Extra-ordinary assumptions could be a particularly large development near the project, impacts from adjacent construction, specific policies or ordinances, or how travelers will react to innovative solutions. Post-Opening Data Collection, which summaries the data needed to verify the traffic forecast and assumptions, and the associated data collection plan that will be executed upon project opening. An annotated outline of the Silver-level archive can be found in Appendix B. Silver-level archiving systems are currently in practice in some places, including Districts 5 of the Florida Department of Transportation and the Wisconsin Department of Transportation.

Traffic Forecasting Accuracy Assessment Research Guidance Document I-57 Gold Archiving Level The third level, termed “Gold”, builds upon the Silver level by focusing on making the traffic forecast reproducible after project opening. In this way, the sources of forecasting error can be definitively identified. The Gold level is recommended for unique projects, innovative projects that have not been previously forecasted, and a sample of typical projects. The Gold level includes the Silver level material plus an electronic archive that allows analysts to completely reproduce the forecast after project opening. The study team found this easier to achieve than originally assumed, as all available models were reasonably reproduced. Gold level archiving practices are used by some sponsors of Federal Transit Administration (FTA) Capital Investment Grant (CIG) projects. CIG grant recipients are required to perform a Before-and-After Study two years after project opening, and although not required to fully reproduce the model originally used to prepare forecasts, they may find this useful to more effectively asses the causes of forecasting errors. Forecast Archive and Information System As described above under the Bronze Archiving Level, we recommend that forecasters archive the basic information of the forecast and the actual traffic volume, making these data available for later analysis. To facilitate the process of compiling and archiving the data, we have developed a forecast archive and information system, which is freely available for future use and expansion. The system is published as an open-source software tool, with more detailed user documentation integrated directly with the software. This allows the user documentation to be updated with any future revisions to the software. This section of the guidance document provides a high-level overview of the software and accompanying data, broken down into three parts:  Development and Design Features: This section describes the development of that system, which provides useful context for the design decisions made in the final implementation. Specifically, we first developed a Microsoft Access database to support the Large-N analysis described in the accompanying technical report. This experience allowed us to design a system that would be more scalable for future use.  Data Specification: This section describes the structure of the data, which we refer to as Forecast Cards. The specification is available at: https://github.com/e- lo/forecastcards.  Data Storage: This section describes options for storing the data once they are in the forecast card format. Currently, a repository provides all of the data used in this research. It is designed so that more data can be added as new projects are planned or opened. The data are available at: https://github.com/gregerhardt/forecastcarddata. Development and Design Features The data for this project were compiled from existing data about forecasts and actual outcomes provided by several agencies. The data provided by each agency was in a different format, and an

Traffic Forecasting Accuracy Assessment Research Guidance Document I-58 early task was to put the data in a common format such that we could analyze the combined database. In defining this common format, we reviewed information from:  Ohio DOT,  Wisconsin DOT,  Michigan DOT,  Virginia DOT,  Florida DOT District 4,  Florida DOT District 5,  Minnesota, and  Kentucky Transportation Cabinet (KYTC) (no database: equivalent single-axle load (ESAL) and traffic projection reports only). Table 6 shows the fields used by each of these sources. These include information in the underlying data, such as traffic forecast reports, that are not recorded in the summary spreadsheets. The table also notes the number of records and unique projects from each source.

Traffic Forecasting Accuracy Assessment Research Guidance Document I-59 Table 6: Common Data Fields in Forecast Accuracy Sources Field Data Source Recommended Ohio DOT Wisconsin DOT Michigan DOT Virginia DOT Florida DOT-D4 Florida DOT-D5* Minnesota Kentucky TC* for Combined Database Number of records 6,229 458 9 1,160 143 50 2,179 n/a Number of unique projects 2,466 132 7 39 134 31 110 Cumulative Spreadsheet or database (flat file) √  √  √  √  √  √  Cumulative Database (relational) √  √  ESAL or technical reports √  √  √  √  Project identification number or code √  √  √  √  √  √  √  √  √  Description, including facility name √  √  √  √  √  √  √  √  Location County √  √  √  √  √  √  √  Mile Post √  √  Other Type of Location √  √    Facilty type or functional class √  √  √  √  √  √  √  Segment identification codes √  √  √  √  √  √  Length       √  √    Project cost             Area type       Type of improvement √  √  √  √  Forecaster (person, agency, company) √  √  √  √  √  Year Forecast Made √  √  √  √  √  √  √  √  Forecast year(s) √  √  Opening Year √  √  √  √  √  Interim Year √  √  √  Design Year √  √  √  √  √    √  √  Unlabeled Year   √  √  Forecast value   √  ADT forecast √  √  √  √  √  √  √  √  √  Peak hour or K-values √  √  √  √  √    √  Turning movement forecast √  Segment or link identification √  √  √  √  √  √  √  √  DHV, Truck and ancillary data (sometimes forecasts, sometimes assumptions) √  √  √  √  √  √  √  Actual value √  ADT √  √  √  √  √  √  √  √  VMT             √  Turning movements   Year of observation √  √  √  √  √  √  √  Single value √  √  √  √  √  √  Multiple values √    √  Methodology Examined and/or Applied √  √  √  Historical trend or regression √  √  √  √  √  √  Population projections √  √  √  √  Travel model √  √  √  √ 

Traffic Forecasting Accuracy Assessment Research Guidance Document I-60 Other √  √  √  √  √  Relative error √  √  √  √  √  GEH √  √  Accuracy ratio √  √  √  Other √  Adjustment for differences in FY, AY √  * Florida DOT D5 and Kentucky Transportation Cabinet provided many traffic forecasts and ESAL reports that were not part of a database. Some sources have a limited amount of information in a spreadsheet or database, but the project information remains in associated ESAL or traffic projection reports:  Florida DOT District 5 provided numerous traffic project reports with reasonably complete information that were later added to the database.  KYTC also provided numerous ESAL and traffic projection reports, but it was difficult to determine when the projects opened and obtain matching count data, so those records were left out of our later analysis.  Virginia DOT provided multiple pieces of information about the project. For instance, there is an excerpt about each project and another file with the traffic counts, and a third spreadsheet with initial accuracy calculations. Including these data requires reviewing the associated reports and information to develop a complete data record, and that task was not undertaken as part of this research.  Data from international projects and from a Deep Dive in Massachusetts were later added to the database. Several observations can be made of these data:  Accuracy seems to be focused strictly on ADT values, even though multiple items are produced in the forecast;  Accuracy also seems to be evaluated at a single point in time – the opening year – with almost no investigation into years after the opening year;  Most sources contain data for most fields, but not necessarily the same fields; and  Sometimes when the fields are common, the definitions of the data they contain are not. For example, consider the type of improvement. One agency may have a binary classification of new road versus existing road, while another may have a more detailed classification that distinguishes between operational improvements, adding lanes, repaving, intersection changes, and other types of improvements. The first two points limited the analysis that could be conducted for this research, and the second two are important to the design of the database. The final column shows the columns that were recommended for inclusion in the combined database. A key goal of the combined database is to ensure a level of data standardization that allows data to be combined and compared across agencies. We accomplished this by implementing an initial combined database in Microsoft Access. This implementation was successful for the purposes of this research, but as we contemplated deploying it for future use, its limitations became clear. Specifically, it would be difficult to share across multiple users in different organizations.

Traffic Forecasting Accuracy Assessment Research Guidance Document I-61 A goal of this research is to establish a process for the continued evaluation of traffic forecast accuracy. To make this happen, it should be easy to conform to this data standard, and different users should be able to store their data as new projects are forecast or open without creating conflicts with other users. Microsoft Access is a desktop-based software package, with the idea that the data sit on a hard drive or a network drive. Any user that wants to add to the data, must have access to that drive, and the whole file gets updated. This makes collaboration difficult. As we identified the goals for the revised archive and information system, we wanted something with the following design features: 1. A standard data specification: A core goal of the archive and information system, as developed in the first version, is to establish a standard data specification for forecast accuracy data. This will allow data from different agencies to be stored in a common format and analyzed jointly. 2. Stable, long term archiving: The purpose of archiving forecasts is to allow them to be evaluated after the project opens. This requires that the solution provide stable storage for a number of years, and not be based on files that could be lost with staff or hardware changes. 3. Multiple users and data sharing: The solution should allow multiple users in different agencies to upload their forecasts and actual data, and should allow the data to be shared in a common format across those agencies. This will allow for combined analysis in the future, which is particularly important if the sample sizes are small for any particular agency. 4. Both a public and a private or local option for data storage: We recognize that some users may not wish to share their data publicly, so want the option to store the data privately or locally. 5. The ability to add files for the silver and gold standards: While we want the data in a tabular format for statistical analysis, we also wish to have a place to store supporting documentation or forecast reports for the silver standard, or model files for the gold standard. These should be linked to the tabular data records so the integrity of the tabular data can be checked against the supporting documentation. 6. Based on mainstream and low-cost software: The solution should be based on software that is mainstream, and does not pose a budgeting obstacle for potential users. As we begin to list these requirements, we looked for similar use cases in related fields. In terms of long-term and stable archiving, a similar problem occurs in scientific publishing. Increasingly, journals are requiring that the data supporting a research paper be archived and made available to future researchers. The preferred method for doing so is by uploading the data to a public data repository, which stores the data long-term with and provides access to download the data through a stable URL. Nature maintains a list of recommended data repositories for different fields at: https://www.nature.com/sdata/policies/repositories. These allow for any type of file to be stored, but are often set up to correspond to a specific snapshot of data, associated with the publication of a paper. Software development has a similar archiving problem, and there are repositories to make open-source software available, much as there are data repositories. However, software usually involves ongoing evolution, either for the development of new versions or to fix bugs, so it needs a strategy for managing those changes. Further complicating matters is that those changes are often made by different individuals, so it is important to develop a strategy to allow multiple users to

Traffic Forecasting Accuracy Assessment Research Guidance Document I-62 contribute to the software without creating conflicts. This task of managing the changes has usually facilitated in a version control system. A version control system tracks the changes made to files, as well as the user who made those changes. It allows the user to revert to a previous version of the files, making it easy to “undo” a change if it introduces an error. Usually the files involved are the software source code, but they can be any file. There are strategies for merging changes made by different users, although this task is easier if the changes are to different files stored in the same repository, rather than to one large file. Git (https://git-scm.com/) is a commonly used and free version control system. A version control system can be installed locally on a single machine, or it can be combined with a software repository which allows the files to be uploaded and archived to remote servers. Git works with either approach. A software repository can be public, and allow anyone to download the files, or it can be made private and password protected, such that the files are only shared with designated users. Github (https://github.com/) is a commonly used software repository that integrates with Git and is free for most users. A number of agencies that generate traffic forecasts use Git and Github to manage the code and inputs for their travel demand models, including the Metropolitan Transportation Commission (https://github.com/BayAreaMetro), the San Francisco County Transportation Authority (https://github.com/sfcta) and the Oregon Department of Transportation (https://github.com/ODOT-PTS). The purpose of the above diversion is to illustrate the commonalities with the problem at hand. A clearly defined data specification, combined with a version control system and a software specification meets the six major design goals outlined above. Further, it takes advantage of existing software tools (Git and Github) where possible. The resulting archive and information system is composed of two parts: one repository to store the data specification and associated scripts, and a second to store the data itself. Data Specification The first repository defines what we refer to as forecast cards. Forecast cards are a data specification for storing key information about travel forecasts and associated outcomes in order to evaluate the performance of a forecast, analyze the collective performance of forecasting systems and institutions, and identify contributing factors to accurate or inaccurate forecasts. Each “card” is a text-based CSV file. This makes them easy to edit without specialized software, it integrates well with version control because any chances can be clearly viewed (version control works on binary files too, but it will simply show that the file has changed, rather than highlighting specific lines or characters that have changed in a text file), and it means that “decks of cards” from different agencies can either treated separately or easily combined into a larger deck. There are five types of forecast cards:  Points of Interest, such as a roadway segment or transit line,  Projects, such as a specific roadway expansion,  Scenarios, including information about the forecasting system,  Forecasts, which are predictions at the points of interest about what the project will do, and

Traffic Forecasting Accuracy Assessment Research Guidance Document I-63  Observations, which are points of data (usually traffic counts) used to evaluate the forecasts Each type of card can be thought of as a table in a relational database, and each card a record in that table. The cards are related to each other as follows:  A point of interest defines a specific physical or planned location, such as Mulberry Road between Busytown and Pleasantville or Mulberry Road between Pleasantville and Workville.  A project is associated with a planned, physical project that may or may not ultimately be built. For example, the Mulberry Road Improvement Project.  A project may be associated with one or more scenarios. For example, it might include an opening year and a design year scenario, a build and a no-build scenario, or a two- lane and a four-lane scenario. Thus, scenarios are likely to correspond to alternatives considered and reported in a planning document.  Each scenario can be associated with one or more forecasts. A forecast is a prediction of a specific outcome, for a specific point of interest, at a specific point in time. For example, a forecast might predict 10,000 ADT (outcome) on Mulberry Road between Busytown and Pleasantville (point of interest) in the year 2020 (point in time). A related forecast might predict 8,000 ADT (outcome) on Mulberry Road between Pleasantville and Workville (point of interest) in the year 2020 (point in time). Both are associated with the four-lane opening year scenario for the Mulberry road Improvement Project.  It is also possible for multiple forecasts to be stored for the same scenario, outcome, point of interest, and point in time. For example, forecasts are commonly revised between initial planning and final design phases or analysts might consider a low, medium and high-growth forecast.  Observations are also associated with a specific outcome, point of interest and point in time, but they need not be associated with a specific project. It is quite possible to record a traffic count without a specific project being built. However, there is an option to match an observation with a specific forecast. Each type of card has some data fields that are required to contain data, and most have others that are optional. Categorical variables have specific categories defined to ensure consistency. For example, the options for project_type are: hov, road-capacity-expansion, transit-priority, road-diet, bike-facility, new-roadway, land-use, and other. Figure 8 shows the forecast card data schema. Required fields are in red and optional fields are in black. Blue text shows the options for categorical variables.

Traffic Forecasting Accuracy Assessment Research Guidance Document I-64 Figure 8: Forecast Card Data Schema

Traffic Forecasting Accuracy Assessment Research Guidance Document I-65 The forecast cards for a project are stored together in a single directory. Figure 9 shows an illustration of the CSV files associated with a single example project. Each rectangle represents a CSV file, with circles representing rows within the CSV file. Because the structure is to have one folder for each project, it is easy to add files beyond the required CSVs to that same folder. This would logically include a traffic forecast report or documentation conforming to the silver standard. For select files, it may also include the files necessary to reproduce the associated model runs, in conformance with the gold standard. This design provides an advantage over a traditional database format in that it is easy to add any type of file and have all the relevant information about a project stored in the same place. Figure 9: Forecast Card Illustration for an Example Project There is a balance in the data specification between keeping the required fields to a minimum, and thus making it easier to use, and being more extensive in case specific users may wish to expand the scope of their analysis. For example, this research focused exclusively on the accuracy of ADT forecasts, but it is reasonable to also consider the accuracy of peak period traffic volume forecasts or of travel time forecasts. Likewise, future users may wish to consider the accuracy of transit forecasts in addition to road traffic forecasts. Therefore, we have aimed to build a degree of flexibility into the data specification. That is why we consider a point of interest instead of a roadway segment, and why we allow several options for the forecast variable. We have not tested the specification for transit forecasts or other uses beyond the scope of this research, and we expect that it may evolve as other seek to extend it. Instructions for getting started with forecast cards and examples are included in the forecast cards repository at: https://github.com/e-lo/forecastcards. The repository also includes a validate_project script that checks to ensure that the forecast cards conform to the data specification. Scripts for combining the data for analysis are discussed in Chapter 4. Data Storage The above section describes the recommended data structure for archiving traffic forecasts and associated observations. It should be noted that the archived forecasts do not require significant amounts of storage, as the project database for this study (with over 16,000 segments representing more than 2,000 projects) required less than 50 megabytes. There is one folder for each project, along with CSV files storing key information conforming to the bronze standard. Additional files may be added to the same directory to meet the silver or gold standard for archiving forecasts. In its most basic form, this is simply a folder on someone’s hard drive or on a local network drive. However, this

Traffic Forecasting Accuracy Assessment Research Guidance Document I-66 does not necessarily ensure the long-term archival of the data, nor does it facilitate sharing outside the organization. Therefore, three options are provided for storing the forecast card data: local storage, a private repository and a public repository. Each are discussed below. Local storage is appropriate for agencies who do not wish to store their data to the cloud, or agencies that wish to upload their data only at regular time intervals (e.g., annual uploads). The data should be stored on a local network drive, and can be edited directly. Even with this option, we recommend that agencies use a version control system, such as Git, to create a local repository. The files are still edited directly, but the version control software will track the changes made and store a record of previous versions in a file on the same drive. This provides a degree of protection in the event that a change inadvertently introduces an error. A private repository is appropriate for agencies who wish to create an off-site archive of the data to protect it, but still want to restrict access to the data to select users. The files would still live on a local drive and be edited directly, but the version control system would be set up to also copy the files to a remote software repository (such as Github) and keep the local and remote versions in sync. Because two copies of the data are stored, it is unlikely they will be lost in the transition of local hardware or due to local hardware failure. This structure will also work well if multiple team members are involved in creating traffic forecasts and adding forecast cards. Jack and Jill can each add their own projects and upload them to the remote server. These changes are saved, along with the timestamp and name/email of the person who made the change. When they do this, the version control system will ensure that they also download any changes made by the other. Access can be granted to users outside the organization, such as consultants who create traffic forecasts, or researchers involved in analyzing the data. Access is only available if the user creates an account and the administrator grants specific read or write permission. A public repository is appropriate for agencies who wish to be fully transparent, and make their data available for use by others who may wish to analyze it. The structure is the same as with a private repository, and write access is restricted to those granted permission, but anyone with access to the internet can download a copy of the data. Each of the agencies who provided data to this project have agreed to have the data included in a public repository. These data are available at: https://github.com/gregerhardt/forecastcarddata. Each project is stored in a separate folder, and each folder starts with the name of the agency that provided the forecast. This makes it easy for data to be combined across multiple agencies, much like data can be combined from multiple users in the private repository option. This is the preferred option, because it maximizes the possibility that other researchers will continue to analyze these data, and that insights can be gained from a larger, combined data set. One limitation to note relates to file sizes. Version control systems store information the changes from all previous versions of files included in the repository. This is easy for text files and other small files, but if the files are extremely large it can be burdensome. Software repositories, such as Github, are typically free for basic users, but there may be limits on the total repository size before incurring data storage fees. Agencies wishing to follow the gold standard may need to account for this, either by paying for file storage, or by setting up the version control system to only store large model files locally.

Traffic Forecasting Accuracy Assessment Research Guidance Document I-67 Instructions for setting up forecast cards with these options are available at: https://github.com/e-lo/forecastcards.

Traffic Forecasting Accuracy Assessment Research Guidance Document I-68 Reporting Accuracy Results We recommend that agencies involved in producing traffic forecasts periodically report the accuracy of those forecasts relative to observed data. We recommend that this reporting include three components: 1. A short (~4 page) summary report updating the overall distribution of forecast error. This summary report would be written for consumers of traffic forecasts so they understand the range of errors to expect. The summary report would be updated approximately every two years as new projects open. 2. Estimating quantile regression models from local data. This step is optional, but doing this would allow the agency to use a more specific range of expected outcomes as a function of the forecast, rather than using the defaults produced by this research. 3. Specific Deep Dives, aimed at understanding the sources of forecast error for either typical or important projects. This step should occur as a precursor to travel model improvement projects, as discussed in Chapter 5. The remainder of this chapter provides guidance on how to go about reporting these results. Segment and Project Level Observations The research presented in the technical report presents results both at the segment level and a project level. A segment corresponds roughly to a link in a travel demand roadway network, and is usually defined as a section of roadway between major intersections. A project can include one or more segments. For example, a road expansion project might include segments that are linearly connected along a corridor. Alternatively, an interchange project might include segments on both directions of the mainline freeway, on the arterial roadway, and on each of the ramps. Thus, depending on the project, segments can be either quite homogenous, or different from each other. It is not immediately clear whether segments or projects should be considered an observation. Our preference is to aggregate the segments to projects before reporting the results, although there may be cases—such as if the interest is in a particular type of facility—where a segment level analysis is warranted. The project level aggregation ensures that projects with more segments do not dominate the results. Our preferred way to aggregate from the segment level to the project level is to sum the vehicle miles traveled (VMT) across segments that have both counts and forecasts. In our data, we do not always have the length of the segment, so we instead average the traffic volumes.

Traffic Forecasting Accuracy Assessment Research Guidance Document I-69 Summary Report There are several metrics used to report accuracy results. A popular metric used to determine the accuracy of traffic forecasts is the “half a lane” criterion. This criterion specifies that the forecast is accurate if the forecast volume varies from measured volume by less than half a lane’s capacity from the constructed facility’s capacity. If the forecast is more than half a lane less than the facility’s capacity, the facility could have been constructed with one fewer lane in each direction. If the forecast was more than half a lane than the facility capacity, the facility needs one additional lane in each direction. Calculating whether a forecast is within a half a lane requires several assumptions, such as the share of the daily traffic that occurs in the peak hour. Details of how we calculated the metric in this research are included in the accompanying Technical Report, but we recognize that agencies may wish to vary those assumptions based on local practice. Past research has commonly reported the percent error, defined as: PEi= Forecast - Counted VolumeCounted Volume *100% (3) Where PEi is the percent error for project i. A limitation of this standard metric is that it uses the counted volume as the reference point and the counted volume is not known until after the project opens. For this reason, we prefer to report the percent difference from forecast (PDFF), measured as: PDFFi= Counted - Forecast VolumeForecast Volume *100% (4) Using either PE or PDFF, a perfectly accurate forecast will produce a value of 0%. The absolute magnitude of the percentages describes to the magnitude of inaccuracy. With PDFF, the value with be negative if the actual outcome is lower than forecast, and the value will be positive if the actual outcome is higher than forecast. In this way, we can use forecast accuracy metrics to show the expected distribution of PDFF, and use that distribution to understand the uncertainty surrounding a forecast. This aligns better with the quantile regression models that we report, where we want to know whether the actual value is likely to be higher or lower than what we are forecasting at the time we make the forecast. As a measure of the average bias in forecasts, we recommend reporting the average PDFF. As a measure of the average error in outcomes, we recommend reporting the mean absolute percent difference from forecast (MAPDFF), which is analogous to the more commonly used mean absolute percent error (MAPE). Both prevent positive and negative values from canceling each other out. The MAPDFF is defined as: Mean Absolute Percent Difference from Forecast MAPDFF = 1 n *∑ |PDFFi|ni=1 (5) We also suggest reporting the 5th and 95th percentiles of the PDFF distribution, which provides a bound within which 90% of projects fall. For multiple projects, we recommend showing the entire distribution of the selected metric(s) in a frequency chart because using only the average or mean error

Traffic Forecasting Accuracy Assessment Research Guidance Document I-70 rates can mask important fluctuations. Figure 10 shows the distribution of the percent difference from forecast from this research. Figure 10: Distribution of Percent Difference from Forecast We recommend that the summary report include the following metrics:  The distribution of percent difference from forecast, similar to Figure 10;  The mean and median percent difference from forecast;  The mean absolute percent difference from forecast;  The 5th and 95th percentile percent difference from forecast; and  The percent of forecasts that are within half a lane. Optionally, agencies may wish to report these same metrics stratified across certain project characteristics to understand how they vary in relation to those characteristics. These details are of greater interest to technical staff and should be included in an appendix to the summary report if they are explored. Project characteristics that agencies may wish to explore include:  Traffic volume;  Functional class;  Improvement type;  Area type;  Forecast method;  Forecast horizon; 𝑃𝐷𝐹𝐹 Actual ForecastForecast ∗ 100

Traffic Forecasting Accuracy Assessment Research Guidance Document I-71  Comparisons to previous summary reports; and  Comparisons to the default data included in this research. For categorical variables, we recommend both a tabular reporting of the results, such as in Table 7, and violin plots, such as Figure 11. A violin plot shows the distribution of the percent difference from forecast on the vertical access, providing a simple way of visualizing the differences across categories. Appendix B of the technical report provides a more detailed anatomy of a violin plot. Table 7: Forecast Inaccuracy by Functional Class (Segment Level Analysis) Functional Class Observations Mean Absolute Percent Difference from Forecast Mean Median Standard Deviation 5 th Percentile 95 th Percentile Interstate or Limited Access Facility 434 12.32 -9.21 -8.48 13.58 -27.81 10.44 Principle Arterial 837 16.95 -9.63 -10.89 19.38 -37.51 23.95 Minor Arterial 404 18.92 -8.26 -10.24 24.54 -41.50 29.26 Major Collector 258 20.67 -10.81 -11.10 26.92 -51.11 23.85 Minor Collector 19 22.53 -12.74 -8.66 24.30 -41.43 28.58 Local 1 46.67 46.67 46.67 46.67 46.67 Unknown Functional Class 1958 32.42 10.69 2.68 53.67 -48.75 86.21 Figure 11: Distribution of Percent Difference from Forecast by Functional Class (Segment Level Analysis)

Traffic Forecasting Accuracy Assessment Research Guidance Document I-72 Updating Quantile Regression Models As discussed in Chapter 2, quantile regression models can be used to estimate the uncertainty window around a forecast. A set of default models is provided along with this guidebook. However, if an agency has collected data on traffic forecast accuracy, the quantile regression models can be re- estimated using local data. Doing this is advantageous because it is based on data that are likely more similar to the types of forecasts that an agency is will continue to perform. The task is to estimate a model of the actual traffic as a function of the forecast traffic. This provides a model that can be used to predict the range of expected traffic if the forecast is known. The recommended model form is: 𝑦 , 𝛼 𝛽 𝑦 𝛾 , 𝑋 , 𝑦 𝛾 , 𝑋 , 𝑦 . . . 𝛾 , 𝑋 , 𝑦 𝜀 , (5) where 𝑦 , is qth percentile of the actual (expected) traffic on project i, 𝑦 is the forecast traffic on project i, and 𝜀 , is a random error term. 𝛼 is an estimated constant and 𝛽 is an estimated slope. 𝑋 , through 𝑋 , are descriptive variables associated with project i, and 𝛾 , through 𝛾 , are estimated model coefficients associated with those descriptive variables and those quantiles. Each is multiplied by 𝑦 which makes the effect of that variable scale with the forecast volume (i.e. change the slope of the line) rather than be additive (i.e. shift the line up or down). For example, consider a median model where 𝛼 is 0, 𝛽 is 1 and there is a single descriptive variable, 𝑋 , , which is a binary flag which is 1 if the forecast is for a new road, and 0 otherwise. If 𝛾 has a value of -0.1 then it means that the median actual value would be 10% lower than the forecast. If 𝛾 has a value of +0.1 then it means that the median actual value would be 10% higher than the forecast. and ε_i is a random error term. At a minimum, models should be estimated for the 5th, 50th and 95th percentiles, with the option to include additional percentiles. The main difference between understanding a standard regression model and understanding a quantile regression model is that for each coefficient (𝛼 , 𝛽 and 𝛾 , through 𝛾 , ) there is one estimated coefficient for each percentile considered (q), rather than a single coefficient. For consistency, we recommend the same model specification across all the percentiles, even if the coefficients are insignificant for some percentiles. Some guidance on expected coefficient values are discussed in the following bullet points. For simplicity, these points refer only to the 5th, 50th, and 95th percentiles, although the expectations are similar for any percentile less than the 50th or greater than the 50th.  If the median is unbiased, 𝛼 should be zero, 𝛼 should be negative and 𝛼 should be positive. All are in the same units as the forecast variable (ADT in this research).  If the median is unbiased, 𝛽 should be one, 𝛽 should be less than one and 𝛽 should be greater than one. Each serves as a scaling factor on the forecast. Values closer to one indicate a narrower range, and values farther from one indicate a wider range.  If a descriptive variable 𝑋 , has no effect, 𝛾 , should be zero.  If 𝛾 , has the same sign for all the percentiles, it indicates that the effect of that variable is to shift or bias the results in that direction. Alternatively, it is possible that only the median estimate is biased, as would be indicated by the value of 𝛾 , .

Traffic Forecasting Accuracy Assessment Research Guidance Document I-73  If 𝛾 , is negative and 𝛾 , is positive, it indicates that the variable serves to expand the range of expected forecasts, and is associated with more uncertainty.  If 𝛾 , is positive and 𝛾 , is negative, it indicates that the variable serves to narrow the range of expected forecasts, and is associated with less uncertainty. The default model specifications can be used as a starting point for local models. Additional variables can be tested and evaluated based on the logic of the resulting coefficients and their statistical significance. Detailed guidance on the art of model specification is beyond the scope of this guidebook. A number of statistical packages can be used to estimate quantile regression models. R was used for the models developed in this research. As described in the next section, Python scripts for estimating basic quantile regression models are included as a starting point for future analyses. It is important that projects used to develop the quantile regression equations be (1) sufficient in quantity to produce statistically significant coefficient estimates and (2) representative of all the types of forecasts made. If an agency does not have a sufficient sample of local projects to support model estimation, it should supplement their local data with data from projects at peer agencies. The data provided with this report can be used. It is also recommended to use a census of all (not a sample) projects to the extent possible. This will avoid “cherry picking” highly-accurate or -inaccurate forecasts. The next section describes scripts that are provided to compile forecast accuracy data in a format suitable for quantile regression estimation. Using the Forecast Archive and Information System Summarizing forecast accuracy metrics as described above is straight-forward if the data are available in a clean format. By “clean”, we mean that the data are provided in a table with one record for each observation (project or segment), and at a minimum with one column for the forecast traffic volume and one column for the actual traffic volume. Other fields can be included if they are of interest for segmenting the data, or as descriptive variables in quantile regression. The data should only include projects that are open and have both forecast and actual traffic volumes, they should avoid duplicates, and they should be filtered to include only the forecasts of interest (for example opening year forecasts if design year is to be excluded). Once they are in this format, standard statistical software or a spreadsheet can be used to calculate the above metrics. As with many statistical analyses, the bulk of the effort is in assembling and cleaning the data. Chapter 3 described the bronze level of archiving forecasts and associated actual data, and it described a forecast archive and information system to facilitate the storage of these data using the forecast cards structure. If agencies follow this standard, they will acquire the data necessary to assess forecast accuracy locally. If they and others share the data, the assessments can combine local and external information. By separating projects into separate folders and the related data into separate CSV files, the forecast cards data structure is optimized to make it easy to store and share the data. However, separate forecast cards are not well suited to analyzing the data—for that, a single combined file with the relevant fields is better. Therefore, we have developed python scripts to convert from forecast cards into a flat file format.

Traffic Forecasting Accuracy Assessment Research Guidance Document I-74 The main data_validation_and_preparation script follows the process used to clean the data for this research and includes the following steps: 1. Get and validate the forecast cards. This step reads the forecast cards from a designated repository and checks that they conform to the data specification and have the required fields coded appropriately. 2. Combine data from multiple sources. In this step, data can be combined from multiple repositories. This could include local data from a hard drive, and data pulled from a shared public repository. 3. Combine and format data for estimation. The data are combined into a single table with one record per observation and the appropriate fields. 4. Filter records. The data are filtered to exclude projects that have not yet opened, do not have observations, or otherwise are problematic. 5. Explore the estimation data set. Print summary statistics for the combined dataset. As an extension to this, the resulting data can be used to estimate new quantile regression models, as opposed to using the default models provided with this research. Users may want to do this either to update the models with new data as more projects open, or to estimate models from local data reflecting the accuracy of their own agency’s forecasts. These may result in uncertainty windows that are narrower or wider than the defaults. Once the data are formatted with the data_validation_and_preparation script, the process of estimating new quantile regression models is similar to estimating linear regression models where the actual value is the dependent variable and the forecast value is the descriptive variable. An estimate_quantiles script is provided to estimate a basic model with just this one variable. The spreadsheet that implements quantile regression models can be updated with new estimation results to test their effect. Instructions for running the data_validation_and_preparation and estimate_quantiles script are available at: https://github.com/e-lo/forecastcards. Deep Dives A Deep Dive is a case study of the accuracy of a forecast for a particular project. It can only be performed after a project has opened to traffic and requires the passage of time from the original forecast. The goal of a Deep Dive is to investigate the sources of forecasting error or reasons for forecasting accuracy of a project, using a retrospective look. Our recommended methodology is: 1. First, record all information about the forecast this should be done shortly just after the forecast has been finalized, as described in Chapter 3 of this guidebook. For a Deep Dive to be successful, the forecast should be archived using either the Silver or Gold levels. 2. Next, collect data about the forecast itself and the inputs and assumptions behind the forecast. The data plan specified in the Silver level annotated outline can be executed after project opening.

Traffic Forecasting Accuracy Assessment Research Guidance Document I-75 3. Then, compare project forecasts with actual values to identify accurately and inaccurately forecasts. The comparison of project forecasts and actual volumes should account for the difference in the assumed and actual opening years, if any. This will avoid focusing on errors caused by construction or funding delays. 4. Compare inputs and assumptions with actual values to categorize the accurate and inaccurate assumptions behind the forecast. An initial comparison can include typical or ordinary inputs and assumptions common in most traffic forecasts: population estimates, auto fuel prices, changes in land use, travel times, and other variables. Since these variables are commonly used in traffic forecasts, it is reasonable to assume that errors in these assumptions will contribute to the inaccuracy of the forecast. Once these are complete, a secondary comparison can be made with project-specific assumptions, such as the model’s grasp of travel markets at the time the forecast is generated and the model’s grasp of expected travel markets. The results of both comparisons should be documented. 5. Adjust the inputs and assumptions and generate adjusted forecasts by re-running the full model or using elasticities. The best method to assess the causes of forecast inaccuracy is to correct the inputs and assumptions and reproduce the forecast using the same method, usually a travel demand model, that developed the original forecast. In this way, the cumulative impacts from the corrected inputs will be estimated, resulting in an adjusted forecast. Ancillary forecasts, which correct just one input, can be produced to enhance understanding of the causes of error. This option is not available for forecasts made using traffic count or population trendlines. For these situations, computing the forecasting error for this method is the only comparison that can be made. Forecasting error from trendlines can be compared to the error from other methods to determine the best methods for certain project types. A good analysis will include the differences in costs and resources needed to generate the forecasts along with the forecast error to produce a complete picture. Elasticities measure the change in the traffic forecast given a known change in an input or assumption to the forecast. Elasticities are usually expressed as ratios. An elasticity of +0.3 means that the traffic forecast would increase by 3% for every 10% increase in the input value. Conversely, an elasticity of -0.2 means that the traffic forecast would decrease by 2% for every 10% increase in the input value. The equation for adjusting the forecast is: Elasticities are general rules of thumb and assume that the forecast changes entirely due to the change in one variable. For the elasticity analysis to be effective, the forecasts must be changed cumulatively for every variable. The following example shows how the computations can be applied. Effect on Forecast 𝑒𝑥𝑝 𝐸𝑙𝑎𝑠𝑡𝑖𝑐𝑖 𝑡𝑦 ∗ 𝑙𝑛 1 𝐶ℎ𝑎𝑛𝑔𝑒 𝑖𝑛 𝑉𝑎𝑙𝑢𝑒 1 Adjusted Forecast 1 𝐸𝑓𝑓𝑒𝑐𝑡 𝑜𝑛 𝐹𝑜𝑟𝑒𝑐𝑎𝑠𝑡 ∗ 𝐴𝑐𝑡𝑢𝑎𝑙 𝐹𝑜𝑟𝑒𝑐𝑎𝑠𝑡 𝑉𝑜𝑙𝑢𝑚𝑒

Traffic Forecasting Accuracy Assessment Research Guidance Document I-76 Figure 12: Example of Adjusting Forecasts using Elasticity Computations Seg#  Items  Actual Value  Forecast  Value  Change  required in  Forecast  Value  Elasticity  Effect on Forecast  Actual  Forecast  Volume  Adj  Forecast  Volume  Remaining %  Error for Adj  Forecast  1  Employment   38,801   48,312  ‐20%   0.30  ‐6%   10,262   9,609  13%  1  Population/Household    78,576   80,854 ‐3%   0.75  ‐2%   9,609   9,405  11%  1  Car Ownership   54,603   56,084 ‐3%   0.30  ‐1%   9,405   9,330  10%  1  Fuel Price/Efficiency    $2.340    $1.820  29%   (0.20)  ‐5%   9,330   8,872  5%  1  Travel Time/Speed   ‐   ‐  0%   (0.60)  0%   8,872   8,872  5%  1  Original Traffic  Forecast   8,474   10,262  21%   N/A   N/A  1  Adjusted Traffic  Forecast  N/A  N/A   N/A   10,262   8,872  5%  6. Compare and analyze the adjusted forecasts with actual values to determine the major sources of error or accuracy. Some key questions might need to be addressed in the analysis are: Would the project decision have changed if the forecast accuracy or reliability were improved? How useful were the forecasts in terms of providing the necessary information to the planned process? Was risk and uncertainty considered in the forecast? How was it communicated? Based on the analysis, changes to the forecasting methodology or model validation methodology can be suggested for future forecasting efforts. 7. Finally, summarize the findings in a summary memo format so that the results can be easily recalled when needed. An annotated outline for Deep Dive analyses can be found in Appendix B. Electronic versions of the outline and spreadsheets can be found <<<here>>>.

Traffic Forecasting Accuracy Assessment Research Guidance Document I-77 Improving Traffic Forecasting Methods We recommend that planners and engineers consider the results of past accuracy assessments in improving traffic forecasting methods. Building from that broad recommendation, this chapter describes three specific ways in which traffic forecast assessments can be used to guide method improvements. We use the term traffic forecasting methods to refer both to the breadth of tools used, and to the full range of activities involved in generating a forecast. Travel demand models are a common method of traffic forecasting, and much of the focus in this chapter is on improving those models. However, it is important to recognize that creating a project-level traffic forecast may also involve creating land-use inputs for those models, assembling inputs to those models, making assumptions about factors such as expected gas prices, and pivoting the results from base-year traffic counts. If the weak points of the forecasting process lay in one of those activities, rather than in the model itself, then that activity should be the focus for improvement. Using Deep Dives to Guide Model Improvements Chapter 4 describes a process for conducting a Deep Dive into the forecast accuracy associated with a specific project. One goal of a Deep Dive is to better understand what may contribute to error in the forecast. For example, the Deep Dives conducted for this research considered how much the traffic forecast would change if it used the actual instead of forecast population and employment inputs, if travel speeds were correctly predicted, or if external traffic were better forecast. If the traffic forecast were to improve a great deal with those changes, then that indicates that improvements in those specific areas may be valuable. Conversely, if the traffic forecast changes very little with improvements in one of those areas, then it may not be worth expending additional resources to further refine that step. For example, if a several Deep Dives were to show that the largest source of error in traffic forecasts is associated with the employment forecasts, it may be more important to improve the socio-economic models than to improve the travel models. It is logical that this task would be performed in prior to a model development project, because that is a time where resources are already being allocated to improving forecasting methods, and this analysis may serve to inform the allocation of those resources. Specifically, we would recommend that a first task in a model development project should be to conduct three to five Deep Dives for projects that are already open, but are similar to those that will be forecast again in the future. These

Traffic Forecasting Accuracy Assessment Research Guidance Document I-78 would be conducted to evaluate the strengths and weaknesses of the existing forecasting process, with the analysis informing where that process most needs improvement. This is not to suggest that such accuracy assessments would be the only mechanism for guiding new model improvements. It remains important that new models be sensitive to the types of policies and projects expected in the future, even if those are different than the types of policies considered in the past. It is also appropriate that new models should incorporate improved methods both for theoretical reasons and in ways that take advantage of new data or computing resources. However, we suggest that past accuracy be considered on par with these other factors. Some may question the value of evaluating past forecasts at precisely the time that a new forecasting tool is being developed. Would the findings not be irrelevant if a new tool is being developed anyway? While we also believe that testing a new model in a similar capacity is important, and we recommend doing so in the next section, forecasting is unique precisely because the analyst does not know the outcome a priori. Therefore, it is an opportunity to learn about what unanticipated changes or risks may have arisen in a way that is not possible when the outcome is known. There may be important insights gained from that opportunity. Project-Level Testing and Validation Our second recommendation focuses on testing and improving newly developed travel models by testing their ability to correctly predict the changes that occur when a project opens. This would rely on projects that have already opened, using data archived as described in Chapter 3. The Travel Model Validation and Reasonability Checking Manual (the Manual) (Cambridge Systematics, Inc. 2010) recommends temporal validation and sensitivity testing, and project-level validation is one specific form of temporal validation. The Manual defines temporal validation tests as comparisons of model forecasts or backcasts against observed travel data, and sensitivity tests as evaluations of model forecasts for years or alternatives for which observed data do not exist. In our experience, we have commonly observed temporal validation tests to occur at a regional level. For example, a travel model developed with a 2015 base year may be applied to 2010 conditions and compared to 2010 traffic counts by area type and facility type. We also observe that sensitivity testing often happens at a project-level, meaning that a specific alternative is coded, and the model is run with and without that alternative, comparing the build and no-build cases to make sure the result is reasonable. Both are valuable, but come with their own limitations. A challenge of temporal validation as described is that many factors change at once—population, employment, networks, technology changes and so forth—making it difficult to isolate the effects of specific projects. In addition, a focus at the regional level can make it difficult to detect the effects of individual projects. In contrast, sensitivity testing of specific alternatives allows the project effects to be isolated, and in some ways is more comparable to the way in which the model will be applied for project-level forecasting. However, sensitivity testing is limited because it does not involve a comparison to observed data. What we propose here is a project-level temporal validation (sometimes referred to as longitudinal or dynamic validation) that involves testing the model for a specific change that has already occurred and comparing the model’s predictions to observed outcomes.

Traffic Forecasting Accuracy Assessment Research Guidance Document I-79 There is precedent for such an approach in the before-and-after studies required by the Federal Transit Administration’s (FTA’s) New Starts program. In order to receive federal funding for new transit infrastructure projects—generally rail or other fixed guideway systems—project sponsors must conduct a before-and-after study to measure the effects of the project. This involves collecting detailed transit ridership data both before and after the project opens. Part of that study involves the evaluation of the transit ridership forecasts (an evaluation of the past forecasts, as described in the previous section). In some cases, such as following the opening of the Central Light Rail Line in Phoenix, that same data has been used to validate new travel models by running those new models with and without the project, and comparing to the change in the observed data. Those comparisons were used to diagnose issues in the new models and guide their resolution. This type of analysis is valuable because we develop models to predict change, and it is appropriate to evaluate them on their ability to predict change. If we are merely interested in what is happening in the base year, we could collect data instead. Such an analysis is predicated on having the data available against which to compare the model results, and it is that data that distinguishes it from a sensitivity test. If forecasting agencies begin to systematically collect data on the projects that they have forecast and that have opened, they will also be compiling a library of relevant projects against which to test new models. A few details matter here in terms of what is collected. For the actual outcomes, there is value in collecting traffic counts and associated data not only after the project opens, but before. This allows the project effects to be better isolated, and is the approach used by Highway England’s Post-Opening Project Evaluations (POPE) (Highways England 2015). Second, archiving appropriate detail about the scope of the project itself is important. A map can be valuable so the networks can be coded appropriately. This would be further facilitated by the use of the Gold archiving standard for select projects. If the model runs for a project are available, it is likely that the inputs in terms of socio-economic data and networks can be converted for use in a new model with modest effort. One of the challenges to backcasting efforts is the effort of coding networks and compiling inputs, and if those are properly archived and available, the cost to do such tests is lower. A logical approach to conducting such tests is to look for symmetry with the Deep Dives described in the previous section. If a model development project starts with an evaluation of the performance of the forecasts created using the old model for 3-5 projects, it can end with an evaluation of the performance of the new model for those same 3-5 projects. If the new methods are better able to replicate the observed changes that occur when those projects open, such a direct comparison to the same set of benchmarks would make a compelling case for the value of the new tools. Large-N Analysis for Method Selection The third way that traffic forecast accuracy evaluations can be used to improve traffic forecasting methods is to use Large-N analysis to determine whether some methods produce more accurate forecasts than others. There are many cases in traffic forecasting where alternative methods can be used. This could be traffic count trends versus a travel model, a 4-step travel model versus an activity-based travel model, a multinomial logit versus a nested logit mode choice model, a static versus dynamic traffic assignment model, and so forth. In many cases there are good reasons to prefer one method over

Traffic Forecasting Accuracy Assessment Research Guidance Document I-80 another. Perhaps one approach has a stronger theoretical foundation. One might provide the sensitivity to test a broader range of policies. For example, a time-of-day choice model might provide the ability to test the effect of peak spreading, whereas static time-of-day factors would not. It may be that one method is preferred due to ease of use or better computational efficiency. These are all legitimate considerations in selecting forecasting methods, but we propose that the ability to produce accurate forecasts should be a key consideration in method selection. Unfortunately, there is limited empirical evidence to guide such decisions. In talking with traffic forecasters, there is plenty of professional experience with the range of such methods, and often a general sense that they work reasonably well, but there is little evidence that has been compiled systematically. This research project represents a start in that direction in that it compiles forecast and actual traffic data at the project-level. We were able to gain some insight into the accuracy of different methods, for example finding that travel models tend to be more accurate than traffic count trends. However, we regret that this analysis remains limited. The challenge is two-fold. First, we were only able to record the method used at a high-level, such as: travel model, traffic count trend, population growth rate, or professional judgment. Even at this high level, there are a substantial number of projects in our database where the method is not recorded. Second, the use of certain methods is not randomly assigned within the data. Instead it is correlated with the agency that produces the forecast, and often with the type of project. This shows up in the difference between some of the older projects in our database, which tend to be larger infrastructure projects, and some of the newer projects which tend to be more routine. From a research design perspective, this introduces a potential for confounding across these variables. If a certain method is used for routine projects and performs better, is it because the method is better or is it because the project is easier to forecast? This research design problem can be managed in one of two ways. First, the confounding factors can be introduced as control variables in the statistical analysis. This is the approach used when estimating quantile regression models for this research, and provides an advantage over the univariate distributions as shown in the violin plots in the Technical Report. Such an approach works best if 1) the data are more complete such that there are fewer fields with missing data, and 2) the variables used in the model are less correlated with each other. Both of these can be improved somewhat if data are added from more diverse sources and if those data are as complete as possible. The second, and more robust mechanism for addressing this is to explicitly design the sample to control for potential confounding effects. One straightforward way to do this is to compare forecasts made by multiple methods for the same project. In such a research design, everything but the method is controlled for. There are some cases where this may occur naturally. For example, in the traffic forecast reports from Florida DOT, the forecaster will use both a traffic count trend and a travel model, then select the forecast they deem most reasonable. Because both are documented, there is an opportunity to go back and compare the two methods for the same project. Similarly, many FTA New Starts forecasts involve a transit ridership forecast developed using a locally- developed travel model, but then checked against FTA’s Simplified Trips-on-Project Software (STOPS) model. This can apply more broadly when an agency is interested in understanding the uncertainty involved in a forecast by checking against an independent method. Such an approach would be greatly facilitated by the development of models, such as STOPS, that can be readily deployed in multiple regions. It is reasonable to envision a semi-standard model that can be deployed in multiple regions with modest effort. In fact, recent TRB reports have focused

Traffic Forecasting Accuracy Assessment Research Guidance Document I-81 on a portion of this problem, including NCHRP 735: Long-Distance and Rural Travel Transferable Parameters for Statewide Travel Forecasting Models (Schiffer 2012) and SHRP 2 Report S2-C10A-RW-2: Transferability of Activity-Based Model Parameters (Gliebe et al. 2014). To be practical, such models would require a certain level of standardization in their input data, with an effort currently underway to define “Datasets of standardized input and observed data in order to facilitate the testing of and compare the performance of various algorithms, approaches, or parameters” (Zephyr Foundation https://zephyrtransport.org/projects/2-network-standard-and-tools/). The eventual goal would be to move towards a set of “interchangeable parts” models where different methods could be readily tested for the same project-level forecast. The ability to test different methods for the same set of forecasts, combined with the ability to evaluate which method forecast better, would provide a strong foundation for guiding the continued improvement of traffic forecasting methods in a scientifically sound manner.

Traffic Forecasting Accuracy Assessment Research Guidance Document I-82 Implementation and Future Research This research provides a starting point for a broader body of work on traffic forecast accuracy. Such assessments have always been limited by available data, largely because it is cumbersome to compile data on forecasts several years after they are made. This document provides guidance on how to archive traffic forecasts and associated data with the goal of making the activity efficient so it will be done systematically. It also provides guidance on how to use those data. The natural next step is to implement these recommendations. Appendix E is an implementation plan which outlines steps to be taken to promote the implementation of these recommendations. Such implementation may itself involve further research. As discussed previously, the data used in this study were selected based on availability, and users may find value in analyzing projects specifically relevant to their situation. The starting point may be further analysis using the data from this research, which is available as described in Chapter 3. Because most of the projects in the database are not yet open, the sample of projects available for analysis increases with each passing year. This may also involve analysis of data compiled by agencies from their own traffic forecasts, which may more directly relevant to their interests. As the analysis of forecast accuracy continues, there will undoubtedly be new insights gained. However, such work need not be limited to the recommendations here. There is ample room to expand the scope of the analysis conducted, with several possible directions listed below. 1. In addition to a comparison of post-opening counts to the project forecast, also compare the pre-opening counts to the base year or no-build forecast. Such an assessment should focus on the change between the pre- and post-opening volumes. 2. Measure forecast versus actual outcomes on screen lines to provide insight into whether any inaccuracy is attributable to a trip table prediction problem or an assignment problem. 3. Evaluate the forecast versus actual outcomes not only on the project itself, but aslo in comparison to broader VMT trends throughout the region. This would provide additional insight into whether any inaccuracy is specific to the project or regional in nature. 4. Consider forecast outcomes beyond ADT, such as peak period traffic volumes, roadway speeds and truck volumes. 5. Expand the research to include additional types of projects, such as transit projects, toll roads and express lanes.

Traffic Forecasting Accuracy Assessment Research Guidance Document I-83 6. Examine design year forecasts in addition to opening year forecasts. 7. Consider side-by-side test of competing methods to directly compare the accuracy of each. 8. Compile the data necessary to test a broader range of methods. 9. Further consider the implications for the large N analysis of missing data fields being correlated with specific agencies providing the data. 10. Test the importance of clustering effects in the data. The statistical methods used here assume that each observation is independent, but in fact some may be correlated. This issue is most prominent in the segment-level analysis, where it is reasonable to expect each segment to be correlated with other segments from the same project. There are good examples of the first four extensions in Highway England’s Post-Opening Project Evaluations (POPE) of Major Schemes. For example, Figure 13 a comparison of the after-minus- before forecast versus the after-minus-before counts, highlighting that this particular forecast does a reasonably good job of predicting the change in traffic. The POPE reports provide numerous examples of post-opening evaluations, not only of the accuracy of forecasts, but also of the effectiveness of the project against its stated goals. They are available here: https://www.gov.uk/government/collections/post-opening-project-evaluation-pope-of-major- schemes. Figure 13: Example Pre- and Post-Opening Traffic Forecast Comparison (Source: Atkins 2017) Additional research along these lines will continue to inform our understanding of the uncertainty around traffic forecasts, as well as provide the opportunity to reduce that uncertainty as well as any bias present.

Traffic Forecasting Accuracy Assessment Research Guidance Document I-84 References Atkins. (2017). Post Opening Project Evaluation: A23 Handcross to Warninglid One Year After Study. Highways England. Cambridge Systematics, Inc. (2010). Travel Model Validation and Reasonableness Checking Manual: Second Edition. Federal Highway Administration. Donnelly, R., Erhardt, G. D., Moeckel, R., and Davidson, W. A. (2010). Advanced practices in travel forecasting. National Cooperative Highway Research Program, NCHRP Synthesis 406, Transportation Research Board. Flyvbjerg, B. (2007). “Policy and planning for large-infrastructure projects: problems, causes, cures.” Environment and Planning B: Planning and Design, 34(4), 578 – 597. Flyvbjerg, B., Holm, S., K, M., and Buhl, S. L. (2006). “Inaccuracy in Traffic Forecasts.” Transport Reviews, 26(1). Galileo Galilei. (1638). Dialogues concerning two new sciences. Macmillan, New York. Gliebe, J., Bradley, M., Ferdous, N., Outwater, M., Lin, H., Chen, J., Strategic Highway Research Program, Strategic Highway Research Program Capacity Focus Area, Transportation Research Board, and National Academies of Sciences, Engineering, and Medicine. (2014). Transferability of Activity-Based Model Parameters. Transportation Research Board, Washington, D.C. Highways England. (2015). Post Opening Project Evaluation (POPE) of Major Schemes: Main Report. International Standards Organization (ISO). (n.d.). Quality management principles. 20. Kim, Y., and Stanton, J. M. (2016). “Institutional and individual factors affecting scientists’ data-sharing behaviors: A multilevel analysis.” Journal of the Association for Information Science and Technology, 67(4), 776–799. Parthasarathi, P., and Levinson, D. (2010). “Post-construction evaluation of traffic forecast accuracy.” Transport Policy, 17(6). Schiffer, R. G. (2012). Long-Distance and Rural Travel Transferable Parameters for Statewide Travel Forecasting Models. NCHRP Report, Transportation Research Board, Washington, D.C. Wachs, M. (1990). “Ethics and Advocacy in Forecasting for Public Policy.” Business and Professional Ethics Journal, 9(1 & 2), 141–157.

Traffic Forecasting Accuracy Assessment Research Guidance Document I-85 Author Acknowledgments Funding: This research was funded by the Transportation Research Board as National Cooperative Highway Research Program (NCHRP) Project 08-110: Traffic Forecast Accuracy Assessment Research. Author Contributions: Greg Erhardt and Dave Schmitt were principal investigators for this research, and are responsible for designing and directing all aspects of it. Jawad Hoque was the primary research assistant throughout the project, conducting much of the Large-N analysis and the Cynthiana Deep Dive. Ankita Chaudhary developed the Large-N database, and conducted the Indian River Street Bridge Deep Dive. Kyeongsu Kim conducted the Central artery Tunnel and US 41 Deep Dives. Sujith Rapolu conducted the Eastown Road Deep Dive. Marty Wachs facilitated the project workshop, helped to craft conclusions and recommendations, and wrote the initial executive summary. Mei Chen oversaw the Large-N statistical analysis, and identified quantile regression as an appropriate tool. Reg Souleyrette directed the Cynthiana Bypass Deep Dive. Steve Weller conducted the Southbay Expressway Deep Dive. Elizabeth Sall led the development of the forecastcards archiving system. Data: Thank you to the following individuals and organizations who provided data to support this research and agreed to make relevant project data publicly available: Mark Byram and Greg Giaimo, Ohio DOT; Chris Chritton, Wisconsin DOT; Pavithra Parthasarathi, Puget Sound Regional Council (PSRC) and David Levinson, University of Sydney (Minnesota data); Don Mayle, Michigan DOT; Shi-Chiang Li and Hui Zhao, Florida DOT District 4; Jason Learned, Florida DOT District 5; Jonathan Reynolds, Kentucky Transportation Cabinet; John Miller, University of Virginia; Morten Skou Nicolaisen, City of Aarhus (European data); Clint Daniels and Peter Stevens, San Diego Association of Governments (Southbay Expressway). Workshop: Thank you to those who participated in the project workshop: Chris Hiebert, Southeast Wisconsin Regional Planning Commission; Brad Lane, Delaware Valley Regional Planning Commission; Nokil Park, Atlanta Regional Council; Thomas Hill, Florida DOT; Mark Byram, Ohio DOT; Amir Shahpar, Virginia DOT; Ed Azimi, Virginia DOT; Chowdhury Siddiqui, South Carolina DOT; Juan Robles, Colorado DOT; Chris Chritton, Wisconsin DOT; Don Mayle, Michigan DOT; Jonathan Reynolds, Kentucky Transportation Cabinet; Lori Duguid, Michael Baker International; Brian Gardner, Federal Highway Administration; Ken Cervenka, Federal Transit Administration; Larry Goldstein, Transportation Research Board.

Traffic Forecasting Accuracy Assessment Research Guidance Document I-86 TRB Workshop: Thank you to the organizers, speakers and participants in the workshop on Progress in Improving Travel Forecasting Accuracy at the 2019 Transportation Research Board Annual Meeting, including: David Hartgen, The Hartgen Group; Kay Axhuasen, ETH-Zurich; Ken Cervenka, Federal Transit Administration; Maren Outwater, RSG; Greg Giaimo, Ohio DOT; Julie Dunbar, Dunbar Transportation Consulting. Other Acknowledgments: Thank you to others who discussed this research with us, including: Xu Zhang, University of Kentucky; David Hartgen, The Hartgen Group; Rob Bain, RB Consult; and John Miller, University of Virginia. We are indebted to all of the individuals listed above, as well as any we may have neglected to mention, for helping to craft this research and the resulting recommendations.

Traffic Forecasting Accuracy Assessment Research Guidance Document I-87 Appendix A: Electronic Resources Several electronic resources accompany this Guidance document, as described below. Quantile Regression Spreadsheet NCHRP 08-110 Quantile Regression Models.xls This spreadsheet demonstrates the application of the quantile regression models, as described in Chapter 2. The first sheet in the file contains additional details on how to use the spreadsheet. ForecastCards Repository https://github.com/e-lo/forecastcards This repository contains the schema for archiving forecasts according to the Bronze standard, as described in Chapter 3. The schema defines the structure of the files and the fields to store. The repository also contains Python scripts to perform several functions: 1. Validate data to ensure that it conforms to the schema. 2. Clean and merge the data into CSV flat file appropriate for statistical analysis and model estimation. 3. Create any additional categorical or derived variables on that flat file. 4. Summarize data and estimate quantile regression models Chapter 3 describes the process at a high level, with more detailed user documentation incorporated into the repository itself. The repository is version controlled, with Release 1.0 corresponding to the publication of this report. This structure ensures both that the release will continue to be available and that future users can build upon it. ForecastCardData Repository https://github.com/gregerhardt/forecastcarddata This repository contains the actual data used in Part 2 of this report. The processes contained in the ForecastCards repository can be used to compile these data into a CSV flat file suitable for reproducing the results presented here. Release 1.0 corresponds to the publication of this report, but

Traffic Forecasting Accuracy Assessment Research Guidance Document I-88 more data can be added, and more data may become relevant as projects continue to open. In this way, we hope it will serve as a living resource. Forecast Archive Annotated Outline Appendix B- Forecast Archive Annotated Outline.docx This document serves as a template for documenting details about a forecast as described in the Silver standard in Chapter 3. It is also included as Appendix B, but is available in Word format for ease of editing. Deep Dive Annotated Outline Appendix C- Deep Dive Annotated Outline.docx This document serves as a template for documenting Deep Dives as described in Chapter 4. It is also included as Appendix C, but is available in Word format for ease of editing.

Traffic Forecasting Accuracy Assessment Research Guidance Document I-89 Appendix B: Traffic Forecast Preservation Annotated Outline (Silver Standard) 1. Introduction This report, written in month-year, documents the traffic forecasts and supporting assumptions for the <project name>. The information in this report will be the primary source of information used to record the accuracy of the traffic forecast(s) and determine whether the assumptions used as a basis for the forecast also were generally accurate. Section 2 describes the project. Section 3 summarizes the project traffic forecasts. Section 4 describes the forecasting method used to develop the traffic forecasts in Section 3. Section 5 enumerates the common and project-specific assumptions. Section 6 describes the data collection plan that will be executed prior to the post-construction forecast analysis. Section 7 provides a list of data sources and references used to develop the forecast. 2. Project Description <Name of the project> is a <type of project [capacity addition, reconstruction, etc.]> located in <city, state>. Traffic forecasts for the project were prepared in YYYY for the YYYY, YYYY and YYYY forecast year(s) for <agency name>. The project is currently planned to open in YYYY. The internal agency tracking number(s) for planning, design and construction phases is NNNNNNNNN. [Include a 1-2 sentence description of the purpose of the project and the need for the traffic forecast] The study area boundaries are <here>, <here>, <here> and <here>. A summary of the project scope goes here. Describe any unique characteristics of the project. Some examples include: first project of its type in the region, first project of its type in decades, and exceptional project length, construction period and/or cost. Describe the travel markets that are expected to comprise the majority of demand on the project. Travel markets are significant quantities of trips that traverse from one geographic area to another. They are typically further characterized by common trip purposes, time periods, line-haul or

Traffic Forecasting Accuracy Assessment Research Guidance Document I-90 circulation/distribution movements, or socio-economic variables. Examples of travel markets include: suburb-to-CBD work trips, external-external trips, gameday traffic, and local shopping trips. Include a map, please. 3. Description of Traffic Forecasts Traffic forecasts were made for NN links in the study area. Unless mentioned otherwise, traffic forecasts discussed here are expressed in Average Annual Daily Traffic (AADT)/AM peak hour/other units. The base year is YYYY. The most recent year traffic counts are available in the project area is YYYY. Forecasts were developed for the YYYY (opening), YYYY (interim), and YYYY (design) years. Describe generally how the traffic forecasts were produced (e.g., model outputs only, post- processed model outputs, traffic counts with growth rate [define growth rate], etc.). If there are existing reports that document the traffic forecasting methodology, please include them in the appendix and reference them in this section. Table 1: Traffic Forecasts Segment # Project Segment and Direction Time of Day Most Recent Traffic Count (Year) Base Year Estimate Opening Year Forecast Interim Year Forecast Design Year Forecast 1 AM Peak Hour C,CCC N,NNN N,NNN N,NNN N,NNN 1 PM Peak Hour C,CCC N,NNN N,NNN N,NNN N,NNN 1 ADT C,CCC N,NNN N,NNN N,NNN N,NNN 2 2 2 Etc. Here is an overall assessment of the forecasts and how they compare to recent or existing traffic counts. Describe the uncertainty windows – the range of forecasts – for the project using the quantile regression models developed by NCHRP 08-110. 4. Forecasting Methodology This section describes the methodology used to develop the traffic forecasts. Common methods include: traffic count trend projections, population growth rate forecasts, and travel models. If standardized methods are used, refer to the type and version and reference already-available documentation. Also enclose this documentation in Section 7.

Traffic Forecasting Accuracy Assessment Research Guidance Document I-91 Describe the method’s understanding of the project. Items include: general understanding of current demand, travel times and the travel markets expected to use the project. Additional items would include how well the model understands the expected changes, including the project itself in terms of overall demand, travel markets, and travel times. Identify important items for which the method does not account. (If much of the information below is already documented in a standardized report format required by the agency, attach that report in the appendix and refer to it in this section as needed.) If forecasts from more than one method were produced for the project, briefly explain why the forecasts in Section 3 were selected over the alternate forecasts. 5. Assumptions This section identifies the exogenous forecasts and project assumptions used in the development of the traffic forecasts. Exogenous forecasts are made outside of the immediate traffic forecasting process. Project assumptions are established during project development and serve as the basis for the traffic forecast. Exogenous forecasts and project assumptions are leading sources of forecast error. An example are population and employment forecasts, which are commonly identified as a major source of traffic forecasting error. These forecasts are usually made by outside planning agencies on a regular basis; that is, they are not prepared for any individual project. During project development, these forecasts are revised to match assumptions documented by the project team. In this example, population and employment forecasts are both an exogenous forecast and a project assumption. Section 5.1 records exogenous forecasts and assumptions common in most traffic forecasts. Section 5.2 records extra-ordinary other project-specific assumptions, including those that are qualitative. Ordinary Assumptions Traffic forecasting accuracy research has identified several exogenous forecasts and project assumptions that commonly are reported as sources of forecast error. This section documents these typical, or ordinary, assumptions. Such assumptions include:  Macro-economic conditions (of the region or study area),  Population and employment forecasts,  Significant changes in land use,  Auto fuel prices,  Toll pricing and sensitivity,  Auto ownership,  Changes in technology,  K-, D- and T-factors,

Traffic Forecasting Accuracy Assessment Research Guidance Document I-92  Travel times within the study area, and  Duration between year forecast produced and opening year. The assumptions can be documented in tabular format. The information should be corridor- specific to the extent possible, with an understanding that the forthcoming post-construction analysis may only be able to gather some information – population, for instance – at aggregate levels. Extra-Ordinary Assumptions This section highlights variables from Section 5.1 that have a significant impact on the traffic forecast values and describes those assumptions in more detail. Additionally, this section can also describe uncommon, or extra-ordinary, assumptions that are specific to the project. This includes elements that may be qualitative, such as how the region may react to newly introduced managed lanes. Other examples may include a particularly large development that is expected to comprise significant demand of the project, impacts from adjacent construction, and specific policies or ordinances (e.g. parking prohibitions). 6. Post-Opening Data Collection This section describes data collection needed to verify the traffic forecast and the key assumptions described in Section 5. This may include collecting traffic counts, documenting key events (e.g., adjacent construction delays), purchasing location-based data, and/or gathering observed speed data. Electronic Appendix: Supporting Files The following materials and methodology files are enclosed electronically: 1. Standardized Traffic Forecasting Reports, if used by the agency for this project 2. All readily-available documents related the forecasting method described in Section 4

I-93 Appendix C: Deep Dive Annotated Outline 1. Introduction <Name of the project> is a <type of project [capacity addition, reconstruction, etc.]> located in <city, state>. This report, written in month-year, assesses the utility, reliability and accuracy of traffic forecasts for the <project name>. Traffic forecasts for the project were prepared in YYYY for the YYYY, YYYY and YYYY forecast year(s). The project opened in YYYY. Traffic counts are available for YYYY-YYYY year(s), all post-opening. Section 2 describes the project. Section 3 compares the predicted and actual traffic volumes for all roadways in the study area where post-opening traffic counts are available. Section 4 enumerates the exogenous forecasts and sources of forecast error for the project. It also includes an assessment of the accuracy of the exogenous forecasts. Section 5 attempts to identify items discussed in Section 4 that are important sources of forecast error and, if so, attempt to quantify how much it would change the forecast if the forecasters had accurate information about the item. Section 6 summarizes the findings from the previous two sections. Section 7 discusses suggested improvements to the forecasting methodology, forecasting practices, and/or validation practices to be used for future projects. Section 8 provides a list of data sources and references used in the development of this report. 2. Project Description The study area boundaries are <here>, <here>, <here> and <here>. A summary of the project scope goes here. Describe any unique characteristics of the project. Some examples include: first project of its type in the region, first project of its type in decades, and exceptional project length, construction period and/or cost. Include a map, please.

Traffic Forecasting Accuracy Assessment Research Guidance Document I-94 3. Predicted-Actual Comparison of Traffic Forecasts There are NN links/roadways in the study area. Traffic forecasts were made for NN links, or PP percent. Describe generally how the traffic forecasts were produced (e.g., model outputs only, post-processed model outputs, traffic counts with growth rate [define growth rate], etc.). There are NN links, or PP percent, with an Annual Average Daily Traffic (AADT) traffic count. The following table lists each of these links with their forecast and observed AADT. Slight differences between forecast year and year traffic forecasts were collected should be noted here.

Table 1: Traffic Volume Accuracy Assessment (columns in yellow require numerical input) Seg# Project Segement and Direction Time of  Day Base Year  Count Base Year  Forecast  (if different) Opening  Year Count Opening Year  Forecast Over- Estimating Rate % Growth  in Count % Growth  in Forecast Sources of data Comments 1 AADT #DIV/0! #DIV/0! #DIV/0! 1 AM Peak #DIV/0! #DIV/0! #DIV/0! 1 PM Peak #DIV/0! #DIV/0! #DIV/0! 2 AADT #DIV/0! #DIV/0! #DIV/0! 2 AM Peak #DIV/0! #DIV/0! #DIV/0! 2 PM Peak #DIV/0! #DIV/0! #DIV/0! 3 AADT #DIV/0! #DIV/0! #DIV/0! 3 AM Peak #DIV/0! #DIV/0! #DIV/0! 3 PM Peak #DIV/0! #DIV/0! #DIV/0! 4 AADT #DIV/0! #DIV/0! #DIV/0! 4 AM Peak #DIV/0! #DIV/0! #DIV/0! 4 PM Peak #DIV/0! #DIV/0! #DIV/0! 5 AADT #DIV/0! #DIV/0! #DIV/0! 5 AM Peak #DIV/0! #DIV/0! #DIV/0! 5 PM Peak #DIV/0! #DIV/0! #DIV/0! 6 AADT #DIV/0! #DIV/0! #DIV/0! 6 AM Peak #DIV/0! #DIV/0! #DIV/0! 6 PM Peak #DIV/0! #DIV/0! #DIV/0! 7 AADT #DIV/0! #DIV/0! #DIV/0! 7 AM Peak #DIV/0! #DIV/0! #DIV/0! 7 PM Peak #DIV/0! #DIV/0! #DIV/0! 8 AADT #DIV/0! #DIV/0! #DIV/0! 8 AM Peak #DIV/0! #DIV/0! #DIV/0! 8 PM Peak #DIV/0! #DIV/0! #DIV/0! Base Year Count: {Report  Name/Website  link/Model Run Details} Base Year Forecast: Opening Year Count: Opening Year Forecast:

Traffic Forecasting Accuracy Assessment Research Guidance Document I-96 Here is an overall assessment of the accuracy of these forecasts. 4. Potential Sources of Forecast Error This section identifies the exogenous forecasts and project assumptions used in the development of the traffic forecasts. Exogenous forecasts are made outside of the immediate traffic forecasting process. Project assumptions are established during project development and serve as the basis for the traffic forecast. Exogenous forecasts and project assumptions are leading sources of forecast error. An example are population and employment forecasts, which are commonly identified as a major source of traffic forecasting error. These forecasts are usually made by outside planning agencies on a regular basis; that is, they are not prepared for any individual project. During project development, these forecasts are revised to match assumptions documented by the project team. In this example, population and employment forecasts are both an exogenous forecast and a project assumption. Past forecasting research has identified several exogenous forecasts and project assumptions as common sources of forecast error, including:  Macro-economic conditions (of the region or study area),  Population and employment forecasts,  Significant changes in land use,  Auto fuel prices,  Tolling pricing, sensitivity and price levels,  Auto ownership,  Changes in technology,  Travel times within the study area, and  Duration between year forecast produced and opening year. The following table lists all exogenous forecasts and project assumptions for which observed data is available. It also includes an assessment of the accuracy of each item. <See Table 2; note where actual data is not available> Assess whether overall accuracy of the sources of forecast error. Identify any model deficiencies/issues, data deficiency/issues and unexpected non- transportation changes that might have contributed to forecast error. If none, state accordingly.

Table 2: Input Accuracy Assessment Table (columns in yellow require input) Items Definition  Quantifiable Important  Factor? Estimated  Opening Year  Values Observed  Opening Year  Values Difference (Est. ‐ Obs.) % Difference Related Comments from the Report Data Sources Employment The actual employment (or GDP) differs from what was projected. Yes Yes/No ‐         #DIV/0! Estimated Value: {Report Name/Website  link/Model Run Details} Observed Value: Population/Household The actual population or households differ from what was projected. Yes ‐         #DIV/0! Estimated Value: Observed Value: Car Ownership Actual car ownership differs from projection. Should note whether car ownership is endogenous or exogenous to the forecast. Yes ‐         #DIV/0! Estimated Value: Observed Value: Fuel Price/Efficiency The average fuel price or fuel efficiency different from expectations. Yes ‐         #DIV/0! Estimated Value: Observed Value: Travel Time/Speed Travel time comparison of the facility itself and alternative routes. Yes ‐         #DIV/0! Estimated Value: Observed Value: Toll Sensitivity/Value of Time The sensitivity to tolls, or the value of the tolls themselves is in error. For example, Anam, S. (2016) study on Coleman Bridge found that the project considered two toll amounts ($1 and $0.75), however by the time of opening/horizon year it got to $0.85 and $2. Yes ‐         #DIV/0! Estimated Value: Observed Value: Macro-economic Conditions Any effect of number of economic recession on forecast accuracy. No ‐         #DIV/0! Estimated Value: Observed Value: Significant Land Use changes Refers to changes in build environment that are not specific to the project (Andersson et al.(2016)). Flyvbjerg et al. (2006) found that 26% of projects experience problems regarding the change in land use. No ‐         #DIV/0! Estimated Value: Observed Value: Changes in Technology Autonomous Vehicles, Automated Tolls No ‐         #DIV/0! Estimated Value: Observed Value: Study-Forecast Duration Number of years between forecast year and base year. According to Anam, S. et al. (2016) as the difference decreases, accuracy increases. Yes ‐         #DIV/0! Estimated Value: Observed Value: Trip Generation/Travel Characteristics The availability of appropriate data and their quality, in particular traffic counts, network characteristics, travel costs etc. No ‐         #DIV/0! Estimated Value: Observed Value: Project Scope The project was built to different specifications than was assumed at the time of the forecast. For example, budget constraints meant that only 4 lanes were built instead of 6. Yes ‐         #DIV/0! Estimated Value: Observed Value: Rest of Network Assumptions There were assumptions about related projects that would be constructed that differed from what was actually built. Yes ‐         #DIV/0! Estimated Value: Observed Value: Model Deficiency/Issues Limitations of the model itself. This could include possible errors, or limitations of the method. For example, the project was built in a tourist area, but the model was not able to account for tourism. No ‐         #DIV/0! Estimated Value: Observed Value: Data Deficiency/Issues Limitations of the data available at the time of the forecast. For example, erroneous or outdated counts were used as the basis for pivoting. No ‐         #DIV/0! Estimated Value: Observed Value: Unexpected Changes In the latter portion of the 20th century, this could include the rise of 2-worker households or other broad social trends. In the 21st century, this could include technology changes, such as self-driving cars. No ‐         #DIV/0! Estimated Value: Observed Value: Other Other issues that are not articulated above. No ‐         #DIV/0! Estimated Value: Observed Value:

Traffic Forecasting Accuracy Assessment Research Guidance Document I-98 5. Contributing Sources to Forecast Error Building upon the items discussed in Section 4.0, this section attempts to identify items that are important sources of forecast error and, if so, attempt to quantify how much it would change the forecast if the forecasters had accurate information about the item. Adjusted forecasts for the critical roadways are computed by applying an elasticity to the relative change between the actual and predicted values for each item in Section 4. Only those items which could be quantified and deemed important for this project were adjusted. The effect on the forecast can be quantified in this way. First, the change in forecast value, a delta between the opening year forecast and the actual observed traffic count in the opening year is calculated. Change in Forecast Value 𝐴𝑐𝑡𝑢𝑎𝑙 𝑉𝑎𝑙𝑢𝑒 𝐹𝑜𝑟𝑒𝑐𝑎𝑠𝑡 𝑉𝑎𝑙𝑢𝑒 𝐹𝑜𝑟𝑒𝑐𝑎𝑠𝑡 𝑉𝑎𝑙𝑢𝑒 Second, a factor of the effect on forecast by exponentiating an elasticity of the common source errors and natural-log of the change rate in forecast value is calculated. This factor is applied to the actual forecast volume to generate an adjusted forecast. Effect on Forecast 𝑒𝑥𝑝 ∗ 1 Adjusted Forecast 1 𝐸𝑓𝑓𝑒𝑐𝑡 𝑜𝑛 𝐹𝑜𝑟𝑒𝑐𝑎𝑠𝑡 ∗ 𝐴𝑐𝑡𝑢𝑎𝑙 𝐹𝑜𝑟𝑒𝑐𝑎𝑠𝑡 𝑉𝑜𝑙𝑢𝑚𝑒 The results of this process are shown in the following table. Discuss insights and findings. <See Table 3: Forecast Adjustment Table based on Elasticities, sorted largest-to-smallest by the “remaining percent difference from forecast” column >

Table 3: Forecast Adjustment Table (Elasticity Adjustments) (repeat for all segments, provide summary adjustments at the final rows of the table) Seg# Items Definition  Is Error Forecast  Value Actual  Value Change in  Forecast Value Elasticity Effect on  Forecast Actual  Forecast  Volume Adj Forecast  Volume Remaining % Error  for Adj Forecast Data  Sources for  Elasticity Comments 1 Employment The actual employment (or GDP) differs from what was projected. Yes/No   ‐    ‐     0% 0.30     0% ‐     ‐    #DIV/0! 1 Population/Household The actual population or households differ from what was projected.   ‐    ‐     0% 0.75     0% ‐     ‐    #DIV/0! 1 Car Ownership Actual car ownership differs from projection. Should note whether car ownership is endogenous or exogenous to the forecast.   ‐    ‐     0% 0% ‐     ‐    #DIV/0! 1 Fuel Price/Efficiency The average fuel price or fuel efficiency different from expectations.   ‐    ‐     0% (0.20)     0% ‐     ‐    #DIV/0! 1 Travel Time/Speed Travel time comparison of the facility itself and alternative routes.   ‐    ‐     0% 0% ‐     ‐    #DIV/0! 1 Toll Sensitivity/Value of Time The sensitivity to tolls, or the value of the tolls themselves is in error. For example, Anam, S. (2016) study on Coleman Bridge found that the project considered two toll amounts ($1 and $0.75), however by the time of opening/horizon year it got to $0.85 and $2.   ‐    ‐     0% 0% ‐     ‐    #DIV/0! 1 Study-Forecast Duration Number of years between forecast year and base year. According to Anam, S. et al. (2016) as the difference decreases, accuracy increases.   ‐    ‐     0% 0% ‐     ‐    #DIV/0! 1 Project Scope The project was built to different specifications than was assumed at the time of the forecast. For example, budget constraints meant that only 4 lanes were built instead of 6.   ‐    ‐     0% 0% ‐     ‐    #DIV/0! 1 Rest of Network Assumptions There were assumptions about related projects that would be constructed that differed from what was actually built.   ‐    ‐     0% 0% ‐     ‐    #DIV/0! 1 Original Traffic Forecast Original Forecasted Volume for Segment 1 0 0 0% N/A N/A 1 Adjusted Traffic Forecast Adjusted Volume for Segment 1 N/A N/A N/A - - #DIV/0! 2 Employment The actual employment (or GDP) differs from what was projected. Yes/No   ‐    ‐     0% 0.30     0% ‐     ‐    #DIV/0! 2 Population/Household The actual population or households differ from what was projected.   ‐    ‐     0% 0.75     0% ‐     ‐    #DIV/0! 2 Car Ownership Actual car ownership differs from projection. Should note whether car ownership is endogenous or exogenous to the forecast.   ‐    ‐     0% 0% ‐     ‐    #DIV/0! 2 Fuel Price/Efficiency The average fuel price or fuel efficiency different from expectations.   ‐    ‐     0% (0.20)     0% ‐     ‐    #DIV/0! 2 Travel Time/Speed Travel time comparison of the facility itself and alternative routes.   ‐    ‐     0% 0% ‐     ‐    #DIV/0! 2 Toll Sensitivity/Value of Time The sensitivity to tolls, or the value of the tolls themselves is in error. For example, Anam, S. (2016) study on Coleman Bridge found that the project considered two toll amounts ($1 and $0.75), however by the time of opening/horizon year it got to $0.85 and $2.   ‐    ‐     0% 0% ‐     ‐    #DIV/0! 2 Study-Forecast Duration Number of years between forecast year and base year. According to Anam, S. et al. (2016) as the difference decreases, accuracy increases.   ‐    ‐     0% 0% ‐     ‐    #DIV/0! 2 Project Scope The project was built to different specifications than was assumed at the time of the forecast. For example, budget constraints meant that only 4 lanes were built instead of 6.   ‐    ‐     0% 0% ‐     ‐    #DIV/0! 2 Rest of Network Assumptions There were assumptions about related projects that would be constructed that differed from what was actually built.   ‐    ‐     0% 0% ‐     ‐    #DIV/0! 2 Original Traffic Forecast Original Forecasted Volume for Segment 2 0 0 0% N/A N/A 2 Adjusted Traffic Forecast Adjusted Volume for Segment 2 N/A N/A N/A - - #DIV/0!

Traffic Forecasting Accuracy Assessment Research Guidance Document I-100 If the travel model or other methodology used to produce the traffic forecasts is available, then re-run the model or methodology, to the extent possible, using corrected exogenous forecasts and project assumptions. Report the results here. If the results are dramatically different from the elasticity- based approach, note this and re-run the model or methodology altering the biggest contributors of forecast error individually. Note the “elasticities” and “cross-elasticities” from this process. <See Table 4: Adjust Forecast Table using the Model>

Table 4: Forecast Adjustment Table (Travel Model Adjustments) (repeat for all segments, provide summary adjustments at the final rows of the table) Seg# Items Changes made in the model Is Error Old Model  Value New Model  Value Old Model  Volume New Model  Volume Observed  Volume Difference (New ‐ Obs.) % Difference from  Observed Volume Comments/  Conclusions 1 Employment Yes/No ‐    ‐   ‐     ‐     ‐    #DIV/0! 1 Population/Household ‐    ‐   ‐     ‐     ‐    #DIV/0! 1 Car Ownership ‐    ‐   ‐     ‐     ‐    #DIV/0! 1 Fuel Price/Efficiency ‐    ‐   ‐     ‐     ‐    #DIV/0! 1 Travel Time/Speed ‐    ‐   ‐     ‐     ‐    #DIV/0! 1 Toll Sensitivity/Value of Time ‐    ‐   ‐     ‐     ‐    #DIV/0! 1 Study-Forecast Duration ‐    ‐   ‐     ‐     ‐    #DIV/0! 1 Project Scope ‐    ‐   ‐     ‐     ‐    #DIV/0! 1 Rest of Network Assumptions ‐    ‐   ‐     ‐     ‐    #DIV/0! 2 Employment Yes/No ‐    ‐   ‐     ‐     ‐    #DIV/0! 2 Population/Household ‐    ‐   ‐     ‐     ‐    #DIV/0! 2 Car Ownership ‐    ‐   ‐     ‐     ‐    #DIV/0! 2 Fuel Price/Efficiency ‐    ‐   ‐     ‐     ‐    #DIV/0! 2 Travel Time/Speed ‐    ‐   ‐     ‐     ‐    #DIV/0! 2 Toll Sensitivity/Value of Time ‐    ‐   ‐     ‐     ‐    #DIV/0! 2 Study-Forecast Duration ‐    ‐   ‐     ‐     ‐    #DIV/0! 2 Project Scope ‐    ‐   ‐     ‐     ‐    #DIV/0! 2 Rest of Network Assumptions ‐    ‐   ‐     ‐     ‐    #DIV/0! 1 ‐     ‐     ‐    #DIV/0! 2 ‐     ‐     ‐    #DIV/0! 3 ‐     ‐     ‐    #DIV/0! 4 ‐     ‐     ‐    #DIV/0! 5 ‐     ‐     ‐    #DIV/0! 6 ‐     ‐     ‐    #DIV/0! 7 ‐     ‐     ‐    #DIV/0! 8 ‐     ‐     ‐    #DIV/0! All Adjustments All the above items (whereever possible) are adjusted to match the actual values

Traffic Forecasting Accuracy Assessment Research Guidance Document I-102 6. Discussion This section discusses how the findings in Section 5 relate to Section 3 and Section 4. This section should then address the following questions:  Would the project decision have changed if the forecast accuracy or reliability were improved?  How useful (what was their utility) were the forecasts in terms of providing the necessary information to the planning process  Was risk and uncertainty considered in the forecast? How was it be considered? How was it communicated? 7. Suggested Changes This section suggests improvements to the:  Forecasting methodology,  Forecasting practices, and/or  Validation practices to be used for future projects. Supporting evidence from Sections 3-6 should be explicitly referenced. 8. Data Sources and References List and number in alphabetical order the data sources and references used to develop this report.

Traffic Forecasting Accuracy Assessment Research Guidance Document I-103 Appendix D: Forecast Card Data Assumptions Due to unavailability of clear information, in most cases, some assumptions have been made. The assumptions are specific to State and are explained in detail in below sub sections. 1. Florida Florida DOT District 4 (D4) and District 5 (D5) data were provided in different formats. D4 information was in excel format while D5 was extracted from scanned pdf reports. There are in total 143 valid records in D4 dataset. The table below lists the important fields in the D4 dataset. Description of the fields is assumed based on its name. CATEGORY FEILD DESCRIPTION GENERAL ROADWAY INFORMATION Segment/ Intersection Description of the segment or intersection where the project is carried out Roadway ID ID assigned to the roadway on which project is located County County in which project was located Roadway Segment Description of the roadway corresponding to Roadway ID DATE Date of Report Date when the report was completed Year of WP +10 (Interim) Interim/Mid-design year Year of WP +10 AADT Forecasted AADT in Interim/Mid-design year Design Year Design year Design Year AADT Forecasted AADT in design year FORECASTED DATA Future Forecasted Year (Opening Year) Opening year Future Forecasted AADT (Opening Year) Forecasted AADT in opening year COUNT STATION FDOT/County Count Station ID# ACTUAL DATA COUNTS Actual AADT Actual AADT in opening year Florida D4 Field Description

Traffic Forecasting Accuracy Assessment Research Guidance Document I-104 Some of the assumptions made for D4 dataset are as follows: 1. Date of report is assumed to be same as the year when forecast was completed. 2. Report finding column is used to get the details on Improvement Type. This information is not always available in Report Finding column. 3. Forecasting Agency for all the projects is assumed to be State DOT. 4. NCHRP Methodology is derived from Method column in the Florida D4 dataset. Method categories are re-assigned into the NCHRP Methodology categories as follows: Method Assumptions NCHRP Methodology Palm Beach Social Economic Data. Population growth rates CGR Compound Growth Rate Traffic Count Trend LGR Linear Growth Rate Traffic Count Trend Cost Feasible model Project-specific travel model GR Growth Rate Traffic Count Trend EGR Traffic Count Trend Historical AADT Traffic Count Trend SERPM Regional Travel Model Regional Travel Model CGR (Average) Historical AADT Historical AADT+SERPM Traffic Count Trend + Regional Travel Model Model SERPM Regional Travel Model Linear Interpolation Traffic Count Trend CGR Historical Data (2005-2010) *Declining Trend Analysis BEBER Population Forecast (0.5%) Traffic Count Trend + Population growth rates CGR/4 Methods Traffic Count Trend CGR Master Plan Traffic Count Trend Interpolation Assumed to be Linear Interpolation Traffic Count Trend Liner GR Assumed to be Linear GR (typo error) Traffic Count Trend No report, no CGR, no K or D factors and no TMS. Just 18 kip. Professional judgement GR and TAZ Mostly TAZ refers to Socio Economic Growth Traffic Count Trend + Population growth rates Florida D4 Methodology Assumptions

Traffic Forecasting Accuracy Assessment Research Guidance Document I-105 2. Michigan The Michigan dataset was provided by Michigan DOT in the form of both pdf reports and excel table. Whenever there was any mismatch with the information in reports and excel table, the information in the reports were considered legit. There are in total 10 records in this dataset. List of important variables in the dataset are as follows: FIELD DESCRIPTION TAR Number Report number – specific to a project location Details on where the project was located Urban/Rural Area type NFC Length Length of the segment Facility Name Name of the facility for which forecasts were developed Facility type Type of the facility for which forecasts were developed (based on the categories in NCHRP Database) Improvement type Type of the improvement proposed in the project. (if numeric then it is based on the categories in NCHRP Database) Forecaster Agency who was responsible for forecasting (based on the categories in NCHRP Database) Forecaster Description Name of the person who forecasted Methodology Method used for forecasting the AADT Post-Processing or alternative Methodology Post-Processing or alternative Methodology used for the forecast (based on the categories in NCHRP Database) Year Forecast Produced Year when the forecast was completed Base Year Base year for the forecast Forecast year Forecast year in comma format Forecast Year Type Forecast year type respective to the forecast year column in comma format Forecast Units Units of final forecasts (based on the categories in NCHRP Database) Year of observation Year for which the actual AADT was reported Traffic Count Reported actual AADT Count StationiD Count Station ID for which the actual AADT is reported Count Units Units of actual traffic counts Source More information on the source of the actual count Michigan Field Description Some of the assumptions made in this dataset are as follows: 1. Project years for TAR# 2293 and 2573 in the excel were not matching with the reports. The project years in the reports were given preference. 2. Improvement types: Bridge replacement, Bridge repair, Pavement design, Crack treatment and Guard rail replacement is considered in the Resurfacing and minor improvement category. 3. For TAR# 2293, there are multiple forecasts for same segment. The results from the latest forecast for the report year of 2013 was uploaded in the NCHRP database.

Traffic Forecasting Accuracy Assessment Research Guidance Document I-106 4. Horizon year is assumed to be referring to Forecast year for 10+ year. 5. If the forecast year type has only one value, then the second year is assigned to subsequent forecast year type. For example, if the Forecast year are “2013, 2018” and Forecast year type is mentioned as “1”, then 2013 is assumed to be Opening year (Type code 1) and 2018 is assumed to be the Interim year (Type code 2). 3. Minnesota The Minnesota dataset was gathered from previous studies (Parthasarathi and Levinson 2010) in the form of excel table. The raw data is available at Minnesota Historical Society Archives and is in the form of scanned pdf reports produced by Minnesota DOT. Count maps used to get the actual AADT information is also provided in scanned pdf format. Date of collection of data is between 2007-2009. Forecast beyond this timeframe are not included in the dataset. There are in total 1,583 valid records. List of important variables in the dataset are as follows: FIELD DESCRIPTION id: Project identification number, generated for analysis purposes reportno: Report number provided by the Minnesota DOT reportdesc: Project description, typically obtained from the Minnesota DOT project report forecastdate: Date of the project report reportyear: Year in which project report was completed forecastyear: Year for which forecasts were created noofyrs: Number of years between the year in which the report was prepared and the forecast year projectstatus Status indicating if the forecasts were for an existing roadway or new roadway highway: Highway for which forecasts were developed from: Segment start location to: Segment end location direction: Direction of the forecasts (Categories: EB - Eastbound, WB - Westbound, NB -Northbound, SB - Southbound) funclass: Roadways classified using the categorization in the Year 2000 Twin Cities Regional Travel Demand Model (Categories Freeway, Expressway, Divided Arterial, Undivided Arterial, Collector) highwaytype: Roadway type classified based on the type of access provided to downtowns of Minneapolis and St. Paul (Categories: Radial, Lateral) seglengthmi: Length of the segment in miles; the start and end location define Segment segmentcity: City where the roadway segment is located; the start and end location define Segment segmentcounty: County where the roadway segment is located; the start and end location define Segment segmentdirection: Roadway direction with respect to the central cities of Minneapolis and St. Paul (Categories: East, West, North, South, Northeast, Northwest, Southeast, Southwest, Middle, Middle North, Middle South) forecastadt: Forecasted Average Daily traffic (ADT), provided as part of the project actualadt: Actual ADT obtained for the segment, obtained from Minnesota DOT projstat: Project status at the time of report preparation (Categories: Existing facility, New facility) Minnesota Field Description

Traffic Forecasting Accuracy Assessment Research Guidance Document I-107 Some of the assumptions made in this dataset are as follows: 1. Records with no or zero value for the Forecasted AADT were removed from the dataset. 2. 99% of the forecasts are produced by State DOT and 1% of the forecast are produced by consultant under the contract with State DOT. Therefore, it is safely assumed that the forecasting agency for all records is State DOT. 3. The blank cell for variables Highway, From, To, Seglengthmi does not necessarily mean a missing value. In most of the cases, the information is only entered in one direction. In such cases, the information in the prior record should be assumed. For example, in figure x.x, the second row is referring to same segment i.e. Proposed road from CR 30 to Scandia Rd. But, it is in reverse direction (west). Example - Missing Segment Values 4. Not much information is available in the reports for Forecast Methodology. Since these are old forecasts, it is assumed that the forecast was made using Traffic Count Trend. 5. Records with no actualadt value means there were no available AADT counts for that year on that segment. Missing traffic count information wherever possible was added using the Count maps. 6. It is also unclear which forecast year (Opening/Mid-Design/Design year) does the forecast belong to. 7. If no county information is available, then the county information was found from google maps using the segment description. 8. Some of the records have Blaine as a county. But, it is not a county and was reassigned to Anoka/Ramsey based on the location of the project. 9. When two counties are mentioned in the segmentcounty column, then only the first county is entered in NCHRP database.

Traffic Forecasting Accuracy Assessment Research Guidance Document I-108 4. Wisconsin Wisconsin DOT provided the project information in a table format in an Excel file. The worksheet “Data” in the Accuracy_Sites_For_Submittal_FINAL.xls is used for the upload process. There are in total 457 valid records. The field description for this table is described below: FIELD DESCRIPTION Control Number Internal ID number for Traffic Forecasting Section use Match Year The first interim forecast year that matches with year (post-project) of traffic count Last Count Year Year of count on which forecast was projected from Time Period (Yrs) Number of Years between Match Year and Last Count Year Report Completed Year of forecast completion County County in which project was located Forecaster Staff member who developed traffic forecast Functional Class Legacy FHWA functional classification of subject project/forecast roadway 1=Rural Interstate 2=Rural Principal Arterial 6=Rural Minor Arterial 7=Rural Major Collector 8=Rural Minor Collector 9=Rural Local 11=Urban Interstate 12=Urban Freeway/Expressway 14=Urban Principal Arterial 16=Urban Minor Arterial 17=Urban Collector 19=Urban Local Actual Volume Observed AADT in match year Forecast Volume Forecasted AADT in match year. % Difference Actual/forecast difference ratio GEH Geoffrey E. Havers statistic (utilizing forecasted volume and actual volume) Model Area? Yes/no flag indicating availability of a Travel Demand Model in subject project/forecast area Difference Percentage difference between actual and forecast volume Abs Difference Absolute value of Difference Last Count Observed AADT in Last Count Year AGR Forecast Annual Growth Rate Site ID WisDOT master traffic count database ID number of the subject traffic count / forecast site Actual AGR Actual Annual Growth Rate Diff AGR Actual AGR - Forecast AGR Pos. Growth Yes/no flag indicating positive growth of Actual AGR Wisconsin Field Description

Traffic Forecasting Accuracy Assessment Research Guidance Document I-109 Before the data was uploaded in the final database some changes were made to the data. 1. Wisconsin Traffic studies distinguishes forecast years into: 1st Interim Year, 2nd Interim year and Design year. Usually, the 1st Interim year is the year open to traffic or construction year. For this database, the first Interim year which is represented by the Match year is assumed to be the Opening year. 2. “Report completed” field is assumed to be the year when forecast was completed and not when report was completed/submitted. 3. It is assumed that all the forecaster were State DOT members/employees. 4. For Built projects, the assumed method for forecasting is using Regional Travel Demand Model only. In other cases, it is combination of Regional Travel Model and Regression Model which is considered in the Traffic count trend methodology. 5. Functional Class in the Wisconsin dataset are reclassified into the NCHRP Functional Categories as shown in the table below. Records with no Functional class code are kept blank in the NCHRP database. Urban collectors are categorized as major collectors in the main database. Wisconsin Functional Class NCHRP Code NCHRP Description Rural Interstate System 1 Interstate or Limited-access facility Rural Principal Arterial 3 Principal Arterial Rural Minor Arterial 4 Minor Arterial Rural Major Collector 5 Major Collector Rural Minor Collector 6 Minor Collector Urban Interstate 1 Interstate or Limited-access facility Urban Freeway/Expressway 1 Interstate or Limited-access facility Urban Principal Arterial 3 Principal Arterial Urban Minor Arterial 4 Minor Arterial Urban Collector 5 Major Collector Urban Local 7 Local Functional Class Classification Assumption 6. Area Type is decided based on the functional classification information. 1 to 9 Functional Class are Rural and rest are assigned as Urban. 7. There was typo error in County name for “Forest” county which was fixed. 8. Toll type is decided based on the facility type. ”Unknown” category is assigned for freeways/expressways functional class.

Traffic Forecasting Accuracy Assessment Research Guidance Document I-110 Appendix E: Implementation Plan The guidance in this document for the continued assessment and improvement of traffic forecasts will only be helpful if it is implemented. The following pages describe potential barriers to implementation, ways past those barriers, and proposes a plan to promote the implementation of this research. 1. Lessons from Data Sharing Research The key part of the successful implementation of this research is the willingness of transportation agencies to share data about their forecasts, as well as the actual outcomes of their projects. In understanding this challenge, we can draw from the lessons of other fields to understand the factors that affect the decision of whether or not to share data, and adapt those lessons to promote data sharing in traffic forecasting. Consider the example of scientists’ willingness to share their research data. These are data sets that would be archived electronically, in addition to published papers, that would allow other scientists to more easily reproduce, review and build upon their work. From a community perspective, data sharing is a clear winner, but some individual scientists may be reluctant to share either because of the additional effort involved, because it opens up the possibility of someone else finding an error they made, or because they may perceive exclusive access to certain data as a competitive advantage. There are similarities to our problem where there are clear community-level benefits to data sharing, but individual agencies may be reluctant either due to the effort or because they may not want to be criticized using their own data. Figure E-1 shows the results of recent research quantifying the factors contributing to scientists’ data sharing behavior (Kim and Stanton 2016). The boxes indicating the factors identified, and the numbers indicate the relative importance of each factor, with larger numbers being more important. Several observations can be made from these results. The two largest factors are scholarly altruism and normative pressure (the norms of whether others in their discipline share data). Both show up as having a larger influence than regulative pressures. In addition, the perceive effort associated with data sharing is an important impediment, while the availability of a data repository has a positive (but insignificant) influence.

Traffic Forecasting Accuracy Assessment Research Guidance Document I-111 Figure E-1 Factors Affecting Data Sharing Behavior (Kim and Stanton 2016) We recognize that there are important differences between the data sharing behavior of individual scientists versus that of transportation agencies—particularly with respect to the perceived risk of criticism. In part, this is because science has a long-standing culture of critical debate—by being transparent and open to argument, it builds the credibility of the science itself. Therefore, part of the solution must be to promote a culture of constructive engagement where we aim to improve and inform rather than to blame. In addition, research into the factors influencing the adoption of new travel forecasting methods (Donnelly et al. 2010) offers similar lessons. In particular, one of the biggest factors influencing agencies decisions to adopt new travel forecasting techniques was the presence of an individual who would serve as a champion. The lesson here is to recruit individuals to champion the implementation of this research. If we take these lessons together, we suggest that the key elements necessary to promote the implementation of this research are:  Develop a working data repository, as described in Chapter 3. Make it widely available and make it easy to use. This serves the dual goals of having a data repository and of reducing the perceived effort associated with data sharing.  Identify and recruit individuals to serve as potential champions for the implementation of this research at their own agencies. Likely candidates include members of the NCHRP oversight panel and participants in the workshop. Make an appeal to these individuals focused on the greater good of the field.

Traffic Forecasting Accuracy Assessment Research Guidance Document I-112  Minimize the perceived risk by promoting a culture of constructive improvement. This can be achieved, in part, by having the early adopters talk publicly about the benefits they see from participating.  Once the early adopters are participating, take advantage of this participation as an emerging norm in the field that will further incentivize others to participate. With a strong focus on this approach, regulative pressure (as shown in Figure E-1) may not be necessary. 2. Impediments to Successful Implementation There are several specific barriers that may impede the successful implementation of this research, but also paths beyond those barriers. Both are discussed below. Barrier 1: Lack of Awareness Potential users will not implement this research if they are not aware of it. Therefore, a strong dissemination push is warranted. This may involve presenting the findings at national and international conferences, local model users’ group meetings, and publishing academic papers derived from this research. Barrier 2: Lack of Resources We must be honest in acknowledging that there is effort involved in implementing the recommendations in this guidance document. While there are also benefits and the potential for a more efficient allocation of resources towards more effective modeling techniques, those agencies responsible for forecasts must still invest the resources to archive and analyze their accuracy. Therefore, the less cumbersome it is to implement the recommendations, the more likely agencies are to do so. During the course of this research, several efforts have been made to make implementation as easy as possible—including developing the archive and information system, and providing annotated outlines for the Silver standard and for conducting deep dives. These tools were tested during the workshop, but would benefit from more extensive beta testing and associate refinement. Barrier 3: Fear of Transparency One potential barrier to implementation is that agencies responsible for traffic forecasting may fear that by examining forecast accuracy, they are opening themselves up to criticism if those forecasts are shown to be inaccurate. Section 1.3.3: Reasons to Implement These Recommendations discusses this issue, as well as the potential for a transparent examination of forecast accuracy to be used as a tool for building credibility. There is an aspect to this issue that relates to culture and norms. As noted in Figure E-1, normative pressure is the biggest influence on scientists willingness to share research data. Normative pressure means that people are more willing to take an action if they see their peers doing so and perceive it to be the norm. It is reasonable to expect similar dynamics at play in traffic forecasting, with a similar effect observed when examining the emergence of activity-based travel demand models (a type of traffic forecasting model that is more methodologically sophisticated than traditional trip-

Traffic Forecasting Accuracy Assessment Research Guidance Document I-113 based travel demand models). The first few activity-based models came slowly, with many potential adopters not wanting to take a risk on a new method. Once a handful of agencies had adopted such models, they talked publicly about their experience and the benefits they saw in doing so. Other agencies felt safe to join and could build upon the development efforts of the early adopters. This led to the implementation of such models accelerating, particularly among larger Metropolitan Planning Organizations. Considering this history, one of the best things that can be done to promote the implementation of this research more broadly is to do everything possible to get a handful of early adopters to implement the recommendations and see success in doing so. Those early adopters will then serve as examples for others, both in terms of further testing and refining the process and in terms of providing safety in numbers. 3. Proposed Activities The following activities are proposed to promote the implementation of this research. Task 1: Dissemination, has already begun during the closing months of this research project. The other tasks would require additional resources. Task 1: Dissemination The dissemination activities aim to increase awareness of this research. They focus on traditional scholarly dissemination strategies, such as conference participation and dissemination through journal articles. Specific activities include:  The research was the focus of a Sunday workshop on “Progress in Improving Travel Forecasting Accuracy” at the 98th Transportation Research Board Annual Meeting in Washington, D.C., January 2019.  This research will be the focus of the closing plenary session at the Transportation Research Board Planning Applications Conference in Portland, Oregon, June 2019.  This research will be presented at the Florida Model Task Force Meeting in July 2019.  The research team has drafted a journal article referencing the NCHRP report and describing the quantile regression analysis. This would be submitted after approval of the final report and in coordination with the program manager. The research team plans to publish additional journal articles referencing the NCHRP report, and identify additional opportunities to present the work. Task 2: Implementation Site Visits As noted above, achieving the first few successful implementations of this research at transportation agencies is critical. One way to achieve these early successes is to identify potential early adopters and provide support to establish the process at those agencies. As currently envisioned, implementation support would involve a 2-day site visits by 1-2 members of the implementation team. During this site visit, the implementation team would work directly with agency staff to teach them how to calculate the uncertainty ranges around forecasts, install the forecast card repository so they

Traffic Forecasting Accuracy Assessment Research Guidance Document I-114 can track their own projects, enter the first few projects into the repository according to the Bronze standard, work through the documentation needs of the Silver standard, and work through an example of analyzing the data using the data from this report. The visit would also include a briefing for management-level staff to discuss the goals and expectations for the effort. Following the site visit, the implementation team would be available for phone/web-conference support in the event that additional issues arise. Task 3: Beta Testing Refinements As with any project that involves software and new processes, early users may identify refinements that would make the process more usable for their own needs. This is a normal and important part of the development process. For this reason, the recommendations were written with the expectation that agencies may adapt them to their specific needs, and the software and data specification are open-source and written to allow for extensions. The early adopters will also serve as beta testers. It would be smart to allow for additional software refinements to the archive and information system, and/or usability improvements built on top of the main software. For example, an agency may find that a graphical user interface would suit their needs better than editing CSV files in the forecast cards system, and it would be a logical extension to write such an interface that reads and writes forecast cards. Alternatively, it may be possible to streamline certain aspects of the data specification, or add to it, depending on the needs identified during the site visits. Task 4: Support and Community We expect implementation to be more successful with a plan for ongoing support, and propose to achieve this primarily through peer-to-peer support. Undoubtedly, users will encounter both issues and insights as they start to track and report forecast accuracy, and there is value in learning from each other as they do so. A natural first step in establishing such a community is through the early adopters, as discussed in Task 2. The site visits set up the initial process, but there must be some reason to continue the process after the site visit is done. Therefore, we propose that those early adopters be asked to participate in a peer exchange. This peer exchange would take place about a year after the site visits are complete, and provide the participants some opportunity to use the process and produce their first forecast accuracy summary report (Recommendation 3). The peer exchange would be modeled after the successful statewide modeling peer exchanges that have happened in recent years. Participants would present their initial results, and it would serve as a forum to share advice and work through any issues. There is also a need to ensure that this community support is sustainable, with part of that sustainability related to establishing an institutional home. One such home may be a TRB subcommittee on forecast accuracy, which could be a joint subcommittee of ADB40: Transportation Demand Forecasting and ADB50: Transportation Planning Applications. There is also an opportunity for the use of pooled funds to ease the burden on participating agencies. While participating agencies must individually implement recommendations 1 and 2, it may be more efficient for a single researcher to produce an annual or biannual forecast accuracy update report. This would use data pooled from across participating agencies, and those agencies might “buy-in” to fund the joint analysis. This may be similar to the summary reports produced by the Highway England’s Post-Opening Project

Traffic Forecasting Accuracy Assessment Research Guidance Document I-115 Evaluations (POPE). Such an effort would benefit from institutional support to avoid navigating the contracting requirements of each of these individually. One possibility for that is the Zephyr Foundation, which was set up to facilitate shared resources and shared software within the travel forecasting community. We propose that the implementation team take the lead in identifying and arranging potential support, and the peer-exchange participants be offered the opportunity to make a decision about which path works best for them.

Next: Part II: Technical Report »
Traffic Forecasting Accuracy Assessment Research Get This Book
×
MyNAP members save 10% online.
Login or Register to save!
Download Free PDF

Accurate traffic forecasts for highway planning and design help ensure that public dollars are spent wisely. Forecasts inform discussions about whether, when, how, and where to invest public resources to manage traffic flow, widen and remodel existing facilities, and where to locate, align, and how to size new ones.

The TRB National Cooperative Highway Research Program's NCHRP Report 934: Traffic Forecasting Accuracy Assessment Research seeks to develop a process and methods by which to analyze and improve the accuracy, reliability, and utility of project-level traffic forecasts.

The report also includes tools for engineers and planners who are involved in generating traffic forecasts: Quantile Regression Models and a Traffic Forecast Accuracy Assessment.

  1. ×

    Welcome to OpenBook!

    You're looking at OpenBook, NAP.edu's online reading room since 1999. Based on feedback from you, our users, we've made some improvements that make it easier than ever to read thousands of publications on our website.

    Do you want to take a quick tour of the OpenBook's features?

    No Thanks Take a Tour »
  2. ×

    Show this book's table of contents, where you can jump to any chapter by name.

    « Back Next »
  3. ×

    ...or use these buttons to go back to the previous chapter or skip to the next one.

    « Back Next »
  4. ×

    Jump up to the previous page or down to the next one. Also, you can type in a page number and press Enter to go directly to that page in the book.

    « Back Next »
  5. ×

    To search the entire text of this book, type in your search term here and press Enter.

    « Back Next »
  6. ×

    Share a link to this book page on your preferred social network or via email.

    « Back Next »
  7. ×

    View our suggested citation for this chapter.

    « Back Next »
  8. ×

    Ready to take your reading offline? Click here to buy this book in print or download it as a free PDF, if available.

    « Back Next »
Stay Connected!