National Academies Press: OpenBook
Page i
Suggested Citation:"Front Matter." National Academies of Sciences, Engineering, and Medicine. 2014. Regional Operations Forums for Advancing Systems Operations, Management, and Reliability. Washington, DC: The National Academies Press. doi: 10.17226/22256.
×
Page R1
Page ii
Suggested Citation:"Front Matter." National Academies of Sciences, Engineering, and Medicine. 2014. Regional Operations Forums for Advancing Systems Operations, Management, and Reliability. Washington, DC: The National Academies Press. doi: 10.17226/22256.
×
Page R2
Page iii
Suggested Citation:"Front Matter." National Academies of Sciences, Engineering, and Medicine. 2014. Regional Operations Forums for Advancing Systems Operations, Management, and Reliability. Washington, DC: The National Academies Press. doi: 10.17226/22256.
×
Page R3
Page iv
Suggested Citation:"Front Matter." National Academies of Sciences, Engineering, and Medicine. 2014. Regional Operations Forums for Advancing Systems Operations, Management, and Reliability. Washington, DC: The National Academies Press. doi: 10.17226/22256.
×
Page R4
Page v
Suggested Citation:"Front Matter." National Academies of Sciences, Engineering, and Medicine. 2014. Regional Operations Forums for Advancing Systems Operations, Management, and Reliability. Washington, DC: The National Academies Press. doi: 10.17226/22256.
×
Page R5
Page vi
Suggested Citation:"Front Matter." National Academies of Sciences, Engineering, and Medicine. 2014. Regional Operations Forums for Advancing Systems Operations, Management, and Reliability. Washington, DC: The National Academies Press. doi: 10.17226/22256.
×
Page R6
Page vii
Suggested Citation:"Front Matter." National Academies of Sciences, Engineering, and Medicine. 2014. Regional Operations Forums for Advancing Systems Operations, Management, and Reliability. Washington, DC: The National Academies Press. doi: 10.17226/22256.
×
Page R7
Page viii
Suggested Citation:"Front Matter." National Academies of Sciences, Engineering, and Medicine. 2014. Regional Operations Forums for Advancing Systems Operations, Management, and Reliability. Washington, DC: The National Academies Press. doi: 10.17226/22256.
×
Page R8
Page ix
Suggested Citation:"Front Matter." National Academies of Sciences, Engineering, and Medicine. 2014. Regional Operations Forums for Advancing Systems Operations, Management, and Reliability. Washington, DC: The National Academies Press. doi: 10.17226/22256.
×
Page R9
Page x
Suggested Citation:"Front Matter." National Academies of Sciences, Engineering, and Medicine. 2014. Regional Operations Forums for Advancing Systems Operations, Management, and Reliability. Washington, DC: The National Academies Press. doi: 10.17226/22256.
×
Page R10
Page xi
Suggested Citation:"Front Matter." National Academies of Sciences, Engineering, and Medicine. 2014. Regional Operations Forums for Advancing Systems Operations, Management, and Reliability. Washington, DC: The National Academies Press. doi: 10.17226/22256.
×
Page R11
Page xii
Suggested Citation:"Front Matter." National Academies of Sciences, Engineering, and Medicine. 2014. Regional Operations Forums for Advancing Systems Operations, Management, and Reliability. Washington, DC: The National Academies Press. doi: 10.17226/22256.
×
Page R12
Page xiii
Suggested Citation:"Front Matter." National Academies of Sciences, Engineering, and Medicine. 2014. Regional Operations Forums for Advancing Systems Operations, Management, and Reliability. Washington, DC: The National Academies Press. doi: 10.17226/22256.
×
Page R13
Page xiv
Suggested Citation:"Front Matter." National Academies of Sciences, Engineering, and Medicine. 2014. Regional Operations Forums for Advancing Systems Operations, Management, and Reliability. Washington, DC: The National Academies Press. doi: 10.17226/22256.
×
Page R14
Page xv
Suggested Citation:"Front Matter." National Academies of Sciences, Engineering, and Medicine. 2014. Regional Operations Forums for Advancing Systems Operations, Management, and Reliability. Washington, DC: The National Academies Press. doi: 10.17226/22256.
×
Page R15
Page xvi
Suggested Citation:"Front Matter." National Academies of Sciences, Engineering, and Medicine. 2014. Regional Operations Forums for Advancing Systems Operations, Management, and Reliability. Washington, DC: The National Academies Press. doi: 10.17226/22256.
×
Page R16
Page xvii
Suggested Citation:"Front Matter." National Academies of Sciences, Engineering, and Medicine. 2014. Regional Operations Forums for Advancing Systems Operations, Management, and Reliability. Washington, DC: The National Academies Press. doi: 10.17226/22256.
×
Page R17
Page xviii
Suggested Citation:"Front Matter." National Academies of Sciences, Engineering, and Medicine. 2014. Regional Operations Forums for Advancing Systems Operations, Management, and Reliability. Washington, DC: The National Academies Press. doi: 10.17226/22256.
×
Page R18
Page xix
Suggested Citation:"Front Matter." National Academies of Sciences, Engineering, and Medicine. 2014. Regional Operations Forums for Advancing Systems Operations, Management, and Reliability. Washington, DC: The National Academies Press. doi: 10.17226/22256.
×
Page R19
Page xx
Suggested Citation:"Front Matter." National Academies of Sciences, Engineering, and Medicine. 2014. Regional Operations Forums for Advancing Systems Operations, Management, and Reliability. Washington, DC: The National Academies Press. doi: 10.17226/22256.
×
Page R20
Page xxi
Suggested Citation:"Front Matter." National Academies of Sciences, Engineering, and Medicine. 2014. Regional Operations Forums for Advancing Systems Operations, Management, and Reliability. Washington, DC: The National Academies Press. doi: 10.17226/22256.
×
Page R21
Page xxii
Suggested Citation:"Front Matter." National Academies of Sciences, Engineering, and Medicine. 2014. Regional Operations Forums for Advancing Systems Operations, Management, and Reliability. Washington, DC: The National Academies Press. doi: 10.17226/22256.
×
Page R22
Page xxiii
Suggested Citation:"Front Matter." National Academies of Sciences, Engineering, and Medicine. 2014. Regional Operations Forums for Advancing Systems Operations, Management, and Reliability. Washington, DC: The National Academies Press. doi: 10.17226/22256.
×
Page R23
Page xxiv
Suggested Citation:"Front Matter." National Academies of Sciences, Engineering, and Medicine. 2014. Regional Operations Forums for Advancing Systems Operations, Management, and Reliability. Washington, DC: The National Academies Press. doi: 10.17226/22256.
×
Page R24
Page xxv
Suggested Citation:"Front Matter." National Academies of Sciences, Engineering, and Medicine. 2014. Regional Operations Forums for Advancing Systems Operations, Management, and Reliability. Washington, DC: The National Academies Press. doi: 10.17226/22256.
×
Page R25
Page xxvi
Suggested Citation:"Front Matter." National Academies of Sciences, Engineering, and Medicine. 2014. Regional Operations Forums for Advancing Systems Operations, Management, and Reliability. Washington, DC: The National Academies Press. doi: 10.17226/22256.
×
Page R26
Page xxvii
Suggested Citation:"Front Matter." National Academies of Sciences, Engineering, and Medicine. 2014. Regional Operations Forums for Advancing Systems Operations, Management, and Reliability. Washington, DC: The National Academies Press. doi: 10.17226/22256.
×
Page R27
Page xxviii
Suggested Citation:"Front Matter." National Academies of Sciences, Engineering, and Medicine. 2014. Regional Operations Forums for Advancing Systems Operations, Management, and Reliability. Washington, DC: The National Academies Press. doi: 10.17226/22256.
×
Page R28
Page xxix
Suggested Citation:"Front Matter." National Academies of Sciences, Engineering, and Medicine. 2014. Regional Operations Forums for Advancing Systems Operations, Management, and Reliability. Washington, DC: The National Academies Press. doi: 10.17226/22256.
×
Page R29
Page xxx
Suggested Citation:"Front Matter." National Academies of Sciences, Engineering, and Medicine. 2014. Regional Operations Forums for Advancing Systems Operations, Management, and Reliability. Washington, DC: The National Academies Press. doi: 10.17226/22256.
×
Page R30
Page xxxi
Suggested Citation:"Front Matter." National Academies of Sciences, Engineering, and Medicine. 2014. Regional Operations Forums for Advancing Systems Operations, Management, and Reliability. Washington, DC: The National Academies Press. doi: 10.17226/22256.
×
Page R31
Page xxxii
Suggested Citation:"Front Matter." National Academies of Sciences, Engineering, and Medicine. 2014. Regional Operations Forums for Advancing Systems Operations, Management, and Reliability. Washington, DC: The National Academies Press. doi: 10.17226/22256.
×
Page R32
Page xxxiii
Suggested Citation:"Front Matter." National Academies of Sciences, Engineering, and Medicine. 2014. Regional Operations Forums for Advancing Systems Operations, Management, and Reliability. Washington, DC: The National Academies Press. doi: 10.17226/22256.
×
Page R33
Page xxxiv
Suggested Citation:"Front Matter." National Academies of Sciences, Engineering, and Medicine. 2014. Regional Operations Forums for Advancing Systems Operations, Management, and Reliability. Washington, DC: The National Academies Press. doi: 10.17226/22256.
×
Page R34
Page xxxv
Suggested Citation:"Front Matter." National Academies of Sciences, Engineering, and Medicine. 2014. Regional Operations Forums for Advancing Systems Operations, Management, and Reliability. Washington, DC: The National Academies Press. doi: 10.17226/22256.
×
Page R35
Page xxxvi
Suggested Citation:"Front Matter." National Academies of Sciences, Engineering, and Medicine. 2014. Regional Operations Forums for Advancing Systems Operations, Management, and Reliability. Washington, DC: The National Academies Press. doi: 10.17226/22256.
×
Page R36
Page xxxvii
Suggested Citation:"Front Matter." National Academies of Sciences, Engineering, and Medicine. 2014. Regional Operations Forums for Advancing Systems Operations, Management, and Reliability. Washington, DC: The National Academies Press. doi: 10.17226/22256.
×
Page R37
Page xxxviii
Suggested Citation:"Front Matter." National Academies of Sciences, Engineering, and Medicine. 2014. Regional Operations Forums for Advancing Systems Operations, Management, and Reliability. Washington, DC: The National Academies Press. doi: 10.17226/22256.
×
Page R38
Page xxxix
Suggested Citation:"Front Matter." National Academies of Sciences, Engineering, and Medicine. 2014. Regional Operations Forums for Advancing Systems Operations, Management, and Reliability. Washington, DC: The National Academies Press. doi: 10.17226/22256.
×
Page R39
Page xl
Suggested Citation:"Front Matter." National Academies of Sciences, Engineering, and Medicine. 2014. Regional Operations Forums for Advancing Systems Operations, Management, and Reliability. Washington, DC: The National Academies Press. doi: 10.17226/22256.
×
Page R40
Page xli
Suggested Citation:"Front Matter." National Academies of Sciences, Engineering, and Medicine. 2014. Regional Operations Forums for Advancing Systems Operations, Management, and Reliability. Washington, DC: The National Academies Press. doi: 10.17226/22256.
×
Page R41
Page xlii
Suggested Citation:"Front Matter." National Academies of Sciences, Engineering, and Medicine. 2014. Regional Operations Forums for Advancing Systems Operations, Management, and Reliability. Washington, DC: The National Academies Press. doi: 10.17226/22256.
×
Page R42
Page xliii
Suggested Citation:"Front Matter." National Academies of Sciences, Engineering, and Medicine. 2014. Regional Operations Forums for Advancing Systems Operations, Management, and Reliability. Washington, DC: The National Academies Press. doi: 10.17226/22256.
×
Page R43
Page xliv
Suggested Citation:"Front Matter." National Academies of Sciences, Engineering, and Medicine. 2014. Regional Operations Forums for Advancing Systems Operations, Management, and Reliability. Washington, DC: The National Academies Press. doi: 10.17226/22256.
×
Page R44
Page xlv
Suggested Citation:"Front Matter." National Academies of Sciences, Engineering, and Medicine. 2014. Regional Operations Forums for Advancing Systems Operations, Management, and Reliability. Washington, DC: The National Academies Press. doi: 10.17226/22256.
×
Page R45
Page xlvi
Suggested Citation:"Front Matter." National Academies of Sciences, Engineering, and Medicine. 2014. Regional Operations Forums for Advancing Systems Operations, Management, and Reliability. Washington, DC: The National Academies Press. doi: 10.17226/22256.
×
Page R46
Page xlvii
Suggested Citation:"Front Matter." National Academies of Sciences, Engineering, and Medicine. 2014. Regional Operations Forums for Advancing Systems Operations, Management, and Reliability. Washington, DC: The National Academies Press. doi: 10.17226/22256.
×
Page R47
Page xlviii
Suggested Citation:"Front Matter." National Academies of Sciences, Engineering, and Medicine. 2014. Regional Operations Forums for Advancing Systems Operations, Management, and Reliability. Washington, DC: The National Academies Press. doi: 10.17226/22256.
×
Page R48
Page xlix
Suggested Citation:"Front Matter." National Academies of Sciences, Engineering, and Medicine. 2014. Regional Operations Forums for Advancing Systems Operations, Management, and Reliability. Washington, DC: The National Academies Press. doi: 10.17226/22256.
×
Page R49
Page l
Suggested Citation:"Front Matter." National Academies of Sciences, Engineering, and Medicine. 2014. Regional Operations Forums for Advancing Systems Operations, Management, and Reliability. Washington, DC: The National Academies Press. doi: 10.17226/22256.
×
Page R50
Page li
Suggested Citation:"Front Matter." National Academies of Sciences, Engineering, and Medicine. 2014. Regional Operations Forums for Advancing Systems Operations, Management, and Reliability. Washington, DC: The National Academies Press. doi: 10.17226/22256.
×
Page R51
Page lii
Suggested Citation:"Front Matter." National Academies of Sciences, Engineering, and Medicine. 2014. Regional Operations Forums for Advancing Systems Operations, Management, and Reliability. Washington, DC: The National Academies Press. doi: 10.17226/22256.
×
Page R52
Page liii
Suggested Citation:"Front Matter." National Academies of Sciences, Engineering, and Medicine. 2014. Regional Operations Forums for Advancing Systems Operations, Management, and Reliability. Washington, DC: The National Academies Press. doi: 10.17226/22256.
×
Page R53
Page liv
Suggested Citation:"Front Matter." National Academies of Sciences, Engineering, and Medicine. 2014. Regional Operations Forums for Advancing Systems Operations, Management, and Reliability. Washington, DC: The National Academies Press. doi: 10.17226/22256.
×
Page R54
Page lv
Suggested Citation:"Front Matter." National Academies of Sciences, Engineering, and Medicine. 2014. Regional Operations Forums for Advancing Systems Operations, Management, and Reliability. Washington, DC: The National Academies Press. doi: 10.17226/22256.
×
Page R55
Page lvi
Suggested Citation:"Front Matter." National Academies of Sciences, Engineering, and Medicine. 2014. Regional Operations Forums for Advancing Systems Operations, Management, and Reliability. Washington, DC: The National Academies Press. doi: 10.17226/22256.
×
Page R56
Page lvii
Suggested Citation:"Front Matter." National Academies of Sciences, Engineering, and Medicine. 2014. Regional Operations Forums for Advancing Systems Operations, Management, and Reliability. Washington, DC: The National Academies Press. doi: 10.17226/22256.
×
Page R57
Page lviii
Suggested Citation:"Front Matter." National Academies of Sciences, Engineering, and Medicine. 2014. Regional Operations Forums for Advancing Systems Operations, Management, and Reliability. Washington, DC: The National Academies Press. doi: 10.17226/22256.
×
Page R58
Page lix
Suggested Citation:"Front Matter." National Academies of Sciences, Engineering, and Medicine. 2014. Regional Operations Forums for Advancing Systems Operations, Management, and Reliability. Washington, DC: The National Academies Press. doi: 10.17226/22256.
×
Page R59
Page lx
Suggested Citation:"Front Matter." National Academies of Sciences, Engineering, and Medicine. 2014. Regional Operations Forums for Advancing Systems Operations, Management, and Reliability. Washington, DC: The National Academies Press. doi: 10.17226/22256.
×
Page R60
Page lxi
Suggested Citation:"Front Matter." National Academies of Sciences, Engineering, and Medicine. 2014. Regional Operations Forums for Advancing Systems Operations, Management, and Reliability. Washington, DC: The National Academies Press. doi: 10.17226/22256.
×
Page R61
Page lxii
Suggested Citation:"Front Matter." National Academies of Sciences, Engineering, and Medicine. 2014. Regional Operations Forums for Advancing Systems Operations, Management, and Reliability. Washington, DC: The National Academies Press. doi: 10.17226/22256.
×
Page R62
Page lxiii
Suggested Citation:"Front Matter." National Academies of Sciences, Engineering, and Medicine. 2014. Regional Operations Forums for Advancing Systems Operations, Management, and Reliability. Washington, DC: The National Academies Press. doi: 10.17226/22256.
×
Page R63
Page lxiv
Suggested Citation:"Front Matter." National Academies of Sciences, Engineering, and Medicine. 2014. Regional Operations Forums for Advancing Systems Operations, Management, and Reliability. Washington, DC: The National Academies Press. doi: 10.17226/22256.
×
Page R64
Page lxv
Suggested Citation:"Front Matter." National Academies of Sciences, Engineering, and Medicine. 2014. Regional Operations Forums for Advancing Systems Operations, Management, and Reliability. Washington, DC: The National Academies Press. doi: 10.17226/22256.
×
Page R65
Page lxvi
Suggested Citation:"Front Matter." National Academies of Sciences, Engineering, and Medicine. 2014. Regional Operations Forums for Advancing Systems Operations, Management, and Reliability. Washington, DC: The National Academies Press. doi: 10.17226/22256.
×
Page R66
Page lxvii
Suggested Citation:"Front Matter." National Academies of Sciences, Engineering, and Medicine. 2014. Regional Operations Forums for Advancing Systems Operations, Management, and Reliability. Washington, DC: The National Academies Press. doi: 10.17226/22256.
×
Page R67
Page lxviii
Suggested Citation:"Front Matter." National Academies of Sciences, Engineering, and Medicine. 2014. Regional Operations Forums for Advancing Systems Operations, Management, and Reliability. Washington, DC: The National Academies Press. doi: 10.17226/22256.
×
Page R68
Page lxix
Suggested Citation:"Front Matter." National Academies of Sciences, Engineering, and Medicine. 2014. Regional Operations Forums for Advancing Systems Operations, Management, and Reliability. Washington, DC: The National Academies Press. doi: 10.17226/22256.
×
Page R69
Page lxx
Suggested Citation:"Front Matter." National Academies of Sciences, Engineering, and Medicine. 2014. Regional Operations Forums for Advancing Systems Operations, Management, and Reliability. Washington, DC: The National Academies Press. doi: 10.17226/22256.
×
Page R70
Page lxxi
Suggested Citation:"Front Matter." National Academies of Sciences, Engineering, and Medicine. 2014. Regional Operations Forums for Advancing Systems Operations, Management, and Reliability. Washington, DC: The National Academies Press. doi: 10.17226/22256.
×
Page R71
Page lxxii
Suggested Citation:"Front Matter." National Academies of Sciences, Engineering, and Medicine. 2014. Regional Operations Forums for Advancing Systems Operations, Management, and Reliability. Washington, DC: The National Academies Press. doi: 10.17226/22256.
×
Page R72
Page lxxiii
Suggested Citation:"Front Matter." National Academies of Sciences, Engineering, and Medicine. 2014. Regional Operations Forums for Advancing Systems Operations, Management, and Reliability. Washington, DC: The National Academies Press. doi: 10.17226/22256.
×
Page R73
Page lxxiv
Suggested Citation:"Front Matter." National Academies of Sciences, Engineering, and Medicine. 2014. Regional Operations Forums for Advancing Systems Operations, Management, and Reliability. Washington, DC: The National Academies Press. doi: 10.17226/22256.
×
Page R74
Page lxxv
Suggested Citation:"Front Matter." National Academies of Sciences, Engineering, and Medicine. 2014. Regional Operations Forums for Advancing Systems Operations, Management, and Reliability. Washington, DC: The National Academies Press. doi: 10.17226/22256.
×
Page R75
Page lxxvi
Suggested Citation:"Front Matter." National Academies of Sciences, Engineering, and Medicine. 2014. Regional Operations Forums for Advancing Systems Operations, Management, and Reliability. Washington, DC: The National Academies Press. doi: 10.17226/22256.
×
Page R76
Page lxxvii
Suggested Citation:"Front Matter." National Academies of Sciences, Engineering, and Medicine. 2014. Regional Operations Forums for Advancing Systems Operations, Management, and Reliability. Washington, DC: The National Academies Press. doi: 10.17226/22256.
×
Page R77
Page lxxviii
Suggested Citation:"Front Matter." National Academies of Sciences, Engineering, and Medicine. 2014. Regional Operations Forums for Advancing Systems Operations, Management, and Reliability. Washington, DC: The National Academies Press. doi: 10.17226/22256.
×
Page R78
Page lxxix
Suggested Citation:"Front Matter." National Academies of Sciences, Engineering, and Medicine. 2014. Regional Operations Forums for Advancing Systems Operations, Management, and Reliability. Washington, DC: The National Academies Press. doi: 10.17226/22256.
×
Page R79
Page lxxx
Suggested Citation:"Front Matter." National Academies of Sciences, Engineering, and Medicine. 2014. Regional Operations Forums for Advancing Systems Operations, Management, and Reliability. Washington, DC: The National Academies Press. doi: 10.17226/22256.
×
Page R80
Page lxxxi
Suggested Citation:"Front Matter." National Academies of Sciences, Engineering, and Medicine. 2014. Regional Operations Forums for Advancing Systems Operations, Management, and Reliability. Washington, DC: The National Academies Press. doi: 10.17226/22256.
×
Page R81
Page lxxxii
Suggested Citation:"Front Matter." National Academies of Sciences, Engineering, and Medicine. 2014. Regional Operations Forums for Advancing Systems Operations, Management, and Reliability. Washington, DC: The National Academies Press. doi: 10.17226/22256.
×
Page R82
Page lxxxiii
Suggested Citation:"Front Matter." National Academies of Sciences, Engineering, and Medicine. 2014. Regional Operations Forums for Advancing Systems Operations, Management, and Reliability. Washington, DC: The National Academies Press. doi: 10.17226/22256.
×
Page R83
Page lxxxiv
Suggested Citation:"Front Matter." National Academies of Sciences, Engineering, and Medicine. 2014. Regional Operations Forums for Advancing Systems Operations, Management, and Reliability. Washington, DC: The National Academies Press. doi: 10.17226/22256.
×
Page R84
Page lxxxv
Suggested Citation:"Front Matter." National Academies of Sciences, Engineering, and Medicine. 2014. Regional Operations Forums for Advancing Systems Operations, Management, and Reliability. Washington, DC: The National Academies Press. doi: 10.17226/22256.
×
Page R85
Page lxxxvi
Suggested Citation:"Front Matter." National Academies of Sciences, Engineering, and Medicine. 2014. Regional Operations Forums for Advancing Systems Operations, Management, and Reliability. Washington, DC: The National Academies Press. doi: 10.17226/22256.
×
Page R86
Page lxxxvii
Suggested Citation:"Front Matter." National Academies of Sciences, Engineering, and Medicine. 2014. Regional Operations Forums for Advancing Systems Operations, Management, and Reliability. Washington, DC: The National Academies Press. doi: 10.17226/22256.
×
Page R87
Page lxxxviii
Suggested Citation:"Front Matter." National Academies of Sciences, Engineering, and Medicine. 2014. Regional Operations Forums for Advancing Systems Operations, Management, and Reliability. Washington, DC: The National Academies Press. doi: 10.17226/22256.
×
Page R88
Page lxxxix
Suggested Citation:"Front Matter." National Academies of Sciences, Engineering, and Medicine. 2014. Regional Operations Forums for Advancing Systems Operations, Management, and Reliability. Washington, DC: The National Academies Press. doi: 10.17226/22256.
×
Page R89
Page xc
Suggested Citation:"Front Matter." National Academies of Sciences, Engineering, and Medicine. 2014. Regional Operations Forums for Advancing Systems Operations, Management, and Reliability. Washington, DC: The National Academies Press. doi: 10.17226/22256.
×
Page R90
Page xci
Suggested Citation:"Front Matter." National Academies of Sciences, Engineering, and Medicine. 2014. Regional Operations Forums for Advancing Systems Operations, Management, and Reliability. Washington, DC: The National Academies Press. doi: 10.17226/22256.
×
Page R91
Page xcii
Suggested Citation:"Front Matter." National Academies of Sciences, Engineering, and Medicine. 2014. Regional Operations Forums for Advancing Systems Operations, Management, and Reliability. Washington, DC: The National Academies Press. doi: 10.17226/22256.
×
Page R92
Page xciii
Suggested Citation:"Front Matter." National Academies of Sciences, Engineering, and Medicine. 2014. Regional Operations Forums for Advancing Systems Operations, Management, and Reliability. Washington, DC: The National Academies Press. doi: 10.17226/22256.
×
Page R93
Page xciv
Suggested Citation:"Front Matter." National Academies of Sciences, Engineering, and Medicine. 2014. Regional Operations Forums for Advancing Systems Operations, Management, and Reliability. Washington, DC: The National Academies Press. doi: 10.17226/22256.
×
Page R94
Page xcv
Suggested Citation:"Front Matter." National Academies of Sciences, Engineering, and Medicine. 2014. Regional Operations Forums for Advancing Systems Operations, Management, and Reliability. Washington, DC: The National Academies Press. doi: 10.17226/22256.
×
Page R95
Page xcvi
Suggested Citation:"Front Matter." National Academies of Sciences, Engineering, and Medicine. 2014. Regional Operations Forums for Advancing Systems Operations, Management, and Reliability. Washington, DC: The National Academies Press. doi: 10.17226/22256.
×
Page R96
Page xcvii
Suggested Citation:"Front Matter." National Academies of Sciences, Engineering, and Medicine. 2014. Regional Operations Forums for Advancing Systems Operations, Management, and Reliability. Washington, DC: The National Academies Press. doi: 10.17226/22256.
×
Page R97
Page xcviii
Suggested Citation:"Front Matter." National Academies of Sciences, Engineering, and Medicine. 2014. Regional Operations Forums for Advancing Systems Operations, Management, and Reliability. Washington, DC: The National Academies Press. doi: 10.17226/22256.
×
Page R98
Page xcix
Suggested Citation:"Front Matter." National Academies of Sciences, Engineering, and Medicine. 2014. Regional Operations Forums for Advancing Systems Operations, Management, and Reliability. Washington, DC: The National Academies Press. doi: 10.17226/22256.
×
Page R99
Page c
Suggested Citation:"Front Matter." National Academies of Sciences, Engineering, and Medicine. 2014. Regional Operations Forums for Advancing Systems Operations, Management, and Reliability. Washington, DC: The National Academies Press. doi: 10.17226/22256.
×
Page R100
Page ci
Suggested Citation:"Front Matter." National Academies of Sciences, Engineering, and Medicine. 2014. Regional Operations Forums for Advancing Systems Operations, Management, and Reliability. Washington, DC: The National Academies Press. doi: 10.17226/22256.
×
Page R101
Page cii
Suggested Citation:"Front Matter." National Academies of Sciences, Engineering, and Medicine. 2014. Regional Operations Forums for Advancing Systems Operations, Management, and Reliability. Washington, DC: The National Academies Press. doi: 10.17226/22256.
×
Page R102
Page ciii
Suggested Citation:"Front Matter." National Academies of Sciences, Engineering, and Medicine. 2014. Regional Operations Forums for Advancing Systems Operations, Management, and Reliability. Washington, DC: The National Academies Press. doi: 10.17226/22256.
×
Page R103
Page civ
Suggested Citation:"Front Matter." National Academies of Sciences, Engineering, and Medicine. 2014. Regional Operations Forums for Advancing Systems Operations, Management, and Reliability. Washington, DC: The National Academies Press. doi: 10.17226/22256.
×
Page R104
Page cv
Suggested Citation:"Front Matter." National Academies of Sciences, Engineering, and Medicine. 2014. Regional Operations Forums for Advancing Systems Operations, Management, and Reliability. Washington, DC: The National Academies Press. doi: 10.17226/22256.
×
Page R105
Page cvi
Suggested Citation:"Front Matter." National Academies of Sciences, Engineering, and Medicine. 2014. Regional Operations Forums for Advancing Systems Operations, Management, and Reliability. Washington, DC: The National Academies Press. doi: 10.17226/22256.
×
Page R106
Page cvii
Suggested Citation:"Front Matter." National Academies of Sciences, Engineering, and Medicine. 2014. Regional Operations Forums for Advancing Systems Operations, Management, and Reliability. Washington, DC: The National Academies Press. doi: 10.17226/22256.
×
Page R107
Page cviii
Suggested Citation:"Front Matter." National Academies of Sciences, Engineering, and Medicine. 2014. Regional Operations Forums for Advancing Systems Operations, Management, and Reliability. Washington, DC: The National Academies Press. doi: 10.17226/22256.
×
Page R108
Page cix
Suggested Citation:"Front Matter." National Academies of Sciences, Engineering, and Medicine. 2014. Regional Operations Forums for Advancing Systems Operations, Management, and Reliability. Washington, DC: The National Academies Press. doi: 10.17226/22256.
×
Page R109
Page cx
Suggested Citation:"Front Matter." National Academies of Sciences, Engineering, and Medicine. 2014. Regional Operations Forums for Advancing Systems Operations, Management, and Reliability. Washington, DC: The National Academies Press. doi: 10.17226/22256.
×
Page R110
Page cxi
Suggested Citation:"Front Matter." National Academies of Sciences, Engineering, and Medicine. 2014. Regional Operations Forums for Advancing Systems Operations, Management, and Reliability. Washington, DC: The National Academies Press. doi: 10.17226/22256.
×
Page R111
Page cxii
Suggested Citation:"Front Matter." National Academies of Sciences, Engineering, and Medicine. 2014. Regional Operations Forums for Advancing Systems Operations, Management, and Reliability. Washington, DC: The National Academies Press. doi: 10.17226/22256.
×
Page R112
Page cxiii
Suggested Citation:"Front Matter." National Academies of Sciences, Engineering, and Medicine. 2014. Regional Operations Forums for Advancing Systems Operations, Management, and Reliability. Washington, DC: The National Academies Press. doi: 10.17226/22256.
×
Page R113
Page cxiv
Suggested Citation:"Front Matter." National Academies of Sciences, Engineering, and Medicine. 2014. Regional Operations Forums for Advancing Systems Operations, Management, and Reliability. Washington, DC: The National Academies Press. doi: 10.17226/22256.
×
Page R114
Page cxv
Suggested Citation:"Front Matter." National Academies of Sciences, Engineering, and Medicine. 2014. Regional Operations Forums for Advancing Systems Operations, Management, and Reliability. Washington, DC: The National Academies Press. doi: 10.17226/22256.
×
Page R115
Page cxvi
Suggested Citation:"Front Matter." National Academies of Sciences, Engineering, and Medicine. 2014. Regional Operations Forums for Advancing Systems Operations, Management, and Reliability. Washington, DC: The National Academies Press. doi: 10.17226/22256.
×
Page R116
Page cxvii
Suggested Citation:"Front Matter." National Academies of Sciences, Engineering, and Medicine. 2014. Regional Operations Forums for Advancing Systems Operations, Management, and Reliability. Washington, DC: The National Academies Press. doi: 10.17226/22256.
×
Page R117
Page cxviii
Suggested Citation:"Front Matter." National Academies of Sciences, Engineering, and Medicine. 2014. Regional Operations Forums for Advancing Systems Operations, Management, and Reliability. Washington, DC: The National Academies Press. doi: 10.17226/22256.
×
Page R118
Page cxix
Suggested Citation:"Front Matter." National Academies of Sciences, Engineering, and Medicine. 2014. Regional Operations Forums for Advancing Systems Operations, Management, and Reliability. Washington, DC: The National Academies Press. doi: 10.17226/22256.
×
Page R119
Page cxx
Suggested Citation:"Front Matter." National Academies of Sciences, Engineering, and Medicine. 2014. Regional Operations Forums for Advancing Systems Operations, Management, and Reliability. Washington, DC: The National Academies Press. doi: 10.17226/22256.
×
Page R120
Page cxxi
Suggested Citation:"Front Matter." National Academies of Sciences, Engineering, and Medicine. 2014. Regional Operations Forums for Advancing Systems Operations, Management, and Reliability. Washington, DC: The National Academies Press. doi: 10.17226/22256.
×
Page R121
Page cxxii
Suggested Citation:"Front Matter." National Academies of Sciences, Engineering, and Medicine. 2014. Regional Operations Forums for Advancing Systems Operations, Management, and Reliability. Washington, DC: The National Academies Press. doi: 10.17226/22256.
×
Page R122
Page cxxiii
Suggested Citation:"Front Matter." National Academies of Sciences, Engineering, and Medicine. 2014. Regional Operations Forums for Advancing Systems Operations, Management, and Reliability. Washington, DC: The National Academies Press. doi: 10.17226/22256.
×
Page R123
Page cxxiv
Suggested Citation:"Front Matter." National Academies of Sciences, Engineering, and Medicine. 2014. Regional Operations Forums for Advancing Systems Operations, Management, and Reliability. Washington, DC: The National Academies Press. doi: 10.17226/22256.
×
Page R124

Below is the uncorrected machine-read text of this chapter, intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text of each book. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.

SHRP 2 Reliability Project L36 Regional Operations Forums for Advancing Systems Operations, Management, and Reliability Kathleen M. Frankle University of Maryland Center for Advanced Transportation Technology Gary Euler Parsons Brinckerhoff Pierre Pretorius Kimley-Horn Emanuel Robinson Westat TRANSPORTATION RESEARCH BOARD Washington, D.C. 2015 www.TRB.org

© 2015 National Academy of Sciences. All rights reserved. ACKNOWLEDGMENTS This work was sponsored by the Federal Highway Administration in cooperation with the American Association of State Highway and Transportation Officials. It was conducted in the second Strategic Highway Research Program, which is administered by the Transportation Research Board of the National Academies. This project was managed by Neil Pedersen, Deputy Director, Implementation and Communication. The research reported on herein was performed by the University of Maryland Center for Advanced Transportation Technology, supported by Parsons Brinckerhoff, Inc.; Kimley-Horn, Inc.; and Westat, Inc. Kathleen M. Frankle, University of Maryland Center for Advanced Transportation Technology, was the Principal Investigator. The other authors of this report are Gary Euler, Parsons Brinckerhoff; Pierre Pretorius, Kimley-Horn; and Emanuel Robinson, Westat. COPYRIGHT INFORMATION Authors herein are responsible for the authenticity of their materials and for obtaining written permissions from publishers or persons who own the copyright to any previously published or copyrighted material used herein. The second Strategic Highway Research Program grants permission to reproduce material in this publication for classroom and not-for-profit purposes. Permission is given with the understanding that none of the material will be used to imply TRB, AASHTO, or FHWA endorsement of a particular product, method, or practice. It is expected that those reproducing material in this document for educational and not-for-profit purposes will give appropriate acknowledgment of the source of any reprinted or reproduced material. For other uses of the material, request permission from SHRP 2. NOTICE The project that is the subject of this document was a part of the second Strategic Highway Research Program, conducted by the Transportation Research Board with the approval of the Governing Board of the National Research Council. The Transportation Research Board of the National Academies, the National Research Council, and the sponsors of the second Strategic Highway Research Program do not endorse products or manufacturers. Trade or manufacturers’ names appear herein solely because they are considered essential to the object of the report.

DISCLAIMER The opinions and conclusions expressed or implied in this document are those of the researchers who performed the research. They are not necessarily those of the second Strategic Highway Research Program, the Transportation Research Board, the National Research Council, or the program sponsors. The information contained in this document was taken directly from the submission of the authors. This material has not been edited by the Transportation Research Board. SPECIAL NOTE: This document IS NOT an official publication of the second Strategic Highway Research Program, the Transportation Research Board, the National Research Council, or the National Academies.

The National Academy of Sciences is a private, nonprofit, self-perpetuating society of distinguished scholars engaged in scientific and engineering research, dedicated to the furtherance of science and technology and to their use for the general welfare. On the authority of the charter granted to it by Congress in 1863, the Academy has a mandate that requires it to advise the federal government on scientific and technical matters. Dr. Ralph J. Cicerone is president of the National Academy of Sciences. The National Academy of Engineering was established in 1964, under the charter of the National Academy of Sciences, as a parallel organization of outstanding engineers. It is autonomous in its administration and in the selection of its members, sharing with the National Academy of Sciences the responsibility for advising the federal government. The National Academy of Engineering also sponsors engineering programs aimed at meeting national needs, encourages education and research, and recognizes the superior achievements of engineers. Dr. C. D. (Dan) Mote, Jr., is president of the National Academy of Engineering. The Institute of Medicine was established in 1970 by the National Academy of Sciences to secure the services of eminent members of appropriate professions in the examination of policy matters pertaining to the health of the public. The Institute acts under the responsibility given to the National Academy of Sciences by its congressional charter to be an adviser to the federal government and, upon its own initiative, to identify issues of medical care, research, and education. Dr. Victor J. Dzau is president of the Institute of Medicine. The National Research Council was organized by the National Academy of Sciences in 1916 to associate the broad community of science and technology with the Academy’s purposes of furthering knowledge and advising the federal government. Functioning in accordance with general policies determined by the Academy, the Council has become the principal operating agency of both the National Academy of Sciences and the National Academy of Engineering in providing services to the government, the public, and the scientific and engineering communities. The Council is administered jointly by both Academies and the Institute of Medicine. Dr. Ralph J. Cicerone and Dr. C.D. (Dan) Mote, Jr., are chair and vice chair, respectively, of the National Research Council. The Transportation Research Board is one of six major divisions of the National Research Council. The mission of the Transportation Research Board is to provide leadership in transportation innovation and progress through research and information exchange, conducted within a setting that is objective, interdisciplinary, and multimodal. The Board’s varied activities annually engage about 7,000 engineers, scientists, and other transportation researchers and practitioners from the public and private sectors and academia, all of whom contribute their expertise in the public interest. The program is supported by state transportation departments, federal agencies including the component administrations of the U.S. Department of Transportation, and other organizations and individuals interested in the development of transportation. www.TRB.org www.national-academies.org

Contents 1 Executive Summary 3 CHAPTER 1 Background 3 Problem Statement 3 Objective 4 CHAPTER 2 Research Approach 4 Introduction 6 CHAPTER 3 Findings and Applications 6 Task 1 – Literature Review of the SO&M State of the Art 10 Task 2 – Regional Operations Forum Agenda 14 Task 3 – Regional Operations Forum Venues for Pilot Deliveries 17 Task 4 – Develop the Regional Operations Forum Curriculum 19 Task 5 – Scheduling and Outreach 21 Task 6 – Regional Operations Forum Pilot Program Delivery 31 Task 7 – Final Report 35 CHAPTER 4 Conclusions and Suggested Research 35 Conclusions 36 Suggested Research 37 Reference A-1 APPENDIX A Final Matrix of ROF Sessions B-1 APPENDIX B Pilot R0F Agendas C-1 APPENDIX C Evaluation Forms D-1 APPENDIX D Scheduling and Outreach Documents E-1 APPENDIX E ROF Evaluation Summaries F-1 APPENDIX F Sustainability Options

1 Executive Summary The overall goal of Project L36, Regional Operations Forums for Advancing Systems Operations, Management, and Reliability, was to advance transportation system management and operations (TSM&O) and serve as a platform for mainstreaming the second Strategic Highway Research Program (SHRP 2) Reliability research results through a regional operations forum (ROF) concept. The term regional describes a geographic area comprising several states or several metropolitan areas within a state. The curriculum offered through the ROFs was intended to provide pertinent education and training on TSM&O business processes; organizational capabilities; operations and planning; and design, technical, and analytical issues to a broad range of transportation agency representatives. The objectives for the project were:  To design an ROF curriculum based on a comprehensive review of available and anticipated research results of the SHRP 2 Reliability program and other information available from TSM&O resources, including federal, state, and local initiatives;  To pilot test the curriculum developed, including: o Continually adjusting the curriculum to incorporate SHRP 2 research outcomes as they became available; and o Conducting an ongoing evaluation of the effectiveness of the curriculum and its delivery; and  To develop a sustainable business model for the ROFs. The project has met both its goals and objectives. Specifically, the following tasks have been accomplished as a result of this research project:  Curriculum was developed and delivered for a 4-day, in-person ROF. Some of the curriculum was included in the form of prestudy materials and recorded speaker sessions that were played on DVDs during the ROF; the remaining sessions were delivered in person.  Five ROF pilots were scheduled, coordinated, and delivered in: o Nashville, Tennessee o Seattle, Washington o Phoenix, Arizona o Concord, New Hampshire o Milwaukee, Wisconsin  An additional ROF was delivered in Orange, California, by the research team and funded by Caltrans.  A 1-day follow-up was scheduled, coordinated, and delivered in Nashville, Tennessee.

2  A report on the future sustainability of the ROF program was developed and presented at the May 2014 Implementation Planning Workshop (IPW).  A section dedicated to the ROFs was created on the Knowledge Transfer System (KTS) website at http://www.tsmoinfo.org/resources/regional_operations_forums.aspx. The website was developed to allow the page to be easily edited without any additional funding required. Based on the results of the independent evaluation done by the research team and the discussions during the IPW in May 2014, the ROF pilots have been a success, and discussions are now focused on how to continue delivery of additional ROFs to other locations. In addition, there are plans to conduct follow-ups in regions that have already participated in one of the ROF pilots.

3 CHAPTER 1 Background Problem Statement Project L36, Regional Operations Forums for Advancing Systems Operations, Management, and Reliability, was expected to advance TSM&O and serve as a platform for mainstreaming SHRP 2 Reliability research results through an ROF concept. The curriculum offered through the ROFs was intended to provide pertinent education and training on TSM&O business processes; organizational capabilities; operations and planning; and design, technical, and analytical issues to a broad range of transportation agency representatives. Objective The objectives for the project were  To design a curriculum for the ROFs based on a comprehensive review of available and anticipated research results of the SHRP 2 Reliability program and other information available from TSM&O resources, including federal, state, and local initiatives; and  To pilot test the curriculum developed, including: o Continually adjusting the curriculum to incorporate SHRP 2 research outcomes as they became available, o Conducting an ongoing evaluation of the effectiveness of the curriculum and its delivery, and o Developing a sustainable business model for the ROFs.

4 CHAPTER 2 Research Approach Introduction The following principles, which were provided in the project request for proposal (RFP), were used as the basis for the development of the ROFs: 1. The ROF curriculum was designed to be beneficial both to practitioners in the field of TSM&O and to attendees whose experience is outside the TSM&O field. 2. The ROFs were designed to take advantage of the variety of experiences and backgrounds that exist among potential attendees, including practitioners and policy makers, various levels of management, various disciplines (planning, engineering, operations, etc.), and various organizations (cities, metropolitan planning organizations [MPOs], departments of transportation [DOTs], law enforcement agencies, fire, emergency response, etc.). 3. The curriculum was designed for flexibility in implementation so that it could be readily tailored to the interests and capabilities of the regions in which the ROFs were delivered and to varying mixes of attendees (for example, ROFs in which attendees are from both state and local organizations and those attended exclusively by state employees). 4. The curriculum was designed to balance presentation and discussion of technical subjects (tools, analytical techniques, devices, standards, testing, etc.) and nontechnical subjects (planning, programming, organization, workforce, funding, collaboration, etc.). The research approach featured the following elements:  Curriculum design based on a comprehensive review of available and anticipated research results of the SHRP 2 Reliability program and a variety of other sources combined with existing TSM&O educational and materials, including the highly successful Operations Academy Senior Management Program;  Pilot testing and evaluation of the ROF curriculum design in five ROFs and one follow- up session around the country. Continual adjustments to curriculum subject matter were made as experience in earlier forums was gained and new information became available; and  Development of information to support decisions being made regarding establishment of a sustainable business model for continued delivery of ROFs beyond this SHRP 2 L36 initiative.

5 Throughout the project, guidance and direction was provided by the SHRP 2 L36 project team consisting of the Transportation Research Board SHRP 2 staff and the L36 Technical Expert Task Group (TETG), along with representatives from the Federal Highway Administration (FHWA), the American Association of State Highway and Transportation Officials (AASHTO), and other public sector and industry representatives.

6 CHAPTER 3 Findings and Applications Task 1—Literature Review of the SO&M State of the Art This task focused on identifying material related to systems operations and management (SO&M) with the greatest relevance to curriculum development for the intended audiences of the ROFs. The material needed to cover a broad set of issues responsive to the range of contexts of targeted ROF attendees, such as urban versus rural settings, weather and transportation network characteristics, level of experience, and resources. The approach to Task 1 took advantage of the literature references that were already included (and synthesized in detail) in the Knowledge Transfer System (KTS) developed under the SHRP 2 L17 project, including references to SHRP 2 products (see http://tsmoinfo.org/documents/). At the start of Task 1, the L17 KTS contained 61 TSM&O and reliability-related resources. To refine the literature review, gaps in the existing KTS subject matter coverage were identified and compared against subject areas/topics deemed most relevant to the ROF agenda and curriculum. Table 3.1 presents a list of curriculum subject areas (in the left column) selected to ensure comprehensive consideration of all aspects of the TSM&O field. These general subject areas are further broken down into more specific topics (in the right column) that identify more detailed aspects. This breakdown reflects a subject/topic structure originally developed under the SHRP 2 L06 project (Institutional Architectures to Advance Operational Strategies) and subsequently used in FHWA-sponsored workshops implementing the results of that research (Organizing for Reliability—Assessment and Implementation Plan Development), as well as the structure of the searchable L17 KTS database. Table 3.1 indicates in italicized font those subject areas or topics that lacked coverage in the KTS database. Those in bold (whether or not italicized) were deemed most relevant to the ROFs. Thus, those terms that are both in bold and italics were the focus of the literature review. Table 3.1. Identification of Key Subject Areas/Topics Subject Areas Topics Managerial Issues 1. Business Processes  Planning  Programming/budget  Performance measurement  Procurement mechanisms and processes  Project development  Integration with safety, maintenance, and new capacity projects

7 2. Performance Measurement  Measures definition  Data collection  Analytics  Utilization 3. Funding  Federal  State  Tolling and pricing  Innovative  Regional/Local 4. Policy/Program/Regulations Development  Federal TSM&O  State TSM&O  Regional TSM&O  Corridor TSM&O 5. Outreach/Marketing  Business cases  Branding  Communicating with media  Social networks Institutional Issues 6. Culture  Leadership  Policy/decision maker support  Customer service  Program hierarchy  Program formality 7. Organization/Workforce  Organizational structures  Reporting relationships  Staff training/certification  Conditions of employment  Career paths  Recruitment/retention  Blended contractor and agency staff 8. Education  Communicating reliability information  Course curricula  Training programs

8 9. Collaboration  State/MPO/local  DOT/Public Safety Agencies & Emergency Response  Public-private Technical Issues and State of Practice Examples 10. Systems and Technology  System architecture  Systems engineering  Verification & validation (V&V)  ITS technology (monitor, control, disseminate)  Standards 11. Active Traffic Management  Lane and speed control  Queue warning  Shoulder use/hard shoulder running  Adaptive ramp metering  Junction control 12. Managed Lanes  HOV  Managed lanes/high-occupancy toll (HOT)  Enforcement 13. Travel Demand Management  Pricing  Vehicle miles traveled (VMT) charging  Dynamic ride matching  Active parking management  Transit signal priority 14. Planned and Unplanned Events  Active transportation and demand management (ATDM)  Integrated corridor management  Traffic incident management  Work zone management  Road weather management  Traffic control device operation  Emergency transportation operations  Transit management operations  Freight management operations 15. Vehicle Technologies  Connected vehicle  Portable devices

9 16. Goods Movement  Importance of reliability  Truck freight information systems  Preferential treatment  Truck parking systems 17. Traveler Information  Mobile applications  Pre-trip  In-vehicle information systems  Roadside information systems  Freight information systems 18. Data and Analytic Tools  Traveler behavior  Traffic dynamics  Reliability predictive methods  Evaluation of operations strategies  Data collection and management  Performance measurement  Economic analysis of reliability  Costs and benefits  Capability maturity model (CMM) Sources that were reviewed or contacted to identify references that addressed the missing key subject areas/topics identified above included: 1. Ongoing SHRP 2 research 2. Other key non-SHRP 2 TSM&O documents already synthesized in the L17 KTS 3. The Transportation Management Center Pooled Fund Study 4. The Operations Academy list server 5. U.S. Department of Transportation (USDOT) websites, primarily the Office of Operations and the RITA Joint Program Office Knowledge Resource site 6. FHWA workshop/training materials 7. The RITA Professional Capacity Building program 8. National Highway Institute course lists 9. Consortium for ITS Training and Education (CITE) courses 10. State DOTs websites

10 11. Discussions with key FHWA personnel Through this approach, 23 new resources were identified and synthesized into the L17 KTS. The addition of the 23 resources brought the total number of resources available for use in ROF agenda and curriculum development (and in the broader TSM&O/Reliability arena) to 84. This process helped ensure that the participants benefited from the latest findings and thinking from the SHRP 2 program and other sources in those areas judged to be most relevant to typical participants. Through use of the KTS’s online search feature, curriculum developers had one-stop access to these documents and could easily incorporate information into the ROF curriculum materials. The process also served to enhance the KTS website through the addition of important new materials. Task 2— Regional Operations Forum Agenda This task focused on development of the agenda framework and draft agenda, identification of instructors and facilitators, and development of the ROF performance measurement system. A. Development of the Agenda Framework, Draft Agenda, and Identification of Instructors and Facilitators The development of the agenda for the forums was an interactive process that spanned the entire length of the project. The topics for the agenda were garnered from two sources: the key topics from the Operations Academy Senior Management Program that were spelled out in the RFP coupled with the topics that were deemed most relevant to the ROFs under Task 1 (see Table 3.1). The research team developed an initial matrix of potential sessions for the ROFs with some topics classified as “core” or required for all ROFs while the other topics were classified as “optional” and could be selected by each ROF location based upon interests of the region. These two types of sessions were designed to provide flexibility to the ROF pilot locations to select sessions most relevant to their regional needs. The research team made sure to include an appropriate mix of managerial, institutional, and technical subjects. The seven columns in the matrix included 1. Session title 2. Description of content to be covered 3. Subject areas relevant to the session (from Task 1 list of potential subjects that could be covered during the ROFs) 4. Related SHRP 2 projects, products, and results that needed to be addressed during the session 5. Products and reports that are included in the SHRP 2 L17 KTS database that could be used to assist in development of the session or as related reading or resources for ROF participants

11 6. Potential delivery mechanisms 7. Suggested content developer and presenter The initial matrix was presented to the TETG in March 2013. Using this initial matrix, the research team developed a first draft of the 4-day ROF agenda. This draft agenda included placeholders for the “core” sessions, as well as a certain number of “optional” sessions that could be selected by the ROF pilot locations. The draft agenda was also presented to the TETG in March 2013. The TETG provided feedback on both the initial version of the matrix and the first draft of the 4-day ROF agenda. Both were edited based on the feedback received. The final version of the matrix is shown in Appendix A. It identified “recommended core” and “optional” sessions and contained the seven columns shown earlier for each session. The final version of the matrix, plus the draft agenda, were then shared with the five ROF locations (see Task 3) for them to select the sessions that they wanted delivered during their particular ROF pilot. During May 2013, the research team reached out to the state leads for all five ROF pilots with the goal of presenting five draft agendas (one for each ROF pilot) to the TETG during their face-to-face meeting in June 2013. Feedback received from the pilot locations and presented during the June 2013 meeting revealed that all pilot locations wanted to include just about every session topic in the agenda. By the end of the 2-day TETG meeting, it was decided that all ROF sessions would have the same agenda, and a final draft agenda was developed by the end of the meeting. The agenda developed at the June 2013 meeting was the one used in the first pilot ROF that was held in Nashville, Tennessee, on September 9 to 13, 2013. It is important to note that the agendas for each pilot ROF location were all slightly different due to the feedback received from the performance measurement system process (see description below). Evaluation was conducted during each of the five ROF pilot offerings, and based on the feedback received from the participants, the agenda, and curriculum materials for the next offering were modified. Appendix B includes a copy of the agendas from each of the five ROF pilots, plus the California offering. Task 3 covers the California ROF. Table 3.2 lists all the sessions from the final agenda, plus prestudy material, and indicates the subject area group and delivery mechanism for each session. The point of the table is to show that the final ROF program included an appropriate mix of the managerial, institutional, and technical subjects that were identified under Task 1. Table 3.2. Agenda Development Framework Session Title Subject Area Group Delivery Mechanism Overview of Operations Institutional Online course taken through prestudy. Capability Maturity Model Managerial Prestudy includes a six-page article on the CMM and a self

12 (CMM) Self Evaluations evaluation is submitted via e-mail to the research team. 2-hour session in person during ROF. Self evaluations submitted are also summarized and reviewed. Planning and Programming for Operations Managerial Overview of “What is Planning for Operations” included in prestudy, plus optional resources documents to review. 2-hour session in person during ROF. Performance Measurement Managerial 4-hour session in person during ROF. Contained a small group exercise. Facilitating Goods Movement through Operations Technical 1-hour videotaped session shown to participants via DVD during the ROF. Q&A followed with instructor or facilitator. Systems Engineering Technical Lessons 1 and 2 of CITE online course “Introduction to Systems Engineering” as part of prestudy. 1-hour session in person to discuss systems engineering processes used by the states. Technical Tour Technical Visit to local transportation management center or project site. Traffic Incident Management, Emergency Operations and Planned Special Events Technical 3-hour session in person during ROF. Contained small group exercises. Safety and Operations * Technical 1-hour recorded session shown to participants via DVD during ROF. Q&A followed with instructor or facilitator. Road Weather Technical 1.25-hour session in person at ROF. Contained a small group exercise. Work Zones Technical 1-hour session in person at ROF. Contained a small group exercise. Traveler Information and Operations Technical 1.5-hour session in person at ROF. Contained a small group exercise. Managing a Corridor (includes ICM, ATM, Managed Lanes) 3-hour session in person during ROF. Contained a small group exercise. How to Organize for Operations (includes Workforce Development) Institutional Chapter in prestudy materials on Workforce Development in “Road Ahead” book. 1.5-hour session in person during ROF. Contained small group exercises/discussions. Communicating the Value of Operations Institutional 1.5-hour session in person during ROF. Contained a small group discussion.

13 *This session was removed from the last two ROF pilots because the video presentation was not effective in conveying the intended information. The content could be presented by a local speaker at future ROF offerings. B. Description of Performance Measurement System for ROF Pilots An impartial evaluation was conducted for each ROF pilot and was critical to the success of the ROF program. The performance measurement process sought to address two distinct aspects of feedback from the participants. First, feedback was sought from participants that would provide the research team with the necessary information to make adjustments to the ROF curriculum and instruction. Second, information was gathered to measure whether the ROF pilots were attaining their stated goals and objectives. The second aspect was assessed using several critical questions related to the conduct of and outcomes that could be attributed to the ROFs, including questions of content focus, learning, addressing participant needs, and lessons learned that could lead to long-term organizational behavior change. The performance measurement system process that was developed and used for the ROF pilots used a modular approach and included the following:  Twice a day, brief survey forms with closed-ended items and open-ended space to include qualitative feedback were used to collect information from the ROF participants for all sessions and speakers. The content was consistent with the Kirkpatrick (2006) four-step approach to collect data to address process and contextual issues, as well as knowledge acquisition and anticipated behavioral changes. The evaluation forms were collected by the logistics coordinator and placed into sealed envelopes. The envelopes were provided to the independent evaluator on the last day of the ROF.  A trained observer and interviewer, who is part of the independent evaluation team, visited each ROF site on the final day to independently collect performance measures. When the ROF pilot was completed on the last day, the evaluator held a brief moderated Connected Vehicles and the Future of Transportation Technical 1-hour videotaped session shown to participants via DVD during ROF. Q&A followed with instructor or facilitator. Team Exercise Institutional Managerial Technical Participants separated so one member of every participating state was a member of each group. Designed to reinforce the CMM model and TSM&O concepts. The background materials on an imaginary state DOT, for the workshop were included as part of the prestudy materials. Agency Implementation Plans Institutional Managerial Participants divided into groups by agency. Designed so states develop action strategies centered on the CMM that they can implement when they go back to their agency.

14 discussion with ROF participants without the instructors, the facilitator, or any other ROF project staff to ensure participants could speak freely. This format allowed for more dynamic feedback than can usually be given in open-ended responses. Participants are often reluctant to write detailed responses but will elaborate more in facilitated guided discussions.  On the final day of the ROF, prior to the moderated discussion, the independent evaluator also asked participants to complete a survey assessing the overall conduct and effectiveness of the ROF.  All performance measures, including notes from observations and discussions, were entered into an electronic format (transcribed notes and ratings entered into a database). The data were kept separate from the rest of the research team and housed at the offices of the independent evaluator to ensure independence.  Overall summaries were developed after each ROF that included both quantitative and qualitative feedback, creating a case study for each ROF. Individual ROF case study reports were timed to allow improvements to be built into the implementation of subsequent ROFs. This emphasis on early discovery of potential weaknesses that can be corrected as the program continues is critical to an approach intended to provide both formative and summative evaluative information.  A follow-up web survey was sent from one to six months after the training and collected information about the retention of knowledge and skills and the use of the knowledge and skills to change operational practices. A copy of all the evaluation forms used for the ROF pilot program can be found in Appendix C. Task 3—Regional Operations Forum Venues for Pilot Deliveries The research team was directed by the TETG to schedule at least one pilot ROF in each of the four AASHTO regions and was provided contacts at four states who had expressed an interest in hosting an ROF. Based on the budget that was developed, the goal was to deliver five pilot ROFs so the last ROF location was identified based on interest from a state DOT and their participation in the Operations Academy Senior Management Program. Based on this information, the research team contacted the five states who had expressed an interest in hosting an ROF pilot. Information was gathered concerning the following:  Solidification of commitment by the host agency and identification of one lead contact;  Identification of other participating states that made sense for a regional forum;  Identification of a lead for the other participating states;  Potential months to hold the ROF based on local weather and events;  Host agency to identify a potential field trip (would affect location of the ROF); and  Potential locations (city and hotel) to hold the ROF.

15 The research team developed a summary document of the five potential ROF locations and submitted it to the TETG to be discussed during their March 2013 teleconference. The TETG was asked if the locations and mix of participating states were acceptable. With only minor adjustments to the initial suggestions, the TETG approved the five pilot locations, which are shown in Table 3.3. Once the TETG approved all the pilot locations, the research team began the process of scheduling the ROFs. This information is contained under Task 5. It should be noted that the budget for the project was developed based on funding a total of 30 participants per ROF pilot. Therefore, the number of participants per state that could be funded by the project varied based on the number of participating states.

16 Table 3.3. Summary of ROF Locations by AASHTO Region Region States Date Location Participants Invited Region 1 New Hampshire (host) Vermont Maine Connecticut Rhode Island Hudson Valley, N.Y. Massachusetts April 28 to May 2, 2014 Concord, N.H. 4 participants per state Region 2 Tennessee (Host) North Carolina Georgia Kentucky Mississippi Missouri September 9 to 13, 2013 Nashville, Tenn. 5 participants per state Region 3 Wisconsin (host) Minnesota Illinois Indiana Michigan Iowa Kansas June 9 to 13, 2014 Milwaukee, Wis. 5 participants per state Region 4 Arizona (host) New Mexico Colorado Utah Nevada Southern California Texas February 10 to 14, 2014 Phoenix, Ariz. 5 participants per state Washington (host) Oregon Idaho Montana Northern California November 4 to 8, 2013 Seattle, Wash. 6 participants per state During the course of the project, the research team was approached by both the California Department of Transportation (Caltrans) and the Colorado DOT to deliver a regional forum specifically for their state. As a result, an ROF was conducted in Orange, California, the week of May 12 to 16, 2014. The forum was presented by the research team, but it was funded by Caltrans. The research team worked with Caltrans staff to walk them through the steps that

17 needed to be accomplished to deliver the ROF. Feedback from the California ROF is included with the evaluation results. The ROF for the Colorado DOT was being planned as this report was being prepared with a tentative date of November 17 to 21, 2014, in Denver, Colorado. Task 4—Develop the Regional Operations Forum Curriculum A. Development of the ROF Curriculum The development of the ROF curriculum began initially in tandem with the activities of Task 2. As previously discussed under Task 2, the major deliverables were the matrix of Recommended Core and Optional Sessions, along with the development of the draft agenda and performance measurement plan. The focus of this task was to develop the course materials for delivery during the ROFs, apply the appropriate learning methods for each topic, and modify curriculum as needed throughout the project. Modifications to the curriculum could result from two different events. The first is the evaluation of the effectiveness of the curriculum and its delivery received from the performance measurement activities that occurred during each ROF. The second stems from the need to incorporate SHRP 2 research outcomes as they became available. The first step in the development of the curriculum happened in April 2013 when the TETG approved the final matrix of Recommended Core and Optional Sessions along with the draft agenda (Task 2). Once that occurred, the research team engaged the subject matter experts on the research team that were identified for each session to begin the development of outlines that would be discussed during the June 2013 face-to-face TETG meeting. Outlines were created for all sessions and included the following information:  Session Title  Outline Developer  Targeted Length of Session  Session Purpose  Desired Outcomes  Content Outline  Related SHRP 2 Reliability Products  Description of Any Exercises or Discussions  Listing of Other Resources During the June 2013 face-to-face TETG meeting, the group reviewed all the outlines in detail. The end results were that some sessions were slightly edited, some were combined, some were cut out completely, and some were sliced up and put into several other sessions. Immediately following the June meeting, the research team modified the session outlines based on the feedback received and began the development of the full content. There were a few outlines that were completely redone and needed TETG approval prior to the content development stage.

18 Each session of the ROF curriculum was developed in a PowerPoint presentation format with detailed speaker notes that fit the topic descriptions and session length. The draft version of the curriculum was delivered to the TETG on August 2, 2013. The TETG members split up the sessions so that all TETG members did not have to review all sessions. Once feedback was received by the TETG on all the sessions, the final version of the curriculum for the first pilot was delivered in Nashville, Tennessee, the week of September 9 to 13, 2013. B. Apply the Appropriate Learning Methods and Media When the curriculum matrix was developed in Task 2, one of the columns of the matrix was potential delivery mechanisms. Table 3.2 shown under Task 2 lists all the sessions from the final agenda plus prestudy material and has a column specifically for delivery mechanism. Because the cost of delivery of the ROFs was a critical issue, both for the pilots and future ROF offerings beyond this project, the delivery mechanism for each session was reviewed carefully by the research team and the TETG. At this point of the ROF program, there were three delivery methods that were utilized: online materials through the prestudy, videotaped sessions of several one-hour speaker sessions, and live delivery during the ROF using an instructor or facilitator. The majority of the content was delivered in person during the ROF. This particular mix worked well for the pilot ROFs, but that does not prohibit the conversion of other curriculum sessions into webinars or online course materials for future delivery of the ROFs. C. Performance Measurement System for Evaluating ROFs and Curriculum Modification As discussed under Task 2 above, all aspects of each ROF were evaluated including prestudy materials, instructors, recorded speaker sessions, technical tour, team exercise, agency implementation plans, and meeting facility. Approximately 1 week following the delivery of every ROF pilot, the research team’s independent evaluator supplied a summary of the evaluation results. These results were distributed to all subject matter experts from the research team who had developed and delivered the ROF curriculum. A conference call was then held to discuss the results and modifications that needed to be made to the curriculum based on the feedback received. At that time, the team also discussed if there were any other modifications that needed to be made based on any new SHRP 2 products or other research that had become available. As expected, modifications to the curriculum following the initial ROF pilot in Nashville, Tennessee, were the greatest. The evaluation process was the same for every ROF pilot delivery, but modifications between sessions became fewer with every offering. It is important to note that besides the evaluation received from the independent evaluator, the research team also received feedback from FHWA representatives who attended the ROF pilots or were subject matter experts. Those comments were also used to modify the ROF curriculum.

19 Task 5—Scheduling and Outreach With the approval of the five pilot locations and participating states accomplished at the end of Task 3, scheduling and outreach for the pilots began in earnest. Activities under this task focused on:  Research, selection, and coordination for meeting space, lodging, and meals;  Identification of participants;  Solidification of the technical tour with the host agency;  Identification and coordination with any local speakers;  Coordination with participants for lodging, prestudy requirements, and ROF logistics; and  Provision to the participants of electronic versions of the curriculum before arriving at the ROF. The research team used the leads that were identified for each participating state to identify appropriate participants for the ROF. This effort proved to be easy in some states and quite challenging in others. The process was easier in states where at least one person from the agency had already attended the Operations Academy Senior Management Program. In those states, the agencies recognized the education and training that would be received at the ROF would be worth the time. Another challenge with participants during this task was that for every ROF pilot, several people dropped out at the last minute (for various reasons), and it was too late in the process to find a replacement. To make the outreach part of this task successful, several documents were developed and used to coordinate and communicate with participants. These documents included the following documents:  ROF Fact Sheet—This document includes the background, description, goals, objective, and target audience of the ROFs. It also includes the specific date and location of the ROF, information on travel, and the draft agenda as an attachment.  Participant Commitment Letter—All participants were required to sign the commitment letter. This was developed for two reasons: first, participants would understand what was expected of them (prestudy and attendance every day during the ROF); and second, to gather information about lodging needs and any special dietary requirements.  PowerPoint Presentation on overall ROF and prestudy requirements—Once all the participants for a specific ROF were identified and signed the Participant Commitment Letter, a webinar was conducted. During the webinar, the research team informed participants about the general goals and objectives of the ROF program as well as specific information about their particular ROF (dates, location, and travel logistics). They also provided a draft agenda, described what to expect each day, and reviewed all

20 the prestudy requirements, including due dates and how to submit the required information.  Prestudy Instructions—These instructions were provided to all participants and spelled out specifically what was required, along with due dates and how to submit the information. This same information was also included in the PowerPoint presentation discussed earlier.  Logistics—A sheet was provided to all participants that included specific information concerning the meeting location of the ROF, shuttle information, meals, technical tour information (if held as an optional activity), and any hotel check in information.  Travel Reimbursement Information—Since all travel costs associated with participation in the ROF were covered by the project, an information sheet was provided to all participants, which included instructions for making travel arrangements, specific travel costs that were covered, and the travel reimbursement process plus form. Generic versions of most of these documents are shown in Appendix D. Electronic versions of all the ROF curriculum sessions were sent to ROF participants prior to the start of the first day of their pilot ROF; hardcopies of instructor material were not provided at any ROF pilot. Initially, participants were provided with access to a web-based folder so they could download the PDF files to their computer or tablet. Eventually, a section was added to the SHRP 2 L17 project KTS website specifically on the regional operations forums. Not only did this provide an easy way to provide the curriculum files to participants, it provided the ROF participants with access to a range of information about the ROFs, including goals and objectives, a detailed agenda, prestudy material, access to the links for the taped video sessions, a calendar, and program contact information. Further, providing the ROF participants with information about the KTS website offered them a valuable resource for finding TSM&O resources, activities, and dialogue. Figure 3.1 provides a screenshot of the ROF home page that is included on the KTS website. The research team did not actually develop the ROF section on the KTS website but provided the information that was included on the site and kept the page updated. Funding for the development of the ROF section was provided under a separate project. The URL for the ROF portion of the KTS is http://www.tsmoinfo.org/resources/regional_operations_forums.aspx.

21 Figure 3.1. Screenshot of the ROF home page created for the Knowledge Transfer System website. Task 6—Regional Operations Forum Pilot Program Delivery A. Delivery of Five Pilot ROFs A total of five ROF pilots were funded and delivered by this project, and a sixth ROF was delivered but was funded directly by Caltrans. Table 3.4 below provides a summary of the ROFs that were delivered.

22 Table 3.4. Summary of ROFs Delivered—Date Order Date Location Participating States/ Jurisdictions # of Graduates September 9 to 13, 2013 Nashville, Tenn. Tennessee (Host) North Carolina Georgia Kentucky Mississippi Missouri 31 November 4 to 8, 2013 Seattle, Wash. Washington (host) Oregon Idaho Montana Northern California 32 February 10 to 14, 2014 Phoenix, Ariz. Arizona (host) New Mexico Colorado Utah Nevada Southern California Texas 34 April 28 to May 2, 2014 Concord, N.H. New Hampshire (host) Vermont Maine Connecticut Rhode Island Hudson Valley, N.Y. Massachusetts 24 May 12 to 16, 2014 Orange, Calif. California DOT City Chula Vista California Highway Patrol El Dorado County Trans Comm METRO SCT Mexico Riverside County Trans Depart Sac County DOT Santa Clara Valley Trans Auth Nevada DOT 39 June 9 to 13, 2014 Milwaukee, Wisc. Wisconsin (host) Minnesota Illinois Indiana Michigan Iowa Kansas 27 The California ROF is mentioned and included in this report for several reasons. First, since it was the first ROF to be completely funded and managed by a state rather than the SHRP 2 L36 project, the funding model was of interest to the sustainability section of this report, which

23 is located under Task 7. Second, it provided the research team experience in working with a state to deliver an ROF and spurred the development of several documents that would assist the state in organizing and delivering the ROF. These documents, available in Appendix D, can be used for future implementation of the ROF program at a state level. Each ROF was staffed during the entire ROF by a facilitator and a logistics coordinator. As anticipated by the research team, the facilitator was a key to success of the ROFs. That person was responsible for keeping the flow of the ROF moving, generating lively discussion and interaction among participants and with instructors, tying together all the ROF sessions, providing a neutral perspective, and answering questions on a variety of ROF topics. The logistics coordinator was equally important, only in different ways. The logistics coordinator made sure that: the meeting room had all required equipment in working order, transportation to the technical tour was confirmed, breaks and meals were prepared and ready as scheduled, any supplies needed for the group exercise were available for participants, participants signed the daily sign in sheet to confirm their daily attendance, CEU certificates were correct and ready to be distributed on the last day of the ROF, and that a group photograph was taken. B. Follow-Up Activities Follow-up activities are an integral part of the ROF program in order to accommodate emerging issues, evolving state of the practice, and peer-to-peer interchange. Below is a list of follow-up activities that have been (or will be) conducted following the initial ROF pilots:  List server (done)—All ROF pilot participants have been added to the existing Operations Academy™ list server. Initially, each ROF had its own list server but there was not any interaction of the groups. Therefore, the research team added them to the Operations Academy™ list server since it is one of the most active and engaging list servers the research team has seen. ROF attendees have been actively participating in the information exchange.  Group on Facebook or LinkedIn (done)—Two of the groups expressed interest in developing a private group on Facebook or LinkedIn. One of the participants from the ROF in Concord, New Hampshire, (April 28 to May 2) has already set up a LinkedIn page in which the research team was invited to participate in as well.  Follow up on Agency Implementation Plans—ROF pilot participants developed an agency implementation plan of action strategies that they presented to the other participants on the last day of each ROF pilot. The research team followed up with the participants from all of the ROF pilots (including California) and asked if they had been able to implement their plans. Responses to this question were almost evenly split with 47.95% of attendees saying “yes” and 52.05% saying “no.” A summary of all survey results is provided in Appendix E.

24  1-Day Follow-up Meeting to Initial ROF Pilot Delivery—Several ROF locations have requested that the same group of participants get back together on an annual basis for information exchange. With the five pilot ROFs completed, the research team determined that there was enough funding remaining in the meeting expenses budget to fund one, 1-day follow-up meeting. Since the first ROF pilot was delivered in Nashville, Tennessee, that was chosen to be the location of the follow-up meeting, scheduled for Tuesday, September 30, 2014. Since networking has been one of the benefits that all ROF pilots have found most valuable, a networking reception was also planned for Monday night. A hotel was secured for a Monday evening reception and a 1-day meeting on Tuesday. The agenda was still being developed as this report was being prepared, but the research team plans to hold a discussion on the successes and/or challenges of implementing the agency action strategies that they developed during the initial ROF in the morning and some additional knowledge transfer activities for the afternoon. Future follow-up activities will need to be addressed through the Operations Center of Excellence or the ROF Implementation Program being planned by FHWA. C. Lessons Learned For the ROF pilots to succeed, it was important for the research team to continually evaluate the ROF pilots and make changes as needed to continually improve what was delivered. The evaluation was done officially through the research team’s independent evaluator (see the evaluation information in Section D below) and was also done by the other members of the research team (instructors, facilitators, logistics coordinator, Principal Investigator) as part of the process of delivering the ROF pilots. Following is list of lessons learned that the research team compiled throughout the life of the project that are important to document for future offerings. These lessons learned were also shared with the TETG and participants of the Implementation Planning Workshop (discussed in Task 7 below).  Sharing of experiences across states must be fostered and managed. The networking aspect of the ROFs was listed as the greatest benefit at every ROF pilot. The curriculum sessions became more and more discussion oriented, and it is important to manage the discussion so there is a balance between providing the participants with all the curriculum materials and allowing the sharing of information among participants.  ROF curriculum material needs to be constantly modified and tailored and adequate resources need to be budgeted for this item. It was important to adjust the curriculum to incorporate SHRP 2 research outcomes and new TSM&O research and resources as they became available as well as to adjust the sessions based on participant feedback from previous ROF pilots. However, it is also important to modify the curriculum based on the specific audience that is participating in the ROF. For example, some regions may be

25 more focused on weather related events while others may be more urban and might want to focus more on corridor management.  Using local representatives to present slides can work well, but it is important that the individuals are knowledgeable on a national level as well as a local level. Local speakers can be more difficult to manage so it is extremely important to coordinate expectations and timing.  The ability and experience of instructors and facilitators is important. They should be knowledgeable on the topic at a national level as well as a regional/local level. They should also be effective presenters and be able to make a good connection with the participants and engage them in discussion.  Recorded speaker sessions can work, but it is extremely important to have a dynamic speaker presenting the material. It is also best if the instructor is available afterwards via phone or Skype to answer questions. Coordination is a challenge, however, as well as the connection required for Skype. A Q&A session could be accomplished through the facilitator if he or she is knowledgeable on the topic. The recorded sessions need to be monitored in case the content needs to be updated.  The technical tour portion of the ROF is essential. The tour is an excellent way to break up the week, and it is a great way for the participating agencies to share information with each other.  ROFs should not be held in a DOT facility. It is much too distracting for local participants if the ROF is held at their place of work. The participants can easily get called into a meeting or other activities. ROFs should be scheduled at a neutral off-site facility, preferably at the same location as the lodging to minimize travel time and transportation costs.  Evaluation is important and necessary, but the participants should not be surveyed more than twice a day or they will get survey fatigue and the results will not be nearly as effective.  Not everyone will complete the prestudy materials, so make sure that the amount is limited.  The agenda is extremely tight and had to be constantly modified to accommodate for discussion between the states and breaks. There is a delicate balance between getting the entire curriculum delivered and providing enough time for discussion, group exercises, and breaks. D. Evaluation Results The following is an evaluation summary from the five ROF pilot locations, plus California. An independent contractor oversaw the evaluation and analysis of data from the five ROF pilots funded by the project. For the pilot conducted in California, the contractor was forwarded a summary of the evaluation results and that summary data has been included in this report. This

26 summary highlights main themes and common findings for both multiple-choice questions and open-ended responses received from the pilot participants. Overall, feedback across all of the ROFs and each individual session was very positive. The most frequently stated positive comment from the ROFs was the peer-to-peer interaction and connections that were made. Respondents stated that they learned a lot from these informal interactions with other agencies. In addition, participants felt strongly that the ROFs were valuable and applicable to their jobs (especially after returning to their jobs). Additionally, the total achieved percentage was calculated for each session, and it is clear from the high percentages received that the overall ROFs were a positive experience for participants. The main criticism was regarding the length of the ROF. These findings indicate a very successful pilot program that offered an immense amount of value to participants, including ways of implementing what they learned as well as general inspiration to improve their organizations. A short overview evaluation summary of all the evaluation results obtained during each ROF of the six ROFs is included in Appendix E, along with detailed results of the follow-up survey that was sent out after the ROFs were completed. Overall Evaluation Themes Applicability. A main question of the participants was whether they could apply what was learned once they returned to their jobs. During immediate feedback given while participating in the ROFs, attendees rated the two applicability items lower overall than most other items. Attendees had trouble thinking of ways to incorporate the ROF knowledge into everyday practice, and this may have been due to feeling their positions lacked the influence for broad organizational changes. In spite of lower applicability ratings, these responses indicated that attendees were inspired to return to their agencies and share the information received. Interestingly, when the follow-up ROF survey was administered, applicability ratings were much stronger—attendees were able to find ways to turn that inspiration into action. An overwhelming majority (over 80%) agreed or strongly agreed with the ROF being directly applicable to their jobs when asked similar questions during the follow-up survey. Figures 3.2 and 3.3 show the data from the follow-up survey questions regarding applicability (see Appendix E for more details). Figure 3.2 shows the distribution of responses on the follow-up survey item, “The training and instruction I received from the ROF has been directly applicable to my job.” Figure 3.3 shows percentage responses to the question, “Are you applying the concepts you learned in your job?”

27 Figure 3.2. Distribution of Follow-Up Responses Figure 3.3. Percentage Responses The follow-up responses indicated that despite low ratings of applicability at the time of the ROF, many attendees found the material to be applicable once they returned to their jobs. It

28 should be noted that the follow-up survey was filled out by participants from all of the ROFs at the same time; therefore, varying amounts of time, from a few weeks to several months, had passed since the ROF session and the response to the survey. It is possible the ratings would increase as later ROF participants have more time to implement action strategies they developed. Given the change in ratings of applicability from time of ROF to follow up, it may be that once attendees get back to their day-to-day jobs, they are able to find areas where the information learned during the ROF could be useful and applicable. In fact, 83.56% of attendees stated that they applied the concepts from the ROFs to their jobs. In the open-ended responses, some attendees gave specific examples of how they applied what they learned:  “It has changed the way that I think about how we impact traffic and has caused me to think about ways to lessen that impact. I have asked my employees to be thinking about the manner in which we impact our customers and how we can do better.”  “Yes—we had a FHWA self-assessment workshop here and the background we had at the ROF helped make me an effective advocate for operations.”  “I will be managing a multistate corridor operations effort. The concepts I learned will help provide a link between planning and operations and identify efforts that we can do across state lines.”  “It has been helpful to start creating new guidelines for integrating operations better into planning, reviewing a new ICM location with understanding of what ICM is and what it requires to be effective and applying the concepts to creation of new reliability PMs.” Value of ROF. In the follow-up survey, attendees were asked two questions related to the effectiveness and value of their experience at the ROF. 88% of attendees replied that they either agreed or strongly agreed with the statement that the ROF was effective. Additionally, nearly all attendees (97%) stated that their experiences in the ROF were valuable. See Appendix E for more details. Below are some highlighted comments:  “I was thinking about moving away from operations in my career and the ROF re- energized me.”  “Great presentations and valuable audience participation.”  “Networking and learning from peers (and in my case, higher-level managers than myself) was the most valuable part of the whole experience; and for the majority of the other aspects of the ROF (lectures, group exercises, etc.), most proved valuable to me in my job.”  “Yes, my eyes were really opened in terms of what other states are doing and how they are doing things. I have learned a different perspective of looking at issues like Operations. I certainly learned a lot from others in my class. That was very good.”

29 Implementation of Action Strategies. During the follow-up surveys, attendees were asked, “Have you been able to address/implement any of the action strategies that you and the other participants from your agency developed?” 47.94% of attendees responded yes to this question (see Appendix E). As noted above, many respondents were relatively recent ROF graduates, so they may not have had time to implement the action strategies yet (and some noted this in comments). Additionally, many provided specific examples of how they implemented the action strategies:  “TIM training throughout the corridors. Coalition building along the corridors.”  “Yes, our primary uses and insights from others have been useful in how we want our ATMS and organization to run. Our largest use of devices is now the Systems Engineering process.”  “We are now pursuing statewide CMM process with FHWA. Our participation in the ROF allowed us to be part of the larger ‘official’ process and our strategies are being rolled up into it.” Peer Interaction. Attendees stated that the most beneficial part of the ROF was peer-to-peer interaction. Many attendees stated that learning what other states are doing was helpful. Both informal discussions and the group activities were received well by the attendees. Positive Experience. The overall evaluation during the ROF received an average percentage of 76.58% across the ROF sessions (see Appendix E). This high percentage reflects the high ratings attendees gave on feedback surveys. Additionally, open-ended comments left by attendees further highlight the positive experience many had at the ROF sessions. Total Achieved Percentages for Session For each ROF, the total achieved percentage was calculated for each session by calculating the total possible points and dividing it by the total achieved points, with a maximum possible total achieved percentage of 100% if every participant gave the highest rating on all survey items. These total achieved percentages were then averaged across all of the ROFs. Table 3.5 displays the mean total achieved percentages by session ranked from highest average rating to lowest average rating. As the ROFs progressed, some sessions were eliminated, changed, and/or added to the program. Therefore, not all of the scores are based on an average across all ROFs as reflected by the Number of Sessions column.

30 Table 3.5. Total Achieved Percentage Across ROF Sessions Name of Session Number of Sessions Total Achieved Percentage Mainstreaming TSM&O in a State Program: A CEO Perspective 1 95.95% Connected Vehicles and the Future of Transportation Speaker 6 85.78% Performance Measurement 6 85.75% Managing a Corridor (includes ICM, ATM, Managed Lanes) 6 85.29% Facilitating Good Movement Through Operations Speaker 5 85.21% Traffic Incident Management, Emergency Operations, and Planned Events 6 84.98% Traveler Information and Operations 6 84.06% Communicating the Value of Operations 6 83.62% Work Zones 4 83.27% Road Weather 4 82.85% Planning and Programming for Operations 6 81.43% How to Organize for Operations and Workforce Development 4 81.09% How to Organize for Operations (was combined with Workforce Development) 2 80.52% Systems Engineering 6 80.51% Review of Capability Maturity Model Self Evaluation 6 80.34% Workforce Development (was combined with How to Organize for Operations) 2 79.10% Transit Speakers 2 77.07% Overview of Operations * 2 76.58% Safety and Operations Speaker 2 72.11% *This session was a review in the first two ROFs since there was a session on the topic in the prestudy. Participants felt there was no reason to cover it again since it was thoroughly covered during the prestudy. Overall, sessions received at least 72% in overall rankings, and over three quarters of the 19 sessions (76.2%) received a rating of greater than 80%. Based on these high rankings, it is clear that attendees had a very positive perception of sessions overall. This was further highlighted by many of the open-ended comments. For example, comments included “great session and information,” “very informative,” and “great week.” Additionally, across all six ROF locations, only a handful of people answered “no” when asked if they would recommend the program to a peer. Adapting ROFs based on early feedback The six SHRP 2 ROFs took place over the course of a ten-month period. The evaluation design was intended to be both iterative and formative, while also providing timely feedback that could be used between each ROF training to improve the process, flow, and content of the training and

31 enhance the training approach based on participant feedback. Overall summaries were developed after each ROF that included both quantitative and qualitative feedback and resulted in a case study for each ROF. Qualitative feedback was collected during participant debriefings at the end of each training session, with only the participants and an evaluation team member in the room to enhance the confidentiality of the information provided. Participants were asked to focus on what worked, what did not work, and what would they change about the training. The feedback from these sessions was combined with qualitative feedback provided in course and overall ROF evaluation forms and was used to supplement quantitative summaries of participant feedback. Within 2 weeks of each ROF course, a summary evaluation report was provided to the training team with specific feedback regarding the conduct and content of the ROF session. This allowed the training team to make iterative adjustments to the training program to address specific concerns raised by participants. Adjustments were then done at both the individual course content and presentation level, as well as at the overall ROF organizational level. This emphasis on early discovery of potential weaknesses that could be corrected as the program continues was critical to an approach intended to provide both formative and summative evaluative information. For example, some evidence of the use of such feedback was the combination of two sessions offered at an earlier ROF into one session for the later ROFs. The total achieved percentage was higher for the combined session than for either of the earlier individual sessions, thus demonstrating the benefit of using feedback from early ROFs to improve later ROF pilots. Task 7— Final Report The final report contains all the information that was required except for all the curriculum materials. All the session PowerPoint slides, prestudy materials, and links to recorded session speakers will be delivered via DVD or flash drive to TRB, and they are all available for download at http://www.tsmoinfo.org/resources/regional_operations_forums.aspx. Implementation Planning Workshop on ROF Sustainability One of the major items to be included in the final report was information on the sustainability of the ROFs beyond the SHRP 2 program support. FHWA has set aside a one million dollar budget to sustain the ROF program in some form and a 2-day Implementation Planning Workshop (IPW) was held in May 2014 to discuss the topic. The research team prepared several PowerPoint presentations and a report on sustainability options that were presented and discussed during the premeeting webinar held on May 13, 2014, and the 2-day workshop held on May 20–21, 2014. The full sustainability options report that was developed by the research team can be found in Appendix F. The objectives of the 2-day IPW were to assess the ROF product that was delivered by the research team, develop goals for potential continued implementation, explore additional opportunities, and develop tactics to facilitate its acceptance and use in the transportation community. The research team participated in the IPW discussions along with participants

32 representing state transportation agencies, metropolitan planning organizations, state highway patrol, the FHWA, AASHTO, and TRB. The group’s first activity was to define a vision of success for the continued ROF product. Together, the group crafted the following: An ongoing Regional Operations Forum program is established with a sustainable funding strategy to support it and participants are proactively engaged in a series of activities to promote regional peer groups and a broadened community of practice nationally. The group then established the following desired outcomes:  Assist with mainstreaming TSM&O by moving the state of the art closer to the state of the practice;  Develop a community of practice, among states and other agencies, by developing a peer network and training the next generation of operations leadership; and  Transfer TSM&O knowledge, including disseminating SHRP 2 solutions and developing a business case for TSM&O programs and activities. The IPW participants noted that it would be desirable to put the need and role for ROFs in the context of a vision for a comprehensive TSM&O curriculum. A comprehensive curriculum would take advantage of other related training programs and leverage other available TSM&O technical resources. As part of a broader TSM&O training program, the ROFs could be an “on-boarding” activity to get individuals familiar with many aspects of TSM&O and to get them involved with the TSM&O community. A defined sequence of subsequent training and peer activities would then continue to advance participants’ skill development. Planning is currently underway to incorporate the ROF into a mass comprehensive TSM&O curriculum to be offered by the new Operations Center of Excellence. The IPW participants then recognized that a sustainable strategy needed to address several key barriers and challenges to the pilot ROF format. These challenges include:  Cost of a 4.5-day in-person event;  Restrictions on out-of-state travel;  Amount of time away from jobs;  Potentially large demand for ROFs given the need for TSM&O training;  Need to create funding mechanisms once SHRP 2 implementation funding ends; and  A more systematic and comprehensive peer group program needs to be established during the ROF implementation phase.

33 To achieve the desired outcomes for ROF implementation, the participants established three key objectives:  Select host organization and conduct additional ROFs;  Refine training options and ROF curriculum; and  Establish and maintain participant peer groups and activities. To address the barriers identified, the sustainable ROF program would need to offer a suite of training options in addition to the pilot ROF program. These options would vary in terms of the amount of in-person time, the delivery mechanism used for different training modules and topics, the cost, and the degree of flexibility for a host agency to help design a specific program. In addition, the content of each training module would need to be updated periodically and more than one team of trainers would need to be available to deliver the training. Finally, a proactive series of peer group activities would need to be established for ROF participants so that the momentum gained during the training is not lost. IPW participants also identified two primary marketing and communications goals during the session:  Develop outreach materials and tools to take to decision makers to ensure that the training is supported (staff time, cost) and available; and  Ensure that those who want to take the training know about it and can access it. The participants identified four target audiences (decision makers, implementers or users, advocates or influencers, and other groups), benefits to these target audiences resulting from use of the ROF product, and suggested key messages based on the benefits identified. A number of marketing tactics were suggested for consideration as part of ROF product implementation. IPW participants recommended that performance measures established for the ROF product be simple and focus on tangible results from specific activities. The measures suggested included:  Degree of interest and participation in ROFs;  Number of new people (not previously engaged in TSM&O activities) attending ROFs;  Amount of involvement in peer networks and activities;  Follow up on agency implementation plans; and  Success stories from participants related to their participation in a ROF or follow-up peer activities. The SHRP 2 Implementation Plan budget available for the continuation of the ROF program is $1,000,000. At the IPW Workshop, participants were asked to recommend budget

34 allocations to six different activities reflecting the effort required and/or the relative priority of each. At the IPW, the need to develop additional training resources was not broken out as a separate budget item, so the allocation to this item is an estimate of the resources that might be required. Table 3.6 summarizes the results. Table 3.6. Recommended Budget Allocation for ROF Continuation Funding Budget Category Percent Allocations 1. Update Training Material 5 2. Establish Suite of ROF Options 10 3. Develop Additional Training Resources 5 4. Conduct Additional ROFs 50 5. Support Peer Groups 20 6. Develop Marketing and Communication 10 Total 100 The IPW participants did not discuss a specific implementation schedule. However, Table 3.7 suggests a schedule for implementation activities reflecting the completion of the ROF project in September 2014 and the proposed launch of the Operations Center of Excellence in January 2015. Table 3.7. Potential Implementation Schedule Task Lead Start (month/year) Complete (month/year) Draft ROF Continuation Implementation Plan AASHTO/FHWA 9/14 10/14 Finalize Implementation Plan 10/14 10/14 Update Existing Material Host organization/ Trainer Ongoing Establish Suite of ROF Options AASHTO/FHWA 11/14 12/14 Develop Additional Training Resources AASHTO/FHWA 11/14 12/14 Conduct Additional ROF Host organization 11/14 12/15 Support Peer Groups 11/14 12/15 Develop Marketing and Communication 10/14 6/15 The IPW Summary Report contains a more detailed description of the discussion and results of the workshop.

35 CHAPTER 4 Conclusions and Suggested Research Conclusions The overall goal of this research project was to advance transportation system management and operations (TSM&O) and serve as a platform for mainstreaming SHRP 2 Reliability research results through the development of a regional operations forum (ROF) program. The curriculum offered through the ROFs was to provide pertinent education and training on TSM&O business processes; organizational capabilities; operations and planning; and design, technical, and analytical issues to a broad range of transportation agency representatives. The objectives for the project were:  To design a curriculum for the ROFs based on a comprehensive review of available and anticipated research results of the SHRP 2 Reliability program and other information available from TSM&O resources, including federal, state, and local initiatives;  To pilot test the curriculum developed, including o Continually adjusting the curriculum to incorporate SHRP 2 research outcomes as they became available and o Conducting an ongoing evaluation of the effectiveness of the curriculum and its delivery;  To develop a sustainable business model for the ROFs The project has met both its goals and objectives. Specifically, the following has been developed and activities have occurred as a result of this research project.  Curriculum was developed and delivered for a 4-day in-person ROF. Some of the curriculum was included as prestudy materials and recorded speaker sessions that were played on DVDs during the ROF; the remaining sessions were delivered in person.  Five pilot ROFs pilots were scheduled, coordinated, and delivered in:  Nashville, Tennessee  Seattle, Washington  Phoenix, Arizona  Concord, New Hampshire  Milwaukee, Wisconsin  An additional ROF was delivered in Orange, California, by the research team and funded by Caltrans.  A 1-day follow-up was scheduled, coordinated, and delivered in Nashville, Tennessee.  A report on the future sustainability of the ROF program was developed and presented at the May 2014 Implementation Planning Workshop.

36  A section dedicated to the ROFs was created on the KTS website at http://www.tsmoinfo.org/resources/regional_operations_forums.aspx. The website was developed to allow the page to be easily edited without any additional funding required. Based on the results of the evaluation done by the research team and the discussions during the IPW in May 2014, the ROF pilots have been a success, and discussions are now focused on how to continue delivery of additional ROFs to other locations, as well to deliver follow-ups in regions that have already participated in one of the ROF pilots. Suggested Research Presented here are several observations of the research team pertaining to issues that we believe should be explored and resolved as the implementation of the ROF program proceeds:  Assessing the Value of Face-to-Face Communications: The research team believes that face-to-face communications, as provided by the format used in the five initial ROF pilot sessions, is the most effective way for the participants to learn, especially from each other. However, the research team also recognizes that factors such as state travel restrictions and time away from the office serve to make a full week of face-to-face learning difficult or impossible. Nevertheless, the research team suggests that an effort be made to assess how much effectiveness is lost with initial ROF options involving less face-to-face time, particularly as it relates to these considerations: o How much learning effectiveness is lost as the amount of face-to-face time decreases or disappears? o What value is placed on the potential loss of learning effectiveness relative to the cost and “less time away from the office” savings inherent in options involving less or no face-to-face time? o How much does less face-to-face time hinder achieving the sustainable community-of-practice objective? Is there a significant difference in the amount of follow-up interactions, including use of the list server and participation in follow-up forums, in cases where the initial forum involved less face-to-face communications?  Keeping Material Current: As noted earlier, it is important to provide resources to continually tailor and update the curriculum material and presentations. Material and presentations should be tailored to the needs of specific audiences, and material should be continually updated to reflect the latest research results and innovative deployment practices. This issue needs to be considered in combination with the next ROF offerings in that different funding and delivery methods will have different ramifications for how this is done. For example, more decentralized methods will benefit from strength in diversity of participation but may lead to divergence and unevenness in the material (or curriculum) that is being presented.

37  Long-Term Sustainability: The IPW Summary Report discusses a number of options for how future ROFs (both initial and follow on) could be paid for and delivered and recommends experimentation with these options using the FHWA implementation budget. Long-term sustainability depends on perceived value and that perception may be different for initial forums than for follow on forums. Perceived value should be assessed in both contexts. Tradeoffs associated with delivery of ROFs centrally (through an organization such as AASHTO or FHWA) or de-centrally (for example, through a network of universities) should also be assessed relative to issues such as management efficiency, diversity of participation, relevance to participants, and quality control.  Peer Activities: The IPW Summary Report presents a number of possibilities for peer activities to support sustainment of a TSM&O community of practice that will be built through the Initial ROFs. The research team believes that these will be critical functions of the emerging Operations Center of Excellence and that peer activities associated with pilot ROF participants will provide a base from which the Center of Excellence can expand offerings and broaden participation. Reference Kirkpatrick, D. L., and J. D. Kirkpatrick. 2006. Evaluating Training Programs: The Four Levels. 3rd ed. Berret-Koehler Publishers, San Francisco.

A-1 APPENDIX A Final Matrix of ROF Sessions  Recommended Core Subjects  Optional Subjects

A-2 FINAL MATRIX of ROF AGENDA SESSIONS Session Title Issues to be Addressed Task 1 Subjects/Topics Related SHRP2 Products/Results Related Literature in KTS Database Delivery Mechanism of Session Content Developer & Presenter RECOMMENDED CORE SUBJECTS Overview of Operations  Describing Management and Operations  Terminology and jargon  Mobility as it relates to the agency and system user  M&O Strategies that have an impact on mobility  Making the business case  Benefits of M&O Introduction to overall topics to be covered. L03: Analytic Procedures for Determining the Impacts of Reliability Mitigation Strategies L11: Evaluating Alternative Operations Strategies to Improve Travel Time Reliability L17: A Framework for Improving Travel Time Reliability C05 - Understanding the Contributions of Operations, Technology, and Design to Meeting Highway Capacity Needs Optimizing the System: Saving Time, Saving Lives (AASHTO) Online material as pre-study Short presentation on day one to summarize/discu ss importance of operations Gary Euler, PB Facilitator will deliver a brief version on day 1. Basically an overview of importance.

A-3 Planning and Programmi ng for Operations Main Issues to be Addressed:  Integrating operations into the metropolitan and statewide transportation plan.  Using an objectives- driven, performance- based approach to plan for operations  Role of the congestion management process (CMP)  Existence of formal Operations Plan or Regional Concept for Transportation Operations (RCTO)  Process and methods for preparing/updating the Operations Plan and participation in the process  Funding sources for operations and programming of operations projects and programs metropolitan/state transportation plans/Operations Plan into State/Regional/Local Transportation Programs  Assessing benefits and Managerial - Business Processes:  Planning  Programming / budget L01: Integrating Business Processes to Improve Reliability L05 (principle one) L06: Institutional Architectures to Advance Operational Strategies L17 GFP #2: Deployment Guidance for TSM&O Strategies C01 - A Framework for Collaborative Decision Making on Additions to Highway Capacity Transportation Management Center Business Planning and Plans Handbook (FHWA) A Primer on Performance- Based Highway Program Management – Examples from Selected States (AASHTO) Best Practices of Rural and Statewide ITS Strategic Planning (FHWA) Congestion Management Process: A Guidebook (FHWA) Statewide Opportunities For Linking Planning and Operations (FHWA) Chart Non- Pre-forum self- assessment using L06 framework Short standardized presentation to frame discussion issues Facilitated discussion among participants aimed at identifying issues and actions Mike Smith/Jocelyn Bauer, SAIC

A-4 costs of operations projects in support of regional or statewide operations goals and performance objectives Constrained Deployment Plan (MDOT) VDOT Nova Smart Travel Program Plan: Executive Summary VDOT Statewide Systems Operations Program Performanc e Measureme nt  Why do performance measurement?  Use of performance measures for performance management  Use of PMs in planning, programming, and procedures improvement  Selecting Operations PMs  Data collection, sharing, processing and reporting  Using PMs in decision making & communications Managerial – Business Processes and Performance Measurement  Measures definition  Data collection  Analytics  Utilization Technical Issues - Data and Analytic Tools L02: Establishing Monitoring Programs for Mobility and Travel Time Reliability L03 L04: Incorporating Reliability Performance Measures in Planning and Operations Modeling Tools L05: Incorporating Reliability Performance Measures into the Transportation Planning and Programming Measuring Performance among State DOTs: Sharing Good Practices – Congestion (NCHRP 20- 24(37)I) Traffic Analysis Toolbox Volume I: Traffic Analysis Tools Primer (FHWA) Cost-Effective Performance Measures for Travel Time Delay, Variation and Reliability (NCHRP 618) Standardized presentation for all ROFs Facilitated discussion among participants aimed at identifying the key features of a performance measurement system:  What to collect  Where to collect it  How to process the information  How the information will be used Tim Lomax, TTI

A-5 Processes L07 L08 L35: Local Methods for Modeling, Economic Evaluation, Justification, and Use of the Value of Travel Time Reliability in Transportation Decision Making L17 GFP #4: Standard Reporting and Evaluation Procedures for TSM&O Strategies L17 GFP #5: Guidebook for Placing a Value on Travel Time Reliability C02 - Performance Measurement Framework for Highway Capacity Decision Making Guide to Effective Freeway Performance Measurement: Final Report and Guidebook (NCHRP WOD 97) NTOC Performance Measurement Initiative 2011 Urban Mobility Report (TTI)  Where should the information be stored Promoting the Value of Operations within an Main Issues to be Addressed:  Fostering support of organizational Institutional - Cultural Issues:  Leadershi p L06: Institutional Architectures to Advance Operational Uses of Social Media in Public Transportation (TCRP Synthesis  Pre-forum self- assessment using L06 Pierre Pretorus/Lisa Burgess, Kimley-Horn

A-6 Agency and Among Regional Decision- makers leadership  Focus on and accountability for performance (performance management)  Building understanding throughout the organization and its stakeholders  Strengthening relationships with other organizations  Taking best advantage of private resources  Cost/Benefit  Policy/de cision maker support  Customer service  Program hierarchy  Program formality Institutional - Collaboration:  State/MP O/local  DOT/Pub lic Safety Agencies  Public- private Strategies L17 GFP #1: TSM&O Briefing Book L17 GFP #6: TSM&O Business Case Primer L31 99) 3rd Annual State DOT Social Media Survey (AASHTO) Developing Transportation Agency Leaders (NCHRP Synthesis 349) Managing Change in State DOTS – Scan 4 of 8: Innovations in DOT Communications , Image, and Positioning (NCHRP Web Document 39) AASHTO SO&M Guidance: Culture Guidance ITS Benefits, Costs, Deployment, and Lessons Learned Desk Reference (FHWA) Regional framework  Short standardized presentation to frame discussion issues  Facilitated discussion among participants aimed at identifying issues and actions Potential Presenter: Doug McDonald, Washington State

A-7 Transportation Operations Collaboration and Coordination: A Primer… (FHWA) Transportation Systems Management & Operations Guidance (AASHTO)

A-8 How to Organize for Operations Main Issues to be Addressed:  Organizational changes and reporting relationships  Stature of Operations within the agency or region  Core capabilities and suitability of job descriptions and career paths  Adequacy of education and training, recruitment and retention opportunities  Role of outsourcing  Identifying key issue, points of interaction and support Institutional – Organization/Wo rkforce:  Organizationa l structures  Reporting relationships  Staff training/certif ication  Career paths  Recruitment/ retention  Blended contractor and agency staff L06: Institutional Architectures to Advance Operational Strategies C01 - A Framework for Collaborative Decision Making on Additions to Highway Capacity A Transportation Executive's Guide to Organizational Improvement (NCHRP 20- 24(42)) AASHTO SO&M Guidance: Organization & Workforce Guidance Attracting, Recruiting, and Retaining Skilled Staff for Transportation System Operations and Management (NCHRP 693) Transportation Systems Management & Operations Guidance (AASHTO) Pre-forum self- assessment using L06 framework Short presentation to frame discussion issues Facilitated discussion among participants aimed at identifying issues and actions Steve Lockwood, PB

A-9 Session Title Issues to be Addressed Task 1 Subjects/Topics Related SHRP2 Reliability Products/Results Related Literature in KTS Database Delivery Mechanism of Session Content Developer & Presenter OPTIONAL SUBJECTS Systems Engineering  The systems engineering process  Concept of Operations and architecture  Functional, performance and technical requirements  Standards, specifications and IT issues  System development  System acceptance testing and documentation Technical Issues – Systems and Technology Metropolitan Transportation Management Center Concepts of Operation (FHWA) Combination of:  Presentations  Class exercises – e.g., drill to prepare functional or performance requirements  Discussion – what’s worked, what hasn’t Available as an on-line course if they want to do it as pre-study Mike Smith, SAIC In general, the following presentations would cover:  A description of the strategy (problem addressed and operational concept)  Procedures & protocols and supporting ITS devices  Issues and implementation considerations  Good practice examples  Benefit and cost information Coordinate d Incident Manageme nt  Importance of TIM and Quick Clearance  What makes a good TIM/QC program  Roles and alignment of partners  Good practice examples Technical Issues – Unplanned, Severe Weather and Planned Events Institutional – Collaboration L12/L32A&B Training for Traffic Incident Responders Sharing Information between Public Safety and Transportation Agencies for Traffic Incident Management Would include  A standardized presentation  Facilitated discussion aimed at Capt. Tom Martin

A-10  National Unified Goal  Incorporating TIM practitioners into transportation planning and project design decisions (NCHRP 520) Information Sharing Guidebook for Transportation Management Centers, Emergency Operations Centers, and Fusion Centers (FHWA) Best Practices in Traffic Incident Management (FHWA) Field Operations Guide for Safety/Service Patrols (FHWA) Traffic Incident Management Handbook (FHWA) identifying actions to take to improve current practices and interagency cooperation Integrated Corridor Manageme nt (including arterial operations)  Overview of ICM  Rationale for ICM  USDOT ICM Initiative o ConOps o ICM AMS  Demo Site ICM Approaches (systems & facilities involved, system architectures, Technical Issues – Systems and Technology L02: Establishing Monitoring Programs for Mobility and Travel Time Reliability L03: Analytic Procedures for Determining the Sharing Information between Public Safety and Transportation Agencies for Traffic Incident Management (NCHRP 520) Would include  a standard presentation  a local presentation  discussion on how to implement this topic at a Pierre Pretorius/Lisa Burgess, Kimley-Horn

A-11 decision support approach, evaluation, & deployment status) Impacts of Reliability Mitigation Strategies L11: Evaluating Alternative Operations Strategies to Improve Travel Time Reliability Public Roads Magazine, Volume 71, Number 5 - Integrated Corridor Management (FHWA) regional/local level Active Traffic Manageme nt  ATM Overview  ATM Strategies o Speed Harmonizatio n o Temporary Shoulder Use o Queue Warning o Dynamic Merge Control o Construction Site Management o Truck Restrictions o Dynamic Rerouting and Traveler Info o Dynamic Lane Markings o Automated Enforcement  ATM in the US  ATM in Europe Technical Issues – Active Traffic Management L02: Establishing Monitoring Programs for Mobility and Travel Time Reliability L03: Analytic Procedures for Determining the Impacts of Reliability Mitigation Strategies L11: Evaluating Alternative Operations Strategies to Improve Travel Time Reliability Active Traffic Management: The Next Step in Congestion Management (FHWA) Sharing Information between Public Safety and Transportation Agencies for Traffic Incident Management (NCHRP 520) Would include  a standard presentation  a local presentation  discussion on how to implement this topic at a regional/local level Les Jacobson, PB Integrating Operations  Includes wild fires, hurricanes, floods, Technical Issues – Planned and NCHRP 20-59(42) Transportation Planning for Severe Weather Would include  a standard Chris Poe, Tony Voigt

A-12 into Emergency Operations rock slides, snow, earthquake, catastrophic infrastructure issue  Discussion on impact on operations  How to go from routine to major response  Coordinating with other states Unplanned Events Planning for Disasters, Emergencies and Significant Events Events in Public Transit Operations (UI Chicago) The Role of Transit in Emergency Evacuation (TRB Special Report 294) Information Sharing Guidebook for Transportation Management Centers, Emergency Operations Centers, and Fusion Centers (FHWA) presentation  a local presentation  discussion on how to implement this topic at a regional/local level and Darrell Borchardt from TTI Road Weather Manageme nt  Weather events  Weather impacts on mobility  RWIS  Clarus  Weather Responsive Traffic Management (WRTM) Strategies and Tools  Benefits of WRTM strategies  Decision Support Systems  Performance Technical Issues – Planned and Unplanned Events L02: Establishing Monitoring Programs for Mobility and Travel Time Reliability L03: Analytic Procedures for Determining the Impacts of Reliability Mitigation Strategies L11: Evaluating Road Weather Management (FHWA) Seasons of Achievement – Accomplishments of the Road Weather Management Program (FHWA) Planning for Severe Weather Would include  a standard presentation  a local presentation  discussion on how to implement this topic at a regional/local level Developer: Leon Osborne, University of North Dakota Presenter: Deepak Gopalakrishan a, Battelle

A-13 Monitoring  Regional/multi- jurisdictional road weather management  Stakeholder/Leveragi ng Resources –S haring weather information between stakeholder groups  Best Practices – US  Best Practices - International Alternative Operations Strategies to Improve Travel Time Reliability Events in Public Transit Operations (UI Chicago) Impact of Freight Goods Movement on Operations  What’s unique about freight  Importance of considering commercial vehicles when developing operational improvements  Impact on safety and congestion  How to work with the trucking industry Technical Issues – Goods Movement L02: Establishing Monitoring Programs for Mobility and Travel Time Reliability L03: Analytic Procedures for Determining the Impacts of Reliability Mitigation Strategies L11: Evaluating Alternative Operations Strategies to Improve Travel Time Reliability Performance Measures for Freight Transportation (NCFRP Report 10) 2011 Urban Mobility Report (TTI) 2011 Congested Corridors Report (TTI) Freight Transportation: National Policy and Strategies Can Help Improve Freight Mobility (GAO) Would include  a standard presentation  speaker – Rebecca Brewster, American Trucking Association  discussion Donald Ludlow (assistance from Monica Isbell and Rich Margiotta), Cambridge Systematics, Inc. Traveler Information  Overview  TI Delivery Technical Issues – Traveler L02: Establishing Monitoring Uses of Social Media in Public Would include  a standard Pierre Pretorius/Lisa

A-14 and Operations o Internet o IVR o PDAs o Media (TV/Radio) o Kiosks/Displa y Boards o Dynamic Message Signs o Telematics o Highway Advisory Radio  511 Systems  TI & Social Media  Public Private Partnerships  Data collection strategies for provision of good TI Information Programs for Mobility and Travel Time Reliability L03: Analytic Procedures for Determining the Impacts of Reliability Mitigation Strategies L11: Evaluating Alternative Operations Strategies to Improve Travel Time Reliability L14: Effectiveness of Different Approaches to Disseminating Traveler Information and Travel Time Reliability Transportation (TCRP Synthesis 99) Impacts of Technology Advancements on Transportation Management Center Operations (FHWA) Impacts of Technology Advancements on Transportation Management Center Operations (FHWA) Real-time Traveler Information Services Business Models: State of the Practice Review (FHWA) presentation  Speaker – TBD  Discussion Burgess, Kimley-Horn Customer Service  Definition of customer service  The intersection of operations and customer service  Principles of customer service (tools, techniques and Managerial Issues - Outreach/Marketi ng Impacts of Technology Advancements on Transportation Management Center Operations (FHWA) Would include  a standard presentation  Homework: review of video by Bob Marbourg, Traffic Lisa Burgess, Kimley-Horn

A-15 attitude – you vs. we)  Evaluation of existing materials  Media interaction  The customer service mentality Managing Change in State DOTS – Scan 4 of 8: Innovations in DOT Communications, Image, and Positioning (NCHRP Web Document 39) Reporter  Discussion on how to implement this topic at a regional/local level OPTIONAL SUBJECTS - SPEAKERS In general, the presentations would focus on:  The latest developments  Examples of successful deployments  Why they were successful Connected Vehicles/ Future of Transportat ion  Connected Vehicle Technology  Connected Vehicle Applications  Policy & Institutional Issues  Ongoing Research  Vehicle Technology & Trends  Looking Forward Technical Issues – Vehicle Technologies AASHTO Connected Vehicle Infrastructure Deployment Analysis (RITA) Impacts of Technology Advancements on Transportation Management Center Operations (FHWA) Would include  A standard presentation  Speaker  Discussion Shelley Row, Shelley Row Associates, LLC Multimodal Manageme nt  Transit – vital component of the transportation network  Bus/Rail operations  Technology – smart transit and informed Statewide Opportunities for Integrating Operations, Safety, and Multimodal Presentation plus a discussion of the management challenges experienced by transit operators John Collura, University of Massachusetts OR Michael Walk,

A-16 riders  Integration of efforts – working with highway agencies Planning: A Reference Manual (FHWA) in dealing with highway agencies Maryland Transit Administratio n Still working this one out. Managed lanes/pricin g  Road User Charging and Toll Collection  Technology Options for Charging  Technology Options for Enforcement  Vehicle Detection and Classification  Central System  Case Studies  Future Developments Technical Issues – Managed Lanes, Travel Demand Management L02: Establishing Monitoring Programs for Mobility and Travel Time Reliability L03: Analytic Procedures for Determining the Impacts of Reliability Mitigation Strategies L11: Evaluating Alternative Operations Strategies to Improve Travel Time Reliability C04 - Improving Our Understanding of How Highway Congestion and Pricing Affect Travel Demand Priced Managed Lane Guide (FHWA) Vehicle Mileage Fee Primer (TxDOT/TTI) Would include  A standard presentation  Speaker – TBD  Discussion Bart Cima and David Kamnitzer from IBI Group

A-17 Safety and Operations  Importance of safety  Balance between safety and throughput  Operational Factors that influence safety Statewide Opportunities for Integrating Operations, Safety, and Multimodal Planning: A Reference Manual (FHWA) Would include  A standard presentation  Discussion Stephen Brich, Kimley-Horn

B-1 APPENDIX B Pilot ROF Agendas  Agenda for Nashville ROF – September 9 to 13, 2013  Agenda for Seattle ROF – November 4 to 8, 2013  Agenda for Phoenix ROF – February 10 to 14, 2014  Agenda for Concord ROF – April 28 to May 2, 2014  Agenda for California ROF – May 12 to 18, 2014  Agenda for Milwaukee ROF – June 9 to 13, 2014

B- 2 AGENDA for NASHVILLE - REGIONAL OPERATIONS FORUM September 9, 2013 September 10, 2013 September 11, 2013 September 12, 2013 September 13, 2013 Monday Tuesday Wednesday Thursday Friday 8:00 AM Optional Technical Tour Systems Engineering Traffic Incident Management includes Road Weather and Emergency Operations Managing a Corridor includes ICM, ATM, Managed Lanes Team Exercise Presentations and Discussion 8:30 AM 9:00 AM Planning and Programming for Operations 9:30 AM Future of Operations Speaker 10:00 AM 10:30 AM Development of Agency Implementation Plans and Reports 11:00 AM 11:30 AM LUNCH LUNCH LUNCH 12:00 PM Freight Speaker Safety & Operations Speaker LUNCH Evaluation 12:30 PM Welcome, Opening Communicating the Value of Operations Presentation of Certificates 1:00 PM Participant Introductions Performance Measurement Planned Events includes Work Zones and Special Events 1:30 PM Overview of Operations 2:00 PM How to Organize for Operations 2:30 PM Review of Capability Self Evaluations 3:00 PM Traveler Information and Operations 3:30 PM Applying What You Learned 4:00 PM 4:30 PM Team Exercise Meetings 5:00 PM Introduction to Team Exercise Team Exercise Meetings Team Exercise Meetings 5:30 PM 6:00 PM DINNER DINNER DINNER DINNER 6:30 PM 7:00 PM Optional Group Activity

B- 3 AGENDA for SEATTLE - REGIONAL OPERATIONS FORUM November 4, 2013 November 5, 2013 November 6, 2013 November 7, 2013 November 8, 2013 Monday Tuesday Wednesday Thursday Friday 7:00 AM Future of Operations Speaker 7:30 AM 8:00 AM Optional Technical Tour Planning and Programming for Operations Traffic Incident Management includes Road Weather and Emergency Operations Managing a Corridor includes ICM, ATM, Managed Lanes Team Exercise Presentations and Discussion 8:30 AM 9:00 AM 9:30 AM Development of Agency Implementation Plans and Reports 10:00 AM 10:30 AM Systems Engineering 11:00 AM 11:30 AM LUNCH LUNCH LUNCH LUNCH Evaluation 12:00 PM Freight Speaker Transit Speakers Safety and Operations Speaker 12:30 PM Welcome, Opening Presentation of Certificates 1:00 PM Participant Introductions Performance Measurement Planned Events includes Work Zones and Special Events Communicating the Value of Operations 1:30 PM Overview of Operations 2:00 PM How to Organize for Operations 2:30 PM Review of Capability Self Evaluations 3:00 PM Traveler Information and Operations Workforce Development 3:30 PM 4:00 PM Applying What You Learned 4:30 PM 5:00 PM Introduction to Team Exercise Team Exercise Meetings Team Exercise Meetings Team Exercise Meetings 5:30 PM 6:00 PM DINNER DINNER DINNER DINNER ON OWN 6:30 PM 7:00 PM Optional Group Activity

B- 4 AGENDA for PHOENIX - REGIONAL OPERATIONS FORUM February 10, 2014 February 11, 2014 February 12, 2014 February 13, 2014 February 14, 2014 Monday Tuesday Wednesday Thursday Friday 8:00 AM Performance Measurement Traffic Incident Management, Emergency Operations and Planned Events Managing a Corridor includes ICM, ATM, Managed Lanes Team Exercise Presentations and Discussion 8:30 AM 9:00 AM 9:30 AM Development of Agency Implementation Plans and Reports 10:00 AM 10:30 AM 11:00 AM Welcome, Opening 11:30 AM Participant Introductions LUNCH LUNCH LUNCH Presentation of Certificates 12:00 PM LUNCH Freight Speaker Safety and Operations Speaker Future of Operations Speaker Evaluation 12:30 PM 1:00 PM Review of Capability Self Evaluations Systems Engineering Team Exercise Meetings Communicating the Value of Operations 1:30 PM 2:00 PM Team Exercise Meetings Road Weather How to Organize for Operations 2:30 PM 3:00 PM Intro to Team Exercise Workforce Development 3:30 PM Planning and Programming for Operations Technical Tour - Arizona DOT Traffic Operations Center Work Zones 4:00 PM Applying What You Learned 4:30 PM Traveler Information and Operations 5:00 PM Team Exercise Meetings 5:30 PM 6:00 PM DINNER DINNER DINNER DINNER ON OWN 6:30 PM 7:00 PM Optional Group Activity

B- 5 AGENDA for CONCORD - REGIONAL OPERATIONS FORUM April 28, 2014 April 29, 2014 April 30, 2014 May 1, 2014 May 2, 2014 Monday Tuesday Wednesday Thursday Friday 7:30 AM Group Photo Team Exercise Presentations 8:00 AM Planning and Programming for Operations Traffic Incident Management, Emergency Operations, and Planned Events Future of Transportation Speaker 8:30 AM 9:00 AM Break 9:30 AM Managing a Corridor includes ICM, ATM, Managed Lanes Mainstreaming M&O in a State Program: A CEO Perspective 10:00 AM Welcome, Opening Break 10:30 AM Participant Introductions Systems Engineering Development of Agency Implementation Plans and Reports 11:00 AM Setting the Stage 11:30 AM LUNCH LUNCH LUNCH LUNCH 12:00 PM Freight Speaker VIDEO Presentation of Certificates 12:30 PM Review of Capability Self Evaluations Road Weather Managing a Corridor, continued Evaluation 1:00 PM Performance Measurement 1:30 PM 2:00 PM Work Zones How to Organize for Operations 2:30 PM Intro to Team Exercise 3:00 PM Break Break 3:30 PM Technical Tour Traveler Information and Operations Communicating the Value of Operations 4:00 PM 4:30 PM Applying What You Learned 5:00 PM Team Exercise Meetings Team Exercise Meetings Agency Meetings on Implementation Plans 5:30 PM 6:00 PM DINNER DINNER DINNER DINNER ON OWN 6:30 PM 7:00 PM Optional Group Activity

B- 6 AGENDA for CALIFORNIA - REGIONAL OPERATIONS FORUM May 12, 2014 May 13, 2014 May 14, 2014 May 15, 2014 May 16, 2014 Monday Tuesday Wednesday Thursday Friday 7:30 AM Group Photo 8:00 AM Technical Tour Performance Measurement Traffic Incident Management, Emergency Operations, and Planned Events Managing a Corridor includes ICM, ATM, Managed Lanes Team Exercise Presentations and Discussion 8:30 AM 9:00 AM 9:30 AM Development of Agency Implementation Plans and Reports 10:00 AM 10:30 AM Break 11:00 AM Welcome, Setting Stage 11:30 AM Participant Introductions LUNCH LUNCH LUNCH Presentation of Certificates 12:00 PM LUNCH Evaluation 12:30 PM Freight Speaker Road Weather Communicating the Value of Operations 1:00 PM Review of Capability Self Evaluations 1:30 PM Systems Engineering 2:00 PM Work Zones Break 2:30 PM Break How to Organize for Operations 3:00 PM Break Future of Operations Break 3:30 PM Planning and Programming for Operations Traveler Information and Operations 4:00 PM Intro to Team Exercise Applying What You Learned 4:30 PM Team Exercise Meetings 5:00 PM Team Exercise Meetings Agency Implementation Plan Meetings 5:30 PM 6:00 PM RECEPTION AT HOTEL DINNER ON OWN DINNER ON OWN DINNER ON OWN 6:30 PM 7:00 PM Optional Group Activity * * Angels vs Tampa Bay Rays Baseball game - 7:05 start time

B- 7 AGENDA for MILWAUKEE - REGIONAL OPERATIONS FORUM June 9, 2014 June 10, 2014 June 11, 2014 June 12, 2014 June 13, 2014 Monday Tuesday Wednesday Thursday Friday 8:00 AM Performance Measurement Traffic Incident Management, Emergency Operations, and Planned Events Managing a Corridor includes ICM, ATM, Managed Lanes Team Exercise Presentations 8:30 AM 9:00 AM Break 9:30 AM Connected Vehicles & Future of Transportation Speaker 10:00 AM 10:30 AM Welcome, Opening Break 11:00 AM Participant Introductions Evaluation Evaluation Agency Implementation Plans Presentations 11:30 AM LUNCH LUNCH LUNCH LUNCH 12:00 PM Freight Speaker VIDEO Presentation of Certificates 12:30 PM Review of Capability Self Evaluations Road Weather Evaluation & Group Photo Evaluation 1:00 PM Systems Engineering How to Organize for Operations 1:30 PM 2:00 PM Team Exercise Meetings Work Zones Break 2:30 PM Introduction to Team Exercise Communicating the Value of Operations 3:00 PM Break 3:30 PM Break Technical Tour Traveler Information and Operations Applying What You Learned 4:00 PM Planning and Programming for Operations 4:30 PM Agency Meetings - Development of Implementation Plans 5:00 PM Team Exercise Meetings 5:30 PM 6:00 PM DINNER DINNER DINNER DINNER ON OWN 6:30 PM 7:00 PM Optional Group Activity

C-1 APPENDIX C Evaluation Forms  Classroom Session Evaluation  Speaker Session Evaluation  ROF Overall Evaluation  Follow-up Program Evaluation

C-2 Classroom Session Evaluation Session Title: __________________________ ROF Location and Date: ________________________ Please take a few moments to complete this evaluation of this session of the SHRP 2 L36 Regional Operations Forums. Your feedback is valuable to us and will help improve future sessions. 1. One specific thing I learned in this session:_______________________________________________________ 2. One specific thing that I will use in my job:_______________________________________________________ N/A = No answer/Not applicable, 1 = Strongly disagree, 2 = Disagree, 3 = Neutral, 4 = Agree, 5 = Strongly agree Using the scale above, please indicate how strongly you agree or disagree with the following statements about this session. Please mark only one answer for each statement using a  or an X. N/ A 1 2 3 4 5 3. The material for this session was well organized and easy to follow. 4. The material can be directly applied to my job. 5. I am more knowledgeable about this subject area after the session. 6. The instructor was knowledgeable in the subject matter covered. 7. The instructor communicated ideas and concepts clearly. 8. The instructor left me with something to think about. 9. The instructor used good examples in the lecture 10. The instructor encouraged attendees to participate. 11. The session contributed to my understanding of the importance and value of a management and operational focus within my organization. 12. As a result of the session I am likely to change the way I plan, program, and organize for systems management & operations in my organization. 13. Overall, the instructor was effective. C1. Additional comments related to the instructor, materials, and delivery of this session:

C-3 Speaker Session Evaluation Speaker Session Title: _______________________________________ ROF Location and Date: _________________________________________ Please take a few moments to complete this evaluation of the speaker session of the SHRP 2 L36 Regional Operations Forums. Your feedback is valuable to us and will help improve future sessions. N/A = No answer/Not applicable, 1 = Strongly disagree, 2 = Disagree, 3 = Neutral, 4 = Agree, 5 = Strongly agree Using the scale above, please indicate how strongly you agree or disagree with the following statements about the speaker today. Please select only one answer for each statement. N/A 1 2 3 4 5 1. The speaker was knowledgeable in the subject. 2. The speaker communicated ideas and concepts clearly. 3. The speaker’s presentation held my interest. 4. The speaker left me with something to think about. 5. The speaker encouraged attendees to participate. 6. The speaker presented ideas relevant to the Regional Operations Forums 7. Overall, the speaker was effective. S1. Additional comments related to the instructor, materials, and delivery of this session:

C-4 Regional Operations Forum Overall Evaluation ROF Location and Date: __________________________________________ Please take a few moments to complete this evaluation of the SHRP 2 L36 Regional Operations Forums. Your feedback is valuable to us and will help improve future sessions. I. General N/A=No answer/Not applicable, 1 = Strongly disagree, 2 = Disagree, 3 = Neutral, 4 = Agree, 5 = Strongly agree Using the scale above, please indicate your overall level of satisfaction with the program you just completed. Please select only one answer to each question. N/A 1 2 3 4 5 1. The structure and organization of the program was good. 2. The materials provided were of high quality. 3. The information provided to me prior to arriving for the first day of the classroom portion was effective. 4. The Forums contributed to my understanding of the importance and value of a management and operational focus within my organization. 5. As a result of the Forum I am highly likely to change the way in which I plan, program, and organize for systems management & operations in my organization. 6. I am likely to use the knowledge and skills I learned in the Forum to effectively use a management and operations perspective to identify and implement cost effective solutions to address traffic problems. 7. As a result of the Forum, I am likely to implement, recommend implementing, or modify the use of performance measures in my transportation network or organization. 8. The Forum encouraged me to help organize or participate in a peer network of regional agencies to share ideas and work together more effectively. 9. As a result of the Forum, I am more aware of the SHRP2 Reliability research and other nationwide systems management and operations research programs. 10. The program met its goal of providing the training and experience necessary for transportation management and operations personnel to effectively facilitate the use of existing transportation infrastructure.

C-5 F1. Is there anything you would change about the program? Yes No If “Yes,” please explain. F2. Would you recommend this program to others? Yes No If “No,” please explain why. F3. Please use this space to provide suggested topics for future SHRP 2 L36 Regional Operations Forums sessions and/or provide additional comments: II. Pre-Study Program N/A=No answer/Not applicable, 1 = Strongly disagree, 2 = Disagree, 3 = Neutral, 4 = Agree, 5 = Strongly agree Using the scale above, please indicate your overall level of satisfaction with the program you just completed. Please select only one answer to each question. N/A 1 2 3 4 5 11. The pre-study material was a necessary foundation for the classroom portion of the program. 12. The exams adequately tested the material you were required to review. 13. You received enough direction regarding your specific agency challenge to begin to address it when you return to your workplace. F4. Please use this space to provide additional comments on the pre-study program. For example, are there topics you would add or subtract from the pre-study program?

C-6 III. Classroom Training and Speakers F5. What did you like least about the classroom training and speakers? F6. What did you like most about the classroom training and speakers? F7. Please use this space to provide additional comments on classroom training and speakers: IV. Accommodations N/A=No answer/Not applicable, 1 = Strongly disagree, 2 = Disagree, 3 = Neutral, 4 = Agree, 5 = Strongly agree Using the scale above, please indicate your overall level of satisfaction with the accommodations. Please select only one answer to each question. N/A 1 2 3 4 5 14. The accommodations were satisfactory. 15. The learning environment, in terms of its conduciveness to studying and completing assignments, was satisfactory. F8. Please use this space to provide additional comments on the accommodations: V. Team Exercise F9. Did the Team Exercise help you apply what you learned in the Regional Operations Forum? If yes, how did the Team Exercise help you apply what you learned? If no, please indicate why it did not?

C-7 VI. Implementing Action Strategies in Your Agency A=No answer/Not applicable, 1 = Strongly disagree, 2 = Disagree, 3 = Neutral, 4 = Agree, 5 = Strongly agree ng please indicate your overall level of satisfaction with tmmodations. Please select nly one answer to each question. N/A 1 2 3 4 5 16. Today you were grouped with the other participants from your state to develop "Agency Implementation Plans.” How likely are you to implement the action strategies that you and your peers developed? F10. Please explain your answer. What conditions or issues would affect the implementation? VII. Technical Tour N/A=No answer/Not applicable, 1 = Strongly disagree, 2 = Disagree, 3 = Neutral, 4 = Agree, 5 = Strongly agree Using the scale above, please indicate your overall level of satisfaction with the accommodations. Please select only one answer to each question. N/A 1 2 3 4 5 17. The Technical Tour on Tuesday afternoon added to the value of the Regional Operations Forum. 18. The Technical Tour should be offered in future Regional Operations Forums. F11. Please explain your answers. What parts of the Technical Tour did you find most and least valuable?

C-8 NOTE: This was formatted for web implementation. Follow-up Program Evaluation Now that you have been back at your job for over six months after participating in the SHRP 2 L36 Regional Operations Forums, we want to hear your thoughts about the effectiveness of the program. Please take a few moments to complete this brief follow-up evaluation. We would really appreciate your feedback and it will help improve future Forums. Thank you! N/A = No answer/Not applicable, 1 = Strongly disagree, 2 = Disagree, 3 = Neutral, 4 = Agree, 5 = Strongly agree Using the scale above, please indicate your overall level of satisfaction with the Regional Operations Forum (ROF). Please select only one answer to each question. N/A 1 2 3 4 5 After several months of being back at my job following the ROF, I believe the ROF was effective. The training and instruction I received from the ROF has been directly applicable to my job. The listserv is an effective tool to share information with other ROF graduates. Would you say your experiences in the ROF were valuable? Why or why not? Are you applying the concepts you learned in your job? Yes No If you answered yes, please explain and provide specific examples. Have you been able to address/implement any of action strategies that you and the other participants from your agency developed? Yes No If you answered yes, please explain and provide specific examples. Would you participate in a one day follow-up workshop that included all the participants from the original ROF that you participated in? Yes No Is there something that could have been more valuable or applicable to your job?

D-1 APPENDIX D Scheduling and Outreach Documents  ROF Generic Fact Sheet  ROF Participation Commitment Form  Pre-ROF Webinar – PowerPoint Presentation on Overall ROF and Prestudy Requirements  ROF Prestudy Instructions

D-2 Regional Operations Forum Generic Fact Sheet Background There will be five pilot Regional Operations Forums (ROFs) conducted around the country as part of the SHRP2 L36 project Regional Operations Forums for Advancing Systems Operations, Management, and Reliability. Description As states confront transportation problems with tighter fiscal budgets, the necessity for better managing and operating the transportation system becomes extremely critical. The ROFs provide practitioners with new and innovative approaches for managing and operating the highway system, drawing from the cutting edge work being carried out under the SHRP2 program and other national programs. Participation in a (ROF) provides the strategies, technologies and practices needed to advance the paradigm shift towards Advancing Systems Operations, Management, and Reliability Goals  To mainstream systems management & operations into the culture of the state departments of transportation by transitioning the state of the art closer to the state of the practice  To strengthen the systems management & operations programs at the state and regional level  To develop a community of practice through the development of a peer network  To provide the next generation of leadership with the critical thinking skills for advancing system management & operations  To provide training on the “best use” of SHRP2 Reliability products Objective Upon completion of the program, participants should be able to:  Understand the importance and value of a management and operational focus within their agency  Know how to plan, program and organize for systems management & operations  More effectively use a management and operations perspective in identifying and implementing cost effective solutions to address transportation problems  Measure the performance of the operations of their transportation network  Work with a peer network of regional agencies to share ideas/information, learn from the activities of others, and work together more effectively.  Apply where applicable, the results of the SHRP2 Reliability research and other nationwide systems management and operations research programs. Target Audience Director/Manager of Traffic Operations District/Division/State Traffic Engineers Regional Director/Operations Chief Director of Maintenance Transportation Operations Center Manager Manager of Traffic Engineering in state or local jurisdiction

D-3 Senior Transportation Planner in state or local jurisdiction or MPO State Police Date/Location Participating States Travel Travel for __________ participants from each state is covered under the project. See travel information. Agenda See attached draft Cost for Extra Attendees (above the ?? paid by project) These costs are per person and payable to Housman and Associates: Hotel: ___________for four nights taxable, $________ for five nights taxable Hotel: $________ for four nights tax exempt, $________ for five nights tax exempt For tax exempt must present tax exempt certificate for verification at check in Meals: $________ (Monday through Friday) These costs are to be paid by the agency: Travel to and from the meeting location Book: “The Road Ahead…Why are we driving 21st Century Cars on 20th Century Roads with 19th Century Thinking?” By Phil Tarnoff, Found on Amazon for $26.49 Note: If you are interested in sending additional attendees, you MUST coordinate with Kathleen Frankle. There is only a limited number of additional seats available. kfrankle@umd.edu, 410-414-2925, cell: 410-303- 4728 FHWA Division Representatives: If a representative from an FHWA division wants to audit the course, that is acceptable but they must pay their own hotel (rate above) and meals costs (per diem amounts). They will need to coordinate with Kathleen Frankle as well.

D-4 Regional Operations Forum (ROF) Participation Commitment Form Contact Information: Name: ___________________________________ Title: _______________________________ Agency: _____________________________________ Phone:______________________________ Mailing Address: ____________________________________________________________________ E-mail Address:______________________________________________________________________ Date: ________________________________________ Location: I plan to attend the Regional Operations Forum.  I understand that about 20 hours of pre-study is required and must be completed by (fill in).  I understand that it is a requirement that I am in attendance every day of the ROF.  I understand that if I am unable to attend, I will notify (fill in) right away so someone else can attend instead.  I understand that this is a unique training opportunity and I am looking forward to participating! Accommodations: _______________________________________________________________________ I am local to the ___________________________ area and would rather stay at home. o I understand that I need to participate in all activities but just won’t need accommodations at the hotel. I will need hotel accommodations. o Check in on Monday, ________________ and checking out Friday__________________ Meals: I have no special needs for meals. I will need vegetarian meals. ---------------------------------------------------------------------------------------------------------------- ---------------

D-5 I am unable to commit my time to attend the four-day ROF and therefore, understand that someone else Please e-mail this completed form to ___________________________________by _________________________ Signature: _____________________________________________________________

D-6

D-7

D-8

D-9

D-10 Regional Operations Forum (ROF) Pre-Study Instructions The pre-study has been divided into several parts. All parts will be averaged together and to account for 15% of your overall grade for the ROF program (see attached grading summary). In order to receive CEUs for your participation in the ROF, there are several items that will be graded during the ROF. These include: the pre-study, workshop, and participation in the action planning process. Part 1 – Challenge within Your Organization You are attending the Regional Operations Forum (ROF) to gain an appreciation for the power of Transportation Systems Management and Operations (TSM&O) and to identify ways in which TSM&O can receive greater emphasis within your organization. Please tell us about the single greatest challenge you are having within your organization relating to increasing its focus on TSM&O. Every ROF participant is expected to complete the attached form on their agency challenge as part of the pre-study program. Your responses will be provided to all the instructors for them to review prior to the ROF so they can address your challenges as part of their sessions. It should be noted that not every instructor will be able to discuss every agency challenge that is submitted. In this way, we will ensure that the presentations are relevant to your specific concerns. Hopefully, we will be able to address all the challenges that are submitted. If not, you will at least be able to learn from some of the other agency challenge discussions. You will not be graded for the content of your submission but you will be given a grade for doing the work. Thus, if you don’t submit it you will receive a zero and if you do submit it you will receive a 100%. Due date: ______________ Part 2 – CMM Self Evaluation An important focus in the ROFs is to support participating agencies in the improvement of their capabilities to deliver effective Transportation Systems Management and Operations (TSM&O) on a continuing basis. Significant SHRP 2 research has identified the reality that program effectiveness – as it varies from agency to agency – is directly related to an agency having in place six key “capabilities”: business processes, systems & technology, performance measurement, culture, organization/workforce, and collaboration—that together are needed to support the development and implementation of effective strategies (such as incident management or special event management, road weather information, etc.). Based on this research, a transportation agency self-evaluation process was developed that has been applied via workshops in 17 states – focusing on state DOTs and their key partners in TSM&O strategy delivery. (These workshops are being continued in another 20 states next year).

D-11 The core of this process is self-evaluation regarding where an agency currently stands regarding its key capabilities in the six dimensions of capability – and identifying practical strategies for going from the mores ad hoc approaches to those that are increasingly managed and fully integrated. CMM Self Evaluation On the first day of the ROF, there will be a session where we will discuss/review the “Capability Maturity Model” self-evaluations. During the session, we will discuss the concept of continuous improvement, key capabilities, levels of capability maturity and what agencies need to do to get to the next level. To prepare for this session, each participant is asked to do three things. 1) Read the attached TRNews article, “Institutional Architectures to Improve Transportation Systems Management and Operations,” that explains the background to the research, findings, key concepts and self-evaluation process. 2) Complete and submit the brief self-evaluation regarding your agency’s state-of-play in each of the six dimensions. You do this by following these steps: a. Go to the AASTHO TSM&O Guide website: http://www.aashtotsmoguidance.org/ b. Click on One-Minute Evaluation and follow the instructions for selecting the current level of capability of you agency according to the criteria (not where you may be heading!!) c. Hit Next – and on the Results Summary page you should see your current levels indicated. d. Hit Save and Send: enter your name and email to mztwiz@umd.edu During the ROF Prior to the ROF, the self-evaluation forms will be compiled by the instructor into averages for all the participating agencies – in order to identify common and differing strengths and weaknesses among the ROF states. States will not be identified by name. The criteria defining levels of maturity will then provide the framework for a discussion of the strategic actions needed to move up to the next level of capability – including examples from participant experience. Given time limits, the focus will be on the 2-3 most critical of the 6 dimensions – those where improvement is essential to increased TSM&O effectiveness. The output of the session will constitute an important part of each state’s “take-away” from the ROF – and will also be useful to define areas where technical support will be the most useful. You will not be graded for the content of your submission but you will be given a grade for doing the work. Thus, if you don’t submit it you will receive a zero and if you do submit it you will receive a 100%. Due date: ______________ Part 3 – “The Road Ahead” by Phil Tarnoff

D-12 You will need to read several chapters of the book “The Road Ahead” by Phil Tarnoff – see list below. The book is being mailed to your office to the address you provided. Once you have completed the required chapters you will go on-line to complete an online exam. Information will be e- mailed to you on how to access the online exam. Please let Kathleen Frankle know if you do not receive the book and/or if you have any questions. Below is a list of the chapters for you to read from the book. Due date: ______________  Chapter 3 – The State of the System – An overview of supply-demand  Chapter 4 – Getting There Alive: An overview of safety  Chapter 6 – A Tale of Two Cultures: A discussion of M&O along with the disconnect with the construction mentality  Chapter 8 – The Twenty-First Century: A discussion of the future  Chapter 9 – Developing the Workforce Part 4 – Overview of Operations We have developed a short session on the Overview of Operations. It is supposed to be an online course but we did not have enough time to put it online since all participants need to review it before your attendance at the ROF in September. Therefore, we are providing you will the slides and the instructor notes. On the first day of the ROF, the facilitator will also spent about an hour going through the most important things that you should have gotten out of the session. When you complete reviewing the session, please complete the exam for this session. It can be found under “My Tests and Surveys.” It will be called “Part 4 Exam – Overview of Operations.” Due date: ______________ Part 5 – Systems Engineering The Consortium for ITS Training and Education (CITE) has an online course called “Introduction to Systems Engineering.” Please review the lessons listed below and complete the online exam. You will need to login using the instructions on the next page to review the materials and take the online exam.  Lesson 1: Overview of Systems Engineering  Lesson 2: The ‘V’ Model Due date: ______________ You are more than welcome to go through the other Lessons in the course if you would like but they are not required.

D-13 Part 6 – Planning and Programming for Operations There are two pieces to this part of the pre-study. Due date: ______________ 1) Read the three (3) page document called “What is Planning for Operations.” 2) There are several questions that we would like for you to answer. In order to do that, you will need to talk to someone in your agency’s planning department. Please bring your answers to these questions to the Monday afternoon session during the ROF called “Planning and Programming for Operations.” The answers to these questions will assist in your participation during the session. RECOMMENDED but not mandatory: 3) Skim the other documents sometime before _______________________. They include:  Extracts from SHRP 2 L05  NC Traffic Plan – Operations Focused Recommendation  Portland Metro TSM&O Plan – Executive Summary  WSDOT Statewide ITS Plan Part 7 – Team Exercise Please read through the background materials for the Team Exercise. You will receive instruction for the Team Exercise on _______________________. There is no exam and nothing to turn in at this point. It is just important to read this material before your first Team Exercise Meeting. Due date: ______________ Web Cast We will be conducting a webcast on __________________. We will provide a summary of the pre-study materials and also allow you to ask any questions about the pre-study or the ROF itself. URL for webinar: _______________________ Call in number: _______________________ Passcode: ____________________ Instructions for Accessing the On-line Materials and Exams 1. Go to the following address: http://citetraining.org/ 2. On the left side of screen, type in your assigned Login Name and Password. You will be sent this information via e-mail. It is usually your first initial_last name. However, there are always some exceptions to that configuration so please be on the lookout for the e- mail. If you don’t get this e-mail, contact Kathy Frankle (see contact information below). 3. Click Login 4. You will see a list of courses that you have access to view

D-14 5. Click on the “Regional Operations Forum” course 6. You should now be in the course 7. On the left side under “Content Navigation” is a list of all six parts of the pre-study. Click on each one separately to obtain the materials for that part. 8. The exams are found on the home page. Click on “My Tests and Surveys.” Questions If you have any questions or problems with anything related to the ROF pre-study, please contact Kathleen Frankle and she will be happy to assist. E-mail: kfrankle@umd.edu, work: 410-414-2925, cell: 410-303-4728

E-1 APPENDIX E Evaluation Summaries  Nashville ROF – September 9 to 13, 2013  Seattle ROF – November 4 to 8, 2013  Phoenix ROF – February 10 to 14, 2014  Concord ROF – April 28 to May 2, 2014  California ROF – May 12 to 18, 2014  Milwaukee ROF – June 9 to 13, 2014  Follow-up Survey Summary

E-2 Nashville, TN: September 9 to 13, 2013 N/A=No answer/Not applicable, 1 = Strongly disagree, 2 = Disagree, 3 = Neutral, 4 = Agree, 5 = Strongly agree N/A 1 2 3 4 5 Blank 1. The structure and organization of the program was good. - - 3 (9.7%) 4 (12.9%) 18 (58.1%) 6 (19.4%) - 2. The materials provided were of high quality. - - 1 (3.2%) 6 (19.4%) 15 (48.4%) 9 (29.0%) - 3. The information provided to me prior to arriving for the first day of the classroom portion was effective. - - - 4 (12.9%) 23 (74.2%) 4 (12.9%) - 4. The Forums contributed to my understanding of the importance and value of a management and operational focus within my organization. - - - 1 (3.2%) 21 (67.7%) 9 (29.0%) - 5. As a result of the Forum I am highly likely to change the way in which I plan, program, and organize for systems management & operations in my organization. 2 (6.5%) - - 2 (6.5%) 20 (64.5%) 7 (22.6%) - 6. I am likely to use the knowledge and skills I learned in the Forum to effectively use a management and operations perspective to identify and implement cost effective solutions to address traffic problems. 3 (9.7%) - - 4 (12.9%) 21 (67.7%) 3 (9.7%) - 7. As a result of the Forum, I am likely to implement, recommend implementing, or modify the use of performance measures in my transportation network or organization. - - 1 (3.2%) 5 (16.1%) 19 (61.3%) 6 (19.4%) - 8. The Forum encouraged me to help organize or participate in a peer network of regional agencies to share ideas and work together more effectively. 1 (3.2%) - - 3 (9.7%) 18 (58.1%) 9 (29.0%) - 9. As a result of the Forum, I am more aware of the SHRP2 Reliability research and other nationwide systems management and operations research programs. - - - 2 (6.5%) 20 (65.5%) 9 (29.0%) - 10. The program met its goal of providing the training and experience necessary for transportation management and operations personnel to effectively facilitate the use of existing transportation infrastructure. - - - 3 (9.7%) 21 (67.7%) 7 (22.6%) -

E-3 11. The pre-study material was a necessary foundation for the classroom portion of the program. 1 (3.2%) - - 4 (12.9%) 23 (74.2%) 3 (9.7%) - 12. The exams adequately tested the material you were required to review. - 1 (3.2%) 3 (9.7%) 6 (19.4%) 16 (51.6%) 5 (16.1%) - 13. You received enough direction regarding your specific agency challenge to begin to address it when you return to your workplace. - 1 93.2%) 2 (6.5%) 7 (22.6%) 17 (54.8%) 4 (12.9%) - 14. The accommodations were satisfactory. 1 (3.2%) - - 4 (12.9%) 19 (61.3%) 6 (19.4%) 1 (3.2%) 15. The learning environment, in terms of its conduciveness to studying and completing assignments, was satisfactory. 1 (3.2%) - - 4 (12.9%) 20 (64.5%) 6 (19.4%) - 16. You were grouped with the other participants from your state to develop "Agency Implementation Plans.” How likely are you to implement the action strategies that you and your peers developed? 1 (3.2%) - 6 (19.4%) 10 (32.3%) 12 (38.7%) 1 (3.2%) 1 (3.2%) 17. The accommodations were satisfactory. - - 5 (16.1%) 8 (25.8%) 13 (41.9%) 5 (16.1%) - 18. The learning environment, in terms of its conduciveness to studying and completing assignments, was satisfactory. - - 3 (9.7%) 5 (16.1%) 18 (58.1%) 5 (16.1%) - 19. How likely are you to implement the action strategies in your agency? - - - 4 (12.9%) 18 (58.1%) 8 (25.8%) 1 (3.2%) Overall ROF Evaluation form responses  Favorable ratings overall—vast majority of responses were “agree” or “strongly agree” for most of the questions. There were several comments about potential improvements though. o Selective paraphrased open ended comments about most liked and least liked aspects:  Too much lecture (multiple—over one-third of responders). Goal of fostering discussions and improvement in DOT operations could be done with less classroom time. Classroom portion was too long (multiple).  Need more breaks/field trips (multiple).  Slide shows were too long and boring (multiple). Speakers had to rush to get through too many slides. “Death by PowerPoint.” “150+ slides.”  Networking with other DOTs was good.  Some speakers were more boring than others, but overall the team was good. Presenters had knowledge, passion, enthusiasm, and were well informed.

E-4  Need a neutral site so staff from home state does not get distractions from work.  Focus on using laptops resulted in participants doing other things on laptops during the sessions.  There were two statements that yielded slightly different responses frequencies: o Q16, pre-study: “I would add or subtract subjects or topic areas to the pre-study program to help participants prepare for the classroom portion of the program.” 19% of respondents disagreed with this statement, and 32% were neutral. 41% agreed or strongly agreed with the statement. o Selective paraphrased open ended comments about the pre-study:  Book and articles were good, but PowerPoint slides did not enhance learning as well. Another said that 110 slide pre-study was too long, with too much emphasis on engineering. Length of time for pre-study was about right.  Access to pre-study materials was not easy. Technology not helpful. CITE training website was difficult to access the first time. Better instructions would help.  Recommend that you start with a TSMO quiz to determine which parts of pre-study are needed.  Less emphasis on organizational needs and more emphasis on value. Need to update Table 4a, b, c, d in the book. Disappointed that after 40 years we could not fill in the effectiveness of this table.  Felt like we went over what we had already studied/knew; book and papers were repeated in the class; pre-study was repetitive;  Book and self-assessment were good.  Pre-study time was a lot but got a lot out of it as preparation for the class.  Exam questions were poorly worded.  Remove the systems portion of the program.  Topic of “maturation” should be introduced in the pre-study.  Pre-study helped prepare class for the right mind set towards operations.  Some modules were too broad. Would like more detail on systems engineering and its architecture.  A lot of tests/requirements were left to the last day before coming to the forum. Spread them out more. o Q17: “The accommodations were satisfactory.” 16% of respondents disagreed with this statement, and 25% were neutral. Only 42% agreed or strongly agreed with the statement. o Selective paraphrased open ended comments about the facilities:  Room too cold (multiple responders-well more than half).  Hotel too far from facility (multiple responders-at least one quarter).  Hotel renovations were an issue.  Room was loud. Had trouble hearing.  Food was good (multiple). However, not enough choices and too much chicken.

E-5 Seattle, Washington: November 4 to 8, 2013 N/A=No answer/Not applicable, 1 = Strongly disagree, 2 = Disagree, 3 = Neutral, 4 = Agree, 5 = Strongly agree N/A 1 2 3 4 5 Blan k 1. The structure and organization of the program was good. - - - 4 (12.5%) 19 (59.4%) 9 (28.1%) - 2. The materials provided were of high quality. - - - 2 (6.2%) 20 (62.5%) 10 (31.2%) - 3. The information provided to me prior to arriving for the first day of the classroom portion was effective. - - 1 (3.1%) 2 (6.2%) 18 (56.2%) 10 (31.2%) 1 (3.1%) 4. The Forums contributed to my understanding of the importance and value of a management and operational focus within my organization. - - - 1 (3.1%) 17 (53.1%) 14 (43.8%) - 5. As a result of the Forum I am highly likely to change the way in which I plan, program, and organize for systems management & operations in my organization. - - 1 (3.1%) 7 (21.9%) 16 (50.0%) 8 (25.0%) - 6. I am likely to use the knowledge and skills I learned in the Forum to effectively use a management and operations perspective to identify and implement cost effective solutions to address traffic problems. - - - 9 (28.1%) 16 (50.0%) 7 (21.9%) - 7. As a result of the Forum, I am likely to implement, recommend implementing, or modify the use of performance measures in my transportation network or organization. - - 1 (3.1%) 4 (12.5%) 17 (53.1%) 10 (31.2%) - 8. The Forum encouraged me to help organize or participate in a peer network of regional agencies to share ideas and work together more effectively. - - - 5 (15.6%) 18 (56.2%) 9 (28.1%) - 9. As a result of the Forum, I am more aware of the SHRP2 Reliability research and other nationwide systems management and operations research programs. - - - 3 (9.4%) 18 (56.2%) 11 (34.4%) - 10. The program met its goal of providing the training and experience necessary for transportation management and operations personnel to effectively facilitate the use of existing transportation infrastructure. - - - 3 (9.4%) 20 (62.5%) 9 (28.1%) - 11. The pre-study material was a necessary foundation for the classroom portion of the program. 1 (3.1%) 1 (3.1%) 3 (9.4%) 4 (12.5%) 14 (43.8%) 8 (25.0%) 1 (3.1%) 12. The exams adequately tested the material you were required to review. 2 (6.2%) - 4 (12.5%) 5 (15.6%) 15 (46.9%) 5 (15.6%) 1 (3.1%)

E-6 13. You received enough direction regarding your specific agency challenge to begin to address it when you return to your workplace. 2 (6.2%) - 2 (6.2%) 3 (9.4%) 18 (56.2%) 6 (18.8%) 1 (3.1%) 14. The accommodations were satisfactory. 2 (6.2%) - 1 (3.1%) 2 (6.2%) 18 (56.2%) 8 (25.0%) 1 (3.1%) 15. The learning environment, in terms of its conduciveness to studying and completing assignments, was satisfactory. 2 (6.2%) - 1 (3.1%) 1 (3.1%) 19 (59.4%) 8 (25.0%) 1 (3.1%) 16. You were grouped with the other participants from your state to develop "Agency Implementation Plans.” How likely are you to implement the action strategies that you and your peers developed? 2 (6.2%) - 1 (3.1%) 6 (18.8%) 15 (46.9%) 6 (18.8%) 2 (6.2%) YES NO Blank F1. Is there anything you would change about the program? 24 (75.0%) 3 (9.4%) 4 (12.5%) F2. Would you recommend this program to others? 29 (90.6%) - 3 (9.4%)  Overall Evaluation o Favorable ratings overall—vast majority of responses were “agree” or “strongly agree” for most of the questions. There were several comments about potential improvements though.  Selective paraphrased open ended comments about most liked and least liked aspects: o There were two statements that yielded slightly different responses frequencies:  Q11: “The pre-study material was a necessary foundation for the classroom portion of the program.” 12.5% of respondents disagreed or strongly disagreed with this statement, and 12.5% were neutral. 68.8% agreed or strongly agreed with the statement.  Selective paraphrased open ended comments about the pre-study:  Q12: “The exams adequately tested the material you were required to review.” 12.5% of respondents disagreed or strongly disagreed with this statement, and 15.6% were neutral. 62.5% agreed or strongly agreed with the statement.  Selective paraphrased open ended comments about the facilities:

E-7 Most frequently cited findings from open-ended items (n equals 3 or more). Note: Some repetition because respondent comments were from different open-ended items:  Long days/sessions (9), even if we knew it was coming. Lots of information to process. Too little time outside of presentations for discussions at lunch and during breaks. Need more/longer breaks (4). It is good to have pre-study to get people on similar page prior to attending (4). Days were too long to absorb/decompress (3).  Need more breaks (3), even for 5 minutes (2). Need a full hour for lunch.  Speakers were extremely knowledgeable/enthusiastic/well-spoken/engaging (7).  Team exercise helped break down and categorize information/understanding of the application of the modules (9). Incremental improvements versus broad infrastructure. Reinforced concepts in a practical, realistic manner (2). The best part was getting to know/having discussions with the other participants in a smaller group session (7).  Networking/open discussions with other states/participants was good (5). Team/group exercises were specifically cited by respondents (3).  More time for team/group exercises during the day (3). Remove the team presentations and night work and replace it with more time for the state action plans (that will be implemented).  Main emphasis on scalability when participants are from rural states was a problem (3).  Main obstacles to implementation were buy-in from management (7). If FHWA makes it a priority, it will be a priority in our state. Getting agreement among the individuals and HQ and regional offices (3) could be difficult.

E-8 Phoenix, Arizona: February 10 to 14, 2014 N/A=No answer/Not applicable, 1 = Strongly disagree, 2 = Disagree, 3 = Neutral, 4 = Agree, 5 = Strongly agree N/A 1 2 3 4 5 Blank Multi 16. The structure and organization of the program was good. - - 3 (9.1%) 3 (9.1%) 19 (57.6%) 8 (24.2%) - 17. The materials provided were of high quality. - - - 3 (9.1%) 24 (72.7%) 6 (18.2%) - - 18. The information provided to me prior to arriving for the first day of the classroom portion was effective. 1 (3.0%) 1 (3.0%) 5 (15.2%) 3 (9.1%) 15 (45.5%) 8 (24.2%) - - 19. The Forums contributed to my understanding of the importance and value of a management and operational focus within my organization. - - 1 (3.0%) 2 (6.1%) 15 (45.5%) 15 (45.5%) - - 20. As a result of the Forum I am highly likely to change the way in which I plan, program, and organize for systems management & operations in my organization. - - 2 (6.1%) 1 (3.0%) 19 (57.6%) 11 (33.3%) - - 21. I am likely to use the knowledge and skills I learned in the Forum to effectively use a management and operations perspective to identify and implement cost effective solutions to address traffic problems. - - 1 (3.0%) 3 (9.1%) 19 (57.6%) 10 (30.3%) - - 22. As a result of the Forum, I am likely to implement, recommend implementing, or modify the use of performance measures in my transportation network or organization. - - 1 (3.0%) 3 (9.1%) 14 (42.4%) 15 (45.5%) - - 23. The Forum encouraged me to help organize or participate in a peer network of regional agencies to share ideas and work together more effectively. - - - 3 (9.1%) 12 (36.4%) 18 (54.5%) - - 24. As a result of the Forum, I am more aware of the SHRP2 Reliability research and other nationwide systems management and operations research programs. - - 2 (6.1%) 5 (15.2%) 17 (51.5%) 9 (27.3%) - - 25. The program met its goal of providing the training and experience necessary for transportation management and operations personnel to effectively facilitate the use of existing transportation infrastructure. - - 1 (3.0%) 5 (15.2%) 18 (54.5%) 9 (27.3%) - -

E-9 26. The pre-study material was a necessary foundation for the classroom portion of the program. 1 (3.0%) 2 (6.1%0 5 (15.2%) 8 (24.2%) 10 (30.3%) 7 (21.2%) - - 27. The exams adequately tested the material you were required to review. 2 (6.1%) 1 (3.0%) 3 (9.1%) 8 (24.2%) 14 (42.4%) 5 (15.2%) - - 28. You received enough direction regarding your specific agency challenge to begin to address it when you return to your workplace. 1 (3.0%) 1 (3.0%) 4 (12.1%) 2 (6.1%) 18 (54.5%) 7 (21.2%) - - 29. The accommodations were satisfactory. 1 (3.0%) 1 (3.0%) - - 11 (33.3%) 20 (60.6%) - - 30. The learning environment, in terms of its conduciveness to studying and completing assignments, was satisfactory. - 1 (3.0%) 1 (3.0%) 2 (6.2%) 11 (33.3%) 18 (54.5%) - - 31. Today you were grouped with the other participants from your state to develop "Agency Implementation Plans.” How likely are you to implement the action strategies that you and your peers developed? - - 5 (15.2%) 2 (6.1%) 12 (36.4%) 14 (42.4%) - - 32. The technical Tour on Tuesday afternoon added to the value of the Regional Operations Forum. 1 (3.0%) - 1 (3.0%) 3 (9.1%) 11 (33.3%) 17 (51.5%) - - 33. The Technical Tour should be offered in future Regional Operations Forums. - - 1 (3.0%) 1 (3.0%) 10 (30.3%) 21 (63.6%) - -  Overall Evaluation o Favorable ratings overall—vast majority of responses were “agree” or “strongly agree” for most of the questions o There were four statements that yielded slightly different response frequencies:  Q3: “The information provided to me prior to arrive for the first day of the classroom portion was effective.” 18.2% of respondents disagreed or strongly disagreed with this statement, and 9.1% were neutral. 69.7% agreed or strongly agreed.  Q11: “The pre-study material was a necessary foundation for the classroom portion of the program.” 21.3% of responded disagreed or strongly disagreed with this statement, and 24.2% were neutral. 51.5% of respondents agreed or strongly agreed.  Q12: “The exams adequately tested the material you were required to review” 12.1% of respondents disagreed or strongly disagreed, 24.2% were neutral. 57.6% of respondents agreed or strongly agreed.  Q13: “You received enough direction regarding your specific agency challenge to begin to address it when you return to your workplace.” 15.1% of respondents disagreed or strongly disagreed, 6.1% were neutral. 75.7% of respondents agreed or strongly agreed.

E-10 Most frequently cited findings from open-ended items (n equals 3 or more). Note: Some repetition because respondent comments were from different open-ended items:  Need a few more breaks because of the length of some of the segments (3).  Need more time for group discussion and sharing (3), including more peer-to-peer sharing (1) and interagency dialogue (1). Wednesday afternoon session [and other sessions] were too long (3) without a break (1). The video on Wednesday contributed to the length issue. Participants should get a chance to guide the emphasis of subjects that are focused on. Group exercises within different presentations could provide collaborative opportunities.  Reading was good (4) but seemed a bit dated (2). Book was a good intro to TSM&O.  Not enough breaks (7). Going 2+ hours with multiple topics is too long (4).  Did not like most of the video presentations (7).  Speakers were knowledgeable (5), most of them are well-prepared (1) and encouraged class participation (1). Speakers were excellent, dynamic (1).  The portion on TIMS (5), and how we can build it to work better with DPS/DOTs (1).  Team exercises (3). Networking (2). Sessions that encouraged peer-to-peer exchanges.  Group discussions (3) and sharing of ideas as the information was presented or shortly after.  Great place (8), took great care of us (1). Well suited for this type of conference (1). Single location of training, lodging, dining, etc. provided a good atmosphere for participants to interact and get to know one another/network (1). John deserves pat on back for the hotel (2).

E-11 Concord, New Hampshire: April 24 to May 2, 2014 N/A=No answer/Not applicable, 1 = Strongly disagree, 2 = Disagree, 3 = Neutral, 4 = Agree, 5 = Strongly agree N/A 1 2 3 4 5 Blank Multi 1. The structure and organization of the program was good. - - - 1 (4.2%) 10 (41.7%) 13 (54.2%) - - 2. The materials provided were of high quality. - - - - 11 (45.8%) 13 (54.2%) - - 3. The information provided to me prior to arriving for the first day of the classroom portion was effective. 1 (4.2%) - - 2 (8.3%) 12 (50.0%) 9 (37.5%) - - 4. The Forums contributed to my understanding of the importance and value of a management and operational focus within my organization. - - - - 8 (33.3%) 16 (66.7%) - - 5. As a result of the Forum I am highly likely to change the way in which I plan, program, and organize for systems management & operations in my organization. - - - 1 (4.2%) 14 (58.3%) 9 (37.5%) - - 6. I am likely to use the knowledge and skills I learned in the Forum to effectively use a management and operations perspective to identify and implement cost effective solutions to address traffic problems. - - - 1 (4.2%) 13 (54.2%) 10 (41.7%) - - 7. As a result of the Forum, I am likely to implement, recommend implementing, or modify the use of performance measures in my transportation network or organization. - - - 1 (4.2%) 13 (54.2%) 10 (41.7%) - - 8. The Forum encouraged me to help organize or participate in a peer network of regional agencies to share ideas and work together more effectively. - - - - 12 (50.0%) 12 (50.0%) - - 9. As a result of the Forum, I am more aware of the SHRP2 Reliability research and other nationwide systems management and operations research programs. - - - 1 (4.2%) 9 (37.5%) 14 (58.3%) - - 10. The program met its goal of providing the training and experience necessary for transportation management and operations personnel to effectively facilitate the use of existing transportation infrastructure. - - - 1 (4.2%) 11 (45.8%) 12 (50.0%) - -

E-12 11. The pre-study material was a necessary foundation for the classroom portion of the program. 1 (4.2%) - - 2 (8.3%) 13 (54.2%) 8 (33.3%) - - 12. The exams adequately tested the material you were required to review. 1 (4.2%) - - 1 (4.2%) 12 (50.0%) 10 (41.7%) - - 13. You received enough direction regarding your specific agency challenge to begin to address it when you return to your workplace. 1 (4.2%) - - 3 (12.5%) 13 (54.2%) 7 (29.2%) - - 14. The accommodations were satisfactory. 1 (4.2%) - - 1 (4.2%) 9 (37.5%) 13 (54.2%) - - 15. The learning environment, in terms of its conduciveness to studying and completing assignments, was satisfactory. - - - 1 (4.2%) 10 (41.7%) 13 (54.2%) - - 16. Today you were grouped with the other participants from your state to develop "Agency Implementation Plans.” How likely are you to implement the action strategies that you and your peers developed? - - - 1 (4.2%) 10 (41.7%) 13 (54.2%) - - 17. The technical Tour on Tuesday afternoon added to the value of the Regional Operations Forum. 1 (4.2%) - - 3 (12.5%) 4 (16.7%) 15 (62.5%) 1 (4.2%) - 18. The Technical Tour should be offered in future Regional Operations Forums. 1 (4.2%) - - 2 (8.3%) 3 (12.5%) 17 (70.8%) 1 (4.2%) - Overall Evaluation o Favorable ratings overall—vast majority of responses were “agree” or “strongly agree” for most of the questions o For nearly all questions, a handful (1–3) people responded “neutral”

E-13 Most frequently cited findings from open-ended items (n equals 3 or more). Note: Some repetition because respondent comments were from different open-ended items:  Want to see the slides for the presentations beforehand or have printouts to take notes on (4).  Useful to have state specific information more applicable to the agencies there, at least for a portion of the sessions (4).  Sessions were too long. There was too much sitting down without time for a break or to stretch (6).  Speakers were very knowledgeable about subject matter (6).  Is important to get this message to senior management, they should be the ones attending this (3). California: May 12 to 16, 2014 N/A=No answer/Not applicable, 1 = Strongly disagree, 2 = Disagree, 3 = Neutral, 4 = Agree, 5 = Strongly agree N/A 1 2 3 4 5 Blank 1. The structure and organization of the program was good. - - 1 (3.4%) 1 (3.4%) 18 (62.1%) 8 (27.6%) 1 (3.4%) 2. The materials provided were of high quality. - - - 3 (10.3%) 15 (51.7%) 10 (34.5%) 1 (3.4%) 3. The information provided to me prior to arriving for the first day of the classroom portion was effective. - - - 3 (10.3%) 13 (44.8%) 12 (41.4%) 1 (3.4%) 4. The Forums contributed to my understanding of the importance and value of a management and operational focus within my organization. - - - 1 (3.4%) 16 (55.2%) 11 (37.9%) 1 (3.4%) 5. As a result of the Forum I am highly likely to change the way in which I plan, program, and organize for systems management & operations in my organization. 4 (13.8%) - - 6 (20.7%) 11 (37.9%) 7 (24.1%) 1 (3.4%) 6. I am likely to use the knowledge and skills I learned in the Forum to effectively use a management and operations perspective to identify and implement cost effective solutions to address traffic problems. 1 (3.4%) - - 2 (6.9%) 16 (55.2%) 9 (31.0%) 1 (3.4%) 7. As a result of the Forum, I am highly likely to implement, recommend implementing, or modify the use of performance measures in my transportation network or organization. 1 (3.4%) - - 4 (13.8%) 12 (41.4%) 11 (37.9%) 1 (3.4%) 8. The Forum encouraged me to help organize or participate in a peer network of regional agencies to share ideas and work together more effectively. 1 (3.4%) - 2 (6.9%) 1 (3.4%) 9 (31.0%) 15 (51.7%) 1 (3.4%)

E-14 9. As a result of the Forum, I am more aware of the SHRP2 Reliability research and other nationwide systems management and operations research programs. - - - 1 (3.4%) 10 (34.5%) 17 (58.6%) 1 (3.4%) 10. The program met its goal of providing the training and experience necessary for transportation management and operations personnel to effectively facilitate the use of existing transportation infrastructure. 2 (6.9%) - - 4 (13.8%) 12 (41.4%) 10 (34.5%) 1 (3.4%) 11. The pre-study material was a necessary foundation for the classroom portion of the program. - 1 (3.4%) 2 (6.9%) 5 (17.2%) 16 (55.2%) 5 (17.2%) - 12. The exams adequately tested the material you were required to review. 1 (3.4%) - 1 (3.4%) 6 (20.7%) 13 (44.8%) 8 (27.6%) - 13. You received enough direction regarding your specific agency challenge to begin to address it when you return to your workplace. 1 (3.4%) 1 (3.4%) 1 (3.4%) 6 (20.7%) 13 (44.8%) 7 (24.1%) - 14. The accommodations were satisfactory. 1 (3.4%) - - 1 (3.4%) 7 (24.1%) 20 (69.0%) - 15. The learning environment, in terms of its conduciveness to studying and completing assignments, was satisfactory. - - - - 12 (41.4%) 17 (58.6%) - 16. Today you were grouped with the other participants from your state to develop "Agency Implementation Plans". How likely are you to implement the action strategies that you and your peers developed? 2 (6.9%) - 1 (3.4%) 5 (17.2%) 14 (48.3%) 7 (24.1%) - 17. The Technical Tour on Tuesday (Monday) afternoon (morning) added to the value of the Regional Operations Forum. - 1 (3.4%) - 4 (13.8%) 6 (20.7%) 15 (51.7%) 3 (10.3%) 18. The Technical Tour should be offered in future Regional Operations Forums. - - 1 (3.4%) 2 (6.9%) 6 (20.7%) 16 (55.2%) 4 (13.8%) Overall Evaluation o Favorable ratings overall—vast majority of response were “agree” or “strongly agree” for most of the questions o For all items, except one, only 1–2 people disagreed or strongly disagreed o One question had three respondents (10.3%) disagreed or strongly disagree. This was Q11: “The pre-study material was a necessary foundation for the classroom portion of the program.” o Additionally, all respondents answered “yes” when asked if they would recommend this program to others in the future.  Most frequently cited findings from open-ended items (n equals 3 or more). Note: Some repetition because respondent comments were from different open-ended items: o Great networking opportunity to promote collaboration (internally and externally)

E-15 o Organize shorter, more focused sessions o Sessions were too long. Need more breaks. o Have more field trips or group interactions to keep things interesting. Too much sitting at one time. Milwaukee, WI: June 9 to 13, 2014 N/A=No answer/Not applicable, 1 = Strongly disagree, 2 = Disagree, 3 = Neutral, 4 = Agree, 5 = Strongly agree N/A 1 2 3 4 5 Blank Multi 1. The structure and organization of the program was good. - - 1 (3.8%) - 16 (61.5%) 9 (34.6%) - - 2. The materials provided were of high quality. - - 1 (3.8%) 1 (3.8%) 16 (61.5%) 8 (30.8%) - - 3. The information provided to me prior to arriving for the first day of the classroom portion was effective. - 1 (3.8%) 1 (3.8%) 3 (11.5%) 17 (65.4%) 4 (15.4%) - - 4. The Forums contributed to my understanding of the importance and value of a management and operational focus within my organization. - 1 (3.8%) - 1 (3.8%) 11 (42.3%) 13 (50.0%) - - 5. As a result of the Forum I am highly likely to change the way in which I plan, program, and organize for systems management & operations in my organization. - 1 (3.8%) 1 (3.8%) 2 (7.7%) 15 (57.7%) 7 (26.9%) - - 6. I am likely to use the knowledge and skills I learned in the Forum to effectively use a management and operations perspective to identify and implement cost effective solutions to address traffic problems. - 1 (3.8%) - 1 (3.8%) 17 (65.4%) 7 (26.9%) - - 7. As a result of the Forum, I am likely to implement, recommend implementing, or modify the use of performance measures in my transportation network or organization. - 1 (3.8%) - 3 (11.5%) 13 (50.0%) 9 (34.6%) - - 8. The Forum encouraged me to help organize or participate in a peer network of regional agencies to share ideas and work together more effectively. - - 1 (3.8%) 5 (19.2%) 8 (30.8%) 12 (46.2%) - - 9. As a result of the Forum, I am more aware of the SHRP2 Reliability research and other nationwide systems management and operations research programs. - - 2 (7.7%) 3 (11.5%) 13 (50.0%) 8 (30.8%) - -

E-16 10. The program met its goal of providing the training and experience necessary for transportation management and operations personnel to effectively facilitate the use of existing transportation infrastructure. - - 1 (3.8%) 2 (7.7%) 17 (65.4%) 6 (23.1%) - - 11. The pre-study material was a necessary foundation for the classroom portion of the program. 1 (3.8%) - - 7 (26.9%) 13 (50.0%) 5 (19.2%) - - 12. The exams adequately tested the material you were required to review. 2 (7.7%) - 1 (3.8%) 4 (15.4%) 15 (57.7%) 4 (15.4%) - - 13. You received enough direction regarding your specific agency challenge to begin to address it when you return to your workplace. 1 (3.8%) - 1 (3.8%) 4 (15.4%) 17 (65.4%) 3 (11.5%) - - 14. The accommodations were satisfactory. - 1 (3.8%) - - 11 (42.3%) 14 (53.8%) - - 15. The learning environment, in terms of its conduciveness to studying and completing assignments, was satisfactory. - 1 (3.8%) - - 14 (53.8%) 11 (42.3%) - - 16. Today you were grouped with the other participants from your state to develop "Agency Implementation Plans.” How likely are you to implement the action strategies that you and your peers developed? - 1 (3.8%) - 2 (7.7%) 13 (50.0%) 9 (34.6%) 1 (3.8%) - 17. The technical Tour on Tuesday afternoon added to the value of the Regional Operations Forum. 1 (7.7%) - - 4 (15.4%) 12 (46.2%) 8 (30.8%) - - 18. The Technical Tour should be offered in future Regional Operations Forums. 1 (3.8%) - - 3 (11.5%) 11 (42.3%) 11 (42.3%) - - Overall Evaluation o Favorable ratings overall—vast majority of response were “agree” or “strongly agree” for most of the questions o Five of the items received slightly different response frequencies than the others:  Item 8: “The Forum encouraged me to help organize or participate in a peer network of regional agencies to share ideas and worked together more effectively.” 3.8% of respondents disagreed and 19.2% remained neutral. 77% of respondents agreed or strongly agreed.  Item 11: “The pre-study material was a necessary foundation for the classroom portion of the program.” 26.9% of respondents remained neutral while 69.2% agreed or strongly agreed.  Item 12: “The exams adequately tested the material you were required to review.” 3.8% of respondents disagreed and 15.4% remained neutral.

E-17  Item 13: “You received enough direction regarding your specific agency challenge to begin to address it when you return to your workplace.” 3.8% of respondents disagreed while 15.4% remained neutral.  Item 17: “The technical Tour on Tuesday afternoon added to the value of the Regional Operations Forum.” 15.4% of respondents remained neutral. Most frequently cited findings from open-ended items (n equals 3 or more). Note: Some repetition because respondent comments were from different open-ended items:  Most participants suggested that the sessions or days be shorter. They found the sessions hard to sit through and the week to be very long.  The PowerPoint presentations sometimes did not match the sessions and made it difficult to follow along.  The interaction with other states and the peer to peer interactions were the best part about the sessions  More so than the presentations, the interactive discussions during the sessions were helpful Follow-Up Survey Summary Sent out July 3, 2014

E-18  Summary o Favorable ratings overall- 88% of attendees agreed or strongly agreed o No participants disagreed or strongly disagreed but 10.67% remained neutral and 1.33% responded “not applicable.”  Selected open-ended responses: o “I was thinking about moving away from operations in my career and the ROF re- energized me.” o “I found the interaction with other states and learning about states best practices were useful.” o “Great presentations and valuable audience participation.” o “We have had some internal discussions on possible organization changes to reflect operations.” o “I came back from the forum with an eagerness to implement ideas we were exposed to, but daily demands and technological limitations have required some initiatives to be tabled for now. Hopefully, we’ll be able to pursue them again in the near future…”  Summary o Favorable ratings overall  82% of attendees agreed or strongly agreed with this item  Less than 3% of attendees disagreed or strongly disagreed

E-19  Selected open ended responses o “It has changed the way that I think about how we impact traffic and has caused me to think about ways to lessen that impact. I have asked my employees to be thinking about the manner in which we impact our customers and how we can do better.” o “Yes, it was good. I learned some new ways of thinking and I learned some new tools.” o “The ROF has been helpful and effective in developing a TSM&O Implementation Plan as a result of the SHRP2 Organizing for Reliability CMM workshops.” o “It let us know how we compare to other DOTs and what needs to be completed to get to the next level of maturity.”  Summary o Responses were varied to this survey item  60% of participants agreed or strongly agreed  About 30% of participants were neutral  Less than 3% of participants disagreed  Selected open-ended responses: o “I have a hard time keeping up with these when there is a barrage of them coming in. I agree that the information is good and it’s a good network to tap into. I wonder if another method might be more effective.”

E-20 o “I think this listerv is a good tool for communication—but it evolved into nearly that level of communication. That volume of messaging via email turns into noise—which I ignore or mass delete on occasion.” o “Been using it since my 2012 full Ops Academy class.”  Summary o Almost all (97%) participants responded “yes.”  Selected open-ended responses: o “Great chance to network with other agencies and share information.” o “Networking and learning from peers (and in my case, higher-level managers than myself) was the most valuable part of the whole experience; and for the majority of the other aspects of the ROF (lectures, group exercises, etc.), most proved valuable to me in my job.” o “Yes, my eyes were really opened in terms of what other states are doing and how they are doing things. I have learned a different perspective of looking at issues like Operations. I certainly learned a lot from others in my class. That was very good.” o “Good exposure to field operations.” o “The interaction of the participants was as valuable as the training. It allowed states to share information and make contacts in adjacent states. This will be a great resource for me in the future.” o “It allowed me to take a step back to see how our state approaches TSPO and to learn from other states as well. The learning material was excellent, but the most

E-21 valuable piece of the experience was the networking and connection with other states/agencies.” o “Strengthened regional professional network, provided insights into the national state of practice.” o “The best part in my opinion was the opportunity to share experiences with peers from other states and benefit from lessons learned.”  Summary o The majority of participants responded “yes” (83.56%), while only 16.44% responded “no.”  Selected open-ended responses o “Yes—we had a FHWA self-assessment workshop here and the background we had at the ROF helped make me an effective advocate for operations.” o “I will be managing a multistate corridor operations effort. The concepts I learned will help provide a link between planning and operations and identify efforts that we can do across state lines.” o “It has been helpful to start creating new guidelines for integrating operations better into planning, reviewing a new ICM location with understanding of what ICM is and what it requires to be effective and applying the concepts to creation of new reliability PM’s.” o “We are applying many of the concepts, including: Systems Engineering, Capability Maturity Model, TSM&O, etc.” o “We now take a more pro-active approach to how we conduct ourselves on the highway performing maintenance work and especially maintenance work that closes the road for short durations. We are already planning to develop performance measures for gauging how well we are doing in several areas. We are starting to develop contingency traffic plans for hotspots and known problem areas.”

E-22  Summary o Responses to this question were almost evenly split with 47.95% of attendees saying “yes” and 52.05% saying “no.”  Selected open-ended responses: o “TIM training throughout the corridors. Coalition building along the corridors.” o “Yes, again our primary uses and insights from others have been useful in how we want our ATMS and organization to run. Our largest use of devices is now the Systems Engineering process.” o “We are now pursuing statewide CMM process with FHWA. Our participation in the ROF allowed us to be part of the larger ‘official’ process and our strategies are being rolled up into it.” o “Moving forward with TIM training in part because of the forum” o Special note: several participant comments explained why they answered “no.” Attendees referenced lack of time and being busy with other work as reasons for not being able to implement action plans.

E-23  Summary o Almost all respondents said that they would participate in a follow-up workshop.  Selected open-ended responses: o “This would be great.” o “If it is only one day. I would be interested to listen to changes or implementation that others have made.” o “Please have it soon!” “That would be beneficial.” o “I think this would be a very useful experience and good for the program.” o “Absolutely…and would encourage such a workshop…” o “This would be a great opportunity for peer discussion.”

F-1 APPENDIX F Sustainability Options

F-2 I. Background Regional operations forums (ROF) were expected to advance transportation systems management and operations (TSM&O) and serve as a platform for mainstreaming SHRP 2 Reliability research. The $1,000,000 SHRP 2 L36 project funded the delivery of five ROFs around the United States. The effort involved in delivery included the development of the curriculum and course material for four full days of training, selection of cost-effective venues in a convenient location, coordination with agencies to identify participants and field visits, securing local speakers, coordination with participants and monitoring of prestudy activities, reimbursement for participant travel costs, and much more. ROFs were offered in the following five locations: Nashville, Tennessee; Seattle, Washington; Phoenix, Arizona; Concord, New Hampshire; and Milwaukee, Wisconsin. One additional ROF was delivered by the research team in Orange, CA that was 100% funded by Caltrans. In order to maximize participation by the states and properly pilot test the ROFs, all costs associated with development and delivery were completely funded by the SHRP 2 L36 project, including travel costs of participants. The project cost for delivery of one ROF was $135,000. According to the contact at Caltrans, the ROF cost them $67,205. However, this latter amount does NOT include travel costs of the participants, staff time by Caltrans to coordinate and manage the ROF, nor coordination with Caltrans and the participants that was conducted by the SHRP 2 L36 research team. It should also be noted that since Caltrans had a $5,000 limit on individual sole source contracts, the facilitator role had to be split by two people and travel time to the ROF had to be donated by the firm. The ROF program was conceived to continue as a sustainable program once the initial pilot offerings were completed: continuing to reach out to new individuals to be trained as well as continuing to train those who already attended the ROFs. The program begins as a forum for a group of +/- 30 participants from states within a region who participate in an initial face-to-face session, and then continue their association and interaction through related follow-up activities involving a mix of web-based and (perhaps) face-to-face activities. The initial ROF pilots reached approximately 187 participants (this “Round One” number includes California participants). In keeping with the initial concept, some level of follow-up activities is expected to be provided in the future. In addition, a new cycle of ROFs and follow- up activities (“Round Two”) is presumed in order to reach beyond the initial participants, in both states covered by the pilot sessions, and in states that have not yet participated. Extending the program beyond the initial five pilot sessions to other states and new participants, plus follow-up activities from the initial ROF participants, introduces sustainability issues. The organization and management of the ROF activity as a sustainable program needs to be considered in its own right, as well as in the context of the development of the System Management and Operations Center of Excellence and its knowledge transfer functions and other American Association of State Highway and Transportation Officials (AASHTO) Subcommittee on Systems Operation and Management and Federal Highway Administration (FHWA) initiatives. II. Sustainable Options This SHRP 2 L36 project includes the development of the initial ROF curriculum, delivery of five pilot sessions, and the examination of the sustainability of ROFs beyond SHRP 2 program support. Based upon the knowledge and experience to date of the L36 research team, this paper therefore addresses the key issues related to sustainability. The primary focus is to identify

F-3 potential options for the sustainability of the ROF program as well as documenting the costs associated with delivering the ROFs and the trade-offs related to knowledge transfer effectiveness of the different options. Both the initial forum and the follow-up activities may be addressed through varying options involving alternative mixes and the extent of knowledge transfer methods such as face-to-face and web-based activities (e.g., webinars and background reading). Cost estimates based on experience to date are presented for the application of the knowledge transfer options appropriate to both an initial forum and follow-up activities. In addition, key issues are addressed related to the appropriateness of knowledge transfer methods and related sponsorship, funding, management, and contracting issues. Decisions regarding the next stage (anything beyond the initial five pilot ROFs) should be made with an understanding of the inherent trade-offs among the various options. The material below succinctly outlines these trade-offs for the purpose of further discussion and exploration with stakeholder organizations. A. Key Objectives and Lessons Learned for Potential ROF Knowledge Transfer Methods Options for the scope, scale, funding, and contracting of future ROFs should be discussed with a clear understanding of the objectives of the ROFs. It is best to review the original objectives as documented through the January 2012 “Workshop Findings and Recommendations” report, prepared by Phil Tarnoff, for the SHRP 2 L36 Technical Expert Task Group. The report included the following key objectives as developed during Task Group discussions: 1. “Organizational Objective: To mainstream operations by moving state of the art closer to state of the practice as transportation organizations transition from capacity building to operations, and to strengthen the SO&M state and regional programs.” How This Objective Has Been Accomplished Through the Pilot ROFs and Lessons Learned  This objective has been accommodated through a multiday curriculum with specific topic modules developed to provide the needed range of substantive knowledge transfer. SHRP 2 research products were included throughout the curriculum.  The material presented and supporting dialogue reflects knowledge of best practices nationwide – accomplished through the involvement of acknowledged national experts.  The TSM&O Capability Maturity Model product of SHRP 2 research has been valuable to participants in assessing the current state of play in their states prior to attending the ROF and then identifying key action strategies that they can tackle to improve the TSM&O capability of their agency. 2. “Personal/Community Objective: To develop a community of practice through the development of a peer network, and to train the next generation of leadership.” How This Objective Has Been Accomplished Through the Pilot ROFs and Lessons Learned  This objective has been accommodated though a multiday ROF format that provides for several days of both formal and informal interaction built around topic module discussion, team exercises, field trip and informal contact, and discussion at breaks and meals.  All ROF participants have been added into the Operations Academy listserv to facilitate the exchange of information among peers.

F-4  From discussion with attendees and the evaluation process, it is evident that ROF participants want to continue the dialogue with their peers. It is important that there is some mechanism for the exchange of information and knowledge transfer to continue for the ROF pilot participants as well as adding other peers to the mix. These activities should take advantage of the full range of face-to-face and web-based options. 3. “Educational Objective: To transfer SO&M knowledge including the dissemination of SHRP 2 solutions, and develop a business case for SO&M programs and activities.” How This Objective Has Been Accomplished Through the Pilot ROFs and Lessons Learned  This objective has been met by a multiday format that provides for coverage of key topic modules. Topics were decided through discussions with the TETG and state DOT representatives.  Experience at the initial ROFs suggests there is value in augmenting the ROF learning experience and responding to changing contexts and issues through a sequence of shorter topic-focused interchanges which may be accommodated via web methods or brief topic- specific meetings/workshops.  The business case for TSM&O has been built up over the course of several days of technical presentations and discussion and reinforced by peer exchange of state experiences 4. “Additional Objective: To create a regional network of academies that complements the National Operations Academy, but with the potential of reaching a broader audience. “ Lessons learned  ROF participants to date have tended to be key technical staff at the activity management level – distinct from Operations Academy attendees that typically include agency middle management at the program level. Instruction material has been tailored throughout to better fit the span of control and understanding of the participants.  The initial round of five ROFs and related follow-on activities will reach about 150 professionals. This cycle needs to be repeated in each region in order to reach even more key staff. As the first round of ROFs moves into follow-up activities, a second round of ROFs plus follow-on activities should be initiated to reach additional key staff and new states.  The follow-on activities of the initial and subsequent ROFs should be folded together to ensure overall coordination of effort and knowledge transfer. B. ROF Delivery Options (Knowledge Transfer) – Initial ROFs The optimal ROF program should include both the delivery of an ROF plus a set of related follow-up activities, which should continue on a multiyear basis. There are three main options for how future ROFs could be delivered. The amount of face time accommodated both between facilitators/instructors and the participants is the most defining feature of these options. The three options cover the range of possibilities and any of the options could be tailored further.

F-5 It should be noted that it might not be necessary to select just one option. ROFs could be delivered differently in different places depending on factors such as the ability to authorize and fund travel, the objectives of the sponsoring agency, the desire to learn from peers in other states, and how difficult it is to conduct each option relative to the contract/payment and funding options presented later. Delivery Option 1 – Continue ROFs in their Current Format The first option is to continue conducting the ROFs as per the five pilot sessions. This option provides for the most face time, not just in the formal instructor-led sessions, but also during the group exercises and during breaks, meals, and free time where experiences often continue to be shared. As a result, this option would be the most effective way for the participants to learn, especially from one another. In addition, the appropriate level of face-to-face communication at the ROFs – both formal and informal builds interpersonal relationships that are essential to support continuing knowledge transfer. The strong relationships developed tend to carry forward and increase the probability of follow-up interactions including use of listserv and follow-up forums. Use of webinars or even video conferencing in lieu of a common physical location would not facilitate the same level of interaction as having people together at the same time and in the same place (and without distractions) discussing subjects of mutual interest and is less likely to result in a sustainable community of practice among peer states. In addition, regarding knowledge transfer, in the case of webinars or online learning materials, there is unfortunately no good way of ensuring that participants actually focus on the material being presented and there are limited opportunities for instructors to tailor presentations and discussion in real time. The cost of this option is the highest because of the travel costs, lodging, meals, and facility rental; however, the costs associated with facilitator/instructor and participant time are similar among all of the options since it will cost roughly the same to prepare and conduct a number of webinar sessions over a longer period of time as it would to do all the sessions face- to-face during a 1-week period. These expenses could be borne by different parties but this will be discussed later. The ROF pilot experience has reinforced the value of conducting the initial activity in a face-to-face format at a sufficient level to achieve the initial objectives related to state of practice knowledge transfer in key areas plus the development of a community of practice (see discussion above). The number of days spent face-to-face versus webinar or online learning should be discussed based on a variety of factors including costs and time away from the office. It should be noted that the approach to the initial five pilot sessions includes significant web-based activities including the use of online resources such as a learning management system, course materials, and the existing SHRP 2 L17 Knowledge Transfer website (www.tsmoinfo.org). As part of ROF meeting preparation process (prestudy), participants used the web to:  Review program and administrative materials  Access background reading  Access course presentation material  Assess their agency level of capability maturity using the one minute AASHTO assessment tool (http://www.aashtotsmoguidance.org/)  Take online exams  Download all session presentations that will be presented during the ROF

F-6 In order to save costs during the ROF delivery, some speaker sessions were either taped and participants shown the video or a local presenter was used to deliver the presentation that was developed by the research team. Delivery Option 2 – Shorter In-person ROF Plus Webinars and Other Web-based Activities This option is a compromise between the Option 1 and 3 (see below). It would involve a series of short webinar/video presentations, and/or web-based activities over a specified period of time followed by a 1- to 3-day workshop to share experiences, conduct group exercises; discussions from activities conducted prior to the face-to-face meeting, and/or develop agency implementation plans. There are a variety of ways to construct this option. It could be done differently in different places depending on the needs/desires/availabilities of each location. It would accommodate some interaction between facilitators/instructors and participants, and among participants. The travel and facility costs would be less than the first option but it would not necessarily be proportional since session content modifications and travel to the meeting site will cost the same regardless of whether it is for 1 day or multiple days. Delivery Option 3 – All Web-based The option includes no in-person interaction at all. The ROF would be completely accomplished through a series of short webinars, video presentations, online courses or other web-based activities over a specified period of time. Interaction with each session instructor would be via webinar, teleconference, or online forum. This would be the lowest cost option since travel and facility rental costs are completely eliminated; however, there will still be costs associated with delivery of any web-based materials. Video sessions could be developed to avoid paying instructors to deliver material in person at workshops; however, video recordings will become dated and funds will be needed to periodically update them. It should be noted that videos might be less effective at holding participants’ attention and fostering discussion. There would still be interaction among the participants via webinars and forum posts, but the time would be limited to faceless web dialogue and both the group discussion and informal interaction to develop a sense of community – which have characterized the pilot ROFs – would be absent. These are important considerations since the in-person sessions during the pilot ROFs include a great deal of group discussion. The participants in the ROFs to date have touted the networking aspect as the most beneficial aspect. With this option, the group exercises would need to be eliminated because it will be difficult to organize and conduct these effectively without face-to-face interaction. All of the informal break/meal/free time interaction opportunities would also be lost. In addition, participant continuity of attendance will be challenging over a series of sessions and participants are much more likely to be distracted while on a webinar – multitasking while supposedly “listening” to the discussions. This option could prove valuable for states that want to participate in an ROF but for whom out- of-state travel cannot be authorized. C. Follow-up Activities Delivery Options As discussed earlier, follow-up activities need to be included as an integral part of the ROF program in order to accommodate emerging issues, evolving state of the practice and peer-to- peer interchange.

F-7 1. Current Approach to Follow up While the first round of ROFs in not complete, some follow-up activities have already taken place or are planned. These are all web based. Below is a list of follow-up activities that have been (or will be) conducted following the initial ROFs.  Listserv (done) – All ROF pilot participants have been added to the existing Operations Academy listserv. Initially, each ROF had its own listserv but there was not any interaction in the group. Therefore, the research team added the participants to the Operations Academy listserv since it is one of the most active and engaging listservs we have seen. ROF attendees have been actively participating in the information exchange.  Group on Facebook or LinkedIn (done) – Two of the groups expressed interest in developing a private group on Facebook or LinkedIn. One of the participants from the ROF in Concord, NH (April 28-May 2) has already set up a LinkedIn page, which the research team was invited to participate in as well. The research team will monitor it to see its effectiveness.  Follow-up on Agency Implementation Plans (to do) – ROF participants developed an agency implementation plan that was presented to the other participants on the last day of the ROF. The research team plans to follow up with the participants from the first three ROFs with an online survey to see how they are doing in implementing their plans (6 months following the ROF). The project will be over before the team is able to follow up with the last two ROFs.  Refereed Dialogue – Consideration has been given as to how to manage live conversations on specific topics with the relevant interested participants 2. Follow-up Delivery Options The same three ranges of options presented in Section B above can be considered for the appropriate mix of follow-up activities beyond the initial ROF. DELIVERY OPTION 1 – IN-PERSON ACTIVITIES Several ROF locations have requested that the same group of participants get back together on an annual basis for information exchange. If this activity actually happens, the research team suggests that there also should be follow-up on the agency implementation plans as well as some additional knowledge transfer activities planned. DELIVERY OPTION 2 – MIX OF IN-PERSON AND WEB-BASED ACTIVITIES This option would consist of a mix of 1- to 2-day face-to-face meetings/workshops along with some web-based activities such as webinars, web-based course materials, videos, web-based knowledge site, etc. This would allow the knowledge transfer process to be more continual rather than just once a year or once every other year. Some of the follow-up activities could be conducted for all ROF alumni at once while other activities could be designed for participants from specific ROFs that were conducted. DELIVERY OPTION 3 – ALL WEB-BASED ACTIVITIES This option would consist solely of web-based activities such as webinars, web-based course materials, videos, web-based knowledge site, etc. This would be the most inexpensive option to continue the knowledge transfer process and would allow for it to be continual rather than just once a year or once every other year. As discussed in Option 2 above, some of the follow-up

F-8 activities could be conducted for all ROF alumni at once while other activities could be designed for participants from specific ROFs that were conducted. Table F.1 provides a summary of the pros and cons of the delivery options discussed above for both the ROF delivery options and follow-up activities.

F-9 o Table F.1. Summary of the Pros and Cons of the Delivery Options Delivery Option 1 Continue ROFs in their Current Format Delivery Option 2 Shorter In-person ROF plus Webinars and Other Web-based Activities Delivery Option 3 All Web-based Pro Con Pro Con Pro Con Initial Forum  Sufficient time to cover key topics  Most conducive to learning – provides for the most interaction (with instructors and among participants)  Best way of ensuring participant focus  Content can be adjusted in real time during ROF to deal with key issues that emerge  Higher cost due to per diem and facility rental  Requires travel authorization and time commitment of 5 consecutive days  Reduced per diem and facility rental cost  Doesn’t require commitment of 5 consecutive days  Potentially more participants and/or a higher level with a shorted number of days out of the office  Less face-to-face time  Requires more complex schedule of participant time – continuity of participation will be more challenging  Less flexible in terms of tailoring material to specific issues that emerge through discussions  Costs associated with creating and hosting webinars  No per diem or facility rental cost  No extended (multiple-day) time commitment  Avoids need to authorize travel  No face-to-face time  Continuity of participation will be even more challenging  Difficult to maintain focus on key issues that emerge from session to session  Includes higher costs of creating and hosting webinars (including instruction material) or videos Delivery Option 1 In-person Activities Delivery Option 2 Mix of In-person and Web-based Activities Delivery Option 3 All Web-based Activities Follow- up Activities  Dedicated face- to-face time for discussion of specific topics and/or group exercises  Strengthening of community network  Higher cost due to facility rental, meals, and lodging  Requires time commitment and travel authorization  Dedicated face- to-face time for discussion of specific topics and/or group exercises  Strengthening of community network  Potentially less cost than Option 1 for facility rental, meals, and lodging  Continuity of participation would be challenging. Many would not participate in web activities – too easy to do other work in office  Costs for creating and hosting webinars and/or videos  Less in-person time for participants to network  Knowledge transfer is conducted in smaller segments so not out of the office for an extended period of time  No per diem or facility rental costs  Avoids need to authorize travel  Continuity of participation would be challenging. Many would not participate – too easy to do other work in office  Costs for creating and hosting webinars and/or videos and other web activities  No in-person time for participants to network

F-10 III. ROF Labor Hours and Costs The cost information presented in Table F.2 is the average cost of delivering one pilot ROF for 30 participants as experienced from managing this project. It should be noted that the development of the ROF content is not included in this cost because for a sustainability discussion, the development costs have already been incurred and sustainability addresses modification of existing content not the development of new content. Table F.2. Pilot Costs for 30 Attendees Labor Hours Costs Instructors (includes content modifications but not travel time) 74 $17,382 Facility and Travel Coordination 115 $8,600 Facilitator (hours don’t include travel time) 40 $10,280 External Evaluation 51 $8,058 Administrative Coordinator 120 $8,100 Travel/Meeting Travel (staff and participants) $8,705 Shuttle Bus $795 Logistics $31,057 Materials/Supplies Meeting Expenses $2,432 Subtotal $95,409 UMD Overhead (subsidized) $39,302 Total $134,711 Below is a description of what each line item includes:  Instructors – Includes time to modify content for the specific ROF along with time to deliver the sessions during the ROF.  Facility and Travel Coordination – This cost is for a specific firm to research acceptable facilities to hold the ROFs; contract for the meeting space, lodging, and meals; make reservations for participants with facility; secure bus for technical tour; prepare name tags and tent cards; attend the ROF to ensure all logistics run as desired; and process and pay all travel for participants.  Facilitator – One person to facilitate the ROF for all days. The facilitator is also the instructor for several sessions.  External Evaluation – The project required that an external evaluator assess the effectiveness of the ROF sessions, instructors, speakers, facility, etc., and determine any changes that should be made to the program. Special Note: This cost could be reduced for future ROFs. All evaluation forms have been developed and the Administrative Coordinator could make sure that the evaluation forms are distributed, collected, and summarized.  Administrative Coordinator – Labor costs for coordinating with a host agency and participants along with setting up and monitoring the prestudy activities.  Travel – Participants, instructors, and facilitator.  Shuttle Bus – For technical tour.

F-11  Logistics Costs - Lodging, facility rental, meals, AV, breaks, breakout rooms.  Meeting Expenses - Books, binders, copies, postage, certificates, and any other miscellaneous meeting costs. Obviously, the costs for a 5-day forum are higher than a shorter 2- or 3-day forum. However, a 3-day forum with web activities that occurred prior to the ROF (additional web activities other than the existing prestudy that participants already do ahead of time), there would be savings on the logistics costs, including fewer days for lodging, facility rental, and per diem. There would also be some savings from instructor time to deliver sessions in person at the ROF since some instructors would not have to travel. However, time would still be incurred for instructors to deliver a knowledge transfer session via webinar or for them to tape a video session. It should be noted that the time savings from instructor travel does not appear as significant as one might think because generally the instructors are not billing all the travel time necessary to physically attend the ROFs in person. The marginal cost differences, therefore, must be weighed against the other objectives of community building, more effective learning through face-to-face interaction between instructors/participants and among participants, and the ability to tailor knowledge transfer in real time in response to participant interactions and discussions. Based on the ROF costs that have been incurred to date, an estimated cost of delivering a 3-day ROF with web-based pre-activities is shown in Table F.3. Please note that this is just an estimate. Table F.3. ROF Estimated Costs for 3-day Program with Web-based Pre-Activities Labor Hours Costs Instructors (includes content modifications but not travel time) 44 $8,700 Facility & Travel Coordination* 99 $7,550 Facilitator (hours don’t include travel time) 24 $6,168 Evaluation Summary (no external) 15 $600 Administrative Coordinator 120 $8,100 Travel/Meeting Travel (staff & Participants) $8,105 Shuttle Bus $795 Logistics $18,634 Materials/Supplies Meeting Expenses $2,432 Video Production/editing (2 @ $2,594) $5,188 Webinars (5 @ $2,275) $11,375 Blended web-based Course** (1 @ $2,275) $2,275 Subtotal $79,922 UMD Overhead $41,560 Total $121,482 *This cost could be eliminated if the host agency were to perform all these functions – see description above. **This assumes that the web-based course is already available and ready for delivery.

F-12 In order to develop an estimate of the costs for everything to be delivered via the web, some decisions had to be made regarding which sessions would be offered in what format. In Table F.4, the research team assigned all the sessions to a format in order to develop costs. This in no way means that the formats chosen are the only format that this session could be offered; it was done specifically to develop an estimate of costs. Table F.4. ROF Estimated Costs for All Web-based Pre-Activities Cost of Delivering Sessions Format Costs Overview of Operations Blended Web-based Course* $2,275 Capability Maturity Model (CMM) Self Evaluations Webinar $2,275 Planning and Programming for Operations Blended Web-based Course* $2,275 Performance Measurement Blended Web-based Course* $2,275 Facilitating Goods Movement through Operations Video $2,594 Systems Engineering Blended Web-based Course* $2,275 Traffic Incident Management, Emergency Operations and Planned Special Events Webinar $2,275 Safety and Operations Webinar $2,275 Road Weather Webinar $2,275 Traveler Information and Operations Webinar $2,275 Managing a Corridor (includes ICM, ATM, Managed Lanes) Webinar $2,275 How to Organize for Operations Blended Web-based Course* $2,275 Communicating the Value of Operations Webinar $2,275 Connected Vehicles and the Future of Transportation Video $2,594 Other Costs Travel (instructors for video production) $500 Session Content Updates (14 sessions @ 6 hours each) $22,512 Administrative Coordinator (14 events @ 10 hours each) $9,450 Subtotal $64,950 UMD Overhead $33,774 Total $98,724 * This assumes that the web-based course is already available and ready for delivery. IV. Management and Funding Options Continued funding support for the ROFs will be a fundamental key to sustainability of the program. There is a spectrum of funding models and options that could potentially enable a sustainable multiyear ROF program. The funding options could be applied to any of the delivery methods described above. They are: Option 1 - Program Totally Managed/Funded by Federal Agency or Industry Organization In this option, a central entity such as AASHTO, FHWA, ITE, a university, or the new Operations Center of Excellence would manage the ROF program and provide full funding for all initial ROF and follow-up activities (could or could not including participant travel and per diem costs). This option is how the five ROF pilots have been offered with FHWA funding made available through SHRP 2.

F-13 In this model, all ROFs would be channeled through a single contract for maximum administrative efficiency and quality control. The central entity would develop an RFP (scope, budget and schedule) for delivering ROF activities – both for the initial ROFs and follow-up activities – and be the contract administrative agency. Each region requesting an ROF would work through the central entity, which would pay for all of the costs associated with delivery. Currently, FHWA has set aside $1,000,000 to initiate the sustainability of the ROF program. This funding is not intended to be renewed on an annual or biannual basis. It is just a one-time funding allocation to keep the ROF momentum going now that the initial ROF pilots have been completed. That funding will allow time for the development of other potential funding sources to be identified. Advantages:  There would be no cost to participating states so that they would be encouraged to participate. With the leadership interest in TSM&O that varies among states, it would allow for broader participation.  Allows the contractor – working with a stakeholder advisory committee – to provide national experts in the relevant topic areas. Through experience with multiple ROFs, these experts would be able to continually improve/tailor the instructor material, forum activities, and procedures. Disadvantages:  The need to secure a sustainable source of funding from outside the participating states to support the program on a continuing basis.  The central entity and its staff are not the direct beneficiaries of the ROFs and would have to continually justify the expenditure of supporting the ROFs for participants from primarily state and local government agencies. Option 2 - Program Totally Managed/Funded by Participating States This option allows a host agency to self-fund and offers an ROF to its own agency employees and /or partner/regional agencies. Since the ROF is completely funded by the agency itself, it would be solely up to the agency to decide who participates. The agency pays for and could utilize the resources, content, and material that has already been developed under the L36 project, but is responsible for organizing the multistate participation, operating expenses (including contracts with instructors and facilitator), coordinating all logistics, coordinating with participants, and complete delivery of the ROF. As with any other funding option, content could be modified and expanded by the agency’s contracted instructors for local experience and needs. An alternative to the host agency doing all the coordinating themselves, a host agency could decide to pay a central entity (AASHTO, FHWA, ITE, or the new Center of Excellence) to do all the coordination activities for them. There would be one contract between the host agency and the central entity, which would handle all activities associated with the development and delivery of the ROF. The host/funding agency would only need to provide input on participants, technical tour, and any local speakers. This method of delivering an ROF was conducted May 12-16, 2014 with Caltrans funding and hosting an ROF. The agency contracted with each instructor individually, handled all coordination for meeting space, identified and coordinated with participants, handled logistics for the technical tour, etc. For some sessions, the content was adapted from that used in the multistate ROFs. The research team had to participate in some activities such as solidifying the

Regional Operations Forums for Advancing Systems Operations, Management, and Reliability Get This Book
×
MyNAP members save 10% online.
Login or Register to save!
Download Free PDF

TRB’s second Strategic Highway Research Program (SHRP 2) Report: Regional Operations Forums for Advancing Systems Operations, Management, and Reliability documents the piloting of the Transportation System Management & Operations (TSM&O) curriculum through a Regional Operations Forum (ROF) concept. In addition to the report, the project also produced a set of PowerPoint presentations to assist with training.

Disclaimer: This software is offered as is, without warranty or promise of support of any kind either expressed or implied. Under no circumstance will the National Academy of Sciences or the Transportation Research Board (collectively "TRB") be liable for any loss or damage caused by the installation or operation of this product. TRB makes no representation or warranty of any kind, expressed or implied, in fact or in law, including without limitation, the warranty of merchantability or the warranty of fitness for a particular purpose, and shall not in any case be liable for any consequential or special damages.

  1. ×

    Welcome to OpenBook!

    You're looking at OpenBook, NAP.edu's online reading room since 1999. Based on feedback from you, our users, we've made some improvements that make it easier than ever to read thousands of publications on our website.

    Do you want to take a quick tour of the OpenBook's features?

    No Thanks Take a Tour »
  2. ×

    Show this book's table of contents, where you can jump to any chapter by name.

    « Back Next »
  3. ×

    ...or use these buttons to go back to the previous chapter or skip to the next one.

    « Back Next »
  4. ×

    Jump up to the previous page or down to the next one. Also, you can type in a page number and press Enter to go directly to that page in the book.

    « Back Next »
  5. ×

    To search the entire text of this book, type in your search term here and press Enter.

    « Back Next »
  6. ×

    Share a link to this book page on your preferred social network or via email.

    « Back Next »
  7. ×

    View our suggested citation for this chapter.

    « Back Next »
  8. ×

    Ready to take your reading offline? Click here to buy this book in print or download it as a free PDF, if available.

    « Back Next »
Stay Connected!