Understanding and Communicating Reliability of Crash Prediction Models (2021)

Below is the uncorrected machine-read text of this chapter, intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text of each book. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.

NCHRP Web-Only Document 303: Understanding and Communicating Reliability of Crash Prediction Models Raghavan Srinivasan Bo Lan Caroline Mozingo The University of North Carolina at Chapel Hill Highway Safety Research Center Chapel Hill, NC James Bonneson Kittelson and Associates, Inc. Portland, OR Craig Lyon Bhagwant Persaud Persaud and Lyon, Inc. Ottawa, ON Geni Bahar NAVIGATS Inc. Toronto, ON Conduct of Research Report for NCHRP Project 17-78 Submitted August 2020 NATIONAL COOPERATIVE HIGHWAY RESEARCH PROGRAM Systematic, well-designed, and implementable research is the most effective way to solve many problems facing state departments of transportation (DOTs) administrators and engineers. Often, highway problems are of local or regional interest and can best be studied by state DOTs individually or in cooperation with their state universities and others. However, the accelerating growth of highway transportation results in increasingly complex problems of wide interest to highway authorities. These problems are best studied through a coordinated program of cooperative research. Recognizing this need, the leadership of the American Association of State Highway and Transportation Officials (AASHTO) in 1962 initiated an objective national highway research program using modern scientific techniques—the National Cooperative Highway Research Program (NCHRP). NCHRP is supported on a continuing basis by funds from participating member states of AASHTO and receives the full cooperation and support of the Federal Highway Administration (FHWA), United States Department of Transportation, under Agreement No. 693JJ31950003. COPYRIGHT INFORMATION Authors herein are responsible for the authenticity of their materials and for obtaining written permissions from publishers or persons who own the copyright to any previously published or copyrighted material used herein. Cooperative Research Programs (CRP) grants permission to reproduce material in this publication for classroom and not-for-profit purposes. Permission is given with the understanding that none of the material will be used to imply TRB, AASHTO, FAA, FHWA, FTA, GHSA, NHTSA, or TDC endorsement of a particular product, method, or practice. It is expected that those reproducing the material in this document for educational and not-for-profit uses will give appropriate acknowledgment of the source of any reprinted or reproduced material. For other uses of the material, request permission from CRP. DISCLAIMER The opinions and conclusions expressed or implied in this report are those of the researchers who performed the research. They are not necessarily those of the Transportation Research Board; the National Academies of Sciences, Engineering, and Medicine; the FHWA; or the program sponsors. The information contained in this document was taken directly from the submission of the author(s). This material has not been edited by TRB.

The National Academy of Sciences was established in 1863 by an Act of Congress, signed by President Lincoln, as a private, non- governmental institution to advise the nation on issues related to science and technology. Members are elected by their peers for outstanding contributions to research. Dr. Marcia McNutt is president. The National Academy of Engineering was established in 1964 under the charter of the National Academy of Sciences to bring the practices of engineering to advising the nation. Members are elected by their peers for extraordinary contributions to engineering. Dr. John L. Anderson is president. The National Academy of Medicine (formerly the Institute of Medicine) was established in 1970 under the charter of the National Academy of Sciences to advise the nation on medical and health issues. Members are elected by their peers for distinguished contributions to medicine and health. Dr. Victor J. Dzau is president. The three Academies work together as the National Academies of Sciences, Engineering, and Medicine to provide independent, objective analysis and advice to the nation and conduct other activities to solve complex problems and inform public policy decisions. The National Academies also encourage education and research, recognize outstanding contributions to knowledge, and increase public understanding in matters of science, engineering, and medicine. Learn more about the National Academies of Sciences, Engineering, and Medicine at www.nationalacademies.org. The Transportation Research Board is one of seven major programs of the National Academies of Sciences, Engineering, and Medicine. The mission of the Transportation Research Board is to provide leadership in transportation improvements and innovation through trusted, timely, impartial, and evidence-based information exchange, research, and advice regarding all modes of transportation. The Board’s varied activities annually engage about 8,000 engineers, scientists, and other transportation researchers and practitioners from the public and private sectors and academia, all of whom contribute their expertise in the public interest. The program is supported by state transportation departments, federal agencies including the component administrations of the U.S. Department of Transportation, and other organizations and individuals interested in the development of transportation. Learn more about the Transportation Research Board at www.TRB.org.

C O O P E R A T I  V E  R E S E A R  C H  P R O G R A M S  CRP STAFF FOR NCHRP WEB-ONLY DOCUMENT 303 Christopher J. Hedges, Director, Cooperative Research Programs Lori L. Sundstrom, Deputy Director, Cooperative Research Programs Waseem Dekelbab, Associate Program Manager, National Cooperative Highway Research Program David Jared, Senior Program Officer Natalie Barnes, Director of Publications Heather DiAngelis, Associate Director of Publications Kami Cabral, Editor Kathleen Mion, Senior Editorial Assistant NCHRP PROJECT 17-78 PANEL Field of Traffic—Area of Safety Tim Harmon, Holly Springs, NC (Chair) Kelly K. Campbell, Idaho Transportation Department, Boise, ID Andrew H. Ceifetz, WSP, Walled Lake, MI Brian Hovanec, Jackson, MS Rahul Jain, District Department of Transportation, Washington, D.C. Dean C. Kanitz, Michigan Department of Transportation, Lansing, MI Andrew G. Kaplan, Port Authority of New York and New Jersey, Hoboken, NJ John McFadden, FHWA Liaison Kelly K. Hardy, AASHTO Liaison AUTHOR ACKNOWLEDGMENTS The authors acknowledge the significant contributions of the agencies and their staffs that contributed data and insights that enhanced this report and the associated guide published as NCHRP Research Report 983: Reliability of Crash Prediction Models: A Guide for Quantifying and Improving the Reliability of Model Results.

iv Contents Summary ............................................................................................................................................. 1  Chapter 1. Introduction and Background ........................................................................................... 3  Chapter 2. Survey of Practitioners .................................................................................................... 15  Chapter 3. Procedures for Quantifying the Reliability of Crash Prediction Model Estimates with a  Focus on Mismatch Between CMFs and SPF Base Conditions .......................................................... 19  Chapter 4. Procedures for Quantifying the Reliability of Crash Prnsiction Model Estimates with a  Focus on Error in Estimated Input Values ......................................................................................... 42  Chapter 5. Procedures for Quantifying the Reliability of Crash Prediction Model Estimates with a  Focus on How the Number of Variables in CPM Affects Reliability .................................................. 51  Chapter 6. Reliability Associated with Using a CPM to Estimate Frequency of Rare Crash Types and  Severities: Overview of the Problem with Possible Solutions ........................................................... 60  Chapter 7. Reliability Associated with Predicting Outside the Range of Independent Variables:  Problem Description and Procedure for Practitioners ...................................................................... 71  Chapter 8. Reliability Associated with Predictions Using Cpms Estimated for Other Facility Types:  Problem Illustration with Possible Solutions ..................................................................................... 88  Chapter 9. Summary and Conclusions .............................................................................................. 98  References ........................................................................................................................................ 99  Appendix A: The Development of Procedures for Quantifying the Reliability of Crash Prediction  Model Estimates with a Focus on Mismatch Between CMfs and SPF Base Conditions .................. 101  Note: The conduct of research report published here is associated with NCHRP Research Report 983: Reliability of Crash Prediction Models: A Guide for Quantifying and Improving the Reliability of Model Results (available on TRB’s website at http://www.trb.org/ by searching on “NCHRP Research Report 983”).

v LIST OF FIGURES Figure 1. Example CURE Plot (Source: Hauer and Bamfo, 1997) ................................................................ 13  Figure 2. CURE Plot with Confidence Limits (Source: Hauer and Bamfo, 1997) ......................................... 14  Figure 3. Crash Type and Severity SPFs for Signalized Intersections that could not be estimated in NCHRP  Project 17‐62. .............................................................................................................................................. 61  Figure 4. Crash Type and Severity SPFs for Signalized Intersections that could not be estimated in NCHRP  Project 17‐62. .............................................................................................................................................. 61  Figure 5. FHWA Calibrator Tool CURE plot of residuals based on modified NCHRP Project 17‐62  estimated base condition SPF (Option 1, Equation 55) predictions (x‐axis) ............................................... 65  Figure 6. FHWA Calibrator Tool CURE plot of residuals based on modified NCHRP Project 17‐62  estimated base condition SPF (Option 2, Equation 56) predictions (x‐axis) ............................................... 66  Figure 7. Calibrator CURE plot of residuals based on original NCHRP Project 17‐62 estimated base  condition SPF (Equation 57) predictions (x‐axis) ........................................................................................ 68  Figure 8. Calibrator CURE plot of residuals based on modified NCHRP Project 17‐62 estimated base  condition SPF (Option 1, Equation 58) predictions (x‐axis) ........................................................................ 69  Figure 9. Calibrator CURE plot of residuals based on modified NCHRP Project 17‐62 estimated base  condition SPF (Option 2, Equation 59) predictions (x‐axis) ........................................................................ 70  Figure 10. CURE Plots for MV Crashes ‐ Option 1 ....................................................................................... 80  Figure 11. CURE Plots for MV Crashes ‐ Option 2 ....................................................................................... 81  Figure 12. CURE Plots for MV Crashes ‐ Option 3 ....................................................................................... 82  Figure 13. CURE Plots for MV Crashes ‐ Option 4 ....................................................................................... 83  Figure 14. CURE Plots for MV Crashes ‐ Option 5 ....................................................................................... 84 

vi LIST OF TABLES Table 1. Model Related Factors Influencing the Reliability of an Estimated Value Using a CPM ................. 6  Table 2. Application Related Factors Influencing the Reliability of an Estimated Value Using a CPM ......... 7  Table 3. Results of Practitioner Survey. ...................................................................................................... 17  Table 4. Factors Related to CMFs that Influence the Reliability of an Estimated Value Using a CPM. ...... 19  Table 5. Summary of Applications Associated with Reliability Reduction in Predicted Value. .................. 21  Table 6. Required Data for Case A. ............................................................................................................. 23  Table 7. Required Data for Case B. ............................................................................................................. 27  Table 8. Required Data for Case C. ............................................................................................................. 31  Table 9. Required Data for Case A Example Application. ........................................................................... 35  Table 10. Required Data for Case B Example Application. ......................................................................... 37  Table 11. Required Data for Case C Example Application. ......................................................................... 40  Table 12. Influence of Error in Estimated Input Values on the Reliability of an Estimated Value Using a  CPM ............................................................................................................................................................. 42  Table 13. Sensitivity Analysis Predictions Evaluation Guidance ................................................................. 46  Table 14. Network Screening Evaluation Guidance .................................................................................... 47  Table 15. Sensitivity Analysis of Predictions Evaluations for Example ....................................................... 49  Table 16. Example Network Screening Results ........................................................................................... 49  Table 17. Network Screening Evaluation for Example ................................................................................ 50  Table 18. Factors Related to How Number of Variables in a CPM Influences the Reliability of an  Estimated Value Using a CPM. .................................................................................................................... 52  Table 19. Goodness‐of‐Fit Evaluation Guidance ......................................................................................... 56  Table 20. Network Screening Evaluation Guidance .................................................................................... 56  Table 21. General GOF Statistics ................................................................................................................. 57  Table 22. General GOF Statistics Cont'd ..................................................................................................... 58  Table 23. Comparison of Network Screening Results by EB as a Percentage ............................................. 58  Table 24. General GOF Evaluation for Example .......................................................................................... 58  Table 25. Network Screening Evaluation for Example ................................................................................ 59  Table 26. Influence Applying the Two‐Stage CPM Approach on the Reliability of an Estimated Frequency  of Rare Crash Types and Severities ............................................................................................................. 60  Table 27. GOF Outputs for the Two Options (SD_KA) ................................................................................ 64  Table 28. GOF Outputs for Original and Options 1 and 2 (SD_KAB) ........................................................... 67  Table 29. Maximum AADT Values for Selected CPMs ................................................................................ 71  Table 30. Bias, Variance, and Repeatability Associated with Predicting Outside the Range of the Input  Variable. ...................................................................................................................................................... 73  Table 31. Summary Statistics for the Data Sets Consisting of CA Rural Flat Highway Segments Used in the  Illustration ................................................................................................................................................... 78  Table 32. Testing Results for Multi‐Vehicle Crashes (Rural 4‐lane flat terrain freeways) .......................... 79  Table 33. Results for Single‐Vehicle Crashes .............................................................................................. 85  Table 34. Results for Total Crashes ............................................................................................................. 86  Table 35. CPMs for Daytime Crashes on Freeway Segments from Ohio and North Carolina .................... 89 

vii Table 36. Bias, Variance, and Repeatability Associated with Predicting Crashes at a Different Facility  Type. ............................................................................................................................................................ 89  Table 37. Facility Types by Area Type, Terrain, and Number of Lanes ....................................................... 92  Table 38. Summary Statistics for Groups 1 and 2 ....................................................................................... 93  Table 39. Summary Statistics for Groups 3 and 4 ....................................................................................... 94  Table 40. Assessment Results for Rural 6‐lane Flat Terrain (Group 1) ....................................................... 95  Table 41. Assessment Results for Urban 4‐lane Flat Terrain (Group 2) ...................................................... 95  Table 42. Assessment Results for Rural 6‐lane Rolling Terrain (Group 3) .................................................. 96  Table 43. Assessment Results for Urban 4‐lane Rolling Terrain ................................................................. 96 

viii LIST OF FIGURES Figure 1. Example CURE Plot (Source: Hauer and Bamfo, 1997) ................................................................ 13  Figure 2. CURE Plot with Confidence Limits (Source: Hauer and Bamfo, 1997) ......................................... 14  Figure 3. Crash Type and Severity SPFs for Signalized Intersections that could not be estimated in NCHRP  Project 17‐62. .............................................................................................................................................. 61  Figure 4. Crash Type and Severity SPFs for Signalized Intersections that could not be estimated in NCHRP  Project 17‐62. .............................................................................................................................................. 61  Figure 5. FHWA Calibrator Tool CURE plot of residuals based on modified NCHRP Project 17‐62  estimated base condition SPF (Option 1, Equation 55) predictions (x‐axis) ............................................... 65  Figure 6. FHWA Calibrator Tool CURE plot of residuals based on modified NCHRP Project 17‐62  estimated base condition SPF (Option 2, Equation 56) predictions (x‐axis) ............................................... 66  Figure 7. Calibrator CURE plot of residuals based on original NCHRP Project 17‐62 estimated base  condition SPF (Equation 57) predictions (x‐axis) ........................................................................................ 68  Figure 8. Calibrator CURE plot of residuals based on modified NCHRP Project 17‐62 estimated base  condition SPF (Option 1, Equation 58) predictions (x‐axis) ........................................................................ 69  Figure 9. Calibrator CURE plot of residuals based on modified NCHRP Project 17‐62 estimated base  condition SPF (Option 2, Equation 59) predictions (x‐axis) ........................................................................ 70  Figure 10. CURE Plots for MV Crashes ‐ Option 1 ....................................................................................... 80  Figure 11. CURE Plots for MV Crashes ‐ Option 2 ....................................................................................... 81  Figure 12. CURE Plots for MV Crashes ‐ Option 3 ....................................................................................... 82  Figure 13. CURE Plots for MV Crashes ‐ Option 4 ....................................................................................... 83  Figure 14. CURE Plots for MV Crashes ‐ Option 5 ....................................................................................... 84