Understanding and Communicating Reliability of Crash Prediction Models (2021)

Below is the uncorrected machine-read text of this chapter, intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text of each book. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.

1 Summary Background The 1st edition of the Highway Safety Manual (AASHTO, 2010) is the product of over 10 years of effort and thousands of volunteer hours to provide fact-based, analytical tools, and techniques to quantify the potential safety impacts of planning, design, operations, and maintenance decisions. Part C of the 1st edition of the HSM contains the predictive methods for rural two-lane roads, rural multilane highways, and urban and suburban arterials. Since the publication of the 1st edition, two chapters with crash prediction models (CPMs) for freeways and ramps that were developed through NCHRP Project 17-45 have been approved for inclusion as a supplement to Part C. The 2nd edition of the HSM is expected to be published in the next two years. The 1st edition of the HSM does not include methods to consistently convey model reliability. During initial implementation of the 1st edition of the HSM, model results have been generated and utilized without fully understanding and communicating the accuracy of the model results, which can erode the credibility of this new and rapidly growing field. Since the publication of the 1st edition of the HSM, the state of the art of safety analysis has progressed and more has been learned about the impact on accuracy of assumptions made during the development of CPMs using HSM procedures. Practitioners are also striving to fully understand and appropriately communicate the benefits of the HSM methods and results derived from these methods. What Is Reliability? In general, the reliability of the prediction from a CPM can be described in terms of bias, variance, and repeatability:  Bias represents the difference between the CPM estimate and the true value.  Variance describes the extent of uncertainty in the estimate due to unexplained or random influences.  Repeatability describes the extent to which multiple analysts using the same CPM with the same training, data sources, and site of interest obtain the same results (as measured by the number of significant figures showing agreement among results). A more reliable estimate has little bias, a smaller variance, and is likely to have results that show several significant figures in agreement (should there be repeated independent applications). This report focuses on how to estimate the bias and the variance for certain conditions. Two categories of factors influence the reliability of a CPM: model-related factors and application- related factors. Model-related factors describe the components of a CPM that is specified by an agency for use by practitioners to evaluate sites in the corresponding jurisdiction. Application-related factors describe techniques that practitioners apply when using a CPM to evaluate a site. The factors in both categories combine to define the reliability of a CPM estimate when it is used to evaluate a given site. In this regard, the estimate’s reliability is likely to vary from site-to-site and analyst-to-analyst, depending on how the model is configured and applied to a given site. Objectives of This Research The objectives of this study are to:  Develop guidance for the quantification of the reliability of CPMs (including crash modification factors, crash modification functions and safety performance functions) for practitioner use;

2  Develop guidance for user interpretation of model reliability; and  Develop guidance for the application of crash prediction models accounting for, but not limited to assumptions, data ranges, and intended and unintended uses. This was a two-phase effort. Phase I included a kickoff call, reviewing and assimilating literature and state of the art, analysis of relevant resource data and identify gaps, develop work plan, develop an annotated outline of the guidance document, develop an annotated outline of the communications plan, develop an interim report, and a face to face interim meeting. Phase II involved the development of the guidance document that accompanies this conduct of research report, development of the communications plan, and final documents and reports. Following a kickoff call with the panel early in this study, the project team decided to conduct a survey of practitioners to obtain insight into their priorities and concerns and used these to identify the research issues to be addressed in Phase II (see Chapter 2 for a summary of the results from the survey). The project team conducted a limited literature review in the beginning of the project, but most of the literature review was specific to the research issues that were addressed in the context of the plans for Phase II. Scope of This Report and Audience Based on a survey conducted in the beginning of NCHRP Project 17-78, “Understanding and Communicating Reliability of Crash Prediction Models,” this report focuses on the following model- related and application-related factors:  Procedures for Quantifying the Reliability of Crash Prediction Model Estimates with a Focus on Mismatch between CMFs and SPF Base Conditions (Chapter 3)  Procedures for Quantifying the Reliability of Crash Prediction Model Estimates with a Focus on Error in Estimated Input Values (Chapter 4)  Procedures for Quantifying the Reliability of Crash Prediction Model Estimates with a Focus on How the Number of Variables in CPM Affects Reliability (Chapter 5)  Reliability Associated with Using a CPM to Estimate Frequency of Rare Crash Types and Severities: Overview of the Problem with Possible Solutions (Chapter 6)  Reliability Associated with Predicting Outside the Range of Independent Variables: Problem Description and Procedure for Practitioners (Chapter 7)  Reliability Associating with Predictions Using CPMs Estimated for Other Facility Types: Problem Illustration with Possible Solutions (Chapter 8) Chapter 2 provides a summary of the results of a survey conducted in Phase 1 to obtain insight into their priorities and concerns. From Chapter 3 through Chapter 8, each chapter provides a description of the problem along with procedures for quantifying the reliability or possible solutions to improve the reliability. The discussion for some of the topics is more involved, e.g., the topic in Chapter 3 involving mismatch between CMFs and SPF base conditions, is very detailed and involved. In this report, each chapter is self-contained thus enabling the users to consult the chapter suitable to their need. The primary audience for this report is practitioners and researchers. These readers should be familiar with the Chapter 3: Fundamentals and Part C Chapters of the AASHTO Highway Safety Manual 1st Edition (AASHTO, 2010). As much as possible, this report does not use complicated jargon. However, some of the procedures described in this document are very detailed and involved and may be require a high level of statistical understanding. Accompanying this report are the following documents:  NCHRP Research Report 983: Reliability of Crash Prediction Models: A Guide for Quantifying and Improving the Reliability of Model Results  A communications plan  A one-page flyer to promote the Guide