Improving Mid-Term, Intermediate, and Long-Range Cost Forecasting for State Transportation Agencies (2020)

Below is the uncorrected machine-read text of this chapter, intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text of each book. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.

46 4. Identification of Suitable Construction Cost Index 4.1 Introduction “Garbage In, Garbage Out” (GIGO) in computer science refers to the fact that invalid or inconsistent inputs in mathematical and computational applications are expected to produce inaccurate or unrealistic results (Farrell 2014). Translated into the context of this study, it means that even if this study were able to produce a methodology able to accurately extract annual inflation rates from any construction cost index (CCI), it still would be useless if it is not applied to a CCI that reasonably represent the intended scope of work. In order to provide effective guidance to state transportation agencies (STAs) on cost forecasting procedures, the research team considered necessary the development of a protocol to assess the degree of suitability of available cost indexing alternatives. The proposed protocol is not aimed to find the best possible CCI. It is instead intended to facilitate a comparative analysis to identify the most suitable alternative among a set of available options, even if all options are traditional CCIs affected by the limitations discussed in Section 1.4.1. This means that the protocol can still be used by STAs that decide not to implement Multilevel Construction Cost Indexes (MCCIs). This chapter summarizes the proposed comparative suitability analysis protocol as it is applied to identify the most suitable cost indexing alternative for each region under each case study. The indexing alternatives considered in this study are the multiple MCCI versions described in Section 3.6.4 and the external and in-house CCIs listed in Table 3.2 in Chapter 3. This chapter only presents an overview of the protocol. NCHRP Research Report 953 contains a more detailed description of the protocol, including instructions to perform the quantitative analysis. 4.2 Purpose of Chapter Besides vetting the proposed protocol for the assessment of cost indexing alternatives, the research efforts presented in this chapter were also intended to answer the following three research questions:  Could different geographic regions within the same state experience significantly different inflationary trends, so that different inflation rates should be applied to different regions?  Is the MCCI system more effective at tracking market changes than traditional cost indexing methodologies?  What price input is the most effective at tracking fluctuations in the construction market? (awarded; average; median; or all unit prices received from both successful and unsuccessful contractors)

47 4.3 Overview of Comparative Suitability Analysis Protocol for Cost Indexes It should be noted that the proposed protocol for the assessment of cost indexes is not intended to evaluate their cost forecasting capabilities. Those capabilities are ultimately given by the forecasting methodologies discussed in the next chapter. The protocol is rather aimed to identify the indexing alternative that most closely resembles the observed behavior of the construction market, which should be the most suitable source of historical pricing data for the intended cost forecasting process. Figure 4.1 illustrates the proposed comparative suitability analysis protocol. Each step in this protocol is discussed in the following sections. Figure 4.1 Cons Indexing Comparative Suitability Analysis Protocol 4.3.1 Representative Pay Items and Analysis Period The pay items used for the comparative analysis of cost indexing alternatives do not necessarily need to include all the MCCI pay items referenced in Chapter 3. Due to time and resource constraints, this study only considered between three to five of the case study agencies’ most relevant pay items to apply the proposed protocol. The selected pay items for each agency are listed in Table 4.1. Further research is required to determine the optimal number of relevant pay items for this analysis. However, a simple computation algorithm would make it easy to consider any number of pay items. Identify a set of  representative pay items  and define the analysis  period Create a bid data point  cloud for each selected pay  item Develop a power regression  curve for each selected pay  item for the first year of the  analysis period  Use power regression curves to  estimate the unit price for each  bid quantity advertised along the  analysis period  Adjust estimated unit prices to  their respective bid dates using  each of the cost indexing  alternatives under consideration Compare adjusted estimated unit  prices and actual awarded prices  with a weighted MAPE value for  each indexing alternative  Identify the cost indexing  alternative with the lowest  adjusted MAPE

48 Table 4.1 Selected Relevant Items for Comparison Analysis Agency Item ID Description Relative Weight Minnesota DOT 2106507/00010 Excavation - Common 17% 2301507/00010 Structural Concrete 33% 2360509/23200 Type SP 12.5 Wearing Course Mixture (3,B) 21% 2360509/23300 Type SP 12.5 Wearing Course Mixture (3,C) 19% 2360509/24500 Type SP 12.5 Wearing Course Mixture (4,E) 10% Colorado DOT 412-00600 Concrete Pavement (6 Inch) 33% 403-00720 Hot Mix Asphalt (Patching) (Asphalt) 56% 203-00010 Unclassified Excavation (Complete In Place) 11% Delaware DOT 202000 Excavation and Embankment 24% 406001 Hot-Mix Patching 54% 503001 Patching C.C.C. Pavement,1.8 m to 6 m, Type A 22% The analysis period used for the identification of the most suitable cost indexing alternative does not need to be the same 20-year period used in Chapter 3 for the creation of MCCIs and used again in Chapter 5 for the assessment cost forecasting methodologies. The analysis period for the protocol presented in this chapter should be long enough to include a good amount of cost indexing data, but not too long, so that, the indexing alternatives are still evaluated on their suitability to the current construction industry. If a given cost index shows the best effectiveness at tracking price fluctuations over the last 20 years, but a different one is found to be more effective over the last 10 years, preference should be given to the latter. Thus, the analysis period for the application of the comparative suitability analysis protocol was about 10 years for each case study agency. 4.3.2 Bid Data Point Clouds “Accuracy” is an abstract concept in construction cost estimating. The market price for a given construction activity cannot be defined by a single “accurate” value. Different contractors competing for the same project commonly submit different sets of prices in their bids. This does not necessarily mean that some of those prices are wrong or inaccurate. There is some natural price variability resulting from the combination of several factors, including the construction means and methods adopted by each contractor, differences in overhead and profit markups, the use of different suppliers and subcontractors, each bidder’s unique perception of risk, and bargaining power of each contractor. Thus, the historical bid data for a given pay item forms a cloud of points fluctuating over time, which represents each item’s unavoidable cost variability. The proposed protocol uses three-dimensional bid data point clouds. The three parameters that give the location of each point in the cloud are: 1) letting data; 2) bid quantity; and 3) recorded awarded unit price. The following steps are aimed to identify the indexing alternative that most closely cut through the middle of those data point clouds.

49 4.3.3 Base Power Regression Curves and Base Unit Price Estimates The base power regression curves used in the comparative analysis of cost indexing alternatives are built with bid data from projects awarded during the first year of the analysis period. For the case studies, one power regression curve was developed for each of the items listed in Table 4.1. For example, the analysis period for the Minnesota Department of Transportation (MnDOT) started in January 2007 and ended in December 2018; therefore, the base power regression curves for each of the five items listed in Table 4.1 were created with historical bid data from 2007. Those curves are then used to estimate unit prices for all bid quantities awarded for each of the selected pay items along the analysis period. Those unit prices are referred to in this report as base unit price estimates. Since the regression curves were developed with data from the first year of the analysis period, all unit price estimates produced with those curves are assumed to yield average unit prices in the middle of that year. Thus, MnDOT’s Structural Concrete base power regression curve was used to estimate a unit price for a quantity awarded in 2018, but the obtained output corresponded to the average price for that amount of structural concrete in mid-2007. 4.3.4 Index-Based Data Point Clouds All base unit price estimates from the previous step are then adjusted to their respective letting dates, creating another data point cloud for each pay item –called index-based data point cloud. Each cost indexing alternative under consideration is used to create a separate set of index-based data point clouds. Each point in the index-based data point clouds has a corresponding point in the original bid data point clouds created with the actual historical bid data. The following steps are intended to measure the average distance between corresponding points provided by each cost indexing option. The shorter the average distance, the larger the overlapping between the bid data and the index-based data point clouds, and the more suitable the cost indexing alternative. 4.3.5 Average Distance Between Bid Data and Index-Based Data Point Clouds and Identification of the Most Suitable Cost Indexing Alternative The average distances between the bid data and the index-based data point clouds were quantified in the form of Mean Absolute Percentage Error (MAPE) values. One MAPE value per pay item. MAPE values are commonly used in the cost estimating literature to measure and compare accuracy between cost estimating approaches (Gardner 2015), but in this study, those values are aimed to indicate the degree of overlap between data point clouds for each selected pay item. For instance, for the case study with the Delaware Department of Transportation (DelDOT), three MAPE values (one per pay item listed in Table 4.1) were calculated with each of the 21 cost indexing alternatives considered for that case study (16 MCCIs from Table 3.9 and 5 existing CCIs from Table 3.2). The MAPE values associated with each cost indexing approach are then combined into a single overall MAPE taking into consideration the relative importance of each pay item (relative weight in Table 4.1). The result of this combination is a weighted MAPE, which is just the weighted average of the pay item MAPE values. The relative weights shown in Table 4.1 were used to calculate the weighted MAPEs for the three case studies. Likewise, the comparative analysis

50 protocol was conducted at the regional level in order to identify the most suitable cost indexing alternative for each of the regions shown in Figures 3.7 to 3.9. Table 4.2 shows the top three cost indexing alternatives identified for each region. The results summarized in this table provide sufficient information to answer the three research questions stated at the beginning of this chapter, and whose answers are discussed in the following section. Table 4.2 Case Study Results - Top Three Cost Indexing Alternatives per Region Agency Region Top Three Indexing Alternatives Minnesota DOT North 1. Statewide MCCI with Awarded Unit Prices 2. North MCCI with Awarded Unit Prices 3. Statewide MCCI with All Unit Prices North Central South Central 1. Statewide MCCI with All Unit Prices 2. Statewide MCCI with Median of Unit Prices per Project 3. Statewide MCCI with Average Unit Prices per Project South 2. Statewide MCCI with All Unit Prices 3. Statewide MCCI with Awarded Unit Prices 4. South MCCI with Median of Unit Prices per Project Colorado DOT Northwest 1. Statewide MCCI with All Unit Prices 2. Northwest MCCI with Awarded Unit Prices 3. Statewide MCCI with Median of Unit Prices per Project Northeast 1. Northeast MCCI with All Unit Prices 2. Statewide MCCI with Median of Unit Prices per Project 3. Northeast MCCI with Awarded Unit Prices Southwest 1. Statewide MCCI with Awarded Unit Prices 2. Statewide MCCI with Median of Unit Prices per Project 3. Statewide MCCI with All Unit Prices Southeast Delaware DOT North 1. Statewide MCCI with Median of Unit Prices per Project 2. Statewide MCCI with All Unit Prices 3. North MCCI with All Unit Prices Central 1. Statewide MCCI with Awarded Unit Prices 2. Statewide MCCI with All Unit Prices 3. Statewide MCCI with Median of Unit Prices per Project South 1. Statewide MCCI with All Unit Prices 2. Statewide MCCI with Median of Unit Prices per Project 3. Statewide MCCI with Awarded Unit Prices 4.4 Chapter Findings This section discusses the three research questions stated in Section 4.2 in the light of the case study results presented in Table 4.2. Could different geographic regions within the same state experience significantly different inflationary trends, so that different inflation rates should be applied to different regions? Table 4.2 shows that statewide MCCIs tend to outperform regional MCCIs, which could be explained by the fact that statewide indexes are developed with larger datasets allowing a more effective representation of construction market changes. The only region that showed an overall

51 better performance of regional indexes was Colorado’s Northeast Region. This could be due to the fact that this is the most densely populated region in Colorado, leading the Colorado Department of Transportation (CDOT) to spend large portions of its annual construction budget in that part of the state, and making its regional historical bid database sufficiently large to produce reliable cost indexes. Although regional MCCIs were not the best option for most regions, the fact that different statewide indexes are more suitable for different regions in the same state allows to conclude that construction activities in different locations could be affected by different inflation patterns. In other words, it would be reasonable to consider the use of different inflation rates for different regions across the state. Those regional inflation rates would be the result from using a different construction cost index in each region. Is the MCCI system more effective at tracking market changes than traditional cost indexing methodologies? In all case studies and geographic regions, the three alternatives with the lowest weighted MAPEs were three MCCIs. Moreover, the research found that the top MCCI always statistically significantly outperformed the most suitable existing CCI (with a 95% confidence level). The research team believes that the ability of MCCIs to meet the matching and proportionality principles discussed in Section 1.4.1 is what makes this alternative cost indexing system more effective at tracing construction price fluctuations than the traditional alternatives. What price input is the most effective at tracking fluctuations in the construction market? Table 4.2 also suggests the greater suitability of two of the four price inputs considered in this study. The most suitable MCCI for 10 of the 11 regions evaluated through the case studies were built with awarded unit prices or all the unit prices received by the agencies. Construction price fluctuations in only one region (DelDOT’s North Region) were better modeled with the median unit prices. The superior performance of the top two price inputs seems to reflect a dilemma considered by the research team at the beginning of the study. The use of awarded prices should be preferred since those are the actual market prices paid by the agency. However, that would limit the amount of available historical bid data to one unit price per contract item per project. A larger dataset, such as the one that would be obtained if all unit prices from successful and unsuccessful bidders are used, could be more effective at modeling relative fluctuations in the construction market. Case study results show that most geographic regions could be associated with one of those two opposing statements. It should be noted that conclusions associated with this research question were based on case study results from only three STAs. The replication of these research efforts among other STAs would allow the formulation of stronger conclusions regarding the selection of appropriate price inputs for the development of MCCIs. The calculation and comparison of weighted MAPEs is the end of the proposed protocol for the comparative analysis of cost indexing alternatives. Once the most suitable cost indexing approach has been identified through the process described in this chapter, the agency can proceed to select the appropriate cost forecasting methodology as discussed in the following chapter.