Click for next page ( 50

The National Academies | 500 Fifth St. N.W. | Washington, D.C. 20001
Copyright © National Academy of Sciences. All rights reserved.
Terms of Use and Privacy Statement

Below are the first 10 and last 10 pages of uncorrected machine-read text (when available) of this chapter, followed by the top 30 algorithmically extracted key phrases from the chapter as a whole.
Intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text on the opening pages of each chapter. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.

Do not use for reproduction, copying, pasting, or reading; exclusively for search engines.

OCR for page 49
49 SECTION 6 TESTING PROCESS 6.1 INITIAL TESTING OF PROTOTYPES sures have been established for pavement ride quality, bridge condition (e.g., the number of functionally obsolete, struc- After prototype AssetManager tools were developed, the turally deficient bridges), and safety (e.g., number of cor- research team tested them. For AssetManager NT, a sample rectable crash sites funded for improvement). scenario was developed using runs from the Pontis bridge The P3 involves a series of tradeoff analyses using MDT's management system and the Deighton dTIMS system (pro- pavement, bridge, congestion, and safety management sys- vided by the Vermont Agency of Transportation). A sample tems. These analyses compare investment levels to perfor- common performance measure across pavements and bridges mance outcomes and seek the distribution of funds that yields (deficiency cost) was calculated for each run and year based the best overall performance. The results of P3 do not deter- on the cost to replace deficient pavements and bridges using mine which specific projects are selected--only the distribu- average unit costs. The input files were developed by enter- tion of funding to districts and work categories and the over- ing management system results (as well as the derived results all system performance expectations associated with that on deficiency costs) into two Excel worksheets and saving distribution. each to CSV format. A system metrics file was developed The AssetManager tools were tested to explore their using data from a sample HPMS file and queries of a sample potential value within the P3 as well as within related efforts Pontis database. A scenario was run, and views were created to assess needs, screen project nominations, and relate can- using the what-if tool. The sample scenario dataset "Sam- didate programs of projects to established work mix and per- ple1" provided with AssetManager NT contains the results of formance objectives. this process. Field testing took place on February 23 through 25, 2004. AssetManager PT was initially tested using a small set of During the site visit, the research team followed test scripts for fabricated data. both tools and recorded MDT staff comments and suggestions. The research team revised both tools as a result of this ini- tial prototype testing period. Subsequently, a formal testing process was undertaken at two panel member states: Montana AssetManager NT Department of Transportation (MDT) and New York State Department of Transportation (NYSDOT). The research team Data Preparation prepared a test plan, including a series of case-oriented test scripts covering each of the steps required to use the tools. MDT ran the PMS 10 times using last year's data set. Budget levels between $50 million annually and $400 mil- lion annually in years 2008 through 2012 were run. (Budget 6.2 MDT FIELD TESTING levels in 2003 through 2007 were constant in all 10 runs, already reflecting programmed projects.) Each run took 10 to Montana is a rural state with a land area of 147,000 square 15 minutes. miles (fourth largest in the nation) and a population of roughly The research team developed a set of Microsoft Access 900,000 (seventh smallest in the nation). The Montana Depart- tables and queries to automate the process of loading spread- ment of Transportation is responsible for maintaining more sheet outputs from MDT's PMS into AssetManager NT. than 10,800 miles of highway and about 2,100 bridges. To pro- These queries were used to produce the necessary input files. vide the Montana Transportation Commission with guidance The research team obtained a copy of the MDT Pontis data- on allocation of available transportation funds, the MDT estab- base and added an option to the Pontis robot to allow the first lished the Performance Programming Process (P3) in 2002. 5 years of each run to be fixed based on programmed projects This process develops a performance-based funding distribu- and variation in budget levels to start in the 6th year. The Pon- tion plan for systems (e.g., Interstate, NHS, and primary), dis- tis robot was then run on the MDT Pontis database to create tricts, and type of work (e.g., roadway reconstruction, rehabil- the system metrics and bridge scenario input files. The bridge itation, resurfacing). Investments for bridges and safety work metrics information was then merged with the pavement met- also are linked to performance objectives. Performance mea- rics information into a single system metrics file.

OCR for page 49
50 The research team used MDT's standard geographic and Staff made the following comments: network categories and performance measures to set up a configuration in AssetManager NT and then created an ini- This tool might be useful for P3 analysis if it could help tial scenario with the pavement and bridge data. Researchers to reduce the current number of PMS runs that must be ran through the test scripts and found and corrected minor done by providing a way to quickly see the impacts of bugs in the configuration screens. different budget allocations on performance. In the initial scenario, the research team observed that, Given that the main challenge in P3 is to allocate a fixed when the budget is fixed for the first 5 years and the what-if budget across work types, networks, and districts in the analysis focuses on the last 5 years, the user still needs to best possible way, this tool would be more helpful if it input the average annual budget for the entire 10-year period. allowed varying allocations of a given budget, rather Because all users may not understand this need, a second sce- than being focused on varying the budget level. nario was created in which the first year of the scenario was This tool appears to be ideally suited for the needs analy- set to 2008 (instead of 2003), and only the data for 2008 sis that is performed at MDT every 2 years, which through 2012 were included. This scenario was useful for involves estimating the amount of funds needed over a looking at annual budget levels over the 5-year period of 10-year period to achieve certain performance objectives. interest; however, it did not allow the entire trend lines from In general, the process to compare two different sets of the present to be seen. resource allocations is awkward (go to resource alloca- On site, MDT staff conducted an additional set of PMS tion screen, set allocations, close that screen, go to the runs (with a different distribution of work across resurfacing, multibudgets, look at results, close that screen, return to rehabilitation, and reconstruction) and created an additional resource allocations, reset values, close, return to multi- NT scenario. budgets, compare . . .). This process is not convenient to use. For this tool to really be used to compare alloca- tions, a split screen is needed so that the user can adjust Testing Results one and see the other change. The user also needs an The research team demonstrated each of the NT views and easy way to generate a tabular report of results for dif- walked MDT staff through using each screen. A few bugs ferent allocations. [These comments were later addressed were identified and logged. by adding the allocation view and incorporating the Two possible applications of the NT tool were identified: resource allocation settings window within the dash- board view.] To facilitate the investment versus performance analy- It is hard to draw definitive conclusions about benefits 3 sis conducted for MDT's P and without actually using the tool as part of the P3 or needs To estimate the investment required over a 10-year analysis process (i.e., putting it to the real test). period to achieve stated performance objectives for the biennial needs analyses. MDT staff suggested several enhancements as a result of the testing process: Staff felt that the tool would definitely be of value for the biennial needs analysis. They felt that the tool would be of Consider modifying the system to handle the case in some, although limited, value for the P3 analysis because it which the first few years of a program are fixed (with pro- does not allow scenarios for different work-type mixes to be grammed projects) and variations need to be tested for the tested, which is a key requirement of P3 analysis. However, last set of years only, but the entire performance trend line analysis of work-type mixes is best accomplished within the still needs to be seen. This case is an extension of the pavement management system itself rather than in Asset- "base" year concept to multiple years (however, there are Manager NT. expenditure levels; they just happen to be fixed). In the end, MDT staff felt that, although AssetManager In the budget view, when switching selections on the NT would not dramatically cut down the amount of effort first tab of the setup, enable automatic population of the required for P3 (such reduction would require some enhance- budget ranges with ntiles (where n is the number of bud- ments to their pavement management system), it could help gets selected), if any of the numbers for budgets do not at the beginning of the process to estimate how much invest- fall within the ranges specified. [This comment was later ment would be required for individual districts and network addressed by adding an auto fill button on this screen.] categories to meet the performance targets. Providing visu- Put the name of the scenario on the views. alization of how sensitive different performance measures For the budget view, include in the report a tabular view are to varying investment levels could potentially be quite of the data shown (i.e., budget level, PM, value, year) helpful. They could not be sure how beneficial the tool would that is exportable to a spreadsheet. Especially with float- be until they actually used it, but they felt that it would be type indicators, it is not easy to read values given the worth giving it a try. axis scaling and labeling.

OCR for page 49
51 Similarly, for the dashboard view, have a tabular report perceive the tool as adding value for this process. In the end, (also exportable to a spreadsheet) with PM, value, net- the decision was made to focus on how planning staff could work category, geographic category, year, and annual use AssetManager PT to better understand the work compo- budget. sition and likely performance implications of the projects On the budget levels tab of the budgeting setup screen, set that were being nominated and selected. Two specific appli- the tab order to navigate across budget levels directly and cations were suggested: not to the colors/thicknesses, etc. (which are relatively rarely changed). [This comment was later addressed.] Screening pavement preservation projects--to help plan- On the resource allocation screen, have the network cat- ning staff recommend which pavement preservation egories, geographic categories, and asset types appear in projects should be advanced into the program (at the the same order as they were entered in the configuration. time of the testing process, MDT was screening pave- [This comment was later addressed.] ment preservation nominations for 2006) and Analyzing work distribution and performance implica- tions of nominations--to compare project mix to the P3 Summary Evaluation recommendations and to explore the likely performance impacts of nominated projects. For this analysis, a "plug" The MDT staff was asked to rate the tools on a scale from value for the pavement preservation category would be 1 to 10, where 10 is the most positive rating. Staff assigned used rather than a value for individual pavement preser- the following summary ratings to AssetManager NT: vation projects, because these projects are on a shorter development cycle than other projects. Potential value of functionality: 7, Ease of data preparation: 89, Data sets were not readily available for loading into the PT User interface: 67 (staff commented that many tool; multiple data sources (e.g., TCP, nominations, PMS, "clicks" were needed to accomplish a given task), and BMS) needed to be merged. This process could be at least Reports/Outputs: 89. partially automated. In the end, staff felt that if there was a "cookbook" procedure for loading the data, the process would AssetManager PT not be overly burdensome. The following issues that occurred with the MDT sources Data Preparation are likely to occur elsewhere as well: AssetManager PT was set up with MDT's network, geo- Project data used for capital programming are not consis- graphic, and project type categories as well as its perfor- tently and accurately tied to location referencing and/or mance measures. cannot be conveniently linked to condition/performance On site, two data sets were created using a set of proposed data from the management systems. This lack will make pavement preservation projects that were being screened. deriving "before" values of performance measures for Preparation of a second data set was begun and then com- projects difficult when the major data source for proj- pleted after the visit. The second data set included a more ects is the capital program. complete set of capital projects from the tentative construc- Different types of projects are on different time cycles tion program. from a budgeting perspective. Smaller preservation proj- ects (e.g., resurfacing projects) are often treated as "plug" line items without specific locations assigned until 1 to Testing Results 2 years before implementation. Programming decisions The research team demonstrated each of the AssetManager about larger capital projects are made further in advance. PT screens and walked MDT staff through using each screen. Obviously, tradeoffs across project categories are not Bugs were identified and logged. possible when the decisions for each category are made The applications for the PT tool were initially less clear at different points in time. than those for the NT tool. The research team discussed how Project data used for capital programming may not be at the purpose of the tool was to provide a better connection a sufficiently disaggregated level for direct input into between the network-level analysis done for P3 and the proj- AssetManager PT. For example, a single project may ects that are actually selected. Staff pointed out that decision- include multiple types of work and may span multiple making about specific projects is highly decentralized in Mon- types of assets and geographic and/or network categories. tana. The PT tool, in theory, could be used by a district to These projects must be split into their component parts help determine which projects to nominate for the program and treated as project packages for purposes of Asset- in a given year, but the staff was skeptical that districts would Manager PT.