Below is the uncorrected machine-read text of this chapter, intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text of each book. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.
34 Chapter 4 describes the development and testing process of the spreadsheet tool applied to the four evaluation elements in the spreadsheet model. To test the spreadsheet model, the Kent Corridor in Washington, and the Merchants Corridor in Illinois and Missouri, described in the previous chapter, were used to produce ranked scores of the at-grade crossings in each corridor. 4.1 Beta Test Process and Application The beta test process included the following steps: ⢠Two nonproject staff who represented future users of the tool were assigned to test rail corridor sites to complete the modules. ⢠After completing the spreadsheet, nonproject staff com- pleted a survey to record opinions and preferences about the spreadsheet. ⢠Project staff reviewed and evaluated quantitative metrics, such as the length of time required to complete each module, criteria scores for the same case study completed by two personnel, and other measurable issues monitored and recorded. ⢠Completed documentation described the modules, including the methodology applied to the two corridor case studies, and summarized the results of the beta test. Spreadsheet Tool Development The project team recognized the primary benefits of a spreadsheet tool rather than a web application, including no ongoing maintenance or hosting costs. In addition, users can use the tool offline and will be able to access multiple input files to run various scenarios. Users may also adjust the model parameters in each scenario. The spreadsheet tool was devel- oped using Microsoft Excel. Beginning in April 2017, the program development team released early versions of the spreadsheet tool for internal members of the team to test and debug (alpha testing). The spreadsheet included the four evaluation models: safety, economic impacts, environment, and community livability. Initial concerns during the alpha testing period included the following: ⢠The time required to download FRA data sets and ques- tions related to the data within the sets was often excessive. Downloading the FRA data sets could take 15 to 20 min- utes, once initiated. Evaluating and scrubbing the data, to ensure that the data were accurate, would take additional hours, depending on the corridor length and the quality of the specific DOT-generated data. ⢠The extensible markup language (XML) data stored in the FRA database has an inconsistent format from state to state. For example, the number of fields for each record often varies. The FRA database has a built-in function eliminating fields with null values. ⢠Descriptor fields for cities and counties in the FRA dataset are not populated, requiring the download of additional tables and a lookup process. Manually populating these fields required a significant amount of postprocessing time for the users, which researchers believe would significantly limit the toolâs usability. Preliminary Project Team Testing The research team began alpha testing the spreadsheet to review functionality, ease of use, built-in assumptions, and consistency in data output for future users. After completing alpha testing, the research team began preliminary beta test- ing the tool, or essentially running scenarios and attempt- ing to break the model and find glitches. Team members reviewed and refined user inputs and the mathematics behind the calculations for each of the four modules to make sure C H A P T E R 4 Beta Testing the RCAT
35 that they followed the logic assumed in the decision process development. A final step in the preliminary beta testing included enhancing the look and the feel of the spreadsheet tool, which is shown in Figure 4-1 with the introduction tab. Beta Testing Once alpha testing of the spreadsheet tool was completed, two nonproject staff were selected to test the modules for the select rail corridor sites in Washington State and the Saint Louis region. The beta testers had a background in transpor- tation planning and GIS, but were not experts in analyzing or manipulating railroad or freight data. Full descriptions of these corridors were presented in Chapter 3. For the beta test, a list of railroad crossings for each candidate corridor was given to the testers. The information was supplied in a separate Excel spreadsheet, apart from the model spreadsheet tool. In addition to the list of railroad crossings, testers were provided with a set of initial instruc- tions and tips developed during alpha testing. The beta testers followed the steps below: 1. For the beta test, the railroad crossings and associated corridor were identified and given to the testers. The next step was for the beta testers to map the crossings in ArcGIS or similar GIS software, using the provided latitude and longitude data in GIS, and to create a ½-mile buffer around the rail line. 2. Using the Excel spreadsheet, the beta testers selected the crossings to include in the module. The program allowed the users to filter by state, division, subdivision, and mile- post to allow for easier selection of the crossings. 3. Once all crossings were selected, the beta testers entered the required data for each of the four modules; this pro- cess requires reports from FRA and other local corridor information. The graphic in Figure 4-2 is a screenshot from the community livability module. Once all data were entered for each module, the program generated final test scores, rankings, and numeric values, as shown graphically in Figure 4-3. The resulting summary data can be displayed in graphs or tables. Figure 4-1. Screenshot of RCAT.
36 Figure 4-2. Community livability module: example of user input. Figure 4-3. Sample final graphic of beta test scores. 4.2 Beta Test Feedback for Improving the Spreadsheet Tool One goal of the beta test was to determine where addi- tional instruction was needed or if modifications were needed within the spreadsheet tool. Two independent beta tests were completed, and similar comments were received from both testers, primarily about the process and ease of instructions for future users. Each of the beta testers produced a list of comments for each scoring module. In total, the beta testers compiled more than 50 comments about problems encountered and opportuni- ties for improvement. Issues ranged from computer soft- ware version compatibility to better data input definitions. Each beta tester successfully completed scoring the assigned corridor, and the model produced a final ranking of the rail- road crossings in both test case corridors. Figure 4-4 shows the final score/ranking from one of the beta tests.
37 Figure 4-4. Beta test: final scores for priority crossings displayed on map.
38 4.3 Beta Test Summary The two beta test corridors scored with the use of the spread- sheet tool provided many suggestions that were incorporated into the final instructions to future users. The value of this feedback was critical to the success of the final spreadsheet tool and to the practical use for community planners across the country. Summary notes for consideration included the following: ⢠Overall, the beta spreadsheet tool developed through this research to evaluate and priority rank roadârail grade separation projects within a defined corridor performed as intended. The tool provided qualitative rankings for roadârail crossings based on safety, economic impacts, community impacts, and environmental impacts. ⢠For consistency within the model, the user guide should suggest readily available data resources for future users. ⢠The time to complete the overall framework depends on how many railroad crossings are included in the study corridor. For example, a corridor with one to six crossings may take approximately 2 to 4 hours to complete if the default values are used within the model. If a user wants to customize the data inputs to ensure the local environment is reflected, then more time is needed. A customized cor- ridor with more than 10 railroad crossings may take 6 to 8 hours after data collection, input, and final outputs.
