Data Management and Governance Practices (2017)

Below is the uncorrected machine-read text of this chapter, intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text of each book. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.

20 This chapter summarizes and discusses the findings of the surveys and follow-up interviews of representatives of state DOTs. It includes the experiences and practices of DOTs in the following topics: • Data governance; • Data warehousing and cloud computing; • Data integration and sharing; and • Data quality assurance. To facilitate responses to the survey, the surveys addressed these topics as they apply to 17 data categories. These categories are sufficiently specific, without being overly detailed, and can be linked to the business functions and project/asset life cycle at transportation agencies. As mentioned, 43 DOTs responded to the Phase 1 survey and 34 DOTs responded to the follow-up survey, representing response rates of 83% and 65%, respectively. Data Governance An important step in data governance is the establishment of a data governance board or council (a group that institutes policies and oversees activities regarding data governance throughout the organization). Interviews with a sample of agencies indicated that this group normally consists of agency executives (e.g., division heads or directors). Figure 11 shows that data governance boards/councils remain rare in DOTs. Of 43 responses received to this question, only eight agencies (19%) indicated the existence of a data governance board/council. However, 16 agencies (37%) indicated that a data governance body is in the develop- ment stage, signifying progress toward implementing data governance. Most of the DOTs that have designated data governance boards and those in the process of devel- oping ones provided a brief description of these boards/councils. The descriptions reveal different names for these bodies, including: • Drive Team—Maine DOT; • Data Governance Committee—Arkansas DOT; • Data Governance Working Group—District of Columbia DOT; • Reliable Organized Accurate Data Sharing (ROADS) Steering Committee (for data sharing)— Florida DOT; • Enterprise Data Sharing and Storage Committee—Kentucky DOT; and • Enterprise Information Governance Group—Washington State DOT. In addition, most DOTs (e.g., Minnesota, New York, Maine, Michigan, Kentucky, Arkansas, and Washington State) have committees or are in the process of designating committees made up of members from different areas of the agency to provide leadership and support in making policies for data-related issues. Some agencies that do not have a fully functional agencywide governance board have structured governance policies for specific data programs. Other agencies designate the responsi- bility of data governance to a single program area (e.g., in Virginia DOT, data governance is managed by the policy division). chapter four State DepartmentS of tranSportation practiceS anD experienceS

21 Data coordinators are individuals or committees that coordinate the organization, sharing, access, and use of multiple data sets within a business area (e.g., asset management, safety). Figure 12 shows that 26 (60%) responding agencies have data coordinators. Eleven (26%) agencies indicated they are in the process of establishing designated data coordinators, and only six (14%) agencies indicated they do not have data coordinators. Agencies who reported having a data governance council/board indicated they also have data coordinators or are in the process of designating one. Respondents from Florida, Illinois, Kentucky, Louisiana, and New York indicated they have designated individuals in different business areas performing the assignments of data coordinators for data sets within their units. In some DOTs, such as Minnesota, Nebraska, and Utah, data stewards perform the assignment of data coordinators. Some respondents mentioned specific data sets with data coordinators, including: • Roadway inventory—Louisiana and Texas DOTs; • Crash and safety data—Iowa, Arizona, and North Dakota DOTs; • Traffic monitoring—North Dakota and Louisiana DOTs; • HPMS—Arizona and Puerto Rico DOTs; • Asset, bridge, and pavement condition—Iowa, Louisiana, North Dakota DOTs; and • Maintenance—Iowa and Louisiana DOTs. The survey asked the respondents to identify data sets (from a list of 17 data sets) for which their agencies have designated stewards. Figure 13 show the data sets and the percentage of DOTs that have designated stewards for the data sets. These results indicate that data collected during the system Does your agency have a designated data governance board/council? 19% (8 agencies) 44% (19 agencies) 37% (16 agencies) Yes No In development FIGURE 11 Data governance boards/councils at DOTs. Does your agency have designated data coordinators? 60% (26 agencies) 14% (6 agencies) 26% (11 agencies) Yes No In development FIGURE 12 Presence of designated data coordinators at DOTs.

22 monitoring phase, specifically pavement inventory and condition, HPMS, and roadway inventory, are more likely to have designated data stewards than are data collected at earlier phases of the asset/project life cycle. Of note is that data sets that reside in data warehouses or marts (discussed in the following section of this synthesis) tend to have data stewards. Conversely, data sets that tend to reside in disparate files (as opposed to data warehouses or marts) are least likely to have designated stewards; examples include real estate data (e.g., property acquisition, agency-owned), pavement work history data, project design and materials data (e.g., design plans, structural design, mix design), and travel modeling data (e.g., household surveys, origin–destination). Some respondents indicated their agencies have designated data stewards for other data not listed in the survey. For example, Puerto Rico DOT provides a steward for historical aerial data, and Alaska designates stewards for road weather information systems (RWIS) used to support winter weather maintenance decisions, seasonal weight restrictions, and travel decisions for the 511 traveler information systems. Arkansas has a steward for all public roads LRS and city/county boundaries data. The follow-up interviews indicated that the data coordinators and stewards tend to be subject matter experts in their business areas (e.g., operations, safety, materials, research, design and engi- neering services, and project development). These individuals hold positions or titles in their busi- ness units such as transportation planner, pavement management engineer, and GIS specialist. One respondent indicated the agency had informal internal training for its data stewards and coordinators. The other interviewed agencies indicated they do not have any training or certification program for their data stewards and coordinators. The development of a document describing the data governance model can serve as a reference and thus assist directly or indirectly in the implementation of data governance. Figure 14 shows that 11 responding agencies (26%) have such a document and 12 (28%) are in the process of developing one (totaling 23 agencies or 54%). However, 20 responding agencies (46%) do not have such a document and are not currently developing one. The survey participants were asked to describe the effect of four factors on limiting progress toward the implementation of data governance in their agencies. These factors are: (1) other mission- related issues are more pressing, (2) hard to justify cost and effort, (3) lack of resources, and (4) lack of staffing. 0 10 20 30 40 50 60 70 80 90 100 Others Project design and materials data Pavement work history data Travel modeling data Real estate data Bridge work history data Transportation improvement programs data Environmental impact and compliance data Project construction data Inventory and condition data for other assets Financial data Contracts/procurement data Traffic monitoring data Crash data Bridge inventory and condition data Roadway inventory HPMS Pavement inventory and condition data % of DOT respondents What data in your agency have designated stewards? FIGURE 13 Presence of designated stewards for data sets at DOTs.

23 Lack of staffing, “other mission-related issues are more pressing,” and lack of resources were commonly described as major factors in limiting progress toward implementing data governance, with 21, 19, and 16 respondents, respectively (Figure 15). Only six of 32 respondents indicated difficulty in justifying cost and effort as a major factor. These results clearly show that lack of staffing is an important factor, whereas difficulty in justifying cost and effort appears to be a much less limiting factor in progress toward data governance at DOTs. Respondents were given the opportunity to list at most two other factors. Other factors that respondents mentioned include lack of departmentwide compliance; lack of enterprise solutions; historic focus on projects (not underlying data); lack of formal governance policies, manuals, standards, and procedures; lack of leadership; lack of understanding of technical needs (geospatial/ data integration/mapping) and how they should be envisioned for the enterprise; and lack of under- standing the magnitude of data being managed by the agency. Finally, the follow-up interviews identified the following hurdles that state DOTs faced while implementing data governance: • Difficulty in getting the business areas to commit staff and time for the data governance working group; • Staff turnover; • Organizational restructuring; and • Difficulty in reaching consensus within the data governance working group. Does your agency have a document that describes its current data governance model? 26% (11 agencies) 46% (20 agencies) 28% (12 agencies) Yes No In development FIGURE 14 DOT practices in documenting data governance models. To what extent do the following factors limit progress on instituting data governance in your agency? 21 16 6 19* 10 14 14 11 1 2 12 3 Lack of staffing Lack of resources Hard to justify the cost and effort Other mission-related issues are more pressing No. of respondents Major factor Minor factor Not an issue *Indicates number of respondents FIGURE 15 Factors limiting progress on implementing data governance in DOTs.

24 Data WarehouSinG anD clouD computinG In response to questions about the use of data warehouses or marts, a wide range of practices across the 17 data sets used in the survey were identified, as shown in Figure 16. Most respondents indicated maintaining system inventory, condition, and performance data in warehouses or marts. These data tend to be collected during the system operation and monitoring phases. Conversely, data collected at the early phases of asset/project life cycle (e.g., real estate data, pavement work history data, project design and materials data, travel modeling data) are less likely to be stored in data warehouses or marts. As noted, data that are likely to be stored in data warehouses or marts tend to have designated stewards. Few other DOTs, such as those in Puerto Rico, Indiana, Alaska, and Arkansas, indicated they maintain other data sets in warehouses or marts. Some of these data sets include historical aerial data maintained by Puerto Rico and RWIS maintained by Alaska. A related question was included in the survey to determine what data are archived to retain historical information. Figure 17 shows that most respondents archive the 17 data sets used in the survey, with the exception of real estate data and travel modeling data. A clear majority of respondents (70% or more) archive the following types of data: pavement inventory and condition, roadway inventory, traffic monitoring, HPMS, project construction, crash, and bridge inventory and condition. The respondents identified other data sets that are archived systematically in their agencies, includ- ing fleet management systems database (Oregon DOT), RWIS (Alaska DOT), and historical aerial data (Puerto Rico DOT). When asked about the volume of data being maintained, the survey revealed that most DOTs do not have reliable estimates. Twenty-six of the 31 responses to this question indicated that such an estimate is not available. This may be because of the use of data silos, which makes it difficult to track all data within an agency as a whole. A few respondents mentioned that their agency has less than 2.2 terabytes of data; whereas, one respondent indicated the agency maintains about 50 terabytes of data. This wide range of data volume may be attributed to differences in the size of the agencies. The use of cloud computing for maintaining transportation data is emerging but is in its infancy, as demonstrated in Figure 18. Of the 31 DOTs that responded to a question about the use of cloud 0 10 20 30 40 50 60 70 80 90 100 Others Real estate data Project design and materials data Travel modeling data Environmental impact and compliance data Pavement work history data Inventory and condition data for other assets Contracts/procurement data Bridge work history data Project construction data Transportation improvement programs data Pavement inventory and condition data Financial data HPMS Crash data Traffic monitoring data Bridge inventory and condition data Roadway inventory % of DOT respondents What data in your agency are maintained in data warehouses or marts? FIGURE 16 Data sets maintained in data warehouses or marts at DOTs.

25 computing, 22 estimated that 1% to 10% of their data (as a whole) are being stored and managed in the cloud. Six agencies indicated that cloud computing is not being used or its use is unknown. When asked about their predictions of the future use of cloud computing for storing and managing transportation data, the responses suggest that this practice is likely to grow in the future (Figure 19). Fourteen of 31 agencies expect that, in the next 5 years, more than 10% of transportation data in their agencies will be stored and managed in commercial cloud computing services. However, the same number of respondents indicated that making this prediction is not possible. Data inteGration anD SharinG The survey included questions to assess the level of data integration across the 17 data sets discussed. The survey also asked the participants to identify methods and tools used for integrating and sharing data within their agencies and with external stakeholders. In all, the respondents identified 120 pairs of data sets that are integrated in their agencies to serve various business requirements. Figure 20 shows pairs of data sets that the majority (16 or more) of Others Travel modeling data Real estate data Environmental impact and compliance data Inventory and condition data for other assets Bridge work history data Transportation improvement programs data Pavement work history data Contracts/procurement data Project design and materials data Financial data Bridge inventory and condition data Crash data Project construction data Roadway inventory HPMS Traffic monitoring data Pavement inventory and condition data % DOT respondents What data in your agency are archived systematically to retain historical information? 0 20 40 60 80 100 FIGURE 17 Use of data archiving to retain historical information at DOTs. Approximately what percentage of your agency’s transportation-related data is currently stored and managed using commercial cloud computing services? 2 agencies 1 agency 22 agencies 3 agencies 3 agencies 21–50% 11–20% 1–10% None Unknown FIGURE 18 Use of cloud computing services at DOTs for storing and managing data.

26 In the next 5 years, what percentage of your agency’s transportation- related data is anticipated to be stored and managed using commercial cloud computing services? 4 agencies 5 agencies 5 agencies 3 agencies 14 agencies More than 50% 21–50% 11–20% 1–10% Unknown FIGURE 19 Future use of cloud computing services for storing and managing transportation data owned by DOTs. B&C A&J A&I A&B A&C A&G A&E No. of DOT respondents Which of these data sets are integrated in your agency to serve various business needs? A. Roadway inventory (e.g., location, classification, geometrics) B. Crash data C. Traffic monitoring data (e.g., speed, volume) D. Travel modeling data (e.g., household surveys, origin–destination) E. Pavement inventory and condition data F. Pavement work history data G. Bridge inventory and condition data H. Bridge work history data I. Inventory and condition data for other assets (e.g., traffic signs, signals, drainage assets) J. Transportation improvement programs data K. Environmental impact and compliance data L. Project design and materials data (e.g., design plans, structural design, mix design) M. Contracts/procurement data (e.g., bid tabs) N. Project construction data (e.g., cost/payments, schedule, material acceptance testing, as-built plans) O. Real estate data (e.g., property acquisition, agency-owned) P. Financial data (e.g., current and historical revenues, expenditures, budgets) 0 5 10 15 20 25 30 FIGURE 20 Pairs of integrated data sets at a majority of responding DOTs. the 31 respondents identified as being integrated in their agencies. The extended form of this figure (showing all pairs of data sets) is presented in Appendix C. In most cases, these integrated pairs involve roadway inventory and another data set collected during system monitoring and operations. In fact, six of the seven most integrated pairs of data sets involve roadway inventory (data set A). Participants were asked to identify pairs of data sets that would be beneficial for their agencies to integrate. Figure 21 shows pairs of data sets which the majority (16 or more) of the 31 respondents identified as potentially beneficial for their agencies to integrate. The extended form of this figure

27 (showing all identified pairs of data sets) is presented in Appendix C. All of these pairs involve road- way inventory (data set A). These results demonstrate the importance of roadway inventory data to multiple business functions at DOTs. However, these results are to be viewed within the context of many of the survey respondents being planners or analysts/managers of planning data, jobs for which roadway inventory plays a major role. The integration of transportation data from different sources requires a common LRM or automated means for converting different LRMs to be compatible. Thus, the survey asked the participants to identify the LRMs used in existing data sets at their agencies. Six LRMs were given: route mile point, route reference post, link-node, route street reference, multilevel linear referencing system, and geo- graphic coordinates. Figure 22 shows that all of these LRMs are being used by state DOTs in various data sets. However, geographic coordinates (e.g., longitude-latitude or state plane coordinates) and FIGURE 21 Pairs of data sets that would be beneficial to integrate, as identified by the majority of responding DOTs. 0 8 16 24 32 Geographic coordinates Route mile point Link–node Route reference post Route street reference Multilevel LRS Other N o. o f a ge nc ie s What location referencing methods are used in your agency? FIGURE 22 LRMs used by DOTs in at least one data set.

28 route mile point are the most commonly used LRMs. The wide use of geographic coordinates may be because of the prevalence of Global Positioning System (GPS) in current data collection technologies. The route mile point LRM is used to represent attributes (called events) on linear features (called routes). Table 5 shows that a similar conclusion can be made about the use of these LRMs for individual data sets. Comments made by some respondents indicated that some agencies use other LRMs, especially for contract/procurement data, project construction data, and travel modeling data. To evaluate agency data sharing methods with external stakeholders, including public and private entities, five options were provided to the respondents to select from: online open access, online preauthorized access, upon request (e.g., data sent by e-mail or a file-sharing service), not shared outside the agency, or data shared through other methods. Some DOT respondents indicated more than one sharing method for some data sets. Figure 23 shows the use of these methods for all data sets, and Table 6 shows the use of these methods for each data set individually. Each responding DOT indicated the use of a combination of these methods to share data with external users. However, the online open access and “upon request” methods are used by most of the responding DOTs (Figure 23). Table 6 shows that online open access and upon request methods are consistently most common when the results are divided by individual data sets. Table 6 also indicates that the most shared data sets through online public access are traffic monitoring, TIPs, and roadway inventory. All respondents indicated that their agencies share roadway inventory and traffic monitoring data sets outside their agencies. Conversely, travel modeling and bridge work history are the data sets least shared with external entities. What location referencing methods are used in your agency? Geographic coordinates Route mile point Link–node Route reference post Route street reference Multilevel LRS Other Roadway inventory 17 27 7 8 13 1 Crash data 22 20 4 7 0 Traffic monitoring data 12 24 5 4 1 Travel modeling data 7 9 9 2 9 9 1 4 HPMS 14 26 7 6 11 0 Pavement inventory and condition data 17 25 5 2 0 Pavement work history data 7 17 2 4 5 Bridge inventory and condition data 24 23 3 2 0 Bridge work history data 13 14 2 1 1 1 9 7 3 Inventory and condition data for other assets 19 19 4 5 17 Transportation improvement programs data 7 18 7 8 Environmental impact and compliance data 12 12 0 8 7 6 2 5 8 8 8 5 8 6 1 3 3 2 8 Contracts/procurement data 6 15 8 3 Project construction data 7 14 0 0 0 7 1 1 2 4 7 TAbLE 5 DOTs’ USAGE OF LRMs IN INDIVIDUAL DATA SETS (by number of respondents)

29 How does your agency share data with outside users (public and private entities)? Online (open access) Upon request Not shared outside agency Online (preauthorized access) Other Roadway inventory 20 25 0 2 Crash data 10 16 3 2 Traffic monitoring data 22 18 0 3 Travel modeling data 2 17 0 HPMS 10 21 4 1 Pavement inventory and condition data 10 22 1 1 Pavement work history data 7 17 3 Bridge inventory and condition data 15 21 1 2 Bridge work history data 5 14 4 Inventory and condition data for other assets 8 17 5 Transportation improvement programs data 20 16 0 1 Environmental impact and compliance data 6 16 5 Contracts/procurement data 11 15 3 2 Project construction data 7 16 6 Financial data 5 15 6 5 4 4 2 0 5 4 4 6 6 4 3 5 3 3 1 4 3 5 2 8 7 TAbLE 6 METHODS USED bY DOTs FOR SHARING INDIVIDUAL DATA SETS WITH ExTERNAL USERS (by number of respondents) Respondents were asked to identify strategies that would improve or have improved data sharing and access at their agencies. Four strategies were presented to the respondents: improved metadata, increased use of web-based data storage and access, improved data management systems, and reduced use of hardware and software that require specialized data format. The respondents were asked to describe these strategies as having major effect, minor factor, no effect, or not applicable. Respondents were also given the opportunity to list two other strategies. FIGURE 23 Data sharing methods used by DOTs for at least one data set. 0 8 16 24 32 Online (open access) Upon request Not shared outside agency Online (preauthorized access) Other N o. o f a ge nc ie s How does your agency share data with outside users (public and private entities)?

30 Most respondents indicated that two of these strategies have a major effect on improving data sharing and access: (1) increased use of web-based data storage and access, and (2) improved database management systems (Figure 24). Other strategies mentioned by respondents include implementation of data governance, establish- ment of standards and requirements for sharing, implementation of civil integrated management, having a data registry, use of enterprise-level business intelligence vendor products, and the presence of a senior management champion. Respondents also listed data management tools most useful for accessing and sharing data within their agencies. These tools can be generally grouped into: • GIS and geospatial and mapping tools; • Tools for integrating different location referencing systems; • Specific tools, such as Structured Query Language (SQL) Server Reporting Services (SSRS), SharePoint, Excel, SAP business Objects Suite, and Oracle business Intelligence Suite; • Data warehouses, including cloud storage; • Representational State Transfer (REST) services (the software architectural style of the World Wide Web); and • ER/Studio (identified by one of the interviewed agencies). This is a commercially available software tool for managing data assets, including documenting data elements and objects; show- ing their sources, interactions, and dependencies; and setting permissions for access controls. Data Quality The participants were asked to identify the extent to which various data quality elements are evalu- ated in their agencies. The quality elements provided in the survey included accuracy, completeness, timeliness, relevancy, consistency, accessibility, and access security. Table 7 summarizes the responses What strategies would improve (or have improved) data sharing and access within your agency? Reduced use of hardware and software that require specialized data format Improved database management systems Increased use of web-based data storage and access Improved metadata No. of DOT respondents Major effect Minor factor No effect Not applicable *Indicates number of respondents 16 23 23 15* 10 6 4 15 3 2 1 2 3 FIGURE 24 Strategies for improving data sharing and access at DOTs. Quality Element Accuracy Completeness Timeliness Relevancy Consistency Accessibility Access Security Evaluated in all or most areas 14 13 13 12 7 8 16 Evaluated in some areas 17 16 21 17 13 19 13 Evaluated in a few areas 3 5 0 5 14 7 5 TAbLE 7 EVALUATION OF DATA QUALITY AT DOTs (by number of respondents)

31 received. No respondents indicated that any of these data quality elements is entirely ignored in their agencies. Most respondents indicated that all of these data quality elements are evaluated in at least some data areas in their agencies. Timeliness, accuracy, and access security are most commonly evaluated. Consistency is the data quality element least evaluated by DOTs. because it can be reasonably assumed that feedback from data users would improve data quality, the survey participants were asked to specify if their agencies have mechanisms in place for incor- porating users’ feedback into the data collection process. Figure 25 shows that 15 of 30 respondents indicated that their agencies have mechanisms in place for incorporating this type of feedback. The respondents described these feedback mechanisms as ad hoc meetings, surveys, steering committees, web forms, and direct e-mails. Does your agency have mechanisms in place for incorporating feedback from data users in your agency into the data collection process? 15 agencies 15 agencies Yes No FIGURE 25 Presence of mechanisms for incorporating users’ feedback into the data collection process.