Guidance for Development and Management of Sustainable Enterprise Information Portals (2018)

Below is the uncorrected machine-read text of this chapter, intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text of each book. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.

47 Design and Implementation for Sustainable DOT EIPs This section highlights how DOT EIP services could be decomposed and implemented fol- lowing the principles of microservices architecture. This section does so by providing the reader with a list of requirements for sustainable DOT EIPs; technology recommendations for building, hosting, and managing microservices; examples of microservices implementations depicted as a series of system engineering diagrams for each of the DOT EIP services described in Table 2-1; and recommendations for migrating services from a monolithic EIP system to a microservices architecture. These recommendations and examples are presented in the following six areas: • Requirements • Operating environment • Technology recommendations for sustainable DOT EIPs • Implementation examples • System diagrams • Migration strategy 5.1 Requirements This section contains an initial list of requirements for sustainable DOT EIPs. A requirement is a capability or function that must be delivered by a system component or components. The list organizes requirements into the following categories: • Functional requirements. These define the functions, or business needs, of a system or of sys- tem components as seen by a system user. These requirements may be calculations, technical details, data manipulations, and other functionalities that define what a system is supposed to accomplish. • Performance requirements. These specify the speed or operational effectiveness of a capabil- ity that must be delivered by the system. • Design requirements. These detail the plans for the software solution and specify architecture design as well as the methodologies, models, or frameworks by which the software is to be developed. • External interface requirements. These specify the hardware, software, communications, user, or database elements to ensure that the system will communicate properly with external components. • Resource requirements. These include the hardware, software, expertise, and organizational components. Table 5-1 lists general requirements that support sustainable DOT EIPs. These requirements serve as a strong starting point for agencies as they move toward the development and manage- ment of their services. S E C T I O N 5

48 Guidance for Development and Management of Sustainable Enterprise Information Portals Functional Requirements F1. The DOT EIP shall contain a service discovery and registrar service F2. The DOT EIP shall contain a persistent link service F3. The DOT EIP shall contain a cache service F4. The DOT EIP shall contain an API gateway service F5. The DOT EIP shall contain a notification service F6. The DOT EIP shall contain a search service F7. The DOT EIP shall contain a data ingestion service F8. The DOT EIP shall contain an access management service F9. The DOT EIP shall contain a content publishing service F10. The DOT EIP shall contain an activity tracking and logging service F11. The DOT EIP shall contain a workflow management service F12. The DOT EIP shall contain a data archiving service F13. The DOT EIP shall contain a metadata management service F14. The DOT EIP shall contain a reporting service F15. The DOT EIP shall contain a file exchange service F16. The DOT EIP shall contain a data analysis service F17. The DOT EIP shall contain a data visualization service F18. The DOT EIP shall contain a video-streaming service F19. The DOT EIP shall contain a computer vision service F20. The DOT EIP shall contain a video transcoding service F21. The DOT EIP shall contain a GIS data analysis service F22. The DOT EIP shall contain a GIS rendering service F23. The DOT EIP shall contain a public web user interface F24. The DOT EIP shall contain a contractor web user interface F25. The DOT EIP shall contain a contractor/vendor procurement user interface F26. The DOT EIP shall contain an administrator web user interface F27. The DOT EIP shall contain a DOT operation user interface F28. The DOT EIP shall contain a DOT librarian user interface F29. The DOT EIP shall contain a DOT analyst user interface F30. The DOT EIP shall contain a DOT maintenance user interface F31. The DOT EIP shall contain a DOT monitoring user interface F32. The DOT EIP shall contain a DOT download user interface F33. The DOT EIP shall contain a DOT map user interface F34. The DOT EIP shall contain a DOT procurement user interface F35. The DOT EIP shall contain a DOT data collection user interface F36. The DOT EIP shall contain a generic DOT employee user interface F37. The DOT EIP shall contain an IT monitoring user interface F38. The DOT EIP shall contain an IT operation user interface F39. The DOT EIP shall contain an IT investigation user interface F40. The DOT EIP shall contain a knowledge consumer web user interface F41. The DOT EIP shall contain a knowledge provider web user interface F42. The DOT EIP shall contain a knowledge expert web user interface F43. The DOT EIP shall contain a state web user interface F44. The DOT EIP shall contain a load balancer service F45. The DOT EIP shall contain an auto-scaler service F46. The DOT EIP shall contain an analytical database service F47. The DOT EIP shall contain an archive storage service F48. The DOT EIP shall contain a content storage service F49. The DOT EIP shall contain a geo-capable Not Only SQL (NoSQL) distributed database F50. The DOT EIP shall contain a geospatial relational database management system service F51. The DOT EIP shall contain a metadata repository service F52. The DOT EIP shall contain a network and application logs service F53. The DOT EIP shall contain a NoSQL distributed database service F54. The DOT EIP shall contain a relational database management system (RDBMS) service Table 5-1. Requirements for sustainable DOT EIPs.

Design and Implementation for Sustainable DOT EIPs 49 Table 5-1. (Continued). Performance Requirements P1. DOT EIP services shall be able to be started rapidly P2. DOT EIP services shall be able to be stopped rapidly P3. DOT EIP services shall be able to be duplicated rapidly P4. DOT EIP components shall be able to be decommissioned rapidly P5. DOT EIP components shall be able to be deployed rapidly P6. DOT EIP components shall be able to be updated rapidly Design Requirements D1. The DOT EIP components shall be distributed D2. The DOT EIP components shall connect to the API gateway service D3. The DOT EIP components shall register to the service discovery service D4. The DOT EIP components shall be managed through the DOT EIP persistent link service D5. The DOT EIP shall follow a microservices architecture pattern D6. The DOT EIP data store shall be part of the microservices architecture D7. The DOT EIP shall use a centralized logging service D8. The DOT EIP shall use a distributed tracking and logging service D9. The DOT EIP shall rely on events notification to communicate between components D10. All DOT EIP services shall adjust to demand spikes automatically D11. All DOT EIP components shall be able to be updated, deployed, and decommissioned without affecting the other DOT EIP components D12. The DOT EIP shall use API standards to design and operate its API gateway D13. The DOT EIP shall use a single data format across the entire system such as JavaScript Object Notation (JSON) or Extensible Markup Language (XML) D14. The DOT EIP shall provide its clients with required libraries as part of its services D15. The DOT EIP shall rely mostly on hyper text transfer protocol (HTTP) and REST for communication between services External Interface Requirements E1. The DOT EIP shall be able to connect to a live sensor feed service E2. The DOT EIP shall be able to connect to a live video feed service E3 . All DOT EIP components shall rely mostly on HTTP and REST for communication with external services Resource Requirements R1. The DOT EIP shall be implemented on a commodity cluster infrastructure (cloud-like) R2. The DOT EIP shall be developed using personnel familiar with cloud technology R3. The DOT EIP shall leverage cross-function teams over the complete product cycle 5.2 Operating Environment Sustainable DOT EIPs should be hosted on either cloud or mixed (cloud and on-premise) infrastructure. The mixed infrastructure approach allows increased scalability and reliability for EIP applications while also allowing sensitive data such as acquisition documents to be kept in house. The recommended approach for providing application deployment and server configura- tion is either a cloud-based configuration management service or an open source configuration management service used in combination with a containerized architecture. The use of an open source continuous integration tool, a hosted continuous integration ser- vice (SaaS), or a proprietary continuous integration tool is recommended for the implementa- tion of sustainable DOT EIPs.

50 Guidance for Development and Management of Sustainable Enterprise Information Portals 5.3 Technology Recommendations for Sustainable DOT EIPs Table 5-2 shows a list of recommended technologies for developing sustainable DOT EIPs following a microservices architecture approach. These technologies were selected based on a survey of industry practitioners and a scoring and ranking of each of the suggested technologies based on its reliability, security, interoperability, usability, scalability, modifiability, reusability, integrability, affordability, and appropriateness for DOT organizations. 5.4 Implementation Examples To help users better understand how microservices can be designed and organized to per- form the various DOT EIP services presented in Table 2-1, Table 5-3 presents a list of nine example scenarios based on the DOT EIP high-level services described in Section 2. In these examples, DOT EIP users interact with various services of the portal to obtain or publish information. How microservices can be created and organized to implement each service can be illus- trated through a compilation of Unified Modeling Language (UML) diagrams. UML is a way of visualizing a software program using a collection of 14 different diagrams. UML is accepted by the Object Management Group (OMG) as the standard for modeling software development. UML diagrams provide a way for individuals involved in the planning and design of a complex enterprise system to visualize user interactions, processes, and the structure of the system they are trying to build to help identify issues early on and ensure a common vision among all parties from the beginning. Due to the large number of diagrams that would be needed to completely represent each of the nine scenarios, the asset management example is offered herein because surveyed DOT stake- holders showed the greatest interest in this service. Typically, UML diagrams begin with the use case; consequently, Section 5.4.1 presents the use case for the asset management and engineering services scenario listed in Table 5-3. Following the use case, Section 5.4.2 provides an enumera- tion of service and interface components, which is the context for the subsequent presentation of the four diagram types—block definition diagrams, component and data flow diagrams, activity diagrams, and sequence diagrams. Similar UML diagrams illustrating the remaining service scenarios listed in Table 5-3 can be found in the contractor’s final report for NCHRP Project 20-103, NCHRP Web-Only Docu- ment 241: Development and Management of Sustainable Enterprise Information Portals (available on www.trb.org). Microsoft Visio files of these diagrams are also available on www.trb.org by searching on NCHRP Research Report 865. 5.4.1 Use Case UML Diagrams A use case diagram at its simplest is a representation of a user’s interaction with a sys- tem. This representation shows the relationship between the user and the different use cases in which the user is involved. Use case diagrams are valuable for visualizing the functional requirements of a system so that these requirements can be translated into design choices and development priorities. Use case diagrams also help identify any internal or external factors that may influence the system and thus should be taken into consideration, and these diagrams provide a good high-level analysis from outside the system. Use case diagrams specify how the system interacts with actors and are not focused on the details of how that functionality is

Design and Implementation for Sustainable DOT EIPs 51 EIP Component Recommended Approach Se rv ic e API Gateway Adopt a cloud API framework. Searching Adopt a cloud enterprise search service. GIS Data Processing and Rendering Adopt an open source scale out GIS server framework in combination with the use of cloud storage. Data Stream Processing Adopt a cloud stream processing framework. Messaging Adopt a cloud messaging framework. Content Delivery Adopt either a cloud content delivery network solution or a commercial distributed content delivery solution. U se r In te rf ac e Web Applications Adopt a cloud-based web framework in combination with an advanced client-side web development framework to support web design capabilities. Similarly, adopt a cloud provider monitoring service for web monitoring. Mobile Applications For design and monitoring capabilities, adopt a cloud mobile application deployment service. Desktop Applications No desktop application should be developed as part of a sustainable DOT EIP. The recommended approach to providing desktop application capabilities in a sustainable DOT EIP API is the use of a content management service-based web application. St or ag e Content Archiving Adopt a mixed storage capability using cloud and on-premise storage. Content Caching Adopt a cloud or a traditional commercial caching and load balancing solution. Content Tagging Use a commercial metadata repository, a custom user-generated metadata repository, or a semantic web solution. D at ab as e GIS Database Services The recommended approach for providing GIS database service capabilities in a sustainable DOT EIP API is the adoption of a cloud or on- premise NoSQL geo-capable database. Document Database Services The recommended approach for providing document database service capabilities in a sustainable DOT EIP API is the adoption of an open source distributed document analysis engine. Image, Video, and Binary Files Database Services The recommended approach for providing image, video, and binary files database service capabilities in a sustainable DOT EIP API is the adoption of a basic cloud storage solution. Financial Data Database Services The recommended approach for providing financial data database service capabilities in a sustainable DOT EIP API is the adoption of a commercial cloud enterprise solution. Sensor Data The recommended approach for providing sensor data database service Database Services capabilities in a sustainable DOT EIP API is the adoption of a cloud streaming framework based solution. M an ag em en t T oo ls Resources and Application Monitoring The recommended approach for providing resources and application monitoring capabilities in a sustainable DOT EIP API is the adoption of a cloud monitoring solution. Resources Provisioning and Scaling The recommended approach for providing resources provisioning and scaling capabilities in a sustainable DOT EIP API is the adoption of either a cloud-based serverless architecture or a cloud or on-premise container- based solution. Content Auditing The recommended approach for providing content auditing service capabilities in a sustainable DOT EIP API is the adoption of cloud-based commercial or open source content auditing and inventory software. Table 5-2. Sustainable DOT EIP technology recommendations. (continued on next page)

52 Guidance for Development and Management of Sustainable Enterprise Information Portals Se cu ri ty a nd Id en ti ty Identity and Access Management The recommended approach for providing identity and access management service capabilities in a sustainable DOT EIP API is the adoption of a cloud access and identity management solution. Security Audit There is no single approach to providing a security audit to a sustainable DOT EIP. Some approaches are tool based while others are human based. These approaches were provided by the survey responders: open source or commercial security evaluation and monitoring tools for network and software layer, internal security audits and assessments, and external party assessments and penetration testing. These approaches should be used together to provide security audits to a sustainable DOT EIP. EIP Component Recommended Approach Table 5-2. (Continued). implemented. Figure 5-1 shows the use case diagram for the assets management and engineer- ing services scenario, while Table 5-4 lists use case actions and describes the actions illustrated in the diagram. 5.4.2 Service Components Block definition, component and data flow, activity and sequence diagrams all contain sim- ple graphical components representing interfaces or microservices involved in the processes described by the diagrams. To better understand the example diagrams, a list of the components included in the diagrams and a brief description of each are provided. Each of these components is structured as a series of identical microservices that can be cloned or turned on/off upon demand. Service Scenario Mapped GIS and Mapping Services DOT GIS expert requests an analysis of a GIS dataset and visualizes the results. Operations and Performance Management Services DOT EIP public user requests a live video stream of a traffic camera on his cell phone. Bid and Contract Services DOT procurement employee runs a procurement status report on the DOT EIP. External Portal Users Services Contractor searches for and downloads documents shared on the DOT portal. Asset Management and Engineering Services DOT maintenance employee archives older asset data. Environmental Services DOT employee monitors and analyzes live data coming from seismic sensors. Document and Library Services DOT librarian uploads documents into a portal library, extracts metadata from the document, and shares its availability with the rest of the DOT. Knowledge Management Services Knowledge provider searches the DOT knowledge management system, contributes to knowledge content, and shares the contribution. IT Services (network and application monitoring) DOT IT experts search network and applications logs for abnormal behaviors, analyze and visualize the data, and share the findings. Table 5-3. Implementation example scenarios.

Design and Implementation for Sustainable DOT EIPs 53 Table 5-5 presents the set of interfaces while Table 5-6 and Table 5-7 present business and data services, respectively. 5.4.2.1 Block Definition Diagrams Block definition diagrams are created to communicate structural information about a system. This design technique creates extensibility in a system design by decoupling the clients of services from any specific implementation of a provider of those services. Block definition diagrams are not tied to any specific stage of the system life cycle or level of design. They are created and referenced to support stakeholder needs analysis, requirements definition, architectural design, performance analysis, test case development, and integration. Block definition diagrams provide a complementary view of an aspect of the system of interest. Figure 5-2 shows an example of a block definition diagram representing the structural organization of the various system compo- nents involved in the asset management and engineering services scenario. The diagram shown in Figure 5-2 deviates from the UML, SysML standard to indicate the ability for each component to scale and duplicate itself by representing them as a series of closely stacked rectangles with a dashed rectangle at the bottom. Figure 5-1. Use case diagram for asset management and engineering services scenario.

54 Guidance for Development and Management of Sustainable Enterprise Information Portals User Actions Action Descriptions illustrated in the Use Case Diagram Set up alert Alerts are set up either by DOT operations or maintenance users to be sent when there is important or relevant information of which other users should be made aware. Receive alert Alerts containing new important or particularly relevant DOT system information are received by DOT operations or maintenance users. Modify asset data DOT maintenance users make any necessary modifications to asset data that has been uploaded. Create new asset DOT maintenance users create new assets in the system as new assets are installed. Disable asset DOT maintenance users disable assets for removal from the system as assets are removed or no longer functional. Upload asset information All relevant asset information is uploaded by DOT maintenance users. Analyze asset data Both DOT operations and maintenance users can analyze asset data as they are uploaded into the system. View asset report DOT operations users view reports on various assets containing the status of installed assets. Monitor asset DOT operations users monitor all installed assets and create reports of their status. Search asset data All asset management users, both internal and external, can search the asset data available. Browse asset data All internal asset management users can browse the asset data available. Text search Text searching allows users to type into a text box and use keywords to narrow the available content to only what is relevant to what they need. Faceted search Faceted searching allows users to apply filters to the data to search through relevant content for what they need. View asset data All DOT and other state users can view asset data. View asset metadata All DOT and other state users can view the asset metadata. Visualize asset data All DOT and other state users can view visualizations of the asset data such as quality assurance control charts. Download asset data When asset data are made available for download, internal users may download a copy. Add new asset The administrator adds new assets data in bulk to the system after they are approved for upload. Decommission asset The administrator decommissions asset(s) as they are confirmed to be no longer in existence. Archive asset data The administrator archives asset data as assets are decommissioned or no longer needed in the system. Manage asset metadata The administrator manages all metadata to ensure that they are still relevant, effective, and up-to-date. Manage asset data storage The administrator manages data storage to ensure that the system’s capacity is not exceeded and data are archived as necessary. Manage asset data access The administrator manages all users’ access to the datasets that they have permission to use. Table 5-4. Asset management and engineering services use case actions and descriptions.

Design and Implementation for Sustainable DOT EIPs 55 interact with the DOT EIP. Public web user interfaces Defines the web interfaces that will be used by the public to interact with the DOT EIP. Contractor web user interfaces Defines the web interfaces that will be used by vendors or DOT contractors to interact with the DOT EIP. Contractor/vendor procurement user interfaces Defines the web interfaces that will be used by vendors or DOT contractors to interact with the procurement section of the DOT EIP. Interface Components Component Description API gateway Allows external applications to call multiple DOT EIP internal components using a single request and then combine the multiple responses into a single response. DOT general user interfaces Defines the web interfaces that will be used by all DOT employees to interact with the DOT EIP. DOT analyst user interfaces Defines the web interfaces that will be used by DOT analysts to interact with the DOT EIP. Knowledge provider web user interfaces Defines the web interfaces that will be used by DOT and non-DOT knowledge providers to interact with the DOT EIP. DOT monitoring user interfaces Defines the web interfaces that will be used by DOT employees dedicated to DOT system monitoring to interact with the DOT EIP. DOT download user interfaces Defines the web interfaces that will be used by DOT employees or non-DOT users to export data from the DOT EIP. DOT map user interfaces Defines the web interfaces that will be used by DOT employees or non-DOT users to map the DOT EIP data. DOT data collection user interfaces Defines the web interfaces that will be used by DOT employees to interact with the DOT EIP during data collection talks or surveys. Knowledge consumer web user interfaces Defines the web interfaces that will be used by DOT knowledge consumers, a subset of DOT employees, to interact with the DOT EIP. Knowledge expert web user interfaces Defines the web interfaces that will be used by DOT knowledge experts to interact with the DOT EIP. DOT librarian user interfaces Defines the web interfaces that will be used by DOT librarians to interact with the DOT EIP. DOT maintenance user interfaces Defines the web interfaces that will be used by DOT maintenance employees to interact with the DOT EIP. DOT operation user interfaces Defines the web interfaces that will be used by DOT operation employees to interact with the DOT EIP. DOT procurement user interfaces Defines the web interfaces that will be used by DOT procurement employees to interact with the DOT EIP during the procurement process. Administrator web user interfaces Defines the web interfaces that will be used by the DOT system administrator to interact with the DOT EIP. IT investigation user interfaces Defines the web interfaces that will be used by IT investigation users, a subset of DOT employees, to interact with the DOT EIP. IT monitoring user interfaces Defines the web interfaces that will be used by IT monitoring users, a subset of DOT employees, to interact with the DOT EIP. IT operation user interfaces Defines the web interfaces that will be used by IT operation users, a subset of DOT employees, to interact with the DOT EIP. State web user interfaces Defines the web interfaces that will be used by state employees to Table 5-5. Interface components with descriptions for use in UML diagrams.

56 Guidance for Development and Management of Sustainable Enterprise Information Portals Internal DOT EIP Business Services Descriptions Access management Provide secure access to all the other DOT EIP components and manage users’ privileges. Analysis management Provide advanced analysis of DOT data such as traffic sensor data or driver’s behavior data to all the other DOT EIP components. Archiving management Provide archiving capabilities to the DOT EIP data storage and databases. Cache management Provide data-caching capabilities to all the other DOT EIP components. Computer vision management Provide image recognition capabilities to the DOT EIP videos and images. File exchange management Provide file upload and download capabilities to all the other DOT EIP components. Geo-analysis management Provide advanced analysis of DOT GIS or LIDAR data to all the other DOT EIP components. Ingestion management Provide large dataset ingestion capabilities to all the other DOT EIP components. Logging management Capture and track the activity of all the other DOT EIP components. Metadata management Maintain the metadata associated with all the DOT EIP data storage and databases. Notification management Provide notifications or alerts to all other DOT EIP components. Operation management Act on DOT systems components by all other DOT EIP components. For example, take a traffic camera offline or reboot a data-capture device. Persistent link management Provide all other DOT EIP components with persistent link pointing to DOT EIP content. This set of services allows DOT EIP content to be moved around without having to update all the other components with its new location. Publishing management Provide all other DOT EIP components with the ability to publish announcements both internally (email, wiki) and externally (social media, blogs). Rendering management Provide graphic rendering services to all other DOT EIP components. For example, the rendering of map tiles from raster data. Reporting management Provide report-generation capabilities to all other DOT EIP components. Search management Provide to all other DOT EIP components search capabilities across all DOT EIP data services. Service discovery management Defines the internal DOT EIP services that will be used as a registrar service for all other DOT EIP components. It will be used by each DOT EIP component to find the location of specific services within the DOT EIP clusters. Transcoding management Provide video transcoding capabilities to all the other DOT EIP components. Video stream management Provide video-streaming capabilities to all the other DOT EIP components. Visualization management Provide all other DOT EIP components with advanced visualization capabilities that they cannot themselves generate. An example could be the visualization of 3-D LIDAR datasets streamed to a mobile device. Workflow management Provide business rules and process management services to all DOT EIP components. Table 5-6. Business services and descriptions for use in UML diagrams.

Design and Implementation for Sustainable DOT EIPs 57 Data Services Descriptions Analytical database services Internal DOT EIP services that will be used to provide advanced data analysis capabilities to other DOT EIP components. An example could be a set of services performing text analysis on DOT documents or training prediction models using historical data. Archive storage services Internal DOT EIP services that will be used to provide other DOT EIP components with archiving capabilities. Content storage services Internal DOT EIP services that will be used to provide other DOT EIP components with content storage and distribution capabilities. Geo-capable NoSQL distributed database Internal DOT EIP services that will be used to provide scalable database services with geospatial search capabilities to other DOT EIP components. Geospatial RDBMS services Internal DOT EIP services that will be used to provide traditional geospatial database services to other DOT EIP components. Live sensor feed services Internal DOT EIP services that will be used to provide DOT EIP components with live sensor data feeds, for example from traffic speed sensors or CO2 sensors. Live video feed services Internal DOT EIP services that will be used to provide DOT EIP components with live video feeds, for example from traffic cameras. Metadata repository services Internal DOT EIP services that will be used to provide DOT EIP components with a set of taxonomies or ontologies to help organize and search DOT EIP content. Network and application logs services Internal DOT EIP services that will be used to provide DOT EIP components with services allowing them to log and track their activities. NoSQL distributed database services Internal DOT EIP services that will be used to provide scalable, non- relational database services to other DOT EIP components. RDBMS services Internal DOT EIP services that will be used to provide traditional relational database services to other DOT EIP components. External asset information services External services that could be used by DOT EIP components to collect asset information, for example from manufacturers’ web services. External document source services External services that could be used by DOT EIP components to collect documents outside of the DOT, for example from online libraries. External GIS data source services External services that could be used by DOT EIP components to collect GIS data outside of the DOT, for example from online GIS data libraries and registries. External knowledge source services External services that could be used by DOT EIP components to collect knowledge outside of the DOT, for example from online knowledge repositories. External state procurement services External services that could be used by DOT EIP components to collect data from other state agencies through state-maintained web services. Table 5-7. Internal and external data services and descriptions for use in UML diagrams. 5.4.2.2 Component and Data Flow Diagrams Component and data flow diagrams are designed to illustrate how data are processed by a sys- tem in terms of inputs and outputs. As the name of this diagram indicates, its focus is on the flow of information—where data come from, where they go, and how they get stored. The microservices architecture pattern greatly increases the amount of data flowing through an application and the complexity of the data exchanged between components, making it a challenge to diagram data flow. For purposes of clarity, the component and data flow diagrams created by the research team do not show all the components involved in the process and do not show every possible data exchange happening between components. Showing every component and every data exchange would render the diagrams unreadable. Instead, the team focused on

58 Guidance for Development and Management of Sustainable Enterprise Information Portals showing the main data exchanges between the service components involved in the high-level use cases and omitted some of the components that were less relevant to the use cases, such as logging and service discovery. The components that were not included in the data flow but are still involved in the process are represented at the top of the diagrams as unconnected hexagons. Figure 5-3 shows an example of a component and data flow diagram illustrating the flow of data among components (microservices) in the asset management and engineering services sce- nario. Note that each component of the component and data flow diagrams has been color coded according to the color pattern that was established in the block definition diagrams to differenti- ate interfaces, business services, and data services. Because of the large amount of data connec- tion between components, the team decided to represent each component as a hexagon rather than as a rectangle to increase the space for representing connection points between components and avoid having too many overlapping arrows on the diagrams. Also, due to the reactive and event-driven nature of the microservices architecture pattern, most data flow between compo- nent services is bilateral. Finally, to increase clarity, the team differentiated some of the data flow representation in the diagrams. Black arrows represent the main application data flow, red arrows represent the API gateway data flow, and dashed arrows represent the security data flow. 5.4.2.3 Activity Diagrams Activity diagrams are designed to represent workflows of stepwise activities and actions that support choice, iteration, and concurrency. Activity diagrams are intended to model both com- putational and organizational processes and show the overall flow of control. Figure 5-2. Asset management and engineering services block definition diagram.

Figure 5-3. Asset management and engineering services component and data flow diagram.

60 Guidance for Development and Management of Sustainable Enterprise Information Portals Figure 5-4 shows an activity diagram representing the possible flow of actions that could be taken during the asset management and engineering services scenario. In creating the activity diagram, the team focused on representing the high-level workflow that would need to be fol- lowed to perform the tasks occurring during each of the implementation examples. The activity diagram design is not intended to be exhaustive. 5.4.2.4 Sequence Diagrams Sequence diagrams are interaction diagrams intended to show how objects operate with one another during a specific scenario and in what order. Sequence diagrams use a Message Sequence Chart construct to represent the system components and their interactions arranged in time sequence. They depict the objects involved in the scenario and the sequence of messages exchanged between the objects needed to carry out the functionality of the scenario. Multiple scenarios and diagrams could be derived from the implementation examples, com- ponents, and data flow identified in the previous diagrams. The research team restricted these Figure 5-4. Asset management and engineering services activity diagram.

Design and Implementation for Sustainable DOT EIPs 61 diagrams to the single scenario of “DOT maintenance employee archives older asset data” for illus- tration. Other use case scenarios for asset management and engineering services would require their respective sequence diagram. Figure 5-5 shows a sequence diagram representing the chronological sequence of interactions between components (microservices) in the “DOT maintenance employee archives older asset data” use case of asset management and engineering services. Note that each component of the sequence diagram has been color coded according to the color pattern established in the block definition diagrams to differentiate interfaces, business services and data services. 5.5 Migration Strategy8 The process of transforming a monolithic EIP into a series of microservices is a form of appli- cation modernization. Traditionally, for a monolithic application, application modernization implies a complete rewrite of the entire application on the new platform. This is often referred 8This section is based on “Refactoring a Monolith into Microservices” (Richardson, 8 March 2016). https://www.nginx.com/ blog/refactoring-a-monolith-into-microservices/ Figure 5-5. Asset management and engineering service sequence diagram (DOT maintenance employee archives older asset data).

62 Guidance for Development and Management of Sustainable Enterprise Information Portals to as the “Big Bang” rewrite. It is often the only option when migrating an application to another monolithic platform; it can be extremely risky, and it will likely lead to failures that will postpone the modernization plan. Migration to microservices does not have to follow a complete application rewrite approach. Microservices-based systems are modular and as such are a perfect fit for an incremental refac- toring of the monolithic application. Monolithic application services can be gradually migrated out of the monolithic application and replaced by a new application consisting of microservices. The newly created service will replace the legacy of service and keep interacting with the remain- ing services operating on the monolithic application. Over time, the amount of functional- ity implemented by the monolithic application shrinks until either it disappears entirely or it becomes just another microservice. This strategy is akin to the challenge of servicing a car while it is being driven down the highway at 70 mph, but it is far less risky than attempting a complete microservices rewrite. 5.5.1 Extracting Services from a Monolithic Application A large, complex monolithic application often consists of tens or hundreds of modules, all of which are candidates for extraction. Figuring out which modules to convert first is often challenging. A good approach is to start with a few modules that are easy to extract. This will allow the development team to build experience with microservices, in general, and, more specifically, build experience with the extraction process. After that, the focus should be on extracting those modules that will give an agency the greatest benefit because converting a module into a microservice can be time consuming. The development team will want to rank modules on the basis of the benefit they would provide if they were migrated to a microservice. It is usually beneficial to extract modules that change frequently. Once a module has been converted into a service, it can be developed and deployed independently of the monolith, which will accelerate development. It is also beneficial to extract modules that have resource requirements significantly differ- ent from those of the rest of the monolith. It is useful, for example, to turn a module that has an in-memory database into a service that can then be deployed on hosts with large amounts of memory. Similarly, it can be worthwhile to extract modules that implement computationally expensive algorithms, since the service can then be deployed on hosts with lots of CPUs. By turning modules with unshared resource requirements into services, the development team can make their application much easier to scale. 5.5.2 How to Extract a Module The first step of extracting a module is to define a coarse-grained interface between the module and the monolith. A coarse-grained interface is most likely a bidirectional API, since the mono- lith will need data owned by the service and vice versa. It is often challenging to implement such an API because of the tangled dependencies and fine-grained interaction patterns between the module and the rest of the application. Business logic is often especially challenging to refactor because of the numerous associations between domain model classes. The development team will often need to make significant code changes to break these dependencies. Figure 5-6 shows the refactoring process. Once the development team implements the coarse-grained interface, the team turns the module into a free-standing service. To do that, the team must write code to enable the monolith and the service to communicate through an API mechanism. Figure 5-6 shows the architecture before, during, and after the refactoring.

Design and Implementation for Sustainable DOT EIPs 63 In Figure 5-6, Module Z is the candidate module to extract. Its components are used by Module X, and Module Z uses Module Y. The first refactoring step is to define a pair of coarse- grained APIs. The first interface is an inbound interface that is used by Module X to invoke Module Z. The second is an outbound interface used by Module Z to invoke Module Y. The second refactoring step is to turn the module into a standalone service. The inbound and outbound interfaces are implemented by code that uses an API mechanism. At this point, the development team needs to build the service by combining Module Z with a microservices framework that handles cross-cutting concerns such as service discovery. Once a module has been extracted, the development team has yet another service that can be developed, deployed, and scaled independently of the monolith and any other services. The team can even rewrite the service from scratch; in this case, the API code that integrates the service with the monolith becomes an anti-corruption layer that translates between the two domain models. Each time the team extracts a service, it takes another step in the direction of microservices. Over time, the monolith will shrink and the number of microservices will increase. Figure 5-6. Monolithic module refactoring.