Below is the uncorrected machine-read text of this chapter, intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text of each book. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.
25 : Principles and Considerations in Developing the Transactional Data Specification The first chapter provided a context for a transactional data specification, describing what it is, its benefits, and the challenges to development and implementation. This chapter begins a more in- depth discussion of the general principles and considerations applicable to data specifications for software applications, as well as functional and technical factors specifically relevant to an approach for a demand responsive transportation transactional specification. In this chapter, the focus switches from all stakeholders to those stakeholders who will actively use the specificationâand mostly likely be involved in some manner in the process of specification development, formal adoption, and subsequent governance. This focus on the stakeholders who will directly use the specification continues through Chapter 4. This chapter covers multiple topics, all of which are important to developing a transactional specification. It first describes core principles for designing the specification. Second, market considerations are discussed. Third, the specific functional requirements for a DRT transactional data specification are explained; a model of the âtrip lifecycleâ is presented to organize functional schema. Fourth, different technical models for implementing a DRT transactional data flow are described. Finally, a preferred technical approach for data communication of specification- compliant transactional messages is selected. In this report, several key terms concerning transactional data specifications are frequently used: ï§ Specification(s) ï§ Data Specification(s) ï§ Transactional Data Specification(s) ï§ Standard(s) (specifically referencing technical standards) Figure 2-1 defines these terms, which are related but distinct. Whichever specific term is being used, readers should understand that the overall context is always one of data specifications that are relevant to transactional processes.
26 2.1 Core Principles for Successful Specification Approaches The six key principles guiding the process of developing a successful specification are shown in Figure 2-2 and discussed in this section. Figure 2-2: Core Principles for the Specification Development â¢ Core Principal #1 Sufficiency. Sufficiency means that: (1) the specification covers all of the typical transactions that occur among parties; and (2) for each of the transactions covered, the specification incorporates all of the data elements needed for each party to accomplish that specific transaction. A specification is sufficient if it is capable of handling all âuse casesâ that Sufficiency Simplicity Flexibility Adaptability Compatibility Technical Appropriateness Specifications, Standards, Transactional Data Specifications: Understanding the Terminology Used in This Report Specification: a set of documented requirements to be satisfied by a material, design, product or service; the act of identifying something precisely or stating a precise requirement. Data Specification: a set of requirements for the types of data and the format of the required data elements for a specific data domain, typically related to a specific functional area. Transactional Data Specification: a data specification focused on enabling functional transactions that require the flow of data among 2 or more software systems. Standard: an established norm or requirement, usually a formal document, that sets forth uniform technical criteria, methods, processes and practices. Formal technical standards will typically have been ratified by some organization which represents the interests of the entities to which the standards apply. F Figure 2-1: Terminology for Specifications
27 typically occur within its transactional domain; each âuse caseâ describes a scenario in which two or more parties interact for purposes of transacting among themselves or exchanging information. â¢ Core Principle #2: Simplicity. Simplicity in the context of a transactional data specification largely connotes the absence of complexity in the message typology and message contents. Ideally, message types are simplified so that no further subdividing of the message would also produce useful messages. At the same time, each fundamental step in the transactional process should be able to be accomplished via a single message (and when appropriate, followed by an acknowledging response). â¢ Core Principle #3: Flexibility. A specification has flexibility when it can accommodate variations in transactional needs among participants. The same message types can support transactions in which only basic data elements are exchanged as well as those in which more detailed, albeit optional, data can be exchanged. â¢ Core Principle #4: Adaptability. A specification has adaptability when it can be modified to meet new needs without being fundamentally altered. For example, if new message types are added to handle transactional situations that are either new or different from prior practice, there should be no need to modify existing transactional interchanges that are not related to the new situations. In more technical terms, the changes should be âbackwards compatibleâ: what currently works continues to work after the specification is modified. â¢ Core Principle #5: Compatibility. A specification can be said to be compatible with existing practice when the types of data interactions it supportsâencompassing data elements, types of data, functional reasons for exchanging data, and means by which parties exchange dataâare already occurring and rely upon existing technical approaches and âprotocolsâ broadly consistent with those defined in the specification. A specification is not compatible with existing practice if it includes a data transmission approach and/or many data elements not commonly used in current practice in the domain which is its focus. â¢ Core Principle #6: Technical Appropriateness. A specification is technically appropriate when it employs the computer languages, technical terminology and technical approaches to data exchange that are in common use in the industry sector which is its focus, with particular reference to the software applications and computer systems of the major participants in that sector. 2.2 Market Considerations In addition to the importance of the specificationâs fidelity to the core principles, it is important that specifications work effectively for the groups who will use them in their day-to-day activities and that they work well with existing technology. The research team identified two important market considerations that could impact the type of specification proposed: 1. Major DRT service providers and DRT technology suppliers must be supportive or at least neutral towards specifications.
28 2. Specifications must build on trusted relationships and provide âgatekeepingâ mechanisms to mediate collaborative system-to-system interactions among parties. Given their potentialâand often actualâimpact on technology suppliers and their customers, new specifications in particular are the opposite of business as usual. They require significant âbuy-inâ from the leading parties, particularly the technology organizations. They also must provide for the detailed information needed by diverse DRT providers so they are functional. The development process needs to include expertise and strong industry/sector understanding on the part of those directly involved. Market is Supportive of or Neutral Towards Specifications To successfully develop and adopt data specifications they must have the support of the key organizations that they directly affect. The organizations principally affected are: â¢ DRT service providers who use the technology systems or components that are the focus of the specification (including public and private providers, private transit service contractors, taxi companies, and ride-hailing services); and â¢ Technology suppliers who work directly with the technology systems, e.g., software applications, hardware and other devices, computing infrastructure. Their products typically have functionality for booking, scheduling, vehicle routing, and/or dispatching. This also includes companies whose technologies are complementary to the core software systems, such as in-vehicle devices and fare payment solutions. The public sector organizations that fund public transit and human service transportation are also key organizations, but their overriding interest is that specifications exist rather than their specific form and detailsâhence they are not a factor in this consideration. If service providers and/or key technology suppliers are strongly supportive of a proposed new specification, this will significantly enhance the likelihood that agreement among participants can be reached on structure and details of the specifications. In interviewing key players, the research team found that the DRT technology industry is neutral about investing in development of specifications, with some in favor but also some large players who do not see the effort as in their interest. Providers, on the other hand, would generally like to have the benefits of transactional specifications but typically do not individually have the knowledge, resources, or organization to effect the change. Trusted Relationships and Gatekeeping Mechanisms In many industries, specification development and evolution is common and includes individuals from service and technology companies who are veterans of the process. This can greatly facilitate the achievement of a positive outcome. Unfortunately, such is not the case for the DRT sector given the absence of any prior experience with developing sector-wide specifications. When starting a new specification process with no prior history, the process itself will need to create the foundation of trust among participants. In such situations, there may be an enhanced need for mechanisms that provide participants with some assurances that technical and
29 business collaboration will proceed in a smooth, business-like manner and lead to actual results. âNeutralâ third parties and âobjectiveâ domain experts who organize and lead the specification process are useful to serve as a neutral party in launching the process. The use of such perceived neutral third parties and domain experts are examples of the âgatekeepingâ mechanisms that are important to mediate collaborative interactions among parties. This will extend to the actual technical details of system to system interactions, and there will be a strong need to ensure that concerns such as data privacy and security are handled appropriately by the technical solution. In some cases, transactions can be routed through âcertifiedâ gateways such as with standards-compliant payment processors, while in other cases it is the trust among the parties themselves that represents the gatekeeping mechanism. As such trust cannot be automatically assumed in a situation of developing new specifications, the ability to structure a process in which interactions among the parties creates such trust becomes very important. Since a data specification typically requires changes to technology systems, the organizations whose systems are affected have an incentive to minimize resources required to achieve compliance with the specification. In practice, this means that compatibility with such factors as the âvocabularyâ describing transactions, the data elements usually included in transactions, and the mechanisms for transmitting data among different applications/systems that are used by the major firms in the market represent the technical path of least resistance for advancing towards agreement on the many detailed elements of the specification. 2.3 Functional Requirements for DRT Data: The Trip Life Cycle In determining the scope of transactional data specifications, it is essential to consider functional requirements that must be supported by the software systems. The specification must support the trip from inception to completionâthe trip lifecycleâand enable the different organizations involved in its booking, planning and execution to share and view all relevant data about the trip during this process. The trip lifecycle is represented with five steps that are shown in Figure 2-3. Figure 2-3: Trip Life Cycle Steps Each of these five steps requires data to be exchanged by participants involved in the process. Some organizations require more data for the transactional process than others, but there is a core set of mandatory data elements needed to support each phase of the trip lifecycle. These data elements are briefly discussed in the following descriptions of each step in the trip lifecycle process. Step 1: Trip booking (reservation) In a trip reservation, data must be generated about the pickup and drop-off locations (addresses) of the traveler; when they wish to travelâwhich could be a requested pickup time or desired Trip Booking Scheduling/ Dispatching Cancellation (optional) Trip Execution Data Reporting
30 delivery time; if they have any special mobility needs (e.g., use a walker or wheelchair) and/or are using a mobility aid of some type; how many people are making the trip; and if there are any special instructions for the service provider and/or its driver, e.g., âpick up customer at back door.â In addition to these mandatory data items, there are other data elements that may be desirable or necessary for some organizations, such as the purpose of the trip or whether it is associated with a specific funding source. Step 2: Trip scheduling & dispatching For trip scheduling, the organization responsible for delivering the trip will determine when the vehicle will arrive at the pickup location, generate an estimate of when it will arrive at the delivery location, and provide information on time windows associated with the pickup and delivery times. Later in the scheduling process, a specific vehicle will be assigned to the trip. The information about the trip will then be dispatched to the vehicleâor more specifically, some type of digital device in the vehicle capable of receiving data messages about the passenger trips assigned to the vehicle and displaying this information to the driver. Step 3: Trip cancellation (optional) To cancel a trip, the only essential data elements are the time when the cancellation occurred and whether it is due to customer request or the inability of the operator to deliver the service. Step 4: Trip execution (delivery) In executing (delivering) a trip, the mandatory data elements are the identity of the traveler, the actual locations and times for pickup and drop-off, how many passengers boarded the vehicle, what fare was paid, and the identity of the service provider, vehicle, and driver that handled the trip. It may also be important to know whether a wheelchair or other mobility aid is used. The real-time location of the vehicle (via GPS readings) during trip execution is also important information for the service provider, and can be used as well as an input to the functional ability to inform the customer of when the vehicle is likely to arrive at their pickup location. Step 5: Trip data reporting In the final step, all pertinent data from the trip execution phase must be generated and sent to the entity that is responsible for the customerâs trip. In addition, the service provider must also communicate the cost (price) of the trip to the organization (or individual) responsible for the payment of the customerâs trip. The actual financial transactions associated with the trip, namely the payment to the service provider by the trip sponsor or individual responsible for this, are clearly important to participants in the system but extend beyond the operational elements of DRT services. As such, actual financial transactions are currently outside of the scope of the specifications described in this report. 2.4 Models for Transactional Data Flow A transactional data specification is a document that contains the necessary information for software developers and technology providers to construct data messages that make it possible for two or more computer applications to inter-operate. While the specification will typically have a precise syntax, with a defined âmessage setâ and required and optional data elements for each âmessage,â there is more than one paradigm for how the applications/systems could interoperate.
31 Although the specification syntax itself does not depend on which of these models is used, the technical approach to specification enforcement (i.e., validation of specification adherence) is strongly influenced by the application interaction paradigm used. There are several useful models for this type of application/system interaction. Three are explained briefly below, as they are the most directly relevant to a transactional data specification for DRT. Model 1: Request/response In a request/response model, one computer software programâcall it Application Aâsends a data message to another computer software programâcall this Application Bârequesting that the latter perform some specific action(s) related to a âbusinessâ transaction or potential transaction. For example, on a stock exchange, the computer system of a financial firmâ Application Aâmay send a request to the stock exchangeâs computer systemâApplication Bâto purchase 10,000 shares of Acme Widget stock at the current market price. Application B acts on the requestâit attempts to match the potential buyerâs request with that of one or more sellers of Acme Widget stock to fulfill the order for 10,000 sharesâand responds to Application A with confirmation that the request has been fulfilled. Alternatively, if there are no available sellers of Acme Widget stock, Application B will respond that the request has been unsuccessful. In this model, Application A cannot make any assumptions about the status of its request until it receives the response, which can be as simple as merely confirming that the request was received. For example, it might be the case that the protocol defined by the specification is for Application B to first acknowledge the request, namely that Application Aâs buy order has been properly received by the stock exchange computer system. Application B would then in a separate message respond with the status of the request, e.g., purchase request fulfilled or unsuccessful. Application A must wait until it receives acknowledgement that its message was properly receivedâand take appropriate action if it does not receive such an acknowledgementâuntil it can update the message status to âdeliveredâ and then expect a response directly regarding the disposition of its request. In this model, each of the collaborating software applications must manage the dialogue between them. The advantage of the request/response paradigm is that both applications can essentially guarantee that they have a consistent view of the status of messages. Only by receiving a response can the initiating application be sure that its message has been received and acted upon and what the disposition of its request was. The request/response approach does not necessarily imply a direct point-to-point data communication relationship between two applications. A message exchange or transaction broker could mediate between two applications and serve as the clearinghouse for message notifications and transmission; it would use the request/response paradigm to communicate with the end-point applications. For example, in this example, there could be third application, Application C, which operates for another financial firm and which also communicates with Application B (the stock exchange). Application C might send a message asking if there are any current potential buyers for Acme Widget stock. Application B can then serve as the transactional intermediary (which
32 is essentially what a stock exchange is) between Applications A and C, sending messages to each to enable them to bring the transaction to a conclusion; each interaction in the process involves a request/response sequence of messages. An example relevant to DRT would be that of a trip exchange, such as has been developed in Denver and is discussed in Chapter 3. In the trip exchange situation, a DRT providerâs technology systemâApplication Aâwould send an unfulfilled trip request to the exchangeâ Application Bâwith the objective of having another DRT provider actually provide the service for the trip in question. Using a request/response approach, the trip exchange would inform Application A via a message that it had successfully received and processed Application Aâs message, enabling the latter to know that there was no issue with the data transmission nor the contents of its message. If Application A did not receive a confirmation, it would be programmed to send the message again. Another DRT providerâs technology systemâApplication Câwould then be able to make a request to the trip exchange to send it any new trip postings since its previous request, and the trip exchange would respond with the trip posted by Application A (and any others that might have been posted). And this process would continue with requests and responses from the interacting software applications. Model 2: Publish and subscribe In a âpublish and subscribeâ model, an application âregistersâ with another system/applicationâsimilar to an individual signing up for an on-line service that requires them to provide an email address and mobile phone number as part of their accountâand subscribes to a service of the latter that informs it of events of interest. The events can be those that are of specific interest to the external application or more general in nature. For example, an individual with an on-line account for a national news service might wish to only be notified of the posting of any new stories about the city they live in, whereas other account holders might be notified of every new posting. It is the responsibility of the external application to determine what to do when it is notified that an event of interest has occurred and it has received the data about that event. For example, consider a situation in which a trip exchange exists for the purpose of enabling service providers and other organizations that have DRT service requests that they cannot themselves fulfill to âbuyâ service from another service provider with available capacity. The external DRT software applications can post trip requests to the trip exchange that are âvisibleâ to any other software applications capable of connecting to the trip exchange (this bears obvious similarities to the stock exchange situation described previously). Each trip request is specific to a service zone in the region in which the trip exchange operates. Using the âpublish and subscribeâ model, the trip exchange would send a message to all âsubscribersâ to a particular service zone notifying them whenever a new trip request for that zone is posted to the exchange. (Additional criteria could also be used to qualify the entities to which the notification is sent.) Depending on the design of the notification service, it might simply notify the external systems of the existence of a new tripâalerting those systems that they should request the new trip informationâor it might include the actual trip request âmessageâ from the originating systemâthat message being in the transactional specification
33 format. In the latter case, this âbroadcastâ functionality would be a core feature of the trip exchangeâs publish and subscribe approach. The advantage of the âpublish and subscribeâ model is that each participating software application is only notified ofâor sent the messages forâthose events it has interest in. In the absence of such a notification function, the external applications would need to periodically interrogate the trip exchange (or other source of potential transactions) and itself determine if the trip requests existing there are ones that it is already aware of or new requests. It can be easily imagined that this could become quite complex if there are a large number of trip requests whose status is frequently changing. With the publish and subscribe model, an application knows that only new or updated requests are publishedâand will be delineated as suchâand it becomes simpler to manage what could be a large number of events of interest. The âpublish and subscribeâ model is robust and has been used in many large-scale data interchange systems, such as for financial markets. It is particularly appropriate for exchange-type systems in which there are many transactions (or opportunities to transact) of potential interest, but different applications are focused on different subsets of those events. A brokerage approachâessentially an exchange-like hub which also makes transactional decisions such as which provider to purchase a trip from (as opposed to merely facilitating transactions in which the end point applications are the decision making points)â could also make good use of the publish and subscribe model. Model 3: Open API Framework An Application Programming Interface (API) is a software module that enables external applications or software systems to inter-operate with the software application providing the API. Over the past 25 years, APIs have become the most common method for business applications to communicate with external software systems/applications. An Open API is simply one in which the technical details of the API are published and available to a large community of other systems. Open APIs are not the same as open source software. There may be certain restrictions on what type of entities qualify to use the API and those entities are likely to be required to agree to terms of service, but in general an open API provides the external world with the ability to transact in some fashion with the host system/application. With an open API approach, each external application is fully responsible for managing its interactions with the system/application that provides the API (i.e., the host system). The host system API makes possible the transactional data flow, typically via a published set of function calls (i.e., message requests) that external applications can invoke to initiate a specified action or to retrieve certain types of information. It bears emphasizing that while the host system API defines the available functionalities, it does not typically manage transactions for the external systems. A directly relevant example of an open API that is heavily used is the Uber platform for ride- hailing services. Uber publishes the specifications for its ride-hailing platformâs API services, which provide external systems with the capability to request and manage trips and obtain information about those trips. (Uber has recently restricted access to who can use its API.)
34 In essence, all of the transactional services directly available to a consumer via the Uber app on their smartphone are also available to third parties via their own software applications: trips can be requested, estimated vehicle arrival times can be obtained, the identity of the vehicle and driver assigned to the trip is available and can be sent to the customer, etc. The major advantage of the open API approach is that a large ecosystem of applications can inter-operate without any formal structure for coordination (beyond the simple permissions to use another software systemâs API). The obvious disadvantage is that in the absence of a âuniversal APIâ for a particular industry or sector, technology providers will to find it necessary to support many different external APIs in order to inter-operate with actual or potential transactional partners. In fact, this can make common data specifications difficult if not impossible except on the terms of an entity that dominates the field, such as Google with its Google Maps API. One further aspect of interest concerning open APIs is that they impose no formal structure on how each external application manages transactional processes. This is a strength, as it provides a great deal of flexibility for the external application in how it implements its own transactional approach, within the constraints of the functions and data made available via the host systemâs API. However, each host system and external application may have a different transactional paradigm, meaning that across many such applications the âviewâ of transactions is unlikely to be identical. If more than a few applications are involved in the transactional ecosystem, this is likely to result in barriers to sector level transactional specifications, as well as the necessity of complicated software solutions to enable transactional data exchanges among multiple applications. These three models are briefly summarized below in Figure 2-4. Figure 2-4: Three Models for Transactional Data Flow Model 1: Request/Response â¢How it works: Request with response from receiver â¢Advantage: Clear sense of status of request for both sender and receiver â¢Disadvantage: Each party must manage every step in data exchange process Model 2: Publish & Subscribe â¢How it works: Applications subscribe to receive requests that meet certain criteria â¢Advantage: Helps manage large volume of potential requests â¢Disadvantage: No explicit model for transactions Model 3: Open API â¢How it works: software module enables external applications to interoperate with the application providing the API â¢Advantage: Large number of applications can inter- operate â¢Disadvantage: Need to support many APIs
35 2.5 Recommended Approach for Specification Transactional Flow Given the relatively undeveloped status of transactional data interaction in the DRT sector, it is prudent to begin with a specification implementation approach that is both agnosticâthat is, does not build on the transactional data practices of a dominant entityâand prescriptive. The latter means that each step in the process is well-defined and that all entities using the specification must adopt the same approach in their external facing transactional interactions. Moreover, given that this is the first step towards a common transactional data specification, the approach should be relatively granular and support data interchanges that are âatomicââeach transactional step is self- contained. These considerations point towards the use of a request-response approach as the core paradigm for managing transactional interactions at this stage of specification implementation. While this is a somewhat older style approach for system-to-system data interaction, it has the advantage that it enforces certainty in transactional data flow, verifying that each step in the process has in fact occurred. Moreover, the request/response approach has proven itself over many years in real life conditions in the demand responsive transportation sector in the Nordic countries. It is also the transactional data backbone of the most advanced multi-participant DRT system in the world, that of FlexDanmark in Denmark. Six regional public transit entities, over 500 transportation service providers, and many local governments and health care organizations use this technology approach to provide up to 24,000 trips per day. The approach relies on the SUTI standard (introduced in Chapter 1 and discussed in some detail in Chapter 3) and embodies the request-response/model of transactional data interchange. In addition, the request/response paradigm can co-exist with the use of publish and subscribe models of transactional data flow so it is flexible. This means that both bi-lateralâdirect data exchange between two applicationsâand multi-lateralâin which an exchange-type mechanism enables multiple applications belonging to DRT service providers and trip sponsors to participate in transactional data flowsâapplication interactions can be supported with an appropriate system design. The request/response model can be used for the former and the publish and subscribe model can be used for the latter. In fact, the trip exchange that has been developed in Denver does use elements of both approaches, as is discussed in Chapter 3. 2.6 Summary This chapter described the core principles and considerations required for development of a transactional data specification. Data specifications should strive to be as parsimonious as possible (simplicity) while maintaining flexibility. Such flexibility is important as the specification must be compatible with existing data systems but also meet the needs of different suppliers and users today and in the future. Market conditions and technical functionality that will affect the development of a transactional data specification for the DRT industry were also discussed; industry buy-in and governance considerations will be critical to the overall success of development and adoption of a specification for DRT transactional data. Finally, different models for supporting data interaction for transactional purposes were considered, and the request- response model selected as most appropriate for the initial implementation of a transactional data specification.