Transit Enterprise Architecture and Planning Framework (2011)

Below is the uncorrected machine-read text of this chapter, intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text of each book. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.

11 To develop an assessment of the state of the practice, the research team reviewed available industry literature and con- ducted telephone interviews with a sample of transit agencies as well as several state DOTs. The literature search and inter- views covered the five major elements included in the TEAP Framework: • Enterprise Architecture and Enterprise Architecture Plan- ning (EA/EAP) • Business Case Methodology (BCM) • ITS Funding Implementation (FI) • Systems Engineering (SE) • Post-Implementation Review (PIR) To provide a reasonable sample of agencies for the tele- phone interviews, a group of 14 transit agencies and three DOTs was selected for interviews. Survey protocols were developed for the interviews. A standard set of interview questions was administered to all the agencies. In addition, some agencies were asked more detailed questions on some Framework areas, if the screening questions discovered issues to probe further and if time was available. The first column in Table 2 shows the agencies and state DOTs that were interviewed. Several agencies were asked more detailed questions about their experience with the Framework areas. The checked columns in Table 2 represent the agencies that were surveyed in more detail on selected topics. The “Gen- eral” column refers to the standard questions asked of every interviewee. Summary of Results Interviews included questions from the general survey as well as more detailed surveys for each of the five Framework areas. The following sections summarize the findings. Enterprise Architecture and Enterprise Architecture Planning (EA/EAP) The scan of the transit industry revealed limited adoption and understanding of EAP. Overall, few “lessons learned” emerged through the industry scan because few organizations engage in planning and documenting their EA. Industry lit- erature related to transit ITS technology deployment is rife with examples about how the lack of enterprise architecture planning is limiting success in system deployments. Among the organizations interviewed, most of the Chief Information Officers (CIOs) or IT managers were familiar with the concept, particularly if they came from other indus- tries. However, few had the resources or management support to undertake a comprehensive enterprise architecture planning process. Fewer were versed on the “segment architecture” approach currently applied by other industries. WMATA and Miami-Dade Transit were the two transit agencies that said they were working on EAP. Details of these two implementa- tions are described in the State of the Practice Synthesis (see Appendix B). Business Case Methodology (BCM) The screening survey included questions about the organi- zation’s use of a BCM, verified terminology, and asked about the use of a Return on Investment (ROI) analysis and other cost-related analyses in justifying an IT/ITS project. Addi- tional follow-up questions were asked of a subset of respon- dents. The details of the responses and resources are available in the State of the Practice Synthesis (see Appendix B). This section includes several questions and a summary of the gen- eral responses. In addition, the section elaborates on several organizations that have a process in place to apply business case analysis as part of their IT/ITS project approval process. C H A P T E R 3 State of the Practice

Does your organization have a process for proposing, justifying, and approving an IT or ITS investment (a BCM)? Approximately one-half of the organizations had some sort of process, whether it was IT/ITS-specific or the general agency budget approval process, for proposing, justifying, and approving IT/ITS investments. Only a few of the agencies had an IT/ITS-specific process that provided templates and guidance for staff that needed to initiate and justify a project. Some respondents said their organization used consultants to build the justification for a project. Another said, “Nobody in our organization formally requires a BCM process, we have standard budget justification forms, but no official BCM doc- ument or process. However, we end up doing some of a BCM’s steps to justify the project to the management and Board as part of the budget process, and because it’s helpful.” TriMet. The public transportation agency for the Portland, OR, metropolitan region (TriMet) felt that the BCM should be simple, clear, flexible, and understood by all the stakehold- ers. Flexibility was important, so the business case could be scaled based on the size and complexity of the project to ensure it would be used for all projects and not be skipped because of an onerous process. Basic templates are available for the Project Charter, the Planning Report (which is shown in Appendix A), Alternatives Analysis, and other aspects of justifying the project. They stated that the analysis should consider all the system life cycle stages, including feasibility, design, development, implementation, operation and main- tenance, and the end of the life cycle when the system is ter- minated or replaced. Further, TriMet has a project sponsor for each project, with “. . . responsibility for approving budget, schedule and scope changes, deciding the issues to be presented to other stakeholders and for accepting the final work product. The sponsor is typically the most senior person from the business unit needing the work who will stay informed of and involved in the project.” In their BCM, the project sponsor has a quick reference document with checklists to help them in their role of facilitating the project’s success. Examples of some of the project sponsors’ checklists, which help them do their job, are included in the State of the Practice Synthesis (Appendix B). WMATA. WMATA is working on the development of an Enterprise Architecture and also has a project management methodology that it uses. As a result, their BCM includes a reference to the Enterprise Architecture. The project manage- ment methodology includes a Business Plan Initiation (BPI) process, although the process does not always require a justi- fication for all projects. The BPI feeds into the capital plan- ning framework for all projects. A streamlined form for the Business Plan Initiation Review process and instructions for completing the form are included in the State of the Practice Synthesis (Appendix B). The form summarizes all the project justification documents. King County Metro (KCM). Over the last 15 years, King County Metro has used a couple of different processes for developing a business case. Currently, KCM must use King County government’s process for justifying and approving IT/ITS projects. The process is described in a 69-page docu- ment titled, “Project Manager Guide to PRB Reviews,” which also references other documents for additional guid- ance (1). Two tables from the Guide, which show the sug- gested deliverables for Phase I (called Project Planning) and for Phase II (called Project Development), which in King 12 Agency General EA/EAP BCM Funding Systems Eng. PIR C-Tran Hampton Roads Iowa DOT Kansas DOT King County Metro Lynx MARTA Miami-Dade NY State DOT Paducah RIPTA River Bend SEPTA TriMet UTA Wichita WMATA ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓✓ ✓ ✓ ✓ ✓ ✓ ✓ Table 2. Transit agencies and state DOTs interviews for industry scan.

County’s process includes the “business case,” are included in the State of the Practice Synthesis (Appendix B). The Proj- ect Planning phase is typically completed as a preliminary request for funding to further build the business case in Phase II. King County employs a gated process, with fund- ing released by project phase. Does your organization use the term “Business Case Methodology”? Only one respondent said that the term “Business Case Methodology” was used in his or her organization. A few respondents wanted to know what the term meant before answering the question. Terms that were used for their agency’s process for approving IT/ITS projects included Business Case, QBC or Quantified Business Case, Phased Gate Review, and BPI or Business Plan Initiation. In a Phased Gate Review process, a management review event occurs between specified project phases to determine if the project should proceed “through the gate” to the next phase. Does your BCM vary by type of system or IT/ITS project? If so, how? Of those agencies with a BCM, all allowed for lesser detail in describing the business case, depending on the size and perceived risk of the project. Some skipped steps when they knew the project was required. Others were acutely aware of the costs of doing the analyses and wanted to keep the level of effort commensurate with the estimated project costs, com- plexity, and risks. King County provided the only form for determining the level of oversight a project might require, which drives the number and detail level of the forms to be submitted. The four categories of factors used to determine project risk rating are project size, project manager experience, team experience, and project type. The Project Size factor rates the project on size, primarily based upon onetime cost estimates and, sec- ondarily, on project duration. The Project Type factor rates the technical complexity of the work being undertaken. If yes, does the BCM consider: (Operations and maintenance costs and requirements, agency architecture, regional ITS architecture, integration options, other enterprise-wide thinking)? All the business case methodologies took into considera- tion operations and maintenance costs. The business case methodologies also considered one or more aspects of the agency architecture and/or the regional ITS architecture. One of the King County BCM forms had a checklist of tech- nical outcomes which included “Leverages and/or extends integration architecture.” WMATA’s Business Plan Initiation form includes “Implement Authority-wide Integration” as an IT priority. In your organization, what have been the benefits and issues pertaining to completing a BCM? TriMet felt that the BCM helps with ensuring a common understanding of the project and helps manage expectations. High-level documentation of the project from the BCM and project management process is available for stakeholders to access (they have it in a database). Standardization of the steps helped simplify training on the process, helped readers quickly find information, and helped somewhat with comparisons between projects. At Metropolitan Atlanta Rapid Transit Authority (MARTA), the head of IT said, “You are relating what you want to do to the business needs, costs, and impacts. You show why the project should be done, not just providing an opinion or gut feel.” Issues pertaining to the BCM included finding the time and resources to do the analyses. Finding the data to do the ROI was also cited as an issue. A concern was stated that some- times, for some projects, the process can take so long that the user requirements and technology options change before the project is started. Does your organization usually perform a Return on Investment (ROI) analysis as part of the IT/ITS project justification process? A majority of the respondents said their agency had con- ducted an ROI analysis on one or more IT/ITS projects. More than one respondent was unclear on the difference between a cost-benefit analysis and an ROI analysis. “ROI analyses” were conducted on key projects at some agencies that did not have a BCM. Conversely, the existence of a BCM at an organ- ization did not mean that an ROI analysis was always com- pleted on a project, although some level of cost analysis was always done. Other cost-related analyses completed when a new project is being justified. Many of the agencies completed some form of a cost-benefit analysis. For a subset, Total Cost of Ownership was calculated. King County has a process for completing a “Quantified Business Case.” Another said, “they consider if the overall cost exceeds the benefits.” 13

Does your agency have a formal process for comparing and selecting among different proposed IT/ITS projects? If a respondent said their organization did not have a BCM, they were not asked this question. Mostly the answer to this question was “no,” although several said that having a stan- dard form for proposing projects helped with the comparison process. TriMet said they had a three-category classification of projects, which are Mandatory, Highly Recommended, and Discretionary. Others said that their organization had tried different approaches but did not currently have a repeatable process in place. MARTA is pleased that the selection of projects is done through the IT Governance committees, which include tran- sit management. At their agency, users prioritize all the IT projects. This relatively new process “ended the old user com- plaints about IT pushing them.” IT/ITS Funding Implementation Transit agencies are applying the full range of financing mechanisms to make IT/ITS investments from large enterprise technology replacement projects to small automated vehicle location (AVL) projects. Pay-Go is the primary financing mech- anism used by most transit agencies. However, comingling of funds and public-private partnerships (PPP) are starting to be used more frequently. For example, Salt Lake City Utah Transit Authority (UTA) co-mingled $12.3 million to acquire an account-based fare col- lection system and a performance reporting system. WMATA is pursuing a public-private partnership to finance, design, implement, operate, maintain, and manage content of a streaming video advertisement and passenger information system called “The Metro Channel.” Southeastern Pennsyl- vania Transportation Authority (SEPTA) is another transit agency considering an ITS public-private partnership, in their effort to replace an antiquated fare collection system. Table 3 summarizes how 12 transit agencies participating in the survey financed their IT/ITS projects. Systems Engineering (SE) In order to determine where transit agencies stand on under- standing and use of Systems Engineering (SE) for ITS project development, a portion of each transit agency interview was devoted to the use of SE. For several of the agencies that had recent experience with the systems engineering process, an additional set of interview questions was posed to assess whether the agencies had seen benefits from their use of the Systems Engineering process, particularly the process recom- mended by the U.S. DOT guidance. The discussion below highlights the key findings from the interviews. Use of the SE Process by Transit Agencies Almost all of the agencies interviewed indicated they used some type of development process or did some aspects of the SE process. Only two answered “no” or “not really” to the basic question, “Do you use a Systems Engineering Process for project/system development?” A closer examination of the interview responses shows that about one-half of the agencies could be described as having a development process, and of these only a couple are really using the SE process. Why the discrepancy? There are several key reasons: • Low level of knowledge of the SE process among agency personnel. In several cases, the agency response was that we do whatever parts of the process the contractor pro- vides. It seems in some cases the agencies are content to rely solely on whatever level of expertise the contractor provides. In one or two of the agencies they specifically hire a contractor to be their system engineer, providing the SE expertise that they lack. • Existing project management or system development processes. Several of the agencies that could be considered more advanced (based on the number and scope of their ITS deployments) have a definite process orientation, but in most cases this orientation is strong on project management (or in one case business management) but not strong in the technical development process that systems engineering represents. Because of the project management focus, these agencies have a structured view of tracking the project’s progress against cost and schedule. They may also have detailed consideration of such cross-cutting activities as risk management. However, what these processes lack is the technical development process, with its Concept of Opera- tions (focusing on the stakeholder needs and the operational scenarios of the systems), formal requirements definition, design tradeoffs, and verification against requirements. They each cover parts of these activities (most often the require- ments definition), but not all of them. 14 Funding Approach Number of Agencies Using Funding Approach, N=12 5 gnicnaniF tbeD 2 gnicnaniF esaeL latipaC Public-Private Partnerships 3 2 tnemecnahnE tiderC 21 oG-yaP 21 gnilgnim-oC Source: TCRP Project J-09, Funding Implementation Survey (January 2009). Table 3. Transit agency funding of IT/ITS projects.

• Transit agencies have in general not been required to use the SE process. Although FTA policy on ITS projects requires an SE analysis for each project using federal funds, the requirements do not cover the full range of the SE process, and can be met by cherry picking info from a far less systematic development process. Two of the agencies inter- viewed were required to closely follow the U.S. DOT SE process. They were developing systems under the Mobility Services for All Americans (MSAA) Initiative grant. The ini- tial phase of these projects developed the Concept of Oper- ations and functional requirements for the system, caused each agency to become knowledgeable of the U.S. DOT SE process and required the agency to utilize the process in the project development. As will be discussed below under “Benefits of Using the SE Process,” both agencies felt it was a worthwhile exercise and plan on using the SE process for future efforts. Benefits of Using the SE Process The question posed to transit staff was, “Have you derived benefits from using the Systems Engineering process?” The answer was a resounding yes. Some of the benefits they iden- tified were: • Using the process helped the agency and the other stakehold- ers go through each step rather than jumping to the end. • The SE process helps the agency keep the project on sched- ule and budget. It allows the agency to have better visibility into the contractor’s progress through the outputs. • Using the process saves the agency a lot of trouble at the backend of the project because the surprises are minimized. • The Concept of Operations made the agency and the rest of the stakeholders more aware of how the parts of the system will integrate and work together. Post-Implementation Analysis (PIA) The transit agencies that were surveyed had varying levels of understanding of post-implementation analysis, or as it is called in other industries, Post Implementation Review (PIR). In addition, post-implementation analysis (PIA) was called different things in the various agencies, so additional prompts and follow-up questions were needed to clarify what was being discussed. Does your agency have a PIA or evaluation phase for IT/ITS projects? With the exception of a few of the transit agencies that were surveyed, most of the respondents described relatively little consistent PIA activity. In a few cases, PIR was confused with system acceptance or project closeout activities. The majority of the agencies surveyed did not have a formal PIA process. Of those that did, it was only sometimes or informally fol- lowed by a subset of those respondents. One respondent said their reports had varying levels of formality, but they usually included lessons learned, performance goals, and compar- isons against initial model forecasts. Terms used to describe PIA activities or processes included post project assessment, benefits realization step, evaluation, feedback, earned value management analysis, and validation. When the transit agency’s PIA had some form of specified pro- cedures, it was generally because the organization’s central IT staff had a System Development Life Cycle (SDLC) methodol- ogy that included a post-project-closeout analysis step. An interesting, related comment from MARTA was that they have hired staff to be an in-house, independent verification group that analyzes a new system prior to system acceptance (they complete the SE verification process step). This group and process have “paid off in dividends.” King County Metro has extensive, detailed documentation and requirements for how project managers will run their IT/ITS projects and document their activities. More informa- tion about the process and the Benefits Realization Report that is due a year after project close-out is in the State of the Practice Synthesis in Appendix B. What is the time frame for measuring/evaluating the results of the IT/ITS project? The time frame for completing PIAs varied, but most were completed within 1 year of system acceptance. The Utah Transit Authority (UTA) has an interesting approach that includes two phases. First, it obtains feedback on the system from the customer within 30 days of system accept- ance. UTA is certified in and applies International Organiza- tion for Standardization (ISO) 9001 Quality Management Standards, so this feedback is part of a regularly followed process. UTA strives to monitor, measure, and report on whether the project met the agreed-upon quality, schedule, and budget expectations defined in the scope, while acknowledging that all categories are subject to change requests that can modify expectations to the scope. UTA has another regular post-implementation practice, although there is no form for it. An IT supervisor or the proj- ect manager always checks back on the new system, generally after it’s running for 3 to 6 months (maximum 1 year) to see if anything else could have been done differently. They look for lessons learned or needed system adjustments, as well as using it as an opportunity to keep up with changing business needs. The King County Metro Transit Signal Priority (TSP) team completes its “before” and “after” data collection efforts 15

immediately surrounding a new installation to have as similar as possible “before” and “after” operating conditions (usually 2 weeks before and 2 weeks after). Who or what is the driver for having a PIA? A variety of reasons were given for doing a PIA. Some agencies cited policy or practice. Another said ISO standards and procedures, as well as it being critical for providing good customer service. Other answers included the following: • Federal requirements • Usually we think it is the right thing to do • Grant requirements • When a project manager pushes for it • When it is a problematic project or one with lots of conflicts • When someone promised cost savings and now we have to find them • We have to justify why it cost so much • We want the lessons learned to improve practices and procedures • We want to know how to improve the system in the enhance- ment phase and if it is needed How are the results used? The most common answer was that the lessons learned were valued for improving future projects. The results were also used to guide the next set of enhancements for the new project or to identify new business requirements. The Utah Transit Authority used the PIR process for several purposes. Documenting PIR results from all of the IT/ITS projects “allows you to go back and see what you did and learn from errors.” From an IT perspective, “one of the best values is the alignment of the requirements and the deliverables (was it that the client changed their mind or that resources changed?). Feedback helps you clearly know what the clients think. It’s time consuming, but good. It just takes lots of time.” The TSP team at King County Metro uses the evaluation results in a number of different ways. They use the feedback for adjusting and fine-tuning the TSP system, for TSP staff train- ing and education, and for determining whether or not to shut down a location with poor performance. In addition, the analyses have helped them contribute to the industry’s knowl- edge about TSP in talks, papers, and during the development of the Transit Communications Interface Profile (TCIP) stan- dards. Finally, they use the evaluation data to help determine where to put the next TSP installation, where to do improve- ments, to estimate how much time each vehicle spends on every block of the street and to provide the data to others in the organization who want it. One of the biggest benefits is that it helped build tools, such as the TSP Interactive Model (cost-benefit model), for creating more effective installations. Does your agency apply the PIA process to all or some of its IT/ITS projects? Three of the agencies said they do some PIA regularly after an IT/ITS project has passed systems acceptance. Most said they would try to do more in the future. What are the biggest issues in completing the analyses? For those agencies that completed post-implementation analyses, time, money, gathering data, and motivation were issues in completing the work. For some, after the project was over, they felt pressure to either work on enhancements or move on to a new project. Another said that it is a struggle to obtain data for a good ROI analysis; they use the cost/benefit analysis portion of the ROI more as a planning tool for decid- ing between implementation options. 16