Click for next page ( 50


The National Academies | 500 Fifth St. N.W. | Washington, D.C. 20001
Copyright © National Academy of Sciences. All rights reserved.
Terms of Use and Privacy Statement



Below are the first 10 and last 10 pages of uncorrected machine-read text (when available) of this chapter, followed by the top 30 algorithmically extracted key phrases from the chapter as a whole.
Intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text on the opening pages of each chapter. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.

Do not use for reproduction, copying, pasting, or reading; exclusively for search engines.

OCR for page 49
49 APPENDIX B State of the Practice Synthesis

OCR for page 49
50 TABLE OF CONTENTS 52 1 Introduction 52 1.1 Scope of State of the Practice Synthesis 52 1.2 Project Overview 52 1.3 Background 53 1.4 The Synthesis Topic Areas 53 1.4.1 Enterprise Architecture (EA) and Enterprise Architecture Planning (EAP) 53 1.4.2 ITS Implementation Funding 54 1.4.3 Business Case Methodology 54 1.4.4 Systems Engineering 54 1.4.5 Post-Implementation Evaluation 55 2 Synthesis Methodology and Industry Scan 55 3 Findings on Transit Enterprise Architecture Planning and Enterprise Architecture 56 3.1 Main Purpose of an Enterprise Architecture 58 3.2 General Approach to EAP/EA Used by Other Industries 58 3.1.1 Federal Enterprise Architecture (FEA) 60 3.1.2 The Open Group Architecture Framework (TOGAF) 60 3.1.3 Industry Implementation Approaches 61 3.1.4 Lessons Learned on EA/EAP from Other Industries 62 3.2 Transit Approaches to EAP/EA 62 3.2.1 Transit EAP/EA: Lessons Learned from the Literature 63 3.2.2 General State of Enterprise Architecture Adoption by Transit Agencies 63 3.2.3 Miami-Dade Transit 64 3.2.4 Washington Metropolitan Area Transit Authority (WMATA) 67 3.2.5 Other Transit Approaches to Enterprise Architecture Planning 72 3.3 Next Steps 72 Chapter 3 Appendix A: Enterprise Architecture Definitions from the Literature 72 Enterprise Architecture 73 Enterprise Architecture Planning 73 Framework 74 Chapter 3 Appendix B: FEA Segment Architecture Description 75 Chapter 3 Appendix C: Description of The Open Group Architecture Framework (TOGAF) 76 4 Findings on Transit IT/ITS Implementation Funding 76 4.1.1 General Findings from the Literature 76 4.1.2 General Approach to Implementation Funding 78 4.2 Transit Capital Investment Needs and Funding Approaches 78 4.2.1 Transit Funding Needs 78 4.2.2 Capital Funding Sources 80 4.2.3 Financing Mechanisms 81 4.2.4 Repayment Streams

OCR for page 49
51 81 4.3 Key Findings on Transit Agencies Implementing IT/ITS Funding 81 4.3.1 Salt Lake City Utah Transit Authority (UTA) 82 4.3.2 Washington Metropolitan Area Transit Authority (WMATA) 82 4.3.3 Metropolitan Atlanta Rapid Transportation Authority (MARTA) 82 4.4 Constraints on Funding Approaches: 82 4.4.1 Eligibility Requirements 83 4.4.2 Marketability 84 4.4.3 Risks and Uncertainties 85 5 Findings on Business Case Methodology in Transit Synthesis 85 5.1 What is a "Business Case"? 85 5.2 What is a "Business Case Methodology"? 85 5.3 Why is having a Business Case Methodology Valuable? 86 5.4 Differences Among Business Case Methodologies 86 5.5 Examples of Possible Topic Areas in a Business Case 87 5.6 Transit BCM Survey Results 89 5.7 Other Approaches to a BCM 91 5.8 Recommended Practices for BCM 92 5.9 Business Case and an Enterprise Architecture 92 5.10 Realizing Benefits 92 5.11 Have Buy-in and Get the Sign-offs 92 Chapter 5 Appendix A: Planning Report Template from TriMet 94 Chapter 5 Appendix B: WMATA's Streamlined Form for the Business Plan Initiation (BPI) Review Process and Instructions for Completing it 103 Chapter 5 Appendix C: KCM's Form for Determining Extent of Oversight Process 109 Chapter 5 Appendix D: King County Suggestions for Project Review Board Deliverables 113 6 Findings on Systems Engineering and Transit 115 6.1 Literature Review 117 6.2 Interview Findings 117 6.2.1 Use of the Systems Engineering Process by Transit Agencies 117 6.2.2 Benefits of Using the Process 118 6.3 Recommendations 118 7 Findings on Post-Implementation Analysis in Transit 118 7.1 Approach/Methodology 118 7.2 What is Post-Implementation Analysis? 119 7.3 PIR Benefits: Why is Post-Implementation Analysis Valuable? 119 7.4 Post-Implementation Review Process Overview 119 7.4.1 Planning the PIR 120 7.4.2 Preparing for the PIR 120 7.4.3 Conducting the PIR 120 7.4.4 Reporting/Follow-up 120 7.5 Transit Survey Findings 121 7.5.1 King County Metro (KCM) 122 7.5.2 Other Transit Discussion of Post-Implementation Analysis 123 7.6 Non-transit Approaches to Post-Implementation Analysis 123 7.7 Issues and Barriers Related to Post-Implementation Analysis 124 7.8 Recommended Practices for Post-Implementation Analysis 125 7.8.1 Checklist for Managers 125 7.8.2 Transit Manager's Roles 126 8 References

OCR for page 49
52 1 Introduction the project implementation effort, and measure results and benefits. 1.1 Scope of State of the Practice Synthesis The project consists of two phases. During Phase I, the This document, the state of the practice synthesis for the Research Team will complete the syntheses and develop the Transit Enterprise Architecture and Planning Framework details of the framework for improving ITS project deploy- project, consists of the deliverables for the following five ments. As early Phase I deliverables, the syntheses describe project tasks. current industry practice through a review of the literature and interviews with transit industry professionals, and identi- Task 1: State of the Practice--Enterprise Architecture fies industry readiness for adopting best practices in the five Task 2: State of the Practice--IT/ITS Funding Implemen- specific disciplines associated with deploying ITS projects. In tation subsequent deliverables, the Transit Enterprise Architecture Task 3: Transit Agency Situational Analysis: State of the and Planning Framework will be described. It will include a Practice--Business Case Methodology high level overview for an executive management audience, Task 4: State of the Practice--Systems Engineering Imple- details on how to develop an enterprise architecture that aligns mentation technology investments with business needs, guidance on how Task 5: State of the Practice--Post Implementation Analysis to show the relationships among ITS business processes, per- formance, information, services and technology, and exam- The five tasks relate to important disciplines that contribute ples and templates. During Phase II, key aspects of the TEAP to the successful planning, funding, development, and deploy- framework will be field tested and demonstrated through the ment of transit Intelligent Transportation Systems (ITS) proj- EA/EAP tool(s) implementation. ects. Chapter 2 includes an overview of the methodology used to develop the syntheses, which included a review of the liter- 1.3 Background ature and interviews with transit agencies and the Depart- ment of Transportation (DOT) for a few States. The synthe- The five disciplines addressed in the syntheses, which will be ses developed for the five tasks are included in Chapters 3 included in a framework for successfully deploying transit ITS through 7. projects, are often poorly understood and executed in transit The purpose of the synthesis tasks was to obtain a better as well as other industries. This is due to several factors: understanding of current industry knowledge and practice in the five topic areas, as well as to observe the state of readiness Lack of time and resources for training on the topics for transit to adopt industry best practices. Subsequent deliv- Lack of time, resources and corporate support for imple- erables will address and recommend best practices within a menting the disciplines framework that blends these five disciplines into a seamless, Lack of materials that tailor the topics for transit to make consistent Framework. them relevant rather than complex and theoretical As competition for limited resources increases, the value 1.2 Project Overview and need for skills in building a good business case, arranging The Transit Enterprise Architecture and Planning Frame- funding, using EAP to improve the value of the investment, work project seeks to provide transit agencies with a managing projects with good systems engineering practices, roadmap, based on a Transit Enterprise Architecture and and proving value with post-implementation analysis, will Planning (TEAP) Framework, to successfully implement increase. Information Technology (IT) and ITS technologies that Further, in transit as well as other industries, the relation- meet their business needs. The project includes a prelimi- ships between the five disciplines are typically not well laid out nary assessment of the industry and tools available (the syn- and understood. In Task 4 of this project, the Framework and thesis tasks) and the development of a framework and a relationships will be described. An enterprise-wide framework process, supported by tools to assist agencies in implement- approach to project planning better enables the identification ing IT/ITS technologies. The Framework and tools will help of the impacts on people, systems and technologies over the transit professionals understand the financial, operational lifecycle process, as well as the meeting of agency requirements. and management impacts of technologies, to help them bet- Specifically, a Framework guides transit in: ter meet their enterprise business process needs and corpo- rate objectives. The Framework will also help guide an agency's Planning how information, services and technology work planning process, improve its understanding of risks, and together across an enterprise to support business processes, validate and verify compliance with its needs, better manage solve problems, and measure performance;

OCR for page 49
53 Promoting information sharing across agency and institu- ITS Implementation Funding (how to pay for it); tional barriers; Business Case Methodology (how well does this project fit Ensuring that IT/ITS projects are defined and staged in a into the your stated priorities; what are the risks, benefits way that ensures best value and supports the successful and costs, and estimated return on investment [ROI]); implementation, operations and maintenance; System Engineering for helping design and manage an ITS Ensuring that the benefits and costs of proposed IT/ITS proj- Project implementation; and ects are understood across the project's lifecycle (including Post-Implementation Analysis for assessing system perform- operations and maintenance), and resources are available to ance (including reviewing the experience for lessons learned) support the program; and meaningful (estimated) ROI. Specifying IT/ITS projects to maximize the IT/ITS invest- ment decisions across the organization; These topic areas and concepts are described in Chapters 3 Ensuring that IT/ITS projects are described to meet stake- through 7. The following highlights the intent and scope of holder needs, requirements are explicitly described, risks are each chapter. identified and mitigated, and the system development process is managed to ensure correct operations and requirements are 1.4.1 Enterprise Architecture (EA) and Enterprise met; and Architecture Planning (EAP) Describing the leadership and organizational structures and processes that ensure that the organization's IT sus- The Enterprise Architecture Planning process is a set of tains and extends corporate strategies and objectives (1). activities used to develop the Enterprise Architecture mod- els, diagrams and descriptions. The process typically is stake- holder driven where the current performance measures, 1.4 The Synthesis Topic Areas business processes, data, applications and technologies The five synthesis topic areas provide tools for planning, that are used in the organization are documented. The developing, deploying, and evaluating the systems and tech- next step consists of documenting where the organization nologies that best meet an organization's objectives. These topic wants to be with respect to its business in the future, about areas, which will become part of the Framework, are: four to five years. The organization consists of the corpo- rate mission, goals, objectives, and the business processes, Enterprise Architecture Planning (EAP) and Enterprise data, applications and technologies that are needed to sup- Architecture (EA) development process (developing the port that vision. This is called the "to-be" architecture. A blueprints); third step is a description of how to get there or a descrip- tion of the "gap" between the current ("as-is") and the future ("to-be"). The Enterprise Architecture, both the "as-is" and "to-be" architectures, are composed of four or Enterprise five models (depending on which Methodology is used-- Architecture see Chapter 3 Appendix A) that are depicted in one or more Planning (EAP) diagrams, policy statements, procedures, inventories or other and EA "artifact." Chapter 3 on EAP/EA describes various IT industry Post- Implementation approaches to EAP processes and EA artifacts. The transit Implementation Funding Analysis Strategies industry does not have a track record in EAP (with only two agencies having documented their enterprise-wide EA); how- ever, some transit agencies are deploying segments of their architectures, through enterprise data, enterprise applica- tions, or during their business case and systems engineering Systems Business Case processes. These examples will be discussed in greater detail Engineering Methods in Chapter 3. 1.4.2 ITS Implementation Funding Chapter 4 on ITS Implementation Funding discusses guide- Figure 1. Synthesis topic areas. lines for obtaining, analyzing, and making use of various

OCR for page 49
54 sources of funding for IT/ITS projects. Like IT projects in gen- ager, or as someone impacted by the operations of the system eral, transportation IT and ITS projects are delivered through (i.e., recipient of information or process coordination part- public leveraging options like bond financing, public-private ner). Customer needs drive the system requirements, or what partnerships, comingled funding, and a variety of Federal, state the system should do. For example, if there is a need to mea- and local funding sources. sure ridership at stops (boardings and alightings) for each trip, Transit agencies are using many of these financing mecha- then there is a corresponding requirement for the Automated nisms to access the various sources of capital for IT/ITS proj- Passenger Counting (APC) system to count boardings and ects. Historically, buy (pay-as-you-go), borrow (issue bonds) alightings at each stop by trip identification. The systems engi- or lease were the primary financing mechanisms used by neering process must ensure that the requirement is described transit agencies. Since the 1990's, creative use of these tradi- in the design, and consequently implemented in the software, tional mechanisms and introduction of public-private part- data collected, stored and reported in a format that is consis- nerships has occurred. Chapter 4 discusses financing mecha- tent with its use as a performance measurement. The steps nisms; in particular, the section describes four categories: prescribed by the Systems Engineering process ensure a struc- debt mechanisms, capital leasing financing, equity and part- tured approach to track the need throughout the development nerships, and credit enhancements. stages. U.S. DOT recognized the potential benefit of the systems Chapter 4 discusses ITS implementation funding and ana- engineering approach for ITS projects and included require- lyzes the best method for obtaining the necessary funds for the ments for the use of the systems engineering process in the selected implementation. Based on the surveys conducted, no FHWA Final Rule/FTA Final Policy on Architecture and one financing method works for all situations, rather financ- Standards that was enacted on January 8, 2001. Chapter 6 dis- ing decisions need to be tailored to the specific project, region cusses the major steps that comprise the Systems Engineering and financial circumstance. analysis process and the results of the transit industry scan that shows the limited understanding and implementation of 1.4.3 Business Case Methodology the policy by transit agencies. A Business Case Methodology (BCM) is a formal analysis used to justify and capture the reasoning for initiating a project. 1.4.5 Post-Implementation Evaluation The business case typically reviews and verifies that (2): Post-implementation analysis or Post Implementation Review (PIR), as it is commonly called in the IT field, is con- The investment has value and importance ducted after a project has been completed. "The purpose of The project will be properly managed the PIR is to evaluate how successfully the project objectives The firm has the capability to deliver the benefits have been met and how effective the project management The firm's dedicated resources are working on the highest practices were in keeping the project on track (3)." value opportunities Projects with inter-dependencies are undertaken in the Chapter 7 discusses what a PIR is and is not. The PIR is not the testing and verification activities that are typically per- optimum sequence." formed in a project acceptance or closeout phase. As an exam- Chapter 5 discusses some of the different business case ple, the AVL system selected to meet the goal of increasing rid- methodologies for justifying IT/ITS investments. Each of the ership may have to be accepted from a vendor if it performs methodologies use somewhat different techniques for build- according to the requirements in the Request for Proposal, it ing the business case and determining return on investment, passes the test plan and the systems engineering verification total cost of ownership, value of investments, risk factors, process. impacts, and opportunities. Some best practices and critical The system, however, may not perform the way the users success factors associated with developing a good business want. Perhaps the business changed or it was specified ambigu- case and business case methodology are included in the ously and/or incorrectly. The PIR occurs after the IT/ITS chapter. system has been incorporated into the business and assesses how well the project meets the users' needs, what needs to be done next, and how well the implementation process 1.4.4 Systems Engineering went. Developing and sharing lessons learned can continu- Systems Engineering is a discipline that attempts to ensure ously improve the agency's project acquisition and manage- that customer needs are implemented in the system that is ment processes. The current practice in the industry, recom- developed. Customer needs are defined by the stakeholders or mended practices, and a checklist for managers is included people who have an interest in the system, as a user, a man- in Chapter 7.

OCR for page 49
55 2 Synthesis Methodology and tice in the past. Today, however, many industries have cre- Industry Scan ated useful templates for documenting an Enterprise Archi- tecture, including the business processes, data, performance To develop an assessment of the state of the practice, the measures and technology characteristics of their industry. research team reviewed available industry literature and The process of creating an EA has also been improved by new conducted telephone interviews with a sample of transit shortcuts and segmenting the enterprise into smaller, more agencies as well as several state DOTs. The literature search manageable chunks. Industries are now starting to realizing and interviews covered the five major elements to be substantial cost and time savings through Enterprise Archi- included in the Transit Enterprise Architecture and Plan- tecture Planning (EAP). ning Framework: The development of an enterprise architecture has shown significant benefits in planning for information technology Enterprise Architecture Planning (EAP) and Enterprise programs and obtaining important information about the Architecture (EA); business. In "The Value of Enterprise Architecture," (4) the ITS Implementation Funding; author reported savings in several areas: Business Case Methodology; Systems Engineering for ITS Project implementation; "Savings of 60% of the man-day efforts needed for collec- Evaluation for post-implementation measurement includ- tion, processing, validation and reporting on the elements ing assessing meaningful (estimated) ROI and performance. of the enterprise architecture--a task done continuously for reasons such as SOX, risk management, data protec- The literature search focused on innovative approaches in tion, user satisfaction etc. These savings alone can finance the IT and transportation industries, and in particular, tran- an EA program sit. Sources included professional journal articles, guidebooks, "A 10% increase in deliverables from investments in IT and tools that are available on the web, including several reports projects by architecturally checking projects in the prepa- published by the Transit Cooperative Research Program ration phase to ensure potential risks are identified and (TCRP). mitigated and also to avoid architectural conflicts during To provide a reasonable sample of agencies for the telephone the execution of IT projects. interviews, a group of 14 transit agencies and three DOTs was "A 10% reduction in yearly operating costs by the discov- selected for interviews. Survey protocols were developed for ery of redundancies or excessive spending in the IT support the interviews. A standard set of interview questions was to business and by standardizing the architecture, which administrated to all the agencies. In addition, some agencies not only leads to cost reduction but also to increased busi- were asked more detailed questions on some Framework ele- ness and IT flexibility and agility." ments, if the screening questions discovered areas to probe further, and if time was available. The first column in Table 1 For example, when a manager wants to save costs by elimi- shows the agencies and state DOTs that were interviewed. nating an ancient data reporting system, it is important to know Several agencies were asked more detailed questions about what parts of the agency access data it manages, which applica- their experience with the Framework topics. The checked tions depend on it, and which customers or users would be columns in Table 1 represent the agencies that were surveyed affected. in more detail on selected topics. This chapter is divided into two sections. First, it discusses the discipline of Enterprise Architecture Planning and the evolution of the leading methodologies. These are formal 3 Findings on Transit Enterprise methods that describe detailed steps, products and building Architecture Planning and blocks that an organization uses to develop an Enterprise Enterprise Architecture Architecture. This chapter presents methodologies for planning and The second part of the chapter discusses how transit cur- documenting an Enterprise Architecture (EA) and discusses rently approaches Enterprise Architecture Planning (EAP) the transit survey findings. Developing an enterprise architec- and the development of Enterprise Architecture. While few ture is often perceived to be an arduous, expensive and transit agencies have enterprise architecture planning processes lengthy process performed by outside consultants who spend in place or have developed an enterprise architecture in the many months at an organization and then provide a huge formal sense, there are ways that organizations are employing report with countless diagrams and, tables that ends up sit- "enterprise thinking" in making technology investment deci- ting on a shelf. The scan of the practice for Information Tech- sions. The brief scan shows that the majority of transit IT pro- nology (IT) and transit ITS shows that this was often the prac- fessionals are not making use of available industry tools to

OCR for page 49
56 Table 1. Transit agencies and state DOTs interviews for industry scan. develop segments of the enterprise architecture, and they ships between data, systems, technologies and business miss capitalizing on the lessons and benefits of "enterprise processes thinking." Increasing business value and effectiveness through improved technology deployment 3.1 Main Purpose of an The proliferation of servers and operating systems that Enterprise Architecture started over two decades ago was one of the earliest drivers of Enterprise Architecture is a structure that provides executive enterprise architecture. Organizations started to have new managers visibility into the overall relationships among their problems as they accumulated different servers, operating people, processes, technologies and performance. It enables system products and versions, database products, and vari- executive managers to plan their technology investment deci- eties of personal computers with different variations of desk- sions to better meet corporate business needs and processes. top software. Soon IT could not keep up; IT staff often didn't The Enterprise Architecture benefits organizations by: know all the software and hardware varieties used through- out the company. The lack of technology planning generated Identifying where to reduce IT costs and complexity, by inefficiencies in the workforce and in the production of increasing the visibility of information flows and relation- needed information. Agencies needed to hire experts to

OCR for page 49
57 maintain all of their equipment; they needed to send their tions, such as the flow and delivery of information among people to training on all the platforms they supported; they subsystems, application versions, and restrictions on use. It needed to add specialists to their PC help desk to support the helps identify integration opportunities and problems, sys- variety of applications. tem dependencies, gaps in functional coverage, the status of Most agencies now have hardware and software inventory systems, and helps ensure that the development and enhance- lists that document every computer, applications, infrastruc- ment of applications align with the business strategies of the ture software and peripherals, as well as a set of systems organization. specifications and standards for software/products used for The Performance Architecture is a relatively newer part of databases, operating systems, web applications, application an EA. It is a standardized framework to measure the perform- development and more. The inventories and standards are ance of major IT investments and their contribution to pro- typical of the elements contained in a "Technology Enterprise gram performance. It includes Mission Goals and Objectives Architecture" layer. Building the lists and standardizing the and Performance Measures. technology was driven by the need to align the technologies There are numerous examples in transit, where an EA would and future investments with the corporate resources (staff, have eliminated ITS project delays and cost overruns. For skills) needed to support them. example, there are instances where transit agencies procured The current methods used to plan for and document enter- automated passenger counting (APC) systems, only to real- prise architectures take this concept further. In addition to ize that they are missing the people or processes to locate, a "Technology Enterprise Architecture" layer, a typical EA track and update the data required as input to the system. includes three to four other interrelated architecture "layers." Had an Enterprise Architecture been in place, it would have In a Government Technology article (5), the author states, shown the connections among the business processes, data ". . . one of the key things to know about enterprise architec- sets and technologies. It would have revealed the missing ture is that it is not `just an IT matter'--it involves the discus- processes and data from the onset, allowing for better plan- sion and clarification of business processes and procedures. ning and budgeting, and successful project delivery. There is no sense building applications and an infrastructure Typically, the Enterprise Architecture (EA) is composed of that simply automate disorganized or inefficient processes, so three major parts: defining and documenting business processes are key com- ponents of a full enterprise architecture undertaking." Current or "As Is" EA The Business Architecture describes the business and Future or "To Be" EA includes details on business processes, work flows, and roles and Gap Analysis with transitional EA models responsibilities needed to meet the business goals and objectives of the organization. It describes the "who, what, where, why, The Enterprise Architecture begins by describing the peo- when and how" business processes are accomplished. The Busi- ple, processes and technologies in use today. It documents ness Architecture helps project developers understand the busi- current processes, technologies and adopted standards and ness, identify stakeholders, find dependencies, and generally identifies problems, bottlenecks and missing linkages among expedites the information gathering tasks needed to develop the enterprise elements. Next, the Enterprise Architecture of requirements. the future identifies how to resolve these problems and struc- The Data Architecture describes the data and data struc- ture an organization where business goals are addressed by tures used by a business and its technology applications. It clearly defined processes, consistent data, easy to use applica- includes the meaning and relationship of information, infor- tions and "well oiled" technology solutions. The Gap Analy- mation on data integration needed by the organization, and sis defines a transitional program that identifies the stages answers the questions of who, what, where, why, when and necessary to move towards the future enterprise. how the data is managed. A Data Architecture can help a tran- The Enterprise Architecture Planning process is the work sit agency minimize ITS project delays due to missing or mis- that is done to develop and update the Enterprise Architec- understood data, such as when a trip planning system is pur- ture. The process is driven by many factors, including map- chased and its operation is delayed because accurate bus stop ping the corporate vision, customer expectations and stake- data is not available. holder needs to the EA. The Services Architecture, which used to be called Appli- The enterprise architecture planning process reveals the cations Architecture, describes the organization's technology pieces and connections to all parts of the business, so that all services and applications, such as web services, Automated the stakeholders along the activity chain see the flows of data, Passenger Counters, customer information systems, inven- work and outcomes. By doing this, other groups within the tory systems, Human Resource systems, etc. The Services organization may benefit from the processes being imple- Architecture contains other information about the applica- mented, such as a process to manage bus stops, or they may

OCR for page 49
58 find a group already doing similar processes. The processes and tional familiarity with common EAP/EA concepts and terms. technologies employed to manage bus stops may be elevated to Some structure and content for this project's Transit Enter- an enterprise activity, thereby aggregating contributions from prise Architecture and Planning Framework will come from multiple departments, benefiting multiple users, eliminating these two EAP/EA approaches. In addition, there are dozens redundancies by creating a single bus stop inventory source of other hybrid approaches and even these two approaches and improving corporate effectiveness through adoption of have influenced each other over the past decade. standard operating procedures. The TOGAF, derived from the Department of Defense The main purpose of an Enterprise Architecture Planning approach, is typically used as a set of templates with step-by- Process (EAP) is: step instructions for developing, planning for and imple- menting a segment of an enterprise architecture. The FEA To engage key stakeholders and IT staff in understanding was originally a set of four reference architecture models and the connections and dependencies among various parts of a process to build a plan to move from the current architec- the business and work together to improve the Enterprise's ture representation to a future architecture vision; as it overall effectiveness by reducing redundancies, leveraging evolved, the reference architecture models grew into a tax- technology investments for multiple processes, and build- onomy that could be used as building blocks to describe seg- ing a seamless information infrastructure. ments of the enterprise; the planning process became a prac- To prioritize enterprise information technology needs with tical roadmap that involved developing a transition plan, respect to the organization's strategic goals and objectives, incorporating core services, building a business case, and devel- particularly as the needs relate to technology investment oping an implementation plan based on a systems engineering decisions (build, operate and maintain). process. The key benefits of an EAP include: 3.1.1 Federal Enterprise Architecture (FEA) Ensuring there is consensus among key decision makers The FEA has grown into a multi-faceted program to define about the organizational objectives, needs, priorities and methods, tools and assessment strategies for the Federal gov- business processes, and how they are served by the technol- ernment to develop Enterprise Architectures that describe ogy investments; business processes for improvements in key areas. In their Ensuring that there is an awareness about how the deci- own words: sions related to technology investments such as business processes that operate and maintain the technology invest- . . . the FEA is entirely business-driven. Its foundation is the ment (including the lack of investment) impact the organ- Business Reference Model, which describes the government's Lines of Business and its services [including financial, HRM, etc. ization, its people, objectives, needs, priorities, and assess- and cross-cutting profiles like geospatial] . . . The outcome of this ment capabilities. effort will be a more citizen-centered, customer-focused govern- ment that maximizes technology investments to better achieve Although the different Enterprise Architecture (EA) and mission outcomes. (6) Enterprise Architecture Planning (EAP) methods in the industry have similar categories and major processes, the Figure 2 shows the business-driven model. The agencies industry is not consistent in the meaning of terms, classifi- share similar functions ("lines of business") such as Financial, cations, and scope of EA and EAP. A more detailed discus- Human Resources, and Homeland Security. They also are in sion of the definitions can be found in the Appendices of need of integrated, cross cutting services ("profiles") such as this chapter. In addition, the appendices include a discus- geospatial, security and records management. This three- sion on how the term "framework" is used with respect to dimensional model shows the inter-relationships and shared EAP/EA. functions among the Federal government departments. When the effort began, the focus was on a development process. The result was an EAP process as shown in Figure 3. 3.2 General Approach to EAP/EA Used by The EAP process for each Federal agency to develop Cur- Other Industries rent and Future architectures, document their standards and This section describes two well-known, non-proprietary policies, and develop transitional plans was daunting. To sup- approaches to EAP/EA, the Federal Enterprise Architecture port the planning process, the FEA working groups devel- (FEA) and The Open Group Architecture Framework oped three Reference Architectures (7) that described Perfor- (TOGAF). The two approaches are introduced to show some mance Measures, high level Lines of Business, and a Data similarities and differences, and to help the reader gain addi- Reference Model (8) that could be used as a reference or tem-

OCR for page 49
59 Architecture Methodology (FSAM) Practice Guidance docu- ment (9). According to the FSAM Overview: A segment architecture is a detailed results-oriented architec- ture (baseline and target) and a transition strategy addressing a vertical or horizontal portion (or segment) of the enterprise. (9) Today, the FEA is composed of Five-layer reference model (performance, business, infor- mation, services, technology) Segment architecture process and guidance Taxonomy for cataloging assets that are part of the EA Process for creating an enterprise architecture Figure 2. FEA Business Reference Architecture. Transitional process for migrating from pre-EA (current) Source: FEA Guidance to post-EA (future) Built in approach for measuring progress and success plate upon which the Government Departments could build (through the performance model) their structures. By 2005, that effort was substituted with a Self-assessment approach for determining success of using new approach. The new emphasis was on scoping the devel- the EA to drive business value opment into smaller, manageable segments. The new approach using Segment Architectures was estab- The approach seems to have paid off; several agencies have lished in 2007, with the publication of the Federal Segment published detailed operational concepts and business processes, Figure 3. FEA Enterprise Architecture Framework from Version 1.1. Source: Version 1.1 FEA Enterprise Architecture Framework

OCR for page 49
117 developing technologies that have great potential for the tran- response was that we do whatever parts of the process the sit industry." contractor provides. It seems in some cases the agencies are One of the key conclusions of the study is that improving content to rely solely on whatever level of expertise the con- transit technology implementation in public transportation tractor provides. In one or two of the agencies they specifi- requires the incorporation of the critical strategies of enter- cally hire a contractor to be their system engineer, providing prise architecture planning (EAP), systems engineering (SE) the SE expertise that they lack. and change management (which is actually a part of the over- Existing project management or system development all systems engineering process but which the study authors processes. Several of the agencies that could be considered broke out separately because of their view of its importance). more advanced (based on the number and scope of their The study reviewed the level of systems engineering knowl- ITS deployments) have a definite process orientation, but edge and usage within transit agencies (primarily from inter- in most cases this orientation is strong on project manage- views and literature review performed prior to 2006) and ment (or in one case business management) but not strong finds a very uneven level of knowledge and usage. Some larger in the technical development process that SE represents. agencies have embraced the process in their technology proj- Because of the project management focus, these agencies ects, but many others have little experience with or knowl- have a structured view of tracking the project's progress edge of the process. against cost and schedule. They may also have detailed An overall summary of the previous literature on systems consideration of such cross-cutting activities as risk man- engineering in transit is that using a systems engineering process agement. However, what these processes lack is the techni- is one of the critical factors in successfully implementing tech- cal development process, with its Concept of Operations nology- (or integration-) based projects; however, only select (focusing on the stakeholder needs and the operational agencies have incorporated the SE process into their develop- scenarios of the systems), formal requirements definition, ment processes, with the majority of transit agencies either design tradeoffs and verification against requirements. turning over responsibility for the process to their contractors They each cover parts of these activities (most often the or not using the process at all. requirements definition) but not all of them. Transit Agencies have in general not been required to use the Systems Engineering process. Although FTA Pol- 6.2 Interview Findings icy (on Architecture and Standards) requires a systems In order to determine where transit agencies stand on under- engineering analysis for each project using federal funds, standing and use of Systems Engineering for ITS project the requirements do not cover the full range of the SE development, a portion of each transit agency interview was process, and can be met by cherry picking info from a far devoted to the use of Systems Engineering. For several of the less systematic development process. Two of the agencies agencies that had recent experience with the systems engi- interviewed were required to closely follow the USDOT neering process, an additional set of interview questions was systems engineering process. They were developing sys- posed to assess whether the agencies had seen benefits from tems under the Mobility Services for All Americans their use of the Systems Engineering process. The discussion (MSAA) Initiative. This effort, which in Phase I developed below highlights the key findings from the interviews. the concept of operations and functional requirements for the system, caused each agency to become knowledgeable of the USDOT SE process and to utilize it in the project 6.2.1 Use of the Systems Engineering Process by development. As will be discussed below under benefits of Transit Agencies the SE process, both agencies felt it was a worthwhile exer- Almost all of the agencies interviewed indicated they used cise and plan on using the Systems Engineering process for some type of development process or did some aspects of the future efforts. Systems Engineering process. Only two answered "no" or "not really" to the basic question, Do you use a Systems Engineering 6.2.2 Benefits of Using the Process Process for project/system development? A closer examination of the interview responses shows that about half of the agencies Have the agencies that have used the Systems Engineering could be described as having a development process, and of process derived benefits from the effort they put into the these only a couple are really using the Systems Engineering process? The answer is a resounding yes. Some of the benefits process. Why the discrepancy? There are several key reasons: they identified were: Low level of knowledge of the systems engineering process Using the process helped the agency and the other stakehold- among agency personnel. In several cases, the agency ers go through each step rather than jumping to the end.

OCR for page 49
118 The SE process helps the agency keep the project on sched- (PIR) was completed. The literature review encompassed post- ule and budget. It allows the agency to have better visibility implementation analyses in transit ITS as well as other fields. into the contractor's progress through the outputs. To supplement the literature review, 14 transit agencies and two Using the process saves the agency a lot of trouble at the state DOTs were surveyed regarding their post-implementation backend of the project because the surprises are minimized. efforts. Because post-implementation analysis is called different The Concept of Operations made the agency and the rest things in different organizations and methodologies, additional of the stakeholders more aware of how the parts of the sys- prompts and follow-up questions were needed to clarify what tem will integrate and work together. was being discussed. 6.3 Recommendations 7.2 What is Post-Implementation Analysis? Based on the observed level of usage of the Systems Engi- Post-implementation analysis or Post Implementation neering process, the following are recommendations for apply- Review (PIR), as it is commonly called in the field of Infor- ing the process to the development of Enterprise Architecture mation Technology (IT), is conducted after a project has been Frameworks. completed. "The purpose of the PIR is to evaluate how suc- cessfully the project objectives have been met and how effec- Agencies should acquire a basic working knowledge of the tive the project management practices were in keeping the Systems Engineering process as it would apply to their project on track." (70) projects. Agency personnel are key participants in the proj- PIR is not the testing and verification activities that are typ- ect implementation process and must have an understand- ically performed in a project acceptance or closeout phase. As ing of the process if they are to successfully deploy tech- an example, a system may have to be accepted from a vendor nology-based systems. This knowledge is available through if it performs according to the requirements. It passes the test training, either in workshop or on-line settings, and agen- plan and the systems engineering verification process. The cies should put plans in place to obtain the necessary skills. system verification step can include both "factory testing" Any agency pursuing development of an Enterprise and "on-site testing" that occur during the initial implemen- Architecture framework should use the Systems Engi- tation or deployment phase. The system, however, may not neering process to plan and perform the development. As perform the way the users want, because either the business mentioned in the TCRP reference above (and as discussed changed or it was specified ambiguously and/or incorrectly. in the results of this current TCRP effort), the two disci- The PIR occurs after the IT/ITS system has been incorporated plines (SE and enterprise architecture) are inextricably into the business. linked and when pursuing enterprise architecture develop- PIR corresponds more closely to the Validation phase of ment it is critical to perform the development using a Sys- the systems engineering process. In the Systems Engineering tems Engineering process. Guidebook for ITS (65), the difference between verification and validation is explained as follows: 7 Findings on Post-Implementation Verification is the process which makes sure that what was Analysis in Transit built matches the requirements. Was the system built the way the requirements and design specified? Was the system built "right"? Objective of Post-Implementation Analysis Synthesis Both the verification and validation processes are important and The purpose of this synthesis is to document the state of the necessary. However, it is the validation which views the system practice in the transit industry with respect to post-implemen- from the system's owner and stakeholder perspective. The veri- fication of the system is viewed from the development team's tation analysis of IT/ITS deployments and to provide high-level perspective. Systems engineering's goal is to unify these views. guidance on how to approach post-implementation analysis. . . . System Validation by system owners and stakeholders com- The synthesis will briefly address the difference between post- pares against needs, goals, and expectations (this evaluation of implementation analysis and project closeout activities. In addi- validity directs the path to system upgrades and enhancements). tion, potential linkages between the post-implementation analysis and other project stages in the Enterprise Architecture The USDOT's Research and Innovative Technology Admin- and Planning Framework will be highlighted. istration's (RITA) web site provides guidance on how to com- plete comprehensive, independent evaluations of ITS projects. The ITS Evaluation Resource Guide at that site (71) expands 7.1 Approach/Methodology and elaborates on recommended evaluation procedures out- To develop this synthesis, a review of the literature on Post- lined in the SAFETEA-LU Reporting and Evaluation Guide- Implementation Analysis and Post Implementation Review lines. If the funding is available for one of these more elaborate

OCR for page 49
119 evaluations, the PIR results would feed into the evaluation. It Provide stakeholders with measures of success to help val- should be noted that a PIR can be very elaborate, but they also idate their decisions and support if the project went well can be completed in relatively simple forms. Generally, the Finally, information from ITS project evaluations can also PIRs are designed to improve the performance of the system or help the industry with subsequent projects by helping the project management procedures at the particular organi- understand the impacts of the technology on transit ser- zation that has implemented the system, rather than provide vices and users, transit organizations and their staff. comprehensive knowledge to the broader industry. Further detail about post-implementation analyses or PIRs 7.4 Post-Implementation Review is included in subsequent sections of this chapter. Process Overview Various organizations and vendors have slightly different 7.3 PIR Benefits: Why is Post- PIR processes, but the core process consists of the following Implementation Analysis Valuable? four steps: Planning, Preparing, Conducting and Reporting/ Post-implementation analysis is valuable for many rea- Following up. Generally, a cross-functional team consisting of sons. The most commonly cited benefit of the PIR process key stakeholders, users and technical experts plans and con- is that it provides managers with critical information on ducts the PIR, unless the capability and resources are available how to modify and improve a recently implemented system for having an independent auditor or evaluator. In that case, the to better meet the needs of the users and the staff responsi- auditor or evaluator works with the team and ensures the data ble for maintaining the system. The PIR process helps iden- collection and analysis integrity. For very small projects, the PIR tify system flaws, requirements that need to be changed, may consist of the business manager or IT manager surveying ways to improve performance and other potential future users and the technical staff and then reporting results. enhancements. The FAA web site includes a moderate-sized list of what a The Information Systems Board (72) of Washington State post-implementation review covers (73): believes that PIR process is valuable because "[w]hat gets measured gets managed. Determining performance measures "Perspective and insight of participants and users, cus- and outcomes at the beginning of a project helps assure that tomers, and stakeholders; the project stays true to the initial purpose and priorities. Original investment expectations including performance, Defining the desired outcomes or acceptance criteria at the investment and operating costs, schedules, benefits, and beginning of the project also clarifies the project's scope. technical capability; Using performance measures ascertains whether the project Actual investment results (e.g., operational performance; did indeed succeed, and provides a starting point for devel- customer, user, and stakeholder satisfaction; investment oping future lessons learned." and operating costs; technical capability; impact on mis- Other benefits of post-implementation analysis include: sion and program measures; unanticipated benefits); Cost and schedule deviations, such as additional "hidden" Identification of "lessons learned" about the technology costs related to investments that have been made to enable Identification of "lessons learned" about the project man- the primary investment; agement process Environmental changes that affected the investment (e.g., Documentation of "what went well" for: political, operational, economic, or technical conditions); awards and team building Original business case assumptions that justified the invest- developing and incorporating best practices into proj- ment program; ect management guidelines Expected next steps for the investment program; sharing with other transit business areas and organiza- Conclusions and learned lessons; and tions in the industry Recommendations to senior management." Improved understanding of the client's business needs Improved effectiveness of the IT organization by incorpo- 7.4.1 Planning the PIR rating PIR lessons and, with time, enhancing its credibility Increased knowledge on how to quantify benefits of ITS Planning begins during final investment analysis in con- projects junction with overall planning for implementation and life- Better investment decisions on future projects from using cycle management of an investment program. Planning for the PIR information the PIR is incorporated into the project plan and funding Ability to provide project sponsors and funding organiza- package. Goals, objectives and anticipated benefits that are tions with evidence of costs and benefits detailed in the Business Case and ROI need to be considered

OCR for page 49
120 in conjunction with the PIR Plan. How will the objectives different things in the various agencies, so additional prompts and benefits realization be measured and analyzed? Is it fea- and follow-up questions were needed to clarify what was sible? Can it be determined if the risks were adequately being discussed. defined and mitigated? Are resources allocated to complete the detailed planning, data collection and analysis steps of Does your agency have a post-implementation analysis or the PIR? evaluation phase for IT/ITS projects? With the exception of a few of the transit agencies that were surveyed, most of the respondents described relatively little 7.4.2 Preparing for the PIR consistent post-implementation analysis activity. In a few This step generally corresponds to the systems engineering cases, PIR was confused with system acceptance or project process's Validation strategy activity. It further defines the closeout activities. The majority of the agencies surveyed did PIR Plan and prepares the survey forms, templates, analysis not have a formal post-implementation analysis process. Of approaches and other resources that will be needed. It defines those that did, it was only sometimes or informally followed in more detail how the PIR or validation will take place and by a subset of those respondents. One respondent said their what resources will be needed. For example, if a before-and- reports had varying levels of formality, but they usually after study design will be used, the "before" data will need to included lessons learned, performance goals and compar- be collected prior to deployment of the system. isons against initial model forecasts. Terms used to describe post-implementation analysis activ- ities or processes included Post Project Assessment, Benefits 7.4.3 Conducting the PIR Realization Step, evaluation, Feedback, earned value manage- Most post-implementation interviews, surveys, "lessons ment analysis and validation. When the transit agency's post- learned" sessions and other data collection activities are con- implementation analysis had some form of proscribed proce- ducted after the IT/ITS system has completed system verifi- dures, it was generally because the organization's central IT cation and acceptance testing. Data collection may have to staff had a System Development Life Cycle (SDLC) method- begin earlier to collect "pre-" or "before" data. ology that included a post-project-closeout analysis step. An interesting, related comment from MARTA was that they have hired staff to be an in-house, independent verifi- 7.4.4 Reporting/Follow-up cation group that analyzes a new system prior to system Data is analyzed, a gap analysis is performed, results are acceptance (they complete the system engineering verifica- documented and they are reported to the project team, stake- tion process step). This group and process have "paid off in holders, transit managers and individuals and organizations dividends." requesting the evaluation results. Ideally, appropriate bene- fits and lessons learned are reported to RITA for incorpora- What is the time frame for measuring/evaluating the tion into the RITA web site databases. results of the IT/ITS project? Transit management and project managers should follow- The time frame for completing post-implementation analy- up on the recommendations and implement changes. Action ses varied, but most were completed within one year of system plans should be developed and implemented. acceptance. If the results from multiple PIRs from different programs The Utah Transit Authority (UTA) has an interesting are reviewed together, the review may identify ways to improve approach that includes two phases. First, it obtains feedback an agency's IT/ITS "investment planning and management on the system from the customer within 30 days of system control processes, enable more accurate estimates of invest- acceptance. UTA is striving for ISO (International Organiza- ment costs and benefits, and ultimately result in better tion for Standardization) consistency, so this feedback is part investment decisions. Results from successive reviews on of a regularly followed process. UTA strives to monitor, meas- singular investment programs enable managers to deter- ure and report on whether the project met the agreed upon mine if actions to improve performance and benefits are quality, schedule and budget expectations defined in the working." (74) scope, while acknowledging that all categories are subject to change requests that can modify expectations to the scope. UTA has another regular post-implementation practice, 7.5 Transit Survey Findings although there is no form for it. An IT supervisor or the project The transit agencies that were surveyed had varying lev- manager always checks back on the new system, generally after els of understanding of post-implementation analysis or it's running for three to six months (maximum one year) to see PIR. In addition, post-implementation analysis was called if anything else could have been done differently. They look for

OCR for page 49
121 lessons learned or needed system adjustments, as well as using lation, where to do improvements, to estimate how much it as an opportunity to keep up with changing business needs. time each vehicle spends on every block of the street and to The King County Metro Transit Signal Priority (TSP) team provide the data to others in the organization who want it. completes its Before and After data collection efforts imme- One of the biggest benefits is that it helped build tools, such diately surrounding a new installation to have as similar as as the TSP Interactive Model (cost-benefit model), for creat- possible "before" and "after" operating conditions (usually ing more effective installations. two weeks before and two weeks after). Does your agency apply the post-implementation analysis Who or what is the driver for having a process to all or some of its IT/ITS projects? post-implementation analysis? Three of the agencies said they did some post-implementa- A variety of reasons were given for doing post-implementa- tion analysis regularly after an IT/ITS project has passed sys- tion analyses. Some agencies cited policy or practice for doing tems acceptance. Most said they would try to do more in the post-implementation analyses. Another said ISO standards and future. procedures, as well as it being critical for providing good cus- tomer service. Other answers included the following: What are the biggest issues in completing the analyses? For those agencies that completed post-implementation Federal requirements analyses, time, money, gathering data, and motivation were "Usually we think it is the right thing to do." issues in completing the work. For some, after the project was Grant requirements over, they felt pressure to either work on enhancements or When a project manager pushes for it move onto a new project. Another said that it is a struggle to When it is a problematic project or one with lots of conflicts obtain data for a good ROI analysis; they use the cost/benefit When someone promised cost savings and now we have to analysis portion of the ROI more as a planning tool for decid- find them ing between implementation options. We had to justify why it cost so much Want the lessons learned to improve practices and proce- 7.5.1 King County Metro (KCM) dures Want to know how to improve the system in the enhance- King County Metro has extensive, detailed documentation ment phase and if it is needed and requirements for how project managers will run their IT/ITS projects, interact with the King County Project Review How are the results used? Board and document their activities. The process is pro- The most common answer was that the lessons learned scribed from the funding request phase through project were valued for improving future projects. The results were close-out, plus a Benefits Realization Report that is due a year also used to guide the next set of enhancements for the new after project close-out. project or to identify new business requirements. The Project Close-out Report is due within a month of the The Utah Transit Authority used the PIR process for sev- final monthly monitoring status report. It is supposed to eral purposes. Documenting PIR results from all of the include: IT/ITS projects "allows you to go back and see what you did and learn from errors." From an IT perspective, "one of the Documentation of the project description, results, variance and a summary explanation of what the project accomplished-- best values is the alignment of the requirements and the highlighting relevant project scope, schedule, and budget infor- deliverables (was it that the client changed their mind or that mation from the close-out documentation. Also includes benefits resources changed?). Feedback helps you clearly know what measurements, lessons learned, records retention, and deliver- the clients think. It's time consuming, but good. It just takes ables turnover. (44) lots of time." The TSP team at King County Metro uses the evaluation Forms and templates are provided to help complete the results in a number of different ways. They use the feedback report. Table 7 lists some of the project areas that are to be for adjusting and fine-tuning the TSP system, for TSP staff considered when developing lessons learned. The project training and education and for determining whether or not teams are also encouraged to describe project practices that to shut down a location with poor performance. In addition, worked well and could be utilized by other projects through- the analyses have helped them contribute to the industry's out the county to improve their performance. knowledge about TSP in talks, papers and during the devel- The Benefits Realization Report is a relatively new require- opment of the TCIP standards. Finally, they use the evalua- ment that aims to identify the benefits and value of the proj- tion data to help determine where to put the next TSP instal- ect after it has been operating for up to a year. In addition, it

OCR for page 49
122 Table 7. Lessons Learned. Lessons Learned-Project Areas to Consider Project Planning Quality Management Implementation Budget Management Communications Support Scope Management Team Management Work Effort Estimating Schedule Management Project Close-out Transition to Production Issues Management Requirements Testing Risk Management Design Other Change Management Development should provide a comparison of the benefits received to the ate aspects of benefits that might have been quantifiable as they value projected by the Business Case. The lessons learned and did not see a need to undertake the additional evaluation." The other data from the project closeout activities are part of the report further states, "Determining costs is complicated, since inputs to the Benefits Realization Report. The key questions some could be attributed to other systems such as the Radio addressed by the report are: system, fare boxes, APC, WAN upgrades, new staff that work on more than one system." Did the project provide quantifiable value to the county or The AVL Synthesis 73 report does list a number of lessons to the public? learned from the post-implementation follow-up, such as Did the project provide non-quantifiable benefits to the "staff resistance to accepting data as valid if it contradicts con- county or to the public? ventional understandings, . . . staff resistance to adopting Did the project provide benefits comparable to those pro- needed changes in operational procedures, . . . a number of jected by the Business Case? integration challenges, . . . the importance of securing partic- ipation from throughout the agency organization, carefully selecting the systems integrator, applying strong project man- 7.5.2 Other Transit Discussion of agement for the implementation, and understanding the sub- Post-Implementation Analysis stantial ongoing effort needed for system management once Transit agencies in the United States are not the only ones it is operational." finding it difficult to complete PIRs on ITS projects. In 2004, A paper from 1998, titled, ITS Benefits: Review of Evaluation Transport Canada published an article titled, Evaluation of Methods and Reported Benefits, (77) provides background infor- the Intelligent Transportation Systems (ITS) Deployment and mation pertaining to ITS evaluations. It summarizes the Integration Program, (75) which examined 12 Canadian ITS reported benefits of a number of ITS systems that have been projects. The report indicated that "some recipients provided deployed and the evaluation methods used to quantify the ITS minimal information in their project evaluations." In the rec- benefits. The report also presents several evaluation frameworks ommendations section, the report states that "In the future, that have been used to evaluate and quantify ITS benefits. Transport Canada should improve the measurement and The US Department of Transportation provides information reporting of results achieved by the ITS projects that it and tools that can help transit agencies with building a business funds. . . . Transport Canada should work with ITS funding case and designing the analyses for the PIR on its RITA website recipients to incorporate appropriate and cost-effective per- (78). For example, under the "ITS Resources" tab, information formance measurements and reporting methodologies to be can be found in the benefits, lessons learned and cost databases able to evaluate the results of ITS projects." that is helpful in planning a PIR. The evaluation support por- In the ITS field of AVL, a 2008 Transit Cooperative Research tion of the website (79) provides guidance on how to complete Program report titled Synthesis 73, AVL Systems for Bus Tran- comprehensive, independent evaluations of ITS projects and sit: Update (76) mentioned challenges with obtaining good provides links to a variety of sample evaluation strategies. post-implementation data. "In many AVL system implemen- Similarly, the IDAS (80) website may help with the design tations, the implementing agency did not systematically evalu- and analysis of some benefits of some ITS projects. "The ITS

OCR for page 49
123 Deployment Analysis System (IDAS) is software developed As another example, the following guidance is included by the Federal Highway Administration that can be used in under the item #1.9, When do we Conduct Post Implemen- planning for Intelligent Transportation System (ITS) deploy- tation Reviews? ments. State, regional and local planners can use IDAS to esti- mate the benefits and costs of ITS investments--which are We conduct post-implementation reviews 6 to 18 months after deployment at an operational site once initial problems are either alternatives to or enhancements of traditional highway worked out and users are generally familiar with the new and transit infrastructure." capability. Timing is crucial and dependent on the status of the investment program. A review conducted too soon may fail to capture full benefits, while a review conducted too late 7.6 Non-transit Approaches to may lose institutional knowledge about the investment and Post-Implementation Analysis recommendations may come too late to influence follow-on installations . . . Other industries and countries have recognized the value of conducting Post-Implementation Analyses and Reviews. Pro- Sample survey forms for the Project Manager, Sponsor, cedures and templates are available from a number of commer- Stakeholders, and Team are available at: cial vendors and consultants. In addition, various states and government organizations have instructions and tools available http://www.its.monash.edu/staff/projects/project-manage- via the Internet. Despite all the available guidance and tem- ment/templates.html. These samples provide additional ideas for plates, many organizations struggle with finding the knowledge, questions to include in a PIR process. time, and resources to complete a PIR on a new system. In addi- tion, some of the barriers and issues cited in the next section Discussion of auditing guidelines for Post Implementation conspire against the successful completions of PIRs. Review from the Information Systems Audit and Control A study conducted in Australia on "The Politics of Post- Association (ISACA) can be found at the ISACA's website. Implementation Reviews" (81, Pages 307319) indicated that This discussion and website is geared more toward auditors "few organizations undertake any substantive form of ex post or someone having to work with auditors that will look at an evaluation." It is one of several studies that state that system- organization's PIR process. It can be found at: atic use of formal evaluation is relatively rare after a system has been accepted. http://www.isaca.org/Template.cfm?Section=home&Template Several of the more accessible web sites that provide guid- =/ContentManagement/ContentDisplay.cfm&ContentID=18682 ance and templates on PIR are briefly discussed below. The Washington State Department of Information Ser- 7.7 Issues and Barriers Related to Post- vices, Information Services Board, provides guidance and Implementation Analysis templates under the topic, "Project Management Frame- work, Closure-Post Implementation Review," at the website: A number of issues that pose challenges to the successful http://isb.wa.gov/tools/pmframework/projectclosure/postim completion of post-implementation analyses were cited both in plementation.aspx the general literature and by the transit agency survey respon- The Federal Aviation Administration also has easy to follow dents. The issues include the following: and understand guidance on PIRs on the web. The initial menu is located at: http://fast.faa.gov/post_implementation/ Lack of knowledge on how to complete a post-implemen- index.htm tation analysis One of the initial menu items, PIR Standard Process Guid- Lack of time and resources to complete the PIR ance, is expanded in Figure 25 to show how the website has "Getting people to understand and care about it" nested the guidance and templates. As an example, the fol- Failure to collect "before" data so that a comparison of per- lowing guidance is included under the item #1.5, Can We formance can be made "pre-" and "post-" implementation Tailor the Review? Prior post-implementation analysis efforts, which tried to Post-implementation reviews are always tailored to the size, "place blame" for aspects of the project that didn't go well, complexity, and importance of an investment program or set of discouraged staff from planning and participating in sub- programs. Activities and costs are scaled appropriately, and may sequent efforts range from periodic surveys or focus-group meetings with users The difficulty of collecting accurate financial and operat- of small, low-cost investment products to multiple site visits by ing data a dedicated cross-functional team of users and stakeholders for Common statistics related to transit may be gathered and large, complex, high-cost investment programs. In all cases, actual operational data from users must be gathered and assessed defined in different ways, both internal to an organization against performance targets. . . . and between transit organizations

OCR for page 49
124 Figure 25. FAA guidance on PIR. (Source: FAA.) Collection and analysis of quantitative data for the purpose "Build the review into program planning from the start of rigorous evaluation is often very expensive. A Canadian during final investment analysis; study on its Intelligent Transportation Systems (ITS) Deploy- Conduct the review against expectations in the original busi- ment and Integration Program found that in some instances, ness case and program baseline; "Quantitative data collection and analysis would require Don't scrimp on resources or effort! This is the last best additional funding beyond that which was provided for the chance for taking corrective action when a program is not actual ITS deployments." (82) performing as intended; Managers who don't want a discoverable record of system Get close with the users; they live it every day and know issues and failures best where we can improve; Major disincentives to further post-implementation analy- Report both the good and the bad; there are always oppor- ses, and the future specification of budget-related benefits, tunities for doing things better; are created when organizations promptly cut all dollar and Ensure issues are handled effectively and that we have a staff savings from a work group that successfully imple- plan for closure; mented efficiencies through ITS improvements. Most Identify next steps clearly; and staff that implement new systems hope to see some of the Follow recommendations and actions to completion." resources that were saved put to use in improving or expanding services in their business area. Other recommendations for improving the success of post- implementation analysis efforts are: 7.8 Recommended Practices for Post- Implementation Analysis TriMet felt it was important to have easy to understand The FAA website, which provides distilled, straightforward procedures and templates for the PIR that require all the guidance on how to conduct Post Implementation Reviews, key steps to be completed but provide some flexibility and (83) includes the following tips for a successful PIR: the ability to scale the effort proportionally.

OCR for page 49
125 At the beginning of the project, and again after project acquire, assess and enhance IT/ITS systems and related busi- closeout, establish expectations and abate fears about the ness practices. PIR process. It is very common for a project to meet This portion of the Checklist will focus on management requirements and pass the project verification phase, while activities that ensure the benefits of completing post- still having issues during the project validation or PIR implementation analyses are realized. A number of the steps phase. This usually occurs because user needs can change also improve the value and success of other phases of an over time and requirements may not have been initially IT/ITS implementation. specified optimally. Involve and obtain input from the customers, the project 7.8.2 Transit Manager's Roles team and other stakeholders. Use a neutral facilitator during the lessons learned session Transit has become more and more dependent on the suc- and avoid placing blame. cessful operation of its automated systems. Managers need Be clear on the project success measures prior to imple- both system performance and critical information from those mentation. IT/ITS systems for effective decision making and for the effi- Use available resources, such as the ITS benefits and les- cient provision of transit service. Transit business area man- sons on the USDOT's Research and Innovative Technol- agers need to play a key role in ensuring the success of these ogy Administration web site, to help develop the project systems. In general, IT/ITS projects cannot be successfully specific list of benefits to assess in the PIR. implemented with only the attention of the IT manager. Choose benefits and performance measures to assess for Manager's Checklist Items which elements data can reasonably be collected. Key roles of the transit management team are to: Remember to collect "before" data at the appropriate time, rather than remembering the task at the end of the project. Ensure a common vision, communicate goals and priori- The time period for collecting the "before" data may vary ties, be champions of integration, provide oversight and by data set. support staff. The transit General Manager and the head of Establish a data collection and analysis methodology for Information Technology have particular responsibility for the project's PIR that addresses what data will be collected, ensuring that an integrated, agency-wide approach is taken how it will be collected, what resources are needed to col- for developing data and information systems solutions. (84) lect it, when it will be collected and how it will be analyzed Ensure that IT/ITS systems support the operational needs of or compared. the agency. The goals of the organization should be one of Selecting an assessment horizon depends on the ITS system the drivers of the IT/ITS project's goals, objectives and being implemented, how stable the initial implementation requirements. was, available resources and the conditions of the operat- Ensure that a realistic evaluation plan, Post-Implementa- ing environment. Some of the assessment should be con- tion Review Plan or Project Validation Plan (depending on ducted after the new system and procedures have had the terminology used by the agency) is developed before the ample time to be integrated into the business. Staff learn- systems development is started, so appropriate "before" ing curves and other project issues may dictate that a num- data can be collected. ber of the benefits be assessed a little later, after the system Ensure that complete financial analyses, such as ROI with has "settled down." cost, benefit, and Total Cost of Ownership considerations UTA felt it was important to link the PIR activity to an are completed during the development of the Business ongoing culture of continuous improvement and the reg- Case. These analyses can be used to assess if the completed ular rechecking of customer needs and expectations. project met or exceeded the original expectations. Incorporate the lessons learned and best practices into the Provide motivation, oversight, and the resources necessary organization's procedures. Share the newly gained knowl- to collect the data. edge that is valuable and constructive. Ensure that the project verification steps, which verify that requirements are met, are completed before system accept- ance and project closeout. 7.8.1 Checklist for Managers After project closeout, ensure that the PIR data collection This Checklist for Managers Section will be used to stim- plan is underway, so the post-implementation analyses can ulate discussion among transit staff participating in Task 4 to be completed. refine best practices for the Framework. The final Checklist Request and review the post-implementation analysis report. for Managers is intended to assist transit managers in Follow-up to make sure appropriate system and process enabling their staff and transit organization to effectively improvement recommendations are implemented.

OCR for page 49
126 8 References 21. Transit Department, Research & Market Strategy Division. "Alter- native Analysis and Recommendation." Seattle Metro, March 1993. 1. Definition of IT Governance adapted from IT Governance Institute. 22. Okunieff, P., N. Neuerburg, Teresa Adams. "Best Practices for 2. Description from http://en.wikipedia.org/wiki/Business_case Using Geographic Data in Transit: A Location Referencing Guide- [November 20, 2008] book Defining Geographic Locations of Bus Stops, Routes and 3. From the Washington State Department of Information Services, other Map Data for ITS, GIS and Operational Efficiencies." FTA- Information Services Board, Project Management Framework, NJ-26-7044-2003.1, US DOT FTA. April 2005. Closure-Post Implementation Review, http://isb.wa.gov/tools/ 23. "10 Areas Where Open Source is Open for Business." From pmframework/projectclosure/postimplementation.aspx http://www.cioinsight.com/c/a/IT-Management/10-Areas-Where- 4. Masling, Erik. "The Value of Enterprise Architecture." http:// Open-Source-is-Open-for-Business/?kc=CIOMINUTE02022009 enterpriseinnovator.com/index.php?articleID=15148§ionID=5, CIO1 [2/2/2009] published 4/21/2008. Extracted 2/24/2009. 24. Federal Enterprise Architecture Framework (v1.1) (http://www. 5. "Enterprise Architecture Demystified" by David Aden, Sep 24, 2008, gao.gov/bestpractices/bpeaguide.pdf) published in Government Technology, http://www.govtech.com/ 25. NASCIO EA Brochure on Enterprise Architecture: Building Better gt/articles/418008 Government through Enterprise Architecture. (www.immagic.com/ 6. "FEA Practice Guidance." OMB. November 2007. [http://www. eLibrary/ARCHIVES/GENERAL/NASCIOUS/N070202B.pdf) cio.gov/index.cfm?function=specdoc&id=FEA Practice Guidance, 26. IEEE Standard 1471-2000 IEEE Recommended Practice for Archi- November 2007&structure=Enterprise Architecture&category= tectural Description of Software-Intensive Systems. Institute of Enterprise Architecture] Electronics and Electrical Engineers, NY, NY, Oct 09, 2000. 7. Consolidated Reference Model Version 2.3 Issued By: OMB-- 27. Spewak, Steven H. and Hill, Steven. Enterprise Architecture Plan- Effective Date: 10.01.2007, 0.655K, PDF (http://www.cio.gov/ ning: Developing a Blueprint for Data, Applications, and Technol- index.cfm?function=specdoc&id=Consolidated Reference Model ogy. John Wiley & Sons, NY, Oct 1, 1992. Version 2.3&structure=Enterprise Architecture&category=Refer 28. Sessions, Roger. "Comparison of the Top Four Enterprise Archi- ence Models) tecture Methodologies" [www.objectwatch.com] 8. "Data Reference Model, Version 2.0," November 17, 2005, is avail- 29. National ITS Architecture. www.iteris.com/itsarch. able at http://www.whitehouse.gov/omb/egov/documents/DRM_ 30. www.fgdc.gov/training/nsdi-training-program/materials/IntroGeo 2_0_Final.pdf BusinessPlanning.ppt [extracted 1/29/09] 9. Federal Segment Architecture Methodology Overview--found at 31. FEA Geospatial Profile, draft version 2.0. http://www.fsam.gov/federal-segment-architecture-methodology- 32. The Role of the Finance Officer in Economic Development, GFOA, toolkit/overview.php February, 2008 10. The Open Group Architecture Framework. (http://www.open 33. Status of the Nation's Highways, Bridges, and Transit: Conditions group.org/togaf/) and Performance Report to Congress, 2006 11. ENTERPRISE ARCHITECTURE DEVELOPMENT TOOL-KIT 34. Transit State of Good Repair: Beginning the Dialogue, October October 2004 v3.0. (http://www.nascio.org/resources/EAre- 2008, FTA 35. Financing Capital Investment: A Primer for the Transit Practi- sources.cfm) tioner, TCRP Report 89, 2003 12. Michigan Enterprise Architecture Framework Work Plan Guide- 36. GASB Statement No. 34: Basic Financial Statements--and Man- lines. 2005 [www.michigan.gov/documents/Appendix_E_Archi agement's Discussion and Analysis--for State and Local Govern- tecture_145041_7.doc] ments, 1999 calling for management of public assets. 13. Mitretek Systems. TCRP Report 84 Volume 5 Concept for an 37. Introduction to Public Finance and Public Transit, FTA, 1993 and e-Transit Reference Enterprise Architecture. Transportation Research Report to Congress on the Costs, Benefits, and Efficiencies of Pub- Board, Washington, DC, 2004. lic-Private Partnerships for Fixed Guideway Capital Projects, FTA, 14. Hwang, M., J. Kemp, E. Lerner-Lam, N. Neuerburg, and P. Oku- 2007 nieff, Advanced Public Transportation Systems: The State of the 38. TCRP Report 89: Financing Capital Investment: A Primer for the Art Update 2006, Federal Transit Administration, Washington, Transit Practitioner, TRB, 2003 D.C., 2006 [Online]. Available: http://www.fta.dot.gov/documents/ 39. TCRP Report 115 Smartcard Interoperability Issues for the Tran- APTS_State_of_the_Art.pdf [accessed June 11, 2007]. sit Industry, TRB, 2006 15. Boyle, Daniel. "TCRP Synthesis 77 Passenger Counting Systems". 40. IT Governance Institute, article http://www.itgi.org/AMTem Transportation Research Board, Washington, DC, 2008. plate.cfm?Section=Deliverables&Template=/ContentManagement/ 16. Parker, Doug. "TCRP Synthesis 73 AVL Systems for Bus Transit: ContentDisplay.cfm&ContentID=24261 Update." Transportation Research Board, Washington, DC, 2008. 41. From: http://en.wikipedia.org/wiki/Business_case, March 1, 2009. 17. Miami-Dade Transit, "Information Technology and Intelligent 42. From Washington State ISB at http://www.isb.wa.gov/tools/ Transportation Systems 20032008 Strategic Plan Summary." pmframework/charter/performancemeasures.aspx (Volumes 12, Appendices AB), January 19, 2004. 43. http://www.fta.dot.gov/documents/Transit_IVBSS_Business_Case_ 18. "Renewing Technology Infrastructure at the Washington Metro- Analysis_Final_Report_9-07.pdf politan Area Transit Authority: Assessment of the Current Enter- 44. King County Office of Information Resource Management's prise Architecture." November 1, 2001. "Project Manager Guide to PRB Reviews", Version 2.0, June 2008 19. "Professionals' Guide to Information Technology Architecture 45. http://www.gcio.nsw.gov.au/documents/ Standards and Services." [2009] 46. From the New South Wales Office of Information and Communi- 20. Seattle Metro Transit Department, Research & Market Strategy Divi- cations Technology, http://www.gcio.nsw.gov.au/documents/ sion. "Geographic Information Systems Project Phase I Feasibility Business_Case_Template.pdf?bcsi_scan_36B076C11B08DC10=0 Study: User Needs Assessment Document". Seattle Metro, July 1992. &bcsi_scan_filename=Business_Case_Template.pdf

OCR for page 49
127 47. http://www.cio.sc.gov/NR/rdonlyres/FDC77967-F99D-4B6B-8119- 68. Mitretek Systems. TCRP Report 84-e-Transit: Electronic Business B4B9A3D6C5CC/0/Business_Case_Template.doc Strategies for Public Transportation-Volume 5 Concept for an e- 48. http://www.fgdc.gov/policyandplanning/future-directions/action- Transit Reference Enterprise Architecture, 2004 plans/draftroiworkbook 69. Burt, Matthew, et al. TCRP Report 84-e-Transit: Electronic Busi- 49. http://www.dir.state.tx.us/pubs/framework/index.htm ness Strategies for Public Transportation-Volume 8 Improving 50. http://www.dir.state.tx.us/pubs/framework/gate1/businesscase/work Public Transportation Technology Implementations and Antici- book.xls pating Emerging Technologies, 2008. 51. http://www.dir.state.tx.us/pubs/framework/gate1/businesscase/ 70. From the Washington State Department of Information Services, instruction.pdf Information Services Board, Project Management Framework, 52. http://fast.faa.gov/lifecycle/lpd/investab.htm Closure-Post Implementation Review, http://isb.wa.gov/tools/ 53. Article by Malcolm Huston at http://statetechmag.com/events/ pmframework/projectclosure/postimplementation.aspx updates/evaluating-it-investments.html 71. http://www.its.dot.gov/evaluation/eguide_resource.htm 54. http://www.ctg.albany.edu/publications/reports/advancing_roi/ 72. Washington State ISB at http://www.isb.wa.gov/tools/pmframe advancing_roi.pdf work/charter/performancemeasures.aspx 55. http://www.ctg.albany.edu/publications/guides/roi/roi.pdf 73. From the FAA category, Standard Process Guidance, Item #1.10: 56. http://www.fta.dot.gov/documents/Final_Report_-_Real-Time_ What Does the Review Cover?, http://fasteditapp.faa.gov/ams/ Systems_ROI_Study.doc do_action?do_action=ListTOC&contentUID=19&contentVersion 57. http://www.fgdc.gov/policyandplanning/future-directions/action- UID=7381 plans/draftroiworkbook 74. From the FAA website on PIR, http://fasteditapp.faa.gov/ams/ 58. Article by Amir Hartman, InformationWeek, April 2, 2007, at: do_action?do_action=ListTOC&contentUID=19 http://www.informationweek.com/news/showArticle.jhtml?article 75. http://www.tc.gc.ca/corporate-services/des/reports/2004/its/menu. ID=198701922&pgno=1&queryText=&isPrev= htm 59. King County Office of Information Resource Management's 76. http://onlinepubs.trb.org/Onlinepubs/tcrp/tcrp_syn_73.pdf "Project Manager Guide to PRB Reviews", Version 2.0, June 77. Turner, Shawn, Wm. R. Stockton, Scott James, Troy Rother, 2008 C. Michael, October 1998, http://citeseerx.ist.psu.edu/viewdoc/ 60. The tables are from Appendix D of the King County Office of Infor- summary?doi=10.1.1.40.9700 mation Resource Management's "Project Manager Guide to PRB 78. Information on the RITA website can be found at: http://www. Reviews", Version 2.0, June 2008 its.dot.gov/about.htm 61. Honour, Eric. "Understanding the Value of Systems Engineering", 79. Evaluation guidance can be found at: http://www.its.dot.gov/ 2004 evaluation/eguide_resource.htm 62. Vu, John D. "Software Process Improvement Journey: From Level 1 80. http://idas.camsys.com/aboutComponents.htm to Level 5" 81. Gwillim, David, Ken Dovey & Bernhard Wieder, The Politics of Post- 63. Barker, Bruce. IBM Commercial Products, 2003 Implementation Reviews, Information Systems Journal, Volume 15 64. FHWA, Systems Engineering for Intelligent Transportation Sys- Issue 4 tems: an Introduction for ITS Professionals, September 2006 82. Evaluation of the Intelligent Transportation Systems (ITS) Deploy- 65. Systems Engineering Guidebook for ITS, Co-sponsored by FHWA ment and Integration Program, March 2004, mhttp://www.tc.gc.ca/ and the California Department of Transportation, is located at: corporate-services/des/reports/2004/its/menu.htm http://www.fhwa.dot.gov/cadiv/segb/ 83. FAA site with guidance on PIR activities is http://fasteditapp. 66. FTA, Advanced Public Transportation Systems: State of the Art faa.gov/ams/do_action Update 2006, FTA-NJ-26-7062-06.1, March 30, 2006. 84. Best Practices for Using Geographic Data in Transit: A Location 67. Parker, Doug. TCRP Synthesis 73, AVL Systems for Bus Transit: Referencing Guidebook, Paula Okunieff, Nancy Neuerburg, and Update, 2008 Teresa Adams, 2003