Click for next page ( 16

The National Academies | 500 Fifth St. N.W. | Washington, D.C. 20001
Copyright © National Academy of Sciences. All rights reserved.
Terms of Use and Privacy Statement

Below are the first 10 and last 10 pages of uncorrected machine-read text (when available) of this chapter, followed by the top 30 algorithmically extracted key phrases from the chapter as a whole.
Intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text on the opening pages of each chapter. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.

Do not use for reproduction, copying, pasting, or reading; exclusively for search engines.

OCR for page 15
15 Transit agencies have in general not been required to system acceptance or project closeout activities. The majority use the SE process. Although FTA policy on ITS projects of the agencies surveyed did not have a formal PIA process. requires an SE analysis for each project using federal funds, Of those that did, it was only sometimes or informally fol- the requirements do not cover the full range of the SE lowed by a subset of those respondents. One respondent said process, and can be met by cherry picking info from a far less their reports had varying levels of formality, but they usually systematic development process. Two of the agencies inter- included lessons learned, performance goals, and compar- viewed were required to closely follow the U.S. DOT SE isons against initial model forecasts. process. They were developing systems under the Mobility Terms used to describe PIA activities or processes included Services for All Americans (MSAA) Initiative grant. The ini- post project assessment, benefits realization step, evaluation, tial phase of these projects developed the Concept of Oper- feedback, earned value management analysis, and validation. ations and functional requirements for the system, caused When the transit agency's PIA had some form of specified pro- each agency to become knowledgeable of the U.S. DOT SE cedures, it was generally because the organization's central IT process and required the agency to utilize the process in the staff had a System Development Life Cycle (SDLC) methodol- project development. As will be discussed below under ogy that included a post-project-closeout analysis step. "Benefits of Using the SE Process," both agencies felt it was An interesting, related comment from MARTA was that a worthwhile exercise and plan on using the SE process for they have hired staff to be an in-house, independent verification future efforts. group that analyzes a new system prior to system acceptance (they complete the SE verification process step). This group and process have "paid off in dividends." Benefits of Using the SE Process King County Metro has extensive, detailed documentation The question posed to transit staff was, "Have you derived and requirements for how project managers will run their benefits from using the Systems Engineering process?" The IT/ITS projects and document their activities. More informa- answer was a resounding yes. Some of the benefits they iden- tion about the process and the Benefits Realization Report tified were: that is due a year after project close-out is in the State of the Practice Synthesis in Appendix B. Using the process helped the agency and the other stakehold- ers go through each step rather than jumping to the end. What is the time frame for The SE process helps the agency keep the project on sched- measuring/evaluating the ule and budget. It allows the agency to have better visibility results of the IT/ITS project? into the contractor's progress through the outputs. Using the process saves the agency a lot of trouble at the The time frame for completing PIAs varied, but most were backend of the project because the surprises are minimized. completed within 1 year of system acceptance. The Concept of Operations made the agency and the rest of The Utah Transit Authority (UTA) has an interesting the stakeholders more aware of how the parts of the system approach that includes two phases. First, it obtains feedback on will integrate and work together. the system from the customer within 30 days of system accept- ance. UTA is certified in and applies International Organiza- tion for Standardization (ISO) 9001 Quality Management Post-Implementation Analysis (PIA) Standards, so this feedback is part of a regularly followed The transit agencies that were surveyed had varying levels process. UTA strives to monitor, measure, and report on of understanding of post-implementation analysis, or as it is whether the project met the agreed-upon quality, schedule, and called in other industries, Post Implementation Review (PIR). budget expectations defined in the scope, while acknowledging In addition, post-implementation analysis (PIA) was called that all categories are subject to change requests that can different things in the various agencies, so additional prompts modify expectations to the scope. and follow-up questions were needed to clarify what was UTA has another regular post-implementation practice, being discussed. although there is no form for it. An IT supervisor or the proj- ect manager always checks back on the new system, generally after it's running for 3 to 6 months (maximum 1 year) to see if Does your agency have a PIA or evaluation anything else could have been done differently. They look for phase for IT/ITS projects? lessons learned or needed system adjustments, as well as using With the exception of a few of the transit agencies that were it as an opportunity to keep up with changing business needs. surveyed, most of the respondents described relatively little The King County Metro Transit Signal Priority (TSP) consistent PIA activity. In a few cases, PIR was confused with team completes its "before" and "after" data collection efforts

OCR for page 15
16 immediately surrounding a new installation to have as similar changed?). Feedback helps you clearly know what the clients as possible "before" and "after" operating conditions (usually think. It's time consuming, but good. It just takes lots of time." 2 weeks before and 2 weeks after). The TSP team at King County Metro uses the evaluation results in a number of different ways. They use the feedback for adjusting and fine-tuning the TSP system, for TSP staff train- Who or what is the driver for having a PIA? ing and education, and for determining whether or not to A variety of reasons were given for doing a PIA. Some shut down a location with poor performance. In addition, the agencies cited policy or practice. Another said ISO standards analyses have helped them contribute to the industry's knowl- and procedures, as well as it being critical for providing good edge about TSP in talks, papers, and during the development customer service. Other answers included the following: of the Transit Communications Interface Profile (TCIP) stan- dards. Finally, they use the evaluation data to help determine Federal requirements where to put the next TSP installation, where to do improve- Usually we think it is the right thing to do ments, to estimate how much time each vehicle spends on Grant requirements every block of the street and to provide the data to others in When a project manager pushes for it the organization who want it. One of the biggest benefits is When it is a problematic project or one with lots of conflicts that it helped build tools, such as the TSP Interactive Model When someone promised cost savings and now we have to (cost-benefit model), for creating more effective installations. find them We have to justify why it cost so much Does your agency apply the PIA process We want the lessons learned to improve practices and to all or some of its IT/ITS projects? procedures We want to know how to improve the system in the enhance- Three of the agencies said they do some PIA regularly after ment phase and if it is needed an IT/ITS project has passed systems acceptance. Most said they would try to do more in the future. How are the results used? What are the biggest issues The most common answer was that the lessons learned were in completing the analyses? valued for improving future projects. The results were also used to guide the next set of enhancements for the new project For those agencies that completed post-implementation or to identify new business requirements. analyses, time, money, gathering data, and motivation were The Utah Transit Authority used the PIR process for several issues in completing the work. For some, after the project was purposes. Documenting PIR results from all of the IT/ITS over, they felt pressure to either work on enhancements or projects "allows you to go back and see what you did and learn move on to a new project. Another said that it is a struggle to from errors." From an IT perspective, "one of the best values obtain data for a good ROI analysis; they use the cost/benefit is the alignment of the requirements and the deliverables analysis portion of the ROI more as a planning tool for decid- (was it that the client changed their mind or that resources ing between implementation options.