Click for next page ( 227


The National Academies | 500 Fifth St. N.W. | Washington, D.C. 20001
Copyright © National Academy of Sciences. All rights reserved.
Terms of Use and Privacy Statement



Below are the first 10 and last 10 pages of uncorrected machine-read text (when available) of this chapter, followed by the top 30 algorithmically extracted key phrases from the chapter as a whole.
Intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text on the opening pages of each chapter. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.

Do not use for reproduction, copying, pasting, or reading; exclusively for search engines.

OCR for page 226
Chapter 7 Procurement and Monitoring 7.3Performance Measures and Evaluation Until the present, car-sharing programs have been the subject of relatively little evaluation and monitoring. Certainly, there has been a host of small- scale surveys, and a small number of larger, more academic projects (see Chapter 4). However, the main performance measure from the point of view of operators and their partners has been the "breath test" in other words, whether the program is still alive and breathing. In some instances, this may be appropriate. If an operator retains a vehicle location without the need for subsidy, it generally means it is being used and that the program is in this respect successful. As the industry matures, though, many partners will face a growing need for more sophisticated performance measures and evaluation techniques. These are important for several reasons: To keep partners on board. Staff at a partner organization will often need to make the case for car-sharing to senior management and/or Board members. Initially, the program may be viewed as experimental, but continued support will often require data. For example, Arlington County is completing its initial evaluation in Spring 2005, in order to support the case for continuing the pro- gram beyond the first year. In other cases, data may be needed before the program even begins. At Tufts University, the decision to support Zipcar was the subject of a two-year debate, amid fears that it could increase emissions. To obtain performance-based funding. Car-sharing qualifies for several sources of transportation funding, but only if the impacts on program goals can be clearly demonstrated and reported. The federal CMAQ program, from which I-GO in Chicago has re- ceived funding, is one of the most significant; CMAQ-funded proj- ects are required to include an assessment of the emissions reduc- tion impact. Other examples include the federal JARC program, which has granted funding to Flexcar in Seattle and City CarShare in San Francisco; and the EPA-administered Clean Air Transporta- tion Communities program, which has benefited Flexcar's pro- grams in Vancouver, WA and Seattle, WA. There are also various local programs using performance-based funding, such as the San Francisco Bay Area's Transportation Fund for Clean Air. To enforce development mitigations. Seattle and Boston, for example, permit car-sharing to be included as a trip reduction strategy in a developer's Transportation Management Plan or Transportation Access Plan Agreement (see Chapter 5). Page September 2005 7-8

OCR for page 226
Car-Sharing: Where and How It Succeeds To determine cost effectiveness. For example, TriMet in Port- land, OR has standards for a minimum number of riders (e.g. 15 roundtrips per day) before it will subsidize vanpools or a Flexcar van shuttle to specific employers. The City of Berkeley conducted a cost comparison when determining whether to outsource a por- tion of its vehicle fleet to City CarShare. To ensure responsible use of public money. If car-sharing is supported by a public agency, whether through cash, in-kind or policy support, it needs to be justified, regardless of the legal or practical reasons for doing so. In other words, evaluations can help to validate the public policy premise for granting support, and assess the extent to which car-sharing is achieving the prom- ised results. Car-sharing is an extremely data-rich environment, particularly when com- pared to transit. The computerized reservations, billing and fleet manage- ment systems used by most large operators allow the automatic reporting of metrics such as vehicle utilization, trip length and revenue. These data also help operators to better understand their customers and different market segments, through examining utilization patterns. In contrast, transit agen- cies are often forced to perform manual counts in order to gain accurate information on basic statistics such as ridership and passenger loads on particular routes. However, quantification of outcomes, such as changes in vehicle owner- ship and travel, still pose difficult challenges for car-sharing operators, as discussed below. Many partner organizations interviewed for this study believe that car-sharing is a "soft measure," the impact of which will never be comprehensively quantified. Evaluations to Date Most partner organizations contacted for this study both through the online survey and interviews consider, on balance, that car-sharing has been successful in helping to achieve their most important goal. Of online respondents who answered this question, nearly 25% said it has been very successful, and a further 45% said it has been fairly successful (Exhibit 7-2). Just one respondent said car-sharing was fairly or very unsuccessful. Page 7-9

OCR for page 226
Chapter 7 Procurement and Monitoring Exhibit 7-2 Evaluation of Success 20 16 Number of Respondents 12 8 4 0 Very successful Fairly successful Neither successful nor Fairly unsuccessful Very unsuccessful unsuccessful This picture of success, however, may be influenced by the nature of the survey respondents; many are "champions" for car-sharing within their organizations. The conclusions also appear to be largely based on qualitative impressions. More than 25% of respondents do not require any monitor- ing or evaluation as a condition of providing support, while many of the remainder request multiple types of reporting (Exhibit 7-3). Reporting was most often done quarterly. Only eight respondents stated they have per- formance standards for their partner car-sharing organization four with formal standards and four with informal standards. Exhibit 7-3 Techniques to Evaluate Success 28 24 20 Number of Respondents 16 12 8 4 0 Utilization Impacts Financial Other None Formal Informal None data evaluations data required Monitoring and Evalution Required Performance Standards Page September 2005 7-10

OCR for page 226
Car-Sharing: Where and How It Succeeds Judging from the interviews, a formal monitoring program is even less common than the survey results imply. The majority of car-sharing partner organizations interviewed do not monitor or evaluate programs in any way, or require data from operators. Examples of partners that do not require monitoring or evaluation from their partner car-sharing organization in- clude TriMet in Portland, OR; the Los Angeles Metropolitan Transportation Authority; the City of Boston; and developers such as JBG Properties. The City of Cambridge, meanwhile, ceased to require data once Zipcar began to pay market rate for parking. Several planning departments find it dif- ficult to track the number of developments that incorporate car-sharing as a transportation mitigation measure. Boston, for example, has a system in place but has not had the staffing to keep the monitoring up-to-date. The City is now setting up a simplified system, where developers will submit their reports on mitigation measures electronically. Other partners rely on more qualitative evaluations as to whether car-shar- ing is meeting their goals. For example, the prime measure of success for The Defender Association in Seattle is the halving in the number of parking spaces leased for employees, from 20 to 10, and the associated ability to continue the transit pass benefit for employees. When information is collected by partners, the material falls into four broad groups: financial and utilization data; trip information; surveys; and inde- pendent evaluations. Each of these is discussed in turn in the following sec- tions. Exhibit 7-4 provides some specific examples of partner organizations that require each type of data. These are not mutually exclusive; several partners ask for information in more than one category. In general, the depth of evaluation depends on the level of support from the partner organization. Partners that do not conduct any evaluation tend to be those that provide lesser degrees of support, such as promotion or a small number of parking spaces, although there are several exceptions to this rule. Partners that provide direct financial support generally ask for more data, particularly financial and utilization information. Financial and Utilization Data Financial and utilization data generally consist of information that is rou- tinely collected and analyzed by car-sharing operators for internal use, and do not impose a new data collection burden. However, such information is often proprietary, particularly in the case of for-profit operators. The degree Page 7-11

OCR for page 226
Chapter 7 Procurement and Monitoring to which it will be disclosed by operators depends on the amount of support that a partner is willing to provide. Non-disclosure agreements or similar undertakings are often required. Utilization and revenue data, such as the number of hours per day each ve- hicle is used, can be used to assess the financial stability of an operator and to evaluate operator requests to move under-utilized vehicles. This type of data tends to be the most closely guarded, as it would be extremely useful to potential competitors. In contrast, operators are usually more than willing to publicize membership numbers. Trip Information Trip information is primarily of interest to business members, who use the information to track utilization and prevent abuse. Most operators provide monthly, itemized invoices indicating the individual user, trip length, dis- tance and time. For example, Swedish Medical Center in Seattle reviews the invoices to see if employees appear to be using the service during non-business hours. The organization's monitoring also led to a policy change; any trip of more than three hours now requires advance permission from the transportation coordinator. The Seattle Defender Association, meanwhile, is interested in using the information to assess productivity and employee work habits for example, the extent to which investigators are in the field. Surveys A small number of partners have asked operators to conduct surveys of their members, in order to gain information on the impacts of car-sharing on travel behavior, vehicle ownership and transit ridership. In most cases, the operator conducts the surveys, generally through e-mail- ing members. The survey instrument is designed collaboratively between the partner and the operator. However, the City of Alexandria, VA, issues its own surveys. These are linked to the City's financial support; in order to receive a free membership, participants must return the surveys. Independent Evaluations In some cases, funding may be available for an independent evaluation, often by a local university. This has the advantage of perceived rigor and objectivity; however, the cost means that it is not normally an option. For example, the City of Chicago found that an independent evaluation to sup- port its CMAQ grant proposal could cost more than the entire grant itself. Page September 2005 7-12

OCR for page 226
Car-Sharing: Where and How It Succeeds Exhibit 7-4 Partner Evaluation Techniques Partner Organization Evaluation Financial and Utilization Data University of Victoria, BC Receives annual report on vehicle utilization, seasonal trends and other data. Arlington County, VA Zipcar and Flexcar report revenue per vehicle on a monthly basis, in order to receive the subsidy (which provides a revenue guarantee see Chapter 5). Ar- lington also collects survey data (Price & Hamilton, 2005). Massachusetts Institute of Receives quarterly report on number of vehicles. Technology City of Chicago Quarterly reports are passed through to Federal Transit Administration as a condition of CMAQ grant funding. The report outlines the demographics of members, car usage, trip destinations, member usage, and emission reductions estimated from car ownership changes. Data are based on (1) baseline data collected at the time of membership enrollment; (2) follow-up surveys; (3) travel diaries from members after 6 and 9 months; and (4) monthly mileage reports. City of Kitchener, ON Receives semiannual report on member numbers, new members and costs. Reviews financial statements. University of Wisconsin Subsidy is based on the quarterly utilization report, which is used to calculate the difference between us- age fees and Community Car's costs to break even. University of North Carolina Zipcar provides monthly utilization data. Trip Information Portland State University Monthly itemized bill indicating miles and hours used, time of trip, and name of user. Seattle Times Defenders Association, Seattle Surveys City of Alexandria, VA Survey questions include commute mode, vehicle ownership, reason for joining, and plans to switch commute mode or sell a car. Paper survey administered by the City. WMATA, Washington, DC Questions asked about frequency of riding transit, member satisfaction, and change in transit usage and vehicle ownership since joining the car-sharing program. E-mail survey to all DC-region members administered by the operator. University of Washington Questions probed reasons for joining, member satisfac- tion, and impact on travel to campus. Online survey to all UW-affiliated Flexcar members, administered by the operator. Responses were tied to date of joining and utilization information held in Flexcar's database. Page 7-13

OCR for page 226
Chapter 7 Procurement and Monitoring Exhibit 7-4 Partner Evaluation Techniques (cont'd) Partner Organization Evaluation City of Chicago Members are screened when they enroll in the I-GO program by asking questions about their demographics, travel mode and travel pattern baseline data, including car ownership (make, model and mileage and type of usage). Follow-up surveys, distributed after 6 and 18 months of membership, request information about members' travel mode choices, travel patterns, and their perceptions of I-GO and its impact, if any, on travel mode choices and travel patterns. Follow-up surveys have parallel questions to the initial survey to easily detect changes over time. After 12 months of membership, travel diaries are also distributed to members. BART, San Francisco Bay Area Questions asked about mode of access to car-sharing, frequency of riding BART, and how members would have made the last trip had City CarShare not been available. E-mail survey to users of car-sharing vehicles at BART stations, administered by the operator and analyzed by a consultant. University of Wisconsin The University asks for results of surveys Community Car conducts of its members. Questions on car-sharing will also be added in the campus' biennial survey. Independent Evaluations King County Metro University of Washington conducted initial evaluation. Metro is currently seeking to fund a second inde- pendent study that will focus on changes in transit ridership, pass sales and achievement against other goals. SANDAG San Diego State University received a separate grant to evaluate the Compass Plus program, which includes Flexcar. City CarShare University of California-Berkeley conducting multi-year study, including travel diaries, funding by the Federal Highway Administration. Principles of Performance Measurement Recent years have seen a growing trend towards the use of increasingly so- phisticated performance measures in the transportation fields. For example, pedestrian, bicycle and transit quality of service measures are now available to complete the traditional focus on vehicular level of service. Agencies such as the San Francisco County Transportation Authority, Florida Department of Transportation and City of Seattle are integrating performance measure- ment into their decision-making processes. At the federal level, changes to transportation and clean air legislation during the 1990s helped encourage Page September 2005 7-14

OCR for page 226
Car-Sharing: Where and How It Succeeds local jurisdictions and Metropolitan Planning Organizations to develop more innovative performance-based methodologies (Ewing, 1995). TCRP has already published two comprehensive references on developing performance measures for transit systems TCRP Report 88: A Guidebook for Developing a Transit Performance-Measurement System (Kittelson & As- sociates et al., 2003a), and TCRP Report 100: Transit Capacity and Quality of Service Manual, 2nd edition (Kittelson & Associates et al., 2003b). As TCRP Report 88 points out, agencies use performance measurement for three main reasons: Because they are required to do so (e.g. for National Transit Data- base reporting) Because it is useful for the agency to do so Because others outside the agency need to know what is going on The same reasoning also applies to car-sharing operators. Although there are fewer required measures when compared to transit, some funding sources have reporting requirements. Measures can be useful internally as a management tool, and in convincing potential partners that car-sharing is a program worth supporting. TCRP Report 88 identifies 11 characteristics of a successful performance measurement system. The following discussion tailors these findings to the field of car-sharing. The characteristics are (Kittelson & Associates et al., 2003a): Stakeholder acceptance. The measures should have broad input and support from a range of stakeholders, including the operator, partners and customers. Linkage to goals. The primary purpose of performance measures is to track progress against goals and objectives. Clarity. Measures should be intuitive for stakeholders to under- stand. Reliability and credibility. Some kinds of data, such as financial information, tend to be more reliable than others. Objectivity is also important: "Measures selected merely to make an agency look good are of little help in identifying areas for improvements," the TCRP guidebook says. Variety of measures. Measures should cover a broad range of areas, such as financial performance and customer satisfaction. Number of measures. Variety must be balanced with the need to avoid overwhelming the end user. Page 7-15

OCR for page 226
Chapter 7 Procurement and Monitoring Level of detail. Measures need to be as simple as possible, while allowing the accurate identification of areas where goals are not being achieved. Flexibility. Measures need to respond to changing goals and ex- ternal factors, while also allowing for historical comparisons. Realism of goals and targets. "Targets should be realistic, but slightly out of reach," in order to spur improvements, the report recommends. Timeliness. This allows quick identification of problem areas, and an appropriate reaction. Integration into agency decision-making. Measures can flag un- der- or over-achieving segments, and allow an appropriate course of action to be determined. Car-Sharing Performance Measures Specific performance measures can be divided into three categories: Internal. These measures are primarily a management issue for individual operators. In most cases, they will not be reported to partners partly for proprietary reasons, and partly because they are of limited relevance. For this reason, only a small selection of possible indicators is described here; for more details, see City CarShare (2005). As discussed above, however, there are some- times exceptions for example, operators may need to report uti- lization or revenue per vehicle when a partner is providing cash subsidy or parking. In these instances, however, the information is usually treated as confidential. Output. These include measures such as member numbers, ve- hicle numbers and coverage, which help measure growth and suc- cess but are not necessarily tied to wider goals. Output measures are also normally proprietary, but are less closely guarded than internal measures. Outcome. These measures are the most important from a part- ner's perspective, but usually the most difficult to measure. They indicate progress in achieving goals, such as impacts on vehicle ownership, parking availability, transit ridership and mobility for low-income households. (See also Shaheen, Schwartz & Wipyews- ki, 2004.) Exhibit 7-5 summarizes a range of performance indicators. The focus is on quantitative measures that can yield measurable results. However, it should be noted that more qualitative measures can also be extremely useful. For ex- Page September 2005 7-16

OCR for page 226
Car-Sharing: Where and How It Succeeds ample, King County Metro has qualitative goals for the car-sharing program, such as political support and success in leveraging private-sector capital. In addition, there is considerable interest in measures of outcomes at a broader scale such as changes in parking availability, traffic congestion or overall vehicle travel at a neighborhood or citywide level. The Philadelphia Parking Authority, for example, plans to use its regular surveys of parking availability to assess whether a portion of any impact can be attributed to the PhillyCarShare program. Most partners, however, are skeptical that car-sharing is a large enough phenomenon for any changes at this scale to be measurable, and there are no examples as yet of success with measures at such a broad scale. WMATA, for example, believes that the ridership gain from car-sharing will be too small to measure directly. Swedish Medical Center in Seattle suggests that while car-sharing helps to reduce barriers to transit use and carpooling, it is difficult to quantify the effect. Exhibit 7-5 Car-Sharing Performance Measures Measure Definition Significance How Measured Internal Measures (normally proprietary) Utilization Revenue hours per vehicle per Core measure of demand and effectiveness, Reservations and month. which helps to assess performance of indi- vehicle logs vidual vehicle locations. Revenue per Revenue (usage charges) per Similar measure to utilization, but helps Reservations and vehicle vehicle per month. control for reduced-rate nighttime trips. vehicle logs Vehicle % reservations denied. Helps assess whether new capacity is Manual reserva- availability needed. Can be separated into members tions logs; dif- whose first choice was denied, and those ficult to measure who could not reserve an acceptable time or with web-based vehicle location for that trip at all. Difficult systems to measure with modern web-based reserva- tions, since members can choose another vehicle or time. Employee Full-time employees per vehicle. Measures the efficiency of staffing; ratio From staffing and overhead should fall as an organization grows. vehicle numbers Member % "very satisfied" or "satis- Simple satisfaction measure. A range of more Member surveys satisfaction fied" with car-sharing service. sophisticated measures are available; see TCRP Report 88 (Kittelson & Associates, 2003a). Member % of members leaving each Customer satisfaction measure. Exit surveys Member database retention year. can probe reasons for leaving. Farebox Ratio of member fees to total Progress towards financial self-sufficiency Financial state- recovery expenses. or profitability. Similar to standard transit ments industry measure. Page 7-17

OCR for page 226
Chapter 7 Procurement and Monitoring Exhibit 7-5 Car-Sharing Performance Measures (cont'd) Measure Definition Significance How Measured Output Measures (usually proprietary) Number of Number of members. Simple measure of size and penetration. Member database members Active mem- % members using service in a Assesses whether members are active users, Reservations logs bers month. dormant or have signed up for "mobility insur- ance." Number of Number of car-sharing vehicles Simple measure of size and penetration. Fleet database vehicles in service. Low-emission % vehicles that are hybrid, Assesses uptake of clean-fuel technology. Fleet database vehicles CNG or electric. Car-sharing Number of (i) approved and (ii) Assesses uptake of car-sharing by develop- Fleet database developments occupied developments that ers. incorporate car-sharing. Outcome Measures Vehicle travel Net change in annual Vehicle Assesses impacts of car-sharing. Member surveys Miles Traveled (VMT) in private or travel diaries vehicles. Vehicle Net change in number of ve- Assesses impacts of car-sharing. Member surveys ownership hicles owned. Where members report having avoided vehicle purchases, this should be reported separately. Emissions Net change in CO2, CO, NOx Assesses impacts of car-sharing. Fleet database or other pollutants. Based and VMT change on change in VMT and fleet calculation, using composition. factors adopted by local air quality regulators Transit rider- Number of new transit trips Assesses impacts of car-sharing. Includes Member surveys ship generated each year. ridership from transit access trips to car- sharing, as well as wider changes in travel behavior. Parking Net reduction in parking Assesses extent to which car-sharing Planning Depart- spaces saved provision in developments that changes the form and auto-orientation of new ment data. Diffi- incorporate car-sharing. development. cult to quantify as parking variances are often granted for multiple reasons. Mobility Perception of increased mobil- Assesses impacts of car-sharing. Member surveys ity among members (e.g. ability to reach new destinations). Fleet savings Annual change in cost of Assesses cost savings from car-sharing. Requires "before" corporate vehicles (rental cars, and "after" data car-sharing, vehicle fleet, and on fleet costs plus mileage reimbursements). costs of car-shar- ing Page September 2005 7-18