Below are the first 10 and last 10 pages of uncorrected machine-read text (when available) of this chapter, followed by the top 30 algorithmically extracted key phrases from the chapter as a whole.
Intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text on the opening pages of each chapter.
Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.
Do not use for reproduction, copying, pasting, or reading; exclusively for search engines.
OCR for page 226
Chapter 7 · Procurement and Monitoring
7.3Performance Measures and Evaluation
Until the present, car-sharing programs have been the subject of relatively
little evaluation and monitoring. Certainly, there has been a host of small-
scale surveys, and a small number of larger, more academic projects (see
Chapter 4). However, the main performance measure from the point of view
of operators and their partners has been the "breath test" in other words,
whether the program is still alive and breathing.
In some instances, this may be appropriate. If an operator retains a vehicle
location without the need for subsidy, it generally means it is being used
and that the program is in this respect successful. As the industry matures,
though, many partners will face a growing need for more sophisticated
performance measures and evaluation techniques. These are important for
several reasons:
· To keep partners on board. Staff at a partner organization will
often need to make the case for car-sharing to senior management
and/or Board members. Initially, the program may be viewed as
experimental, but continued support will often require data. For
example, Arlington County is completing its initial evaluation in
Spring 2005, in order to support the case for continuing the pro-
gram beyond the first year. In other cases, data may be needed
before the program even begins. At Tufts University, the decision
to support Zipcar was the subject of a two-year debate, amid fears
that it could increase emissions.
· To obtain performance-based funding. Car-sharing qualifies for
several sources of transportation funding, but only if the impacts
on program goals can be clearly demonstrated and reported. The
federal CMAQ program, from which I-GO in Chicago has re-
ceived funding, is one of the most significant; CMAQ-funded proj-
ects are required to include an assessment of the emissions reduc-
tion impact. Other examples include the federal JARC program,
which has granted funding to Flexcar in Seattle and City CarShare
in San Francisco; and the EPA-administered Clean Air Transporta-
tion Communities program, which has benefited Flexcar's pro-
grams in Vancouver, WA and Seattle, WA. There are also various
local programs using performance-based funding, such as the San
Francisco Bay Area's Transportation Fund for Clean Air.
· To enforce development mitigations. Seattle and Boston, for
example, permit car-sharing to be included as a trip reduction
strategy in a developer's Transportation Management Plan or
Transportation Access Plan Agreement (see Chapter 5).
Page
September 2005
7-8
OCR for page 227
Car-Sharing: Where and How It Succeeds
· To determine cost effectiveness. For example, TriMet in Port-
land, OR has standards for a minimum number of riders (e.g. 15
roundtrips per day) before it will subsidize vanpools or a Flexcar
van shuttle to specific employers. The City of Berkeley conducted
a cost comparison when determining whether to outsource a por-
tion of its vehicle fleet to City CarShare.
· To ensure responsible use of public money. If car-sharing is
supported by a public agency, whether through cash, in-kind or
policy support, it needs to be justified, regardless of the legal or
practical reasons for doing so. In other words, evaluations can
help to validate the public policy premise for granting support,
and assess the extent to which car-sharing is achieving the prom-
ised results.
Car-sharing is an extremely data-rich environment, particularly when com-
pared to transit. The computerized reservations, billing and fleet manage-
ment systems used by most large operators allow the automatic reporting of
metrics such as vehicle utilization, trip length and revenue. These data also
help operators to better understand their customers and different market
segments, through examining utilization patterns. In contrast, transit agen-
cies are often forced to perform manual counts in order to gain accurate
information on basic statistics such as ridership and passenger loads on
particular routes.
However, quantification of outcomes, such as changes in vehicle owner-
ship and travel, still pose difficult challenges for car-sharing operators, as
discussed below. Many partner organizations interviewed for this study
believe that car-sharing is a "soft measure," the impact of which will never
be comprehensively quantified.
Evaluations to Date
Most partner organizations contacted for this study both through the
online survey and interviews consider, on balance, that car-sharing has
been successful in helping to achieve their most important goal. Of online
respondents who answered this question, nearly 25% said it has been very
successful, and a further 45% said it has been fairly successful (Exhibit 7-2).
Just one respondent said car-sharing was fairly or very unsuccessful.
Page
7-9
OCR for page 228
Chapter 7 · Procurement and Monitoring
Exhibit 7-2 Evaluation of Success
20
16
Number of Respondents
12
8
4
0
Very successful Fairly successful Neither successful nor Fairly unsuccessful Very unsuccessful
unsuccessful
This picture of success, however, may be influenced by the nature of the
survey respondents; many are "champions" for car-sharing within their
organizations. The conclusions also appear to be largely based on qualitative
impressions. More than 25% of respondents do not require any monitor-
ing or evaluation as a condition of providing support, while many of the
remainder request multiple types of reporting (Exhibit 7-3). Reporting was
most often done quarterly. Only eight respondents stated they have per-
formance standards for their partner car-sharing organization four with
formal standards and four with informal standards.
Exhibit 7-3 Techniques to Evaluate Success
28
24
20
Number of Respondents
16
12
8
4
0
Utilization Impacts Financial Other None Formal Informal None
data evaluations data required
Monitoring and Evalution Required Performance Standards
Page
September 2005
7-10
OCR for page 229
Car-Sharing: Where and How It Succeeds
Judging from the interviews, a formal monitoring program is even less
common than the survey results imply. The majority of car-sharing partner
organizations interviewed do not monitor or evaluate programs in any way,
or require data from operators. Examples of partners that do not require
monitoring or evaluation from their partner car-sharing organization in-
clude TriMet in Portland, OR; the Los Angeles Metropolitan Transportation
Authority; the City of Boston; and developers such as JBG Properties. The
City of Cambridge, meanwhile, ceased to require data once Zipcar began
to pay market rate for parking. Several planning departments find it dif-
ficult to track the number of developments that incorporate car-sharing as
a transportation mitigation measure. Boston, for example, has a system in
place but has not had the staffing to keep the monitoring up-to-date. The
City is now setting up a simplified system, where developers will submit
their reports on mitigation measures electronically.
Other partners rely on more qualitative evaluations as to whether car-shar-
ing is meeting their goals. For example, the prime measure of success for
The Defender Association in Seattle is the halving in the number of parking
spaces leased for employees, from 20 to 10, and the associated ability to
continue the transit pass benefit for employees.
When information is collected by partners, the material falls into four broad
groups: financial and utilization data; trip information; surveys; and inde-
pendent evaluations. Each of these is discussed in turn in the following sec-
tions. Exhibit 7-4 provides some specific examples of partner organizations
that require each type of data. These are not mutually exclusive; several
partners ask for information in more than one category.
In general, the depth of evaluation depends on the level of support from
the partner organization. Partners that do not conduct any evaluation tend
to be those that provide lesser degrees of support, such as promotion or a
small number of parking spaces, although there are several exceptions to
this rule. Partners that provide direct financial support generally ask for
more data, particularly financial and utilization information.
Financial and Utilization Data
Financial and utilization data generally consist of information that is rou-
tinely collected and analyzed by car-sharing operators for internal use, and
do not impose a new data collection burden. However, such information is
often proprietary, particularly in the case of for-profit operators. The degree
Page
7-11
OCR for page 230
Chapter 7 · Procurement and Monitoring
to which it will be disclosed by operators depends on the amount of support
that a partner is willing to provide. Non-disclosure agreements or similar
undertakings are often required.
Utilization and revenue data, such as the number of hours per day each ve-
hicle is used, can be used to assess the financial stability of an operator and
to evaluate operator requests to move under-utilized vehicles. This type of
data tends to be the most closely guarded, as it would be extremely useful
to potential competitors. In contrast, operators are usually more than willing
to publicize membership numbers.
Trip Information
Trip information is primarily of interest to business members, who use the
information to track utilization and prevent abuse. Most operators provide
monthly, itemized invoices indicating the individual user, trip length, dis-
tance and time.
For example, Swedish Medical Center in Seattle reviews the invoices to see
if employees appear to be using the service during non-business hours.
The organization's monitoring also led to a policy change; any trip of more
than three hours now requires advance permission from the transportation
coordinator. The Seattle Defender Association, meanwhile, is interested in
using the information to assess productivity and employee work habits for
example, the extent to which investigators are in the field.
Surveys
A small number of partners have asked operators to conduct surveys of
their members, in order to gain information on the impacts of car-sharing
on travel behavior, vehicle ownership and transit ridership.
In most cases, the operator conducts the surveys, generally through e-mail-
ing members. The survey instrument is designed collaboratively between
the partner and the operator. However, the City of Alexandria, VA, issues
its own surveys. These are linked to the City's financial support; in order to
receive a free membership, participants must return the surveys.
Independent Evaluations
In some cases, funding may be available for an independent evaluation,
often by a local university. This has the advantage of perceived rigor and
objectivity; however, the cost means that it is not normally an option. For
example, the City of Chicago found that an independent evaluation to sup-
port its CMAQ grant proposal could cost more than the entire grant itself.
Page
September 2005
7-12
OCR for page 231
Car-Sharing: Where and How It Succeeds
Exhibit 7-4 Partner Evaluation Techniques
Partner Organization Evaluation
Financial and Utilization Data
University of Victoria, BC Receives annual report on vehicle utilization, seasonal
trends and other data.
Arlington County, VA Zipcar and Flexcar report revenue per vehicle on a
monthly basis, in order to receive the subsidy (which
provides a revenue guarantee see Chapter 5). Ar-
lington also collects survey data (Price & Hamilton,
2005).
Massachusetts Institute of Receives quarterly report on number of vehicles.
Technology
City of Chicago Quarterly reports are passed through to Federal Transit
Administration as a condition of CMAQ grant funding.
The report outlines the demographics of members, car
usage, trip destinations, member usage, and emission
reductions estimated from car ownership changes.
Data are based on (1) baseline data collected at the
time of membership enrollment; (2) follow-up surveys;
(3) travel diaries from members after 6 and 9 months;
and (4) monthly mileage reports.
City of Kitchener, ON Receives semiannual report on member numbers, new
members and costs. Reviews financial statements.
University of Wisconsin Subsidy is based on the quarterly utilization report,
which is used to calculate the difference between us-
age fees and Community Car's costs to break even.
University of North Carolina Zipcar provides monthly utilization data.
Trip Information
Portland State University Monthly itemized bill indicating miles and hours used,
time of trip, and name of user.
Seattle Times
Defenders Association, Seattle
Surveys
City of Alexandria, VA Survey questions include commute mode, vehicle
ownership, reason for joining, and plans to switch
commute mode or sell a car. Paper survey administered
by the City.
WMATA, Washington, DC Questions asked about frequency of riding transit,
member satisfaction, and change in transit usage
and vehicle ownership since joining the car-sharing
program. E-mail survey to all DC-region members
administered by the operator.
University of Washington Questions probed reasons for joining, member satisfac-
tion, and impact on travel to campus. Online survey to
all UW-affiliated Flexcar members, administered by the
operator. Responses were tied to date of joining and
utilization information held in Flexcar's database.
Page
7-13
OCR for page 232
Chapter 7 · Procurement and Monitoring
Exhibit 7-4 Partner Evaluation Techniques (cont'd)
Partner Organization Evaluation
City of Chicago Members are screened when they enroll in the I-GO
program by asking questions about their demographics,
travel mode and travel pattern baseline data, including
car ownership (make, model and mileage and type of
usage). Follow-up surveys, distributed after 6 and
18 months of membership, request information about
members' travel mode choices, travel patterns, and
their perceptions of I-GO and its impact, if any, on
travel mode choices and travel patterns. Follow-up
surveys have parallel questions to the initial survey
to easily detect changes over time. After 12 months
of membership, travel diaries are also distributed to
members.
BART, San Francisco Bay Area Questions asked about mode of access to car-sharing,
frequency of riding BART, and how members would
have made the last trip had City CarShare not been
available. E-mail survey to users of car-sharing vehicles
at BART stations, administered by the operator and
analyzed by a consultant.
University of Wisconsin The University asks for results of surveys Community
Car conducts of its members. Questions on car-sharing
will also be added in the campus' biennial survey.
Independent Evaluations
King County Metro University of Washington conducted initial evaluation.
Metro is currently seeking to fund a second inde-
pendent study that will focus on changes in transit
ridership, pass sales and achievement against other
goals.
SANDAG San Diego State University received a separate grant
to evaluate the Compass Plus program, which includes
Flexcar.
City CarShare University of California-Berkeley conducting multi-year
study, including travel diaries, funding by the Federal
Highway Administration.
Principles of Performance Measurement
Recent years have seen a growing trend towards the use of increasingly so-
phisticated performance measures in the transportation fields. For example,
pedestrian, bicycle and transit quality of service measures are now available
to complete the traditional focus on vehicular level of service. Agencies such
as the San Francisco County Transportation Authority, Florida Department
of Transportation and City of Seattle are integrating performance measure-
ment into their decision-making processes. At the federal level, changes to
transportation and clean air legislation during the 1990s helped encourage
Page
September 2005
7-14
OCR for page 233
Car-Sharing: Where and How It Succeeds
local jurisdictions and Metropolitan Planning Organizations to develop more
innovative performance-based methodologies (Ewing, 1995).
TCRP has already published two comprehensive references on developing
performance measures for transit systems TCRP Report 88: A Guidebook
for Developing a Transit Performance-Measurement System (Kittelson & As-
sociates et al., 2003a), and TCRP Report 100: Transit Capacity and Quality of
Service Manual, 2nd edition (Kittelson & Associates et al., 2003b). As TCRP
Report 88 points out, agencies use performance measurement for three main
reasons:
· Because they are required to do so (e.g. for National Transit Data-
base reporting)
· Because it is useful for the agency to do so
· Because others outside the agency need to know what is going on
The same reasoning also applies to car-sharing operators. Although there
are fewer required measures when compared to transit, some funding
sources have reporting requirements. Measures can be useful internally as
a management tool, and in convincing potential partners that car-sharing
is a program worth supporting.
TCRP Report 88 identifies 11 characteristics of a successful performance
measurement system. The following discussion tailors these findings to
the field of car-sharing. The characteristics are (Kittelson & Associates et
al., 2003a):
· Stakeholder acceptance. The measures should have broad input
and support from a range of stakeholders, including the operator,
partners and customers.
· Linkage to goals. The primary purpose of performance measures
is to track progress against goals and objectives.
· Clarity. Measures should be intuitive for stakeholders to under-
stand.
· Reliability and credibility. Some kinds of data, such as financial
information, tend to be more reliable than others. Objectivity is
also important: "Measures selected merely to make an agency
look good are of little help in identifying areas for improvements,"
the TCRP guidebook says.
· Variety of measures. Measures should cover a broad range of
areas, such as financial performance and customer satisfaction.
· Number of measures. Variety must be balanced with the need to
avoid overwhelming the end user.
Page
7-15
OCR for page 234
Chapter 7 · Procurement and Monitoring
· Level of detail. Measures need to be as simple as possible, while
allowing the accurate identification of areas where goals are not
being achieved.
· Flexibility. Measures need to respond to changing goals and ex-
ternal factors, while also allowing for historical comparisons.
· Realism of goals and targets. "Targets should be realistic, but
slightly out of reach," in order to spur improvements, the report
recommends.
· Timeliness. This allows quick identification of problem areas,
and an appropriate reaction.
· Integration into agency decision-making. Measures can flag un-
der- or over-achieving segments, and allow an appropriate course
of action to be determined.
Car-Sharing Performance Measures
Specific performance measures can be divided into three categories:
· Internal. These measures are primarily a management issue for
individual operators. In most cases, they will not be reported to
partners partly for proprietary reasons, and partly because they
are of limited relevance. For this reason, only a small selection of
possible indicators is described here; for more details, see City
CarShare (2005). As discussed above, however, there are some-
times exceptions for example, operators may need to report uti-
lization or revenue per vehicle when a partner is providing cash
subsidy or parking. In these instances, however, the information
is usually treated as confidential.
· Output. These include measures such as member numbers, ve-
hicle numbers and coverage, which help measure growth and suc-
cess but are not necessarily tied to wider goals. Output measures
are also normally proprietary, but are less closely guarded than
internal measures.
· Outcome. These measures are the most important from a part-
ner's perspective, but usually the most difficult to measure. They
indicate progress in achieving goals, such as impacts on vehicle
ownership, parking availability, transit ridership and mobility for
low-income households. (See also Shaheen, Schwartz & Wipyews-
ki, 2004.)
Exhibit 7-5 summarizes a range of performance indicators. The focus is on
quantitative measures that can yield measurable results. However, it should
be noted that more qualitative measures can also be extremely useful. For ex-
Page
September 2005
7-16
OCR for page 235
Car-Sharing: Where and How It Succeeds
ample, King County Metro has qualitative goals for the car-sharing program,
such as political support and success in leveraging private-sector capital.
In addition, there is considerable interest in measures of outcomes at a
broader scale such as changes in parking availability, traffic congestion or
overall vehicle travel at a neighborhood or citywide level. The Philadelphia
Parking Authority, for example, plans to use its regular surveys of parking
availability to assess whether a portion of any impact can be attributed to
the PhillyCarShare program.
Most partners, however, are skeptical that car-sharing is a large enough
phenomenon for any changes at this scale to be measurable, and there are
no examples as yet of success with measures at such a broad scale. WMATA,
for example, believes that the ridership gain from car-sharing will be too
small to measure directly. Swedish Medical Center in Seattle suggests that
while car-sharing helps to reduce barriers to transit use and carpooling, it
is difficult to quantify the effect.
Exhibit 7-5 Car-Sharing Performance Measures
Measure Definition Significance How Measured
Internal Measures (normally proprietary)
Utilization Revenue hours per vehicle per Core measure of demand and effectiveness, Reservations and
month. which helps to assess performance of indi- vehicle logs
vidual vehicle locations.
Revenue per Revenue (usage charges) per Similar measure to utilization, but helps Reservations and
vehicle vehicle per month. control for reduced-rate nighttime trips. vehicle logs
Vehicle % reservations denied. Helps assess whether new capacity is Manual reserva-
availability needed. Can be separated into members tions logs; dif-
whose first choice was denied, and those ficult to measure
who could not reserve an acceptable time or with web-based
vehicle location for that trip at all. Difficult systems
to measure with modern web-based reserva-
tions, since members can choose another
vehicle or time.
Employee Full-time employees per vehicle. Measures the efficiency of staffing; ratio From staffing and
overhead should fall as an organization grows. vehicle numbers
Member % "very satisfied" or "satis- Simple satisfaction measure. A range of more Member surveys
satisfaction fied" with car-sharing service. sophisticated measures are available; see
TCRP Report 88 (Kittelson & Associates,
2003a).
Member % of members leaving each Customer satisfaction measure. Exit surveys Member database
retention year. can probe reasons for leaving.
Farebox Ratio of member fees to total Progress towards financial self-sufficiency Financial state-
recovery expenses. or profitability. Similar to standard transit ments
industry measure.
Page
7-17
OCR for page 236
Chapter 7 · Procurement and Monitoring
Exhibit 7-5 Car-Sharing Performance Measures (cont'd)
Measure Definition Significance How Measured
Output Measures (usually proprietary)
Number of Number of members. Simple measure of size and penetration. Member database
members
Active mem- % members using service in a Assesses whether members are active users, Reservations logs
bers month. dormant or have signed up for "mobility insur-
ance."
Number of Number of car-sharing vehicles Simple measure of size and penetration. Fleet database
vehicles in service.
Low-emission % vehicles that are hybrid, Assesses uptake of clean-fuel technology. Fleet database
vehicles CNG or electric.
Car-sharing Number of (i) approved and (ii) Assesses uptake of car-sharing by develop- Fleet database
developments occupied developments that ers.
incorporate car-sharing.
Outcome Measures
Vehicle travel Net change in annual Vehicle Assesses impacts of car-sharing. Member surveys
Miles Traveled (VMT) in private or travel diaries
vehicles.
Vehicle Net change in number of ve- Assesses impacts of car-sharing. Member surveys
ownership hicles owned. Where members
report having avoided vehicle
purchases, this should be
reported separately.
Emissions Net change in CO2, CO, NOx Assesses impacts of car-sharing. Fleet database
or other pollutants. Based and VMT change
on change in VMT and fleet calculation, using
composition. factors adopted
by local air quality
regulators
Transit rider- Number of new transit trips Assesses impacts of car-sharing. Includes Member surveys
ship generated each year. ridership from transit access trips to car-
sharing, as well as wider changes in travel
behavior.
Parking Net reduction in parking Assesses extent to which car-sharing Planning Depart-
spaces saved provision in developments that changes the form and auto-orientation of new ment data. Diffi-
incorporate car-sharing. development. cult to quantify as
parking variances
are often granted
for multiple
reasons.
Mobility Perception of increased mobil- Assesses impacts of car-sharing. Member surveys
ity among members (e.g. ability
to reach new destinations).
Fleet savings Annual change in cost of Assesses cost savings from car-sharing. Requires "before"
corporate vehicles (rental cars, and "after" data
car-sharing, vehicle fleet, and on fleet costs plus
mileage reimbursements). costs of car-shar-
ing
Page
September 2005
7-18
