John Vangelov, Ford Motor Company
John Vangelov manages the embedded modem features group at Ford Motor Company. He shared a history of automotive electronics before discussing the update challenges faced in the automotive sector today, including the limitations to network capabilities and the challenges of software delivery methods.
When they were first developed in the 1980s, electronic control units (ECUs) were housed on chips that were either erasable-programmable read-only memory (EPROM) or masked read-only memory (MROM), Vangelov said. EPROMs were slow to program but could be erased and reused, although the process was tedious. MROMs were a better value, because they could be programmed on a large scale, but they couldn’t be updated or rewritten. There was also EEPROM (electronic EPROMs), which was cost-prohibitive for wider use but suitable for storing essential automotive configurations or calibrations.
Wherever software was housed in the early days, this software was very difficult to alter once it was in vehicle production, Vangelov said. Most issues were addressed via
direct part replacement. Customers were notified of required changes and brought their vehicles into the dealer, where the ECUs were physically replaced.
As new automotive features were introduced, new software systems were developed. First came the engine controller, which optimizes car performance by reading sensors and sending controls to the car engine; the controller was invented to help cars meet Clean Air Act requirements. “The engine controller was really the inflection point where we started bringing in a lot of electronics,” Vangelov reflected.
Next, centralized body control modules appeared, which included new electronic features such as interior lights, air conditioning, and one-touch power windows, mirrors, and locks. These control modules replaced separate mechanical controllers for each accessory. ECUs were also central to anti-lock brakes, which monitor wheel speed and control the brake valves to make driving safer.
In the late 1990s and early 2000s, ECU processors started using flash memory, said Vangelov, and engineers developed several paths for updates to the operating system (OS) and applications, such as on-board diagnostics (OBD) and BDM or JTAG connection ports. Some ECUs had programmable pins on the BDM and JTAG ports, which raised security concerns and were eventually pulled from production, Vangelov noted.
During this time, designers were integrating more and more ECUs into car features and rapidly increasing their capabilities, requiring far more memory and processing power than ever before. A modern seat controller, for example, uses pulse counts and a memory switch to store a driver’s preferred seat position; steering wheels and interior temperature can similarly be controlled with ECUs.
As the applications and the OSs became more sophisticated, software code did, too. ECUs are now written in higher-level languages (such C or C++), Vangelov observed, and require more memory and processing power. At the same time, ECUs decentralized some functions by leveraging the increasing network communication and processing from other computerized modules in a vehicle. This increased communication, however, meant that cars needed more internal bandwidth.
To meet the continually increasing demand for new features and the software to support them, the automotive industry created guidelines for three components: common library components, a common OS, and communication and diagnostic standards.
In the late 1990s and 2000s, Vangelov said, car makers began to allow mechanics to use diagnostic testing tools to install software updates. Small updates took a few minutes, but depending on how much software was in a car, some updates could take more than an hour. In order to shorten that process, car makers then introduced ECUs that could be updated wirelessly, a significant shift that opened many new opportunities.
Today, limited networking capabilities are a significant challenge for Ford, Vangelov said. While FlexRay, automotive Ethernet, and MOST150 connections boast data speeds of 10, 100, and 140 megabits per second, respectively, those high-speed networks are not available for the OBD connector, so they can’t be used to deliver a software update to a car’s ECU via a diagnostic tool. Controller Area Network (CAN), the one most commonly used by Ford, Vangelov said, provides automotive software updates at about 500 kilobits per second.
In 1999, diagnostic capabilities were standardized through the International Organization for Standardization (ISO) 14230-Keyword Protocol 2000. Before that, diagnostics were disjointed and often proprietary, Vangelov noted. In 2013, Unified Diagnostic Services and ISO 15765 developed and defined diagnostic services for software update deliveries. Diagnostic tools are the primary method for delivering the majority of the ECU software updates, Vangelov said, although there are several other methods, including CAN, USB (mostly for updating the “infotainment” systems), Wi-Fi, cellular, and Ford’s SYNC 3 system. Vangelov noted that Ford has been using Wi-Fi to synchronize system updates with its assembly plants since 2010, which helps reduce the need for immediate or complex updates after a car has been purchased.
Size is a growing challenge for delivering software updates in the automotive industry, Vangelov said. A car today can have more than 10 million lines of code, compared to about 50,000 or so lines in the 1980s. Updates now often require multiple decentralized ECUs to be updated at the same time, which requires coordination and increases the size of the software delivery. This, in turn, means more time and bandwidth are required.
That growth shows no signs of stopping. Software size and complexity will continue to increase as advanced driver assistance systems, autonomous vehicle mapping data, connectivity applications, and other current capabilities continue to evolve.
Another challenge is software integration. The operating systems, their services, and the application logic are often not optimized for software deliveries. As a result, Vangelov said, “If I have to deliver an update to a module, I’m updating everything.” One opportunity for the industry to improve update deliveries would be to structure the architectures in a way that allows updating of single ECUs, he said. This would
drastically reduce the time it takes to deliver updates, without losing any functionality of other components.
Steven Lipner, an independent consultant, asked Vangelov to delve deeper into this problem in the discussion. Vangelov said that some OS components are better partitioned, but other systems have a conglomerated binary that requires coordination for updates. Also, software updates themselves are often structured in a way that only allows delivery of the entire package at once, which takes a long time. Ideally, each specific component’s software could be partitioned, Vangelov said, but unfortunately ECUs are not written that way, and even if they were, that could create a different problem because multiple parts and subparts would all need to be individually managed and updated across a distributed system of ECUs.
A car’s power supply presents additional challenges. As computerized as cars have become, the vast majority are powered by a lead acid battery and internal combustion engine, a power supply system that is simply not optimized for long software updates without additional charging. Very large updates could risk draining the battery, which could erase the module being updated and even deprogram some of the car’s ECUs, making them inoperable, Vangelov said.
As in many industries, legacy products pose another software challenge for automotive ECUs. They are not typically included in verification and validation cycles done for new product releases and so must be pursued in a different development track, Vangelov noted. A further complication is that any update issued to ECUs in older vehicles must be verified to meet requirements for cars on the road today, which could be different from the requirements for which the systems were initially designed years ago. In addition to the development challenges this raises, software updates extensive enough to meet new requirements can sometimes exceed the capabilities or the memory available on the original system, making it essentially impossible to implement the fix, he said.
The technology supporting software update delivery is also constantly changing. Companies take away old products or services in order to free up space for new product offerings. For example, he noted, AT&T recently retired its 2G network, and customers with 2G-connected phones can no longer receive updates. It may be only a matter of time before 3G or 4G, which some cars use for software updates, meet a similar fate. Even before technologies become obsolete, regional differences can undermine their utility for delivering software updates reliably. Globally and even domestically, cellular
service can be inconsistent or intermittent, which can affect update speed, time, or even whether Ford can connect to a vehicle at all.
User experience is another important factor, and the challenges for the automotive industry are in some ways reflective of those seen in the technology industry more broadly. While “tech guys” like Vangelov are interested in the latest and greatest software capabilities, many average users just want their devices to work, without having to know how it works or navigate cumbersome solutions, he said.
One important need is to better communicate the need for software updates. “We really need to focus on how we drive the expectations for software updates to our customers,” Vangelov said. Other changes could help make the update process more palatable for drivers. For example, many devices have to be offline while they are updated, which means a car can’t be driven at all. Longer update times also mean that certain accessory functions, like the radio, could be unavailable during drives. Minimizing the update time would improve the update experience for customers.
In the discussion, Will Drewry, Google, Inc., followed up on this point, asking whether doubling the amount of storage would help to improve the user experience of updates by allowing the car to operate the old system while the new one is being installed. Vangelov said that Ford’s new SYNC 3 system uses this approach to run updates in the background while all of the car’s functions remain in use.
Forum Chair Fred Schneider asked Vangelov to speak to the issue of security, inquiring specifically about whether it is possible to “brick” a car. Vangelov explained that when a car gets an update, it’s not an update for “an entire computer” but rather “a conglomeration of multiple ECUs.” In order to brick a vehicle, everything would have to be updated at the same time and the lead acid battery would be drained in the process. If, in the middle of updating certain critical components, such as the engine controller itself, the update were interrupted and it couldn’t resume operations, that would leave an opening, he said. Ford is aware of this possibility and is looking for ways to strengthen security—not just for safety-critical components, but ensuring that the update mechanism is robust against all potential failure modes for all ECU modules.
Vangelov emphasized that update security is a primary aspect of his work at Ford. For example, Ford does verify keys on ECUs and signs binaries that are delivered over the air. Before a software update can be downloaded, a user must first go through several steps that determine both an update’s validity and the validity of the vehicle and user.
Tadayoshi Kohno, University of Washington, asked if Ford was concerned about the potential impacts of aftermarket components such as stereos or replacement transmissions in terms of software compatibility or security. Vangelov agreed that this phenomenon poses a problem, irrespective of how software updates are delivered. Before any updates are delivered, the system is designed to conduct an interrogation of all a vehicle’s parts, which should help determine whether a new component could compromise an update, or if it will react safely and as expected. “If the interrogation results in something that seems foreign for that particular vehicle, it’s not a build that we recognize, we will actually not deliver the update because we don’t have a deterministic understanding of what the expectation would be if you were to deliver it downstream,” said Vangelov.
Peter Swire, Georgia Institute of Technology, asked Vangelov to explain whether the connections among different ECUs in a car could allow a hacker to get further into the car’s controls after breaching one component. Vangelov explained that while the ECUs do interact, that interaction is at the networking signal communication level. For example, an anti-lock braking system (ABS) module can read certain speed information from the CAN bus. However, that doesn’t mean the ABS module can alter what is on the CAN bus, and so if the ABS module were being updated, the update couldn’t affect the CAN bus, because it’s not involved in the update. If, however, the update was meant to fix a communication problem with the ABS module reading the CAN bus data, then the two systems would have to be fixed together.
Swire asked if there were one main component of a car, central to its operation and connected to all the other accessory components, whose software, if hacked, could allow the car to be turned into a weapon. Vangelov replied that there are some systems, particularly those related to powertrain systems or the chassis, that are obviously essential to a car’s operation. Some models have these systems segregated and some don’t, and Vangelov noted that he could see a situation “potentially go awry if there were a dependency between the two and they weren’t both updated at the same time.”