7
Updates in the Consumer Electronics Industry
Will Drewry, Google, Inc.
Will Drewry is a principal software engineer at Google, Inc., where he is responsible for ensuring privacy and security protections on consumer products. He described the challenges involved in building and securing consumer electronics, including the use of off-the-shelf components, business-side considerations at play, and a variety of technical factors.
Drewry explained that the products he has worked on typically include an operating system and an application layer on a physical device such as a phone or laptop, although these products have a variety of update strategies. While one product line receives feature and security updates every 6 weeks, another has monthly security updates and less frequent feature releases, and still other products follow an “update-on-demand” schedule.
CHALLENGES FROM USING OFF-THE-SHELF COMPONENTS
While the makers of operating systems and Internet of Things (IoT) devices intend to make secure products from the beginning, it can be difficult to do when working with commercial, off-the-shelf parts and trying to keep up with the speed of production
required to survive in the technology industry, Drewry said. In this environment, the potential for updates can be seen as a “crutch,” because security can always, in theory, be improved after deployment.
Drewry offered a hypothetical example of a new device with an application platform that can add more and more features over time. A designer could choose to start layering in security in the browser, the application layer, or somewhere else. The security process might be rushed if the manufacturer wants to bring this device to market quickly and knows updates can be made later. Most builders use best practices such as compartmentalization, tracking information flow, and balancing when assets are available, but no code is invulnerable.
Many products, including all of the ones Drewry has worked on, use open-source components like Linux kernels, he said. The kernel has to be inspected to determine if it’s trustworthy, and kernels often lack a full annotation of security bugs; because it’s unclear which updates include security fixes and which do not, Drewry emphasized his belief that it is important to update the Linux kernel frequently.
Security is difficult when working with commercial off-the-shelf parts and trying to keep up with the speed of production.
Drewry also oversees the hardware components of these devices, where the main processor has to interact with the software kernel being used. This is complicated by the fact that these processors are built by third parties and may not be compatible with a certain kernel, or the company may not offer direct customer service. Processors are often manufactured for short-term use; for example, a camera component on sale now will likely be replaced with a newer one within a couple of years. In that instance, if Drewry is using that older camera in his hardware, he will have to update the kernel, which becomes an increasingly complex process, or ignore the updates.
The Linux kernel isn’t the deepest layer to be concerned about, either. Mobile or device processor vendors layer software on top of a chip—that may itself be modified or adapted, creating an opening for problems. A few years ago, for example, Intel released microcode updates that could inadvertently alter instruction codes, causing some alarm. The full-featured operating systems sold today in “Trusted Execution Environments” may not always be trustworthy, Drewry noted. Computer chip makers feel pressure to include software on their products as a way to differentiate themselves from others, although Drewry would prefer that they come without it. “You can’t even pay them to take it off,” Drewry said. “I keep trying.”
All of these extra layers from the chip up through the software pose added risks. “I think there are lots of ways you can ameliorate that risk, but the farther up you go, it gets harder,” Drewry said.
BUSINESS-SIDE CONSIDERATIONS
Drewry faces these challenges from the perspective of an industry giant; he noted that the many smaller companies and individuals, such as those launching products with funding from Kickstarter.com, for example, rarely have the resources needed to invest heavily in software security. Instead, they are typically entirely dependent on the protections that are already built into off-the-shelf components available from third parties. Google has sufficient resources to spend time compartmentalizing software that it purchases from code that it writes in-house. The company also isolates the Linux kernel from other layers, and invests in projects like the Kernel Self-Protection Project (supported by Linux’s Core Infrastructure Initiative, described by van Someren) that increase kernel security. But this process can be messy even within a company like Google, and navigating such protections could be much harder for smaller companies.
The mechanisms to support the sort of secure development life cycle discussed throughout the workshop could soak up a lot of a company’s bandwidth, Drewry emphasized. Creating updates is a business expense. Key management is also necessary to keep updates secure, but that brings in more people, more technology, and enormous expenses, “just to get a modicum of assurance,” Drewry noted.
Because companies want their customers to use the updates, it is also important to carefully manage how and when they are deployed; updates can’t be so frequent that users ignore them, and they can’t cause frustrating instabilities. In addition, as mentioned previously in the workshop, even the most secure updates in the world still need a secure delivery route to the client, who then has to deploy them correctly. A company’s “update agent” might need to build a secure connection to send the update, creating another layer that has to be implemented, protected, and given privileged access to the relevant devices.
Drewry shared an experience he had shipping an update for Chromebooks: To avoid the situation where users cannot update without logging in (and thus being immediately vulnerable to an attack), Google devised a process by which users could get an update on the log-in screen. This also avoided customers having to be walked through a tedious workaround to install the necessary updates and finally log in.
Experiences like that demonstrate that in addition to thinking about how to provide security, developers must consider how that security can be maintained within
a system that comprises multiple interconnected layers, he said. In a world in which even the chips have software, nothing can be taken for granted. Windows 10 recently added the “Windows Trusted Boot.” Older TiVo devices used to rely on the kernel to check that certain key files had not been altered, and Google is constantly checking all the software signatures and configurations every time a device is restarted, not just when software is first loaded.
ADDITIONAL TECHNICAL COMPLEXITIES
Rollback protection, a mechanism to prevent the system from reverting to an older version of the software, raises additional challenges, Drewry said. The record of having converted to the new version must be stored in a way that will prevent tampering, or at least show evidence of having been tampered with, and this protection must be stored in an appropriate place. One of the most common choices is to use Replay Protected Memory Block (RPMB), built on software and firmware, in the main storage device. That is still problematic, he noted, due to the increased reliance on the storage device vendor’s software development and key management and update practices. Because processors run on a process size that is too small to include flash storage, this functionality cannot be put on the chip itself. As a result, “We lack this essentially critical secure storage for making decisions,” Drewry said, and developers have become reliant on third parties that are part of their trusted network.
Another complex issue is the “single signer” problem. Although users might be protected from unauthorized attacks, what if the signer is the one compromised? “We haven’t actually solved the problem of protecting them against us,” Drewry said, emphasizing that even 1-bit code errors can become vulnerabilities in the right context.
Drewry discussed the importance, and challenges, of preventing targeted attacks. Although Google focuses on making it difficult for hackers to target specific users, it’s still fairly easy to identify people with just a few pieces of information. Removing unique identifiers from the system can help make it harder for attackers to target a specific victim.
Broadly speaking, while software transparency may be an excellent reactive mechanism and make attacks more complicated, Drewry emphasized that it is not proactive and cannot protect users from having their identity or information compromised in the first place. It is important for companies to seek a more proactive solution, he said. One avenue that could improve security updates is finding the right incentives so that designers want to build more secure systems, he suggested. Drewry believes incentives for chip makers, operating system designers, and other key players
will be a strong motivation. “There is room in the industry to provide these tools,” he said, adding that Linux and big companies alike could help move the process along.
Eric Grosse, an independent consultant, asked Drewry if there were best practices the community could use to prevent a new device, fresh out of the box, from being immediately attacked (as Ed Paradise had described in his presentation), while still in need of a software update. Is there a way, he wondered, to build in special protections to keep a device from being penetrated on its very first startup?
Drewry expressed agreement that initial startup could present added vulnerabilities, offering as an example the story of Chromebooks that were manufactured in large numbers and then ultimately sold by resellers up to 2 years later. Seeing the risk that these machines would be significantly more vulnerable to attack if the on-launch software update took longer than 30 minutes, the company focused on creating an update that could be complete within that time window. Another solution being used increasingly is to develop systems that update the software on devices while they are waiting in a factory or warehouse so that the software isn’t too outdated by the time the device reaches the customer, he said.
