6


Ensuring Robust Firmware Updates

Dave Whitehead and Edmund Schweitzer, Schweitzer Engineering Laboratories

Dave Whitehead, vice president of research and development for Schweitzer Engineering Laboratories (SEL), presented a detailed look at SEL’s industry space, products, and update challenges. Edmund Schweitzer, president and board chairman of SEL, was on hand to answer questions.

SEL is a software and firmware company that specializes in inventing, designing, building, and supporting electric power systems for industrial companies and global utilities. With a mission to make electric power safer, more reliable, and more economical, Whitehead described SEL as “the brains of the electric power systems.” While all of the design, manufacturing, and support is done within the United States, SEL serves clients in about 160 countries, making it a global company.

UNIQUE ATTRIBUTES OF SEL’S MARKET SPACE

SEL makes a suite of products that are meant to work in concert to support electrical power systems. The company’s protection and control devices measure electric current and voltage, determining at every millisecond whether a circuit breaker or a power line needs to be tripped. SEL also makes automation equipment, such as special-purpose

computers that interpret data and make higher-level decisions about power use, passing those decisions up to an information center or down to a protection device. In order to keep these systems secure, SEL also builds switches, firewalls, and gateways. Finally, SEL makes operations software that enables client technicians to make decisions, such as determining when to send control commands to shut a power line down for maintenance. “It’s a very layered approach to how we go about controlling and operating the power systems,” Whitehead said.

Electric power systems are managing electric power and transmission concepts that are more than 100 years old and were created well before the Internet and firmware. Whitehead noted that this is actually good news for security: If power systems lose communication or other computerized controls, they can still remain functional. On the other hand, our strong reliance on electricity today means the stakes are extremely high. “When we mess up, letters get written to the President . . . so we take this stuff very seriously.” Whitehead said.

Whitehead noted that SEL has several advantages that more general software companies like Microsoft or Cisco do not. For example, it has the luxury of building its systems to do one thing only, which is to protect power systems: “We don’t have to be all things to everyone,” he said. SEL benefits from the ability to tailor its systems to their intended use and only include those necessary features. In addition, SEL systems are created to be static. Client utilities expect to use their designs for 20 years, with minimum upgrades, because once the systems are built, they are considered “done.” Therefore, in general, SEL does not anticipate frequent firmware updates, although he noted that one of their devices does use a Microsoft product, so they must be aware of any patches and notify affected customers accordingly.

SEL’S APPROACH TO SECURITY

Whitehead described security as integral to the SEL product life cycle and said this ethos is reflected in well-defined, robust engineering principles implemented all the way to Capability Maturity Model level 5. Founder Ed Schweitzer drew on his previous experience at the Department of Defense to place security at the forefront of the company’s products from the get-go. For example, from its first product 30 years ago to today, SEL has had two levels of access to its devices: one for a technician to view data, and another that allows a higher-level user to set and configure a device.

In addition, SEL engineers own every piece of code in its devices and have revision control for every device in its factories. They also use automated test code analysis and human teams to review every piece of firmware top to bottom. They create single binary

files that match a firmware product to its version number, allowing them to track and fix any security problems. The final step in product development is negative testing, in which engineers deliberately try to break every aspect of the firmware. “We really want to beat up the product before we ever release it to any of our customers, because . . . once I release a product out into the electric sector, we’ve got one chance to get it right,” Whitehead said.

SEL also provides a mechanism for customers to verify that all firmware is from SEL before installation, which further enhances security. SEL also now includes signatures in its software, preventing it from being loaded onto SEL firmware until it passes cryptography checks, Whitehead said.

MANAGING FIRMWARE AND SOFTWARE UPDATES

SEL takes a somewhat unique approach to managing firmware upgrades: The company itself keeps track of which customers are using every piece of firmware, so it can alert those customers when updates are needed. SEL also provides education and training for customers to implement SEL products securely, even providing cybersecurity services tailored to each client’s needs.

Firmware management has evolved since SEL began. In the 1980s, firmware was manually upgraded—an engineer had to physically remove an old memory chip from a device and add a new one. In the 1990s, flash memory enabled firmware updates through communication ports. In the 2000s, substations began using Ethernet, which made upgrades easier but also presented cybersecurity challenges, and so the Internet was not often used for firmware upgrades. While SEL could push out firmware upgrades today online, because of the security risk, it avoids this approach, particularly for upgrades to sensitive functionality like protection devices. A technician will instead visit a client site, take the whole device out of service to mitigate any security risks or errors from the upgrade process, and then manually install new firmware.

If a defect is found, SEL will resolve it, push out updates, and notify customers through a security bulletin. Whitehead noted that the company issues a security bulletin each month whether there have been any updates issued or not; this helps customers prove to regulators that their systems are up to date. If there is a security defect, the client has 30 days to evaluate whether or not it affects its equipment, and if so, whether the equipment needs to be taken offline for servicing, which can be a major undertaking involving a planned outage. The firmware is then upgraded, tested, and verified before being put back into service.

Prompted by a question from Richard Danzig, Johns Hopkins University Applied Physics Laboratory, Whitehead elaborated on this 30-day window during the discussion. The window starts when SEL notifies a client about an available fix for a discovered vulnerability. The client then has 30 days to decide whether the vulnerability applies to them, what steps they will take, and how they will mitigate any risk. While the actual fix does not have to be implemented within 30 days, the operator must be able to demonstrate a plan by the end of that period and potentially justify any delays in an audit. While clients may prioritize certain updates over others, by and large they are very responsive to these notifications, Whitehead said.

SEL also creates updates that provide new features, usually in two to four releases per year. These deployments are handled separately from security updates so that clients can respond to the two types of updates accordingly. Whitehead noted that policy-driven software updates can be problematic, because policies can never truly fit every possible situation and sometimes can end up driving out other creative solutions to problems. In his view, the U.S. government should inform power system operators of security threats, but leave specific decisions about features or security updates to those who own the system’s assets.

OTHER MECHANISMS FOR MAINTAINING SECURE SYSTEMS

While power systems operate on networks, Whitehead stressed that there are critical steps to ensuring these networks cannot be penetrated by attackers via the Internet. First and foremost, he said, operators should never connect their systems to the Internet. They should also conduct audits to ensure there are no such connections. SEL clients maintain private, secured networks for their control networks that are completely separate from any other corporate information technology functions, said Whitehead.

In addition to layering and separating different parts of a system and properly educating clients on security defenses, Whitehead noted that SEL also scrutinizes and verifies every aspect of its supply chain in order to track parts, enhance security, and build trust. Other SEL methods for increasing security include maintaining private security plans, compartmentalizing systems, and monitoring and investigating any unusual events.

Because they involve physical systems, one advantage of SEL products, Whitehead noted, is that it is easier to identify problems compared to, for example, online banking, where it can be hard to determine what exactly has happened when there is a breach. Because all of the systems in use are constantly measuring the power system, information about any unusual occurrence can be validated among the systems.

THE “AURORA” VULNERABILITY

At Danzig’s request, Whitehead and Schweitzer described an incident known as Aurora and discussed its security implications. In a power system, a circuit breaker has two parts. When a breaker is opened, the frequency and phase must be the same on both sides; if they aren’t, it can cause excessive torques in the rotating machinery and cause damage. At Con Edison, Schweitzer’s grandfather created a way to record every instance of this occurrence after observing that operators would sometimes intentionally cause it to happen in order to enjoy the noise and drama of watching the breaker synch up again.

An intentional demonstration of this phenomenon, called the Aurora vulnerability, in 2007 raised fears that it could be exploited to break electric power systems. In the incident, operators set up a 4-megawatt diesel generator incorrectly on purpose and then physically manipulated it to trigger the Aurora phenomenon. The incident was not caused by a firmware vulnerability, Schweitzer said, and using this approach to attack a power system would require such a long chain of events that the company does not view it as a significant vulnerability.

FOSTERING DEVELOPMENT EXPERTISE AND BROADER LESSONS FOR THE SOFTWARE INDUSTRY

Bob Blakley, CitiGroup, expressed his admiration for SEL’s robust development process and asked Whitehead how the company manages to recruit qualified developers. Whitehead explained that a close association with Washington State University allows SEL to tap engineering and computer science talent early and train young developers on the company’s methodical engineering design process. While the universities teach the fundamentals of mathematics and engineering principles, Whitehead emphasized that industry is where design and practice are taught, so SEL expects to need to train recruits on its process.

Tadayoshi Kohno, University of Washington, asked if SEL represented the industry as a whole, or is ahead of its competitors, and whether there are any specific lessons the SEL experience could offer for the software industry more broadly. Schweitzer responded that SEL evaluates its products in comparison to those of competitors and remains confident in its work. In terms of broader lessons, Schweitzer noted that the intelligence community could be more involved, and that industry-wide sharing of information about threats could also improve security. For example, SEL writes many technical papers every

year that become resources for the broader community. One, for example, focused on how the company would know if power systems had been hacked, concluding that because it’s easy to measure use in power systems, a hack would be fairly obvious. Other resources include SEL’s in-depth training and its Modern Solutions to Power Systems Conference. Schweitzer noted that this year’s conference theme, “The Roots of Cyber Insecurity,” could be particularly relevant to this discussion.