National Academies Press: OpenBook

The Internet Under Crisis Conditions: Learning from September 11 (2003)

Chapter: 2 The Network Experience

« Previous: 1 Introduction
Suggested Citation:"2 The Network Experience." National Research Council. 2003. The Internet Under Crisis Conditions: Learning from September 11. Washington, DC: The National Academies Press. doi: 10.17226/10569.
×

2
The Network Experience

September 11, 2001, started out more or less routinely on the Internet. Early Tuesday morning is a common time for Internet service providers (ISPs) to schedule maintenance activities on their network, and on that particular Tuesday there were some instances of delay or packet loss between 2:00 A.M. and 5:00 A.M., when Verizon updated software on East Coast frame-relay switches and other ISPs made changes in their networks. But by 6:00 A.M. Eastern time, it appears that the Internet routing and traffic loads were normal for the start of a workday.

That normalcy would be shattered for the Internet, as for so many other operations, when American Airlines Flight 11 crashed into the World Trade Center’s North Tower at 8:46 A.M. Within minutes, major online news sites were struggling to serve between 3 and 10 times their normal load as Internet users sought details. One news Web site estimated that traffic to its Web servers was doubling every 7 minutes, beginning around 8:50 A.M., until about 9:30 A.M.

By just after 9 A.M., when United Airlines Flight 175 crashed into the World Trade Center’s South Tower, the Web sites of CNN, MSNBC, the New York Times, Yahoo! News, and others were observed to be slowing significantly. The cause would later be reported to have been the loads on these sites’ servers, not connectivity problems in reaching servers across the Internet. Then the South Tower collapsed, damaging equipment and circuits in the Trade Center complex. The subsequent collapse of the North Tower, the collapse of World Trade Center Building 7 (a 47-story structure), damage to the neighboring Verizon central office, and power

Suggested Citation:"2 The Network Experience." National Research Council. 2003. The Internet Under Crisis Conditions: Learning from September 11. Washington, DC: The National Academies Press. doi: 10.17226/10569.
×

cuts in Lower Manhattan all had disruptive effects on the Internet and other communications systems.

How did the Internet’s communications infrastructure in particular experience all these events? How much did the events in New York City, and in Washington, D.C., affect the movement of data throughout the Internet? How were ISPs affected by the events of September 11? How serious were the impacts? What actions did ISPs (and others) take in response?

This chapter sets out to answer those questions, as best they can be answered with the available information. Data pertaining to the Internet operations that day were of two types: quantitative data on the system as a whole and on the response of particular networks, and anecdotal reports from network operators, users, and news media that help provide context and possible explanations for the changes on the Internet, both at the macro and micro levels, that were deemed necessary after the attacks.

How comprehensive and authoritative is this information? Some of it—for example, data on changes in the Internet’s routing configurations— permit the overall impact on the Internet to be measured. Reports on specific incidents, on the other hand, do not allow generalizations about the whole system, though they do provide insights into the kinds of local problems that could arise in the future and the responses that may mitigate them. Still, the participation in this study of several national ISPs and one New York regional ISP, together with the anecdotal information obtained though informal information-sharing relationships within the Internet operator community, permit at least a reasonable sampling of the overall experience. In addition, user surveys taken by the Pew Internet and American Life project allowed the committee to relate reported user behavior to some ISP measurements.1 However, in a number of instances, data that would inform the committee’s understanding of what transpired on and shortly after September 11 were lacking (a detailed discussion of Internet-measurement issues is presented in Chapter 5).

OVERVIEW OF DAMAGE AND IMPAIRMENT

The terrorist attacks in New York City caused an immediate disruption in communications within the World Trade Center complex. Soon thereafter, the collapse of the Twin Towers damaged and destroyed equipment of several wireless providers and some data circuits serving the

1  

Lee Raime and Bente Kalsnes. 2001. The Commons of the Tragedy: How the Internet Was Used by Millions After the Terror Attacks to Grieve, Console, Share News, and Debate the Country’s Response. Pew Internet & American Life Project, Washington, D.C., October 10. Available online at <http://www.pewinternet.org/reports/toc.asp?Report=46>.

Suggested Citation:"2 The Network Experience." National Research Council. 2003. The Internet Under Crisis Conditions: Learning from September 11. Washington, DC: The National Academies Press. doi: 10.17226/10569.
×

New York City area, the northeastern United States, and Europe. During the rest of the day on September 11, local power failures caused temporary equipment outages.

These events had several types of effects on ISPs and Internet users, including the following:

  • Loss of Internet connectivity in the vicinity of the attacks. The effects in New York City were extensive as a result of the catastrophic damage at the World Trade Center site, the large number of nearby institutions, and the important role that New York City plays in the Internet infrastructure. Two main factors contributed to the loss of Internet connectivity—the permanent destruction of networking equipment at the site and the loss of power and cooling in adjacent areas for varying lengths of time. (By contrast, the attack in Washington, D.C., did not appear to have a direct influence on network connectivity for institutions outside the Pentagon.)

  • Connectivity loss at “out of town” locations in the United States and in other parts of the world. Several ISPs elsewhere in the United States and overseas experienced connectivity problems resulting from the loss of fiber-optic lines that ran through Manhattan and the temporary disruption of access to Manhattan-based services. (The experience of these networks and providers offers insights into how to plan for future incidents.)

  • Surges in demand for some Internet services. As word of the attacks began to spread, Internet users turned to a variety of news sites for more information. The greatly increased load on these sites made it difficult for all requests to be met.

COLLAPSE OF NORTH AND SOUTH TOWERS

Internet facilities were destroyed when the World Trade Center’s Twin Towers collapsed. Several ISPs, including AT&T Local Systems, Genuity, Verizon, and WorldCom, had points of presence (POPs)—facilities at which customers are connected to an ISP’s network—located in the Trade Center complex. Also, a number of fiber-optic cables ran through the complex in conduits, and circuits of one major telecommunications carrier ran through the Port Authority Trans-Hudson (PATH) sub-site rail tubes that link Manhattan and New Jersey. MFN alone reported the loss of more than 1,300 optical fibers as a result of the towers’ collapse.

Building 7 Collapse and Damage to Verizon Central Office

At approximately 3:45 P.M., ISPs received reports of a fire in or near the Verizon central office at 140 West Street. From a local perspective, the biggest effects probably came from the collapse of Building 7 of the Trade

Suggested Citation:"2 The Network Experience." National Research Council. 2003. The Internet Under Crisis Conditions: Learning from September 11. Washington, DC: The National Academies Press. doi: 10.17226/10569.
×

Center complex at about 5:20 P.M.; this collapse caused extensive damage in Verizon’s central office, across the street at 140 West Street, crushing the walls and cable vaults and causing the office to begin to flood. The result was disrupted service over the course of the evening. Some 14,000 business and 20,000 residential customers lost telephone service (approximately 300,000 voice circuits). Data communications, with a total capacity equivalent to 3.6 million 56 kilobit-per-second (kbps) circuits (or 90 OC 48 SONET links), were also disrupted. Ultimately, all customers directly connected to equipment located at West Street lost Internet service. Several competitive local exchange carriers (CLECs) and ISPs also had equipment in the West Street building, and service in their networks was affected as well.

Damage to 140 West Street also caused further damage to fiber links already compromised by the collapse of the Twin Towers. In some cases the fiber-optic infrastructure had self-healed by routing around the damage done by the Towers’ collapse; the SONET fiber-optic rings commonly used for metropolitan-area networks can be configured to automatically recover in the event of a single cut in the ring. But the infrastructure was not designed to heal from a second break in the fiber.

As a result of these events, Internet connectivity to several universities, medical colleges, and hospitals, and to the city government’s official Web site, was interrupted. ISPs took a number of steps to restore connectivity, as described below.

Electrical Power at Co-location Sites in Lower Manhattan

In addition to the direct effects from the collapse of the Twin Towers and Building 7, there were indirect effects of the attacks, especially regarding electrical power. These disruptions had consequences for other critical telecommunications facilities, even those located outside the area of the attacks’ direct physical impacts.

To be sure, telecommunications facilities operators make provisions for power failures. ISP co-location facilities and telephone central offices contain backup batteries and generators. The exact battery time and fuel capacity of individual offices is not public information, but they generally are provisioned for between 8 and 72 hours of backup in case of commercial electric-utility failure.

Most facilities routinely test their backup systems to ensure that they work. However, it is still not uncommon for a backup system to fail to start up correctly when regular power fails.2 Still, by 4:35 P.M., several

2  

Participants at the committee’s meeting in Washington, D.C., estimated that backup systems fail to start correctly in about 1 out of 10 tests.

Suggested Citation:"2 The Network Experience." National Research Council. 2003. The Internet Under Crisis Conditions: Learning from September 11. Washington, DC: The National Academies Press. doi: 10.17226/10569.
×

Internet co-location facilities near the World Trade Center complex were operating on backup generators because commercial utility power had failed or been turned off (by Consolidated Edison, the local electric utility).

Luckily enough, the backup power systems at all the co-location and phone facilities in Lower Manhattan apparently turned on properly when Consolidated Edison was forced to turn off power just before 10:00 P.M. on September 11. However, as the power outage extended over multiple days (past the planned life of the backup power systems), maintaining power became a serious issue as batteries expired and backup generators ran out of fuel.

Following the loss of the grid electricity supply, some fiber-optic links, which depend on electrically powered termination devices at each end for sending and receiving pulses of light over the fiber, failed. It is standard practice to attach these devices to batteries, which provide several hours of backup power. However, several providers apparently failed to anticipate significantly longer power outages, such as providing additional backup using electrical generators. Consequently, in the early morning of September 12, some minor perturbations in Internet connectivity occurred in the New York City area when the backup batteries supplying optical devices ran out of power themselves.

Although power problems did not persist beyond the month of September 2001, some longer-term effects may still remain. In particular, some networking equipment in the area around the World Trade Center site may have sustained damage from overheating, caused by the loss of power to cooling systems and excessive dust in the air conditioning. One concern from such incidents is that the affected equipment, though still operating, could be less reliable in the future.

INTERNET-WIDE (GLOBAL) PHENOMENA

Routing and Reachability

Changes in routing information exchanged using the Border Gateway Protocol (BGP) indicate changes to the Internet Protocol (IP) routing topology (see Chapter 5 for more details). Route withdrawals (indicating that a path to a group of Internet addresses is no longer available) and advertisements (indicating how to reach a group of Internet addresses) are routine events. They occur, for example, as circuits go up and down and when network operators make changes in their networks to adjust traffic flows across different routes.

The collapse of the South Tower was the first event to cause visible effects on global routing. Figure 2.1 (prepared by Renesys Corporation, using data collected by the RIPE [Réseaux IP Européens] Network Coor-

Suggested Citation:"2 The Network Experience." National Research Council. 2003. The Internet Under Crisis Conditions: Learning from September 11. Washington, DC: The National Academies Press. doi: 10.17226/10569.
×

FIGURE 2.1 Number of reachable network prefixes as reported by several BGP core routers from the United States, Europe, and Japan from September 8 to September 22, 2001. (All times GMT.) Major events are marked on the plot. SOURCE: Renesys Corporation analysis of RIPE RIS archives.

Suggested Citation:"2 The Network Experience." National Research Council. 2003. The Internet Under Crisis Conditions: Learning from September 11. Washington, DC: The National Academies Press. doi: 10.17226/10569.
×

dination Center from several sources) shows changes to the BGP routing table. Events are reflected in reachability measures as well: Matrix NetSystems recorded a brief 8 percent decrease in the ability to reach (ping) a select number of sites on the Internet in the minutes following the collapse of the first tower (see Figure 2.2).3 A loss of this magnitude for an extended period of time would generally be considered a serious problem, but its occurrence for a brief period of minutes is less so—and certainly not unprecedented. Data from a full month show other dips in reachability, but of a smaller magnitude (see Figure 2.3).

Internet routing and reachability measurements returned nearly to normal within 15 minutes of the collapse of the South Tower. However, Matrix NetSystems and Telstra BGP data show that on September 11, about 1 to 2 percent of the approximately 105,000 routes did not return to normal for almost 24 hours. Some of these routes were for businesses located in the World Trade Center complex. Interestingly, others were associated with ISPs in other countries—Italy, Germany, Romania, and South Africa, for example. The collapse of the North Tower appears to have caused some transatlantic circuits to fail, and these ISPs obviously depended on their New York City links for more than just connectivity to the United States (see Box 2.1).

An analysis of the BGP message activity measured during and after September 11 shows that some global routing “events” (spikes in the volume of BGP messages) did take place because of outages caused by the attacks. However, the magnitude of these events was quite modest.

Overall, the rate of BGP routing advertisements and withdrawals suggests that the Internet was actually more stable than normal on September 11. One possible reason for this overall stability is that network operators understandably tend to avoid optional maintenance and hardware or software changes during emergencies. Anecdotal information from network operators also suggests that many operators were watching the news instead of making normal changes to their routers. The most significant traffic and routing events occurred several hours after the attacks; they resulted from damage to the Verizon central office at 140 West Street and power failures at the Telehouse 25 Broadway Internet co-location facility. Several of these events are visible in Figure 2.1.

3  

Different sets of hosts used by Matrix NetSystems display “normal” reachability levels of less than 100 percent because the list of hosts in each set is kept constant to enable comparisons over long intervals and some hosts on the list no longer exist.

Suggested Citation:"2 The Network Experience." National Research Council. 2003. The Internet Under Crisis Conditions: Learning from September 11. Washington, DC: The National Academies Press. doi: 10.17226/10569.
×

FIGURE 2.2 Reachability of four representative sets of Internet hosts on September 11, 2001. SOURCE: Matrix NetSystems, Inc.

FIGURE 2.3 Reachability of four representative sets of Internet hosts during 12 days in September, 2001. SOURCE: Matrix NetSystems, Inc.

Suggested Citation:"2 The Network Experience." National Research Council. 2003. The Internet Under Crisis Conditions: Learning from September 11. Washington, DC: The National Academies Press. doi: 10.17226/10569.
×

BOX 2.1 How Damaged Cables in New York City Could Affect the Internet in Other Countries

It may seem surprising that events in New York City could have disrupted Internet connectivity far from U.S. shores, but one explanation is the structure of the international telecommunications market. The pricing and availability of international phone circuits are complex and do not necessarily reflect such simple measures as distance. They do reflect such factors as treaties, other historical ties between countries, and geography. For example, it is often easier to run a cable under water than across land. Also, in many cases, it is much less expensive for an Internet service provider (ISP) in country A to connect with an ISP in neighboring country B by leasing a line to the United States (or, in some cases, to the United Kingdom) than simply by leasing a line that runs directly from A to B. As a result, many regions choose to interconnect their various ISPs in the United States. New York City (and London) are key interconnection points for Africa and parts of Europe. Miami, Florida, is a major interconnection point for Central and South America. It is this counterintuitive interconnection pattern that explains why the collapse of the World Trade Center affected networks in Italy, Germany, Romania, and South Africa.

Traffic Load Across the Internet

Active ping-style probes are used by a number of entities to monitor the Internet. Data from these sources showed only a small loss in overall connectivity during September 11 and a corresponding slight increase in packet delay times and loss. One example, collected by the Cooperative Association for Internet Data Analysis (CAIDA), is shown in Figure 2.4. Measurements such as these reflect a sort of global average, highlighting the fact that from a global traffic perspective, the events of September 11 were actually quite localized in scope.

These observations are supported by passive measurements of packet traffic. Reports of several ISPs that participated in the committee’s workshop indicated that the total level of Internet traffic in fact dropped slightly on September 11 compared with that on the previous Tuesday. The normal Internet pattern, by contrast, is for traffic volume to increase slightly each week.

One ISP provided workshop participants with detailed information about traffic on its backbone that confirmed the general reports received from other ISPs. The ISP providing the detailed information did not experience any unusual peak traffic loads, delay, or loss within its backbone. Nor did it report any unusual routing instability there. Impacts were confined to the edges of its network, such as customer-access lines. This view is supported by data from Yahoo, which averaged roughly 1

Suggested Citation:"2 The Network Experience." National Research Council. 2003. The Internet Under Crisis Conditions: Learning from September 11. Washington, DC: The National Academies Press. doi: 10.17226/10569.
×

FIGURE 2.4 Reachability seen by various CAIDA monitors in September 2001. SOURCE: CAIDA.

billion page views per day at that time. Traffic to news sites jumped threefold on September 11, and queries related to a search for news jumped 50-fold; yet the overall traffic was just slightly lower than normal.

Not all ISPs reported lower traffic levels, however. Some that specialize in content delivery (i.e., ISPs that combine regional or national networks and high-performance Web servers to provide high-performance Web hosting) saw a large increase in traffic. One nationwide content-distribution network (Akamai) saw traffic jump 350 percent above normal, likely reflecting increased interest on September 11 in particular content, such as news images, and additional use of its service by some major content providers.

Another measure of Internet use is the rate at which dial-up users log in to their ISPs. Consistent with reports that overall traffic declined, data from America Online (AOL) show log-ins on September 11 falling below the rate on September 10 during the period immediately following the plane crashes and during the evening hours (see Figure 2.5). Two plausible explanations are that Internet news sites were experiencing high congestion levels and that users were watching television to obtain news and information.

Suggested Citation:"2 The Network Experience." National Research Council. 2003. The Internet Under Crisis Conditions: Learning from September 11. Washington, DC: The National Academies Press. doi: 10.17226/10569.
×

FIGURE 2.5 AOL log-ins per minute on September 10 and September 11, 2001. SOURCE: Geraldine MacDonald, America Online.

Domain Name System

Not all Internet applications and services were affected significantly by the crisis. An example is the Domain Name System (DNS)—an important Internet service that looks up a name (e.g., <www.example.com>) in its databases and returns the Internet Protocol address (e.g., 190.0.34.72) associated with that name.4 This process is known as name resolution. Measurements of DNS activity during September 11 show that the load on the root servers was normal to light—most likely because caching of domain names on end-hosts is typically very effective in reducing load on the system, especially when most users are accessing commonly requested sites during a crisis event. There was, however, a DNS problem in South Africa stemming from the loss of connectivity in New York, as described in the following section.

SPECIFIC NONLOCAL EFFECTS

As noted earlier, the damage around the World Trade Center complex had impacts worldwide. In addition to interruptions in local ISPs’

4  

W. Stevens. 1994. TCP/IP Illustrated, Vol. 1: The Protocols. Addison-Wesley, Boston.

Suggested Citation:"2 The Network Experience." National Research Council. 2003. The Internet Under Crisis Conditions: Learning from September 11. Washington, DC: The National Academies Press. doi: 10.17226/10569.
×

connectivity to the rest of the Internet, there were more complex—and surprising—effects on the connectivity of providers, some of them located well outside New York City. These effects, resulting from subtle interdependencies between different systems and protocols, included dial-up access problems for ISPs with POPs located in New York City; loss of connectivity for networks in such disparate places as Romania and at the European Organization for Nuclear Research (CERN) in Geneva, Switzerland; and DNS problems in South Africa. Several of these problems are detailed below.

Difficulties Accessing POPs

ISPs that offer dial-up connectivity must provide several facilities, including modem banks and POPs. Customers prefer to make a local (nontoll) call to connect to their ISPs, so the providers seek to have modem banks in most local calling areas. Once connected through a modem bank, customers must authenticate themselves with the ISP before being connected to the Internet or allowed to use ISP-provided services such as e-mail or customer-created Web pages. Building a network that provides these capabilities in each local calling area is expensive and hard to maintain, so it is common practice for ISPs to simply connect a set of modem banks to a single POP (a practice known as backhauling, because the ISP “hauls the data back” to a common point). In this way, the equipment required at each modem bank is kept to a minimum. Because such POPs often cover several states or even a larger region, and because several providers’ POPs for the northeastern United States happened to be located in New York City, some customers in other states found that even though they could establish a dial-up connection, they could not connect to the Internet. Their local modem banks, unbeknownst to them, were connected to a New York City POP that was out of service.

Disruption of the DNS in South Africa

Internet disruptions in New York City led to at least one protocol-related delayed reaction—disruption of DNS service—far away. Some users in South Africa reported difficulty resolving domain names ending in .za, the top-level domain for South Africa, in the days following September 11. As a result, they could not access Internet services (such as Web servers) within the country, despite the fact that there were no physical network disruptions in South Africa itself at that time.

The answer to how this happened even though networks and DNS servers in South Africa continued to operate lies in the design of the DNS. To reduce the load on name servers, the DNS supports automatic caching

Suggested Citation:"2 The Network Experience." National Research Council. 2003. The Internet Under Crisis Conditions: Learning from September 11. Washington, DC: The National Academies Press. doi: 10.17226/10569.
×

of frequently used names, allowing common requests to be handled with information in caches operated by a user’s ISP. Only requests that are not cached are passed to higher-level servers, and if no cached information is found, ultimately the request is passed to the DNS’s root servers. However, this caching does not completely isolate these users from loss of connectivity with the root servers. To ensure that updated information propagates throughout the Internet, DNS entries have an expiration date associated with them. Once an entry expires, a cache flushes the stored information and requests a fresh copy. If the root server cannot be reached, this flushed information cannot be restored.

Interdependency in Hospital Wireless Networks

Another surprising problem resulted from the loss of Internet connectivity in several hospitals. Today, many hospitals rely on handheld computers and wireless connectivity to provide doctors with bedside access to hospital databases (for receiving updated laboratory reports, for example). It turns out that by contracting with an outside carrier to provide this wireless connectivity, the hospitals introduced a dependence on Internet links. Thus if the Internet link between the hospital and the wireless carrier fails, the wireless devices will lose access even to internal databases.

Surmounting interdependencies such as these is important, because they can lead to surprising failure modes, and it is difficult—because interdependencies can arise from effects ordinarily hidden by the layered structure of the Internet architecture, or by the tendency of commercial Internet services to keep private (for competitive reasons) the specifics about their underlying interconnection structures. These interdependencies can become even more difficult to comprehend in the face of technologies that complicate the basic structure of Internet connectivity, such as virtual private networks, private address realms interconnected using network address translators, overlay networks, firewalls, or transparent proxies. In some cases, such technologies make interconnection easier by hiding internal details of the networks. At the same time, their presence can complicate interconnection by imposing additional hurdles that must be negotiated beyond the basic Internet Protocol connectivity.

RESTORATION EFFORTS

Despite the physical destruction in New York City on September 11, Internet connectivity was quickly restored for many of the affected institutions. Probably the most enduring image of data-communications restoration was the tremendous effort to put the New York Stock Exchange

Suggested Citation:"2 The Network Experience." National Research Council. 2003. The Internet Under Crisis Conditions: Learning from September 11. Washington, DC: The National Academies Press. doi: 10.17226/10569.
×

“back in business.” But a number of other efforts to restore Internet connectivity were mounted rapidly. These activities, which involved the use of alternate paths to the Internet as well as the rapid deployment of new infrastructure, were usually less visible and more improvised, though generally effective.

ISP Cooperation

As the events of September 11 unfolded, many Internet service providers took steps to ensure that their networks would continue to run smoothly. Their actions ranged from increasing staff at network operations centers to coordinating with other ISPs to assure connectivity.

Ordinarily competitive service providers cooperated to restore connectivity lost after Verizon’s 140 West Street facility was damaged. For example, NYSERNet (a nonprofit networking consortium) and Applied Theory (a for-profit spin-off of NYSERNet) reconfigured routers, shared their circuits, and made use of other circuits as far away as Buffalo, New York, to restore service to medical institutions in New York City; this roundabout approach was necessary because the two organizations’ personnel could not enter the cordoned-off area south of 14th Street to access equipment.

One result was that NYC.gov, the official New York City Web site, was back in service on September 12 at 8:22 P.M. After a few days, access restrictions in Lower Manhattan were relaxed, and NYSERNet and Applied Theory staff could enter facilities in Lower Manhattan to reconfigure capacity on other SONET rings and restore service to schools, hospitals, and city governments on Long Island. Similar cooperation was reported in London and Amsterdam as ISPs made use of interconnection facilities in those cities to reconnect networks that normally would link in New York City.

ISPs encountered some glitches as they sought to communicate with each other to coordinate their activities. The ISPs typically have the phone numbers of one another’s network operations centers so that their staffs can cooperate during major outages. In most cases, these are toll-free numbers. But during the middle of the day on September 11, toll-free dialing on the WorldCom telephone network was disrupted (though the toll-free service of other major providers remained operational) as a result of link failures and an increased volume of phone calls.5 Some ISPs then

5  

WorldCom estimated that 187,465 toll-free queries failed during an approximately 12-hour period, starting at 9:00 A.M. on September 11. According to WorldCom’s outage report to the Federal Communications Commission: “The root cause of the problem has been isolated to message congestion between the Signal Control Points (SCPs) in West Orange,

Suggested Citation:"2 The Network Experience." National Research Council. 2003. The Internet Under Crisis Conditions: Learning from September 11. Washington, DC: The National Academies Press. doi: 10.17226/10569.
×

found themselves scrambling to exchange non-toll-free numbers that they normally would not have expected to need.

Some ISPs also discovered problems with their contingency plans. Not all ISPs require that their equipment automatically restart in case of a power failure; their expectation had been that in the very rare event of both primary and backup power failure, they would be able to manually restart their systems once power was restored. But the power outage at the Telehouse facility and its location in a limited-access area made those expectations untenable. The lesson is, in fact, not new. Telephone companies and some of the more savvy ISPs already knew it. Nonetheless, the incident points to the need for all telecommunications providers to consider such contingencies and to be equipped to deal with them.

Another operational challenge faced by ISPs working to restore or maintain their networks was that of basic support for personnel. Given local business closures in the wake of the attacks and the extended shifts that staff were required to work, some ISPs found it difficult to obtain food service for them.

Improvising to Restore Connectivity

In a number of cases, improvised links allowed connectivity to be restored. Some of these efforts relied on the Internet’s architecture, which is compatible with almost any sort of communications link and accommodates almost any sort of service. For example:

  • Wireless data links using unlicensed spectrum were used to reestablish customer connectivity from sites in Lower Manhattan to sites slightly farther north, where Internet links were undamaged.

  • Time Warner Cable (TWC) deployed cable modem service to provide connectivity in state and local government offices. For example, when WNYW (Fox) and Police Plaza lost their digital subscriber line (DSL) connections, TWC replaced the Internet connectivity with Road Runner cable modem service. TWC also supplied New York City’s morgue with

   

New Jersey, Dominguez Hills, California, and Irving, Texas and the switch network result-ing from a mass calling event following the terrorist attacks in New York and Washington, D.C. Multiple link failures contributed to the congestion. As a result of the congestion, SCP Servers were intermittently unable to respond to queries, causing toll free service calls from the switch network to time out and fail.” (WorldCom. 2001. Final Service Disruption Report, No. 01-149. WorldCom, Washington, D.C., October 11. Available online at <http://www.fcc.gov/Bureaus/Engineering_Technology/Filings/Network_Outage/2001/reports/01-149.pdf>.)

Suggested Citation:"2 The Network Experience." National Research Council. 2003. The Internet Under Crisis Conditions: Learning from September 11. Washington, DC: The National Academies Press. doi: 10.17226/10569.
×

static IP addresses so that it could create its own networks, and cable and Road Runner service were installed at the Staten Island Red Cross. Voice-over-IP (VoIP) phones running over the cable network were used to support communications among city and state offices.6

  • Internet phones were used in New York City as a way of circumventing problems with the local and long-distance phone networks. Organizations making these facilities available included local universities (chiefly for their students), Time Warner, and Cisco.

  • In Washington, D.C., a temporary IP infrastructure was deployed at the old Naval Research Laboratory facility to provide communications services for the Department of Defense units from the Pentagon.

In addition, many users communicated using unconventional means. Institutions that had lost network connectivity were given the temporary use of other offices with Internet access through the generosity of other companies and universities.

Also, once it was discovered by the late afternoon of September 11 that making long-distance calls was often easier in New York City than placing local calls, ISPs began to advise residents having difficulty dialing into their Internet service providers to make long-distance phone calls in order to reach modems in other locations. Anecdotal accounts suggest that a large number of people did just that.

THE EXPERIENCES OF OTHER COMMUNICATIONS NETWORKS: TELEPHONE, WIRELESS VOICE AND DATA, AND BROADCASTING

The Internet was only one of several communications systems affected by the events of September 11. To place its experience in context, this section provides a brief overview of the other communications networks’ efforts that day.

Telephone

As an indication of how quickly news travels (and a testament to how well the communications infrastructure was working), changes came swiftly. Apparently the first communications impact outside New York City occurred right after United Airlines Flight 175 struck the South Tower at 9:02 A.M., as the load on some telephone switches in northern New

6  

“World Trade Center Tragedy: Time Warner Cable of New York City & NY1 News Efforts.” Time Warner Cable News Release, September 14, 2001.

Suggested Citation:"2 The Network Experience." National Research Council. 2003. The Internet Under Crisis Conditions: Learning from September 11. Washington, DC: The National Academies Press. doi: 10.17226/10569.
×

Jersey impaired parts of the national toll-free calling system (discussed above, in the context of ISP coordination). Because of congestion, the Government Emergency Telecommunication System, which provides authorized users with priority access to telephone circuits, was employed by government officials. By about 9:15 A.M., the situation was quite exceptional. In New York City, call volumes were making it increasingly difficult to call into or out of the city.

Outside the city, the impact was more muted. With the exception of some difficulty with toll-free numbers, the telephone network was working well (although the load was higher than normal). The crash of American Airlines Flight 77 into the Pentagon at 9:37 A.M. did not dramatically change this picture, except to increase call volume into and out of the Washington, D.C., area, thereby adding the nation’s capital to the list of difficult-to-call places. About the same time, as it was learned that the two New York planes had originated in Boston, call loads in Boston also grew.

At noon, telephone traffic remained high. Indeed, the number of telephone calls reported to have been completed by Verizon on September 11 was approximately double that of a typical day. However, it appears that difficulties in calling out of New York City may have eased by then—for example, AOL recommended that users unable to connect to its New York modem banks with a local number try calling a long-distance access number to get online; this advice apparently worked for many users.

At 5:20 P.M., World Trade Center Building 7 collapsed, severely damaging Verizon’s 140 West Street central office. Although some of the equipment in the building continued to run for several more hours, local telephone connectivity through the exchange was ultimately lost. Unfortunately, the damage to the building included the disabling of some of the monitoring equipment, so it is not possible to determine exactly how telephone service degraded over the evening of September 11. Another effect of the damage to 140 West Street was to further damage fiber links already affected by the collapse of the Twin Towers, causing additional connectivity losses.

Cellular Telephones

As word of the attacks spread, the cellular telephone system in the northeastern United States began to be heavily loaded. Nationally, call volume rose 50 percent above normal. One can compare this rate with the 30 percent increase typical on Mother’s Day, the canonical example of an exceptional calling day. Cellular telephone systems are usually engi-

Suggested Citation:"2 The Network Experience." National Research Council. 2003. The Internet Under Crisis Conditions: Learning from September 11. Washington, DC: The National Academies Press. doi: 10.17226/10569.
×

neered to support traffic during busy hours with only about 4 percent of calls being blocked (given a busy signal).

Regionally, the experience was even more dramatic. New York City had a 400 percent increase in call attempts during the day. At about 11:00 A.M., the volume was up 1,300 percent for at least one major carrier. Washington, D.C., had a 125 percent increase for the day. New England as a whole saw a 75 percent increase. The cellular system was not engineered for these loads, so call-blocking rates grew accordingly. In New York, 75 percent of calls were blocked (92 percent at the morning peak). In Washington, D.C., 56 percent of calls were blocked.

Wireless Internet—using such devices as Research in Motion’s Blackberry—also rose on September 11. Traffic surged by 60 percent around 10:00 A.M. and stayed high through the early afternoon.7

Although there have been reports that a large number of cellular phone sites were disabled by the collapse of the Twin Towers, the industry maintains that only five sites were damaged in the attacks. In any case, by late afternoon on September 11, a combination of damage to telephone lines and the loss of power caused 160 cell sites in Lower Manhattan to become inoperable (slightly under 5 percent of the New York City cellular infrastructure). Over the hours and days that followed, the cellular operators adopted a variety of measures, such as the installation of temporary sites and the use of alternate radio frequencies, to restore (or in some cases, such as at the Pennsylvania crash site, to increase) capacity. In Lower Manhattan, full capacity was restored within a week.8

Broadcast Television and Radio

Transmission facilities of 9 of the 14 local-area television stations, along with those of 5 local radio stations, were lost when the North Tower of the World Trade Center was destroyed. Of the stations that lost their transmission facilities, only 2 were able to quickly restore service—WCBSTV (Channel 2), which switched to a full-power backup antenna at the Empire State Building, and WXTV (Channel 41). For households that subscribed to cable, there was much less impact: most television stations deliver their feeds to cable operators directly, by way of fiber or microwave links, and the New York cable-system operators reported no service interruptions outside Lower Manhattan. (However, the impact on broad

7  

From reports by carriers to the Federal Communications Commission.

8  

Data from Kathryn Condello, 2001, “Wireless Industry: Impact and Recovery Efforts Summary Report” (presentation to the Network Reliability and Interoperability Council), Cellular Telecommunications and Internet Association, Washington, D.C., October 30.

Suggested Citation:"2 The Network Experience." National Research Council. 2003. The Internet Under Crisis Conditions: Learning from September 11. Washington, DC: The National Academies Press. doi: 10.17226/10569.
×

cast viewers was greater in New York City than it might have been in other metropolitan areas, as household cable penetration is only about 50 percent, significantly less than the nationwide average of 70 percent.)

To speed the restoration of broadcast service, the Federal Communications Commission gave stations temporary authority to locate replacement transmitters at any reasonable site, provided they would not cause interference with other stations’ activities. Shortly after the World Trade Center buildings collapsed, a number of broadcasters set up transmitters at a tower in Alpine, New Jersey. Since then, six networks have relocated transmitters to the Empire State Building, and two have remained at the Alpine site. Broadcasters do not consider this pair of sites adequate, however, for the long term: the Empire State Building does not have enough physical or electrical capacity for all of the broadcasters, and the Alpine tower, by virtue of its relatively modest height and remote location, does not serve as sizable a market as the World Trade Center site did. Efforts are now under way to select one or more permanent transmitter sites that are more suitable.

Suggested Citation:"2 The Network Experience." National Research Council. 2003. The Internet Under Crisis Conditions: Learning from September 11. Washington, DC: The National Academies Press. doi: 10.17226/10569.
×
Page 21
Suggested Citation:"2 The Network Experience." National Research Council. 2003. The Internet Under Crisis Conditions: Learning from September 11. Washington, DC: The National Academies Press. doi: 10.17226/10569.
×
Page 22
Suggested Citation:"2 The Network Experience." National Research Council. 2003. The Internet Under Crisis Conditions: Learning from September 11. Washington, DC: The National Academies Press. doi: 10.17226/10569.
×
Page 23
Suggested Citation:"2 The Network Experience." National Research Council. 2003. The Internet Under Crisis Conditions: Learning from September 11. Washington, DC: The National Academies Press. doi: 10.17226/10569.
×
Page 24
Suggested Citation:"2 The Network Experience." National Research Council. 2003. The Internet Under Crisis Conditions: Learning from September 11. Washington, DC: The National Academies Press. doi: 10.17226/10569.
×
Page 25
Suggested Citation:"2 The Network Experience." National Research Council. 2003. The Internet Under Crisis Conditions: Learning from September 11. Washington, DC: The National Academies Press. doi: 10.17226/10569.
×
Page 26
Suggested Citation:"2 The Network Experience." National Research Council. 2003. The Internet Under Crisis Conditions: Learning from September 11. Washington, DC: The National Academies Press. doi: 10.17226/10569.
×
Page 27
Suggested Citation:"2 The Network Experience." National Research Council. 2003. The Internet Under Crisis Conditions: Learning from September 11. Washington, DC: The National Academies Press. doi: 10.17226/10569.
×
Page 28
Suggested Citation:"2 The Network Experience." National Research Council. 2003. The Internet Under Crisis Conditions: Learning from September 11. Washington, DC: The National Academies Press. doi: 10.17226/10569.
×
Page 29
Suggested Citation:"2 The Network Experience." National Research Council. 2003. The Internet Under Crisis Conditions: Learning from September 11. Washington, DC: The National Academies Press. doi: 10.17226/10569.
×
Page 30
Suggested Citation:"2 The Network Experience." National Research Council. 2003. The Internet Under Crisis Conditions: Learning from September 11. Washington, DC: The National Academies Press. doi: 10.17226/10569.
×
Page 31
Suggested Citation:"2 The Network Experience." National Research Council. 2003. The Internet Under Crisis Conditions: Learning from September 11. Washington, DC: The National Academies Press. doi: 10.17226/10569.
×
Page 32
Suggested Citation:"2 The Network Experience." National Research Council. 2003. The Internet Under Crisis Conditions: Learning from September 11. Washington, DC: The National Academies Press. doi: 10.17226/10569.
×
Page 33
Suggested Citation:"2 The Network Experience." National Research Council. 2003. The Internet Under Crisis Conditions: Learning from September 11. Washington, DC: The National Academies Press. doi: 10.17226/10569.
×
Page 34
Suggested Citation:"2 The Network Experience." National Research Council. 2003. The Internet Under Crisis Conditions: Learning from September 11. Washington, DC: The National Academies Press. doi: 10.17226/10569.
×
Page 35
Suggested Citation:"2 The Network Experience." National Research Council. 2003. The Internet Under Crisis Conditions: Learning from September 11. Washington, DC: The National Academies Press. doi: 10.17226/10569.
×
Page 36
Suggested Citation:"2 The Network Experience." National Research Council. 2003. The Internet Under Crisis Conditions: Learning from September 11. Washington, DC: The National Academies Press. doi: 10.17226/10569.
×
Page 37
Suggested Citation:"2 The Network Experience." National Research Council. 2003. The Internet Under Crisis Conditions: Learning from September 11. Washington, DC: The National Academies Press. doi: 10.17226/10569.
×
Page 38
Suggested Citation:"2 The Network Experience." National Research Council. 2003. The Internet Under Crisis Conditions: Learning from September 11. Washington, DC: The National Academies Press. doi: 10.17226/10569.
×
Page 39
Next: 3 The User Experience »
The Internet Under Crisis Conditions: Learning from September 11 Get This Book
×
Buy Paperback | $29.00 Buy Ebook | $23.99
MyNAP members save 10% online.
Login or Register to save!
Download Free PDF

This report presents findings of a workshop featuring representatives of Internet Service Providers and others with access to data and insights about how the Internet performed on and immediately after the September 11 attacks. People who design and operate networks were asked to share data and their own preliminary analyses among participants in a closed workshop. They and networking researchers evaluated these inputs to synthesize lessons learned and derive suggestions for improvements in technology, procedures, and, as appropriate, policy.

  1. ×

    Welcome to OpenBook!

    You're looking at OpenBook, NAP.edu's online reading room since 1999. Based on feedback from you, our users, we've made some improvements that make it easier than ever to read thousands of publications on our website.

    Do you want to take a quick tour of the OpenBook's features?

    No Thanks Take a Tour »
  2. ×

    Show this book's table of contents, where you can jump to any chapter by name.

    « Back Next »
  3. ×

    ...or use these buttons to go back to the previous chapter or skip to the next one.

    « Back Next »
  4. ×

    Jump up to the previous page or down to the next one. Also, you can type in a page number and press Enter to go directly to that page in the book.

    « Back Next »
  5. ×

    Switch between the Original Pages, where you can read the report as it appeared in print, and Text Pages for the web version, where you can highlight and search the text.

    « Back Next »
  6. ×

    To search the entire text of this book, type in your search term here and press Enter.

    « Back Next »
  7. ×

    Share a link to this book page on your preferred social network or via email.

    « Back Next »
  8. ×

    View our suggested citation for this chapter.

    « Back Next »
  9. ×

    Ready to take your reading offline? Click here to buy this book in print or download it as a free PDF, if available.

    « Back Next »
Stay Connected!