eWEEK content and product recommendations are editorially independent. We may make money when you click on links to our partners. Learn More.
Verizons phone cables were crushed and submerged in water after 7 World Trade Center crashed. Finally, its rehabilitation work is just about complete.
By the end of this year, more than 1,000 customer-service staffers at Verizon Communications in New York City will face one of the toughest mornings of their lives.
Theyll return to work for the first time to 140 West St., the building that stands across the street from where the twin towers of the World Trade Center fell on Sept. 11, 2001. The building may still be erect, but it has taken two years to make it habitable after sustaining severe damage from the towers collapse—a catastrophic event that took out a vital nerve center in Verizons network in Lower Manhattan.
That 140 West remained structurally intact is testament to its bunker-like construction. Its next-door neighbor, 7 World Trade Center, fell right against it. Steel beams crashed through an underground vault that held all of the wiring at 140 West, virtually destroying the buildings links to Verizons network. Cables were crushed or submerged in floodwaters. Verizon executives and workers had to wear moon-suits to enter the building.
“Words cant describe what was going on,” Paul LaCouture, Verizons Network Services Group president, would say nine months later.
Today, Verizon continues to retool its network in Lower Manhattan. While the physical restoration of switches, underground cables and other network equipment at 140 West was finished more than a year ago, Verizon still has some 600 technicians working on what it calls “de-hubbing”—off-loading some of the network traffic handled at 140 West to other switching offices in Manhattan. That work is expected to be finished sometime next year.
Verizon also is shoring up facilities across its network, which spans about 1.5 million square miles and includes parts or all of 29 states, the District of Columbia and the Commonwealth of Puerto Rico. The overarching goal is to minimize the kind of disruption that occurred in Lower Manhattan, when service to some 4.4 million voice and data lines was cut off for days, weeks and, in some cases, months.
With one exception, Verizons network passed an unscheduled stress test on Aug. 14, when a major power failure knocked out electrical service to New York City and a large chunk of the Eastern seaboard for more than a day. Emergency generators kicked in, and wire-line phone service continued uninterrupted for Verizons local customers. The only glitch: The failure of a Verizon backup diesel generator in Brooklyn forced New York City emergency vehicles to use battery backup devices to handle 911 calls. The blackout reinforced key lessons learned from Sept. 11.
Lesson 1
: Its Good to Have An Army”>
Lesson 1: Its Good to Have An Army
The World Trade Center events tested Verizon to an extent unimaginable on Sept. 10, 2001. Yet, less than a week after the twin towers fell, Verizon supplied the New York Stock Exchange with enough telecom service to reopen for business.
The damage inflicted by the falling towers caused a complete failure at 140 West—200,000 voice lines, 150,000 business lines, and the equivalent of about 4 million data circuits went out of commission.
The worst damage occurred underground. Verizon inspectors found about 300 of the 500 underground cables that fed into 140 West were damaged or rendered inoperable because of water damage from broken mains at Ground Zero.
The steel beam that crashed through the underground vault cut off a key connection point between 140 West and Verizons local network. The air compressor system in 140 West that kept those conduits clear was badly damaged. Overtaxed power generators eventually gave out.
Verizon deployed 1,600 technicians to rig up a replacement network. The upper floors of 140 West had to be cleaned up enough to get some of the network equipment running again. Emergency power had to be reestablished. New cables were run out the buildings windows and down to the street. Those cables then had to be spliced, wire by wire, to undamaged conduits a few blocks away from the destruction.
But, Verizon also had to keep the aboveground conduits clear. “We built an entirely new air compressor system from scratch,” says Joe DeMauro, a Verizon regional president responsible for the outside wiring plant in Lower Manhattan.
All told, about 3,000 field technicians and managers were involved in the restoration work in Lower Manhattan. Few companies could manage to muster that kind of muscle.
In fact, if there were another failure of this magnitude today, Verizon might not have the same muscle. With revenue from domestic telecom operations in decline, Verizon has trimmed its workforce by nearly 15,000, to about 225,000, in the past year. But unions are offering stiff resistance to further cutbacks. In July, an arbitrator ruled that Verizon violated its union contract last December when it laid off some 2,300 technicians and operators in New York.
Lesson 2
: Redundancy Matters”>
Lesson 2: Redundancy Matters
Muscle power was aided immeasurably by one feature Verizon built into its network before Sept. 11.
Verizon had installed redundant connections between its central offices so a cable failure on one route wouldnt disrupt communications. Although the redundancy did not allow Verizon to immediately restore service, technicians ran aboveground cables to an undamaged portion of a redundant line that connected 140 West to a central office on Canal Street, several blocks north of Ground Zero.
Without the redundancy, Verizon would have taken weeks to dig up streets and connect 140 West to the Canal Street facility—assuming equipment was available to dig the trenches.
After Sept. 11, Verizon put 18 optical communications rings into place in Lower Manhattan. With these rings, which use Synchronous Optical Network (SONET) technology, a failure between two points can be overcome simply by reversing the direction of traffic.
Even with the SONET rings, Verizons data network in Lower Manhattan is not fail-safe. A knockout of 140 West will still cause the network to fail. But now Verizons network will operate if damage or disaster disables part of a ring.
Lesson 3
: More Eggs, More Baskets”>
Lesson 3: More Eggs, More Baskets
Prior to the attacks, 140 West supplied phone service to about 14,000 business and 20,000 residential customers in Lower Manhattan. In redesigning its Lower Manhattan network, Verizon moved as many of those residential lines as possible out of 140 West, says William Hummel, Verizons director of business recovery and continuity services. For example, lines from Battery Park City, a mostly residential development, were rerouted to one of the smaller switching facilities.
Verizon also lowered the call volume at 140 West by changing the way it routes some voice traffic. Before the attacks, calls from Lower Manhattan to New Jersey were sent through 140 West. Now, more of those calls travel through a less busy central office in midtown Manhattan.
The company says it is extending the “de-hubbing” project to some other high-volume central offices. Still, there are limits to how much traffic Verizon can offload. For the most part, lines to financial firms in the Wall Street area still have to terminate at 140 West. Even with the de-hubbing, 140 West will handle more than 90% of the data traffic volume that passed through the facility before the 9/11 attacks.
Verizon is counting on redundant cables and the optical rings to keep data moving if a network failure occurs.
Lesson 4
: Dont Wait for the Unimaginable”>
Lesson 4: Dont Wait for the Unimaginable
Before 9/11, Verizons security operations group had launched a “site-hardening” project to improve physical security at its central offices, which now total close to 1,800 buildings. The terrorist attacks dramatically accelerated that project, according to Roger Kochman, Verizon director of security operations.
Under the project, the group looks at everything from access by individuals to the facility, to the ability of the building to withstand a physical assault from the outside. Remedies range from improving surveillance with closed-circuit cameras and better background checks for employees to retrofitting buildings with bomb-resistant outer casings.
Immediately after the 9/11 attacks, Verizon installed temporary concrete barriers to block access to some of its larger buildings.
Verizon has identified its 100 most active network centers and is shoring up physical security at those installations first. It plans to have its top 200 central offices hardened by 2007, at an average cost of about $1 million per building.
Lesson 5
: Life Goes On”>
Lesson 5: Life Goes On
As devastating as the attacks were, Verizon couldnt just put its strategic operations on hold. The company is under pressure to improve performance in its basic wired telephone business.
While fires still burned, Verizon took the wraps off a project it hopes will cut operating costs in a key operation: its call centers. Instead of talking to live operators, customers are being directed to the Web to handle routine service questions, such as whether payments have been received.
The charge for creating such “e-services” fell primarily to Verizons technology department in Dallas, run at the time by Shaygan Kheradpir, now Verizons chief information officer. In summer 2000, the team, led by Fari Ebrahimi, started mapping out a platform that would consolidate close to 100 different billing, provisioning and other back-office systems.
The team built the platform on Internet communication protocols. For every incompatible existing computer system, Verizons team created an Internet overlay.
The main goal of the “Verizon.com Project” was what Ebrahimi calls a “no-touch” transaction. The customer gains secure access to the Verizon Web site to request new or repair service. Data from that transaction gets placed in an Internet Protocol (IP) “envelope” that then gets routed to every relevant application, such as ordering or billing—without human intervention.
Ebrahimi says the no-touch goal isnt always met. But Verizon has plenty of reason to be optimistic. Verizon claims about 10% of its customers now use the Web site to pay bills, and Ebrahimi expects that mark to reach 30% or more by 2005.
Electronic bill payment is a huge money-saver for Verizon. Paper bills can cost anywhere from $2 to $3 to distribute and process.
Savings add up when customer inquiries are handled by the Web, too. A phone call to a human service rep costs Verizon from $5 to $10, while the cost of processing an online transaction is under $1, according to Ebrahimi.
Most importantly, perhaps, the more customer services that can be handled over the Web, the more dispersed Verizons computer systems and network facilities can be.
That kind of distribution should further improve Verizons ability to recover if catastrophe strikes again.
Verizon Communications Base Case
Verizon Communications Base Case
Headquarters: 1095 Avenue of the Americas, New York, NY 10036
Phone: (212) 395-2121
Business: Sells wired and wireless voice and data services to business and residential customers.
Chief Information Officer: Shaygan Kheradpir
Chief Technology Officer: Mark Wegleitner
Financials: $33.3 billion in revenue for the first six months of 2003; $2.7 billion in net income; 8.2% net income margin.
Challenges: Rebuild telecom network in Lower Manhattan to recover from Sept. 11, 2001, terrorist attacks; improve physical security at more than 1,700 central switching offices scattered across 29 states; cut operating costs by moving customer service from call centers to the Web.
Baseline Goals:
- Complete rebuilding of the Lower Manhattan network in 2004.
- Complete “site hardening” of 200 biggest revenue-generating central offices by 2007.
- Get 30% or more customers to pay bills via the e-services Web platform by 2005, up from 10% today.