The VOIP Story of 9/11, Updated

By Ellen Muraskin  |  Posted 2004-09-12

The VOIP Story of 9/11, Updated

The VOIP deployment of the Borough of Manhattan Community College was planned to start as a 50-phone trial, thrown as an accessory project into a large network conversion. When Sept. 11 took out the neighborhoods Verizon central office, that pilot turned into the colleges emergency phone system. Today, the trial IP PBX and phones have grown in size and scope to run a mix of over 900 IP, digital and analog phones. Call center functionality now is being added that will let supervisors draft more staff to answer calls as needed.

On Sept. 11, 2001, the City of New York, or CUNY, college closest to Ground Zero was the Borough of Manhattan Community College. Its main campus on Chambers Street was four blocks north of where the Twin Towers stood. The Verizon central office switching its Centrex lines was the old New York Telephone building on West Street, the one famously gored by the falling girders of Tower 7, which crumbled that afternoon. Tower 7 also took out all 15 floors and 70 classrooms of BMCCs newly renovated Fiterman building.

In those first apocalyptic days after Sept. 11, a good part of the recovery story had to do with telecommunications. Verizons emergency measures included hooking 100 phone lines from BMCCs demarcation point to a truck-mounted, microwave-connected CO (central office). But these were not to be used for the college: They were needed by Port Authority police and emergency medical workers.

Joe Giummo, the colleges director of IT, had inherited what he calls the "telecom stepchild" only a few months before. His first priorities after the towers fell, after assuring his own safety and that of his family and team, was restoring the colleges Internet presence, as a means of keeping faculty and students updated, and then reallocating surviving facilities.

Sometime during that day, he discovered that he had a voice backup in a surviving data line, used for a video feed from BMCCs media center in the main buildings fifth floor. This was a WorldCom Inc. line, running from the great telecom peering site a little further uptown, at 60 Hudson. "WorldCom showed up, looked at the feed, confirmed it was theirs, and looked at a couple of manholes in the street. We were immediately surrounded by people with guns," Giummo recalls.

The other piece of luck was the fact that six months earlier, the college had begun converting its network from a 25M-bps ATM (asynchronous transfer mode) core to 100M-bps Ethernet. This cutover involved the installation, by Alcatels eBusiness Networking Division, of OmniCore 5052 switches and Omni S/R-9 switch routers. As part of that cutover, BMCC started a small-scale Alcatel VOIP (voice over IP) trial: an M1/OmniPCX PBX, with 50 Alcatel Reflex IP phones, in its computer center.

The colleges Centrex contract had been due to expire, and "we figured a small pilot project would get us started," says Giummo. "We had estimated finishing the pilot sometime by Thanksgiving or Christmas 2001."

Next Page: The pilot project gets drafted into service.

The VOIP Story of 9/11, Updated - Page 2

Sept. 11 pushed that date way up. The fifth-floor fiber feed was quickly strung down to the first floor, the computer-room trial PBX hooked up to the OmniCore network that had just been installed, and according to VP of Administration and Planning Scott Anderson, appearing in a testimonial video running on Alcatels site, the VOIP switch "immediately handshaked" with it.

The two surviving T-1s worth of bandwidth served 60 contingency voice lines. During the week, this M1 Omni PCX extended phone service into makeshift classrooms in cafeterias and reception halls. Another M1 was installed in space loaned to BMCC by CUNY uptown, and a T-1 was leased to connect the two. All phones hooked up through the network Ethernet jacks, with PCs connecting to the phones.

Over the following days and weeks, Giummos system integrator hustled to get more phones and a larger IP PBX—the M2—from Alcatels Calabasas, Calif., location. Additional phones and an M2 were rushed from France, arriving at the Red Zone at 10 p.m. in trucks that the National Guard had to inspect. The two surviving WorldCom T-1s were bulked up to a DS3; eight PRI (24-line, primary-rate interface) T-1s serving voice, two T-1s to serve voice and data to the uptown cluster, and 32 POTS (Plain Old Telephone Service) lines from Verizon for backup.

Today, IP phones on BMCCs system number around 758, according to Giummo. The M2 OmniPCX, which was outgrown, was upgraded to an M3 in early 2004, although the line cards were moved from the old cabinets to the new. Like its predecessors, it accepts a mix of Alcatel digital and IP phones, as well as "Brand X" analog phones.

"We do a mix because if we have a failure, we always have some type of phone service available," says Giummo. Conferencing ability has gone from three or four parties to 29, which should eliminate fees to third-party bridging services. The whole system, as always, gateways out to the PSTN on premises: Except for the intercampus WAN, the IP telephony is limited to the campus itself. To the world at large, BMCCs system looks like any other business phone system and is therefore no more vulnerable to denial-of-service attacks or virus attacks from the outside.

A new campus building on Murray Street replaces the downed Fiterman Hall, and because it has unobstructed line-of-sight to the main building three blocks away, BMCC has installed a 1-Gigabit optical laser connection to serve as a backup to the T-1s that link the two. Now in test, its not yet working according to plan. "We can accommodate 46 calls to that building on the two T-1s; the 47th and 48th calls overflow to the laser link. But I want that link to take over completely if the T-1s fail," says Giummo. So far, it doesnt. "Im comfortable that Alcatel and Verizon [the present system integrator] will work it out. We may have to reverse the failover, though, from optical to LAN."

The ROI forecasts that put the Alcatel M2 payback at two and a half years or less have worked out, Giummo reports: The phone service bill has gone down from $25,000 monthly, at all-Centrex, to $15,000 with the premises switch. Users favorite features, Giummo reports, continue to be the dial-by-name ability, using the phones pull-out keypad, and the ease of forwarding voice mails.

Like so many industries and businesses, BMCC has shelved 2001s plan to automate registration through the phone. Self-service in registration and routine queries has been put on the Web, where the visual GUI is so much easier to grasp than recorded and concatenated voice instructions. Plans to install a call center system have gone ahead, though; 30 seats of Alcatels Genesys Call Center Express are in the scope-of-work stage.

"Our call volume is so tremendous at certain periods," Giummo explains, "that we were losing a lot of calls." They just didnt know about it before their own switchs management system could report the fact. "We never knew [calls were being abandoned while holding], because it was happening down the block at Verizons switch."

Genesys server is scheduled to be running by next spring semester, or fall 2005 at the latest. The system will support the "virtual call center," in which those best equipped to answer specific questions on registration, or financial aid, or transcripts, can be pressed into service to answer calls from wherever they work, and calls can be routed to appropriate groups. This is an especially cost-effective way to keep up a level of student service during crunch periods.

Next Page: Automated phone notifications if your financial aid hasnt arrived.

The VOIP Story of 9/11, Updated - Page 3

The Genesys product will also support automated out-dial and delivery of phone announcements to groups of students who may be missing transcripts or tuition payments, for example. This, too, should save money on mailings and head off some inbound call traffic.

Today, BMCCs executives are not hard sells when it comes to disaster planning. Between 125 and 145 Verizon Centrex accounts, by Giummos reckoning, are still kept strictly for backup at a flat rate of $12.95 per line. "Just in case a backhoe goes through my T-1s. Redundancy is very important to us. My management understands the critical nature of backup support, and that doesnt come free."

VOIP/Telecom Topic Center Editor Ellen Muraskin can be reached at Muraskin has been observing and illuminating the murky intersection of computer intelligence and telephony since 1993. She reaches for her VOIP line when the rain makes her POTS line buzz.

Check out eWEEK.coms VOIP & Telephony Center at for the latest news, views and analysis on voice over IP and telephony.

Rocket Fuel