According to published carrier reports filed with the Federal Communications Commission, a variety of things, from natural disasters to careless maintenance, can result in disruptive and costly network outages. But sometimes, as in the case of a Verizon Communications Inc. central office last spring, the cause can be deliberate.
As first reported in eWEEK in May, the co-location floor of a Verizon central office in New York was burglarized, leaving a handful of carriers—all Verizon competitors—without network service to the large Midtown Manhattan market for, in some cases, an entire business day.
Since the theft and after numerous interviews with people at the scene the night of May 2, eWEEK has learned more details of the heist, the cost of which New York City Police Department officials pegged at $433,000.
Of primary concern for all those contacted for this story was the obvious lack of even the most basic security measures in the building, and the colo office in particular. Of the four separate doors to the 8th floor colo office, which housed networking racks for Sprint Corp., Qwest Communications International Inc., XO Communications Inc. and Looking Glass Networks Inc., the main door lacked a working lock. Furthermore, it was routinely left open.
Second, all the network racks in the colo office that were burglarized were secured by simple Allen wrench bolts.
An IT administrator for one of the colo companies, who was called in on the night of May 2 to troubleshoot the alarm, said what he found—rows of empty racks with doors flung open—was surreal. “My jaw literally dropped. I said no way,” said the administrator, who requested anonymity.
In addition to the absence of a security system, the only guard on duty at the building that Sunday night was a Verizon security official primarily responsible for monitoring the fire alarm system. No night watchman walks the floors.
Check out eWEEK.coms Security Center at http://security.eweek.com for the latest security news, reviews and analysis.
Be sure to add our eWEEK.com developer and Web services news feed to your RSS newsreader or My Yahoo page
Next Page: No working cameras the night of the theft.
No Working Cameras
Adding insult to injury, the buildings main entrance security cameras on the night of May 2 were ripped out, awaiting upgrade.
While Verizon officials in New York could not discuss the ongoing investigation, spokesperson Daniel Diaz Zapata insisted, “Security is an integral part of what we do in order to secure the integrity of our network.”
Zapata added that the 38th Street central office has security personnel on duty throughout the day and a corporate security officer with a dual role of fire safety through the night, seven days a week.
Insiders said that since the theft, Verizon added new outside cameras and installed an access keypad on the main colo door. But in recent weeks, even the new keypad has been deemed ineffective, as the door has been left open at times.
All told, some 51 pieces of networking gear, including DS3 cards, transporters and an array of other types of circuit boards, were stolen from the four carriers—enough equipment to fill two duffel bags, insiders said.
Click here
to read how the FCC recently stopped providing network outage reports citing potential threats from terrorists.
One of the many affected network customers in May was Ziff Davis Media Inc., publisher of eWEEK.
As of last week, the case was still open with the NYPDs 17th Precinct as a grand larceny.
Shoddy security at Verizon central offices is nothing new. The company itself offered a detailed list of complaints from colo carriers, ranging from unauthorized entry to theft, in an Aug. 2, 2002, report to the Massachusetts Department of Telecommunications and Energy when arguing to evict Competitive Local Exchange Carriers.
Market researcher Infonetics Research Inc. released a report in February estimating that overall network downtime costs companies an average of 3.6 percent of their annual revenues. It also stated that 58 percent of downtime is the result of outages.
Additional reporting by Caron Carlson