When businesses across the country tried to access the Internet Oct. 2, many found connections sluggish or blocked due to outages on WorldCom Inc.s UUNet backbone. Experts trace the overall cause to faulty network maintenance, particularly at peering points; some say such troubles are likely to recur but theres little anybody other than the carriers can do about it.
Critics charge that the federal government, which is active and vocal about monitoring for terrorism and other malicious attacks online, does little to safeguard against more mundane weaknesses such as those that felled UUNet this month.
Tom Ohlsson, vice president of marketing and business development at Matrix NetSystems Inc., of Austin, Texas, attributed the Oct. 2 outages ultimately to WorldComs reduction in staff responsible for operating and maintaining the network. “Most carriers Internet performance has degraded about 40 percent since the beginning of the year,” Ohlsson said.
Matrix monitors the Internet with approximately 150 workstations located throughout the world, which continuously scan between 10,000 and 30,000 unique Internet destinations, such as service providers core routers and edge routers. The primary purpose of the network is to provide a real-time model of the Internet to enterprises to ensure that they use the most efficient routes.
Once WorldCom had the disruptions under control late in the day Oct. 2, it announced the likely cause was data loaded incorrectly into routing tables (which are updated daily), but experts say more was at play.
“What happened is that some very tired WorldCom engineers were loading not just routing tables [but also an operating system upgrade],” Ohlsson said. “It was bad software. We detected an almost- instantaneous packet-loss increase.”
A healthy network experiences 0.01 to 0.02 percent packet loss, according to Ohlsson. “WorldCom went from the low zeroes to 20 percent packet loss in approximately 15 minutes. Youre basically blocking all traffic at that level,” he said.
And WorldComs problems have a ripple effect on other providers that rely on its backbone services. While the carrier claimed just 20 percent of its customer base was disrupted, that figure does not include backbone traffic, peering traffic, or the secondary or tertiary customer base.
The American Disabilities Association, in Birmingham, Ala., was intermittently blocked from the Internet through its BellSouth Corp. digital subscriber line service because BellSouth relies on UUNet as its global service provider, according to Bill Freeman, association president. “When I asked for the estimated time for resolution, the answer I repeatedly received [from BellSouth] was something like We dont know because we dont have control over UUNet,” Freeman said. “Is it really responsible ISP policy to have all your Internet eggs in the UUNet basket?”
Since WorldComs financial troubles surfaced last spring, competing carriers have made a rallying cry for enterprises to invest in redundant services, but it turns out that they do not always have sufficient backup plans themselves.
Eric Small, president of BellSouth.net, in Atlanta, said the companys first point of entry to major Internet backbones is through UUNet, and to establish a redundant first point of entry would require adding an extra backbone charge on the customers bill.
To avoid major connectivity problems in the future, the company is building a BellSouth Regional Internet Backbone, consolidating its existing IP networks onto a common platform. It will then have multiple first points of entry to major backbones, Small said.
According to Ohlsson and other industry experts, the Internet service interruptions of Oct. 2 are likely to recur if the industry doesnt begin to invest more resources in maintaining the network.
“This is the type of thing that [a community of cyber-incident response centers] would note and distribute for analysis,” said Mary Alice Johnson, spokeswoman at the General Services Administration, in Washington. “Thats not the type of group that likes a lot of publicity.”
The Federal Communications Commission strictly monitors the quality and reliability of voice networks, but it does not have rules in place to do the same for the Internet. FCC officials are considering whether to begin establishing such rules, said Jeff Goldthorp, head of the FCCs network technology division, in Washington. “Currently, there are no rules that would require an ISP like UUNet to submit outage data,” Goldthorp said. “The staff has been looking at issues like these and [asking], Are we doing enough?”
In July, lawmakers called FCC Chairman Michael Powell to Capitol Hill to explain what the government can do to prevent major outages. Powell told Congress that carriers are prohibited from discontinuing service without proper notice. However, he added, it is not clear whether this requirement applies to data service providers, and he asked Congress to clarify and bolster his authority in that regard.
Related Stories:
- WorldCom Internet Services Restored
- Special Report: Bushs Cyber-Security Plan