eWEEK content and product recommendations are editorially independent. We may make money when you click on links to our partners. Learn More.
The future of every business is, or will be, tied to the Internet in one way or another. Its the best medium for nearly instant and universal communication that the world has ever known, and its an essential means of connecting businesses to their suppliers, their customers and to the world at large.
But underneath this sweeping digital panorama is a dirty, little secret—the infrastructure that has become the backbone of e-commerce and a focal point for most businesses is unpredictable, unstable and, in some cases, unsustainable.
A two-month investigation by Sm@rt Partner shows the Internet is at risk from unexpected outages caused by everything from seismic disasters to system overloads, terrorism and just plain human error. Security is spotty, standards still dont exist in many arenas, and, because of the distributed ownership of the Internet itself, getting those standards in place is anything but easy.
“The Internet should be just like the telephone network,” says Steve MacKay, chief system architect at Sun Microsystems. “But if you look at the architecture of the data center and the computing environment, [its not developed to that point]. One of our big service provider customers has 30 to 40 percent of their gear down at any one time—theyre either upgrading or doing maintenance, or somethings crashed—and thats invisible to their e-mail and Web service. But theyre the exception. Try having five or 10 friends send you e-mails at the same time—you wont get them at the same time, because some will inevitably pass through servers that are down and the servers will just hold them.”
Ex-Cisco CTO Judy Estrin relayed a similar message to a spellbound audience in San Jose, Calif., last October. She proclaimed the Internet to be at a “crisis point” and called for architectural leadership. “Internet time is not an excuse not to think,” she said. “Its time to make the Internet better, not just faster.”
Shake On It Silicon Valley—sitting squarely between two major earthquake faults—is bordered by at least three major Internet exchanges—WorldComs MAE West in downtown San Jose; the Palo Alto Internet Exchange; and Equinix, which sits to the south near the railroad tracks on land owned by IBM. A fourth exchange, owned by Pacific Bell, is located in San Francisco.
The San Francisco Bay Area also is home to thousands of square feet of server farms. Exodus, which is headquartered in Santa Clara and claims to host 49 of the top 100 Web sites, was declared part of the U.S. “critical infrastructure” under former President Clinton, and was deemed worthy of federal protection in case of cyber or physical attack.
Arguments run back and forth about what damage a major earthquake really could inflict, but the concentration of exchanges and hosting facilities certainly poses a risk of some disruption. The Feb. 28 quake in Olympia, Wash., was strong enough to give Teleglobes Seattle facility some problems, cause several of Sprints routers in Seattle to become temporarily unreachable, and shut down the Seattle Times Web site, according to Peter Salus of Matrix.Net.
The physical infrastructure of the Internet is growing so fast and is so intermeshed that anything short of an extended nuclear war probably wouldnt knock it out completely; although knocking out an area like the West Coast would cause traffic fluctuations that could affect Internet performance for days. This is a far cry from the late 1980s, when popular lore has it that someone at the NASA Ames Research Center in Mountain View, Calif., pulled a plug and disconnected all of Sweden. Still, accidents like this sometimes occur. Last month, China Telecom lost substantial communication bandwidth to Japan and the United States when an undersea fiber-optic cable was severed.
A majority of Internet traffic from Europe to Asia still flows through the United States. But while some areas have direct experience with the dangers of such concentrated network infrastructure, there are contrasting stories, like in Yugoslavia. That countrys network remained intact throughout the Bosnian war, even though it suffered some damage.
Find the Missing Pieces Getting to the bottom of the Internets stability is a topic most companies are loath to discuss. No ISP or backbone provider contacted by Sm@rt Partner would talk in any detail about the health of its network or its peering relationships—the secret contracts by which it exchanges traffic with other providers so that data can flow unimpeded across the entire Internet. No hosting facility would acknowledge concerns about power in case of an extended blackout or rolling outages.
And except for eBay, no Web site would discuss, or even acknowledge, whether it had backup plans. EBay says its now infamous 22-hour outage in 1999 was due to the fact that its entire business was hosted on one Sun Microsystems server, which Sun says was felled by operator error. Several vendors, including Sun and Oracle, spent several days at eBay conquering the problem, and eBay says extensive backup and recovery plans are now in place.
Those types of concerns recently led Center 7 to withdraw from several partnerships with data centers and ship its equipment back to headquarters. Center 7, which offers enterprise management services over the Internet, now runs its software out of its own data center in Utah, a telco-style cement>> bunker with satellite backup. CEO Kelly Phillipps says that the facility cost twice as much per square foot as the data centers of some of his ex-partners, but the price, he adds, was well worth it.
“Every problem you could find, we found, from electrical to air conditioning to the network,” Phillipps says. “Their security was wide open in comparison, and our computer got up to 100 degrees. I think if wed gone with Exodus or Digex—that tier of [co-location facility]—we would have been OK, but when you get to the guys like AT&T and Qwest, these problems are pretty common. Their business is on such a commodity scale.” (AT&T counters that its security is unmatched, and that its conservative in how it builds its facilities. Qwest says it invests “significant resources” in its hosting business).
Phillipps has decided theres a market in offering extra protection for data center customers, so Center 7 is working with Opto 22 to monitor backup power generators and other non-IT devices for customers in other co-location facilities.
Its not just the physical parameters that are problematic, either. Ben Reytblat of business service provider Quadrix Solutions claims that the staffing inside the co-location facilities is important. He says the turnover rate of a co-location facility is a good indicator of the quality of management. Reytblat spent a year looking for a suitable hosting environment for his New Jersey-based operation, eventually partnering with Level 3. “Youve got to be paranoid,” he says.
Stand in Line The National Science Foundations decision to privatize the Internet—completed in 1995—and the build out that followed are creating more traffic than anyone imagined. Companies have thrown products and services of questionable value into the market. Rock concerts and lingerie shows are being broadcast on a network designed to exchange research data.
Add to that a plethora of startups with conflicting technologies and conflicting ideas that are trying to fix the Internet—bypass the congested exchange points, deliver the big media files, revamp the broken process by which ISPs peer. Some are capital-intensive; few are profitable.
Consider InterNAP, whose CEO canceled four interviews with Sm@rt Partner, after the company suffered through the Seattle earthquake and then announced that it would miss first-quarter revenue estimates and was cutting costs, which included laying off 65 workers. A spokesman said the CEO was too busy raising money to talk. InterNAP has agreements with backbone providers and has developed software that calculates the most efficient route across the Internet at a guaranteed level of service, bypassing congested exchanges. It functions as both a co-location service and an ISP. But several analysts downgraded the stock after questioning whether the value of InterNAPs service will justify its costs.
Hold the Door Security at important Internet nexus points varies widely, from the fortress-like approach taken by Equinix and Exodus; to the Palo Alto Internet Exchange, also known as PAIX (which excludes financial customers, and so does not have to meet their more demanding requirements); to MAE West.
MAE West is in an office building in the middle of downtown San Jose, and is an important factor in the build out of Internet infrastructure in this area. It began in the early 1990s as a joint effort by several ISPs and subsequently was designated as a network access point by the National Science Foundation. WorldCom acquired MAE West as part of its acquisition of MFS Communications in 1996.
Although WorldCom denied Sm@rt Partner a tour for security reasons, we were still able to take public elevators to three separate floors designated as MAE West facilities and walk around unescorted, both before and after business hours. Equipment was placed behind locked doors that were accessible by card key. For legal reasons, we chose not to tailgate our way inside.
A spokeswoman claims we reached MAE Wests co-location facilities, but not its switches, which are guarded by a human 24 hours a day and accessible only to employees. She also says MAE West discusses security with its customers individually and continues to upgrade it as needed. She denies assertions from Equinix and PAIX that service providers are bypassing MAE West because its equipment is outdated. WorldCom claims to carry “a major portion” of U.S. Internet traffic through its three MAE facilities in California, Texas and Virginia.
Big Bucks, Big Changes Nevertheless, big ideas on what to do about the Internet keep flowing. Some very young companies are tackling very big ideas, and as they do, the Internet changes shape. Mike Gaddis founded CoreExpress after trying and failing to reform the system by which ISPs peer. His company launched last May with $573 million in funding. Former FCC Chairman Reed Hundt sits on the board.
CoreExpress is building a national extranet that spans nine regional centers. VP Greg Davis describes the centers as “super-POPs”—telco-like facilities with big metal doors concealing routers and switches that only engineers are allowed to touch. The company is using fiber from Level 3 and Williams Communications and has interconnections with backbones from AT&T, Sprint, Genuity and UUNet. It is about to announce commercial availability of a new service—CoreExpress Extranet—and hopes to interconnect hundreds more ISPs.
CoreExpress plans to resell its service through these ISPs, whose customers could connect to the CoreExpress network with their own routers and T1 lines and monitor their datas progress at every step. Davis says CoreExpress will not compete with ISPs by undercutting their networks or by selling add-on services such as security. In fact, by using MPLS (Multi-Protocol Label Switching)—a technology Davis claims many ISPs already support—CoreExpress can signal to ISPs which data is premium and pay them to treat it accordingly.
“What the Internet has created is the tragedy of the commons,” he says. “The originating carrier is paid by the customer, keeps all the money, and tries to get rid of the data as fast as it can with hot-potato routing. The receiving carrier has no incentive to carry that data with any degree of quality.”
Jay Adelson, another battle-hardened Internet veteran, also sees himself as owning a part of the Internets core. His current company, Equinix, is a combination Internet exchange point, co-location facility and hosting service that strives to provide a neutral meeting place for Internet companies of every stripe. They include backbone, service and e-commerce providers; data centers; content delivery networks; and so on. Equinix is public and has facilities in six U.S. cities with plans for international expansion—it has raised more than $830 million in funding since Adelson co-founded it in 1998.
“I cant categorize all the companies anymore,” says Adelson, who also founded one of the first ISPs. “All I can do is have freedom of offerings on the open market. In the old days [when the Internet was privatized], ISPs went to the early exchange points. But when the exchange points got too congested and politically compromised, the ISPs moved out. Now theyre moving into Equinix—they can peer, and they can get fiber deployed faster here. If Victorias Secret needs 10 gigs of bandwidth, we can run it into their cage overnight, and take it down, too.”
Adelsons former boss, Paul Vixie, questions Equinixs neutrality. Vixie hired Adelson and Equinix co-founder Al Avery in the mid-1990s to help launch PAIX, which began as a research project inside Digital Equipment Corp. and subsequently became the first Internet exchange not controlled by a telco. PAIX wanted to combat the telcos growing power over the Internets infrastructure, and it pioneered the practice of putting competing telcos into the same facility and offering ISPs a choice of carriers.
In 1999, PAIX was acquired by AboveNet, which sells co-location services, and AboveNet in turn was acquired by Metromedia Fiber Network (MFN). But Vixie claims nothing has changed. PAIX still publishes the names of all of its customers on its Web site>> and is under no pressure to become “spectacularly profitable,” Vixie says, despite MFNs plans to expand PAIX across the United States and into Europe.
“Its a pity that Digital did not see the wisdom of building dozens or hundreds more PAIXes, because thats how I think peering will develop,” he says. “There will be thousands of such places, all run by different people, and the network will be very robust because there will be lots of variety. No one can take it down with a single error. No one company can do it all. Al and Jay got fed up and thought they could do it themselves, but after you get involved with VCs and investors, its hard to sell the vision you need to make that possible. They are selling more than they are peering. They are competing with their customers.”
Adelson says Equinix does not compete with its customers and had no comment on leaving PAIX.
Standards Bearers Regulation of the Internet is not a popular idea, and companies continue to find clever ways to manipulate the Internets infrastructure to achieve their goals. Startup Slam Dunk plans to guarantee the arrival of transactions across the Internet by duplicating them, wrapping them in XML envelopes, and sending them across the two best routes. The messages race to reach their destination; the loser is destroyed. “That way if a backhoe takes out a Sprint cable, were protected,” says marketing director Jeff OMara. “Everything is mirrored in two locations.”
Startup WebEver distributes a Web site across servers throughout the world and then uses software agents to calculate which server can deliver the content fastest at the moment someone makes a request. Bay Area users tend to receive their content from servers in Tokyo because of congestion at MAE West, says business development director Steve Byrnes. Delivery of dynamically generated and interactive content has become a major standards battle—with camps led by Cisco, Inktomi and Akamai—and ultimately will become part of the Internets infrastructure.
It is likely that as companies develop new technology, some of the Internets current problems will fade away. Bernard Daines, who founded start-up Worldwide Packets, claims there will be little need for big server farms, as companies like his bring broadband access into homes. He also believes that the amount of Internet traffic generated as a result will force the industry to reform peering. “Its backwards now,” he says. “Ill be able to go to AOL and say, I have a million homes on Gigabit Ethernet. What will you pay to access them? “
Peer-to-peer computing—in which computers communicate directly by behaving as either clients or servers—also will diminish the need for server farms, although it is too soon to tell how those changes will affect Internet traffic patterns. But since peer-to-peer generates traffic at the edge of the Internet, some architectural changes are bound to happen.
But everyone agrees the Internet cannot become ubiquitous until certain problems—from security, to privacy, to coping with the consequences of an open Internet culture—are solved. The questions are, how many years will that take, how much pain will be suffered in the meantime, and how much money will be thrown at the problem? “In any given week there are different obstacles to be overcome, although I cant think of any completely unsolvable problem,” says Vixie. “Well get through it.”
Estrin appears less sanguine. “There are not short-term economic benefits to doing things right,” she told her audience in October. “But remember there are long-term benefits because customers work in the long term. It doesnt do anybody any good when these things break.”
And surely, things are going to break.