When the Dish Network's storage architecture started to crack, this storage consultant was there to pick up the pieces.Sometimes, you can grow too fast. Just ask Satellite TV provider EchoStar Communications (better known as the DISH Network). The companys backup and storage needs are expected to soar from 18 terabytes in 1999 to more than 41 terabytes by the end of 2001. (A terabyte is equal to one trillion bytes.) EchoStars original storage architecture couldnt keep up with the explosion of data.
"In early 1999, they were struggling with lots of local tape drives," recalls Lee Abrahamson, SAN consulting manager for Computer Network Technology (CNT). EchoStar had hundreds of production servers (mostly HP-9000s running HP-UX) on a 100Mbps Ethernet network. Each server connected via SCSI to its own EMC Symmetrix disk array or a StorageTek tape library.
The fragmented tape space was severely underutilized. Inherent single points of failure left the disaster-recovery plan vulnerable. Adding another tape drive for every new server was expensive. Backup traffic drained bandwidth from the production servers. Scheduling and performing backup sessions became a personnel nightmare. "They were even buying robots to load tapes," says Abrahamson, "but even so, they had high turnover" among system administrators.
EchoStar knew it had to revise its storage plan. Says Rick Nelson, the companys senior IT architect: "We turned to CNT after we realized that our first [conceptualization of a] SAN solution, based on Fibre Channel-to-SCSI bridges that connected backup servers to tape drives, turned out to have some immediate scaling problems. We could see that scaling issues—and the resulting cost inefficiencies—would make that solution extremely undesirable in a short amount of time."
Four-Step SAN Plan Nelson needed a long-term backup, storage and disaster-recovery solution. He turned to CNT because the integrator has a 17-year track record in storage network design. He had a clear vision of what his ultimate SAN would include, broken out into four phases.
1. A local tape-backup SAN, which would provide 24-hour backup windows with zero impact on EchoStars production networks.
2. Remote disk-mirroring over IP, which would provide rapid recovery of critical data from EchoStars disaster-recovery data center in Wyoming, more than 100 miles from the Colorado headquarters.
3. Remote tape vaulting over IP.
4. Local serverless backup linking disk storage directly to tape, which would free up servers and reduce backup traffic loads on EchoStars production servers.
The first phase was completed last December, after several months of testing and tweaking.
"We had to spend a couple of months certifying HPs Fibre Channel connectivity with our UltraNet Open Systems Director," says Abrahamson. HPs server Host Bus Adapter cards had some compatibility problems with non-HP gear, which required upgrading to newer cards and applying some software patches.
But ultimately, the testing and benchmarking program verified that EchoStar could achieve a 700 percent performance boost using its existing HP fabric and Open Systems Director (OSD), part of CNTs UltraNet family of SAN products.
The OSD is a multigigabit midplane switching platform. It allows Fibre Channel and SCSI SAN technologies to be shared and mixed across Windows NT and Unix systems. UltraNet SAN Software (USS) drives OSDs six- or 12-slot chassis. The USS package controls configuration, operation and management of the switch, system-service monitors and other core components. The OSD includes WAN interface support for 10/100 Ethernet, ATM/Sonet fiber (OC-3) and T3/E3.
"The OSD provides extremely flexible connectivity options, which is just what EchoStar needed," says Abrahamson. "Also, it enables more efficient utilization of bandwidth from backup servers" by giving EchoStar more control over the mix of tape and disk devices deployed on each Fibre Channel connection into a backup server.
EchoStars new local tape-backup SAN is a far cry from its old architecture (see diagram, at right). The EMC Symmetrix disk arrays now hold mirror images of the production servers data, taking a load off the production servers and LAN. The disks connect to backup servers via Ultranet Fibre Channel Switches (made by Brocade Communications). Several Open Systems Directors steer backup data to 24 StorageTek 9840 and DLT7000 tape libraries.
Go the Distance The second phase of EchoStars SAN strategy is a remote disaster-recovery center, located 100 miles from headquarters. Distance was a challenge, since Fibre Channel network runs are usually limited to about six miles.
But CNTs UltraNet Storage Director enables Fibre Channel over IP networks. It converts data frames to IP packets, handling the end-to-end connections, packet ordering, flow control, error correction and reconversion at the destination. Combined with EMCs Symmetrix Remote Data Facility (SRDF) software, it enables automatic mirroring of EMC enterprise storage systems like EchoStars disk arrays over wide areas—very wide areas. Another CNT customer, P&O Nedlloyd Container Line, recently began mirroring databases and applications between its U.S. and U.K. data centers.
EchoStars Nelson began testing the SAN-over-IP solution using T1 lines last fall. He expects to fully implement the system at the end of January using EchoStars private IP network.
Ready for the Storage Boom? An enterprisewide SAN project like this one often sells for $1 million to $2 million, according to CNT executives. Nelson expects a 14- to 16-month payback time, mainly in productivity gains, reduced downtime and faster data recovery.
"We may tackle their NT side next," says Abrahamson. That would require an entirely separate SAN, and another nice paycheck for CNT.
Data Storage, Data Backup and Storage Virtualization 
