Inside iSCSI: Low-Cost Storage Networking

Networked Storage Backgrounder

With the growth of e-commerce and the Internet, storage has suddenly taken on a whole new importance. Once a boring appendage to the server, data storage has become a strategic asset, tapped 24 hours a day by multiple applications and servers, and mined for valuable personalization information and business intelligence. As companies increasingly digitize their corporate documents, work with large image and video files, and replicate data to alternate locations, they are seeing their storage requirements double or triple in a single year. Controlling this explosive growth and making storage available on a 24/7 basis to employees, partners, and customers across the globe presents IT with major management challenges and expense.

To get the storage monster under control and fulfill its more stringent availability requirements, many companies are consolidating their numerous server attached storage devices into fibre channel storage area networks (SANs)– full scale networks, separate from the communications network that are dedicated to storage and only storage (see our stories Introduction to SANs and Build Your Own SAN for solid background on SAN technology). Once separated from the server, storage can be pooled, partitioned, and allocated to servers and applications at will. SANs enable LAN-free and serverless backup, which lets IT centralize network backup and take it off the communications network. SANs also vastly simplify high availability implementations such as clustering, replication, and off-site backup and mirroring. In other cases, storage is being consolidated into network-attached storage devices (storage appliances that attach directly to the Ethernet LAN and serve files to a variety of client operating systems).

Fibre Channel is a great technology for SANs because its fast– up to 2Gbps today, has much lower overhead than Ethernet, and uses the block-based transactions important for datacenter applications. Its also relatively easy to integrate legacy SCSI storage devices into the SAN storage pool. And Fibre Channel SANs are very reliable typically touting high availability metrics. Unlike server-attached storage, theres no need to shut down or reboot a server in order to add storage to a SAN. You add a storage device and its immediately available to all SAN-attached servers. And if a particular server dies, the SAN storage is still accessible by other servers.

Unfortunately Fibre Channel has yet to take off in small and medium sized organizations for a number of reasons. For one thing, implementing Fibre Channel means learning a whole new, fairly complex infrastructure, with its own cards, hubs, switches, and cabling from companies with unfamiliar names like Gadzoox and Brocade. Fibre Channel standards havent evolved to the level of Ethernet, in which cards, hubs, and NICs from different manufacturers work together effortlessly. Interoperability is better than it once was, but in most cases, once you set up a Fibre Channel infrastructure using products from a single vendor, your best bet by far is to stick with that vendor for future purchases. Managing Fibre Channel SANs requires a whole different set of skills and training, and the pool of Fibre Channel experts is considerably smaller than its Ethernet equivalent. Plus Fibre Channel hardware is expensive.

Enter iSCSI

SAN vendors have been looking for alternatives to Fibre Channel that can combine the familiarity, bandwidth, and low cost of Ethernet with the efficient, speedy, block-based data transfer performance of a SAN. As you can imagine, lots of alternatives have come up to fill the gap, but the one that appears ready to take off is called SCSI over IP, or iSCSI.

Well go into more details of iSCSI below, and you can read our How iSCSI Works sidebar, but overall its a pretty simple concept— its the mapping of SCSI commands (data and status) over a TCP/IP network. In an iSCSI based system, the user or application creates a command to store data on (or get data from) a SCSI-based storage system. The operating system processes the request and converts it to SCSI commands, which are then sent to software or an iSCSI card. The card or software encapsulates the data into a serial string of bytes preceded by an iSCSI header, and passes it to a TCP layer where it is packetized/encapsulated, and if necessary, encrypted for transfer over the network At the other end of the connection a storage controller decrypts the information in the packets, then uses the information in the iSCSI header to send the SCSI command/data to a SCSI drive, which performs the requested functions. If the request is for data retrieval, the data is encapsulated and returned.

The concept of iSCSI is simple, but very powerful— you get all the benefits of block-based transactions and SCSI intelligence and compatibility, with the distance, bandwidth, standardization, low cost, and huge base of expertise available for Ethernet. What about all that TCP overhead? Theoretically you could use existing Ethernet cards with specialty iSCSI drivers to run SCSI over TCP/IP, but the TCP processing will eat CPU cycles for lunch and bog down performance. Instead, a whole new crop of cards that perform TCP protocol processing and sometimes iSCSI processing on board through special ASICs will rev up performance to Fibre Channel levels and beyond.

iSCSI Benefits

The potential benefits of iSCSI are many:

• iSCSI gives you the block-level access and storage I/O intelligence of SCSI, but transforms your SCSI storage devices from server-attached to network-attached, where they can be managed, partitioned, and allocated to servers and applications much more efficiently.
• You get much better reliability and availability than you do with regular SCSI. As with Fibre Channel SANs, theres no need to take down a server to add storage to an iSCSI SAN, and once added, that storage can be made available to any SAN-attached server.

• As with Fibre Channel, you can incorporate all your existing SCSI RAID devices, tape libraries, and other storage into your iSCSI SAN, as long as you provide them with specialized adapters that can extract the SCSI commands from incoming IP packets.
• Instead of the 15-device limit imposed by SCSI, iSCSI should let you connect hundreds or thousands of storage devices to the SAN.
• Youll be able to build your iSCSI SAN at considerably less expense than a Fibre Channel SAN, because you can use the same standardized, inexpensive Ethernet hubs and switches you use for your current communications network.
• You can use a single WAN or even LAN connection to transport both file and block-based data. If you need redundant connections with failover capability, using the same infrastructure for both saves even more.
• Configured as a separate network, iSCSI can provide the same benefits of LAN-free and server-free backup as Fibre Channel.
• You wont be trapped with one or two Fibre-Channel specific vendors (as current Fibre Channel users may be today). Instead you can mix and match hardware from your favorite networking vendors.
• Investment in training should be minimal, as iSCSI is based on the familiar technologies of SCSI, Ethernet, and TCP/IP.
• Whereas Fibre Channel SANs have distance limits of approximately 6 miles without repeaters that limit them to campus or, in some cases, metropolitan SANS, iSCSIs use of TCP/IP lets you create SANs that span countries, continents, or the entire planet, using T-1, T-2, ATM, Frame Relay, DSL, ATM, and various types of Virtual Private Network connections over the Internet. This vastly simplifies off-site backup, replication, and mirroring over large distances. Backup and disaster recovery strategies and policies have risen in prominence since the World Trade Center disaster, and iSCSI provides excellent capabilities in this area. It also makes it easier to outsource storage from an iSCSI-enabled storage provider.

• You can manage your iSCSI network with the same network performance and management tools youve used for your communications network. With Fibre Channel, you have to master a whole new set of tools, which are frequently less refined than their TCP/IP and Ethernet cousins.
• Encapsulating SCSI commands in TCP/IP lets you use multicasting to send the same commands to multiple devices at the same time, a major advantage for companies that need the high availability provided by mirrored storage centers dispersed around the country or across the world.
• Fibre Channel security is just about nonexistent, but iSCSI can theoretically take advantage of the TCP/IP security infrastructure for encryption and protection, including IPSec. This is particularly important in high security environments like the government and financial community. If necessary, you can also implement IP-based quality-of-service functionality.
• iSCSI is plenty fast today, taking advantage of high speed Gigabit Ethernet, ATM, DSL, and DWDM connections today, and 10 Gbit Ethernet tomorrow.
• iSCSI will accelerate the integration of Network-Attached Storage (which is file-based and runs over Ethernet), and storage area networks (which are block based and currently run over Fibre Channel). NAS tends to be more appropriate for file access, while SANs make sense for datacenter applications. However the two technologies have been converging, with NAS systems backing up to SANs using Fibre Channel adapters. With iSCSI, both could run over the same infrastructure if appropriate.

iSCSI Ramping Up

The existing storage area network marketplace is dominated by Fibre Channel SANs at the high end. Large-scale enterprises and service providers have the need, resources, and expertise to build and manage two separate network infrastructures (storage and standard data communications). Medium-sized businesses and even some larger enterprises and small businesses have eyed the benefits of SANs hungrily, but so far have been reluctant to jump on board for lack of resources, expertise, and courage. For these users, iSCSI may provide the answer theyve been looking for. As a result, interest in iSCSI has taken off this year, and the standard is progressing pretty quickly.

Over 60 companies have jumped on board the iSCSI Working Group of the Storage Networking Industry Association (SNIA). Intel, Adaptec, Emulex, Alacritech, and QLogic have shipped host bus adapters, with TCP/IP and iSCSI protocol processing on board using ASICs. IBM has an iSCSI storage system, and Cisco has two iSCSI to Fibre Channel routers. Numerous other vendors have also rolled out iSCSI hardware or plan to soon (see Companies to Watch list below).

In early September 2002, the Storage Network Industry Association IP Storage Forum (SNIA IPS Forum) and its member companies announced that the iSCSI standard had completed the IETF IPS Working Groups “last call,” which means the standard was technically complete with only minor editorial comments remaining. It was then submitted to the IETF Steering Group (IESG) for review, which usually takes from three to six months, according to Ahmad Zamer, iSCSI subgroup chairman for the SNIA IPS Forum. This means that a final standard should be approved by February 2003. There have already been several iSCSI plugfests to test product compatibility and interoperability, mostly at the University of New Hampshire. Cisco offers a number of universal drivers for Ethernet cards, and Intels Architecture Lab released an open source reference platform for storage device vendors to help them develop switches, routers, and adapters that comply with the spec.

iSCSI Deployment Issues and

Summary”>

With any new technology, most analysts and industry players predict that iSCSI will take at least another 18 to 24 months to achieve a significant level of maturity. Also, with any new technology, performance, price, and compatibility quirks take time to work themselves out. With all its disadvantages, Fibre Channel is still the more mature technology today. Even iSCSIs proponents admit that while its current performance is decent, its not yet up to the level of Fibre Channel. Nor can it be with Ethernet maxing out at 1Gbps and FC available up to 2GBps today. Tests run on Ciscos SN5428 Storage router by Eweek have found iSCSI throughputs up to 82MB per second compared to 164MB per second for 2Gbps Fibre Channel. Keep in mind, however, that performance should improve when 10Gbit Ethernet becomes widely available. And the types of storage management and storage virtualization programs that are available in the world of Fibre Channel are in their infancy in iSCSI. As such, iSCSI will probably have a phased rollout.

At first, a number of shops currently running existing fibre channel SANs will probably use iSCSI to connect two Fibre Channel SANs over a WAN, allowing their SANs to span continents where necessary, in order to benefit from SAN high availability functions like mirroring and backup over longer distances than Fibre Channel can provide. This scenario would require an iSCSI/FC router at each end of the connection. They may also use iSCSI to provide branch offices without Fibre Channel functionality access to the corporate office Fibre Channel SANs. In these configurations an iSCSI-to-Fibre Channel Router will do the translation on the corporate office end.

SANs that currently use the more mature Fibre Channel standard over shorter distances can be expected to stick with Fibre Channel for at least a few more years to come.

When it comes to long distance connections, iSCSI has two other standards to compete with: Fibre Channel over IP (FCIP), backed by Brocade, Cisco, Nortel, Lucent and others, and the Internet Fibre Channel Protocol (iFCP), backed by Cisco, Nortel, Quantum, Nishan Networks, and Sun. FCIP essentially tunnels Fibre Channel in IP. IFCP maps Fibre Channel end devices to IP using the same protocol iSCSI uses and requires gateways at both ends of the connection. (See the SNIA IP Storage Tutorial for more details on these alternatives.)

As the iSCSI standard, hardware and software matures, youll probably see smaller and less tech savvy medium sized businesses implement local small-scale iSCSI SANs, since the hardware and software will be more familiar and less expensive than its Fibre Channel cousins. Analysts predict that over the next four or five years, iSCSI SANs will become the technology of choice for most organizations implementing a SAN from scratch.

Storage service providers are expected to have a particular interest in iSCSI, especially with the advent of 10Gb Ethernet, which will give iSCSI tremendous bandwidth. Theoretically, a hosting firm that needs an extra 100TB for a particular project could plug into a high-speed long distance storage network and outsource that 100TB from a service provider in minutes, rather than taking the weeks or months to purchase the extra storage and put it online.

iSCSI networking hardware will consist of network cards with special TCP/IP and iSCSI processing ASICs and iSCSI drivers for the servers, existing Ethernet switches and hubs, and specialized iSCSI routers and switches that will interface between the Ethernet LAN and Fibre Channel or iSCSI SANs. Target storage devices will most likely be fitted with iSCSI channel adapters that can extract SCSI commands from incoming IP packets.

In summary, the biggest benefit of iSCSI technology as it matures, is that users will be able to take the SAN plunge without having to invest and train in a completely separate networking infrastructure. Organizations that have shied away from the hassles of Fibre Channel will undoubtedly be tempted to use iSCSI to take advantages of the benefits of storage area networks. And as organizations expand their corporate Internet, intranet, and extranet sites and applications, SANs will increasingly be the only viable solution for taming the ever expanding storage beast.

SideBar

: How iSCSI Works”>

As its name implies, iSCSI is based on the existing SCSI (Small Computer Systems Interface) standards currently used for communication among servers and their attached storage devices. SCSI has two types of devices– SCSI initiators (usually servers, which start the communications by issuing the commands to be executed) and SCSI Targets (usually storage devices, which respond to the initiators and carry out the commands).

Targets consist of a number of “logical units” (LUs) that are directly addressable and execute the SCSI commands. Commands are communicated via a structure of “Command Descriptor Blocks” (CDBs) and are often combined into “tasks.”

SCSI Command Descriptor Block (CDB) Format

Courtesy of Cisco Sytems, Inc.

Commands are executed with data phases, in which the data travels from the initiator to the target (typical of a WRITE command), or from the target to the initiator (as in a READ command), and with status phases, in which the target completes the operation and terminates the SCSI command or task. iSCSI gives SCSI initiators and targets unique URL-like names, and provides a method for their discovery and mutual authentication. These names can be assigned by OS vendors, NIC and driver vendors, device vendors, gateway vendors, or even service providers and customers.

SideBar

: How iSCSI Works (Continued)”>

Applications typically issue requests, which are built into SCSI CDBs by SCSI drivers, and the requests are forwarded to the iSCSI layer. SCSI drivers also receive CDBs from the iSCSI layer and forward the data to the application. When the SCSI CDBs pass from the SCSI layer to the iSCSI transport layer, they are encapsulated into an iSCSI Protocol Data Unit (PDU) and forwarded to the Transmission Control Protocol (TCP) layer for packetization and transport. With a READ command, for example, the iSCSI transport layer extracts the CDB from the PDU that comes from the TCP layer and forwards it to the SCSI layer.
Overall structure of a PDU template is as follows

Courtesy of Cisco Sytems, Inc.

Before sending SCSI commands to a target, iSCSI initiators establish an iSCSI “session,” which consists of one or more TCP connections over a well known TCP port, established by the initiator. Targets listen over the TCP ports for incoming connections. The initiator then begins a login phase that authenticates the initiator and target, negotiates the session, and identifies the connection as an iSCSI session.

After the login process is finished, the initiator can send SCSI commands and data to the LUs on the target by encapsulating them into iSCSI messages. iSCSI usually numbers the commands that are passing from the initiator to the target, carried by the iSCSI PDU as Command Sequence Numbers (CmdSN), which are unique to each session. The iSCSI target layer delivers the commands in the order indicated by the CmdSN. Responses are also numbered by iSCSI and carried by the iSCSI PDU as the Status Sequence Numbers (StatSN). The initiator maintains an Expected Status Sequence Number (ExpStatSN) to acknowledge status. If the StatSN and ExpStatSN are different, it usually indicates a failed connection. The Domain Name Service (DNS) can be used to resolve the iSCSI URL to one or more IP addresses.

Basic Header Segment (BHS) for SCSI (Initiator) Command

Courtesy of Cisco Sytems, Inc.

Basic Header Segment (BHS) for SCSI (Target) Response

Courtesy of Cisco Sytems, Inc.

SideBar

: Companies to Watch”>

Adaptec
691 South Milpitas Blvd
Milpitas, CA 95035
www.adaptec.com
(408) 945-8600

Makes ASA-7211 iSCSI Adapter, an ASIC-based iSCSI HBA with TCP/IP and iSCSI protocol processing onboard, as well as ASICs for iSCSI adapters and routers.

Agilent Technologies, Inc.
Semiconductor Products Group
3175 Bowers Ave. MS 86F
Santa Clara, CA 95054
(800) 235-0312,(408) 654-8675
www.agilent.com/semiconductors

Agilents Semiconductor Products Group makes the ANIC-2101A and ANIC-2103 1Gbps iSCSI PCI adapters, both of which include onboard TCP/IP and iSCSI protocol processing.

Alacritech, Inc.
234 E. Gish Road
San Jose, CA 95112
www.alacritech.com
(408) 287-9997

Alacritechs 1000×1 Single-Port Server and Storage Accelerator is an iSCSI Gigabit NIC that can perform both IP storage and standard networking functions using an onboard ASIC. Also makes a dual and quad port 100Mbit iSCSI accelerator cards.

Cisco Systems, Inc.
170 West Tasman Drive
San Jose, Calif. 95134
www.cisco.com
(408) 526-4000

Cisco has two storage routers for connecting iSCSI and Fibre Channel, the SN 5420 Storage Router, which has one Fibre Channel and one Gigabit Ethernet port, and the newer workgroup oriented SN 5428 Storage Router with eight 1GBps/2GBps autosensing ports and two Gigabit Ethernet ports.

CNT
6000 Nathan Lane North
Minneapolis, MN 55442
www.cnt.com
(800) 638-8324
International (612) 268-6600

Currently shipping, the UltraNet Edge Storage Router uses FCIP for connecting two Fibre Channel SANS. Shipping in Q 3 , 2002, the UltraNet Edge Storage Router will add FC over ATM connectivity.

Emulex Corporation
3535 Harbor Blvd.
Costa Mesa, CA 92626
www.emulex.com
(800) 854-7112 , (714) 662-5600

The Emulex GN9000/SI is a 64-bit PCI iSCSI host bus adapter that connects to gigabit Ethernet and processes the iSCSI and TCP/IP protocols in firmware. Shipping “after ratification of the iSCSI standard later this year.”

Entrada Networks
12 Morgan
Irvine, CA 92618
(949) 588-2070
www.entradanet.com

Makes SilverLine 222, an FCIP storage switch with two Fibre Channel, two gigabit Ethernet, and two WAN/MAN ports.

Eurologic Systems
1300 Massachusetts Ave.
Boxborough, MA 01719
www.eurologic.com
(800) 231-4070

Makes network storage for OEMs and vertical markets. The Elantra iCS 2100 IP Storage System is an iSCSI storage enclosure that holds up to 1TB in 14 drives and comes with IP security and storage virtualization software.

FalconStor Software
125 Baylis Road, Suite 140
Melville, NY 11747
www.falconstor.com
(631) 777-5188

FalconStor makes IPStor iSCSI and Fibre Channel compatible storage management software.

Finisar, SAN/LAN Performance Tools Division
1389 Moffett Park Drive
Sunnyvale, CA 94089-1133
www.finisar.com
(408) 400-1000

Finisar makes the GTX Gigabit Ethernet/iSCSI Protocol Analyzer.

IBM
New Orchard Road
Armonk, NY 10504.
(800) IBM-4YOU, (404) 238-1234.
www.ibm.com

IBM makes the IP Storage 200i family of iSCSI disk systems.

Intel Corporation
2200 Mission College Blvd,
P.O. Box 58119
Santa Clara, CA 95052-8119
www.intel.com
(408) 765-8080

The Intel PRO/1000 T IP Storage Adapter