The Chicago Mercantile Exchange is undergoing an unprecedented growth period, with the volume of trading doubling every year since Sept. 11, 2001. The exchange handles commodity futures (for example, live cattle and lumber), foreign exchange (such as the euro and the peso), equity index funds (like Nasdaq and the S&P 500) and one of the largest futures products in the world, the Eurodollar short-term interest rate contract.
All together, the CME handles nearly 3.7 million contracts at a notional value of nearly $4 trillion every day.
Why such a boom in futures? According to the CME, more and more people have been learning about derivatives, aka futures.
In addition, the CME has undergone a hub-based expansion to a handful of European markets, giving foreign traders direct access to their marketplace, which simply wasnt available before.
The combination of increased education, volatility and international expansion have contributed to the CMEs dramatic trading increase, said Allan Schoenberg, associate director of corporate communications at the CME.
To deal with the tremendous upswing, the CME had to increase capacity. Such an overhaul was going to require a massive networking restructure and the need to revisit how the CME stores and backs up all its trades and communications.
The exchange took a twofold approach to rebuilding. It relegated networking duties to the expertise and quality assurance efforts of its internal staff. For storage and redundancy, it turned to an integrator the exchange had trusted since 1998.
Given the in-house expertise and the intimate knowledge of every hop on the network, the internal teams took on the responsibility for improving speed, thereby increasing capacity. The six teams analyzed every step of the trading process and looked for where they could trim the fat: a few milliseconds here, a few milliseconds there.
“Things that were done in the beginning because it was the most robust way to do it—we found other ways of keeping robustness but doing it quicker,” said CMEs director of distributed computing, Joe Panfil.
The biggest improvements came in the restructuring of the CMEs architecture, Panfil said. For example, Panfils teams would look at the inbound path of an order before it reaches the matching engine. Theyd notice that it might be currently touching five servers. Theyd ask themselves, “How can we get it so it only touches two servers before it hits the engine?”
Other improvements included upgrading Cisco Systems Inc. 5500 switches to 6500s, optimizing applications, continually upgrading the Tandem match engine and a massive conversion to Linux, Panfil said.
The last step was to split interfaces on individual server boxes where appropriate. Instead of having a box do double duty by pushing and pulling data, Panfil said, the teams would split interfaces by dedicating one box to sending data and another to receiving data.
While the internal teams were improving speed and capacity, the CME looked outside for help on redundancy and storage.
The CMEs high requirements for business continuity and disaster recovery are similar to those of other financial institutions. The new trading volume introduced new redundancy and reliability concerns. But when the CME went public in December 2002, data protection was no longer just a good practice—it became a federal requirement.
Unlike when the exchange was a not-for-profit organization, the CME became subject to the Sarbanes-Oxley Act and other SEC (Securities and Exchange Commission) and CFTC (Commodity Futures Trading Commission) regulations for storage and retrieval of historical financial records.
Since the late 90s, the CME has relied on Cluster Server and NetBackup, an open-systems clustering solution and a data protection application, respectively, from Veritas Software Corp., of Mountain View, Calif.
The CME had been satisfied with the high availability and storage protection that Veritas products provided, so in 2003 the exchange turned to the integrator to take another look at the new CME.
“One of the first questions I ask customers when I meet with them is, What percent of your restores are done in the first 60 days of the datas life?” said Mike Speiser, vice president of product management and product marketing at Veritas.
Repeated answers to that question revealed that more than 75 percent of recovered data is retrieved within 60 days of its creation. That recovered data could be a corrupt file, a virus or some files that someone deleted accidentally. “Thats the vast majority of restores,” said Speiser.
As a result of this finding, Veritas helped the CME switch from a tape-only data protection system to a hybrid disk/tape-based system late last year. The disk-based portion allows for fast reads and writes for quick restores during that common 60-day retrieval period.
The tapes are for true disaster recovery, when you have to get back everything, which accounts for only 10 percent to 25 percent of restores, said Speiser.
For the disk-based portion, the CME deployed the Revolution 200T, a VTL (virtual tape library) system from Copan Systems Inc., of Longmont, Colo. To the CMEs well-defined backup processes, the VTL looks just like a tape drive.
But the VTL is actually a disk array that delivers fast random-seek recording and retrieval. “You get all the benefits of disk without having to change your processes,” Speiser said.
The Copan system was just one of a multitude of hardware devices running on different operating platforms, such as Unix, Windows, Novell, Linux and Oracle.
“We use a multiple-vendor approach to all technology acquisitions. We have proven that this always gets us the best, most cost-effective solution,” said Craig Taylor, associate director of open systems at the CME.
In this diverse platform environment, NetBackup could speak to all hardware and platforms and remained a good fit to protect all the CMEs 40TB of data, said Speiser.
Expanding the exchange with its new speed and protection capabilities hasnt been easy.
“The biggest pinch point weve seen is on our wide-area network,” said Panfil, referring to the CMEs international customers. As quote streams from overseas got larger and larger, there was a noticeable pinch. The exchange worked through this crunch, said Panfil, by opening up the WAN gateway with global hubs. Available in a handful of metropolitan areas in Europe and soon in Singapore, each hub offered customers direct links to the CME.
Panfil said he believes the CME can handle 10 more years of this doubling growth. Bandwidth capacity is currently in the 10 percent range or lower. It jumps up to 50 percent or 60 percent only when there are spikes in activity. Panfil said he understands and appreciates the demands of the exchange.
“It really isnt acceptable to say, Were busy today, so instead of getting a 50- millisecond round-trip time, youre going to get a 500-millisecond round-trip time,” Panfil said. “[Federal Reserve Board Chairman Alan] Greenspan will speak, and well get this huge spike in activity. Were always accommodating for that spike.”
David Spark is a free-lance writer in San Francisco. He can be contacted at [email protected].