At a time when IT executives are finding users less and less tolerant of network delays, CIO William Morgan has perhaps the least tolerant set of users in the world.
Morgan runs technology operations for the 215-year-old Philadelphia Stock Exchange, the nations first stock exchange. But its age doesnt make its daily load any easier, with networks having to handle 120,000 messages per second and peaks of 200,000 messages per second.
But because it is a financial exchange, any delay—even a half-second—is not acceptable.
“We measure transactions in milliseconds these days. This business cant tolerate delays: Were pricing customer orders,” Morgan said. “Its survival for us.”
In recent years, financial activity has pushed that IT demand much higher. “If you go back five or six years, probably the number [of messages per second] was 10,000 or less,” which is one-twelfth todays volume, Morgan said.
Morgan delivers that real-time speed with some homegrown applications sitting atop Sun Microsystems Solaris 10 servers, Stratus fault-tolerant servers in a Nortel network.
The CIO argues strongly for using as much standardized software as possible; the exchanges Web site runs on Windows, and e-mail is using Microsoft Outlook.
“We use the standard Windows environment for all that, but not for our trading. On the trading side, there simply arent many packages,” he said.
“There are many for broker dealers, but a select for exchanges. There arent that many exchanges and, because of the custom nature of each exchanges business, its very hard to find an off-the-shelf” package.
Having the network deliver all of those messages per second—Morgans people stress test their system with 200,000 messages per second—is only part of the battle.
After the messages are delivered, they have to be stored, catalogued and archived. These days, thats about one-half billion messages every day.
All things considered, Morgan said, the storage is the easy part. “Data storage for us during the day is not the challenge. The challenge for us is retention,” he said. “This is more about cost, given our size and the challenges.”
At about 490 million messages a day, the government-required seven years of message retention adds up quickly.
The exchange handles messages in two ways, splitting them into recent messages (about three months worth) where the data needs to be readily accessible and the remainder that can be held in offsite storage about 25 miles away from headquarters.
Much of the data is managed in SANs (Storage Area Networks) with about 10TB of storage at headquarters.
The balancing act of when to have data no longer be so readily available is primarily a budget issue.
The data that is kept super-accessible costs a lot more to maintain. “The longer you wait for it, the cheaper the storage,” Morgan said.
