eWEEK content and product recommendations are editorially independent. We may make money when you click on links to our partners. Learn More.
ETrade Financial saved $13 million annually, realized a boost in performance and changed the way it thinks about development by moving from a Sun Solaris infrastructure built up during the stock market boom to an open-source stack running on x86-based IBM hardware.
Building on work it began in late 2001, the financial services company has moved its open-source migration up the stack and is now taking the lessons it has learned through its experience with open source to optimize internal development processes.
Those lessons will be critical as ETrade assimilates acquisitions and continues to grow to compete with rivals such as Ameritrade.
In December, ETrade bought BrownCo, the online trading arm of JP Morgan Chase, for $1.6 billion. In October, ETrade closed the purchase of Harrisdirect, another online broker, for $700 million.
On Jan. 23, ETrade reported that 2005 revenue was $1.7 billion, up from $1.4 billion in 2004. Net income was $430 million, up from $380 million a year ago.
“It was a win-win-win,” said ETrade CIO Greg Framke, of the company’s open-source experience. “You can work a whole career in technology and maybe get one of those, so it was very gratifying to have made that bet early and to have reaped those benefits very early, and it basically just fed our appetite for continuing down the open-source path.”
In the late 90’s, the dot-com bubble was rapidly expanding, with ETrade right in the center of it. The online company’s trading volume and the areas in which it did business were expanding at a rapid pace, and so, too, were its hardware requirements.
ETrade, like many other financial services companies at the time, managed the growth by adding capacity in the form of large servers from Sun Microsystems running Sun’s proprietary Solaris operating system.
Lee Thompson, ETrade’s vice president of architecture, likened the rows of Sun servers to the aisles in a large supermarket: “You could go into an ETrade data center, and you could imagine yourself looking down the aisles of shelves,” said Thompson in Austin, Texas. “Instead of food, you would see Sun Enterprise 4500s-three full rows of them.”
During the fall of 2001, a set of circumstances prompted ETrade’s IT organization to investigate a move from the Sun Solaris architecture that had been serving the company’s crown-jewels application, ETrade.com, to a mostly open-source stack anchored by Linux.
One of the drivers, said Thompson, was a cooling stock market and its effect on ETrade’s bottom line. “Trading volumes were above 300,000 average trades a day, and that dropped to something in the range of 55,000 a day in 2001,” he said. “Our cash flow was affected, obviously, quite dramatically.”
ETrade’s engineers were aware of Linux as an emerging alternative to proprietary Unix operating systems. “At the time, I knew that [Amazon.com] had already moved over to Linux, and Yahoo was on a FreeBSD stack since its inception,” said Thompson.
“And so I knew that, at some point, architecturally, an open-source operating system would be of some interest.”
Also prompting ETrade to move off the Sun servers was the addition of symmetric multiprocessing and a 32-bit message queue to the Linux 2.4.7 kernel that shipped with Red Hat Linux 7.2, the distribution that ETrade was testing.
Without those features, Linux could not serve as a suitable porting target for ETrade’s applications, according to Thompson.
Last but not least was the public embrace of Linux by two industry powerhouses: IBM and Hewlett-Packard. With their support, said Framke, moving to Linux didn’t mean that ETrade had to move out on its own.
“Fortunately for us, HP and IBM came out and said, We’re going to stand behind Linux,” said Framke in New York. “The second that happened, we made our Linux decision, and we rolled it out very aggressively, starting in winter 2001.”
First, however, the IT organization had to get ETrade.com up and running on Linux.
Thompson assigned an architecture team from the IT organization to study the feasibility of porting ETrade.com’s central components to a Linux and open-source stack.
As part of the study, the team replaced ETrade’s iPlanet (now Sun) Web server and application server with open-source equivalents running on Red Hat Linux 7.2.
“We grabbed Apache, and we grabbed the Jakarta Tomcat servlet system, and we ported a representative stack of our application-our authentication, quote services, product services, some of our trading services and the servlets that rendered the HTML-over to this new stack,” said Thompson.
The team also replaced the Solaris version of BEA Tuxedo with the Linux version of the transaction management system, sticking with Solaris for the Sybase database portion of the platform.
Tests showed that the Linux box could handle about 180 users at a time, compared with 300 to 400 users on one of ETrade’s Sun 4500 servers. Performance on the Linux box dropped sharply when there were more than about 180 users, but, before that point, the Linux box performed faster than the Sun servers.
After the Linux box’s price was factored in, ETrade had itself a new computing platform.
Next Page: The little machine that could.
The Little Machine that
Could”>
“I demonstrated this on Halloween 2001, and it was pretty simple,” said Thompson. “These Sun 4500s were costing about a quarter of a million dollars apiece, and we could run 300 to 400 users on them. And here’s this little box that cost us about $4,000. It was running faster, but you could only run 180 people on it.
“Josh Levine, the CIO at the time, said, Well, then we just need to get two, for $8,000. Do it-put a box in production. So, it was a fairly short meeting.”
With this mandate, ETrade’s architecture department got back to work, putting the new stack through a series of tests. In December 2001, they put the Linux box into production, alongside the rows of Sun 4500’s that sat behind ETrade’s load balancers.
“Heres this little $4,000 box, playing, literally, with the big boys of computing and doing very, very well,” said Thompson.
“Everybody was pretty floored by this-a lot of the systems engineers who do our production operation were all going over to the box and couldn’t believe how well this little machine was doing.”
When Thompson and his team began porting the ETrade.com application to Linux, they first copied the whole code base into a separate Linux branch.
It was in this branch that the team made the adjustments necessary for the code to run properly on Linux. For example, the move from Solaris to Linux also involved moving from the Sun Studio C compiler to GCC, or GNU Compiler Collection, the dominant Linux and open-source compiler, and the team had to sort through syntax errors turned up by the compiler change.
During the next couple of months, the team ported the rest of the ETrade.com application. In the spring of 2002, the team brought in 50 IBM x330s servers and, in stages, shifted traffic through their load balancers from the rows of Sun machines to the rack and a half or so of x86 Linux boxes.
The Sun servers were reallocated, retired or sold to other companies.
While cost savings was the primary motivation for exploring an open-source x86 migration, ETrade found that the move also yielded significant performance increases.
The open-source migration also afforded ETrade a higher level of flexibility. This was especially important because ETrade was making many acquisitions at this time, according to Framke.
“Linux fit our model perfectly for … scaling horizontally rather than vertically,” he said. “It gave us the flexibility to bring a lot of products online that were not just brokerage in nature.”
Flush with its initial open-source success, ETrade’s IT organization was enthusiastic about expanding the model, setting its sights in 2004 on the BEA Tuxedo transaction management server.
“We’re happy to pay for software where it really helps us out, but, on the other hand, if we find an open-source alternative where we’re not paying for licensing, that’s the way we’ll go,” said Framke.
ETrade is about halfway through the development of the Tuxedo replacement, which is expected to save the company $2 million a year.
Coming off the open-source migration experience, Thompson was interested in examining the processes that produced the software hed just finished deploying so successfully.
In a test environment, Thompson installed Gentoo Linux, a distribution that exposes its users to the breakneck rate of change that open-source software undergoes.
Despite these massive changes, however, the system on which Thompson was running Gentoo was relatively stable. Knowing that managing change better and faster than the competition is a key to success, Thompson and Framke sought to find out how open sources methodology-its “secret sauce”-could be applied in ETrade’s own development team.
The secret, ETrade learned, is that open-source projects are organized modularly, with limited commit rights. ETrade is now charting a course for reorganizing the company’s internal development processes to match the way that open-source projects and Linux distributions are organized.
“As our code base grows, as the business gets more complex, it’ll become harder for us to achieve the same productivity and rate of change that we’ve enjoyed in the last few years,” said Framke.
“We want to break ETrade.com up into separate packages, to allow different packages to evolve at a different rate of change. This is not how we do software today at all-basically, the whole application moves forward.”
ETrade also is interested in open-sourcing some of its internally developed components. Framke and Thompson believe they have something to offer, stemming from ETrade’s early bets on SOA (service-oriented architecture).
“We’ve benefited in so many ways from what community processes have taught us,” said Thompson, “and I think its just natural that there might be something that we do that the open-source community may be interested in.”
Senior Analyst Jason Brooks is at jason_brooks@ziffdavis.com.