ETrade Goes into the Great Wide Open

Case Study: Getting rid of its Sun Solaris infrastructure and moving open source up the stack saved ETrade millions of dollars and changed how the company approaches development.

ETrade Financial saved $13 million annually, realized a boost in performance and changed the way it thinks about development by moving from a Sun Solaris infrastructure built up during the stock market boom to an open-source stack running on x86-based IBM hardware.

Building on work it began in late 2001, the financial services company has moved its open-source migration up the stack and is now taking the lessons it has learned through its experience with open source to optimize internal development processes.

Those lessons will be critical as ETrade assimilates acquisitions and continues to grow to compete with rivals such as Ameritrade.

In December, ETrade bought BrownCo, the online trading arm of JP Morgan Chase, for $1.6 billion. In October, ETrade closed the purchase of Harrisdirect, another online broker, for $700 million.

On Jan. 23, ETrade reported that 2005 revenue was $1.7 billion, up from $1.4 billion in 2004. Net income was $430 million, up from $380 million a year ago.

"It was a win-win-win," said ETrade CIO Greg Framke, of the company's open-source experience. "You can work a whole career in technology and maybe get one of those, so it was very gratifying to have made that bet early and to have reaped those benefits very early, and it basically just fed our appetite for continuing down the open-source path."

In the late 90's, the dot-com bubble was rapidly expanding, with ETrade right in the center of it. The online company's trading volume and the areas in which it did business were expanding at a rapid pace, and so, too, were its hardware requirements.

ETrade, like many other financial services companies at the time, managed the growth by adding capacity in the form of large servers from Sun Microsystems running Sun's proprietary Solaris operating system.

Lee Thompson, ETrade's vice president of architecture, likened the rows of Sun servers to the aisles in a large supermarket: "You could go into an ETrade data center, and you could imagine yourself looking down the aisles of shelves," said Thompson in Austin, Texas. "Instead of food, you would see Sun Enterprise 4500s-three full rows of them."

During the fall of 2001, a set of circumstances prompted ETrade's IT organization to investigate a move from the Sun Solaris architecture that had been serving the company's crown-jewels application,, to a mostly open-source stack anchored by Linux.

One of the drivers, said Thompson, was a cooling stock market and its effect on ETrade's bottom line. "Trading volumes were above 300,000 average trades a day, and that dropped to something in the range of 55,000 a day in 2001," he said. "Our cash flow was affected, obviously, quite dramatically."

ETrade's engineers were aware of Linux as an emerging alternative to proprietary Unix operating systems. "At the time, I knew that [] had already moved over to Linux, and Yahoo was on a FreeBSD stack since its inception," said Thompson.

"And so I knew that, at some point, architecturally, an open-source operating system would be of some interest."

/zimages/6/28571.gifLinux joins the software establishment. Click here to read more.

Also prompting ETrade to move off the Sun servers was the addition of symmetric multiprocessing and a 32-bit message queue to the Linux 2.4.7 kernel that shipped with Red Hat Linux 7.2, the distribution that ETrade was testing.

Without those features, Linux could not serve as a suitable porting target for ETrade's applications, according to Thompson.

Last but not least was the public embrace of Linux by two industry powerhouses: IBM and Hewlett-Packard. With their support, said Framke, moving to Linux didn't mean that ETrade had to move out on its own.

"Fortunately for us, HP and IBM came out and said, We're going to stand behind Linux," said Framke in New York. "The second that happened, we made our Linux decision, and we rolled it out very aggressively, starting in winter 2001."

First, however, the IT organization had to get up and running on Linux.

Thompson assigned an architecture team from the IT organization to study the feasibility of porting's central components to a Linux and open-source stack.

As part of the study, the team replaced ETrade's iPlanet (now Sun) Web server and application server with open-source equivalents running on Red Hat Linux 7.2.

"We grabbed Apache, and we grabbed the Jakarta Tomcat servlet system, and we ported a representative stack of our application-our authentication, quote services, product services, some of our trading services and the servlets that rendered the HTML-over to this new stack," said Thompson.

The team also replaced the Solaris version of BEA Tuxedo with the Linux version of the transaction management system, sticking with Solaris for the Sybase database portion of the platform.

Tests showed that the Linux box could handle about 180 users at a time, compared with 300 to 400 users on one of ETrade's Sun 4500 servers. Performance on the Linux box dropped sharply when there were more than about 180 users, but, before that point, the Linux box performed faster than the Sun servers.

After the Linux box's price was factored in, ETrade had itself a new computing platform.

Next Page: The little machine that could.