ETrade Goes into the Great Wide Open

In the late 90's, the dot-com bubble was rapidly expanding, with ETrade right in the center of it. The online company's trading volume and the areas in which it did business were expanding at a rapid pace, and so, too, were its hardware requirements.

ETrade, like many other financial services companies at the time, managed the growth by adding capacity in the form of large servers from Sun Microsystems running Sun's proprietary Solaris operating system.

Lee Thompson, ETrade's vice president of architecture, likened the rows of Sun servers to the aisles in a large supermarket: "You could go into an ETrade data center, and you could imagine yourself looking down the aisles of shelves," said Thompson in Austin, Texas. "Instead of food, you would see Sun Enterprise 4500s—three full rows of them."

During the fall of 2001, a set of circumstances prompted ETrade's IT organization to investigate a move from the Sun Solaris architecture that had been serving the company's crown-jewels application, ETrade.com, to a mostly open-source stack anchored by Linux.

One of the drivers, said Thompson, was a cooling stock market and its effect on ETrade's bottom line. "Trading volumes were above 300,000 average trades a day, and that dropped to something in the range of 55,000 a day in 2001," he said. "Our cash flow was affected, obviously, quite dramatically."

ETrade's engineers were aware of Linux as an emerging alternative to proprietary Unix operating systems. "At the time, I knew that [Amazon.com] had already moved over to Linux, and Yahoo was on a FreeBSD stack since its inception," said Thompson.

"And so I knew that, at some point, architecturally, an open-source operating system would be of some interest."

Linux joins the software establishment. Click here to read more.

Also prompting ETrade to move off the Sun servers was the addition of symmetric multiprocessing and a 32-bit message queue to the Linux 2.4.7 kernel that shipped with Red Hat Linux 7.2, the distribution that ETrade was testing.

Without those features, Linux could not serve as a suitable porting target for ETrade's applications, according to Thompson.

Last but not least was the public embrace of Linux by two industry powerhouses: IBM and Hewlett-Packard. With their support, said Framke, moving to Linux didn't mean that ETrade had to move out on its own.

"Fortunately for us, HP and IBM came out and said, We're going to stand behind Linux," said Framke in New York. "The second that happened, we made our Linux decision, and we rolled it out very aggressively, starting in winter 2001."

First, however, the IT organization had to get ETrade.com up and running on Linux.

Thompson assigned an architecture team from the IT organization to study the feasibility of porting ETrade.com's central components to a Linux and open-source stack.

As part of the study, the team replaced ETrade's iPlanet (now Sun) Web server and application server with open-source equivalents running on Red Hat Linux 7.2.

"We grabbed Apache, and we grabbed the Jakarta Tomcat servlet system, and we ported a representative stack of our application—our authentication, quote services, product services, some of our trading services and the servlets that rendered the HTML—over to this new stack," said Thompson.

The team also replaced the Solaris version of BEA Tuxedo with the Linux version of the transaction management system, sticking with Solaris for the Sybase database portion of the platform.

Tests showed that the Linux box could handle about 180 users at a time, compared with 300 to 400 users on one of ETrade's Sun 4500 servers. Performance on the Linux box dropped sharply when there were more than about 180 users, but, before that point, the Linux box performed faster than the Sun servers.

After the Linux box's price was factored in, ETrade had itself a new computing platform.

Next Page: The little machine that could.