Devils and Details of Benchmark Tests

Eweek Labs web server benchmark, like most benchmark efforts, was an arduous series of trial-and-error events in which we used all our resources to squeeze as much performance as possible out of our Tux, Apache and IIS-based Web servers.

eWeek Labs analysts, like most technophiles, were intrigued and amazed by the extraordinary numbers that Tux 1.0 initially achieved more than a year ago on the SPECWeb 99 benchmark. We chose to use eTesting Labs WebBench benchmark platform to see if we could repeat the phenomenal performance numbers shown by the SPECWeb 99 benchmark at our Foster City, Calif., labs facility. (eTesting Labs is owned by Ziff Davis Media Inc., which also publishes eWeek.)

We used WebBenchs dynamic mix, which has roughly 30 percent dynamic content and 70 percent static content.

Unfortunately, the standard WebBench Apache dynamic module uses thoroughly outdated Common Gateway Interface code, which really didnt give us a fair apples-to-apples comparison with the in-process ISAPI (Internet Server API) module that was created for Windows 2000s IIS (Internet Information Server) Web server.

To make sure the platforms were running comparable workloads, we collaborated with performance engineers from Dell Computer Corp., as well as with engineers at Red Hat Inc. (the developer of Tux), Apache Software Foundation and Microsoft Corp., to create dynamic content modules for each of our Web server options (Tux alone, Tux with a dynamic Apache back end and IIS). The code is posted at www.eweek.com/links.

Although the modules we used produce the same HTML as the standard WebBench modules, they were developed independently of WebBench code, so our tests are not official WebBench results, nor should they be compared with other WebBench tests.

The most unexpected testing issue that came up was our inability to saturate the Tux-based Web server on our standard test server configuration because it was so fast, a problem that forced us to remove two processors from the four-way Dell PowerEdge 6400 server (which was equipped with two Gigabit NICs and 2GB of RAM). This was the only way the testbed of 80 workstations could max out the Tux server (see benchmark chart).

Throughout the benchmark tests, we made sure that client load and network throughput were not performance-limiting factors and that each test kept both server CPUs running at 100 percent of capacity.