A Blurred Benchmark

By John Taschek  |  Posted 2003-05-05 Print this article Print

Benchmarks could be useful for direct comparisons of hardware, but vendors almost always take them out of context.

AMDs Opteron microprocessor is by far the most interesting thing to happen in the server chip world in the last two years. It put Intel into reaction mode, forcing the company to change how it would optimize 32-bit applications on the Itanium 2 by introducing the IA-32 Execution Layer. The Opteron has also left Sun with a conundrum: an x86 processor not made by Intel that is faster than its vaunted UltraSPARC.

Unfortunately, AMD and a partner, RackSaver, have clouded the Opterons rollout with a benchmark that leaves many questions unanswered.

Benchmarks are useful tools. Theyre good for capacity planning, driver optimization and software tuning. They could be useful for direct comparisons of hardware, but the vendors involved almost always take publicly disclosed benchmarks out of context. We expect this from Intel, Sun, Oracle and

Microsoft. Now we can expect it from AMD and RackSaver, thanks to their benchmarks of a system using the Transaction Processing Performance Councils TPC-C benchmark.

The TPC-C requires that two performance numbers be released: the transactions per minute and the cost per transaction per minute, known as the price/performance score. Through RackSaver, AMD posted TPC-C results on the same day that Dell posted Xeon results.

On the two TPC-C scores, the RackSaver Opteron-powered system cranked out 82,226.46 tpm. A Dell Xeon system (not the latest generation) cranked out 78,116.87. The numbers show that the Opteron—a 64-bit processor—has performance that should be similar to the current-generation Xeons in 32-bit performance.

No company will be generating 80,000 tpm any time soon, especially on a single system. The London Stock Exchange is in the 24,000-tpm range now. Its ridiculous, too, to see Microsoft and HP tout their late-April 658,277-tpm score on a single unclustered system, but thats life in the benchmark lane.

The more important metric is the price per transaction per minute, and thats where things get murky.

The RackSaver price per tpmC is only $2.76—an amazing cost in the TPC world. Remember, only three years ago, the cheapest $/tpmC was about $110. Of course, tuning and configuration optimization have lowered the cost as much as improvements in processor and system design. RackSaver, however, is taking advantage of tuning, optimization, fast processors and pricing magic. The RackSaver system—a four-processor system sans attached storage—costs $49,000, not a bad price. Its about the same cost as the Dell Xeon system—but the Dell was not similarly configured.

Where the magic happens is with storage, typically the most expensive part of the machine. The total cost of RackSaver storage was $88,000, half the cost of the Dell PowerVault Disk Subsystem, even though the Dell and RackSaver systems had roughly comparable implementations.

If the Dell system had used the RackSaver storage array, the cost per tpmC would have been about $3.80, still significantly higher than the RackSaver but more reasonable.

The Opteron system, however, also benefited from whopping memory price differentials. A good $71,000 went to 32GB of memory on the Dell Xeon system. It appears that Dell and RackSaver believed they had to cache big chunks of the 500GB-plus database into RAM to get their scores. The Opteron server had the same amount of memory as the Dell Xeon server, but the RackSaver memory was included in the price of the system. So the memory for the Dell system alone costs more than the RackSaver server.

Finally, RackSaver used four dual-processor Athlon RackSaver clients for a cost of $7,628, while the Dell configuration used six single-processor clients for a cost of $23,208. Most of the Dell cost went to the SANBlade adapters. Its quite possible that four dual-processor Athlons with downgraded video options cost under $8,000, but, still, the Dell design costs $15,000 more for a client configuration that is substantially less powerful than RackSavers.

The Opteron will surely find its way into servers that deserve your attention. But until a fair apples-to-apples benchmark appears, you may never know how good it really is.

As the director of eWEEK Labs, John manages a staff that tests and analyzes a wide range of corporate technology products. He has been instrumental in expanding eWEEK Labs' analyses into actual user environments, and has continually engineered the Labs for accurate portrayal of true enterprise infrastructures. John also writes eWEEK's 'Wide Angle' column, which challenges readers interested in enterprise products and strategies to reconsider old assumptions and think about existing IT problems in new ways. Prior to his tenure at eWEEK, which started in 1994, Taschek headed up the performance testing lab at PC/Computing magazine (now called Smart Business). Taschek got his start in IT in Washington D.C., holding various technical positions at the National Alliance of Business and the Department of Housing and Urban Development. There, he and his colleagues assisted the government office with integrating the Windows desktop operating system with HUD's legacy mainframe and mid-range servers.

Submit a Comment

Loading Comments...
Manage your Newsletters: Login   Register My Newsletters

Rocket Fuel