New Web Services Benchmark Under Fire

A new Web services application benchmark formula under development by an organization known more for its hardware and database benchmarks is giving some large software companies fits.

The Standard Performance Evaluation Corp., of Warrenton, Va., is expanding its reach into applications and crafting a new benchmark called AppPlatform. The benchmark, which software buyers would use to help price and sell products, will consider issues typical in customers applications, such as transactions, persistence services, Web services and messaging, said SPEC officials.

AppPlatform, which does not yet have a ship date, will also measure the scalability and performance of platforms, such as J2EE (Java 2 Platform, Enterprise Edition) and .Net, running Web services. When finished, it will be the first benchmark that tries to test both J2EE and .Net applications on an equal footing.

The effort, a first for the application side of the industry, has drawn many top-tier application developers.

Where does Web services stand in the enterprise? Click here to find out.

Not surprisingly, there is discontent over several issues. The main disagreement is over the inclusion of a price/performance metric. According to some involved in the SPEC process, such a metric would be unfair to products such as IBMs mainframe applications, which offer robust performance but are expensive.

But Microsoft Corp., also involved in the process, has proposed that the metric be included, as price/performance is one of the arguments the Redmond, Wash., company said it can win in the enterprise against competing mainframe, Unix and Linux systems.

Last June, IBM, of Armonk, N.Y., started its campaign to ban price/performance information in the benchmark.

SPECs AppPlatform subcommittee includes IBM, Microsoft, BEA Systems Inc., Sybase Inc., Intel Corp., Borland Software Corp. and Oracle Corp.

According to sources close to the SPEC steering committee vote, Intel, HP and Dell Inc. voted to remove the ban on price/performance data, while Sun Microsystems Inc., IBM, Advanced Micro Devices Inc. and Silicon Graphics Inc. voted to keep the ban. As a result, there will be no price/ performance information in the benchmark, just data on absolute performance.

IBM officials would not comment for this story. Industry experts said customers may suffer.

"Without the price/performance information, the benchmark will not give insight to the total cost of the performance," said Stephen Forte, chief technology officer at Corzen Inc., of New York. "Without the price/performance benchmark, as a CTO who truly cares about cost before performance, it is hard to understand what legitimate reason vendors could have for siding against the inclusion of price/performance information, other than trying to cloak the true cost of high-end systems."