Page Two

By Timothy Dyck  |  Posted 2003-05-12 Print this article Print

The new specification also requires something that I regard as the single most important component of a good benchmark—full disclosure. In fact, the benchmark Web site containing the specification, database schema, sample database data, static HTML, and several Java and .Net-based implementations will be made public at the same time as the publication of this column. Its at

To its credit, The Middleware Company published full details on its October test. Every benchmark, no matter how well or how poorly designed, models a very particular usage pattern and workload. Organizations with similar workloads will find the benchmark details and results highly valuable; organizations using different application architectures will know how to read the results in context. In both cases, full disclosure is key.

The new specification is a functional specification, not an implementation specification, as is another major effort, SPECjAppServer. Because its specification includes J2EE code, it can be run only by J2EE application servers. The Middleware Companys effort defines exactly how the application needs to function as well as the back-end database design, but it doesnt specify the language used or other implementation details.

Adopting a vendor- and platform-independent benchmark is something the entire application server industry has long needed to grow up and do.

While I think the Transaction Processing Performance Council and its similarly platform-agnostic TPC-W test is the right long-term home for application server benchmarking efforts, The Middleware Companys specification provides a firm basis from which to move forward. Its certainly good enough to provide credible results already, and the company is planning on using the specification to carry out a J2EE-versus-.Net retest in the next few months.

Iterative improvements based on public comment are how good benchmarking happens. Sail on—honor and recognition await!

Read more Tech Directions columns:

Timothy Dyck is a Senior Analyst with eWEEK Labs. He has been testing and reviewing application server, database and middleware products and technologies for eWEEK since 1996. Prior to joining eWEEK, he worked at the LAN and WAN network operations center for a large telecommunications firm, in operating systems and development tools technical marketing for a large software company and in the IT department at a government agency. He has an honors bachelors degree of mathematics in computer science from the University of Waterloo in Waterloo, Ontario, Canada, and a masters of arts degree in journalism from the University of Western Ontario in London, Ontario, Canada.

Submit a Comment

Loading Comments...
Manage your Newsletters: Login   Register My Newsletters

Rocket Fuel