The new specification also requires something that I regard as the single most important component of a good benchmarkfull disclosure. In fact, the benchmark Web site containing the specification, database schema, sample database data, static HTML, and several Java and .Net-based implementations will be made public at the same time as the publication of this column. Its at www.middleware-company.com/casestudy. To its credit, The Middleware Company published full details on its October test. Every benchmark, no matter how well or how poorly designed, models a very particular usage pattern and workload. Organizations with similar workloads will find the benchmark details and results highly valuable; organizations using different application architectures will know how to read the results in context. In both cases, full disclosure is key.Adopting a vendor- and platform-independent benchmark is something the entire application server industry has long needed to grow up and do. While I think the Transaction Processing Performance Council and its similarly platform-agnostic TPC-W test is the right long-term home for application server benchmarking efforts, The Middleware Companys specification provides a firm basis from which to move forward. Its certainly good enough to provide credible results already, and the company is planning on using the specification to carry out a J2EE-versus-.Net retest in the next few months. Iterative improvements based on public comment are how good benchmarking happens. Sail onhonor and recognition await! Read more Tech Directions columns:
The new specification is a functional specification, not an implementation specification, as is another major effort, SPECjAppServer. Because its specification includes J2EE code, it can be run only by J2EE application servers. The Middleware Companys effort defines exactly how the application needs to function as well as the back-end database design, but it doesnt specify the language used or other implementation details.