Close
  • Latest News
  • Artificial Intelligence
  • Video
  • Big Data and Analytics
  • Cloud
  • Networking
  • Cybersecurity
  • Applications
  • IT Management
  • Storage
  • Sponsored
  • Mobile
  • Small Business
  • Development
  • Database
  • Servers
  • Android
  • Apple
  • Innovation
  • Blogs
  • PC Hardware
  • Reviews
  • Search Engines
  • Virtualization
Read Down
Sign in
Close
Welcome!Log into your account
Forgot your password?
Read Down
Password recovery
Recover your password
Close
Search
Logo
Logo
  • Latest News
  • Artificial Intelligence
  • Video
  • Big Data and Analytics
  • Cloud
  • Networking
  • Cybersecurity
  • Applications
  • IT Management
  • Storage
  • Sponsored
  • Mobile
  • Small Business
  • Development
  • Database
  • Servers
  • Android
  • Apple
  • Innovation
  • Blogs
  • PC Hardware
  • Reviews
  • Search Engines
  • Virtualization
More
    Home Development
    • Development

    Benchmarks Wanted

    Written by

    Timothy Dyck
    Published May 12, 2003
    Share
    Facebook
    Twitter
    Linkedin

      eWEEK content and product recommendations are editorially independent. We may make money when you click on links to our partners. Learn More.

      “Men wanted for hazardous journey. Low wages, bitter cold, long hours of complete darkness. Safe return doubtful. Honour and recognition in event of success.”

      That famous advertisement for Antarctic exploration is attributed to Ernest Shackleton, but it could just as well appeal to those embarking on a benchmarking program. Ive spent my share of hours in cold, dim server rooms trying to warm my hands over the flicker of drive array LEDs. Benchmarking is very hard but, if successful, worthwhile. Thats why we do it, and its why Im glad to see a significant effort righting itself.

      Java training and consultancy company The Middleware Company first set sail for parts unknown in October when it published the results of a benchmark comparing two J2EE (Java 2 Platform, Enterprise Edition) application servers with Microsofts .Net 1.0 and .Net 1.1 release candidate run-times. When the results—with the .Net platform on top—came out, an uproar in the Java community ensued. The fact that Microsoft had funded the test didnt help.

      Now The Middleware Company has set out on a second effort designed to be more rigorous, accurate and fair than the first, which did many things right but also had notable problems.

      Two problems stood out. First, the .Net version did more data caching than the J2EE version. Second, the J2EE version was based on an old code base that still used bean-managed persistence rather than the more modern container-managed persistence.

      Last month, Salil Deshpande, president of The Middleware Company, contacted me with the specifics of his companys retest plan, which fixes both these technical problems and makes two major procedural improvements as well.

      First, this round is a collaborative process that incorporates feedback from a group of outside experts who have been working together on this since February. Second, the benchmark is now based on a formal specification that defines exactly how the test application needs to behave.

      “We created an expert group in February and invited these experts to review a functional spec,” said Deshpande. “We started with [Suns sample application] Pet Store but nailed down enough things—database tables, relationships, what data can be cached, for how long and where, what pages can look like. We fixed all the moving parts.”

      Page Two

      The new specification also requires something that I regard as the single most important component of a good benchmark—full disclosure. In fact, the benchmark Web site containing the specification, database schema, sample database data, static HTML, and several Java and .Net-based implementations will be made public at the same time as the publication of this column. Its at www.middleware-company.com/casestudy.

      To its credit, The Middleware Company published full details on its October test. Every benchmark, no matter how well or how poorly designed, models a very particular usage pattern and workload. Organizations with similar workloads will find the benchmark details and results highly valuable; organizations using different application architectures will know how to read the results in context. In both cases, full disclosure is key.

      The new specification is a functional specification, not an implementation specification, as is another major effort, SPECjAppServer. Because its specification includes J2EE code, it can be run only by J2EE application servers. The Middleware Companys effort defines exactly how the application needs to function as well as the back-end database design, but it doesnt specify the language used or other implementation details.

      Adopting a vendor- and platform-independent benchmark is something the entire application server industry has long needed to grow up and do.

      While I think the Transaction Processing Performance Council and its similarly platform-agnostic TPC-W test is the right long-term home for application server benchmarking efforts, The Middleware Companys specification provides a firm basis from which to move forward. Its certainly good enough to provide credible results already, and the company is planning on using the specification to carry out a J2EE-versus-.Net retest in the next few months.

      Iterative improvements based on public comment are how good benchmarking happens. Sail on—honor and recognition await!

      Read more Tech Directions columns:

      Timothy Dyck
      Timothy Dyck
      Timothy Dyck is a Senior Analyst with eWEEK Labs. He has been testing and reviewing application server, database and middleware products and technologies for eWEEK since 1996. Prior to joining eWEEK, he worked at the LAN and WAN network operations center for a large telecommunications firm, in operating systems and development tools technical marketing for a large software company and in the IT department at a government agency. He has an honors bachelors degree of mathematics in computer science from the University of Waterloo in Waterloo, Ontario, Canada, and a masters of arts degree in journalism from the University of Western Ontario in London, Ontario, Canada.

      Get the Free Newsletter!

      Subscribe to Daily Tech Insider for top news, trends & analysis

      Get the Free Newsletter!

      Subscribe to Daily Tech Insider for top news, trends & analysis

      MOST POPULAR ARTICLES

      Artificial Intelligence

      9 Best AI 3D Generators You Need...

      Sam Rinko - June 25, 2024 0
      AI 3D Generators are powerful tools for many different industries. Discover the best AI 3D Generators, and learn which is best for your specific use case.
      Read more
      Cloud

      RingCentral Expands Its Collaboration Platform

      Zeus Kerravala - November 22, 2023 0
      RingCentral adds AI-enabled contact center and hybrid event products to its suite of collaboration services.
      Read more
      Artificial Intelligence

      8 Best AI Data Analytics Software &...

      Aminu Abdullahi - January 18, 2024 0
      Learn the top AI data analytics software to use. Compare AI data analytics solutions & features to make the best choice for your business.
      Read more
      Latest News

      Zeus Kerravala on Networking: Multicloud, 5G, and...

      James Maguire - December 16, 2022 0
      I spoke with Zeus Kerravala, industry analyst at ZK Research, about the rapid changes in enterprise networking, as tech advances and digital transformation prompt...
      Read more
      Video

      Datadog President Amit Agarwal on Trends in...

      James Maguire - November 11, 2022 0
      I spoke with Amit Agarwal, President of Datadog, about infrastructure observability, from current trends to key challenges to the future of this rapidly growing...
      Read more
      Logo

      eWeek has the latest technology news and analysis, buying guides, and product reviews for IT professionals and technology buyers. The site’s focus is on innovative solutions and covering in-depth technical content. eWeek stays on the cutting edge of technology news and IT trends through interviews and expert analysis. Gain insight from top innovators and thought leaders in the fields of IT, business, enterprise software, startups, and more.

      Facebook
      Linkedin
      RSS
      Twitter
      Youtube

      Advertisers

      Advertise with TechnologyAdvice on eWeek and our other IT-focused platforms.

      Advertise with Us

      Menu

      • About eWeek
      • Subscribe to our Newsletter
      • Latest News

      Our Brands

      • Privacy Policy
      • Terms
      • About
      • Contact
      • Advertise
      • Sitemap
      • California – Do Not Sell My Information

      Property of TechnologyAdvice.
      © 2024 TechnologyAdvice. All Rights Reserved

      Advertiser Disclosure: Some of the products that appear on this site are from companies from which TechnologyAdvice receives compensation. This compensation may impact how and where products appear on this site including, for example, the order in which they appear. TechnologyAdvice does not include all companies or all types of products available in the marketplace.

      ×