Close
  • Latest News
  • Artificial Intelligence
  • Big Data and Analytics
  • Cloud
  • Networking
  • Cybersecurity
  • Applications
  • IT Management
  • Storage
  • Sponsored
  • Mobile
  • Small Business
  • Development
  • Database
  • Servers
  • Android
  • Apple
  • Innovation
  • Blogs
  • PC Hardware
  • Reviews
  • Search Engines
  • Virtualization
Read Down
Sign in
Close
Welcome!Log into your account
Forgot your password?
Read Down
Password recovery
Recover your password
Close
Search
Logo
Logo
  • Latest News
  • Artificial Intelligence
  • Big Data and Analytics
  • Cloud
  • Networking
  • Cybersecurity
  • Applications
  • IT Management
  • Storage
  • Sponsored
  • Mobile
  • Small Business
  • Development
  • Database
  • Servers
  • Android
  • Apple
  • Innovation
  • Blogs
  • PC Hardware
  • Reviews
  • Search Engines
  • Virtualization
More
    Home Development
    • Development

    There Aint No Business Like Benchmarking

    By
    Timothy Dyck
    -
    December 23, 2002
    Share
    Facebook
    Twitter
    Linkedin

      An old Net adage is that the best way to get a quick answer to your question is to post a wrong answer to it. Nothing draws the lurkers out like the chance to correct someone elses mistake.

      Posting benchmark results online are a prima facie example—on first glance, most people who feel that their platform investment or judgment decisions are being threatened can find something obvious that was missed.

      That was never more evident than with a J2EE vs. Microsoft .Net Framework benchmark that The Middleware Company (a J2EE training and consulting company) organized. The company also runs TheServerSide.com, a popular J2EE news portal and discussion site.

      The benchmark (documentation, scripts and code can be downloaded from http://www.middleware-company.com/j2eedotnetbench/) was based on the now infamous Petstore application that Sun originally developed as a teaching tool for J2EE technologies.

      Microsoft has used Petstore in the past to do similar comparisons, and has been legitimately criticized for using the Petstore code because the application was not designed for performance (although the .Net version certainly was).

      In this test, two J2EE experts from the Middleware Company did the J2EE tuning using two different (unnamed) J2EE application servers on both Windows and Linux to get Petstore up to snuff. They made huge improvements in the code: According to their paper, their version was roughly 17 times faster than the original Sun version when they were finished tuning it. But Windows 2000 and the .Net Framework still came out significantly faster as well as more stable in the test when running these particular versions of this application.

      There certainly was a measure of vitriol poured out by TheServerSide.com posters on this benchmark—a feeling of our own betraying us. However, in more than 130,000 words of postings (warning: that link will take a while to load), a number of good criticisms were also made about how the J2EE application could be made faster.

      Besides the carping—Java only comes into its stride on Solaris, or on machines with more than eight CPUs, and so on—there were good criticisms: The .Net and J2EE applications were run with different databases; the J2EE application used bean-managed entity bean persistence instead of container-managed entity bean persistence (many posters thought container management would be faster); and that entity beans should be avoided entirely in favor of a straight JDBC-based approach.

      Rickard Öberg, whose Java credentials look pretty strong to me, posted a detailed list of performance improvements that could be made to the Middleware Companys version of Petstore.

      These kinds of things are to be expected in any complex benchmark and dont make this effort invalid. I know what its like to spend 100-hour weeks in a benchmark lab tweaking head-aching combinations of code and settings to get the best possible performance out of every tested product.

      In my book, any benchmark whose configuration setting and code is published so others can build on the work is a good benchmark.

      The Middleware Company is considering a follow-on test incorporating these optimizations. Good for it. I hope it does so, and I dont think theres any problem with credibility here. As long as everything is made public, people can read the specs and learn their own lessons from the data.

      The best things about benchmarks arent even the numbers: full disclosure reports are a fascinating education in how to tune, in what little things can cause systems to fail, and in how to write performance-sensitive code. They are also a great source for packaged scripts and a methodology people can use in their own testing. I get a lot of mail from readers thanking us for posting our own benchmark scripts and configurations online.

      A larger lesson here is that J2EE vendors are doing their part to bring Petstore pain upon themselves. The main Java application server benchmark, SPECjAppServer2002, former known as ECPerf, cannot be run by non-J2EE vendors because the benchmark is defined using J2EE-specific code. What choice does Microsoft have but to use another application?

      If SPECjAppServer2002 were made more generic, or if application server vendors would start posting TPC-W numbers, then benchmark battles would be fought in a more neutral forum. Instead of saber rattling and angry denouncements, lets see some code and lets see some numbers.

      Our next eLABorations column will appear on Friday, Dec. 27. West Coast Technical Director Timothy Dyck can be reached at timothy_dyck@ziffdavis.com.

      Timothy Dyck
      Timothy Dyck is a Senior Analyst with eWEEK Labs. He has been testing and reviewing application server, database and middleware products and technologies for eWEEK since 1996. Prior to joining eWEEK, he worked at the LAN and WAN network operations center for a large telecommunications firm, in operating systems and development tools technical marketing for a large software company and in the IT department at a government agency. He has an honors bachelors degree of mathematics in computer science from the University of Waterloo in Waterloo, Ontario, Canada, and a masters of arts degree in journalism from the University of Western Ontario in London, Ontario, Canada.
      Get the Free Newsletter!
      Subscribe to Daily Tech Insider for top news, trends & analysis
      This email address is invalid.
      Get the Free Newsletter!
      Subscribe to Daily Tech Insider for top news, trends & analysis
      This email address is invalid.

      MOST POPULAR ARTICLES

      Latest News

      Zeus Kerravala on Networking: Multicloud, 5G, and...

      James Maguire - December 16, 2022 0
      I spoke with Zeus Kerravala, industry analyst at ZK Research, about the rapid changes in enterprise networking, as tech advances and digital transformation prompt...
      Read more
      Applications

      Datadog President Amit Agarwal on Trends in...

      James Maguire - November 11, 2022 0
      I spoke with Amit Agarwal, President of Datadog, about infrastructure observability, from current trends to key challenges to the future of this rapidly growing...
      Read more
      IT Management

      Intuit’s Nhung Ho on AI for the...

      James Maguire - May 13, 2022 0
      I spoke with Nhung Ho, Vice President of AI at Intuit, about adoption of AI in the small and medium-sized business market, and how...
      Read more
      Applications

      Kyndryl’s Nicolas Sekkaki on Handling AI and...

      James Maguire - November 9, 2022 0
      I spoke with Nicolas Sekkaki, Group Practice Leader for Applications, Data and AI at Kyndryl, about how companies can boost both their AI and...
      Read more
      Cloud

      IGEL CEO Jed Ayres on Edge and...

      James Maguire - June 14, 2022 0
      I spoke with Jed Ayres, CEO of IGEL, about the endpoint sector, and an open source OS for the cloud; we also spoke about...
      Read more
      Logo

      eWeek has the latest technology news and analysis, buying guides, and product reviews for IT professionals and technology buyers. The site’s focus is on innovative solutions and covering in-depth technical content. eWeek stays on the cutting edge of technology news and IT trends through interviews and expert analysis. Gain insight from top innovators and thought leaders in the fields of IT, business, enterprise software, startups, and more.

      Facebook
      Linkedin
      RSS
      Twitter
      Youtube

      Advertisers

      Advertise with TechnologyAdvice on eWeek and our other IT-focused platforms.

      Advertise with Us

      Menu

      • About eWeek
      • Subscribe to our Newsletter
      • Latest News

      Our Brands

      • Privacy Policy
      • Terms
      • About
      • Contact
      • Advertise
      • Sitemap
      • California – Do Not Sell My Information

      Property of TechnologyAdvice.
      © 2022 TechnologyAdvice. All Rights Reserved

      Advertiser Disclosure: Some of the products that appear on this site are from companies from which TechnologyAdvice receives compensation. This compensation may impact how and where products appear on this site including, for example, the order in which they appear. TechnologyAdvice does not include all companies or all types of products available in the marketplace.

      ×