Close
  • Latest News
  • Artificial Intelligence
  • Big Data and Analytics
  • Cloud
  • Networking
  • Cybersecurity
  • Applications
  • IT Management
  • Storage
  • Sponsored
  • Mobile
  • Small Business
  • Development
  • Database
  • Servers
  • Android
  • Apple
  • Innovation
  • Blogs
  • PC Hardware
  • Reviews
  • Search Engines
  • Virtualization
Read Down
Sign in
Close
Welcome!Log into your account
Forgot your password?
Read Down
Password recovery
Recover your password
Close
Search
Logo
Logo
  • Latest News
  • Artificial Intelligence
  • Big Data and Analytics
  • Cloud
  • Networking
  • Cybersecurity
  • Applications
  • IT Management
  • Storage
  • Sponsored
  • Mobile
  • Small Business
  • Development
  • Database
  • Servers
  • Android
  • Apple
  • Innovation
  • Blogs
  • PC Hardware
  • Reviews
  • Search Engines
  • Virtualization
More
    Home Development
    • Development

    Benchmarks Wanted

    By
    Timothy Dyck
    -
    May 12, 2003
    Share
    Facebook
    Twitter
    Linkedin

      “Men wanted for hazardous journey. Low wages, bitter cold, long hours of complete darkness. Safe return doubtful. Honour and recognition in event of success.”

      That famous advertisement for Antarctic exploration is attributed to Ernest Shackleton, but it could just as well appeal to those embarking on a benchmarking program. Ive spent my share of hours in cold, dim server rooms trying to warm my hands over the flicker of drive array LEDs. Benchmarking is very hard but, if successful, worthwhile. Thats why we do it, and its why Im glad to see a significant effort righting itself.

      Java training and consultancy company The Middleware Company first set sail for parts unknown in October when it published the results of a benchmark comparing two J2EE (Java 2 Platform, Enterprise Edition) application servers with Microsofts .Net 1.0 and .Net 1.1 release candidate run-times. When the results—with the .Net platform on top—came out, an uproar in the Java community ensued. The fact that Microsoft had funded the test didnt help.

      Now The Middleware Company has set out on a second effort designed to be more rigorous, accurate and fair than the first, which did many things right but also had notable problems.

      Two problems stood out. First, the .Net version did more data caching than the J2EE version. Second, the J2EE version was based on an old code base that still used bean-managed persistence rather than the more modern container-managed persistence.

      Last month, Salil Deshpande, president of The Middleware Company, contacted me with the specifics of his companys retest plan, which fixes both these technical problems and makes two major procedural improvements as well.

      First, this round is a collaborative process that incorporates feedback from a group of outside experts who have been working together on this since February. Second, the benchmark is now based on a formal specification that defines exactly how the test application needs to behave.

      “We created an expert group in February and invited these experts to review a functional spec,” said Deshpande. “We started with [Suns sample application] Pet Store but nailed down enough things—database tables, relationships, what data can be cached, for how long and where, what pages can look like. We fixed all the moving parts.”

      Page Two

      The new specification also requires something that I regard as the single most important component of a good benchmark—full disclosure. In fact, the benchmark Web site containing the specification, database schema, sample database data, static HTML, and several Java and .Net-based implementations will be made public at the same time as the publication of this column. Its at www.middleware-company.com/casestudy.

      To its credit, The Middleware Company published full details on its October test. Every benchmark, no matter how well or how poorly designed, models a very particular usage pattern and workload. Organizations with similar workloads will find the benchmark details and results highly valuable; organizations using different application architectures will know how to read the results in context. In both cases, full disclosure is key.

      The new specification is a functional specification, not an implementation specification, as is another major effort, SPECjAppServer. Because its specification includes J2EE code, it can be run only by J2EE application servers. The Middleware Companys effort defines exactly how the application needs to function as well as the back-end database design, but it doesnt specify the language used or other implementation details.

      Adopting a vendor- and platform-independent benchmark is something the entire application server industry has long needed to grow up and do.

      While I think the Transaction Processing Performance Council and its similarly platform-agnostic TPC-W test is the right long-term home for application server benchmarking efforts, The Middleware Companys specification provides a firm basis from which to move forward. Its certainly good enough to provide credible results already, and the company is planning on using the specification to carry out a J2EE-versus-.Net retest in the next few months.

      Iterative improvements based on public comment are how good benchmarking happens. Sail on—honor and recognition await!

      Read more Tech Directions columns:

      Timothy Dyck
      Timothy Dyck is a Senior Analyst with eWEEK Labs. He has been testing and reviewing application server, database and middleware products and technologies for eWEEK since 1996. Prior to joining eWEEK, he worked at the LAN and WAN network operations center for a large telecommunications firm, in operating systems and development tools technical marketing for a large software company and in the IT department at a government agency. He has an honors bachelors degree of mathematics in computer science from the University of Waterloo in Waterloo, Ontario, Canada, and a masters of arts degree in journalism from the University of Western Ontario in London, Ontario, Canada.
      Get the Free Newsletter!
      Subscribe to Daily Tech Insider for top news, trends & analysis
      This email address is invalid.
      Get the Free Newsletter!
      Subscribe to Daily Tech Insider for top news, trends & analysis
      This email address is invalid.

      MOST POPULAR ARTICLES

      Latest News

      Zeus Kerravala on Networking: Multicloud, 5G, and...

      James Maguire - December 16, 2022 0
      I spoke with Zeus Kerravala, industry analyst at ZK Research, about the rapid changes in enterprise networking, as tech advances and digital transformation prompt...
      Read more
      Applications

      Datadog President Amit Agarwal on Trends in...

      James Maguire - November 11, 2022 0
      I spoke with Amit Agarwal, President of Datadog, about infrastructure observability, from current trends to key challenges to the future of this rapidly growing...
      Read more
      IT Management

      Intuit’s Nhung Ho on AI for the...

      James Maguire - May 13, 2022 0
      I spoke with Nhung Ho, Vice President of AI at Intuit, about adoption of AI in the small and medium-sized business market, and how...
      Read more
      Applications

      Kyndryl’s Nicolas Sekkaki on Handling AI and...

      James Maguire - November 9, 2022 0
      I spoke with Nicolas Sekkaki, Group Practice Leader for Applications, Data and AI at Kyndryl, about how companies can boost both their AI and...
      Read more
      Cloud

      IGEL CEO Jed Ayres on Edge and...

      James Maguire - June 14, 2022 0
      I spoke with Jed Ayres, CEO of IGEL, about the endpoint sector, and an open source OS for the cloud; we also spoke about...
      Read more
      Logo

      eWeek has the latest technology news and analysis, buying guides, and product reviews for IT professionals and technology buyers. The site’s focus is on innovative solutions and covering in-depth technical content. eWeek stays on the cutting edge of technology news and IT trends through interviews and expert analysis. Gain insight from top innovators and thought leaders in the fields of IT, business, enterprise software, startups, and more.

      Facebook
      Linkedin
      RSS
      Twitter
      Youtube

      Advertisers

      Advertise with TechnologyAdvice on eWeek and our other IT-focused platforms.

      Advertise with Us

      Menu

      • About eWeek
      • Subscribe to our Newsletter
      • Latest News

      Our Brands

      • Privacy Policy
      • Terms
      • About
      • Contact
      • Advertise
      • Sitemap
      • California – Do Not Sell My Information

      Property of TechnologyAdvice.
      © 2022 TechnologyAdvice. All Rights Reserved

      Advertiser Disclosure: Some of the products that appear on this site are from companies from which TechnologyAdvice receives compensation. This compensation may impact how and where products appear on this site including, for example, the order in which they appear. TechnologyAdvice does not include all companies or all types of products available in the marketplace.

      ×