Close
  • Latest News
  • Artificial Intelligence
  • Big Data and Analytics
  • Cloud
  • Networking
  • Cybersecurity
  • Applications
  • IT Management
  • Storage
  • Sponsored
  • Mobile
  • Small Business
  • Development
  • Database
  • Servers
  • Android
  • Apple
  • Innovation
  • Blogs
  • PC Hardware
  • Reviews
  • Search Engines
  • Virtualization
Read Down
Sign in
Close
Welcome!Log into your account
Forgot your password?
Read Down
Password recovery
Recover your password
Close
Search
Logo
Logo
  • Latest News
  • Artificial Intelligence
  • Big Data and Analytics
  • Cloud
  • Networking
  • Cybersecurity
  • Applications
  • IT Management
  • Storage
  • Sponsored
  • Mobile
  • Small Business
  • Development
  • Database
  • Servers
  • Android
  • Apple
  • Innovation
  • Blogs
  • PC Hardware
  • Reviews
  • Search Engines
  • Virtualization
More
    Home Development
    • Development

    Where are the App Server Benchmarks?

    By
    Timothy Dyck
    -
    January 6, 2003
    Share
    Facebook
    Twitter
    Linkedin

      Standard benchmarks are one important way customers make buying decisions, and the J2EE market is an ideal level playing field for this effort. After all, where features are comparable, we need ways to discover what the best implementations are.

      The Transaction Processing Council has TPC-W, a transactional Web e-commerce benchmark. Its an online bookstore Web application, and like other TPC benchmarks, it has been carefully designed. Although the TPC-W specification is precise in terms of functional requirements, it provides a lot of freedom in the choice of technologies used in tests. However, the specification even states (in Clause 1.2.10): “The use of commercially available products is encouraged” for the application program layer.

      Yet, every one of the 14 posted TPC-W results (from several vendors, not just Microsoft Corp.) uses what Id call a benchmark special: Web application logic is written in C++ using Internet Server API DLLs. HTML is even hard-coded into C++ printf statements.

      This practice makes TPC-W a fine hardware and database benchmark but a useless application server benchmark. Theres no reason it couldnt be a good one if vendors would only have the guts to use their own application server products in their TPC-W benchmarks (which is how they would advise a real customer to deploy a Web storefront).

      Standard Performance Evaluation Corp.s application server benchmark is SpecjAppServer, a renamed, updated version of ECPerf benchmark, which was originally developed by the Java Community Process. SpecjAppServer is also a good benchmark but is specific to J2EE (Java 2 Enterprise Edition). SpecjAppServer is much narrower in focus than TPC-W because vendors must use the exact code Spec provides. This provides an excellent J2EE apples-to-apples test if application server vendors use comparable hardware, but it excludes non-J2EE vendors.

      That leaves us with a number of one-off efforts, the most recent of which is a Microsoft-.Net-versus-J2EE benchmark published by The Middleware Co. in October and based on the J2EE sample application Petstore. (Visit www.middleware-company.com/j2eedotnetbench to download the code and configurations and to read the book-length message threads on the benchmark and responses from the company.)

      Microsofts .Net Framework came out faster in the test, and message threads are filled with the usual personal attacks, as well as suggestions on how the Java code could have been made faster. (Two frequent suggestions are to use container-managed entity beans instead of bean-managed entity beans and to avoid Enterprise JavaBeans entirely.)

      I think this was a very worthwhile benchmark project, and I applaud the effort and tuning care taken by Middleware staff. Sure, there are things that could be improved on, and, hopefully, a later benchmark will do so. But this is still a respectable step forward, especially because J2EE vendors seem to be studiously avoiding benchmarks where they can be compared with competing application server technologies.

      Any benchmark (by anyone) for which source code and configuration files are published is a worthwhile benchmark in my view.

      Timothy Dyck
      Timothy Dyck is a Senior Analyst with eWEEK Labs. He has been testing and reviewing application server, database and middleware products and technologies for eWEEK since 1996. Prior to joining eWEEK, he worked at the LAN and WAN network operations center for a large telecommunications firm, in operating systems and development tools technical marketing for a large software company and in the IT department at a government agency. He has an honors bachelors degree of mathematics in computer science from the University of Waterloo in Waterloo, Ontario, Canada, and a masters of arts degree in journalism from the University of Western Ontario in London, Ontario, Canada.
      Get the Free Newsletter!
      Subscribe to Daily Tech Insider for top news, trends & analysis
      This email address is invalid.
      Get the Free Newsletter!
      Subscribe to Daily Tech Insider for top news, trends & analysis
      This email address is invalid.

      MOST POPULAR ARTICLES

      Latest News

      Zeus Kerravala on Networking: Multicloud, 5G, and...

      James Maguire - December 16, 2022 0
      I spoke with Zeus Kerravala, industry analyst at ZK Research, about the rapid changes in enterprise networking, as tech advances and digital transformation prompt...
      Read more
      Applications

      Datadog President Amit Agarwal on Trends in...

      James Maguire - November 11, 2022 0
      I spoke with Amit Agarwal, President of Datadog, about infrastructure observability, from current trends to key challenges to the future of this rapidly growing...
      Read more
      IT Management

      Intuit’s Nhung Ho on AI for the...

      James Maguire - May 13, 2022 0
      I spoke with Nhung Ho, Vice President of AI at Intuit, about adoption of AI in the small and medium-sized business market, and how...
      Read more
      Applications

      Kyndryl’s Nicolas Sekkaki on Handling AI and...

      James Maguire - November 9, 2022 0
      I spoke with Nicolas Sekkaki, Group Practice Leader for Applications, Data and AI at Kyndryl, about how companies can boost both their AI and...
      Read more
      Cloud

      IGEL CEO Jed Ayres on Edge and...

      James Maguire - June 14, 2022 0
      I spoke with Jed Ayres, CEO of IGEL, about the endpoint sector, and an open source OS for the cloud; we also spoke about...
      Read more
      Logo

      eWeek has the latest technology news and analysis, buying guides, and product reviews for IT professionals and technology buyers. The site’s focus is on innovative solutions and covering in-depth technical content. eWeek stays on the cutting edge of technology news and IT trends through interviews and expert analysis. Gain insight from top innovators and thought leaders in the fields of IT, business, enterprise software, startups, and more.

      Facebook
      Linkedin
      RSS
      Twitter
      Youtube

      Advertisers

      Advertise with TechnologyAdvice on eWeek and our other IT-focused platforms.

      Advertise with Us

      Menu

      • About eWeek
      • Subscribe to our Newsletter
      • Latest News

      Our Brands

      • Privacy Policy
      • Terms
      • About
      • Contact
      • Advertise
      • Sitemap
      • California – Do Not Sell My Information

      Property of TechnologyAdvice.
      © 2022 TechnologyAdvice. All Rights Reserved

      Advertiser Disclosure: Some of the products that appear on this site are from companies from which TechnologyAdvice receives compensation. This compensation may impact how and where products appear on this site including, for example, the order in which they appear. TechnologyAdvice does not include all companies or all types of products available in the marketplace.

      ×