Whats in a Benchmark?

Whats in a benchmark? Typically, its whatever the testing party darn well pleases. Benchmarks can be made to look like any other statistical data—you can manipulate things to say what you want them to.

Not so in the case of Microsofts .Net StockTrader application, which the software giant used to test the performance of a .Net-based application against a Java-based application from IBM.

Microsoft used its .Net StockTrader application to show that .Net and WFC (Windows Communication Foundation), along with Windows Server 2003, could compete with and even surpass Java-based transaction processing applications, while also demonstrating interoperability with different platforms, said Greg Leake, Microsofts technical marketing manager for .Net.

The benchmark compares .Net StockTrader against IBMs IBM Trade 6.1 performance application. Trade 6.1 is a J2EE (Java 2 Platform, Enterprise Edition)-based application IBM, of Armonk, N.Y., uses as a best-practice performance sample application and capacity testing tool for IBM WebSphere 6.1. The application is available for free and can be downloaded from the IBM WebSphere performance site.

Microsoft took Trade 6.1 and created software that is functionally equivalent to the Trade 6.1 application, both in terms of user functionality and middle-tier database access, transactional and messaging behavior, Leake said.

Resource Library:

The .Net StockTrader provides a blueprint for how companies can advance their business through Microsofts application platform technologies, including WCF, to create SOA (service-oriented architecture) solutions. Microsoft, of Redmond, Wash., officially released .Net StockTrader at the Securities Industry and Financial Markets Association, or SIFMA, technology conference in New York June 19. And because performance and agility are key concerns when building mission-critical applications, .Net StockTrader also includes instructions for performance testing across distributed environments.

The .Net StockTrader as well as the overall benchmark results are published on the MSDN (Microsoft Developer Network) as a best-practice .Net enterprise application. Leake said the .Net StockTrader is an SOA that uses WCF for its underlying remoting and messaging architecture. The user interface is an ASP.Net/C# Web application that is equivalent to the Trade 6.1 JSP (Java Server Pages) application. Moreover, the middle-tier services, written in C#, mirror the functionality and transactional characteristics of the back-end Trade 6.1 services that are based on J2EE and the IBM WebSphere application server.

The benchmark focuses on three core workloads: Web services, message-oriented transaction processing and data-driven Web applications with middle-tier transaction services and data access, Leake said.

Microsoft officials said the comparison is a fair one based on the design of the configurations.

"Since each application supports many different configurations, it is very important to understand what constitutes a fair comparison," said a Microsoft whitepaper describing the comparison. "First, the configurations compared must be equivalent. This means that the applications must produce the exact same functionality and processing behavior in the configurations compared. You cannot, for example, compare one application running with one-phase transactions between the message queue and the database, while running the other application with a two-phase commit across these distributed resources. The .NET StockTrader, while based on .NET and not J2EE, was designed to mirror most of the Trade 6.1 configurations possible with this testing goal in mind."

Also, because both the J2EE-based Trade 6.1 and .Net-based StockTrader applications expose their middle-tier services as industry-standard Web services, the two applications can be integrated with no code changes required, Leake said.

The JSP Trade 6.1 front-end application can use the .Net middle-tier services and messaging capabilities of the .Net StockTrader; and the ASP.Net StockTrader front-end application can use the EJB (Enterprise JavaBeans)-based WebSphere Trade 6.1 middle-tier services and messaging capabilities. This interoperability is possible with the .Net StockTrader since WCF, Microsofts new component remoting and distributed application programming model, is fundamentally based on open Web service standards, including SOAP (Simple Object Access Protocol), XML and the latest WS-* industry standards, he said.

In the Microsoft benchmark testing, the .Net solution outperformed WebSphere by as much as 285 percent, Leake said.

"We took Trade 6.1 and migrated it over to .Net, and we had two goals in mind," Leake said. "We wanted to migrate it around WCF and show how we can integrate it through Web services; and the second thing we wanted to do was to use this opportunity to see how WCF stacked up against WebSphere. We used the same hardware configuration—an AMD four-processor system."

What the test also showed is that Microsoft "can compete with Windows Server and .Net versus their [IBMs] current infrastructure," Leake said. "We can bring in Windows Server and get better throughput at a fraction of the cost."

In particular, he said Microsofts solution showed a significant price advantage over IBMs.

"We did a cost-per-transactions analysis, and we came in between one-half and one-fifth of the cost" of IBMs solution, Leake said.

John Rymer, an analyst with Forrester Research, said the Microsoft benchmark study should be taken seriously.

"The comparison confirms several of the pricing distinctions between WebSphere and Windows Server that we know exist, barring discounts and other sales tactics," Rymer said. "The descriptions of how both applications were both built and tested are complete."

He also noted that Leake knows both environments very well. "His test should not be interpreted as a reflection of Java/J2EE vs. .Net, but only of WebSphere V6 on a key sample app with a functionally equivalent app in .Net. I expect critics of the test to point out better ways of implementing IBM Trade 6.1. Given the wide availability of other frameworks for Java/J2EE, this is a fair comment."

Meanwhile, Leake said, "Wed love to see IBM come back and help us see how to better tune WebSphere."

As far as he is concerned, the whole scenario is a testament to Microsofts thrust around interoperability via Web services.

Again, I ask whats in a benchmark. Ask Leake, he wants to show you.

"All the code is published," he said. "And we show how to interoperate with non-Microsoft platforms, such as IBM mainframes and Linux systems."

Check out eWEEK.coms for the latest news, reviews and analysis in programming environments and developer tools.