What is IT Benchmarking?

IT benchmarking is the process of using standardized software, representing a known workload, to evaluate system performance. Benchmarks are designed to represent customer workloads, such as database or Web applications. They enable a variety of hardware and software configurations to be compared. Many benchmarks are integrated with cost components, so price and performance can be evaluated.

Performance benchmarks can be likened to government mileage estimates for automobiles. Actual performance in a customer environment with a customer workload will be different. Just because a particular database benchmark says a configuration can support 5,000 concurrent users or 8,000 transactions per second, does not mean that it is what a customer will experience with their own configuration. Some planners consider it a rule of thumb that actual results are unlikely to exceed published results. The major components of a benchmark are:

1) a workload, with associated metric(s)

2) a set of conditions, commonly called “run rules”

3) reporting requirements

Predict Performance with Benchmarking

For performance analysts and capacity planners, benchmarks can enhance the ability to estimate system hardware requirements and predict performance.  Commercial capacity planning software base the what-if analysis of performance scenarios on published benchmark results. 

The number of possible benchmarks is only limited by the imagination, but they fall into three general categories:

1) industry-standard

2) vendor-oriented

3) customer-sponsored or internal benchmarking

Industry-Standard Benchmarking

ISBs (industry-standard benchmarks) are developed, maintained and regulated by independent organizations. These benchmarks typically execute on a wide variety of hardware and software combinations. The most well-known ISB organizations are the SPEC (Standard Performance Evaluation Corporation) and the TPC (Transaction Processing Council).

Typically, hardware and software vendors are heavily represented in the membership of these organizations. The groups solicit input from members and the IT community when benchmarks are created and updated to reflect changes in the marketplace. Some common ISBs are:

• TPC-C, representing a database transaction workload

• SPEC jAppServer, representing a multitier, Java 2 Platform, Enterprise Edition application server workload

• SPEC CPU2006, representing CPU-intensive workloads

• SPC (Storage Performance Council), representing storage-intensive workloads

Often, benchmark organizations require license fees or membership dues to provide benchmark software. Corporate Data Center Operations  joined SPEC and SPC several years ago for access to the benchmark software.