How to Understand and Use Benchmarking

By Jennifer Sutherland and James Yaple

It’s 3 a.m. and you just can’t go to sleep. Your recommended server configuration is not providing the service levels that you promised. Customers are complaining about slow response times. Your manager wants to understand why you did not do your “due diligence.”

Thank goodness, this example (this time) was only an imagined fiasco. But this scenario is certainly real all too often. So, what are the industry standards for predicting server performance without actually having the servers delivered and installed? Answers that start with “The vendor told me” or “I read on the Web” just aren’t going to cut it. You need a proven, systematic approach that reduces risk and tilts the odds in favor of success—not just for you, but for your company.

What Does Benchmarking Do?

That’s where benchmarking comes in. Benchmarking measures the actual performance of an IT configuration, be it a storage subsystem, CPU or application software. Contrasted with modeling (which attempts to predict a result), benchmarking is the validation of a performance model or hypothesis. Like the state of Missouri’s motto, benchmark testing is the way to “show me” real results.  When managers are looking for actual numbers, benchmarking provides them.

Benchmarking, in general, establishes a point of comparison and is present in many daily activities. Benchmark comparisons can be subjective or objective, and qualitative or quantitative. Subjective and qualitative comparisons may generate statements such as “I don’t like sushi” to “Vendor A’s management software is easier to use.” Objective and quantitative comparisons are more along the lines of “The average home in Madison, Wis., is 109 percent of the cost of a similar home in Austin, Texas.