Benchmarks Challenged

So now CEOs have to sign off on their companies financial statements—on pain of imprisonment. I wish industry marketers were held to the same standard.

The latest "benchmarketing" flap concerns Intel and AMD. According to AMD, BAPCo, a nonprofit consortium, has unfairly stacked the deck in Intels favor.

BAPCo was founded by vendors and publishing companies, including Intel, but not AMD. BAPCos flagship Sysmark test is used by many companies to make buying decisions. Sysmark emulates how a computer is really used by timing common tasks. But the 2002 version, said AMD, has been unfairly redesigned to favor Intels Pentium 4 CPU.

AMD identified 14 tests that were removed from the latest version of the benchmark—each of which favored AMDs Athlon processor. AMD also highlighted a number of new tasks—all of which favored the Pentium 4. In Excel, according to AMD, the Pentium 4-friendly sort function now constitutes more than 90 percent of the test. That seems a tad high. Whether Intel unduly influenced BAPCo, or its benchmark toadies screwed up, I smell a rat. AMD has taken the high road and has joined BAPCo and will address the problem from within.

Its not just AMD vs. Intel. Rumors abound in the Valley about unethical behavior in the high-end 3-D graphics market, and database vendors are often charged with manipulating results so everyone ends up a winner.

Ethical lapses doomed Enron and WorldCom, and now corporate chieftains are only slightly more popular than cockroaches and Bud Selig. Buyers used to trust benchmarks, but these shenanigans may well taint the entire industry. Its too bad because I had thought wed be above the fray. Unfortunately, lying to potential customers isnt a crime. Yet.

Whom do you trust? Jim Louderback, editor in chief of Ziff Davis Internet, can be reached at jim_louderback@ziffdavis.com.