A Serious Test
?"> But Barkai, whose functions include interacting with the HPC community, said the Linpack benchmark isnt considered to be a serious test by researchers, as the benchmark provides a well-organized series of equations to stress the floating-point unit. Real-world tests include integer functions, and often include seemingly random data. "Linpack is clearlyand Im being cynical herea tool for marketing people who like simple messages: 1, 2, 3," Barkai said. Rather, Barkai added, the benchmark is a simpler, more practical tool for funding agencies. "It boils down to a very concise message that can be passed on, that can make a case for funding," he said.The HPC Challenge benchmarks were created by Jack Dongarra, also the founder of the Linpack benchmark. Dongarra, a distinguished professor at the University of Tennessee and the director of the Center for Information Technology Research and the Innovative Computing Laboratory there, created the HPC Challenge numbers to add several new data points to the mix. The effort is funded by DARPA and the Department of Energy, among other agencies. Eight tests have been added to the benchmark already, which is scheduled for completion early next year, Dongarra said. "We need to move beyond one measurement," Dongarra said. "One measurement is one stake in the ground. We need to put in a round collection of things, to test more features in computers, to allow people to select the features that best match their application span." Each test is self-administered, according to very specific rules, Dongarra said. Each participant is allowed to submit scores from both an unoptimized "base" model as well as an optimized run, where some limited substitutions of code are allowed. The HPCC team then checks the final code and tests to make sure that the tests were run according to the rules. However, Dongarra said the accumulated tests will not be compiled into a single number, one that could rate machines a la the Top500 list. The number of benchmarks may remain fixed, although the HPCC organizers say its going to be tough to anticipate all of the new features that will be added to upcoming systems. Like all benchmarks, the most useful is data. Intel Itanium- and Xeon-based machines dominate the Top500 list, which Intel officials have said is evidence that the architecture scales well and is cheaper than so-called "proprietary" systems using processors designed by Sun Microsystems Inc. and Hewlett-Packard Co. Under the HPCC benchmark, however, machines from Cray Inc. and NEC Inc. play a much larger role. The submitted systems include the Army High Performance Computer Research Centers submission of a 1,024 0.6GHz Alpha 21164 cluster, although most submissions include 256 to 512 processors. Still, most of the tests include some gaps, evidence that the benchmark has evolved over time. NASAs Brooks said he plans to run the HPCC benchmarks and submit the results, although the tests will require him to squeeze in the benchmarks around his users. "We anticipate HPCC having a long lifetime," Dongarra said. "Linpack has had a long lifespan, almost 30 years. We want to make sure that what were doing now will exist for a long period of time, which is why were going slowly." Next Page: ApexMap emerges.
Even the maintainer of the Top500 list says the benchmarks utility is of limited use. "Linpack only measures a single spot on the high end of the performance scalethats why companies like it," said Erich Strohmaier, the maintainer of the list and a computer scientist in the Future Technologies Group of the Computational Research Division at Lawrence-Berkeley National Laboratory in Berkeley, Calif. "Customers have to be very careful when they use it, so that they dont expect the same measure of performance."