Alternative Supercomputer Metrics Sought
Two alternative methods of benchmarking supercomputers are nearing completion, as the most popular metric for benchmarking performance is applied to generate a new list of the world's most powerful systems.Two alternative methods of benchmarking supercomputers are nearing completion, as the most popular metric for benchmarking performance is being applied Monday to generate a new list of the worlds most powerful systems. The latest revision of the Top500 list of supercomputers is expected to be released late Monday before the SC2004 supercomputing show kicks off in Pittsburgh. NASAs SGI-based "Columbia" supercomputer and the University of California/Lawrence Livermore Labs IBM BlueGene/L supercomputer are likely candidates to sit atop the rankings, although NECs recently announced SX-6s 65-teraflop performance could top both of them when NEC administers the Top500 benchmark, known as Linpack. The trouble with the list, critics say, is that it only tracks results from a single test, something that is usually considered to be an imprecise assessment of a systems performance. To address Linpacks limitations, two other initiatives have surfaced: the HPC Challenge benchmark, which has begun generating a significant sample of results, and ApexMap, whose code is now available for public download. The HPC Challenge data is currently in a "0.7 beta" stage; the final version should be released early in 2005, according to Jack Dongarra, the author of Linpack and one of the coordinators of the HPCC effort.
Other, lesser-known metrics are also available. The Top Application Performers list at Purdue University uses the Standard Performance Evaluation Corporations HPC2002 realistic application benchmark, but the list appears to not have been updated in some time. The Department of Defenses Advanced Research Project Agency (DARPA) Information Processing Technology Office is also involved in the creation of performance metrics, although the agency is working toward developing measurements of supercomputer deployment and execution, according to Bob Graybill, a program manager there.