The White House view could help accelerate acceptance of other metrics that can determine system performance.
Alternative to the Linpack tests are emerging. For instance, an international team led by Sandia National Laboratories released a new rating system, Graph 500, in November.
Richard Murphy, a researcher at Sandia who helped develop Graph 500, said the benchmark is complementary to the one used by the Top500 operation.
"Top500 ranks compute intensive applications, and the point of Graph 500 is to rank data intensive applications, which are a major new scientific and engineering research challenge," said Murphy said in an e-mail response to questions from Computerworld.
Business areas that the Graph 500 metric could benefit include cybersecurity, data enrichment, medical informatics and social networks, said Murphy.
The Graph 500 benchmark had nine submission to the first list released in November, "which is about what we were hoping for," said Murphy.
"There were probably another 8-10 groups that inquired about submitting but didn't do so because they didn't know how to baseline the performance numbers they got. Basically, because nobody had put a performance stake in the ground, they didn't know if they performed well or not and didn't want to risk submitting," he added.
Graph 500 measures both performance and problem size, which means a smaller HPC system could outperform a larger one, but only to solve a smaller problem in many cases.
Jack Dongarra, a professor of computer science at the University of Tennessee and a distinguished research staff member at Oak Ridge National Laboratory and one of the international researchers who maintains the Top500 list, said he agrees with the White House report. A benchmark he helped developed, the High Performance Computing Challenge (HPCC), tries to address this problem by completing seven performance tests to measure multiple aspects of a system. HPCC was originally developed for DARPA.
"The benchmark that goes into the generating the Top 500 is only one number measuring only one aspect of a system," said Dongarra, in response to email questions. "The Graph 500 will have the same problem. HPCC attempts to provide many more numbers and test points," he said.
Patrick Thibodeau covers SaaS and enterprise applications, outsourcing, government IT policies, data centers and IT workforce issues for Computerworld. Follow Patrick on Twitter at 
@DCgov, or subscribe to Patrick's RSS feed 
. His e-mail address is pthibodeau@computerworld.com.
Read more about High Performance Computing in Computerworld's High Performance Computing Topic Center.
Free Insider GuideCopyright © 1994 - 2013 Computerworld Inc. All rights reserved. Reproduction in whole or in part in any form or medium without express written permission of Computerworld Inc. is prohibited. Computerworld and Computerworld.com and the respective logos are trademarks of International Data Group Inc.
