Cray to build supercomputer with new AMD chips

Cray to use AMD's new 16-core chip to build an 11.5 petaflop supercomputer for science

Cray Inc. is building a supercomputer for federally funded scientific research under a contract valued at $188 million that was originally won by IBM.

This system, which will be capable of a peak performance of 11.5 petaflops when it is completed next year, will be built with 49,000 processors from Advanced Micro Devices, Cray said in its announcement Monday. A petaflop equals one quadrillion floating point operations per second.

The AMD chips that will be used in this system were released Monday as the new 6200 and 4200 series Opteron processors, formerly code-named Interlagos.

Cray and AMD weren't originally part of the supercomputing project called "Blue Waters." In August, IBM ended a four-year contract to build a petaflop-scale supercomputer at the University of Illinois' National Center for Supercomputing Applications (NCSA) over the project's scope and its cost. The termination was announced jointly after both sides said they "could not come to a mutually agreed-on plan."

This left the NCSA without a computer for its 88,000 square-foot data center as well as for its 25 scientific teams building applications to scale across tens of thousands of compute cores.

After the termination of the IBM contract, Bill Kramer, deputy director of the Blue Waters project, said "we went into full mobilization mode throughout the entire university" to find a replacement.

The National Science Foundation, the project's major funder, approved a new hardware plan last week, Kramer said.

Cray was hired, in part, because it could meet the original schedule to have a system running in 2012 and do it within the initial overall project budget of $208 million. Cray is deploying about 300 of its XE6 cabinets, which will run the AMD processors and a future version of its XK6 supercomputer with Nvidia's Tesla GPU chips.

"This is going to be the largest x86 system in the world," said Barry Bolding, vice president of storage and data management at Cray.

In October, Cray was awarded a $97 million contract from the U.S. Department of Energy's Oak Ridge National Laboratory to build a system, also using AMD chips and Nvidia GPUs, that may eventually deliver up to 20 petaflops.

The system will be installed in phases, with an initial deployment as early as January. It is expected to be completed by the middle of next year.

The high-performance computer (HPC) market has always been important to chip makers, but especially AMD. When it released Opteron in 2003, the first 64-bit x86 compatible server chip, some of its initial big sales were to HPC users.

This latest release increases the number of processor cores from 12 to 16.

Nathan Brookwood, principal analyst at Insight 64, said AMD's chip offers more performance in the same socket, which means existing users can upgrade, but at the same power needs.

"This means that data centers that are constrained in terms of space or power can accommodate increased workloads within the same space and power envelopes," Brookwood said.

Another improvement is to the chip's computation capability. The chip's floating point, or numerically intensive performance, "has been greatly enhanced," from an already strong position, "due to a new arrangement that allows two integer cores to share one beefed-up floating point unit," Brookwood said.

If only one of the two integer cores needs the floating point unit, it can get more than twice the performance of what it could achieve with the older design, Brookwood said. That older design "basically married each integer core with a dedicated floating point unit that was less capable than the new enhanced model."

Some users may even choose to disable half the integer cores in order to ensure that each remaining core gets the full benefit of the bulkier floating point unit, Brookwood said.

The NCSA system will be focusing on floating point performance.

Bolding said the chips have 16 integer cores, but what they anticipate is that the applications will be using one of the integer cores but both of the floating point units, so for the predominate workload the chip is an essentially eight-integer core processor.

The chip runs up to 3.3 GHz but can reach 3.7 GHz, with a feature that AMD calls "Turbo Boost," which can take power that might have been allocated for a core that might not be completely busy and apply it to a core that needs a boost.

The Blue Waters computer will be in a data center than has 30,000 square feet of raised floor space. The data center has enough power to support 24 MW, and the system will be liquid-cooled. The facility will include cooling towers that will allow it to use naturally cooled water, instead of mechanical cooling, when conditions are right.

Larger supercomputers allow scientists to increase the resolution of their problem solving. They can assemble ever larger numbers of atoms together to model how they perform, and increase their ability to improve such things as weather forecasting.

The Blue Waters system will be used to study a wide range of science, from astrophysics, earthquake behavior, climate analysis and how diseases and computer viruses spread.

"We expect it do a tremendous amount for science," Kramer said.

Patrick Thibodeau covers SaaS and enterprise applications, outsourcing, government IT policies, data centers and IT workforce issues for Computerworld. Follow Patrick on Twitter at @DCgov or subscribe to Patrick's RSS feed . His e-mail address is

Copyright © 2011 IDG Communications, Inc.

7 inconvenient truths about the hybrid work trend
Shop Tech Products at Amazon