Beating Server I/O Bottlenecks

As server processor speeds push the 2-GHz mark and enterprises look to Web-enable applications that require greater server bandwidth, network administrators are running into I/O bottlenecks created by the current Peripheral Component Interconnect (PCI) bus.

Enter InfiniBand. This emerging server I/O specification promises to widen the pipeline between clustered servers by replacing current 1GB/sec. PCI-X and Gigabit Ethernet technologies with an external, switched serial I/O fabric that supports data rates of 2.5G to 10G bit/sec. (and eventually 30G bit/sec.) in each direction.

Unlike the PCI busmaster design, InfiniBand will allow multiple concurrent links between devices, increasing throughput. And it will support high-speed interconnection with external target devices, such as RAID storage, up to one kilometer away. InfiniBand will also let network administrators separate servers from storage and other I/O devices in order to scale those resources separately over a low-latency, switched network spanning the data center.

Physically, InfiniBand is a simple parallel connection. The more copper wires (up to 48), the higher bandwidth you get, says John Enck, an analyst at Gartner Inc. in Stamford, Conn. [QuickStudy, October 23, 2000]. InfiniBand's initial design, similar in topology to Fibre Channel, will connect host channel adapters (HCA) in servers to devices with embedded target channel adapters by way of intermediate InfiniBand switches.

The technology will first appear in Intel-based server designs but should migrate into midrange and high-end systems. Vendors plan InfiniBand products that will link directly into a server's CPU and memory subsystems, but development of such servers is behind schedule, Enck says.

The first InfiniBand products, due later this year, will be HCAs that plug into existing PCI and PCI-X slots. They will allow multiple connections and separate scaling of resources, but gains will be limited until new designs arrive in 2003 or 2004.

Fitting In

How will InfiniBand affect the data center? It will likely extend, not replace, existing interconnects. "You'll have InfiniBand going out of servers [and] into a gateway device that will then come out Fibre Channel or Gigabit Ethernet or some other connectivity into the network, into the storage-area network or the network-attached storage," Enck says.

InfiniBand will likely be used in densely clustered blade-server racks, says Vernon Turner, an analyst at IDC in Framingham, Mass.

Some early adopters are already planning for InfiniBand. Nathan McQueen, streaming media systems architect at the University of Washington in Seattle, needs plenty of bandwidth as he prepares to open the institution's archives to other universities and the California state government. McQueen hopes to use five to 10 InfiniBand gateway switches to let his DB2 database and Web servers access shared Fibre Channel-attached storage. Within a year, he anticipates Internet sharing of 1 million to 2 million records per year.

McQueen says he believes he'll have easier configuration and greater speed and scalability with InfiniBand than with a Fibre Channel network.

Enck says InfiniBand will benefit applications such as McQueen's because it enables more powerful I/O connectivity for small-profile servers. "That's where the value proposition is strongest," he says.


Tech Specs

I/O Evolution in the Data Center

MAXIMUM I/O PERFORMANCE 500MB/sec. 1GB/sec. 500MB/sec. per two-wire link (250MB/sec. each way); up to 6GB/sec. on 12 links
LATENCY High High Very low
SCALABILITY Limited to available internal slots Limited to available internal slots Extends outside the server box; up to 64,000 devices per subnet
BEST APPLICATION Legacy PCI bus Faster version of the PCI parallel bus architecture High-speed serial bus architecture designed for external I/O connectivity

*Note: The recently announced PCI-x 2.0 specification will run at 2GB/sec.

Read accompanying stories:

Copyright © 2002 IDG Communications, Inc.

7 inconvenient truths about the hybrid work trend
Shop Tech Products at Amazon