Ted H. Kim's Weblog

Friday June 11, 2004

So what is InfiniBand?

InfiniBand (IB) is a high speed interconnect defined by the InfniniBand Trade Association. The links are point to point (switched), bi-directional using 2.5 Gb signalling rate and 8b/10b encoding. So the basic "1x" cable is capable of 250MB/sec in both directions simultaneously. The other bundle sizes are "4x" (1GB/sec), "12x" (3GB/sec). The next release (1.2) will also add an "8x" bundle and allow for the signalling speed to be doubled (DDR) or quadrupled (QDR). The links can be implemented in a printed circuit board (20+ inches), copper (10 meters), fiber (up to 10 kilometers).

You asemble an IB subnet by connecting up your links to adapter ports and switches (you can have 48K total ports and switches in a subnet). The topology can be arbitrary though obviously some arrangements (star, fat tree, ring, etc.) are more interesting than others.

IB host adapters (called Host Channel Adapters (HCAs)have a protocol processing engine in them that implements a hardware queue (actually a pair of them) to accept commands for each communication endpoint in the adapter. Further, there are other queues which notify the adapter user when commands are done. Queues are used so the commands can be performed asynchronously by the hardware without a need to wait around for the operations to complete. The commands themselves send and receive messages or perform RDMA (more on that at a later time).

There is lots more in the 2000 odd pages of specifications, including: module form factors, hardware and power management, virtual lanes, different transport services, multicast, subnet management, routers, and so on. Maybe we will go into some of that, but this is enough for a starting point.

Technorati Tags: InfiniBand

(2004-06-11 11:40:08.0) Permalink Comments [1]