Cables, cables everywhere
Now that we've explained a little about the X6275 and QNEM leaf switch, how do we start to build up a simple, expandable, infiniband based compute cluster. Lets start small, and then grow over time. I'll only talk in terms of full racks, remember that a fully configured Sun Blade 6048 Chasis contains 48 of the X6275 blade. Each physical blade had two independent Infiniband ports, one per compute node.
The leaf switch (QNEM) attaches directly to all 24 of the X6275 blades. No cables for this connection, as compared to standard cabling and leaf switches. Then, each QNEM connects to the Sun Datacenter Infiniband Switch 648, using 8 of the 12x Infiniband
Interconnect System. Each of these cables is actually 3 cables in one, and uses only 1 connector at each end. (i.e. each cable is really 3 cables).
There are 4 QNEMs in each 6048 rack, so a setup like this only needs 32 cables (4 QNEMs X 8 cables per QNEM). On the Sun Datacenter Infiniband Switch 648 side, only 2 of the line cards are needed. Each line card in the 648 contains 24 connectors (of the 12x just mentioned). To get to all 32 required connectors just mentioned, a second line card is required. ( Roundup (# QNEM ports wired/24) )
So there you have
it, only 32 cables needed to have Infiniband to all 96 compute nodes in
a Sun Blade 6048 Chasis. Larger designs described in the future. This is 1/6 the cables that would be needed with conventional leaf switches and lower density datacenter IB switches. A total of 192 cables would be needed, 1 from each server to a leaf switch (96) and then from the leaf switch to the datacenter switch (96).
A 12X optical IB cable.
