Tuesday June 14, 2005

InfiniBand support in OpenSolaris

Sun has actually been working on InfiniBand for a number of years, and really has been involved since day one with the specification. A number of Sun engineers wrote parts of the IB spec, and indeed might have started InfiniBand in the first place.

Now that OpenSolaris is available, much of the IB code that Sun has been developing will now be available externally for the first time. There now exists the OpenIB effort, which provides IB support to the linux environment. The code from Sun in OpenSolaris was developed separately from the OpenIB effort, and it'll be interesting to see how the two projects interact in the future.

For OpenSolaris the kernel portions of the IB support live under: usr/src/uts/common/io/ib.

So we basically have the following breakdown of components in the Solaris IB world:

• InfiniBand Transport Layer (ibtl)"
• InfiniBand Management Transport Framework (ibmf)"
• InfiniBand Communication Manager (ibcm)"
• InfiniBand Device Manager (ibdm)"
• InfiniBand Nexus Driver (ibnex)"
• IP over InfiniBand driver (ibd)"

For those astute viewers, you may notice something missing, namely an HCA driver. For the time being, our HCA driver source and our uDAPL implementation (userland support) is restricted due to our 3rd party agreements. Stay tuned as this may change in the future.

You may want to checkout Nitin Hande for his talk on setting up some IPoIB interfaces. Also, Ted Kim has long been involved with things.

So, most of my effort has actually been involved in the HCA driver implementation, and it is a little disappointing to me that I cannot at this time actually trace-through any details since the source is not available.

But just generally, it may be interesting to take a look at: inc.flg. This file is a "Teamware" file that allows you to identify the list of directories or files to "bringover". So in this case, if you were to bringover usr/src/uts/common/io/ib, all files identified in the inc.flg would be updated. In the CVS sense, this might be similar to the concept of a module list -- that by specifying one module name, you can also have it bring over other modules as well. But it's interesting in this context, as this is basically the selection of files that are required in order to build in the IB kernel modules.

I completed the first putback to Solaris 10 of InfiniBand support nearly 2 years ago, on August 6th, 2003. This involved some 170 new files, and 21 updates of existing files.

If you have a piece of IB HCA hardware in your system, things will attach and work for you. And if you don't have the hardware, you won't even notice the drivers are there.

It's been a fun ride so far, we'll see where things bring us..
-srust

Technorati Tag: OpenSolaris
Technorati Tag: Solaris