srust blog

srust blog
Stephen Rust's Weblog

All | General

Tuesday June 14, 2005

InfiniBand support in OpenSolaris InfiniBand support in OpenSolaris

Sun has actually been working on InfiniBand for a number of years, and really has been involved since day one with the specification. A number of Sun engineers wrote parts of the IB spec, and indeed might have started InfiniBand in the first place.

Now that OpenSolaris is available, much of the IB code that Sun has been developing will now be available externally for the first time. There now exists the OpenIB effort, which provides IB support to the linux environment. The code from Sun in OpenSolaris was developed separately from the OpenIB effort, and it'll be interesting to see how the two projects interact in the future.

For OpenSolaris the kernel portions of the IB support live under: usr/src/uts/common/io/ib.

So we basically have the following breakdown of components in the Solaris IB world:

For those astute viewers, you may notice something missing, namely an HCA driver. For the time being, our HCA driver source and our uDAPL implementation (userland support) is restricted due to our 3rd party agreements. Stay tuned as this may change in the future.

You may want to checkout Nitin Hande for his talk on setting up some IPoIB interfaces. Also, Ted Kim has long been involved with things.

So, most of my effort has actually been involved in the HCA driver implementation, and it is a little disappointing to me that I cannot at this time actually trace-through any details since the source is not available.

But just generally, it may be interesting to take a look at: inc.flg. This file is a "Teamware" file that allows you to identify the list of directories or files to "bringover". So in this case, if you were to bringover usr/src/uts/common/io/ib, all files identified in the inc.flg would be updated. In the CVS sense, this might be similar to the concept of a module list -- that by specifying one module name, you can also have it bring over other modules as well. But it's interesting in this context, as this is basically the selection of files that are required in order to build in the IB kernel modules.

I completed the first putback to Solaris 10 of InfiniBand support nearly 2 years ago, on August 6th, 2003. This involved some 170 new files, and 21 updates of existing files.

If you have a piece of IB HCA hardware in your system, things will attach and work for you. And if you don't have the hardware, you won't even notice the drivers are there.

It's been a fun ride so far, we'll see where things bring us..
-srust

Technorati Tag: OpenSolaris
Technorati Tag: Solaris

( Jun 14 2005, 08:11:52 AM PDT ) Permalink Comments [10]

Monday June 06, 2005

sun is cool

My name is Stephen Rust, and I have been with Sun for just about 5 years, working through various platform software groups (various because of multiple re-orgs). My group has basically been responsible for platform specific kernel pieces in support of these platforms. I have personally written code used in the Sun Fire 280R, Sun Fire 480, Sun Fire V880 as far back as Solaris 8. Although most recently delivering the biggest chunk of my efforts to Solaris 10.

The biggest thing I've been working on all this time has been InfiniBand. Although this is not really platform specific, it is hardware specific, and is actually meant to be used in many kinds of platforms, SPARC and x86/amd64 (x64?) alike. I have been involved in one way or another with a large portion of the InfiniBand framework that is now in Solaris, while most of my development has been with the IB HCA driver for Solaris 10 and beyond. It's been a long, interesting road and hopefully we'll be seeing a lot of good things to come out of InfiniBand in the future.

Stay tuned,
Steve

( Jun 06 2005, 12:59:54 PM PDT ) Permalink Comments [2]