Ted H. Kim's Weblog

Friday October 09, 2009

InfiniBand on OpenStorage, etc.

Some exciting news: InfiniBand has been added to the OpenStorage offerings starting with the Q3.2009. Check out Cindi's blog here.

Of course, there have improvements for InfiniBand support in Solaris 10 10/09 release migrating over from the stuff in Nevada.

I have been told the X4239A DDR FEM HCA is really for the X6450 blade.

And there have been more Sun QDR switch models now too.

And of course, if what I say is contradicted elsewhere by official stuff, I am probably wrong!

Technorati Tags: InfiniBand

(2009-10-09 16:58:32.0) Permalink Comments [2]

Friday May 29, 2009

Solaris/IB and Open Source

OpenSolaris is known for being open source. In the InfiniBand area, the framework and most Upper Level Protocols (ULPs) were open source right from the start of OpenSolaris. However, we had to keep the Host Channel Adapter (HCA) drivers in closed source because of prior legal agreements. Well this took a while to sort out, and then finally make the changes to the source tree.

So I am happy to announce that the HCA drivers are now open source. You can look at them here.

There is still part of one ULP (Sockets Direct Protocol) which is in closed source, but that is the only remaining Solaris/IB item in closed source. Unfortunately, it is not likely to change any time soon, because of the way SDP was licensed. For similar reasons, even Linux/OFED has some problems in this area.

Technorati Tags: InfiniBand

(2009-05-29 10:54:31.0) Permalink

Wednesday May 27, 2009

SRP target

As noted here, the SRP target for COMSTAR just went back to Solaris Nevada, which is of course the base for future OpenSolaris releases. So now, Solaris can be an SRP target as well as an iSER target or an NFS/RDMA server. So you can get your blocks and files through all the popular flavors of RDMA protocols on IB.

Technorati Tags: InfiniBand

(2009-05-27 14:25:31.0) Permalink

Thursday May 14, 2009

Java can use SDP

Here is an item that was recently pointed out to me. You can now configure Java to use Sockets Direct Protocol (SDP) on InfiniBand, see this article.

Technorati Tags: InfiniBand

(2009-05-14 14:37:16.0) Permalink

Monday May 11, 2009

DDR FEM

Along with all the other Nehalem announcements related to IB was one other IB related item which I haven't talked about yet. That's the X4239A DDR Fabric Expansion Module (FEM). It's an HCA with PCIe 2.0 I/O interfaces and dual DDR 4x IB ports for connection to the in-chasis IB fabrics.

Now the x6275 blade has a QDR HCAs built in. So what would you do with this item? After looking here, it appears that you use it with the x6270 blade to connect them to the QNEM.

Technorati Tags: InfiniBand

(2009-05-11 15:55:24.0) Permalink

Friday May 01, 2009

ConnectX driver in Solaris 10

With the release of Solaris 10 5/09 (aka Solaris 10 Update 7), the ConnectX HCA driver (called "hermon") finally makes it's appearance in Solaris 10. It has been available in OpenSolaris since Nevada build 100. But now it finally makes its way to the Solaris 10. Download Solaris 10 5/09 here. And of course it's free.

Technorati Tags: InfiniBand

(2009-05-01 13:58:28.0) Permalink

Tuesday April 14, 2009

iSER

As other folks have mentioned, iSCSI Extensions for RDMA (iSER) was recently integrated into build 112 of OpenSolaris. Congratulations to the iSER team! The general background on iSER is here with the Solaris architecture specifics here. But the general idea is iSCSI with InfiniBand RDMA to drive up performance.

Technorati Tags: InfiniBand

(2009-04-14 15:25:50.0) Permalink

IB QDR and Nehalem (updated)

In general, InfiniBand DDR HCAs were already limited by the IO bus under PCIe 1.X. With today's Nehalem (Xeon 5500 series) related announcements come a bunch of PCIe 2.0 servers (including x2270, x4170, x4270, x4275 servers plus x6270 and x6275 blades). Now x4 DDR should be able to operate at full speed with a PCIe 2.0 x8 slot and x4 QDR is now realistic (though probably still IO bus limited). The x6275 blade even has on-board QDR HCAs installed. For some great shots of the boards, look here.

So in addition to earlier HCA offerings, we now have this QDR HCA story:

Sun Option	Form Factor	Chip	IO Bus	IB Ports (per HCA)	Card Memory	Solaris Driver
X4237A	Card	ConnectX-IB	PCIe 2.0 x8	Two x4 QDR	Memfree	hermon
X4219A-Z	Express Module

Edit: I think I might finally understand this now! The X5500A SunBlade 6048 QDR NEM has switches, but no HCAs. This design is different from how the DDR NEM works, which has the HCAs in it too. Instead, the QDR NEM is meant to be used with on-board HCAs like with the x6275 blade.

But both IB NEMs for the SunBlade 6048 are meant to be used as part of an overall Sun Constellation system with the final stage of the fat-tree (or torus for the QDR one) switch network incorporated directly in the NEM itself.

And of course, if there is any contradiction, I am sure the official stuff takes precedence over anything I say.

Technorati Tags: InfiniBand

(2009-04-14 14:02:21.0) Permalink Comments [1]

Friday February 06, 2009

IB Updates 3 (OFUV UD)

Finally a timely announcement! Yesterday, Solaris InfiniBand Updates 3 was released. The big addition over the previous Updates 2.1 is the first increment of Open Fabrics User Verbs or OFUV.

What's OFUV you ask? Well, here is the story. Quite a while ago, we delivered uDAPL into Solaris. But over time, with the growing popularity of the Linux OFED InfiniBand releases, OFUV became the pre-dominant OS Bypass InfiniBand framework.

Now the question which comes up from time to time is, "why don't you just port OFED?" The simplified explanation is that it's quite hard to do a straight port for the kernel parts, unless you can completely emulate Linux kernel internals. So instead for the kernel-based InfiniBand services, we are trying to provide equivalent stuff but not necessarily a straight port. But on the userland side of things, it's much more practical to do a port.

So with this new software, we have done a port of userland OFUV APIs on Solaris. Now it's a big project, so this first increment has the Unreliable Datagram (UD) support (based on OFED 1.3 verbs and rdma_cm) and associated drivers (for hermon and tavor), utilities, etc. While the verbs exist for reliable connected (RC) support, the rdma_cm support is not yet in there. Following phases will round out the offering with rdma_cm support for RC and other stuff. And of course, sometime we have to update to OFED 1.4. But finally we have the first part.

So what do you do with userland UD? Lately, it has been used to do high volume, low latency messaging systems on top of InfiniBand, especially in financial services. So stay tuned ...

Technorati Tags: InfiniBand

(2009-02-06 16:56:40.0) Permalink Comments [4]

Wednesday January 28, 2009

Catching up on IB news

Last month, I failed to note that IB Updates 2.1 was released. This rev basically rolls IB Updates 2 forward to Solaris 10 10/2008 (Update 6). More discussion at this blog entry.

Technorati Tags: InfiniBand

(2009-01-28 16:31:37.0) Permalink

Friday October 03, 2008

ConnectX in Nevada

Some other folks have already mentioned this, but for completeness, I will point it out too. The ConnectX HCA driver has gone back to the Nevada source tree. So expect it to eventually come out through OpenSolaris. Plus this is a step towards getting it into the Solaris 10 Updates too.

The integration message is here.

Technorati Tags: InfiniBand

(2008-10-03 15:47:13.0) Permalink

Monday August 25, 2008

Updated NFS/RDMA

Just recently, the Updated NFS/RDMA integrated into Nevada. The history of NFS/RDMA goes back years in Solaris, when Sun originally put in a pre-standard version into Solaris. More recently, work in the IETF sought to standardize how NFS over RDMA would work (the drafts are now moving towards RFC status). The integration now gives Nevada an implementation that corresponds with the new IETF version (replacing the old pre-standard version).

In practical terms, saying over RDMA is a code for InfiniBand and the slowly emerging iWARP. So what the integration into Nevada means is Solaris will now have the updated IETF NFS/RDMA version for InfiniBand with the door open for iWARP when that comes along. And of course, Nevada is the codeword for propagating into the OpenSolaris releases.

Just to recap where we are on InfiniBand ULPs, Solaris now has implementations of IPonIB, SDP, RDS, NFS, SRP, uDAPL (used for MPI). The HCA story is here and here. And more is coming, including a port of the Open Fabrics Userland Verbs, iSER and certification on various SPARC platforms. So stay tuned for exciting days ahead...

Technorati Tags: InfiniBand

(2008-08-25 11:19:55.0) Permalink

Monday July 21, 2008

IB Updates 2

Last week we released Solaris InfiniBand Updates 2 through the Sun Downloads site. This replaces the earlier Solaris InfiniBand Updates 1. The new version supports the Mellanox ConnectX HCA as well as the cards supported in the earlier version. The new update is based on Solaris 10 5/08, which is the latest update of Solaris 10. Support for flashing the HCA firmware and for uDAPL (used by Sun's ClusterTools MPI) is also included. Sun options X4217A-Z, X4216A-Z and X5196A-Z are all based on the ConnectX HCA as well as other IHV cards.

Note that further down the line, various components of the download release will make their way into the mainline Solaris releases. But this download release makes them available now for those who are eager to try them out. Whenever components go back into mainline Solaris, they will be removed from this download release series.

Download Solaris InfiniBand Updates 2 here. And, of course, the software is free.

Technorati Tags: InfiniBand

(2008-07-21 17:06:47.0) Permalink

Friday June 27, 2008

Sun IB News Round-Up 06.2008

So just in case you missed these...

• The Ranger system at University of Texas using the Sun Constellation system clocked in as number 4 on the Top 500 for June 2008.

• Sun announced a smaller NanoMagnum switch for the Sun Constellation system.

• Solaris 10 5/08 released with Sockets Direct Protocol (SDP) support and the fwflash utility for updating HCA firmware. Both are also in OpenSolaris 05.2008. And you can get either version of Solaris for free.

• ClusterTools 8 Early Access 2 released. ClusterTools is Sun's version of OpenMPI and of course has IB support. This version has both Solaris and Linux support. And of course it's free.

Technorati Tags: InfiniBand

(2008-06-27 16:23:30.0) Permalink

HCA Line-up

To state the obvious, I have not blogged in a while. But here is an article I have wanted to do for some time. Occasionally, people wonder which HCAs are offered by Sun. So here is a summary. Yes there are numerous IHV cards out there. But these HCAs are the ones you can get off the Sun price list.

Sun Option	Form Factor	Chip	IO Bus	IB Ports (per HCA)	Card Memory	Solaris Driver
X1333A-4	Card	MT23108 InfiniHost	PCI-X 133	Dual x4 SDR	256 MB	tavor
X1236A-Z	Card	MT25208 InfiniHost III Ex	PCIe x8	Dual x4 SDR	256 MB	tavor
X1288A-Z	Express Module
X1289A-Z	SunBlade 8000 NEM (10 HCAs)	MT25204 InfiniHost III Lx	PCIe x8	One x4 DDR	Memfree	arbel
X4217A-Z	Card	ConnectX-IB	PCIe x8	Dual x4 DDR	Memfree	hermon
X4216A-Z	Express Module
X5196A-Z	SunBlade 6048 NEM (12 HCAs + 2 switches)

Now all the notes and explanation: The Express Module is the PCIe hot-plug form factor used by Sun blade servers. NEM is our Network Express Module form factor which plugs into all the blades in a shelf. And because the NEMs have multiple HCAs in them, you have to note that the IB Ports column is listing the ports per HCA. The term Memfree means it uses host memory instead of using memory on the card. The arbel driver is currently available through our IB Updates 1. The hermon driver is NOT out yet, but it's coming; and before you ask I can't say any more than that. The X5196A-Z has the final stage of the switch setup for a Sun Constellation system built into the NEM.

If you are trying to use third party cards, the thing that might confuse you is that two different drivers are used with InfiniHost III Ex cards. If there is card memory, the Solaris tavor driver is used. On the other hand, Memfree cards use the Solaris arbel driver.

And of course, if there is any contradiction, I am sure the official stuff takes precedence over anything I say.

Technorati Tags: InfiniBand

(2008-06-27 15:31:58.0) Permalink