Ted H. Kim's Weblog

Musings of a Random Dude

Main | Next page »

Friday January 29, 2010

Disclaimer

Now that we are under Oracle's blog policy, I should state this:

The views expressed on this blog are my own and do not necessarily reflect the views of Oracle.

EDITED: Previously I said I was going to move personal stuff into another blog. It has been clarified that I probably don't have to move anything as the blog still is mostly business content. I think the real problem is the same as always, not enough time to spend on the blog, much less move stuff around.

(2010-01-29 14:51:44.0) Permalink Comments [0]

Friday October 09, 2009

InfiniBand on OpenStorage, etc.

Some exciting news: InfiniBand has been added to the OpenStorage offerings starting with the Q3.2009. Check out Cindi's blog here.

Of course, there have improvements for InfiniBand support in Solaris 10 10/09 release migrating over from the stuff in Nevada.

I have been told the X4239A DDR FEM HCA is really for the X6450 blade.

And there have been more Sun QDR switch models now too.

And of course, if what I say is contradicted elsewhere by official stuff, I am probably wrong!

Technorati Tags: InfiniBand

(2009-10-09 16:58:32.0) Permalink Comments [2]

Friday May 29, 2009

Solaris/IB and Open Source

OpenSolaris is known for being open source. In the InfiniBand area, the framework and most Upper Level Protocols (ULPs) were open source right from the start of OpenSolaris. However, we had to keep the Host Channel Adapter (HCA) drivers in closed source because of prior legal agreements. Well this took a while to sort out, and then finally make the changes to the source tree.

So I am happy to announce that the HCA drivers are now open source. You can look at them here.

There is still part of one ULP (Sockets Direct Protocol) which is in closed source, but that is the only remaining Solaris/IB item in closed source. Unfortunately, it is not likely to change any time soon, because of the way SDP was licensed. For similar reasons, even Linux/OFED has some problems in this area.

Technorati Tags: InfiniBand

(2009-05-29 10:54:31.0) Permalink

Wednesday May 27, 2009

SRP target

As noted here, the SRP target for COMSTAR just went back to Solaris Nevada, which is of course the base for future OpenSolaris releases. So now, Solaris can be an SRP target as well as an iSER target or an NFS/RDMA server. So you can get your blocks and files through all the popular flavors of RDMA protocols on IB.

Technorati Tags: InfiniBand

(2009-05-27 14:25:31.0) Permalink

Thursday May 14, 2009

Java can use SDP

Here is an item that was recently pointed out to me. You can now configure Java to use Sockets Direct Protocol (SDP) on InfiniBand, see this article.

Technorati Tags: InfiniBand

(2009-05-14 14:37:16.0) Permalink

Monday May 11, 2009

DDR FEM

Along with all the other Nehalem announcements related to IB was one other IB related item which I haven't talked about yet. That's the X4239A DDR Fabric Expansion Module (FEM). It's an HCA with PCIe 2.0 I/O interfaces and dual DDR 4x IB ports for connection to the in-chasis IB fabrics.

Now the x6275 blade has a QDR HCAs built in. So what would you do with this item? After looking here, it appears that you use it with the x6270 blade to connect them to the QNEM.

Technorati Tags: InfiniBand

(2009-05-11 15:55:24.0) Permalink

Friday May 01, 2009

ConnectX driver in Solaris 10

With the release of Solaris 10 5/09 (aka Solaris 10 Update 7), the ConnectX HCA driver (called "hermon") finally makes it's appearance in Solaris 10. It has been available in OpenSolaris since Nevada build 100. But now it finally makes its way to the Solaris 10. Download Solaris 10 5/09 here. And of course it's free.

Technorati Tags: InfiniBand

(2009-05-01 13:58:28.0) Permalink

Tuesday April 14, 2009

iSER

As other folks have mentioned, iSCSI Extensions for RDMA (iSER) was recently integrated into build 112 of OpenSolaris. Congratulations to the iSER team! The general background on iSER is here with the Solaris architecture specifics here. But the general idea is iSCSI with InfiniBand RDMA to drive up performance.

Technorati Tags: InfiniBand

(2009-04-14 15:25:50.0) Permalink

IB QDR and Nehalem (updated)

In general, InfiniBand DDR HCAs were already limited by the IO bus under PCIe 1.X. With today's Nehalem (Xeon 5500 series) related announcements come a bunch of PCIe 2.0 servers (including x2270, x4170, x4270, x4275 servers plus x6270 and x6275 blades). Now x4 DDR should be able to operate at full speed with a PCIe 2.0 x8 slot and x4 QDR is now realistic (though probably still IO bus limited). The x6275 blade even has on-board QDR HCAs installed. For some great shots of the boards, look here.

So in addition to earlier HCA offerings, we now have this QDR HCA story:

Sun Option Form Factor Chip IO Bus IB Ports
(per HCA) Card Memory Solaris Driver

X4237A Card ConnectX-IB PCIe 2.0 x8 Two x4 QDR Memfree hermon

X4219A-Z Express Module

Edit: I think I might finally understand this now! The X5500A SunBlade 6048 QDR NEM has switches, but no HCAs. This design is different from how the DDR NEM works, which has the HCAs in it too. Instead, the QDR NEM is meant to be used with on-board HCAs like with the x6275 blade.

But both IB NEMs for the SunBlade 6048 are meant to be used as part of an overall Sun Constellation system with the final stage of the fat-tree (or torus for the QDR one) switch network incorporated directly in the NEM itself.

And of course, if there is any contradiction, I am sure the official stuff takes precedence over anything I say.

Technorati Tags: InfiniBand

(2009-04-14 14:02:21.0) Permalink Comments [1]

Friday February 06, 2009

IB Updates 3 (OFUV UD)

Finally a timely announcement! Yesterday, Solaris InfiniBand Updates 3 was released. The big addition over the previous Updates 2.1 is the first increment of Open Fabrics User Verbs or OFUV.

What's OFUV you ask? Well, here is the story. Quite a while ago, we delivered uDAPL into Solaris. But over time, with the growing popularity of the Linux OFED InfiniBand releases, OFUV became the pre-dominant OS Bypass InfiniBand framework.

Now the question which comes up from time to time is, "why don't you just port OFED?" The simplified explanation is that it's quite hard to do a straight port for the kernel parts, unless you can completely emulate Linux kernel internals. So instead for the kernel-based InfiniBand services, we are trying to provide equivalent stuff but not necessarily a straight port. But on the userland side of things, it's much more practical to do a port.

So with this new software, we have done a port of userland OFUV APIs on Solaris. Now it's a big project, so this first increment has the Unreliable Datagram (UD) support (based on OFED 1.3 verbs and rdma_cm) and associated drivers (for hermon and tavor), utilities, etc. While the verbs exist for reliable connected (RC) support, the rdma_cm support is not yet in there. Following phases will round out the offering with rdma_cm support for RC and other stuff. And of course, sometime we have to update to OFED 1.4. But finally we have the first part.

So what do you do with userland UD? Lately, it has been used to do high volume, low latency messaging systems on top of InfiniBand, especially in financial services. So stay tuned ...

Technorati Tags: InfiniBand

(2009-02-06 16:56:40.0) Permalink Comments [4]

Wednesday January 28, 2009

Catching up on IB news

Last month, I failed to note that IB Updates 2.1 was released. This rev basically rolls IB Updates 2 forward to Solaris 10 10/2008 (Update 6). More discussion at this blog entry.

Technorati Tags: InfiniBand

(2009-01-28 16:31:37.0) Permalink

Friday October 03, 2008

ConnectX in Nevada

Some other folks have already mentioned this, but for completeness, I will point it out too. The ConnectX HCA driver has gone back to the Nevada source tree. So expect it to eventually come out through OpenSolaris. Plus this is a step towards getting it into the Solaris 10 Updates too.

The integration message is here.

Technorati Tags: InfiniBand

(2008-10-03 15:47:13.0) Permalink

Monday August 25, 2008

Updated NFS/RDMA

Just recently, the Updated NFS/RDMA integrated into Nevada. The history of NFS/RDMA goes back years in Solaris, when Sun originally put in a pre-standard version into Solaris. More recently, work in the IETF sought to standardize how NFS over RDMA would work (the drafts are now moving towards RFC status). The integration now gives Nevada an implementation that corresponds with the new IETF version (replacing the old pre-standard version).

In practical terms, saying over RDMA is a code for InfiniBand and the slowly emerging iWARP. So what the integration into Nevada means is Solaris will now have the updated IETF NFS/RDMA version for InfiniBand with the door open for iWARP when that comes along. And of course, Nevada is the codeword for propagating into the OpenSolaris releases.

Just to recap where we are on InfiniBand ULPs, Solaris now has implementations of IPonIB, SDP, RDS, NFS, SRP, uDAPL (used for MPI). The HCA story is here and here. And more is coming, including a port of the Open Fabrics Userland Verbs, iSER and certification on various SPARC platforms. So stay tuned for exciting days ahead...

Technorati Tags: InfiniBand

(2008-08-25 11:19:55.0) Permalink

Monday July 21, 2008

IB Updates 2

Last week we released Solaris InfiniBand Updates 2 through the Sun Downloads site. This replaces the earlier Solaris InfiniBand Updates 1. The new version supports the Mellanox ConnectX HCA as well as the cards supported in the earlier version. The new update is based on Solaris 10 5/08, which is the latest update of Solaris 10. Support for flashing the HCA firmware and for uDAPL (used by Sun's ClusterTools MPI) is also included. Sun options X4217A-Z, X4216A-Z and X5196A-Z are all based on the ConnectX HCA as well as other IHV cards.

Note that further down the line, various components of the download release will make their way into the mainline Solaris releases. But this download release makes them available now for those who are eager to try them out. Whenever components go back into mainline Solaris, they will be removed from this download release series.

Download Solaris InfiniBand Updates 2 here. And, of course, the software is free.

Technorati Tags: InfiniBand

(2008-07-21 17:06:47.0) Permalink

Friday June 27, 2008

Sun IB News Round-Up 06.2008

So just in case you missed these...

The Ranger system at University of Texas using the Sun Constellation system clocked in as number 4 on the Top 500 for June 2008.
Sun announced a smaller NanoMagnum switch for the Sun Constellation system.
Solaris 10 5/08 released with Sockets Direct Protocol (SDP) support and the fwflash utility for updating HCA firmware. Both are also in OpenSolaris 05.2008. And you can get either version of Solaris for free.
ClusterTools 8 Early Access 2 released. ClusterTools is Sun's version of OpenMPI and of course has IB support. This version has both Solaris and Linux support. And of course it's free .