Friday October 03, 2008 Ted H. Kim's Weblog
Musings of a Random Dude
Friday October 03, 2008 Some other folks have already mentioned this, but for completeness, I will point it out too. The ConnectX HCA driver has gone back to the Nevada source tree. So expect it to eventually come out through OpenSolaris. Plus this is a step towards getting it into the Solaris 10 Updates too.
The integration message is here.
Technorati Tags: InfiniBand
(2008-10-03 15:47:13.0) Permalink Comments [0]
Monday August 25, 2008 Just recently, the Updated NFS/RDMA integrated into Nevada. The history of NFS/RDMA goes back years in Solaris, when Sun originally put in a pre-standard version into Solaris. More recently, work in the IETF sought to standardize how NFS over RDMA would work (the drafts are now moving towards RFC status). The integration now gives Nevada an implementation that corresponds with the new IETF version (replacing the old pre-standard version).
In practical terms, saying over RDMA is a code for InfiniBand and the slowly emerging iWARP. So what the integration into Nevada means is Solaris will now have the updated IETF NFS/RDMA version for InfiniBand with the door open for iWARP when that comes along. And of course, Nevada is the codeword for propagating into the OpenSolaris releases.
Just to recap where we are on InfiniBand ULPs, Solaris now has implementations of IPonIB, SDP, RDS, NFS, SRP, uDAPL (used for MPI). The HCA story is here and here. And more is coming, including a port of the Open Fabrics Userland Verbs, iSER and certification on various SPARC platforms. So stay tuned for exciting days ahead...
Technorati Tags: InfiniBand
(2008-08-25 11:19:55.0) Permalink Comments [0]
Monday July 21, 2008 Last week we released Solaris InfiniBand Updates 2 through the Sun Downloads site. This replaces the earlier Solaris InfiniBand Updates 1. The new version supports the Mellanox ConnectX HCA as well as the cards supported in the earlier version. The new update is based on Solaris 10 5/08, which is the latest update of Solaris 10. Support for flashing the HCA firmware and for uDAPL (used by Sun's ClusterTools MPI) is also included. Sun options X4217A-Z, X4216A-Z and X5196A-Z are all based on the ConnectX HCA as well as other IHV cards.
Note that further down the line, various components of the download release will make their way into the mainline Solaris releases. But this download release makes them available now for those who are eager to try them out. Whenever components go back into mainline Solaris, they will be removed from this download release series.
Download Solaris InfiniBand Updates 2 here. And, of course, the software is free.
Technorati Tags: InfiniBand
(2008-07-21 17:06:47.0) Permalink Comments [0]
Friday June 27, 2008 So just in case you missed these...
The Ranger system at University of Texas using the Sun Constellation system clocked in as number 4 on the Top 500 for June 2008.
Sun announced a smaller NanoMagnum switch for the Sun Constellation system.
Solaris 10 5/08 released with Sockets Direct Protocol (SDP) support and the fwflash utility for updating HCA firmware. Both are also in OpenSolaris 05.2008. And you can get either version of Solaris for free.
ClusterTools 8 Early Access 2 released. ClusterTools is Sun's version of OpenMPI and of course has IB support. This version has both Solaris and Linux support. And of course it's free.
Technorati Tags: InfiniBand
(2008-06-27 16:23:30.0) Permalink
To state the obvious, I have not blogged in a while. But here is an article I have wanted to do for some time. Occasionally, people wonder which HCAs are offered by Sun. So here is a summary. Yes there are numerous IHV cards out there. But these HCAs are the ones you can get off the Sun price list.
| Sun Option | Form Factor | Chip | IO Bus | IB Ports (per HCA) | Card Memory | Solaris Driver |
|---|---|---|---|---|---|---|
| X1333A-4 | Card | MT23108 InfiniHost | PCI-X 133 | Dual x4 SDR | 256 MB | tavor |
| X1236A-Z | Card | MT25208 InfiniHost III Ex | PCIe x8 | Dual x4 SDR | 256 MB | tavor |
| X1288A-Z | Express Module | |||||
| X1289A-Z | SunBlade 8000 NEM (10 HCAs) | MT25204 InfiniHost III Lx | PCIe x8 | One x4 DDR | Memfree | arbel |
| X4217A-Z | Card | ConnectX-IB | PCIe x8 | Dual x4 DDR | Memfree | hermon |
| X4216A-Z | Express Module | |||||
| X5196A-Z | SunBlade 6048 NEM (12 HCAs + 2 switches) |
Now all the notes and explanation: The Express Module is the PCIe hot-plug form factor used by Sun blade servers. NEM is our Network Express Module form factor which plugs into all the blades in a shelf. And because the NEMs have multiple HCAs in them, you have to note that the IB Ports column is listing the ports per HCA. The term Memfree means it uses host memory instead of using memory on the card. The arbel driver is currently available through our IB Updates 1. The hermon driver is NOT out yet, but it's coming; and before you ask I can't say any more than that. The X5196A-Z has the final stage of the switch setup for a Sun Constellation system built into the NEM.
If you are trying to use third party cards, the thing that might confuse you is that two different drivers are used with InfiniHost III Ex cards. If there is card memory, the Solaris tavor driver is used. On the other hand, Memfree cards use the Solaris arbel driver.
And of course, if there is any contradiction, I am sure the official stuff takes precedence over anything I say.
Technorati Tags: InfiniBand
(2008-06-27 15:31:58.0) Permalink
Thursday February 07, 2008 Here are some recent blog entries featuring Sun hardware using InfiniBand in action. In each case, InfiniBand is being used with Sun hardware to challenge for world records.
More Ranger Facts and Figures gives a run down on the stats for the huge TACC installation which includes 62,976 CPU cores in the system. We won't know how it places on the Top500 until June.
A trio of stories about world record benchmarks using Sun gear connected by InfiniBand: ABAQUS v6.7, EXA PowerFLOW, Fluent.
Technorati Tags: InfiniBand
(2008-02-07 17:42:17.0) Permalink
Back when we came out with our first Solaris HCA driver, InfiniBand only had what is now called single data rate (SDR). Now the situation is different. Not only are double (DDR) and quad (QDR) rates spec'ed for IB, DDR is common out in the customer base. To address this, we have fixes that are going to be released in the next update of Solaris 10, which means they are also in OpenSolaris already. But to get it out faster, the fixes are also being pushed out in official Solaris patches.
So if you have a DDR capable HCA that runs our "tavor" driver, try out the patch (sparc: 126647-02, x86: 126648-02). Of course, remember to be sure your switch is configured for DDR too. Folks using Sun Connection Update Manager should have seen these come out already in the patch stream. BTW, if you are wondering about the "arbel" driver, these fixes are already in our recently released InfiniBand Updates 1.
I should also mention that this patch contains some other fixes which improve our uDAPL performance (used by Sun's MPI implementation in ClusterTools) as well as other fixes which might be of interest to our Solaris IB users. Check it out.
Technorati Tags: InfiniBand
(2008-02-07 17:13:11.0) Permalink
Friday January 25, 2008 Recently, some parts of the Sun download site were reorganized. Now, it is a bit easier to find some InfiniBand items, since they are now in the InfiniBand section under the Hardware Drivers category.
Currently, we have some Linux stuff there as well as these add-ons to the base InfiniBand support in Solaris:
Solaris InfiniBand Updates 1 - which I talk more about here
SCSI RDMA Protocol (SRP) Driver
Firmware Flash Update Tool - to update your HCA firmware when necessary
Of course, this is not the complete list of downloads with InfiniBand content. I don't totally understand the classification system, but I guess it depends on whether they are primarily InfiniBand or more generic than that.
So for example, there is HPC ClusterTools 7.1 which Sun's MPI product which also has InfiniBand support and can be downloaded. But it's in a different section. I think there are some others like this, but you get the idea.
Oh yeah, did I mention that all the downloads I mentioned above are all FREE.
Technorati Tags: InfiniBand
(2008-01-25 13:34:53.0) Permalink
Monday January 14, 2008 Not much has been posted yet, but I should point out that there is now an OpenSolaris project for iWARP.
Look here: RDMA Offload Framework (ROF)
Technorati Tags: iWARP
(2008-01-14 13:18:58.0) Permalink
Just wanted to point out some OpenSolaris projects for InfiniBand ...
IP on InfiniBand - Of course, IPonIB is already in Solaris, but this OpenSolaris project is about futures.
iSCSI Extensions for RDMA (iSER) - This project includes both initiator and target.
NFS over RDMA - Of course, this is both client and server.
OpenFabrics User Verbs - This project is a port of the OpenFabric's userland verb interface to Solaris. Once this is done, many apps which run on OFED could then be ported to run on Solaris.
Technorati Tags: InfiniBand
(2008-01-14 13:10:04.0) Permalink
Friday January 11, 2008 Like many other bloggers, I have to simply own up to the reality that blogging is bursty. Often it's a particular event rather than a lifestyle habit that motivates posting.
Well, this time I am motivated by the release of Solaris InfiniBand Updates 1. If you want cut to the chase, then get it here. But if you have no idea what this is about, then read on ...
This product is a FREE software download containing additional software for Solaris 10 InfiniBand. To run it, you need Solaris 10 8 / 07 (Update 4). From the name, you can probably guess that it will not be the only release in this series. You may wonder if this software is also going back into the other Solaris release trains. The answer is YES. It is our intention to eventually fold all this stuff back into the mainstream Solaris releases. So it is going to end up there. But for now, to satisfy the demands of our rapidly growing InfiniBand market in both HPC and the Enterprise, it's coming out NOW to minimize delay in getting to the market.
So what's in this release? Let me be the first to admit the name of the product is something only an accountant would like. The name really does not tell you what's in it. This product contains support for MemFree InfiniBand HCAs based on the Mellanox MT25204 and MT25208 chips (these PCI IDs: 15b3,5e8d 15b3,6274 15b3,6828). For the non-IB crowd, "MemFree" means that the HCA uses host memory instead of memory on the I/O adapter card for the HCA state (so it's LESS EXPENSIVE). So in particular the Sun 10-Port 4x DDR IB NEM for the SunBlade 8000/8000P (option X1289A-Z) is supported (see here, link to a drawing of the NEM). Of course, there are also numerous third party cards also using these Mellanox chips. Note that this software is only for the "MemFree" cards, other Sun software should be used for the cards (and Express Modules) with on-board memory.
In other Sun InfiniBand news, I should also point out that ClusterTools 7.1 (Sun's MPI based on OpenMPI 1.2.4) has been released. See this article.
Technorati Tags: InfiniBand
(2008-01-11 10:39:21.0) Permalink Comments [2]
Wednesday June 27, 2007 Others have mentioned it, but in case you missed it ...
Folks may want to check out the YouTube video of the unveiling of Sun's 3456 port InfiniBand switch.
If you are wondering what you do with such a big switch, you build an HPC setup something like this.
Technorati Tags: InfiniBand
(2007-06-27 15:09:20.0) Permalink
Friday April 06, 2007  | Previously I talked about Early Access versions of Sun ClusterTools 7. Now the production version has been released here. Just to repeat again, this version is based on OpenMPI and works on Solaris for both SPARC and x64. As well as InfiniBand, it also supports TCP, shared memory and has a Myrinet MX module too. |
Technorati Tags: InfiniBand
(2007-04-06 10:43:52.0) Permalink
Friday January 19, 2007 ClusterTools 7 Early Access (2)
For those following along, there is an update available for Sun's MPI product-- ClusterTools -- on InfiniBand. This update is a refresh of the ClusterTools 7 Early Access release. This OpenMPI based release includes a uDAPL BTL with support for both send/recv and RDMA to improve IB performance, plus other bug fixes and improvements. Plus like most of our software, it's free. So download away.
Technorati Tags: InfiniBand
(2007-01-19 14:03:18.0) Permalink
Wednesday November 15, 2006 A while ago, I described RNICPI. I would have liked to have said it's taking over the world. But alas, that is not to be. I really did want RNICPI to succeed. After all, I am the co-chair of the group that authored it. Now, that is not to say there won't be some RNICPI implementations. I wish them luck.
 | But as far as Sun is concerned, we have decided that the industry momentum is increasingly behind the Open Fabrics Alliance (OFA). Now OFA was originally called OpenIB because of it's InfiniBand roots. And most of its software is still heavily influenced by that. But it also has iWARP support under development. There are some significant milestones still to come before OFA iWARP support can be considered mature, but it's coming along. |
Now if you have been reading my blog, you know I am a Solaris developer. Folks familiar with OFA know it's based around Linux (and there is a Windows version too!). So there are various issues involved with leveraging OFA stuff for iWARP on Solaris and making it all work together. But that is what we are doing. And since it's SC06 time, I guess I shouldn't delay blogging about this any longer.
So there you have it. We are trying to take advantage of the growing interest in iWARP based on OFA code and interfaces. Many of the iWARP IHVs have OFA code internally or under development. We feel that OFA is going to become the de facto standard for iWARP software.
More on this story to come ...
Now I would be remiss if I didn't offer all the standard disclaimers. YMMV. I don't time travel. But who knows, I might still be right. :-)
Technorati Tags: iWARP
(2006-11-15 16:00:00.0) Permalink
Calendar
| « October 2008 | ||||||
| Sun | Mon | Tue | Wed | Thu | Fri | Sat |
|---|---|---|---|---|---|---|
1 | 2 | 3 | 4 | |||
5 | 6 | 7 | 8 | 9 | 10 | 11 |
12 | 13 | 14 | 15 | 16 | 17 | 18 |
19 | 20 | 21 | 22 | 23 | 24 | 25 |
26 | 27 | 28 | 29 | 30 | 31 | |
| Today | ||||||
RSS Feeds
All
/Boardgames
/Books
/General
/InfiniBand
/IO
/iWARP