Thursday February 07, 2008 Ted H. Kim's Weblog
Musings of a Random Dude
Thursday February 07, 2008 Here are some recent blog entries featuring Sun hardware using InfiniBand in action. In each case, InfiniBand is being used with Sun hardware to challenge for world records.
More Ranger Facts and Figures gives a run down on the stats for the huge TACC installation which includes 62,976 CPU cores in the system. We won't know how it places on the Top500 until June.
A trio of stories about world record benchmarks using Sun gear connected by InfiniBand: ABAQUS v6.7, EXA PowerFLOW, Fluent.
Technorati Tags: InfiniBand
(2008-02-07 17:42:17.0) Permalink
Back when we came out with our first Solaris HCA driver, InfiniBand only had what is now called single data rate (SDR). Now the situation is different. Not only are double (DDR) and quad (QDR) rates spec'ed for IB, DDR is common out in the customer base. To address this, we have fixes that are going to be released in the next update of Solaris 10, which means they are also in OpenSolaris already. But to get it out faster, the fixes are also being pushed out in official Solaris patches.
So if you have a DDR capable HCA that runs our "tavor" driver, try out the patch (sparc: 126647-02, x86: 126648-02). Of course, remember to be sure your switch is configured for DDR too. Folks using Sun Connection Update Manager should have seen these come out already in the patch stream. BTW, if you are wondering about the "arbel" driver, these fixes are already in our recently released InfiniBand Updates 1.
I should also mention that this patch contains some other fixes which improve our uDAPL performance (used by Sun's MPI implementation in ClusterTools) as well as other fixes which might be of interest to our Solaris IB users. Check it out.
Technorati Tags: InfiniBand
(2008-02-07 17:13:11.0) Permalink
Friday January 25, 2008 Recently, some parts of the Sun download site were reorganized. Now, it is a bit easier to find some InfiniBand items, since they are now in the InfiniBand section under the Hardware Drivers category.
Currently, we have some Linux stuff there as well as these add-ons to the base InfiniBand support in Solaris:
Solaris InfiniBand Updates 1 - which I talk more about here
SCSI RDMA Protocol (SRP) Driver
Firmware Flash Update Tool - to update your HCA firmware when necessary
Of course, this is not the complete list of downloads with InfiniBand content. I don't totally understand the classification system, but I guess it depends on whether they are primarily InfiniBand or more generic than that.
So for example, there is HPC ClusterTools 7.1 which Sun's MPI product which also has InfiniBand support and can be downloaded. But it's in a different section. I think there are some others like this, but you get the idea.
Oh yeah, did I mention that all the downloads I mentioned above are all FREE.
Technorati Tags: InfiniBand
(2008-01-25 13:34:53.0) Permalink
Monday January 14, 2008 Just wanted to point out some OpenSolaris projects for InfiniBand ...
IP on InfiniBand - Of course, IPonIB is already in Solaris, but this OpenSolaris project is about futures.
iSCSI Extensions for RDMA (iSER) - This project includes both initiator and target.
NFS over RDMA - Of course, this is both client and server.
OpenFabrics User Verbs - This project is a port of the OpenFabric's userland verb interface to Solaris. Once this is done, many apps which run on OFED could then be ported to run on Solaris.
Technorati Tags: InfiniBand
(2008-01-14 13:10:04.0) Permalink
Friday January 11, 2008 Like many other bloggers, I have to simply own up to the reality that blogging is bursty. Often it's a particular event rather than a lifestyle habit that motivates posting.
Well, this time I am motivated by the release of Solaris InfiniBand Updates 1. If you want cut to the chase, then get it here. But if you have no idea what this is about, then read on ...
This product is a FREE software download containing additional software for Solaris 10 InfiniBand. To run it, you need Solaris 10 8 / 07 (Update 4). From the name, you can probably guess that it will not be the only release in this series. You may wonder if this software is also going back into the other Solaris release trains. The answer is YES. It is our intention to eventually fold all this stuff back into the mainstream Solaris releases. So it is going to end up there. But for now, to satisfy the demands of our rapidly growing InfiniBand market in both HPC and the Enterprise, it's coming out NOW to minimize delay in getting to the market.
So what's in this release? Let me be the first to admit the name of the product is something only an accountant would like. The name really does not tell you what's in it. This product contains support for MemFree InfiniBand HCAs based on the Mellanox MT25204 and MT25208 chips (these PCI IDs: 15b3,5e8d 15b3,6274 15b3,6828). For the non-IB crowd, "MemFree" means that the HCA uses host memory instead of memory on the I/O adapter card for the HCA state (so it's LESS EXPENSIVE). So in particular the Sun 10-Port 4x DDR IB NEM for the SunBlade 8000/8000P (option X1289A-Z) is supported (see here, link to a drawing of the NEM). Of course, there are also numerous third party cards also using these Mellanox chips. Note that this software is only for the "MemFree" cards, other Sun software should be used for the cards (and Express Modules) with on-board memory.
In other Sun InfiniBand news, I should also point out that ClusterTools 7.1 (Sun's MPI based on OpenMPI 1.2.4) has been released. See this article.
Technorati Tags: InfiniBand
(2008-01-11 10:39:21.0) Permalink Comments [2]
Wednesday June 27, 2007 Others have mentioned it, but in case you missed it ...
Folks may want to check out the YouTube video of the unveiling of Sun's 3456 port InfiniBand switch.
If you are wondering what you do with such a big switch, you build an HPC setup something like this.
Technorati Tags: InfiniBand
(2007-06-27 15:09:20.0) Permalink
Friday April 06, 2007  | Previously I talked about Early Access versions of Sun ClusterTools 7. Now the production version has been released here. Just to repeat again, this version is based on OpenMPI and works on Solaris for both SPARC and x64. As well as InfiniBand, it also supports TCP, shared memory and has a Myrinet MX module too. |
Technorati Tags: InfiniBand
(2007-04-06 10:43:52.0) Permalink
Friday January 19, 2007 ClusterTools 7 Early Access (2)
For those following along, there is an update available for Sun's MPI product-- ClusterTools -- on InfiniBand. This update is a refresh of the ClusterTools 7 Early Access release. This OpenMPI based release includes a uDAPL BTL with support for both send/recv and RDMA to improve IB performance, plus other bug fixes and improvements. Plus like most of our software, it's free. So download away.
Technorati Tags: InfiniBand
(2007-01-19 14:03:18.0) Permalink
Friday November 03, 2006 I offer this anecdotal data point on the availability of InfiniBand:
InfiniBand on Amazon
How about this refurbished switch and module-box item for around $1500! Standard Disclaimer: I don't endorse this particular item or the company offering the item. But I am trying to claim IB is creeping into the mainstream.
InfiniBand on eBay
Currently, this item
seems to be the best deal on eBay for HCAs. Wow, $50 new! Standard Disclaimer: I don't endorse this particular item or the person offering the item. But I do hope you get my point about falling prices and availability.
Technorati Tags: InfiniBand
(2006-11-03 12:28:25.0) Permalink
Friday September 22, 2006 So it goes it without saying that I should blog more. But just to remind people about what is going on, I thought I would highlight some recent Solaris InfiniBand news.
Folks might be wondering about our MPI story on InfiniBand. For a while now, Ohio State's MVAPICH has run on top of our Solaris uDAPL. More recently, Scali has announced support for their implementation of MPI on Solaris 10 on our Opteron machines. And an Open MPI version is in the works for the official Sun product.
Finally, I should mention that the Solaris InfiniBand SCSI RDMA Protocol (SRP) driver 1.0 is now available for free download. So download away and start using IB SRP storage. As you might imagine, other IB storage options are also under development, but nothing has been officially announced yet.
Technorati Tags: InfiniBand
(2006-09-22 13:00:00.0) Permalink
Wednesday April 19, 2006  | Sun just recently put PCI-Express HCAs on the pricelist. This product uses an 8x PCI-E interface and should help in cases where folks were bandwidth limited by PCI-X or PCI buses. This HCA has 256MB on-card and is also available in low-profile. Take a look here. |
Technorati Tags: InfiniBand
(2006-04-19 12:00:00.0) Permalink
Monday January 30, 2006 Just an update on InfiniBand certified vendors ...
Fans of Voltaire equipment will be delighted to see that the company's products are now listed on the Sun website as Solaris Ready here.
The overall list of certified InfiniBand vendors is here.
Technorati Tags: InfiniBand
(2006-01-30 10:00:00.0) Permalink
Wednesday December 14, 2005 Voltaire, iSER, IP addressing & InfiniBand
A few updates on InfiniBand topics ...
I have been asked privately about Voltaire, another IB vendor. While they aren't on the official Sun website (yet?), I should point out that Voltaire gear has been used in a number of Sun deals already. For example, the massive TiTech cluster uses their ISR9288 switch (up to 288 ports!).
It also looks like the latest IETF draft for iSER does manage to cough up a few words about InfiniBand. Check out Appendix B of the October 2005 draft. This is quite a break from IETF tradition to even hint that a world outside of IP even exists! Anyway, to fit it all together, the stuff that is not in Appendix B (for InfiniBand iSER) is being defined in IBTA Software Working Group (SWG).
Also, the SWG took on the related task of defining an IP address mapping for InfiniBand services. So how to handle the IP addressing for the InfiniBand version of iSER, NFS, etc. will be mapped out.
Technorati Tags: InfiniBand
(2005-12-14 17:00:00.0) Permalink
Wednesday November 16, 2005 Folks may be wondering which InfiniBand solutions work with Sun stuff. Here are some URLs to check out for this information cross-indexed by technology and vendor off the Sun website:
- 3rd Party InfiniBand Summary
- Vendor: Cisco (TopSpin) information
- Vendor: Silverstorm Technologies information
Also, just recently a big Sun x64 cluster (10,480 Opterons) using InfiniBand was announced at Tokyo Institute of Technology. They are aiming to be in the top 5 of the Top500 list.
My thanks to Carl Hensler for providing this information.
Stay tuned for more announcements ...
Technorati Tags: InfiniBand
(2005-11-16 10:10:10.0) Permalink
Wednesday October 12, 2005 For years, the InfiniBand (IB) community has debated which storage protocol to use. One candidate is the SCSI RDMA Protocol (SRP). The other one is an IB version of iSCSI Extensions for RDMA (iSER) protocol. SRP was worked on and approved by T10. There was also a companion IB mapping added to the FC HBA API in T11. SRP products are shipping. But unfortunately, the follow-on SRP2 standards effort died.
iSCSI is an IETF protocol based on IP. As you would expect, it is well integrated into the rest of the IP networking world. Overall, that fact helps give iSCSI based approaches an advantage over SRP when it comes to discovery and management. The iSER version of iSCSI was developed to exploit iWARP's RDMA capabilities over IP.
IB shares a common RDMA heritage with iWARP. So the obvious thing to do is to run iSER over IB as well. In fact, that approach has been done in shipping product already. The thing that was missing was standardization of the mapping. Otherwise, it was a private, vendor-specific thing. iSCSI itself was defined in IETF. So some folks tried to get the IETF documents to describe the IB specific mapping. But IETF rejected that on the grounds that the material was not IP related.
So now, the effort to define the mapping has come to the InfiniBand Trade Association (IBTA). Just recently, the Software Working Group (SWG) of the IBTA approved a project to define the official iSER on IB mapping. As Sun's representative to the SWG, I will be participating in that work.
What I think this means in the long run is that we will see a migration in IB storage towards iSER. As with any transition, there are existing solutions which work. So I don't expect SRP to go away immediately. But over time, I do expect the mix to tilt more towards iSER, especially because of it's commonality with iWARP.
Technorati Tags: InfiniBand
(2005-10-12 14:48:56.0) Permalink
Calendar
| « November 2009 | ||||||
| Sun | Mon | Tue | Wed | Thu | Fri | Sat |
|---|---|---|---|---|---|---|
1 | 2 | 3 | 4 | 5 | 6 | 7 |
8 | 9 | 10 | 11 | 12 | 13 | 14 |
15 | 16 | 17 | 18 | 19 | 20 | 21 |
22 | 23 | 24 | 25 | 26 | 27 | 28 |
29 | 30 | |||||
| Today | ||||||
RSS Feeds
All
/Boardgames
/Books
/General
/InfiniBand
/IO
/iWARP