Ted H. Kim's Weblog

Monday September 27, 2004

Too busy to write on InfiniBand?

Argh! There is always too little time to write everything I want to blog on. So instead of another excuse about why I am not writing today, I will instead link to another interesting InfiniBand article on our blog site which lists products from various InfiniBand vendors and talks about how to assemble it into a respectable HPC cluster. (If you want take a look at all of Dr. Hung-Sheng Tsao's InfiniBand Sun Blog, then go here.)

These examples are not really big InfiniBand clusters. If you want to look at a big one, take a look at this 1105-node cluster. Also there is an IB NASA cluster of 20 big machines that is under construction with 10,240 processors.

Another Sun blogger on IB is Tom Duffy.

Technorati Tags: InfiniBand

(2004-09-27 16:24:20.0) Permalink

Tuesday September 14, 2004

What can you do with InfiniBand?

Okay so it's time to post again on the InfiniBand topic. You might be wondering what can InfiniBand (IB) really be used for. In theory, InfiniBand could be used for a lot of things, including I/O, clustering, etc. But it is far more interesting to look at the cases where folks have actually gone forward with developing the Upper Level Protocols (ULPs) to use it.

For general networking, IP is king. So naturally, folks have implemented Internet Protocol on InfiniBand or IPonIB. Once you have done that your whole IP based stack (including TCP and whatever is above that) can ride on top. So for compatibility of IP infrastructure, this item is a must have. But IPonIB does not really use IB to its full advantage for performance. For one thing, it does not use RDMA, and often times deficiencies in the network stack prevent the full bandwidth potential from being used.

For reliable, connection-oriented services, most people think in terms of the socket interface on top of TCP/IP. InfiniBand has an equivalent called the "Sockets Direct Protocol" or SDP. SDP is described in chapter A4 of the volume 1 version 1.1 specification available from the InfiniBand Trade Association. SDP is not 100% compatible with all socket options, but it does support most common options. The advantage of SDP is that it can take advantage of the underlying reliability of IB's hardware implemented reliable connected transport type, so that software overhead for reliability is avoided. Further, the SDP protocol can use RDMA, though the socket interface must be extended to include such features as asynchronous operation. An effort to standardize these extensions is ongoing in the Socket API Extensions Work Group of the Interconnect Software Consortium (ICSC).

Then there are various sorts of existing protocols which have mapped onto IB and retooled to use RDMA to their advantage. For file access there is NFS over RDMA and "Direct Access File System" or DAFS. My understanding is that NFS and DAFS are merging together into a common protocol being developed in the IETF NFS v4 working group.

For block storage, there is SCSI RDMA Protocol or SRP and its successor SRP-2. Since much of block storage is actually Fibre Channel, there also an InfiniBand mapping for the Fibre Channel HBA API or FC-HBA to support storage management. Unfortunately, I have heard there are some hitches with SRP-2 going forward. Another obvious thing to do for block storage is to map iSCSI to IB, especially the RDMA version called iSCSI Extensions for RDMA or iSER being standardized in the IP Storage Working Group.The main problem is that InfiniBand iSCSI or iSER is not standardized anywhere. Also, there are some issues which need to be resolved with how you map the iSNS or SLP name services to IB. You can use companion IPonIB based services or do a more direct mapping to IB identifiers, but something has to be done given the basic assumption of IP based naming for iSCSI.

There always non-standard, proprietary uses of IB as well; but these are by their very nature vendor specific. But you might expect some IB native I/O device drivers to use this approach.

Another way to use InfiniBand is in an OS Bypass mode. In this type of architecture, the resources of an InfiniBand adapter are mapped directly into an application process. The application can talk directly to the hardware without any OS overhead. Obviously, you still have to involve the OS in the setup process or you have lost all your protections and policy. But once things are setup, this arrangement allows for the fastest possible access to the hardware. There are two APIs designed to enable this: uDAPL from the DAT Collaborative and ITAPI from ICSC Interconnect Transport API Working Group. An OS Bypass type of arrangement is attractive to HPTC people who want to implement the Message Passing Interface or MPI on IB.

Both uDAPL and ITAPI essentially provide a one to one mapping to IB objects in an OS bypass mode. The drawback is that this programming model is a specialized one, which many applications might not want to spend the effort to port to. Therefore, there is a tradeoff on squeezing every bit of performance from IB versus the learning curve/porting cost. In some cases, its better to choose the SDP/Extended Sockets approach to have a model closer to sockets. Except for the connection setup, these APIs do not really dictate a ULP per se. So its certainly possible that any or all of the previously mentioned ULPs could be done in an OS Bypass manner. It's just a matter of what makes sense in the overall system.

Technorati Tags: InfiniBand

(2004-09-14 13:17:56.0) Permalink

Monday September 13, 2004

The Arab-Israeli Wars

In Sepember, I decided to switch gears and read some post-WWII military history. The Arab-Israeli Wars by Chaim Herzog was another case of finally getting around to reading a book that had been sitting on my shelf for a while. Herzog himself was a participant in some of the wars described in the book as well as a former Israeli president. The book covers the period from Israel's independence in 1948 to the Lebanon campaign of 1982.

I don't know many topics which generate as much heated controversy as the conflicts of Middle East. A lot of this has to do with the fact that the underlying issues are still not resolved, and in various ways the conflicts are still on going. Given the environment, it not clear whether the participants can or would really give the detailed candid accounts that one would like. In some cases, it would be politically incorrect, and in others it would betray some military secrets. Further, the public does not have access to the level of documentation that exists about much of WWII (operational orders, daily unit reports, etc.). It's not even clear that the detailed documentation still exists. So any military history about the subject is going to be incomplete and probably without a sense of definitive perspective.

With all these limitations, how does The Arab-Israeli Wars do? I think the book gives an excellent account of the operational level history from the Israeli point of view. The author gives his opinion on many issues such as the cause of the wars, and while he acknowledges other views, it's clear what his viewpoint is. There is nothing wrong with that per se, though some readers may strongly disagree with his views. At various points, he notes Israeli mistakes and some personality issues of the commanders.

The strength of the book is in the accounts describing the Israeli unit operations: where the units were, what they did, their movement, attacks and orders. I found myself wanting to know a similar level of military detail about the Arab military operations. But as noted before, that information was probably not possible to get. However it would have have been nice, if the author clarified exactly what he was able to use in that area. The maps in the book are relatively clear, but there are some passages where it is hard to relate back to the maps.

I did not find the analysis in the book to be as strong as the operational accounts. In particular, the book was written only a year or two after the 1982 Lebanon operation. I think even Israelis may have issues with his analysis of that campaign's results. But even for other wars, I found myself wanting a deeper analysis. For example, the author rarely talks about the relative strengths and weaknesses of the military technology of each side and its impact. A lot could be written about topics such as wire-guided anti-tank missiles, UAVs, ECM, etc. and their effect on the campaigns. Nor does he talk much about tactical doctrine of the participants.

Overall, I enjoyed reading The Arab-Israeli Wars. But at the same time, I realize it is not a definitive account. One fairly old board wargame that covers part of the same subject (1956, 1967, 1973) is Sinai. However, since it has been such a long time since I played the game, I can't really give it an unqualified recommendation. I will need to try it again sometime to see.

Technorati Tags: books, boardgames

(2004-09-13 13:12:09.0) Permalink

Wednesday September 08, 2004

The Battle of Kursk

Since I liked the last Glantz book, I decided to try another one in August. This time I tried The Battle of Kursk by David M. Glantz and Jonathan M. House. I have only read Jonathan House as a co-author with Glantz, though apparently House has some books where he is the sole author. Nevertheless, it was Glantz's name which brought me to this book.

Like his other books, Glantz once again looks into the recently opened Soviet archives to give a fresh look at WWII Eastern front battles. If Stalingrad was the turning point, Kursk is generally known as the battle where the strategic initiative permanently shifted to the Soviet Union. The Eastern front in WWII was full of titanic, intense battles. Nevertheless, Kursk seems to have special legendary status associated with it. It was not just intense, but intense over a very large area. The campaign includes Prohorovka which is reputed to be the largest tank battle in history. The forces included the cream of Germany's mobile forces with the latest weapons rushed into production (including Elephants and the famous Panther with an immature drive train). On the Soviet side, defense was not only in depth to an unprecedented degree, but backed with huge reserves (multiple fronts). The defenses included six defensive belts, including lines that sealed off the entire Kursk salient, guarding against the possibility of the Germans exploiting a successful attack.

The Battle of Kursk gives an interesting description of the battle. What this account provides is a bit more insight into the actions of the Soviet side based on the records from the archives. Of course, it's hard to match the impact of the revelations about Operation Mars in Zhukov's Greatest Defeat. So what unfolds is interesting, but not really that surprising. As expected, the battle was a huge, intense struggle. But according to this book, while Prohorovka was a big tank battle, it is not as big as previous accounts like to paint it. Also, Glantz makes it clear that Kursk is really just the prelude to an overall strategic Soviet offensive plan for the summer of 1943. The offensives at Orel and Kharkov coming so quickly after Kursk are not surprising once seen in that light.

Glantz also tries to address some of the questions surrounding the battle. In this area, I think the book is not as strong as in the descriptions of the battle itself. For me, his analysis is interesting and somewhat convincing, but ultimately not yet the last word on these issues. For starters, why did the German attacks not succeed? In Glantz's opinion, it was the diffusion of effort necessary to fend off the continuous counter-attacks on the flanks. Of course, there is the larger question of whether getting past Obian, Prohorovka or Kursk would really let the German mobile forces break into the open. Even if they did, would that be decisive with the northern pincer stalled? Glantz notes that there were further defensive lines, though they were not as well prepared; but on the other hand, there were still substantial reserves to defend and counter-attack with. So while he doesn't flat out state it, I feel Glantz thinks the Kursk attacks did not have much chance to succeed.

An often repeated claim is that the German Kursk attack could have succeeded if launched in May as originally planned. Glantz makes the point that while Soviet forces were strengthened during the delay, the German forces also were being refitted and equipped with the latest weapons. (Oddly enough, the book has a rather poor picture of the Panther tank.) Perhaps, one part of the delay that was fatal was waiting so long that events outside of the Eastern front (Western Allies invasion of Sicily) had an impact.

The maps in this book are very much like that in Zhukov's Greatest Defeat and When Titans Clashed. Overall they are good, but at Kursk, there are so many Soviet units packed in to the battle that the maps are look very crowded and sometimes hard to read. Overall, though, I found the book very interesting and certainly something any Eastern front military history buff ought to look at.

Again, I would have liked to have mentioned a board wargame tie-in. But I don't have enough experience with the Kursk games out there to recommend one. Further, I am not sure the games will reflect the new information from this book. I am thinking of trying Clash of the Titans which is based on the famous Ring of Fire system. Certainly, I can recommend the latter game, if you want to take a look at the 4th Battle of Kharkov in the aftermath of Kursk.

Technorati Tags: books, boardgames

(2004-09-08 13:00:00.0) Permalink

Wednesday September 01, 2004

Zhukov's Greatest Defeat

Yup, more military history -- I guess I am sort of on a military history kick. Back in July, I tackled another one. Zhukov's Greatest Defeat by David M. Glantz is another one that's been on my shelf a while. Previously, I had read Glantz's When Titans Clashed and was favorably impressed, so I bought some more of his books.

Glantz is known for writing Eastern front WWII books using Soviet archive material which was previously unavailable. This book is in the same vein. The problem with many older works on the Eastern front is that the Soviet side of the story has not been available. Before the fall of the Soviet Union, a lot of official history was propaganda, and often not very credible or substantiated. Even where it was accurate, it was difficult to come by the detailed background information. So it was often the case that Eastern front military history was written based on the German historical records (which the Allies seized at the end of the war and made publically available) or German memoirs. As might be expected, some of these memoirs have revised history to make their authors look better. Only recently, has it been possible to get a balanced look at the historical material. It should also be noted that this process of releasing Soviet archive material is not yet complete. It's not even clear that what has been released has been fully analyzed either.

Zhukov is known to many as the savior of the Soviet Union in WWII. Stalin certainly had the will to survive, but many credit Zhukov as providing the necessary strategic military skill to beat the Germans, especially after the purges of the Red Army officer corps. Zhukov is credited with a key role in defeating the Germans in the Battle for Moscow in late 1941. From that time on, Zhukov is seen as the top Soviet military commander, stopping the Germans at Stalingrad and directing the unstoppable Juggernaut from Kursk on. The impression is one of an undefeated leader. Sure, there were hickups along the way (like Izyum in 1942), but the basic impression remains.

Zhukov's Greatest Defeat challenges this popular view. The book reveals the previously covered up Operation Mars. Despite the vast size of the operation, it has been omitted from Soviet era official writings with only the briefest hints in fragmentary pieces. Only the opening of the Soviet archives allowed the real story to emerge. While the Germans recorded accounts of intense fighting, it seems they also did not know the true nature of the battle either.

Operation Mars was intended to encircle and destroy the German Ninth Army in the Rzhev salient near Moscow in late 1942, just as Operation Uranus encircled and destroyed the German Sixth Army at Stalingrad. Operation Uranus was followed up by Operation Saturn, Ring, etc. to produce the total collapse of the southern Army Groups which was only halted by Manstein's famous Backhand Blow. In a similar way, Mars was to be followed up by Operation Jupiter to totally destroy Army Group Center.

In the end, German reserves held, though seemingly just barely. Many German forces were severely mauled and some of the reserves needed to be pulled out for lengthy refitting. But the Soviet side experienced appalling losses with 335,000 casualties out of about 670,000 committed to this operation. About 100,000 of the total were deaths. And this all in one month of fighting. To put the collosal scale of the Eastern Front in perspective, these figures (both total casualties and deaths) are roughly one third the amount the US suffered in all of WWII in about four years of fighting. In the end, the Rzhev salient was evacuated some time later, so it doesn't even seem like the Germans had a lasting victory.

In this book, Glantz imagines what the commanders must have felt like. And so from time to time, he adds some brief passages of fictionalized first person narrative. But he carefully contains that writing to convey a mood and never to convey the actual facts of the campaign. As you might expect the extremely intense nature of Eastern front combat is very evident in this book. At times, it can feel overwhemling. The maps in this book are fairly good. The geography is done on a gray scale background with the unit designations printed in white in the foreground. And best of all, the maps don't miss items mentioned in the battle descriptions.

While we know Zhukov as a relentless commander, this book also shows his ruthless and sometimes petty side. It's not clear to me though whether this side of Zhukov was simply a product of Stalinist times or whether his inherent human failings were coming out. It is well-known that Stalin pitted Zhukov against Konev in the Berlin campaign. In Operation Mars, Zhukov is jealous of Vasilevsky's success in Stalingrad. Further, Stalin exploits the rivalry by dangling reserve armies as the prize between the two. When Mars starts to break down, Zhukov gambles more desperately by pushing his commanders harder, despite the human and material costs. It seems his reputation is more important to him. And finally, when Mars is a foregone conclusion, Zhukov becomes vindictive and punishes failure by making them continue futile attacks with a horrific cost. To top it off, several commanders are removed to both shift the blame and as revenge.

I would have liked to mention a tie-in board wargame, but I don't know of any published game which covers it. In summary, I would recommend this book to any serious WWII Eastern front military history buff. Glantz has found what is perhaps the biggest revelation yet to be revealed from the Soviet military archives on WWII.

Technorati Tags: books

(2004-09-01 13:05:07.0) Permalink