A Penguin in Purple Clothing

Wednesday September 15, 2004

OpenIB on sparc64/linux

One of my skunx works projects for a while has been to play with sparc64/linux. I have a couple of old e250's in the lab and a blade 100 on my desk that run Debian Sid sparc64. Well, since Roland wanted to test more platforms for openib, I said I would check out sparc64. I got it to compile with a few minor tweaks.

On Monday, I spent a bit of time trying to debug an interaction between openib on Linux and Sun's Infiniband subnet manager (ibsrm) on Solaris. I was getting byte swap issues on the wire because I was using a x86_64 client with a sparc64 host (x86_64 is little endian, sparc64 big).

I thought it might be a problem with ibsrm, so I asked Jeremy from the East coast to take a look. While he was setting up a Linux machine to test with it, I thought I would try a different tack.

If I ran openib on sparc64, there wouldn't be an endian issue, so I thought I might get a little further.

I needed to upgrade the firmware on the Tavor IB card and I had two options: pull the card and stick it into a Solaris box or try to get the recently open sourced tavor flash tool working on sparc64/Linux. The latter seemed more interesting, plus I wouldn't have to go into the lab and touch hardware :-) After Roland helped worked out a struct packing issue, the tool compiled and I took the leap. Luckily, it worked and I didn't end up with a dead card.

Next was to get the tavor driver working on sparc64. This required a bit more, as the openib code mostly assumed that the PCI configuration was x86ish. Sparc64 has an IOMMU and only allows you to DMA into a 32bit address space. In any event, once again Roland pulled through with a quick hack to take out the x86 assumptions the driver loaded properly.

Next on the plate: get IPoIB working with IBSRM.

(2004-09-15 18:35:17.0) Permalink