Matt Ingenthron's Stream of Consciousness

X2100 SMDC best practices for serial over lan with ipmitool

Well, John, you beat me to the punch. I had a great blog entry hanging out in my grey matter on using Solaris 10 with the SMDC. Still, there are a few nuggets I can relay.

First, the prereqs. This all assumes:

• Solaris 10 06/06 (but pretty much any Solaris with new boot [i.e. GRUB] should be the same)
• BIOS rev 1.1.1 and SMDC firmware 1.14 (all off of the 1.4 supplemental CD, also required if you plan to use ZFS)
• ipmitool 1.8.8 (the stuff in Solaris 10 U2 is 1.8.4, it requires 1.8.5 or later)

The release notes, are mostly complete, but have a couple of errors and don't necessarily explain the "why" in some areas. I've already asked for some fixes against the release notes. Blogs are faster though. :) Just don't expect it to be a maintained doc. Check that link for the latest. The revision of the doc as of this writing is -15.

Now the errors. The release notes tell you there are edits required in the GRUB menu.lst. It turns out, you don't need to make any edits to GRUB. They tell you to set GRUB to use the serial device. That's not necessary. So the lines:

serial --unit=0 --speed=9600
terminal serial 

aren't necessary. GRUB will talk to the BIOS, and the BIOS will talk to the SMDC if you've set it up that way per the BIOS settings mentioned release notes.

The release notes also tell you to edit the multiboot line as follows:

kernel /platform/i86pc/multiboot -B console=ttya

That change isn't necessary if you edit the bootenv.rc (either manually or through the eeprom command). Multiboot reads the bootenv.rc, so no changes are necessary there.

Finally, this one is a bit of a major error, you need to use bootadm to determine where the menu.lst is. You cannot necessarily rely on it being in the directory mentioned. It's not as major an error if you read the comments in the file-- but I find not everyone reads documentation. :) Besides, if you take my approach, you don't end up changing it at all anyway.

The other thing required at this point is to edit the asy.conf. This is due to a BIOS bug on the X2100. If you have the BIOS in "redirect over serial" mode, it puts the real, onboard serial device entry in the ACPI table. If you put it in "redirect over SMDC" mode, it does not put the entry for the serial device in the ACPI table, so you need to tickle the asy driver to use the ISA device. This will change in OpenSolaris shortly (if not already) and will be in Solaris 10U3, it's being tracked as Bug ID 6416708. Other platforms have similar issues, so the driver is being modified to look for the common serial ports. The ISA address and interrupts of the serial ports haven't changed for many, many years, so this probably makes sense. I'm told this affects the IBM BladeCenter and Dell PowerEdge 1855MC system's virtual serial consoles (though, those use ttyb).

The downside is that until this fix is available, without a specially built boot archive or modified jumpstart, you won't be able to perform an install over the SMDC. So, until this fix hits, install over the serial or with a BIOS console (keyboard/mouse/video), then make the change to the asy.conf.

Finally, I believe (but don't know for certain) you need to run bootadm update-archive to be sure you'll have a nice fresh archive for your next boot. Solaris will do this on shutdown, but I figure it's best to do it now, just in case you don't have a clean shutdown.

After that, you'll have a nice, working impitool serial over lan. A command similar to:

ipmitool -H  -U Admin tsol

gave me a nice working serial over lan.

There is one lingering issue though. The bug mentioned in the release notes about the bge driver had been fixed, but there seems to continue to be a bge problem. I found that after using the tsol for a little while, the whole SMDC would disappear unless I excluded drv/bge. So, add the following to /etc/system before you reboot:

exclude: drv/bge

That will elimininate sideband usage of the bge from Solaris, but you'll have a stable SMDC.

---

Update 2006-07-26: A new bug for the bge sideband usage has been opened as Bug ID 6453203

---

( Jul 25 2006, 02:41:35 PM PDT ) Permalink