Dan Mick's Little Shop of Hints

Diagnosing kernel hangs/panics with kmdb and moddebug

If you experience hangs or panics during Solaris boot, whether it's during installation or after you've already installed, using the kernel debugger can be a big help in collecting the first set of "what happened" information.

The kernel debugger is named "kmdb" in Solaris 10 and later, and is invoked by supplying the '-k' switch in the kernel boot arguments. So a common request from a kernel engineer starting to examine a problem is often "try booting with kmdb".

Sometimes it's useful to either set a breakpoint to pause the kernel startup and examine something, or to just set a kernel variable to enable or disable a feature, or enable debugging output. If you use -k to invoke kmdb, but also supply the '-d' switch, the debugger will be entered before the kernel really starts to do anything of consequence, so that you can set kernel variables or breakpoints.

So "booting with the -kd flags" is the key to "booting under the kernel debugger". Now, how do we do that?

Kernel debugging with GRUB-boot systems

On modern Solaris and OpenSolaris systems, GRUB is used to boot; to enable the kernel debugger, you add -kd arguments to the "kernel" (or "kernel$") line in the GRUB menu entry. When presented with the GRUB menu, hit 'e' to edit the entry, highlight the kernel line, and hit 'e' again to edit it; add the -kd arguments just after the /platform/i86pc/kernel/$ISADIR/unix argument, so that it says

kernel$ /platform/i86pc/kernel/$ISADIR/unix -kd

and then hit 'b' to boot that edited menu entry. '-k' means "start the debugger"; '-d' means "immediately enter the debugger after loading the kernel". After some booting status, you'll see the kernel debugger announce itself like this:

[0]>

(The number in square brackets is the CPU that is running the kernel debugger; that number might change for later entries into the debugger.)

Now we're in the kernel debugger

There are two good reasons to run under the kernel debugger:

1. If we panic, the panic can be examined before reboot; you can get stack backtraces and get some idea of which section of code might be at fault.
2. Now we can set kernel variables, set breakpoints, etc. to affect the kernel run.

Obviously, there's a lot you can do in a kernel debugger, and I'm only touching on it here, but here are two good ones:

1. For investigating hangs: try turning on module debugging output. You can set the value of a kernel variable by using the '/W' command ("write a 32-bit value"). Here's how you set moddebug to 0x80000000, and then continue execution of the kernel:

   [0]> moddebug/W 80000000
   [0]> :c
   

   That will give you debug output for each kernel module that loads. (see /usr/include/sys/modctl.h, near the bottom, for moddebug flag information. I find 0x80000000 is the only one I really ever use.)
2. To collect information about panics: when the kernel panics, it will drop into the debugger, and print some interesting information; however, usually the most interesting thing, first, is the stack backtrace; this shows, in reverse order, all the functions that were active at the time of panic. To generate a stack backtrace, use

   [0]> $c
   

   A few other very useful information commands during a panic are

   ::msgbuf

   which will show you the last things the kernel printed onscreen, and

   ::status

   which shows a summary of the state of the machine in panic.
3. If you're running the kernel while the kernel debugger is active, and you experience a hang, you may be able to break into the debugger to examine the system state; you can do this by pressing the <F1> and <A> keys at the same time (a sort of "F1-shifted-A" keypress). (On SPARC systems, this key sequence is <Stop>-<A>.) This should give you the same debugger prompt as above, although on a multi-CPU system you may see the CPU number in the prompt is something other than 0. Once in the kernel debugger, you can get a stack backtrace as above; you can also use ::switch to change the CPU and get stack backtraces on the different CPU, which might shed more light on the hang. For instance, if you break into the debugger on CPU 1, you could switch to CPU 0 with

   [1]> 0::switch
   

There's obviously a lot more you can do with the kernel debugger, but these small tips will sometimes help get from a "I have no idea what to do" to "I have a few ideas to try that might let me continue to boot or install", which can make all the difference.

Technorati Tag: opensolaris solaris ( Jun 15 2005, 04:26:17 PM PDT ) Permalink Comments [5]

PCI device identification and driver binding in Solaris

PCI device identification and driver binding in Solaris

A PCI device has a bunch of device identification numbers associated with it, which generic code can retrieve. I've listed them here in most-specific to least-specific order, by their Solaris property name (shown in the prtconf -pv output, which is why we always always always ask for that when diagnosing driver-binding problems):

1. revision (not useful on its own)
2. vendor-id, device-id (the usual lone source of driver binding)
3. subsystem-vendor-id, subsystem-id (the usual source of the "name" property, and hence the usual source of the /devices node name)
4. class-code

The revision number is only useful in conjunction with vendor-id, device-id.

Entry 3, the subsystem, is nearly useless for every purpose, as many machines now use the same subsystem ID for every motherboard device, and if not, at least the same subsystem-vendor-id. Sun had originally interpreted subsystem to be more specific than vendor-id, device-id, but that's not how the industry ended up adopting it. (as usual, the spec was unclear as to its intent).

The only things Solaris normally uses for binding device drivers are 2 and 4.

The way Solaris driver binding works is: for every element in the compatible property, in order, a) look for a same-named driver; if it's there, use it; if not, b) look for a same-named device alias, and get the driver field out of it; if it's there, use it. That's it. (Note that I'm specifically talking about Solaris, nothing to do with bootconf or the DCA.)

So most devices are bound through the vendor-id, device-id pair. Some devices and drivers are generic enough so that one driver is able to run an entire class of devices (say, for instance, pci-ide); in that case, the class-code can be used. But for the most part, vendor-id,device-id is what you want in /etc/driver_aliases, and it's always the right thing to talk about when you're trying to describe which device you have to someone else.

The Broadcom device aliases were added with both vendor-id, device-id and subsystem-vendor-id, subsystem-id, the intent being to try to bind the bge device driver only to particular boards and motherboards we had tested explicitly. (Opinions differed as to whether this was a good idea.) Since then, I believe the motion is back to just vendor-id,device-id, but if you see device aliases for bge with four numbers, that's why. They'll still work with two numbers, just not as pickily.

Now obviously this opens up the possibility that more than one alias might match for a particular set of numbers in the PCI device...but that's why we specify what's in the compatible property, not what's in the device. The compatible property is always constructed in a specific order, and as of s10_37, contains the following (intentionally-redundant) elements for PCI devices:

         *   (possibly) node-name       (0)
         *   pciVVVV,DDDD.SSSS.ssss.RR  (1)
         *   pciVVVV,DDDD.SSSS.ssss     (2)
         *   pciSSSS,ssss               (3)
         *   pciVVVV,DDDD.RR            (4)
         *   pciVVVV,DDDD               (5)
         *   pciclass,CCSSPP            (6)
         *   pciclass,CCSS              (7)

(VVVV is vendor-id, DDDD is device-id, SSSS is subsystem-vendor-id, ssss is subsystem-id, RR revision, CC major class number, SS subclass number, PP programming-interface-byte)

Form 0 is there for certain special devices, to "override" the normal matching, mostly older devices. Then, as you can see, we sorta go from most-specific to least-specific, which is the intent of the compatible property on any bus, PCI being no exception. The exception to that order is number 3, which had to be where it is because of the original definition of the compatible property in the original IEEE1275 spec, which all this is based on. But it's OK, because we (as noted above) virtually never use it for binding drivers anyway; we almost-always use 5 or 6/7, and sometimes 2.

---

OpenSolaris Technorati Tag: OpenSolaris Technorati Tag: Solaris ( Jun 14 2005, 11:53:28 AM PDT ) Permalink Comments [3]

Removing silly warnings from xmms's libSolaris.so output plugin

Here's a procedure for patching /opt/sfw/lib/xmms/Output/libSolaris.so, to get rid of the warnings that appear on the console if you're using a soundcard xmms doesn't know by name:

** WARNING **: solaris output: Unknown sound card type: SUNW,audio810

** WARNING **: solaris output: Assuming capable of the bitstream

If you get those, and are annoyed, and don't want to be bothered rebuilding xmms with the latest plugin source, try this:

$ su -
# cd /opt/sfw/lib/xmms/Output
# cp libSolaris.so libSolaris.so.orig
# mdb -w libSolaris.so
> 442d?5B

0x442d:         ff      75      b0      8d      83 

If you don't see these values, stop now

> 442d?v e9 26 00 00 00
0x442d:         0xff    =       0xe9
0x442e:         0x75    =       0x26
0x442f:         0xb0    =       0x0
0x4430:         0x8d    =       0x0
0x4431:         0x83    =       0x0
> $q

Restart xmms, and you should be done. If anything goes wrong, of course, just copy libSolaris.so.orig back to libSolaris.so.

(What this does is to add a "jmp" around the two calls to g_warning() (really a #define for g_log()) for those two warnings.) ( Jun 08 2005, 11:47:32 PM PDT ) Permalink Comments [0]