Solaris Kernel Nomad

kmdb is the replacement for kadb, and serves as the in-situ kernel debugger for Solaris 10 and beyond. kadb has been removed. It is no more. Periodically I visit its grave to make sure that the nails are still in tightly. Conveniently, kadb's grave is next to Lisa Fisher's, so I can check both at the same time.

First, a few words about the major differences between kmdb and kadb, and then a brief guide to the source layout for those interested in visiting that twisty little maze. This description assumes that the reader is already familiar with the use of kmdb, and further assumes that the reader is one of the four people on the planet who cares how it does its dirty work.

One of the primary design goals behind kmdb was to make it dynamically loadable. Users should be able to load it when they want it, and should similarly be able to make it go away when they're done. This is in contrast to kadb, which could only be loaded at boot, and couldn't be unloaded without a reboot. Another main goal was to move as much of the platform- and cpu-specific stuff into the kernel as humanly possible. By doing this, we stabilize the debugger, and put the hardware-specific goo where it can be handled by an entity (the kernel) designed to deal with cpu-specific and platform-specific modules. This is in contrast to kadb, which either ignored the problem completely or delivered separate platform-specific binaries to get around the problem.

Boot is a terrifying place. We'll stick with SPARC for this description, because a) SPARC boot is less scary than x86, and b) SPARC kmdb has to care about cpu-specific and platform-specific stuff to a much greater extent than does x86 kmdb.

On a SPARC machine, the booter does its thing, loads a standalone (normally unix), loads the interpreter for that standalone (krtld), jumps to the interpreter, and away we go. To allow for the debugging of the earliest stages of boot (i.e. krtld before it passes control to unix`_start), kadb is itself a standalone. When the user utters boot kadb, the booter loads the standalone called kadb. kadb doesn't have an interpreter, so the booter passes control directly to it. kadb does its initialization, and then must duplicate everything the booter would have done to get unix loaded. That is, it loads unix, loads krtld, and finally jumps to krtld.

As if that wasn't enough, there's yet another wrinkle. kadb can't be unloaded, so whichever kadb gets loaded (remember, we don't know which unix will be loaded yet) needed to be able to deal with both 32-bit and 64-bit kernels. In practice, that meant a 64-bit kadb that knew how to talk to a 32-bit kernel.

kmdb is designed to make all of that go away. kmdb is a normal kernel module, and is loaded by krtld after unix, genunix, the CPU module, and the platmod have been loaded. This means that kmdb can't be used to debug early stages of krtld, but the advantages outweigh that cost. The bitness of the kernel is known at that point, so the corresponding version of kmdb can be loaded. The CPU module and platform modules are loaded, so kmdb can link against them, and can use functions that live in those modules. kmdb doesn't have to load anything, so it doesn't essentially have to be its own linker the way kadb did. Finally, the fact that it's a normal kernel module (actually two, but we won't get into that here) means that it can be relocated, and can thus be loaded well after boot by mdb -K (this is referred to as mod-loaded kmdb). Unfortunately, due to the way memory is allocated at the beginning of time, boot-loaded kmdb can't be unloaded, but mod-loaded kmdb can.

kmdb lives in usr/src/cmd/mdb, and is intertwined with mdb. At present, the Makefiles are a bit of a mess, and need to be simplified again. sparc/v9/kmdb/sun4u and sparc/v9/kmdb/sun4v are of particular concern, and will go away again in the fullness of time. kmdbmod should really be built from somewhere under sun4u and sun4v, but currently isn't. While there are a few #ifdefs here and there, the overwhelming majority of source code segregation between debuggers is done by simply keeping the differences in separate files. Due in no small part to the very clean mdb API layers, this separation can be accomplished with very little duplication of code. Rather than having tortured common source files that look like our headers, the two debuggers simply provide different implementations of the various layers. All very neat and tidy. Except, well, where it's not.

As described above, kmdb must be loadable both at boot and on-demand. The easiest way to accomplish this was to separate kmdb into two modules -- kmdbmod and kmdb. The former, which lives in misc, contains the debugger itself, as well as virtually all of the supporting goo. The latter, which lives in drv, exists solely to let kmdbmod present a driveresque face to the world. The driver publishes the /devices entry that eventually becomes /dev/kmdb, and passes the debugger-control ioctls over to kmdbmod. When the debugger is loaded at boot, krtld loads kmdbmod as a primary. The kmdb driver module loads later, and attaches itself to the loaded kmdbmod. When the debugger is loaded via mdb -K, the driver module loads first, and pulls in kmdbmod as a dependency. The source for the driver module lives in usr/src/uts/common/kmdb. The driver module is referred to as kdrv, to avoid confusion with what we call the driver portion of kmdbmod.

All the magic happens in kmdbmod, which is itself subdivided into two parts. There's the bit referred to as "the debugger", which contains all of the code that's run when the debugger has control of the world. The debugger is highly self-contained, and doesn't reference any symbols outside of itself. Any communication with the outside world is done via ops vectors. The remaining portion of kmdbmod is referred to as "the driver" (not to be confused with kdrv, the kmdb driver module), and exists for two reasons. First, it configures the world for kmdb (allocates a memory region for the debugger, sets boothowto, etc). Upon unload, it reverses this process. Second, the driver performs tasks requested of it by the debugger -- tasks that can't be performed by the debugger because they must be accomplished while the world is running. An example of such a task is the loading of a debugger module (dmod), which requires the services of the module subsystem, I/O devices, and so forth.

The division of code between the debugger and the driver (again, within kmdbmod) is pretty simple. Routines that comprise the driver have names that begin with kctl_. Everything else lives in the debugger.

But alas, work calls. A future blog entry will, if there's interest, discuss the following truly thrilling topics: