Wednesday September 30, 2009

CPU used up after an upgrade

Just a quick blog, so that hopefully when you google for this you will find something and not spend hours debugging it.

I just upgraded to OpenSolaris B123, and everything worked fine, however when I logged out and back in again (something I rarely do, so it might have been present before), the machine was on it's knees. As usual I started the debugging process

# prstat

show svc.configd at the top

# truss -p `pgrep svc.configd`

 just showed it running door_return() a lot. This implies that something is talking to svc.configd.

On reflection I could have dtraced the door_call and door_return calls and gathered the execname of the process it was talking to, but I spotted before that, that my pid values were going up quickly. So clearly there was a process starting up, talking to svc.configd and exiting and repeating.

 After a few minutes digging around I found

desktop-print-management-applet calling ospm (OpenSolaris print manager) which was doing svcprop.

A colleague then pointed me to http://defect.opensolaris.org/bz/show_bug.cgi?id=10863 which described the problem and gives a workaround

Fortunately this should be fixed soon

Tuesday September 15, 2009

Debugging fmd plugins

This is really just a note to myself as I keep forgetting the options. I'm developing some new plugins for fmd. When they don't work, there's loads of additional data you can get out which isn't there by default, and you can add debug print statements to your code like

fmd_hdl_debug(hdl, "Crash dump instance %ld\n",
	    cdp->scd_panic_instance);

But these won't be visible unless you fmd in a debug mode. First you need to disable fmd

# svcadm disable fmd

Then you need to run fmd with the right options. Running it in the forground also helps with starting restarting it

# /usr/lib/fm/fmd/fmd -o fg=true -o client.debug=true

If you want to see what fmd it's self is doing add the -o debug=all flag

# /usr/lib/fm/fmd/fmd -o fg=true -o client.debug=true -o debug=all

 Then you see those lovely debug messages.