Monday September 24, 2007 Finding application memory leaks
Today, I unlocked my screensaver and found that I couldn't launch anything from the GNOME panel launchbar. Everything reported fork failing from "out of memory".
Hm.
A coworker had had a similar problem, and it went away when, in trying to debug it, we had to kill gnome-panel.
Well, let's look at gnome-panel's pmap output: gee, its heap is over 1GB. (!)
So, that's probably a memory leak. How to find it? With some quick IRCs to some helpful GNOME engineers (GMan and Laca, to the rescue, as so often happens), I discover that I can remove gnome-panel from my session so it doesn't auto-restart when I kill it, by using
gnome-session-remove gnome-panel
(from a command line, of course, because...I can't launch anything). First,
though, I copied down its current arguments for restart, which were
--session default1. Then, I'm ready to try out libumem.
Adam has a wonderful introduction, Jonathan Adams wrote a really nice reference on the topic of libumem and mdb, and there are plenty of other examples of use around. My particular method was to enter the command
LD_PRELOAD=/usr/lib/libumem.so.1 UMEM_DEBUG=default gnome-panel --session default1
and then use mdb -p $(pgrep gnome-panel) to start up an mdb and access libumem's debugging features.
::findleaks immediately showed some false hits, and ::umem_verify was clean, so I waited for a little while and saw no further leaks in ::findleaks. Then I started a terminal from my custom terminal launcher on the panel. Bam! New leaks from ::findleaks! After messing about with manual ::bufctl_audits on the bufctl addresses in ::findleaks output, I decided that ::help findleaks might be handy, and indeed, it showed the -d option which put it all together, showing stack backtraces for the leaked buffers, which allowed us to examine the code and spot the leaks.
GMan and I were quickly able to find a bug in libgnome-desktop and another in gnome-panel itself (a fix to one of the Sun patches for gnome-panel, in panel_lockdown_is_forbidden_key_file). I was able to verify that all the leaked buffers sprang from one of those two sources.
I can't even begin to tell you how much faster and more fun that was with libumem than without. If you have any sort of allocation problem at all, you should seriously consider libumem debugging...free with Solaris.
( Sep 24 2007, 09:59:18 PM PDT ) Permalink Comments [0]Ever had your system constantly transmitting network packets, and had a hard time finding out who or why?
Recently my system was sending loads and loads of DNS requests for a system I knew not to be on the network (an old system of mine that had been decommissioned). I saw the DNS requests with snoop(1m), but had no idea which process was doing the job.
Enter dtrace. 15 seconds of experimentation: first, look to see if there are functions being called with "udp" in their name:
dtrace -n 'fbt::*udp*:entry'
Yes, there certainly are. OK, modify the above to suppress the default print with -q, and add what I'm interested in, straight out of built-in variables:
dtrace -q -n 'fbt::*udp*:entry{printf("%s from %s(%d)\n", probefunc, execname, pid);}'
(Turns out it was automountd. A quick service disable/enable, and it stopped.) Another mysterious system behavior immediately found with dtrace. ( Jun 21 2007, 02:30:30 PM PDT ) Permalink
Making backspace be your default delete character
One consistently-annoying thing about Solaris is the default terminal control-character settings, which cause Backspace *not* to erase a character in most shells (without someone somewhere executing an "stty erase ^H" to fix it up). I'm not going to claim that the default is rational, or try to speculate about where it arises, although I will say that it's being looked at.
However, today, we discovered that it's relatively easy to change; the initial control-character settings are set up by the ldterm module (the "line-discipline" is the one that establishes all the normal character-editing modes that many shells use when using "cooked-mode" terminal I/O). ldterm, it turns out, reads them from a property in the /kernel/drv/options.conf file, called ttymodes. It's encoded, but it represents the full termio settings, as stty -g would output them. The default looks like this:
ttymodes="2502:1805:bd:8a3b:3:1c:7f:15:4:0:0:0:11:13:1a:19:12:f:17:16";
If you change that 7f (the ASCII code for Del) to 8 (the ASCII code for Backspace), and reboot, then Backspace works as you expect, as soon as you log in, in all shells, thank you very much, the way God intended. ( Oct 20 2006, 05:34:12 PM PDT ) Permalink Comments [4]
Dell USB keyboard volume keys hack
If you have a Dell SK-8125 keyboard (101-key, black, built in 2-port hub, with a silver band at the top and 8 silver application keys, the right three of which are Volume Up/Down/Mute), this hack might interest you.
It turns out Solaris attaches a hid driver instance to the special "consumer control" device that makes up the 8 multimedia keys, but there's no client driver module that interprets the data, and in fact if no one opens that hid device, the data isn't ever generated.
But if you *do* open it, it turns out reading the keys is easy...and since /dev/audioctl is also an easy way to do volume, the following silly userland program has me using volume keys, when I run it as dellusb /dev/usb/hid2.
Maybe if you have a similar keyboard, you can hack it, so I've left the debug dump. This Dell sends 4 bytes for make and break, and doesn't seem to change anything but the second byte, which is a bitmask of keys pressed... pretty easy.
#include <stdio.h>
#include <stdlib.h>
#include <unistd.h>
#include <fcntl.h>
#include <sys/audio.h>
#define VOLDOWN 128
#define VOLUP 64
#define MUTE 32
#define HOME 16
#define RELOAD 8
#define CANCEL 4
#define FORWARD 2
#define BACK 1
void volevent(unsigned char mask);
void dump_event(unsigned char mask);
int
main(int argc, char **argv)
{
int fd;
char errmsg[80];
unsigned char kbuf[4];
unsigned char mask;
if (argc < 2) {
fprintf(stderr, "usage: dellusb <hid-device>\n");
exit(1);
}
if ((fd = open(argv[1], O_RDONLY)) < 0) {
sprintf(errmsg, "open %s", argv[1]);
perror(errmsg);
exit(1);
}
while (1) {
read(fd, kbuf, sizeof(kbuf));
mask = kbuf[1];
if (!mask)
continue;
#if 0
dump_event(mask);
#else
if (mask & (VOLUP | VOLDOWN | MUTE))
volevent(mask);
#endif
}
}
void
volevent(unsigned char mask)
{
struct audio_info i;
static uint_t gain_at_mute = 0;
int fd;
/* get audio info, no matter what we have to do */
if ((fd = open("/dev/audioctl", O_RDWR)) < 0) {
perror("open /dev/audioctl");
return;
}
ioctl(fd, AUDIO_GETINFO, &i);
switch (mask & (VOLUP | VOLDOWN | MUTE)) {
case VOLUP:
i.play.gain += 5;
if (i.play.gain > AUDIO_MAX_GAIN)
i.play.gain = AUDIO_MAX_GAIN;
break;
case VOLDOWN:
/* gain is a uint_t, requiring this silly dance */
if (i.play.gain <= 5)
i.play.gain = 0;
else
i.play.gain -= 5;
break;
case MUTE:
if (gain_at_mute) {
i.play.gain = gain_at_mute;
gain_at_mute = 0;
} else {
gain_at_mute = i.play.gain;
i.play.gain = 0;
}
break;
default:
goto out;
}
ioctl(fd, AUDIO_SETINFO, &i);
out:
close(fd);
return;
}
void
dump_event(unsigned char mask)
{
if (mask & VOLDOWN) printf("VOLDOWN ");
if (mask & VOLUP) printf("VOLUP ");
if (mask & MUTE) printf("MUTE ");
if (mask & HOME) printf("HOME ");
if (mask & RELOAD) printf("RELOAD ");
if (mask & CANCEL) printf("CANCEL ");
if (mask & FORWARD) printf("FORWARD ");
if (mask & BACK) printf("BACK ");
if (mask) printf("\n");
return;
}
( Jun 30 2006, 06:50:35 PM PDT )
Permalink
Diagnosing kernel hangs/panics with kmdb and moddebug
If you experience hangs or panics during Solaris boot, whether it's during installation or after you've already installed, using the kernel debugger can be a big help in collecting the first set of "what happened" information.
The kernel debugger is named "kmdb" in Solaris 10 and later, and is invoked by supplying the '-k' switch in the kernel boot arguments. So a common request from a kernel engineer starting to examine a problem is often "try booting with kmdb".
Sometimes it's useful to either set a breakpoint to pause the kernel startup and examine something, or to just set a kernel variable to enable or disable a feature, or enable debugging output. If you use -k to invoke kmdb, but also supply the '-d' switch, the debugger will be entered before the kernel really starts to do anything of consequence, so that you can set kernel variables or breakpoints.
So "booting with the -kd flags" is the key to "booting under the kernel debugger". Now, how do we do that?
To enter the debugger with Solaris 10, enter "b -kd" to the appropriate
prompt; this is slightly different whether you're installing or booting
an already-installed system:
Install:
Select the type of installation you want to perform:
1 Solaris Interactive
2 Custom JumpStart
3 Solaris Interactive Text (Desktop session)
4 Solaris Interactive Text (Console session)
Enter the number of your choice followed by the <ENTER> key.
Alternatively, enter custom boot arguments directly.
If you wait for 30 seconds without typing anything,
an interactive installation will be started.
Select type of installation:
Installed system:
Type b [file-name] [boot-flags] <ENTER> to boot with options
or i <ENTER> to enter boot interpreter
or <ENTER> to boot with defaults'
<<< timeout in 5 seconds >>>"
Select (b)oot or (i)nterpreter:
Either way, you'll drop into the kernel debugger in short order, which will announce itself with this prompt:
[0]>
(The number in square brackets is the CPU that is running the kernel debugger; that number might change for later entries into the debugger.)
If instead, you're doing this with Software Express build later than 05/05, where GRUB is used to boot Solaris, you add the -kd to the "kernel" line in the GRUB menu entry (you can edit GRUB menu entries for this boot by using the GRUB menu interface, and the 'e' (for edit) key).
[0]> moddebug/W 80000000 [0]> :cThat will give you debug output for each kernel module that loads. (see /usr/include/sys/modctl.h, near the bottom, for moddebug flag information. I find 0x80000000 is the only one I really ever use.)
[0]> $c
A few other very useful information commands during a panic are
::msgbufwhich will show you the last things the kernel printed onscreen, and
::statuswhich shows a summary of the state of the machine in panic.
[1]> 0::switch
There's obviously a lot more you can do with the kernel debugger, but these small tips will
sometimes help get from a "I have no idea what to do" to "I have a few ideas to try that might
let me continue to boot or install", which can make all the difference.
Technorati Tag:
opensolaris
solaris
( Jun 15 2005, 04:26:17 PM PDT )
Permalink
Comments [5]
PCI device identification and driver binding in Solaris
A PCI device has a bunch of device identification numbers associated
with it, which generic code can retrieve. I've listed them here in
most-specific to least-specific order, by their Solaris property name
(shown in the prtconf -pv output, which is why we always always always
ask for that when diagnosing driver-binding problems):
The revision number is only useful in conjunction with vendor-id, device-id.
Entry 3, the subsystem, is nearly useless for every purpose, as many machines now use the same subsystem ID for every motherboard device, and if not, at least the same subsystem-vendor-id. Sun had originally interpreted subsystem to be more specific than vendor-id, device-id, but that's not how the industry ended up adopting it. (as usual, the spec was unclear as to its intent).
The only things Solaris normally uses for binding device drivers are 2 and 4.
The way Solaris driver binding works is: for every element in the compatible property, in order, a) look for a same-named driver; if it's there, use it; if not, b) look for a same-named device alias, and get the driver field out of it; if it's there, use it. That's it. (Note that I'm specifically talking about Solaris, nothing to do with bootconf or the DCA.)
So most devices are bound through the vendor-id, device-id pair. Some devices and drivers are generic enough so that one driver is able to run an entire class of devices (say, for instance, pci-ide); in that case, the class-code can be used. But for the most part, vendor-id,device-id is what you want in /etc/driver_aliases, and it's always the right thing to talk about when you're trying to describe which device you have to someone else.
The Broadcom device aliases were added with both vendor-id, device-id and subsystem-vendor-id, subsystem-id, the intent being to try to bind the bge device driver only to particular boards and motherboards we had tested explicitly. (Opinions differed as to whether this was a good idea.) Since then, I believe the motion is back to just vendor-id,device-id, but if you see device aliases for bge with four numbers, that's why. They'll still work with two numbers, just not as pickily.
Now obviously this opens up the possibility that more than one alias might match for a particular set of numbers in the PCI device...but that's why we specify what's in the compatible property, not what's in the device. The compatible property is always constructed in a specific order, and as of s10_37, contains the following (intentionally-redundant) elements for PCI devices:
* (possibly) node-name (0)
* pciVVVV,DDDD.SSSS.ssss.RR (1)
* pciVVVV,DDDD.SSSS.ssss (2)
* pciSSSS,ssss (3)
* pciVVVV,DDDD.RR (4)
* pciVVVV,DDDD (5)
* pciclass,CCSSPP (6)
* pciclass,CCSS (7)
(VVVV is vendor-id, DDDD is device-id, SSSS is subsystem-vendor-id, ssss is subsystem-id, RR revision, CC major class number, SS subclass number, PP programming-interface-byte)
Form 0 is there for certain special devices, to "override" the normal matching, mostly older devices. Then, as you can see, we sorta go from most-specific to least-specific, which is the intent of the compatible property on any bus, PCI being no exception. The exception to that order is number 3, which had to be where it is because of the original definition of the compatible property in the original IEEE1275 spec, which all this is based on. But it's OK, because we (as noted above) virtually never use it for binding drivers anyway; we almost-always use 5 or 6/7, and sometimes 2.
Removing silly warnings from xmms's libSolaris.so output plugin
Here's a procedure for patching /opt/sfw/lib/xmms/Output/libSolaris.so, to get rid of the warnings that appear on the console if you're using a soundcard xmms doesn't know by name:
** WARNING **: solaris output: Unknown sound card type: SUNW,audio810 ** WARNING **: solaris output: Assuming capable of the bitstream
If you get those, and are annoyed, and don't want to be bothered rebuilding xmms with the latest plugin source, try this:
$ su - # cd /opt/sfw/lib/xmms/Output # cp libSolaris.so libSolaris.so.orig # mdb -w libSolaris.so > 442d?5B 0x442d: ff 75 b0 8d 83
If you don't see these values, stop now
> 442d?v e9 26 00 00 00 0x442d: 0xff = 0xe9 0x442e: 0x75 = 0x26 0x442f: 0xb0 = 0x0 0x4430: 0x8d = 0x0 0x4431: 0x83 = 0x0 > $q
Restart xmms, and you should be done. If anything goes wrong, of course, just copy libSolaris.so.orig back to libSolaris.so.
(What this does is to add a "jmp" around the two calls to g_warning() (really a #define for g_log()) for those two warnings.) ( Jun 08 2005, 11:47:32 PM PDT ) Permalink Comments [0]
Patch for dmidecode-2.[6-8] to run on Solaris x86
Here's a patch for dmidecode, to allow it to run on Solaris x86. The patch applies to at least 2.6 and 2.8, so I'm assuming 2.7 works too.
------- config.h ------- --- /tmp/sccs.shaGFO Fri Apr 1 22:29:27 2005 +++ config.h Wed Mar 16 18:26:52 2005 @@ -6,8 +6,12 @@ #ifdef __BEOS__ #define DEFAULT_MEM_DEV "/dev/misc/mem" #else +#if defined (__sun) && (defined(__i386) || defined(__amd64)) +#define DEFAULT_MEM_DEV "/dev/xsvc" +#else #define DEFAULT_MEM_DEV "/dev/mem" #endif +#endif /* Use mmap or not */ #ifndef __BEOS__
That's it. That's all there is to it. (dmidecode and biosdecode are useful for giving you manufacturer-type info encoded into your BIOS about your machine, its BIOS, jumpers, devices, slots, memory sticks, etc.
Note that in Solaris Nevada and a future Solaris 10 Update, the "smbios" command takes over this functionality. ( Apr 01 2005, 10:30:28 PM PST ) Permalink Comments [5]
Sometimes CSS is annoying. (We have a web-based tool for reviewing code, and when its output is put in a site that has CSS, its appearance is changed enough to be annoying).
So here are two tips:
System events comprise a mechanism for kernel code to signal up the tree to anyone who might be listening: other kernel agents, userland code, etc. Here's a very stupid demo program (Solaris-10-or-later only), just to save you some typing, that demonstrates what an event looks like. Try running this and then plugging/unplugging a USB device.
See libsysevent(3LIB) for a description of the userland interface used in this demo program, and syseventd(1M) for a description of the userland daemon.
If this little sample piques your interest, check out what you could do with syseventadm(1m) and some shell scripts, perhaps involving zenity(1)...
Compile with cc evprint.c -o evprint -lsysevent -lnvpair
#include <unistd.h>
#include <libsysevent.h>
#include <libnvpair.h>
typedef void (sehfn)(sysevent_t *ev);
sehfn handler;
int
main(int argc, char **argv)
{
sysevent_handle_t *seh;
const char *subclass_list[] = {"EC_SUB_ALL"};
seh = sysevent_bind_handle(handler);
if (seh == NULL) {
perror("sysevent_bind_handle");
exit(1);
}
if (sysevent_subscribe_event(seh, EC_ALL, subclass_list, 1) != 0) {
perror("sysevent_subscribe_event");
exit(1);
}
while (1)
pause();
}
void
handler(sysevent_t *ev)
{
nvlist_t *nvlist;
nvpair_t *nvpp;
char *class, *subclass;
unsigned int n, i;
boolean_t bv, *ba;
int8_t i8v, *i8a;
int16_t i16v, *i16a;
int32_t i32v, *i32a;
int64_t i64v, *i64a;
uint8_t ui8v, *ui8a;
uint16_t ui16v, *ui16a;
uint32_t ui32v, *ui32a;
uint64_t ui64v, *ui64a;
char *str, **sa;
str = (char *)malloc(100);
class = sysevent_get_class_name(ev);
subclass = sysevent_get_subclass_name(ev);
printf("\n*** event: class '%s', subclass '%s'\n", class, subclass);
if (sysevent_get_attr_list(ev, &nvlist) != 0) {
printf("no nvlist\n");
return;
}
nvpp = NULL;
while ((nvpp = nvlist_next_nvpair(nvlist, nvpp)) != NULL) {
printf("%s: ", nvpair_name(nvpp));
switch (nvpair_type(nvpp)) {
case DATA_TYPE_BOOLEAN:
printf("true\n");
break;
case DATA_TYPE_BOOLEAN_VALUE:
nvpair_value_boolean_value(nvpp, &bv);
printf("boolean %s\n", bv ? "false" : "true");
break;
case DATA_TYPE_INT8:
nvpair_value_int8(nvpp, &i8v);
printf("int8 %d\n", i8v);
break;
case DATA_TYPE_BYTE:
case DATA_TYPE_UINT8:
nvpair_value_uint8(nvpp, &ui8v);
printf("uint8 %d\n", ui8v);
break;
case DATA_TYPE_INT16:
nvpair_value_int16(nvpp, &i16v);
printf("int16 %d\n", i16v);
break;
case DATA_TYPE_UINT16:
nvpair_value_uint16(nvpp, &ui16v);
printf("uint16 %d\n", ui16v);
break;
case DATA_TYPE_INT32:
nvpair_value_int32(nvpp, &i32v);
printf("int32 %d\n", i32v);
break;
case DATA_TYPE_UINT32:
nvpair_value_uint32(nvpp, &ui32v);
printf("uint32 %d\n", ui32v);
break;
case DATA_TYPE_INT64:
nvpair_value_int64(nvpp, &i64v);
printf("int64 %d\n", i64v);
break;
case DATA_TYPE_UINT64:
nvpair_value_uint64(nvpp, &ui64v);
printf("uint64 %d\n", ui64v);
break;
case DATA_TYPE_STRING:
nvpair_value_string(nvpp, &str);
printf("string '%s'\n", str);
break;
case DATA_TYPE_NVLIST:
printf("nvlist\n");
break;
case DATA_TYPE_BOOLEAN_ARRAY:
printf("boolean array: {");
nvpair_value_boolean_array(nvpp, &ba, &n);
for (i = 0; i < n; i++)
printf("%s ", ba[i] ? "true" : "false");
printf("\n");
break;
case DATA_TYPE_BYTE_ARRAY:
printf("byte array: {");
nvpair_value_byte_array(nvpp, &ui8a, &n);
for (i = 0; i < n; i++)
printf("0x%x ", ui8a[i]);
printf("}\n");
break;
case DATA_TYPE_INT8_ARRAY:
printf("int8 array: {");
nvpair_value_int8_array(nvpp, &i8a, &n);
for (i = 0; i < n; i++)
printf("%d ", i8a[i]);
printf("}\n");
break;
case DATA_TYPE_UINT8_ARRAY:
printf("uint8 array: {");
nvpair_value_uint8_array(nvpp, &ui8a, &n);
for (i = 0; i < n; i++)
printf("0x%x ", ui8a[i]);
printf("}\n");
break;
case DATA_TYPE_INT16_ARRAY:
printf("int16 array: {");
nvpair_value_int16_array(nvpp, &i16a, &n);
for (i = 0; i < n; i++)
printf("%d ", i16a[i]);
printf("}\n");
break;
case DATA_TYPE_UINT16_ARRAY:
printf("uint16 array: {");
nvpair_value_uint16_array(nvpp, &ui16a, &n);
for (i = 0; i < n; i++)
printf("%d ", ui16a[i]);
printf("}\n");
break;
case DATA_TYPE_INT32_ARRAY:
printf("int32 array: {");
nvpair_value_int32_array(nvpp, &i32a, &n);
for (i = 0; i < n; i++)
printf("%d, ", i32a[i]);
printf("}\n");
break;
case DATA_TYPE_UINT32_ARRAY:
printf("uint32 array: {");
nvpair_value_uint32_array(nvpp, &ui32a, &n);
for (i = 0; i < n; i++)
printf("%d, ", ui32a[i]);
printf("}\n");
break;
case DATA_TYPE_INT64_ARRAY:
printf("int64 array: {");
nvpair_value_int64_array(nvpp, &iamp;64a, &n);
for (i = 0; i < n; i++)
printf("%lld, ", i64a[i]);
printf("}\n");
break;
case DATA_TYPE_UINT64_ARRAY:
printf("uint64 array: {");
nvpair_value_uint64_array(nvpp, &ui64a, &n);
for (i = 0; i < n; i++)
printf("%lld, ", ui64a[i]);
printf("}\n");
break;
case DATA_TYPE_STRING_ARRAY:
printf("string array: {");
nvpair_value_string_array(nvpp, &sa, &n);
for (i = 0; i < n; i++)
printf("'%s', ", sa[i]);
printf("}\n");
break;
case DATA_TYPE_NVLIST_ARRAY:
printf("nvlist_array\n");
break;
default:
printf("type unknown\n");
break;
}
}
}
( Mar 31 2005, 06:56:05 PM PST )
Permalink
Comments [2]
prtpci: digest and display prtconf -pv output
Here's a tool (prtpci.tar.Z) for digesting PCI information from prtconf -pv output.
There are several tools around that will show you a PCI manifest; this one
It's useful to me, and I hope it is useful to you. Here's a little sample output:
3/0xb/0 1095,3114 (1095,3114)
Silicon Image, Inc. (formerly CMD Technology Inc) SiI 3114 [SATALink/SATARaid] Serial ATA Controller
class 1/80/0: Mass storage controller/Unknown mass storage controller
BAR[0]: I/O 0xbc00 0x8
BAR[1]: I/O 0xb802 0x1
BAR[2]: I/O 0xb400 0x8
BAR[3]: I/O 0xb002 0x1
BAR[4]: I/O 0xac00 0x10
BAR[5]: I/O 0xfc8ffc00 0x400
ROM: 32-bit memory 0xfc800000 0x80000
3/0xc/0 104c,8023 (10f1,2885)
Texas Instruments TSB43AB22/A IEEE-1394a-2000 Controller (PHY/Link)
class c/0/10: Serial bus controller/FireWire (IEEE 1394)
BAR[0]: 32-bit memory 0xfc8ff000 0x800
BAR[1]: 32-bit memory 0xfc8f8000 0x4000
( Mar 17 2005, 06:43:47 PM PST )
Permalink
Comments [7]