Chris Quenelle's Weblog
Thoughts on developer tools.

Thursday October 05, 2006

malloc interposition can't possibly work reliabily

In Solaris, many of the routines called from libc are "direct bound" so that references from inside libc will always find the function implementations that are inside libc. This approach prevents outside libraries from interposing (substituting) different implmentations of common functions. The largest exception to this is the malloc family of routines. The malloc routines (when called by libc, for example from strdup) MUST call an externally supplied malloc routine, if one is supplied via LD_PRELOAD or library link ordering.

There is a huge gotcha related to malloc interposing. If you get a pointer from malloc, you have to free it using the free routine in the same library that allocated it. But how do you guarantee that? If every program has libc, then every program will have at least one allocator in it. Any program that uses libumem will have at least two (one from libumu and one from libc). If the user wants to LD_PRELOAD their own memory checker library, it just gets worse.

It gets even worse because malloc libraries implement many additional routines to allocate memory. Let's say my app calls valloc in libc. Let's say I want to interpose libmymalloc because it has a spiffy new memory checker that I want to use. Now let's say libmymalloc doesn't include a definition for valloc. My app will crash, because valloc gets memory from the libc pool, and free will free it to the libmymalloc pool.

At this point there are people who will say: "Easy, just make sure they all implement the same set of functions." Well yes, that would solve the problem, if there were a way to do this. But there is no standard for what this list of routines is. Memory allocation libraries are useful because they offer additional functionality beyond the plain malloc and free in libc. So they will always be adding functions that are not in anyone else's implementation. If I write my app to use libmalloc_alpha.so, and someone interposes libmalloc_beta.so, then all the custom functions in libmalloc_alpha.so that I was calling will still go to libmalloc_alpha.so, but all the customary ones will go to libmalloc_beta.so. The result is undefined.

Unfortunately the idea that you can replace a memory allocator library by just interposing a different one is a widely known "fact". You can read about how this bit the Solaris libraries in bugid 4846556. The problem came up in comp.unix.solaris recently as well.

Dbx can crash with g++ symbols

Sun Studio 11 dbx (and the one in latest express release) has a bug that can make dbx crash when it reads g++ information. (The problem is in the g++ demangler library that dbx uses) The bug was fixed recently, but so far it is not patched in Sun Studio 11.  Nobody with a support contract seems to have run into the problem yet.  It should be fixed in the next express release, but that is probably a month or two away.

If dbx crashes on a program that has g++ code in it, look at the stack trace from the dbx core file,  and if it crashed inside some functions like:

__1cU__gnu2_lib_demanglerKparse_type6MrnK__lcstring_b_v_

then you're probably hitting this bug.  There are two bugids associated with the same problem. 6457910 and 6390216.  Remember, if you have a service contract, and you run into these bugs, you can request a patch. Unfortunately the rest of our users have to wait for the next express release.  I'm trying to push for a Sun Studio 11 patch, but so far it hasn't happened.