Morgan Herrington's Blog Adventures in Porting and Tuning

I tend to take for granted that applications will be compatible between successive Solaris releases, but this behavior also depends on the applications "playing by the rules". On the theory that it's sometimes useful to see what went wrong, I thought I might describe a few situations where software failed on Solaris 10 (because of faulty assumptions in the applications).

Abusing the ABI: One of the first portability failures I experienced on Solaris 10 was when an application core dumped because it used the SPARC register %g7. This register has always been reserved for system use by the SPARC ABI, but for well over a decade developers have known that it was only used by the thread library (to point to the user-level thread structure). This lead to the assumption that it could freely be used by single-threaded programs. However, because of the Process Model Unification in Solaris 10, even single threaded applications now have to obey the restriction.

Fortunately, this problem wasn't too difficult to diagnose because the instruction which caused the segmentation violation was using register %g7. Still, this illustrates that point that code should be written based on the documentation (using guaranteed interfaces) rather than on how things are currently implemented (even if they've been that way for a long time).

Assuming Implementation Details: One of the software vendors I've been working with develops on Solaris 8 and then qualifies their applications for Solaris 8, 9, and 10. Because of that trailing development environment, I only recently ran across a change which has been in the Solaris linker for a few years. As an optimization, the linker compresses an object file's symbol table by merging symbols which have the same tail string. My understanding is that this is a significant optimization for the system libraries because many symbols are nearly duplicated, for example, "memcpy" and "_memcpy".

This linker change was invisible to most applications, but ran afoul of a particular vendor's obfuscation utility which is used to make it more difficult to reverse engineer binaries. When every symbol still had a unique string table entry, the utility could simply replace a name like "get_license" with some undecipherable string like "xr56j". Unfortunately, when the string table is compressed, this technique also garbles any symbol which has the same tail string, for example, "license", or even "nse".

Fortunately, the linker provides an easy work around in the form of the -z nocompstrtab option (which inhibits the default compression of the string table).

Neglecting Optimizations: Another optimization-related failure occurred with an improvement in the Sun Studio compilers. In this case, the vendor includes some support functions which are not called during normal operation (but are available, for example, to customer support engineers from within "dbx"). [This is not exactly how the software vendor uses these symbols, but this explanation will still illustrate the problem.] The support functions are encapsulated into an archive library and pulled into the main application with references like:

    static int DEBUG=0;
    ...snip...
    if (DEBUG==1) {
        diagnostic_function1();
        diagnostic_function2();
        diagnostic_function3();
    }

In all of their previous build environments (including Solaris 8 with Sun Studio 8), this successfully referenced the functions and linked them in from the archive library. However, on Solaris 10 using Sun Studio 11, the compiler correctly notes that "DEBUG" will always logically be zero, so the references are optimized away (thus the functions are not even pulled in from the archive library). This happened to be a quiet failure because everything seemed fine during the build, but failed later when the diagnostic functions were needed.

A bit of inspection with /usr/ccs/bin/nm revealed the problem, and the software vendor agreed that the compiler was well within its rights to remove these references. The point, though, is that the code could only work by depending on the lower level of optimization provided by the older compiler releases.

So What? These are just cautionary examples showing how easy it is to (inadvertently or deliberately) slip off the righteous path of chip/OS/language standards and end up with a non-portable application. In each case, the software had worked for several years on prior releases of Solaris (and/or other platforms), so they superficially looked like problems with the new OS release.