Ali Bahrami

Changing ELF Runpaths (Code Included)

A recent change to Solaris ELF files makes it possible to change the runpath of a dynamic executable or sharable object, something that has not been safely possible up until now. This change 80is currently found in Solaris Nevada (the current development version of Solaris) and in OpenSolaris. It is not yet available in Solaris 10, but in time will appear in the standard shipping Solaris as well.

This seems like a good time to talk about runpaths and the business of how the runtime linker finds dependencies. I also provide a small program named rpath that you can use to modify the runpaths in your file (assuming they were linked under Nevada or OpenSolaris).

The Runpath Problem

The runtime linker looks in the following places, in the order listed, to find the sharable objects it loads into a process at startup time:

• If LD_LIBRARY_PATH (or the related LD_LIBRARY_PATH_32 and LD_LIBRARY_PATH_64) environment variables are defined, the directories they specify are searched to resolve any remaining dependencies.

• If the executable, or any sharable objects that are loaded, contain a runpath, the directories it specifies are searched to resolve dependencies for those objects.

• Finally, it searches two default locations for any remaining dependencies: /lib and /usr/lib (or /lib/64 and /usr/lib/64 for 64-bit code).

As if this isn't complicated enough, it should be noted that the crle command can be used to set values for LD_LIBRARY_PATH, and the default directories.

The above scheme offers a great deal of flexibility, and it usually works well. There is however one notable exception — the "Runpath Problem". The problem is that many objects are not built with a correct runpath, and once an object has been built, it has not been possible to change it. It is common to find objects where the runpath is correct on the system the object was built on, but not on the system where it is installed. Usually, we deal with this all too common situation by setting LD_LIBRARY_PATH, or by creating a linker configuration file with crle. Such solutions have serious downsides, as detailed in an earlier blog entry by Rod Evans entitled "LD_LIBRARY_PATH - just say no".

Both approaches will cause unrelated programs to look in unnecessary additional directories for their dependencies. At best, this imposes unnecessary overhead on their operation. At worst, they may end up binding to the wrong version of a given library, leading to mysterious and hard to debug failures. The environment variable approach is simply too broad.

One important technique that people sometimes use, is to set the environment variables in a wrapper shell script, that may look something like:

#!/bin/sh
#
# Run myapp, setting LD_LIBRARY_PATH so it will run

LD_LIBRARY_PATH="/this/that/theother:/someplace/else"
export LD_LIBRARY_PATH

exec /usr/local/myapp

This is a huge improvement over simply setting LD_CONFIG or LD_LIBRARY_PATH in your shell login config script (.profile, .cshrc, .bashrc, etc), for many reasons:

• Reduces the scope of influence to only cover the application (and its children — see below)
• Doesn't require each user to modify their login script(s)
• Can be managed in a central location

It isn't perfect though. If the program in question should happen to run any child processes (and this is more common than many realize), those child processes will inherit the LD_CONFIG and LD_LIBRARY_PATH settings you've established for this one program. This leak may, or may not, cause problems depending on what programs are run.

It would be far better to modify the object in question and set a runpath that accurately reflects the actual location of its dependencies. The effect of a runpath is limited to the file that contains it, so this solution does not "bleed through" to unrelated files, and it imposes no unnecessary overhead on the general operation of the system. This would be a superior solution if it were possible. However it hasn't been an option until recently.

How Runpaths Are Implemented

Every dynamic executable contains a dynamic section. This is an array of items which convey the information required by the dynamic linker (ld.so.1) to do its work. If an object has a runpath, there will be a DT_RUNPATH and/or DT_RPATH item in the dynamic section (there is more than one of these for historical reasons). As an example, lets examine crle:

% elfdump -d /usr/bin/crle | grep 'R*PATH'
       [4]  RUNPATH           0x612               $ORIGIN/../lib
       [5]  RPATH             0x612               $ORIGIN/../lib

The string (in this case, "$ORIGIN/../lib") is not actually stored in the dynamic section. Rather, it is contained in the dynamic string table (.dynstr). The value 0x612 is the offset within string table at which the desired string starts.

A string table is a section that contains NULL terminated strings, one immediately following the other. To access a given string, you add the offset of the string within the section to the base of the section data area. Consider a string table that contains the names of two variables "var1", and "var2" and a runpath "$ORIGIN/../lib". By ELF convention, string tables always have a 0-length NULL terminated string in the first position. In C language notation, we might declare the contents of the resulting string table section containing these 4 strings as

"\0var1\0var2\0$ORIGIN/../lib"

The indexes of the 4 strings in our table are [0], [1], [6], and [11], and any item in the dynamic section or the dynamic symbol table that needs one of these strings will specify it using the appropriate index. An interesting result of the way that string tables are designed is that that every single offset into a string table represents a usable string. Although our intent with the C string above was to represent 4 strings, it actually contains 23 potential strings (26 if you count the duplicate NULL strings), and not just the 4 we intentionally inserted. Listing them by offset, they are:

[0]  ""

[1]  "var1"
[2]  "ar1"
[3]  "r1"
[4]  "1"
[5]  ""

[6]  "var2"
[7]  "ar2"
[8]  "r2"
[9]  "2"
[10] ""

[11] "$ORIGIN/../lib"
[12] "ORIGIN/../lib"
[13] "RIGIN/../lib"
[14] "IGIN/../lib"
[15] "GIN/../lib"
[16] "IN/../lib"
[17] "N/../lib"
[18] "/../lib"
[19] "../lib"
[20] "./lib"
[21] "/lib"
[22] "lib"
[23] "ib"
[24] "b"
[25] ""

This is a very efficient scheme, since each string can appear once in the string table, and multiple ELF items can refer to it. Also, it allows fixed size things, like ELF symbols or dynamic section entries, to efficiently reference variable length strings. There are two things to note, however:

1. For a given string, there is no way to tell if it is referenced, where it is referenced from, or how many references there are.
2. There is no room to add new strings to a string table.

The options for modifying a runpath in this situation are limited:

• Any string already in the string table, as with the 23 options listed in our example above, can be safely set as a runpath, by simply changing the offset in the runpath DT entries. Note that most string table strings are variable and file (not directory) names that are not likely to make useful path strings. This option is unlikely to help.

• You might overwrite the existing path string with a new string of equal or shorter length in the (usually true) belief that nothing else is accessing that particular string. It is a simple matter using a binary aware editor to locate and overwrite the existing string. This usually works, but if there is another part of the file accessing that string, this change will break it. We cannot recommend or stand behind this, even though we have done it ourselves for one-off experiments (never in a shipping product).

• A better approach would be to add a new string to the end of the section, and then change the offset in the dynamic section to use it. Traditionally, ELF files have not had any extra room in the string table section to allow this.

As a result, it has not been possible to support the modification of the runpath in an existing object up until recently.

Making Room

I recently integrated a change to Solaris Nevada (and OpenSolaris) to add a little unused space to our ELF files, in order to facilitate a limited amount of post-link modification:

PSARC 2007/127 Reserved space for editing ELF dynamic sections
6516118 Reserved space needed in ELF dynamic section and
        string table

This change does two things:

1. Adds some extra NULL bytes to the end of every dynamic string table. (The current value is 512 bytes, but this can change in the future).

2. Adds a new dynamic section entry named DT_SUNW_STRPAD to keep track of the size of the unused space at the end of the dynamic string table.

3. Adds some extra (currently 10) unused DT_NULL entries at the end of the dynamic section.

This additional space is small enough that it doesn't increase the size of real world objects by a significant amount. Though small, it gives us a lot of new flexibility. The room in the string table allows for the safe addition of a moderate number of new strings. The additional null DT entries allow us to add a DT_RUNPATH item if the file doesn't already have one to modify. Looking at crle again:

% elfdump -d /usr/bin/crle | egrep 'R*PATH|STRPAD'
       [4]  RUNPATH           0x612               $ORIGIN/../lib
       [5]  RPATH             0x612               $ORIGIN/../lib
      [32]  SUNW_STRPAD       0x200               

The SUNW_STRPAD entry tells us that the dynamic string table has 512 (0x200) bytes of unused space available at the end of its data area.

The way this works is very simple: If a file lacks a DT_SUNW_STRPAD dynamic entry, then we know that it is an older file, and that the dynamic string table does not have any extra space. If it does have a DT_SUNW_STRPAD, then its value tells us how much room is available. In this case, we can add the string, modify the DT_RUNPATH items, and reduce the DT_SUNW_STRPAD value by the number of bytes we used.

If the value in DT_SUNW_STRPAD is too small for our new string, then we are out of luck and cannot add it. This extra room should help in the vast majority of cases, but as with any such approach, there are limits. We recommend the use of the special $ORIGIN token, both because it is a great way to organize objects, and because it is short.

The rpath Utility

Eventually, Solaris will ship with a standard utility for modifying runpaths. However, there is no need to wait. I have written an unofficial test program I call 'rpath' that you can download and build. To build rpath, you will need a version of Solaris Nevada newer than build 61, or a recent version of OpenSolaris. To check your system, try:

% grep DT_SUNW_STRPAD /usr/include/sys/link.h
#define DT_SUNW_STRPAD  0x60000019 /* # of unused bytes at the */

If your grep doesn't find DT_SUNW_STRPAD, your system lacks the necessary support.

To build rpath, unpack the compressed tar file and type 'make'. If you are using gcc, first edit the Makefile and uncomment the CC line:

% gunzip < rpath.tgz | tar xvpf -
% cd rpath
% make

rpath is used as follows:

NAME
     rpath - set/get runpath of ELF dynamic objects

SYNOPSIS
     rpath [-dr] file [runpath]

DESCRIPTION
     rpath can display,  modify,  or  delete  the  runpath  of  a
     dynamic ELF object.

     If called without a runpath  argument  and  without  the  -r
     option,  the  current runpath, if any, is written to stdout.
     If -r is specified, the existing runpath is removed. If run-
     path  is  supplied,  the runpath of the object is set to the
     new value.


OPTIONS
     The following options are supported:

     -d  Cause detailed ELF information about the  ELF  file  and
         the changes being made to it to be written to stderr.

     -r  Instead of adding or modifying the file  runpath,  rpath
         removes  any  DT_RPATH  or  DT_RUNPATH  entries from the
         dynamic section of  the  file.  This  action  completely
         removes  any existing from the file. When this option is
         used, rpath does not allow the runpath argument.

Using rpath

Let's use rpath to look at its own runpath. We will see that it doesn't have one, something that can be verified using elfdump:

% rpath rpath
% elfdump -d rpath | egrep 'R*PATH|STRPAD'
      [28]  SUNW_STRPAD       0x200               

Now, let's add a runpath to it:

% rpath rpath pointless:runpath
% rpath rpath
pointless:runpath
% elfdump -d rpath | egrep 'R*PATH|STRPAD'
      [28]  SUNW_STRPAD       0x1ee               
      [30]  RUNPATH           0x33f      pointless:runpath

Notice that the amount of unused space reported by SUNW_STRPAD has gone down from 512 (0x200) to 494 (0x1ee) bytes, a reduction of 18 bytes. This makes sense, since we added a 17 character string, and we must add a NULL termination.

We can observe the runtime linker looking in 'pointless' and 'runpath' as it loads rpath (note: output is edited for width):

% LD_DEBUG=libs ./rpath 
13707: 
13707: hardware capabilities - 0x25ff7  [ AHF SSE3 SSE2 
       SSE FXSR AMD_3DNowx AMD_3DNow AMD_MMX MMX CMOV
       AMD_SYSC CX8 TSC FPU ]
13707: 
13707: 
13707: configuration file=/var/ld/ld.config: unable to
       process file
13707: 
13707: 
13707: find object=libelf.so.1; searching
13707:  search path=pointless:runpath  (RUNPATH/RPATH
                                        from file rpath)
13707:  trying path=pointless/libelf.so.1
13707:  trying path=runpath/libelf.so.1
13707:  search path=/lib  (default)
13707:  search path=/usr/lib  (default)
13707:  trying path=/lib/libelf.so.1
13707: 
13707: find object=libc.so.1; searching
13707:  search path=pointless:runpath  (RUNPATH/RPATH from
                                        file rpath)
13707:  trying path=pointless/libc.so.1
13707:  trying path=runpath/libc.so.1
13707:  search path=/lib  (default)
13707:  search path=/usr/lib  (default)
13707:  trying path=/lib/libc.so.1
13707: 
13707: find object=libc.so.1; searching
13707:  search path=/lib  (default)
13707:  search path=/usr/lib  (default)
13707:  trying path=/lib/libc.so.1
13707: 
13707: 1: 
13707: 1: transferring control: rpath
13707: 1: 
usage: rpath [-dr] file [runpath]
13707: 1: 

Finally, we'll remove the runpath we just added:

% rpath -r rpath
% rpath rpath
% elfdump -d rpath | egrep 'R*PATH|STRPAD'
      [28]  SUNW_STRPAD       0x1ee               

Note that even though the runpath is gone, the amount of available extra space in the dynamic string section did not go back up from 494 (0x1ee) to 512 (0x200). Adding strings is a one way operation. Once they are added, they are permanent. So even though you now have the ability to add strings of moderate length, you won't want to do it indiscriminately.

On the plus side, you can always re-add the same runpath back without using any more space:

% rpath rpath pointless:runpath
% rpath rpath
pointless:runpath
% elfdump -d rpath | egrep 'R*PATH|STRPAD'
      [28]  SUNW_STRPAD       0x1ee               
      [30]  RUNPATH           0x33f      pointless:runpath

rpath found that the string 'pointless:runpath' was already in the string table, so it used it without inserting another copy.

Conclusions

Our best advice has always been that the LD_LIBRARY_PATH environment variable should not be used to work around objects with bad or missing runpaths. It is best to rebuild such objects and set the runpath correctly. This hasn't changed, and you should always do so if you can.

The problem with that advice is that there are times when all you have is the object, and no option to rebuild. In that case, LD_LIBRARY_PATH has been a necessary evil (and one that we've been glad to have). With the advent of objects that can have their runpaths modified, we now have a better answer, and the use of LD_LIBRARY_PATH for this purpose should be allowed to slowly fade away.

Technorati Tag: OpenSolaris
Technorati Tag: Solaris

Posted at 03:49PM Jun 12, 2007 by ali in Sun  |  Comments[3]