Trond Norbye's Weblog

Creating a relocatable binary

When you create a large project you would most likely want arrange your code into logical parts and create various libraries containing logical units of code. In the "good old days" this would normally be an archive of object files created by using ar. ex:

trond@opensolaris> ar -r myarchive.a foo1.o foo2.o foo3.o
ar: creating myarchive.a

You would pass the archive to the archive to the linker when you linked your program like:

trond@opensolaris> cc -o myprog main.o myarchive.a

The linker would now search for a definition for all of the undefined symbols in main.o in myarchive.a. (please note that if you named your archive libmyarchive you could also pass it to the linker as -lmyarchive). This is what we call static linking, but static linking have some obvious drawbacks:

• If you find a bug in the code in a library, you need to relink all applications that link with the library to fix them.
• At runtime all programs that include functions from the library will have their own copy of the function loaded into memory.

A better solution is to create relocatable objects (aka shared libraries, dll etc), and link the application with those instead (and if you look at a default installation of Solaris you will not find any of the system libraries as static archives). Unfortunately relocatable objects have it's own problems, and that is what I'll address in the rest of this blog post.

So let's go ahead and create a small example to look at the problems and how to solve them. I have created two small source files: lib.c contains the function I want in my library, and main.c contains my application:

trond@opensolaris> cat lib.c
#include <stdio.h>

void my_print(int val) {
   fprintf(stdout, "The value is %d\n", val);
}

We compile this into a relocatable object by using the -Kpic option to cc, and create a shared object by using the -G.

trond@opensolaris> cc -c -Kpic lib.c
trond@opensolaris> cc -o libfoo.so -G lib.o

The next thing we need to do is to compile our main program and link it with the library.

trond@opensolaris> cat main.c
extern void my_print(int);

int main(int argc, char **argv) {
   my_print(argc);

   return 0;
}

trond@opensolaris> cc -c main.c
trond@opensolaris> cc -o myprog main.o -lfoo
ld: fatal: library -lfoo: not found
ld: fatal: file processing errors. No output written to myprog

This doesn't work! Why? By default the linker will only look for shared libraries in /lib[/64] and /usr/lib[/64] (see Directories Searched by the Runtime Linker for more information. To instruct the linker to search the current directory we need to pass -Lpath to the link step (or we could use LD_LIBRARY_PATH, but you don't want to use LD_LIBRARY_PATH... Why? ask Google: LD_LIBRARY_PATH evil)

trond@opensolaris> cc -o myprog main.o -L. -lfoo

It compiled just fine, so let's try start it:

trond@opensolaris> ./myprog
ld.so.1: myprog: fatal: libfoo.so: open failed: No such file or directory

I see multiple solutions to this problem:

• Copy libfoo.so to /usr/lib, but it really doesn't belong in there...
• Create a wrapper script that exports LD_LIBRARY_PATH and invokes my binary (but I would prefer to avoid using LD_LIBRARY_PATH if it is possible
• Use crle (this is the Solaris version of ldconf) to include the directory containing my library into the list the runtime linker will search for libraries in. I don't think this is a good idea as well, because I don't think all of the other programs want my library (and this feels a bit like a really global LD_LIBRARY_PATH hack...)
• Use -R to insert a runtime search path into my binary. Now this looks like a really good idea! let's try it!

trond@opensolaris> cc -o myprog main.o -L. -R. -lfoo
trond@opensolaris> ./myprog
The value is 1

So I guess I'm done now.. Let's install my binary in my $HOME/bin directory and try it :-)

trond@opensolaris> cp myprog libfoo.so ~/bin
trond@opensolaris> myprog
ld.so.1: myprog: fatal: libfoo.so: open failed: No such file or directory

WHAT! It doesn't work!! So what happens when we try to run the binary? let's look:

trond@opensolaris> truss /home/trond/bin/myprog
execve("/home/trond/bin/myprog", 0x08047704, 0x0804770C)  argc = 1
mmap(0x00000000, 4096, PROT_READ|PROT_WRITE|PROT_EXEC, MAP_PRIVATE|MAP_ANON, -1, 0) = 0xFEFB0000
resolvepath("/usr/lib/ld.so.1", "/lib/ld.so.1", 1023) = 12
resolvepath("/home/trond/bin/myprog", "/home/trond/bin/myprog", 1023) = 30
stat64("/home/trond/bin/myprog", 0x08047328) = 0
open("/var/ld/ld.config", O_RDONLY)		Err#2 ENOENT
sysconfig(_CONFIG_PAGESIZE)			= 4096
stat64("./libfoo.so", 0x08046B08)		Err#2 ENOENT
stat64("/lib/libfoo.so", 0x08046B08)		Err#2 ENOENT
stat64("/usr/lib/libfoo.so", 0x08046B08)	Err#2 ENOENT
ld.so.1: myprog: fatal: libfoo.so: open failed: No such file or directory
write(2, " l d . s o . 1 :   m y p".., 74)	= 74
lwp_self()					= 1

It tries to look for libfoo.so in the current directory!!! let's take a peak in the binary (I have bolded out the interesting pieces):

trond@opensolaris> dump -Lv myprog 

myprog:

  **** DYNAMIC SECTION INFORMATION ****
.dynamic:
[INDEX]	Tag         Value
[1]	NEEDED          libfoo.so
[2]	NEEDED          libc.so.1
[3]	INIT            0x8050a20
[4]	FINI            0x8050a3c
[5]	RUNPATH         .
[6]	RPATH           .
[7]	HASH            0x8050118
[8]	STRTAB          0x805042c
[9]	STRSZ           0x37f
[10]	SYMTAB          0x805028c
[11]	SYMENT          0x10
[12]	SUNW_SYMTAB     0x80501fc
[13]	SUNW_SYMSZ      0x230
[14]	SUNW_SORTENT    0x4
[15]	SUNW_SYMSORT    0x8050810
[16]	SUNW_SYMSORTSZ  0x38
[17]	CHECKSUM        0x83d2
[18]	VERNEED         0x80507ac
[19]	VERNEEDNUM      0x1
[20]	PLTSZ           0x30
[21]	PLTREL          0x11
[22]	JMPREL          0x8050850
[23]	REL             0x8050848
[24]	RELSZ           0x38
[25]	RELENT          0x8
[26]	DEBUG           0
[27]	FEATURE_1       PARINIT
[28]	SUNW_CAP        0x8050108
[29]	FLAGS           0
[30]	FLAGS_1         0
[31]	SUNW_STRPAD     0x200
[32]	SUNW_LDMACH     EM_386
[33]	PLTGOT          0x8060a5c

Well, let's re-link our application with the correct directory specified to -R

trond@opensolaris> cc -o myprog main.o -L. -R/home/trond/bin -lfoo
trond@opensolaris> cp myprog /home/trond/bin
trond@opensolaris> myprog
The value is 1

This doesn't feel very user-friendly.. I have hard-coded in my binary where it should search for it's shared libraries. If all users install the binaries in the standard directories such as /opt/foo/bin/myfoo it would work, but it doesn't seem flexible. Luckily for us the run-time linker in Solaris may help us out here, so that we can instruct the runtime linker to search for shared libraries relative to where the binary is installed. If we use the special token $ORIGIN the runtime linker will replace that with the location of the binary. So let's relink our application once more, but this time we want to move the library to $HOME/lib, and keep the binary in $HOME/bin.

trond@opensolaris> mv /home/trond/bin/libfoo.so /home/trond/lib
trond@opensolaris> cc -o myprog main.o -L. -R\$ORIGIN/../lib -lfoo
trond@opensolaris> cp myprog /home/trond/bin
trond@opensolaris> myprog
The value is 1

So let's use truss to see what's happening :-)

trond@opensolaris> truss /home/trond/bin/myprog
execve("/home/trond/bin/myprog", 0x08047708, 0x08047710)  argc = 1
mmap(0x00000000, 4096, PROT_READ|PROT_WRITE|PROT_EXEC, MAP_PRIVATE|MAP_ANON, -1, 0) = 0xFEFB0000
resolvepath("/usr/lib/ld.so.1", "/lib/ld.so.1", 1023) = 12
resolvepath("/home/trond/bin/myprog", "/home/trond/bin/myprog", 1023) = 25
stat64("/home/trond/bin/myprog", 0x0804732C)	= 0
open("/var/ld/ld.config", O_RDONLY)		Err#2 ENOENT
sysconfig(_CONFIG_PAGESIZE)			= 4096
stat64("/home/trond/bin/../lib/libfoo.so", 0x08046B0C) = 0
resolvepath("/home/trond/bin/../lib/libfoo.so", "/home/trond/lib/libfoo.so", 1023) = 28
open("/home/trond/bin/../lib/libfoo.so", O_RDONLY) = 3
mmap(0x00010000, 32768, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_ALIGN, 3, 0) = 0xFEFA0000
mmap(0x00010000, 69632, PROT_NONE, MAP_PRIVATE|MAP_NORESERVE|MAP_ANON|MAP_ALIGN, -1, 0) = 0xFEF80000
mmap(0xFEF80000, 1489, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_FIXED|MAP_TEXT, 3, 0) = 0xFEF80000
mmap(0xFEF90000, 1812, PROT_READ|PROT_WRITE|PROT_EXEC, MAP_PRIVATE|MAP_FIXED|MAP_INITDATA, 3, 0) = 0xFEF90000
munmap(0xFEF81000, 61440)			= 0
memcntl(0xFEF80000, 1300, MC_ADVISE, MADV_WILLNEED, 0, 0) = 0
close(3)					= 0
stat64("/home/trond/bin/../lib/libc.so.1", 0x08046B0C) Err#2 ENOENT
stat64("/lib/libc.so.1", 0x08046B0C)		= 0
resolvepath("/lib/libc.so.1", "/lib/libc.so.1", 1023) = 14
open("/lib/libc.so.1", O_RDONLY)		= 3
mmap(0xFEFA0000, 32768, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_FIXED, 3, 0) = 0xFEFA0000
mmap(0x00010000, 1409024, PROT_NONE, MAP_PRIVATE|MAP_NORESERVE|MAP_ANON|MAP_ALIGN, -1, 0) = 0xFEE20000
mmap(0xFEE20000, 1305977, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_FIXED|MAP_TEXT, 3, 0) = 0xFEE20000
mmap(0xFEF6F000, 28320, PROT_READ|PROT_WRITE|PROT_EXEC, MAP_PRIVATE|MAP_FIXED|MAP_INITDATA, 3, 1306624) = 0xFEF6F000
mmap(0xFEF76000, 6328, PROT_READ|PROT_WRITE|PROT_EXEC, MAP_PRIVATE|MAP_FIXED|MAP_ANON, -1, 0) = 0xFEF76000
munmap(0xFEF5F000, 65536)			= 0
memcntl(0xFEE20000, 188300, MC_ADVISE, MADV_WILLNEED, 0, 0) = 0
close(3)					= 0
mmap(0x00010000, 24576, PROT_READ|PROT_WRITE|PROT_EXEC, MAP_PRIVATE|MAP_ANON|MAP_ALIGN, -1, 0) = 0xFEE10000
munmap(0xFEFA0000, 32768)			= 0
getcontext(0x0804718C)
getrlimit(RLIMIT_STACK, 0x08047184)		= 0
getpid()					= 10032 [10031]
lwp_private(0, 1, 0xFEE12A00)			= 0x000001C3
setustack(0xFEE12A60)
sysi86(SI86FPSTART, 0xFEF76FCC, 0x0000133F, 0x00001F80) = 0x00000001
ioctl(1, TCGETA, 0x080467A0)			= 0
fstat64(1, 0x08046700)				= 0
The value is 1
write(1, " T h e   v a l u e   i s".., 15)	= 15
_exit(0)

In the beginning of the blog I said that one of the benefits of using shared objects was that we didn't have to relink our application if we found a bug in the library we needed to fix, so I thought we should show that as well. So let's modify the library:

trond@opensolaris> cat lib.c
#include <stdio.h>

void my_print(int val) {
   fprintf(stdout, "The end...\n");
}
trond@opensolaris> cc -c -Kpic lib.c
trond@opensolaris> cc -o libfoo.so -G lib.o
trond@opensolaris> cp libfoo.so /home/trond/lib
trond@opensolaris> myprog
The end!!!

So that's it for this time :-)

Posted at 03:23PM Feb 25, 2009 by trond in OpenSolaris  |  Comments[0]