.gnu_debuglink or Debugging system libraries with source code
What is .gnu_debuglink?
It is a section in an ELF file containing
- name of another ELF file and
- crc32 checksum of that file.
It is used as a pointer to a file with separate debug info and can be created with
objcopy --add-gnu-debuglink=<debug info file>
command (more on creation subject in the next post).
For example, let's find out where libc debug info resides on x86 Ubuntu
9.04:
1. Locate libc itself by using ldd on a random binary:
$ ldd /bin/ls | grep libc libc.so.6 => /lib/tls/i686/cmov/libc.so.6 (0xb7ee0000)
2. Dump contents of its .gnu_debuglink section:
$ objdump -s -j .gnu_debuglink /lib/tls/i686/cmov/libc.so.6 /lib/tls/i686/cmov/libc.so.6: file format elf32-i386 Contents of section .gnu_debuglink: 0000 6c696263 2d322e39 2e736f00 0d8f898d libc-2.9.so.....
So name of the file with debug info is libc-2.9.so; the rest must be
checksum. We'll leave that for debugger to verify.
Debug info files are usually stored under /usr/lib/debug plus dirname
of their "parent" library; in our case it is
/usr/lib/debug/lib/tls/i686/cmov/libc-2.9.so
How to obtain debug info for a library?
Debug packages do not get installed
automatically. Let's install one for libc:
$ apt-get install libc6-debug
Package with debug info is usually (always?) named like
<pkgname>-debug, so it is easy to find.
After installation, libc-2.9.so appears in the location we predicted:
$ file /usr/lib/debug/lib/tls/i686/cmov/libc-2.9.so
/usr/lib/debug/lib/tls/i686/cmov/libc-2.9.so: ELF 32-bit LSB shared object, Intel 80386, version 1 (SYSV), dynamically linked (uses shared libs), for GNU/Linux 2.6.15, not stripped
It has DWARF debug info as can be seen by doing
$ readelf --debug-dump=info /usr/lib/debug/lib/tls/i686/cmov/libc-2.9.so | less
Where is the source code?
Debug info is not enough - one needs source code to debug. The
source code is again distributed separately and can be downloaded with
apt-get source command:
$ cd /export/home/maxim/work $ apt-get source libc6-dev
This will create /export/home/maxim/work/glibc-2.9/ directory and
populate it with glibc source code.
How all this works together or How to step into libc function code with dbx?
Brute-force attempt gives us nothig:
(dbx) step dbx: warning: can't find file "/build/buildd/glibc-2.9/misc/../sysdeps/unix/sysv/gethostname.c"
stopped in __gethostname at line 35 in file "gethostname.c"
Obviously, dbx was able to read .gnu_debuglink as it knows where the source
code for __gethostname() is. It may even be enough if all you need is
correct parameter name/values in stack trace:
=>[1] libc.so.6:__gethostname(name = 0xbfd08780 "...", len = 1024U), line 35 in "gethostname.c"
But since we already downloaded the source code, let's point dbx to it:
(dbx) pathmap /build/buildd/glibc-2.9/ /export/home/maxim/work/glibc-2.9/
pathmap command is described in `help pathmap' topic; here's an excerpt:
Establish a new mapping from <from> to <to> where <from> and <to> are filepath prefixes. <from> refers to the filepath compiled into the executable or object file and <to> refers to the file path at debug time.
The result is immediately visible:
(dbx) list
35 if (uname (&buf))
36 return -1;
37
38 node_len = strlen (buf.nodename) + 1;
39 memcpy (name, buf.nodename, len < node_len ? len : node_len);
40
41 if (node_len > len)
42 {
43 __set_errno (ENAMETOOLONG);
44 return -1;
We have source code of a libc function - gethostname().
What about gdb?
A tale about Linux system wouldn't be complete without a story about gdb.
Gdb also supports .gnu_debuglink (obviously), but has a different
command for source code path mapping.
Here's how to step into gethostname() using gdb (in the same environment, e.g. debug info and source code are present):
$ gdb a.out
...
(gdb) set substitute-path /build/buildd/glibc-2.9/ /export/home/maxim/work/glibc-2.9/
(gdb) start
...
(gdb) s
__gethostname (name=0xbfb70910 "...", len=1024)
at ../sysdeps/unix/sysv/gethostname.c:35
35 if (uname (&buf))
(gdb) list
30 size_t len;
31 {
32 struct utsname buf;
33 size_t node_len;
34
35 if (uname (&buf))
36 return -1;
37
38 node_len = strlen (buf.nodename) + 1;
39 memcpy (name, buf.nodename, len < node_len ? len : node_len);
NB: you have to set substitute-path before starting debug session;
when I tried to set it at the same time as in dbx - after stepping
into gethostname() - I got nothing. Besides, with gdb you have to figure
out "from" part of substitute-path by yourself somehow as it doesn't
give you a hint like dbx; here's what gdb prints without proper
substitute-path set:
__gethostname (name=0xbfaf8090 "...", len=1024) at ../sysdeps/unix/sysv/gethostname.c:35 35 ../sysdeps/unix/sysv/gethostname.c: No such file or directory. in ../sysdeps/unix/sysv/gethostname.c
References:
- In dbx, type help separate debug info
- Debugging a Program With dbx, Using a Separate Debug File section
- Understanding ELF using readelf and objdump
