Ali Bahrami
- All
- Sun
ld Is Now A Cross Link-Editor
Until yesterday, ld, the Solaris link-editor, was a native linker. This means that it was only able to link objects for the same machine that the linker was running on. To link sparc objects required the use of of a sparc machine, and x86 objects required an x86 system.
With the integration of
PSARC 2008/179 cross link-editorinto Solaris Nevada build 87, the Solaris ld became a cross link-editor. Now, ld running on any type of hardware can link objects for any of the systems Solaris supports. This is currently sparc and x86, but there are people playing with OpenSolaris on other hardware too, so who knows what we might end up with?
6671255 link-editor should support cross linking
The user interface to this new capability is a simple extension of ld's traditional behavior. Traditionally, ld establishes the class (whether the object is 32 or 64-bit) of the object being generated by examining the first ELF object processed from the command line. In the case where an object is being built solely from an archive library or a mapfile, the -64 command line option is available to explicitly override this default. We have extended this to also determine the machine type of the object to produce:
- The class and machine type of the resulting object default to those of the first ELF object processed from the command line.
- The new '-z target=' command line option can be used to explicitly specify the machine type.
Of course, it's a rare program that doesn't link against at least one system library. You're going to need libc, if nothing else. To do a successful cross-link, you'll need to have an image of the root filesystem for a system of the target type. There are many ways to do this. For testing purposes, I used a sparc and x86 system, using NFS to allow each system to see the root filesystem of the other.
Even though we now have a cross link-editor, we expect that the vast majority of links will be native, for the machine running the linker. We decided to pursue cross linking anyway, for two reasons:
- To lower the bar for OpenSolaris ports: People have had some success using the GNU ld to port OpenSolaris, but it is difficult to get very far that way. The code in Solaris depends on link-editor features that are specific to the Solaris ld, so using GNU ld involves a fair amount of hacking around these things to make progress, and the farther up the stack you go from the kernel to userland, the harder it gets. We hope that providing a better framework for adding targets to the Solaris ld will help such efforts. It should now be possible (though still not trivial!) to add support for a new target to the Solaris linker running on sparc or x86, and then use the resulting system to cross-build for the new platform.
- To allow the use of fast/cheap commodity desktop systems to build objects for other systems, be they large expensive systems, or small embedded devices.
A cross link-editor is a significant step, but is of little use unless you also have a cross-compiler and assembler. The GNU gcc compiler can be built as a cross compiler, and that should be very helpful for OpenSolaris ports. However, the Sun Studio compilers are native compilers, so it will still be awhile before I can use my amd64 desktop to build a sparc version of Solaris as part of work at Sun. We've taken a first step. It will be interesting to see what follows.
Technorati Tag:
OpenSolaris
Technorati Tag:
Solaris
Posted at 05:56PM Mar 19, 2008 by ali in Sun | Comments[2]
Avoiding LD_LIBRARY_PATH: The Options
With the introduction of the elfedit utility into Solaris, we have a new answer to the age old question of how to avoid everyones favorite way to get into trouble, the LD_LIBRARY_PATH environment variable. This seems like an appropriate time to revisit this topic.
LD_LIBRARY_PATH Seems Useful. What's the Problem?
The problem is that LD_LIBRARY_PATH is a crude tool, and cannot be easily targeted at a problem program without also hitting other innocent programs. Sometimes this overspray is harmless (it costs some time, but doesn't break anything). Other times, it causes a program to link to the wrong version of something, and that program dies in mysterious ways.Historically, inappropriate use of LD_LIBRARY_PATH might be the #1 one way to get yourself into trouble in an ELF environment. In particular, people who redistribute binaries with instructions for their users to set LD_LIBRARY_PATH in their shell startup scripts are unleashing forces beyond their control. Experience tells us that such use is destined to end badly.
This subject has been written about many times by many people. My colleague Rod Evans wrote about this ( LD_LIBRARY_PATH - just say no) for one of his first blog entries.
If you need additional convincing on this point, here are some suggested Google searches you might want to try:
LD_LIBRARY_PATH problem LD_LIBRARY_PATH bad LD_LIBRARY_PATH evil LD_LIBRARY_PATH darkest hell
If LD_LIBRARY_PATH is so bad, why does its use persist? Simply because it is the option of last resort, used when everything else has failed. We probably can't eliminate it, but we should strive to reduce its use to the bare minimum.
How to Use, and How To Avoid Using LD_LIBRARY_PATH
The best way to use LD_LIBRARY_PATH is interactively, as a short term aid for testing or development. A developer might use it to point his test program at an alternative version of a library. Beyond that, the less you use it, the better off you'll be. With that in mind, here is a list of ways to avoid LD_LIBRARY_PATH. The items are ordered from best to worst, with the best option right at the top:- Explicitly set the correct runpath for the objects you
build. If you have the ability to relink the object, you
can always do this, and no other workaround is needed.
To set a runpath in an object, use the -R compiler/linker option.
One common problem that people run into with a built in runpath is the use of an absolute path (e.g. /usr/local/lib). Absolute paths are no problem for the well known system libraries, because their location is fixed by convention as well as by standards. However, they can be trouble for libraries supplied by third parties and installed onto the system. Usually the user has a choice of where such applications are installed, their home directory, or /usr/local being two of the more popular places. An application that hard wires the location of user installed libraries cannot handle this. The solution in this case is to use the $ORIGIN token in those runpaths. The $ORIGIN token, which refers to the directory in which the using object resides, can be used to set a non-absolute runpath that will work in any location, as long as the desired libraries reside at a known location relative to the using program. Fortunately, this is often the case.
For example, consider the case of a 32-bit application named myapp, which relies on a sharable library named mylib.so, as well as on the standard system libraries found in /lib and /usr/lib. The -R option to put the runpath into myapp that will look in these places would be:
This allows myapp and mylib.so to be installed anywhere, as long as they are kept in the same positions relative to each other.-R '$ORIGIN/../lib:/lib:/usr/lib'
Even for system libraries, the use of $ORIGIN can be useful. We use it for all of the linker components in the system. For instance:
By setting the runpath using $ORIGIN instead of simply hardwiring the well known location /lib, we make it easier to test a tree of alternative linker components, such as results when we do a full build of the Solaris ON consolidation. We know that when we run a test copy of ld, that it will use the related libraries that were built with it, instead of binding to the installed system libraries.% elfdump -d /usr/bin/ld | grep RUNPATH [7] RUNPATH 0x2e6 $ORIGIN/../../libThere is one exception to the advice to make heavy use of $ORIGIN. The runtime linker will not expand tokens like $ORIGIN for secure (setuid) applications. This should not be a problem in the vast majority of cases.
- Many times, the problem comes in the form of open source software that explicitly sets the runpath to an incorrect value for Solaris. Can you fix the configuration script and contribute the change back to the package maintainer? You'll be doing lots of people a favor if you do.
- If you have an object with a bad runpath (or no runpath) and the
object cannot be rebuilt, it may be possible to alter its runpath
using the elfedit command. Using the myapp example from the
previous item:
For this option to be possible, you need to be running a recent version of Solaris that has elfedit, and your object has to have been linked by a version of Solaris that has the necessary extra room. Quoting from the elfedit manpage:elfedit -e 'dyn:runpath $ORIGIN/../lib:/lib:/usr/lib' myapp
The desired string must already exist in the dynamic string table, or there must be enough reserved space within this section for the new string to be added. If your object has a string table reservation area, the value of the .dynamic DT_SUNW_STRPAD element indicates the size of the area. The following elfedit command can be used to check this:
% elfedit -r -e 'dyn:tag DT_SUNW_STRPAD' file
The dynamic section must already have a runpath element, or there must be an unused dynamic slot available where one can be inserted. To test for the presence of an existing runpath:
% elfedit -r -e 'dyn:runpath' file
A dynamic section uses an element of type DT_NULL to terminate the array found in that section. The final DT_NULL cannot be changed, but if there are more than one of these, elfedit can convert one of them into a runpath element. To test for extra dynamic slots:
% elfedit -r -e 'dyn:tag DT_NULL' file
- If your application was linked with the -c option to the linker,
then you can use the crle command to alter the configuration file
associated with the application and change the settings for
LD_LIBRARY_PATH that are applied for that application. This is a
pretty good solution, but is limited by its complexity, and by the
fact that the person who linked the object needs to have thought
ahead far enough to provide for this option. Odds are that they
didn't. If they had, they might just as well have set the
runpath correctly in the first place, eliminating the need for
anything else.
You can use crle with an application that was not linked with -c, either by setting the LD_CONFIG environment variable, or by modifying the global system configuration file. However, both of these options suffer from the same issues as the LD_LIBRARY_PATH environment variable: They are too coarse grained to be applied to a single application in a targeted way.
- If none of the above are possible, then you are indeed stuck with
LD_LIBRARY_PATH. In this case, the goal should be to minimize the
number of applications that see this environment variable. You should
never set it in your interactive shell environment (via whatever
dot file your shell supports: .profile, .login, .cshrc, .basrc, etc...).
Instead, put it in a wrapper shell script that you use to run the
specific program.
The use of a wrapper script is a pretty safe way to use LD_LIBRARY_PATH, but you should be aware of one limitation of this approach: If the program being wrapped starts any other programs, then those programs will see the LD_LIBRARY_PATH environment variable. Since programs starting other programs is a common Unix technique, this form of leakage can be more common that you might realize.
Technorati Tag:
OpenSolaris
Technorati Tag:
Solaris
Posted at 05:10PM Nov 02, 2007 by ali in Sun | Comments[2]
Introducing elfedit: A Tool For Modifying Existing ELF Objects
Back in June, I wrote about changes we've recently made to Solaris ELF objects that allow their runpaths to be modified without having to rebuild the object. In that posting, I alluded to work that I was then doing when I said "Eventually, Solaris will ship with a standard utility for modifying runpaths". I am happy to say that this has come to pass. I recently integrated /usr/bin/elfedit into build 75 of Solaris Nevada with:
elfedit can indeed modify the runpath in an object, but it is considerably more general than that. elfedit is a tool for examining and modifying the ELF metadata that resides within ELF objects. It can be used as a batch mode tool from shell scripts, makefiles, etc, or as an interactive tool, for examining and exploring objects. elfedit has a modular design, and ships with a set of standard modules for performing common edits. This design makes it easy to add new functionality by adding additional modules.PSARC 2007/509 elfedit 6234471 need a way to edit ELF objects
Prior to elfedit, making these sorts of modifications required the user to write a program, usually in C using libelf. elfedit raises the programming level required to do this significantly. Many operations can be done using existing elfedit commands. For those that cannot, it is far easier to write an elfedit module to add the ability than it is to write a standalone program.
We envision elfedit being used to solve the following sorts of problems:
[Small Fixups]Every elfedit module contains documentation for the commands it provides. This information is displayed using the built in help command, in a format that is based on that of Solaris manpages. The help strings in the standard elfedit modules supplied with Solaris are internationalized using the same i18n mechanisms employed by the rest of the linker software found under usr/src/cmd/sgs. Hence, all elfedit modules supplied by Sun will have complete documentation, and will support the necessary language locales.To correct minor issues in a built file that cannot be easily rebuilt, or for which sources are not available.[Better Way To Support Specialized Rarely Used Features]Probably the most notable such item is the ability to alter the runpath of objects built following the integration of
The ability to do this is a "Frequently Asked Question" for which there has previously been no good answer. This feature is expected to be used nearly as soon as it is available, to fix the runpaths of FOSS (free open source software) built for Solaris, which often has the wrong runpaths set.PSARC 2007/127 Reserved space for editing ELF dynamic sections 6516118 Reserved space needed in ELF dynamic section and string tableAnother common situation is when programmers forget to explicitly add the libraries they depend on to the link line, relying on indirect dependencies to make things work. elfedit can be used to add NEEDED dependencies to an existing object's dynamic section, making the dependencies explicit.
As an avenue for delivering small features to change some object attributes without the need to add additional complex and specialized features to ld and ld.so.1.[Linker Development]For example, we have had requests to allow a mechanism to ld that would allow the user to override the hardware capability bits that are placed in the object by the compiler. Such a feature would be complex to document and burdens already complex commands with features that are rarely used. Such features are a natural fit to elfedit. (See the elfedit(1) manpage for an example of modifying the hardware capabilities).
We sometimes work on linker features that require objects with new values or flag bits that the compilers do not yet generate. elfedit allows us to set arbitrary values for such items quickly, and without having to write a program.[Linker Testing]Many bugs involve an object that is broken in some way. Once the bug is fixed, we need an object broken in that particular way for our test suite. There are several problems that arise:
- Cataloging and archiving broken objects is time consuming and error prone.
- Producing similarly broken objects for different platforms is not always possible.
- As new platforms appear, we end up with coverage gaps where some platforms can do a given test and others cannot.
elfedit gives us the ability to build a simple object, and then break it intentionally in a specific and controlled manner. Tests can then be self contained, requiring no external data, and applicable to all relevant platforms.
elfedit's ability to extract specific bits of data from an object is very useful for object and linker testing.
As with any program that changes the contents of an ELF file, changes to an object by elfedit will invalidate any pre-existing elfsign signature. Assuming the changes are understood and acceptable to the signing authority, such objects will need to be signed after the edits are done.
Modular Design And Extensibility
elfedit has a modular design, reflecting our own experience with dynamic linking, and influenced heavily by the design of mdb, the modular debugger.The elfedit program contains the code that handles the details of reading objects, executing commands to modify them, and saving the results. Very little of the code that performs the actual edits is found in elfedit itself. Rather, the commands exist in modules, which are sharable objects with a well defined elfedit-specific interface. elfedit loads needed modules on demand when a command from the module is executed. These modules are self contained, and include their own documentation in a standard format that elfedit can display using its help command.
The module forms a namespace for the commands that it supplies. Each module delivers a set of commands, focused on related functionality. A command is specified by combining the module and command names with a colon (:) delimiter, with no intervening whitespace. For example, dyn:runpath refers to the runpath command provided by the dyn module.
Module names must be unique. The command names within a given module are unique within that module, but the same command names can be used in more than one module. For example, most modules contain a command named 'dump', which is used to provide an elfedump-style view of the data.
We have adopted the following general rules of thumb for naming modules and commands:
- The module name reflects the part of the ELF format the module addresses (ehdr, phdr, shdr, ...)
- Commands that directly access a field in an ELF structure are given the name of the field (e.g. ehdr:e_flags).
- Commands that are higher level have a simple descriptive name that reflects their purpose (e.g. dyn:runpath).
Give 'Em Enough Rope
elfedit is a tool for linker development and testing. As such, it follows the Unix tradition of doing what it's told, without a lot of noise. This is great if you are doing linker research & development, or testing. We commonly need to intentionally set ELF metadata to undefined or even "wrong" values. However, it follows that elfedit won't prevent you from making nonsensical or otherwise incorrect changes to your ELF objects.For example, X86 objects have little endian byteorder (ELFDATA2LSB):
% file /usr/bin/ls
/usr/bin/ls: ELF 32-bit LSB executable 80386 Version 1 [FPU],
dynamically linked, not stripped, no debugging information available
% elfedit -e 'ehdr:ei_data elfdata2MSB' /usr/bin/ls /tmp/badls
% file /tmp/badls
/tmp/badls: ELF 32-bit MSB executable 80386 Version 1 [FPU],
dynamically linked, not stripped, no debugging information available
% /tmp/badls /tmp/badls: cannot execute
This is really nothing to be worried about. If you are using elfedit's low level operations that allow arbitrary changes to individual ELF fields, then you need to know enough about the ELF format to make these changes properly. Most people will use elfedit for the high level operations such as changing runpaths. The high level operations are safe, and do not require expert knowledge to use.
If you are making those low level changes, the Solaris Linkers and Libraries Guide can be very helpful.
Learning More
elfedit is a standard part of the Solaris development branch, the code that will eventually ship from Sun as the next version of Solaris. It is also available as part of OpenSolaris. It is not part of Solaris 10 or earlier Solaris releases. If you are using a recent Solaris distribution, such as Solaris Express Developer Edition then elfedit should be already present on your system.The elfedit(1) manpage describes the utility in more detail, and gives three examples that should be of general interest:
- Changing runpaths
- Changing hardware/software capability bits
- Reading specific data, without having to grep the output of elfdump.
Technorati Tag:
OpenSolaris
Technorati Tag:
Solaris
Posted at 10:34AM Nov 02, 2007 by ali in Sun |
What Are Fake ELF Section Headers?
I'd like to take a moment to explain an unusual feature we added to elfdump last summer. The -P option tells elfdump to ignore the section headers in the file (if any) and to instead generate a "fake" set from the program headers. So, what are fake section headers, and why would you want them?
Earlier this year, there was an "incident" in which a previously unknown hole in the Solaris telnet daemon was used by a worm. As soon as we got a copy of this worm, we tried to examine it with elfdump to see what we might learn:
% elfdump zoneadmd
ELF Header
ei_magic: { 0x7f, E, L, F }
ei_class: ELFCLASS32 ei_data: ELFDATA2LSB
e_machine: EM_386 e_version: EV_CURRENT
e_type: ET_EXEC
e_flags: 0
e_entry: 0x80512d4 e_ehsize: 52 e_shstrndx: 0
e_shoff: 0 e_shentsize: 0 e_shnum: 0
e_phoff: 0x34 e_phentsize: 32 e_phnum: 5
Program Header[0]:
p_vaddr: 0x8050034 p_flags: [ PF_X PF_R ]
p_paddr: 0 p_type: [ PT_PHDR ]
p_filesz: 0xa0 p_memsz: 0xa0
p_offset: 0x34 p_align: 0
Program Header[1]:
p_vaddr: 0 p_flags: [ PF_R ]
p_paddr: 0 p_type: [ PT_INTERP ]
p_filesz: 0x11 p_memsz: 0
p_offset: 0xd4 p_align: 0
Program Header[2]:
p_vaddr: 0x8050000 p_flags: [ PF_X PF_R ]
p_paddr: 0 p_type: [ PT_LOAD ]
p_filesz: 0x6491 p_memsz: 0x6491
p_offset: 0 p_align: 0x10000
Program Header[3]:
p_vaddr: 0x8066494 p_flags: [ PF_X PF_W PF_R ]
p_paddr: 0 p_type: [ PT_LOAD ]
p_filesz: 0x3e0 p_memsz: 0xc10
p_offset: 0x6494 p_align: 0x10000
Program Header[4]:
p_vaddr: 0x80665c4 p_flags: [ PF_X PF_W PF_R ]
p_paddr: 0 p_type: [ PT_DYNAMIC ]
p_filesz: 0xd8 p_memsz: 0
p_offset: 0x65c4 p_align: 0
If you look at the ELF header, you'll see that our bad guy has set the e_shnum, e_shoff, and e_shentsize fields to zero. These fields are used to locate the section headers for an ELF object. The section headers in turn contain the information needed to look deeper into an object. Section headers are not used to run a program, only to examine it. Zeroing them is a crude, but effective way to obscure what's inside. ELF objects are just files after all, and anyone with write access can modify them. It's not unheard of to modify an ELF object using a binary capable editor like emacs.
Fortunately, the design of ELF makes it difficult to actually hide what an object calls from other sharable objects. And since the system call stubs are all located in libc, you can't hide the system calls your code makes. Here is one way to look inside:
Unlike some other object systems, ELF inter-object references are always looked up by name at runtime. The runtime linker hashes the name and looks it up on the first reference. So if you want to actually call something outside of your own object, you have to call it by its real name. This information is located via the object's program headers, and unlike the section headers, they need to be reasonably accurate for the object to work.% ldd -r -e LD_DEBUG=bindings zoneadmd 2>&1 | fgrep "binding file=zoneadmd" 04992: binding file=zoneadmd to 0x0 (undefined weak): symbol `__deregister_frame_info_bases' 04992: binding file=zoneadmd to 0x0 (undefined weak): symbol `__register_frame_info_bases' 04992: binding file=zoneadmd to 0x0 (undefined weak): symbol `_Jv_RegisterClasses' 04992: binding file=zoneadmd to file=/lib/libc.so.1: symbol `_environ' 04992: binding file=zoneadmd to file=/lib/libc.so.1: symbol `__iob' 04992: binding file=zoneadmd to file=/lib/libc.so.1: symbol `_cleanup' 04992: binding file=zoneadmd to file=/lib/libc.so.1: symbol `atexit' 04992: binding file=zoneadmd to file=/lib/libc.so.1: symbol `__fpstart' 04992: binding file=zoneadmd to file=/lib/libc.so.1: symbol `exit' 04992: binding file=zoneadmd to 0x0 (undefined weak): symbol `__deregister_frame_info_bases' 04992: binding file=zoneadmd to 0x0 (undefined weak): symbol `_Jv_RegisterClasses' 04992: binding file=zoneadmd to 0x0 (undefined weak): symbol `__register_frame_info_bases' 04992: binding file=zoneadmd to file=/lib/libc.so.1: symbol `getenv' 04992: binding file=zoneadmd to file=/lib/libc.so.1: symbol `setsid' 04992: binding file=zoneadmd to file=/lib/libc.so.1: symbol `printf' 04992: binding file=zoneadmd to file=/lib/libc.so.1: symbol `fflush' 04992: binding file=zoneadmd to file=/lib/libc.so.1: symbol `signal' 04992: binding file=zoneadmd to file=/lib/libc.so.1: symbol `pthread_create' 04992: binding file=zoneadmd to file=/lib/libc.so.1: symbol `pthread_join' 04992: binding file=zoneadmd to file=/lib/libc.so.1: symbol `malloc' 04992: binding file=zoneadmd to file=/lib/libc.so.1: symbol `pthread_cancel' 04992: binding file=zoneadmd to file=/lib/libc.so.1: symbol `free' 04992: binding file=zoneadmd to file=/lib/libc.so.1: symbol `close' 04992: binding file=zoneadmd to file=/lib/libc.so.1: symbol `snprintf' 04992: binding file=zoneadmd to file=/lib/libnsl.so.1: symbol `inet_addr' 04992: binding file=zoneadmd to file=/lib/libnsl.so.1: symbol `gethostbyname' 04992: binding file=zoneadmd to file=/lib/libc.so.1: symbol `bcopy' 04992: binding file=zoneadmd to file=/lib/libsocket.so.1: symbol `ntohl' 04992: binding file=zoneadmd to file=/lib/libsocket.so.1: symbol `socket' 04992: binding file=zoneadmd to file=/lib/libsocket.so.1: symbol `htons' 04992: binding file=zoneadmd to file=/lib/libsocket.so.1: symbol `htonl' 04992: binding file=zoneadmd to file=/lib/libsocket.so.1: symbol `connect' 04992: binding file=zoneadmd to file=/lib/libc.so.1: symbol `___errno' 04992: binding file=zoneadmd to file=/lib/libsocket.so.1: symbol `getsockopt' 04992: binding file=zoneadmd to file=/lib/libc.so.1: symbol `fcntl' 04992: binding file=zoneadmd to file=/lib/libc.so.1: symbol `select' 04992: binding file=zoneadmd to file=/lib/libc.so.1: symbol `write' 04992: binding file=zoneadmd to file=/lib/libc.so.1: symbol `read' 04992: binding file=zoneadmd to file=/lib/libc.so.1: symbol `strcpy' 04992: binding file=zoneadmd to file=/lib/libc.so.1: symbol `gettimeofday' 04992: binding file=zoneadmd to file=/lib/libc.so.1: symbol `strstr' 04992: binding file=zoneadmd to file=/lib/libc.so.1: symbol `mkstemp' 04992: binding file=zoneadmd to file=/lib/libc.so.1: symbol `fdopen' 04992: binding file=zoneadmd to file=/lib/libc.so.1: symbol `fopen' 04992: binding file=zoneadmd to file=/lib/libc.so.1: symbol `unlink' 04992: binding file=zoneadmd to file=/lib/libc.so.1: symbol `fputs' 04992: binding file=zoneadmd to file=/lib/libc.so.1: symbol `fputc' 04992: binding file=zoneadmd to file=/lib/libc.so.1: symbol `lseek' 04992: binding file=zoneadmd to file=/lib/libc.so.1: symbol `fclose' 04992: binding file=zoneadmd to file=/lib/libc.so.1: symbol `fprintf' 04992: binding file=zoneadmd to file=/lib/libc.so.1: symbol `fread' 04992: binding file=zoneadmd to file=/lib/libc.so.1: symbol `putc' 04992: binding file=zoneadmd to file=/lib/libc.so.1: symbol `_xstat' 04992: binding file=zoneadmd to file=/lib/libc.so.1: symbol `_lxstat' 04992: binding file=zoneadmd to file=/lib/libc.so.1: symbol `_fxstat' 04992: binding file=zoneadmd to file=/lib/libc.so.1: symbol `_xmknod' 04992: binding file=zoneadmd to file=/lib/libc.so.1: symbol `open' 04992: binding file=zoneadmd to file=/lib/libc.so.1: symbol `mmap' 04992: binding file=zoneadmd to file=/lib/libc.so.1: symbol `strrchr' 04992: binding file=zoneadmd to file=/lib/libc.so.1: symbol `nanosleep' 04992: binding file=zoneadmd to file=/lib/libc.so.1: symbol `fork' 04992: binding file=zoneadmd to file=/lib/libc.so.1: symbol `dup2' 04992: binding file=zoneadmd to file=/lib/libc.so.1: symbol `pipe' 04992: binding file=zoneadmd to file=/lib/libc.so.1: symbol `execve' 04992: binding file=zoneadmd to file=/lib/libc.so.1: symbol `kill' 04992: binding file=zoneadmd to file=/lib/libc.so.1: symbol `waitpid' 04992: binding file=zoneadmd to file=/lib/libc.so.1: symbol `localtime_r' 04992: binding file=zoneadmd to file=/lib/libc.so.1: symbol `utimes' 04992: binding file=zoneadmd to file=/lib/libc.so.1: symbol `strchr' 04992: binding file=zoneadmd to file=/lib/libc.so.1: symbol `sscanf' 04992: binding file=zoneadmd to file=/lib/libc.so.1: symbol `strtoul' 04992: binding file=zoneadmd to file=/lib/libc.so.1: symbol `rename' 04992: binding file=zoneadmd to file=/lib/libc.so.1: symbol `chmod' 04992: binding file=zoneadmd to file=/lib/libc.so.1: symbol `execl' 04992: binding file=zoneadmd to file=/lib/libc.so.1: symbol `lockf'
The above experience led us to consider a new feature for elfdump. What if we started with the program headers, and generated a set of "fake" section headers based on the information they contain? Obviously the information available would be reduced in comparison to the real section headers, because the program headers only contain the information needed to run the object. Nonetheless, it would certainly be better than nothing in the case where the section headers are gone. And what about the case where they are present, but we fear that they have been maliciously modified? The information from the "fake" section headers could be compared to that from the actual section headers.
As a result of this worm episode and the aftermath, I added the -P option to elfdump last July:
With an object that has section headers, fake section headers will not be used unless you explicitly use the -P option. If an object doesn't have any section headers, then elfdump automatically turns on the -P option for you.PSARC 2007/395 Add -P option to elfdump 6530249 elfdump should handle ELF files with no section header table
Let's use the new elfdump with fake section headers to examine the telnet worm. I apologize for the length of this output, but the length underscores the point — there is a lot of information that we can recover from this damaged object:
% elfdump zoneadmd
ELF Header
ei_magic: { 0x7f, E, L, F }
ei_class: ELFCLASS32 ei_data: ELFDATA2LSB
e_machine: EM_386 e_version: EV_CURRENT
e_type: ET_EXEC
e_flags: 0
e_entry: 0x80512d4 e_ehsize: 52 e_shstrndx: 0
e_shoff: 0 e_shentsize: 0 e_shnum: 0 (see shdr[0].sh_size)
e_phoff: 0x34 e_phentsize: 32 e_phnum: 5
Program Header[0]:
p_vaddr: 0x8050034 p_flags: [ PF_X PF_R ]
p_paddr: 0 p_type: [ PT_PHDR ]
p_filesz: 0xa0 p_memsz: 0xa0
p_offset: 0x34 p_align: 0
Program Header[1]:
p_vaddr: 0 p_flags: [ PF_R ]
p_paddr: 0 p_type: [ PT_INTERP ]
p_filesz: 0x11 p_memsz: 0
p_offset: 0xd4 p_align: 0
Program Header[2]:
p_vaddr: 0x8050000 p_flags: [ PF_X PF_R ]
p_paddr: 0 p_type: [ PT_LOAD ]
p_filesz: 0x6491 p_memsz: 0x6491
p_offset: 0 p_align: 0x10000
Program Header[3]:
p_vaddr: 0x8066494 p_flags: [ PF_X PF_W PF_R ]
p_paddr: 0 p_type: [ PT_LOAD ]
p_filesz: 0x3e0 p_memsz: 0xc10
p_offset: 0x6494 p_align: 0x10000
Program Header[4]:
p_vaddr: 0x80665c4 p_flags: [ PF_X PF_W PF_R ]
p_paddr: 0 p_type: [ PT_DYNAMIC ]
p_filesz: 0xd8 p_memsz: 0
p_offset: 0x65c4 p_align: 0
Section Header[1]: sh_name: .dynamic(phdr)
sh_addr: 0x80665c4 sh_flags: [ SHF_WRITE SHF_ALLOC ]
sh_size: 0xd8 sh_type: [ SHT_DYNAMIC ]
sh_offset: 0x65c4 sh_entsize: 0x8 (27 entries)
sh_link: 2 sh_info: 0
sh_addralign: 0x4
Section Header[2]: sh_name: .dynstr(phdr)
sh_addr: 0x8050890 sh_flags: [ SHF_ALLOC ]
sh_size: 0x2da sh_type: [ SHT_STRTAB ]
sh_offset: 0x890 sh_entsize: 0
sh_link: 0 sh_info: 0
sh_addralign: 0x1
Section Header[3]: sh_name: .dynsym(phdr)
sh_addr: 0x8050380 sh_flags: [ SHF_ALLOC ]
sh_size: 0x510 sh_type: [ SHT_DYNSYM ]
sh_offset: 0x380 sh_entsize: 0x10 (81 entries)
sh_link: 2 sh_info: 1
sh_addralign: 0x4
Section Header[4]: sh_name: .hash(phdr)
sh_addr: 0x80500e8 sh_flags: [ SHF_ALLOC ]
sh_size: 0x298 sh_type: [ SHT_HASH ]
sh_offset: 0xe8 sh_entsize: 0x4 (166 entries)
sh_link: 3 sh_info: 0
sh_addralign: 0x4
Section Header[5]: sh_name: .SUNW_version(phdr)
sh_addr: 0x8050b6c sh_flags: [ SHF_ALLOC ]
sh_size: 0xa0 sh_type: [ SHT_SUNW_verneed ]
sh_offset: 0xb6c sh_entsize: 0x1 (160 entries)
sh_link: 2 sh_info: 5
sh_addralign: 0x4
Section Header[6]: sh_name: .interp(phdr)
sh_addr: 0x80500d4 sh_flags: [ SHF_ALLOC ]
sh_size: 0x11 sh_type: [ SHT_PROGBITS ]
sh_offset: 0xd4 sh_entsize: 0
sh_link: 0 sh_info: 0
sh_addralign: 0x1
Section Header[7]: sh_name: .rel(phdr)
sh_addr: 0x8050c0c sh_flags: [ SHF_ALLOC ]
sh_size: 0x258 sh_type: [ SHT_REL ]
sh_offset: 0xc0c sh_entsize: 0x8 (75 entries)
sh_link: 3 sh_info: 0
sh_addralign: 0x4
Interpreter Section: .interp(phdr)
/usr/lib/ld.so.1
Version Needed Section: .SUNW_version(phdr)
file version
libnsl.so.1 SUNW_0.7
libsocket.so.1 SUNW_0.7
librt.so.1 SUNW_1.2
libpthread.so.1 SUNW_1.2
libc.so.1 SUNW_1.1
Symbol Table Section: .dynsym(phdr)
index value size type bind oth ver shndx name
[0] 0x00000000 0x00000000 NOTY LOCL D 0 UNDEF
[1] 0x08067088 0x00000004 OBJT WEAK D 0 22 environ
[2] 0x080511f4 0x00000000 FUNC GLOB D 0 UNDEF dup2
[3] 0x00000000 0x00000000 NOTY WEAK D 0 UNDEF _Jv_RegisterClasses
[4] 0x080510a4 0x00000000 FUNC GLOB D 0 UNDEF strstr
[5] 0x08050f64 0x00000000 FUNC GLOB D 0 UNDEF pthread_cancel
[6] 0x080511a4 0x00000000 FUNC GLOB D 0 UNDEF open
[7] 0x080511d4 0x00000000 FUNC GLOB D 0 UNDEF nanosleep
[8] 0x08051244 0x00000000 FUNC GLOB D 0 UNDEF localtime_r
[9] 0x080665c4 0x00000000 OBJT GLOB D 0 15 _DYNAMIC
[10] 0x08050ea4 0x00000000 FUNC GLOB D 0 UNDEF exit
[11] 0x08051174 0x00000000 FUNC GLOB D 0 UNDEF _lxstat
[12] 0x080670a4 0x00000000 OBJT GLOB D 0 22 _end
[13] 0x08050fd4 0x00000000 FUNC GLOB D 0 UNDEF ntohl
[14] 0x080510e4 0x00000000 FUNC GLOB D 0 UNDEF unlink
[15] 0x08051114 0x00000000 FUNC GLOB D 0 UNDEF lseek
[16] 0x08050ee4 0x00000000 FUNC GLOB D 0 UNDEF getenv
[17] 0x08051234 0x00000000 FUNC GLOB D 0 UNDEF waitpid
[18] 0x080510d4 0x00000000 FUNC GLOB D 0 UNDEF fopen
[19] 0x08051164 0x00000000 FUNC GLOB D 0 UNDEF _xstat
[20] 0x08051074 0x00000000 FUNC GLOB D 0 UNDEF read
[21] 0x080511e4 0x00000000 FUNC GLOB D 0 UNDEF fork
[22] 0x08051094 0x00000000 FUNC GLOB D 0 UNDEF gettimeofday
[23] 0x08050fe4 0x00000000 FUNC GLOB D 0 UNDEF socket
[24] 0x00000000 0x00000000 NOTY WEAK D 0 UNDEF __deregister_frame_info_bases
[25] 0x08051294 0x00000000 FUNC GLOB D 0 UNDEF rename
[26] 0x08050f54 0x00000000 FUNC GLOB D 0 UNDEF malloc
[27] 0x080511b4 0x00000000 FUNC GLOB D 0 UNDEF mmap
[28] 0x08050f94 0x00000000 FUNC GLOB D 0 UNDEF snprintf
[29] 0x08051284 0x00000000 FUNC GLOB D 0 UNDEF strtoul
[30] 0x08051264 0x00000000 FUNC GLOB D 0 UNDEF strchr
[31] 0x08051274 0x00000000 FUNC GLOB D 0 UNDEF sscanf
[32] 0x08051224 0x00000000 FUNC GLOB D 0 UNDEF kill
[33] 0x08051254 0x00000000 FUNC GLOB D 0 UNDEF utimes
[34] 0x08051184 0x00000000 FUNC GLOB D 0 UNDEF _fxstat
[35] 0x08050e94 0x00000000 FUNC GLOB D 0 UNDEF __fpstart
[36] 0x08051124 0x00000000 FUNC GLOB D 0 UNDEF fclose
[37] 0x080510b4 0x00000000 FUNC GLOB D 0 UNDEF mkstemp
[38] 0x08066494 0x00000000 OBJT GLOB D 0 14 _GLOBAL_OFFSET_TABLE_
[39] 0x080512b4 0x00000000 FUNC GLOB D 0 UNDEF execl
[40] 0x08051214 0x00000000 FUNC GLOB D 0 UNDEF execve
[41] 0x08051144 0x00000000 FUNC GLOB D 0 UNDEF fread
[42] 0x08050e74 0x00000000 FUNC WEAK D 0 UNDEF _cleanup
[43] 0x08050f24 0x00000000 FUNC GLOB D 0 UNDEF signal
[44] 0x08051064 0x00000000 FUNC GLOB D 0 UNDEF write
[45] 0x080510c4 0x00000000 FUNC GLOB D 0 UNDEF fdopen
[46] 0x08050e64 0x00000000 OBJT GLOB D 0 9 _PROCEDURE_LINKAGE_TABLE_
[47] 0x08051154 0x00000000 FUNC GLOB D 0 UNDEF putc
[48] 0x08056491 0x00000000 OBJT GLOB D 0 13 _etext
[49] 0x08050ef4 0x00000000 FUNC GLOB D 0 UNDEF setsid
[50] 0x080512a4 0x00000000 FUNC GLOB D 0 UNDEF chmod
[51] 0x08051194 0x00000000 FUNC GLOB D 0 UNDEF _xmknod
[52] 0x08066874 0x00000000 OBJT GLOB D 0 21 _edata
[53] 0x08051054 0x00000000 FUNC GLOB D 0 UNDEF select
[54] 0x080668a0 0x000003c0 OBJT WEAK D 0 22 _iob
[55] 0x08051014 0x00000000 FUNC GLOB D 0 UNDEF connect
[56] 0x08050fb4 0x00000000 FUNC GLOB D 0 UNDEF gethostbyname
[57] 0x080511c4 0x00000000 FUNC GLOB D 0 UNDEF strrchr
[58] 0x080668a0 0x000003c0 OBJT GLOB D 0 22 __iob
[59] 0x08050f14 0x00000000 FUNC GLOB D 0 UNDEF fflush
[60] 0x08051034 0x00000000 FUNC GLOB D 0 UNDEF getsockopt
[61] 0x08051044 0x00000000 FUNC GLOB D 0 UNDEF fcntl
[62] 0x080512c4 0x00000000 FUNC GLOB D 0 UNDEF lockf
[63] 0x08050fa4 0x00000000 FUNC GLOB D 0 UNDEF inet_addr
[64] 0x08050f44 0x00000000 FUNC GLOB D 0 UNDEF pthread_join
[65] 0x08051024 0x00000000 FUNC GLOB D 0 UNDEF ___errno
[66] 0x08051104 0x00000000 FUNC GLOB D 0 UNDEF fputc
[67] 0x08050e84 0x00000000 FUNC GLOB D 0 UNDEF atexit
[68] 0x08050f04 0x00000000 FUNC GLOB D 0 UNDEF printf
[69] 0x00000000 0x00000000 NOTY WEAK D 0 UNDEF __register_frame_info_bases
[70] 0x08051204 0x00000000 FUNC GLOB D 0 UNDEF pipe
[71] 0x08051004 0x00000000 FUNC GLOB D 0 UNDEF htonl
[72] 0x08050fc4 0x00000000 FUNC GLOB D 0 UNDEF bcopy
[73] 0x08050ff4 0x00000000 FUNC GLOB D 0 UNDEF htons
[74] 0x08050f34 0x00000000 FUNC GLOB D 0 UNDEF pthread_create
[75] 0x08067088 0x00000004 OBJT GLOB D 0 22 _environ
[76] 0x08050f84 0x00000000 FUNC GLOB D 0 UNDEF close
[77] 0x08050f74 0x00000000 FUNC GLOB D 0 UNDEF free
[78] 0x080510f4 0x00000000 FUNC GLOB D 0 UNDEF fputs
[79] 0x08051134 0x00000000 FUNC GLOB D 0 UNDEF fprintf
[80] 0x08051084 0x00000000 FUNC GLOB D 0 UNDEF strcpy
Hash Section: .hash(phdr)
bucket symndx name
0 [1] environ
[2] dup2
1 [3] _Jv_RegisterClasses
[4] strstr
2 [5] pthread_cancel
[6] open
[7] nanosleep
[8] localtime_r
3 [9] _DYNAMIC
4 [10] exit
6 [11] _lxstat
10 [12] _end
15 [13] ntohl
[14] unlink
18 [15] lseek
[16] getenv
19 [17] waitpid
20 [18] fopen
21 [19] _xstat
[20] read
22 [21] fork
23 [22] gettimeofday
[23] socket
[24] __deregister_frame_info_bases
24 [25] rename
[26] malloc
27 [27] mmap
28 [28] snprintf
29 [29] strtoul
[30] strchr
30 [31] sscanf
[32] kill
31 [33] utimes
33 [34] _fxstat
[35] __fpstart
[36] fclose
34 [37] mkstemp
35 [38] _GLOBAL_OFFSET_TABLE_
38 [39] execl
39 [40] execve
[41] fread
41 [42] _cleanup
43 [43] signal
[44] write
44 [45] fdopen
46 [46] _PROCEDURE_LINKAGE_TABLE_
[47] putc
[48] _etext
47 [49] setsid
48 [50] chmod
49 [51] _xmknod
52 [52] _edata
[53] select
[54] _iob
53 [55] connect
55 [56] gethostbyname
[57] strrchr
59 [58] __iob
[59] fflush
62 [60] getsockopt
[61] fcntl
[62] lockf
63 [63] inet_addr
[64] pthread_join
[65] ___errno
64 [66] fputc
[67] atexit
65 [68] printf
66 [69] __register_frame_info_bases
[70] pipe
[71] htonl
71 [72] bcopy
73 [73] htons
76 [74] pthread_create
77 [75] _environ
[76] close
78 [77] free
80 [78] fputs
[79] fprintf
81 [80] strcpy
35 buckets contain 0 symbols
25 buckets contain 1 symbols
15 buckets contain 2 symbols
7 buckets contain 3 symbols
1 buckets contain 4 symbols
83 buckets 80 symbols (globals)
Relocation Section: .rel(phdr)
type offset section symbol
R_386_GLOB_DAT 0x80664b0 .rel(phdr) __deregister_frame_info_bases
R_386_GLOB_DAT 0x80664b8 .rel(phdr) __register_frame_info_bases
R_386_GLOB_DAT 0x80664bc .rel(phdr) _Jv_RegisterClasses
R_386_COPY 0x8067088 .rel(phdr) _environ
R_386_COPY 0x80668a0 .rel(phdr) __iob
R_386_JMP_SLOT 0x80664a0 .rel(phdr) _cleanup
R_386_JMP_SLOT 0x80664a4 .rel(phdr) atexit
R_386_JMP_SLOT 0x80664a8 .rel(phdr) __fpstart
R_386_JMP_SLOT 0x80664ac .rel(phdr) exit
R_386_JMP_SLOT 0x80664b4 .rel(phdr) __deregister_frame_info_bases
R_386_JMP_SLOT 0x80664c0 .rel(phdr) _Jv_RegisterClasses
R_386_JMP_SLOT 0x80664c4 .rel(phdr) __register_frame_info_bases
R_386_JMP_SLOT 0x80664c8 .rel(phdr) getenv
R_386_JMP_SLOT 0x80664cc .rel(phdr) setsid
R_386_JMP_SLOT 0x80664d0 .rel(phdr) printf
R_386_JMP_SLOT 0x80664d4 .rel(phdr) fflush
R_386_JMP_SLOT 0x80664d8 .rel(phdr) signal
R_386_JMP_SLOT 0x80664dc .rel(phdr) pthread_create
R_386_JMP_SLOT 0x80664e0 .rel(phdr) pthread_join
R_386_JMP_SLOT 0x80664e4 .rel(phdr) malloc
R_386_JMP_SLOT 0x80664e8 .rel(phdr) pthread_cancel
R_386_JMP_SLOT 0x80664ec .rel(phdr) free
R_386_JMP_SLOT 0x80664f0 .rel(phdr) close
R_386_JMP_SLOT 0x80664f4 .rel(phdr) snprintf
R_386_JMP_SLOT 0x80664f8 .rel(phdr) inet_addr
R_386_JMP_SLOT 0x80664fc .rel(phdr) gethostbyname
R_386_JMP_SLOT 0x8066500 .rel(phdr) bcopy
R_386_JMP_SLOT 0x8066504 .rel(phdr) ntohl
R_386_JMP_SLOT 0x8066508 .rel(phdr) socket
R_386_JMP_SLOT 0x806650c .rel(phdr) htons
R_386_JMP_SLOT 0x8066510 .rel(phdr) htonl
R_386_JMP_SLOT 0x8066514 .rel(phdr) connect
R_386_JMP_SLOT 0x8066518 .rel(phdr) ___errno
R_386_JMP_SLOT 0x806651c .rel(phdr) getsockopt
R_386_JMP_SLOT 0x8066520 .rel(phdr) fcntl
R_386_JMP_SLOT 0x8066524 .rel(phdr) select
R_386_JMP_SLOT 0x8066528 .rel(phdr) write
R_386_JMP_SLOT 0x806652c .rel(phdr) read
R_386_JMP_SLOT 0x8066530 .rel(phdr) strcpy
R_386_JMP_SLOT 0x8066534 .rel(phdr) gettimeofday
R_386_JMP_SLOT 0x8066538 .rel(phdr) strstr
R_386_JMP_SLOT 0x806653c .rel(phdr) mkstemp
R_386_JMP_SLOT 0x8066540 .rel(phdr) fdopen
R_386_JMP_SLOT 0x8066544 .rel(phdr) fopen
R_386_JMP_SLOT 0x8066548 .rel(phdr) unlink
R_386_JMP_SLOT 0x806654c .rel(phdr) fputs
R_386_JMP_SLOT 0x8066550 .rel(phdr) fputc
R_386_JMP_SLOT 0x8066554 .rel(phdr) lseek
R_386_JMP_SLOT 0x8066558 .rel(phdr) fclose
R_386_JMP_SLOT 0x806655c .rel(phdr) fprintf
R_386_JMP_SLOT 0x8066560 .rel(phdr) fread
R_386_JMP_SLOT 0x8066564 .rel(phdr) putc
R_386_JMP_SLOT 0x8066568 .rel(phdr) _xstat
R_386_JMP_SLOT 0x806656c .rel(phdr) _lxstat
R_386_JMP_SLOT 0x8066570 .rel(phdr) _fxstat
R_386_JMP_SLOT 0x8066574 .rel(phdr) _xmknod
R_386_JMP_SLOT 0x8066578 .rel(phdr) open
R_386_JMP_SLOT 0x806657c .rel(phdr) mmap
R_386_JMP_SLOT 0x8066580 .rel(phdr) strrchr
R_386_JMP_SLOT 0x8066584 .rel(phdr) nanosleep
R_386_JMP_SLOT 0x8066588 .rel(phdr) fork
R_386_JMP_SLOT 0x806658c .rel(phdr) dup2
R_386_JMP_SLOT 0x8066590 .rel(phdr) pipe
R_386_JMP_SLOT 0x8066594 .rel(phdr) execve
R_386_JMP_SLOT 0x8066598 .rel(phdr) kill
R_386_JMP_SLOT 0x806659c .rel(phdr) waitpid
R_386_JMP_SLOT 0x80665a0 .rel(phdr) localtime_r
R_386_JMP_SLOT 0x80665a4 .rel(phdr) utimes
R_386_JMP_SLOT 0x80665a8 .rel(phdr) strchr
R_386_JMP_SLOT 0x80665ac .rel(phdr) sscanf
R_386_JMP_SLOT 0x80665b0 .rel(phdr) strtoul
R_386_JMP_SLOT 0x80665b4 .rel(phdr) rename
R_386_JMP_SLOT 0x80665b8 .rel(phdr) chmod
R_386_JMP_SLOT 0x80665bc .rel(phdr) execl
R_386_JMP_SLOT 0x80665c0 .rel(phdr) lockf
Dynamic Section: .dynamic(phdr)
index tag value
[0] NEEDED 0x27f libnsl.so.1
[1] NEEDED 0x294 libsocket.so.1
[2] NEEDED 0x2a3 librt.so.1
[3] NEEDED 0x2b7 libpthread.so.1
[4] NEEDED 0x2c7 libc.so.1
[5] INIT 0x80538fc
[6] FINI 0x8053909
[7] HASH 0x80500e8
[8] STRTAB 0x8050890
[9] STRSZ 0x2da
[10] SYMTAB 0x8050380
[11] SYMENT 0x10
[12] CHECKSUM 0x3126
[13] VERNEED 0x8050b6c
[14] VERNEEDNUM 0x5
[15] PLTRELSZ 0x230
[16] PLTREL 0x11
[17] JMPREL 0x8050c34
[18] REL 0x8050c0c
[19] RELSZ 0x258
[20] RELENT 0x8
[21] DEBUG 0
[22] FEATURE_1 0x1 [ PARINIT ]
[23] FLAGS 0 0
[24] FLAGS_1 0 0
[25] PLTGOT 0x8066494
[26] NULL 0
Technorati Tag:
OpenSolaris
Technorati Tag:
Solaris
Posted at 09:56AM Nov 01, 2007 by ali in Sun |
Changing ELF Runpaths (Code Included)
A recent change to Solaris ELF files makes it possible to change the runpath of a dynamic executable or sharable object, something that has not been safely possible up until now. This change 80is currently found in Solaris Nevada (the current development version of Solaris) and in OpenSolaris. It is not yet available in Solaris 10, but in time will appear in the standard shipping Solaris as well.
This seems like a good time to talk about runpaths and the business of how the runtime linker finds dependencies. I also provide a small program named rpath that you can use to modify the runpaths in your file (assuming they were linked under Nevada or OpenSolaris).
The Runpath Problem
The runtime linker looks in the following places, in the order listed, to find the sharable objects it loads into a process at startup time:- If LD_LIBRARY_PATH (or the related LD_LIBRARY_PATH_32 and LD_LIBRARY_PATH_64) environment variables are defined, the directories they specify are searched to resolve any remaining dependencies.
- If the executable, or any sharable objects that are loaded, contain a runpath, the directories it specifies are searched to resolve dependencies for those objects.
- Finally, it searches two default locations for any remaining dependencies: /lib and /usr/lib (or /lib/64 and /usr/lib/64 for 64-bit code).
The above scheme offers a great deal of flexibility, and it usually works well. There is however one notable exception — the "Runpath Problem". The problem is that many objects are not built with a correct runpath, and once an object has been built, it has not been possible to change it. It is common to find objects where the runpath is correct on the system the object was built on, but not on the system where it is installed. Usually, we deal with this all too common situation by setting LD_LIBRARY_PATH, or by creating a linker configuration file with crle. Such solutions have serious downsides, as detailed in an earlier blog entry by Rod Evans entitled "LD_LIBRARY_PATH - just say no".
Both approaches will cause unrelated programs to look in unnecessary additional directories for their dependencies. At best, this imposes unnecessary overhead on their operation. At worst, they may end up binding to the wrong version of a given library, leading to mysterious and hard to debug failures. The environment variable approach is simply too broad.
One important technique that people sometimes use, is to set the environment variables in a wrapper shell script, that may look something like:
This is a huge improvement over simply setting LD_CONFIG or LD_LIBRARY_PATH in your shell login config script (.profile, .cshrc, .bashrc, etc), for many reasons:#!/bin/sh # # Run myapp, setting LD_LIBRARY_PATH so it will run LD_LIBRARY_PATH="/this/that/theother:/someplace/else" export LD_LIBRARY_PATH exec /usr/local/myapp
- Reduces the scope of influence to only cover the application (and its children — see below)
- Doesn't require each user to modify their login script(s)
- Can be managed in a central location
It would be far better to modify the object in question and set a runpath that accurately reflects the actual location of its dependencies. The effect of a runpath is limited to the file that contains it, so this solution does not "bleed through" to unrelated files, and it imposes no unnecessary overhead on the general operation of the system. This would be a superior solution if it were possible. However it hasn't been an option until recently.
How Runpaths Are Implemented
Every dynamic executable contains a dynamic section. This is an array of items which convey the information required by the dynamic linker (ld.so.1) to do its work. If an object has a runpath, there will be a DT_RUNPATH and/or DT_RPATH item in the dynamic section (there is more than one of these for historical reasons). As an example, lets examine crle:
% elfdump -d /usr/bin/crle | grep 'R*PATH'
[4] RUNPATH 0x612 $ORIGIN/../lib
[5] RPATH 0x612 $ORIGIN/../lib
A string table is a section that contains NULL terminated strings, one immediately following the other. To access a given string, you add the offset of the string within the section to the base of the section data area. Consider a string table that contains the names of two variables "var1", and "var2" and a runpath "$ORIGIN/../lib". By ELF convention, string tables always have a 0-length NULL terminated string in the first position. In C language notation, we might declare the contents of the resulting string table section containing these 4 strings as
The indexes of the 4 strings in our table are [0], [1], [6], and [11], and any item in the dynamic section or the dynamic symbol table that needs one of these strings will specify it using the appropriate index. An interesting result of the way that string tables are designed is that that every single offset into a string table represents a usable string. Although our intent with the C string above was to represent 4 strings, it actually contains 23 potential strings (26 if you count the duplicate NULL strings), and not just the 4 we intentionally inserted. Listing them by offset, they are:"\0var1\0var2\0$ORIGIN/../lib"
This is a very efficient scheme, since each string can appear once in the string table, and multiple ELF items can refer to it. Also, it allows fixed size things, like ELF symbols or dynamic section entries, to efficiently reference variable length strings. There are two things to note, however:[0] "" [1] "var1" [2] "ar1" [3] "r1" [4] "1" [5] "" [6] "var2" [7] "ar2" [8] "r2" [9] "2" [10] "" [11] "$ORIGIN/../lib" [12] "ORIGIN/../lib" [13] "RIGIN/../lib" [14] "IGIN/../lib" [15] "GIN/../lib" [16] "IN/../lib" [17] "N/../lib" [18] "/../lib" [19] "../lib" [20] "./lib" [21] "/lib" [22] "lib" [23] "ib" [24] "b" [25] ""
- For a given string, there is no way to tell if it is referenced, where it is referenced from, or how many references there are.
- There is no room to add new strings to a string table.
The options for modifying a runpath in this situation are limited:
- Any string already in the string table, as with the 23 options listed in our example above, can be safely set as a runpath, by simply changing the offset in the runpath DT entries. Note that most string table strings are variable and file (not directory) names that are not likely to make useful path strings. This option is unlikely to help.
- You might overwrite the existing path string with a new string of equal or shorter length in the (usually true) belief that nothing else is accessing that particular string. It is a simple matter using a binary aware editor to locate and overwrite the existing string. This usually works, but if there is another part of the file accessing that string, this change will break it. We cannot recommend or stand behind this, even though we have done it ourselves for one-off experiments (never in a shipping product).
- A better approach would be to add a new string to the end of the section, and then change the offset in the dynamic section to use it. Traditionally, ELF files have not had any extra room in the string table section to allow this.
As a result, it has not been possible to support the modification of the runpath in an existing object up until recently.
Making Room
I recently integrated a change to Solaris Nevada (and OpenSolaris) to add a little unused space to our ELF files, in order to facilitate a limited amount of post-link modification:
PSARC 2007/127 Reserved space for editing ELF dynamic sections
6516118 Reserved space needed in ELF dynamic section and
string table
- Adds some extra NULL bytes to the end of every dynamic string table. (The current value is 512 bytes, but this can change in the future).
- Adds a new dynamic section entry named DT_SUNW_STRPAD to keep track of the size of the unused space at the end of the dynamic string table.
- Adds some extra (currently 10) unused DT_NULL entries at the end of the dynamic section.
% elfdump -d /usr/bin/crle | egrep 'R*PATH|STRPAD'
[4] RUNPATH 0x612 $ORIGIN/../lib
[5] RPATH 0x612 $ORIGIN/../lib
[32] SUNW_STRPAD 0x200
The way this works is very simple: If a file lacks a DT_SUNW_STRPAD dynamic entry, then we know that it is an older file, and that the dynamic string table does not have any extra space. If it does have a DT_SUNW_STRPAD, then its value tells us how much room is available. In this case, we can add the string, modify the DT_RUNPATH items, and reduce the DT_SUNW_STRPAD value by the number of bytes we used.
If the value in DT_SUNW_STRPAD is too small for our new string, then we are out of luck and cannot add it. This extra room should help in the vast majority of cases, but as with any such approach, there are limits. We recommend the use of the special $ORIGIN token, both because it is a great way to organize objects, and because it is short.
The rpath Utility
Eventually, Solaris will ship with a standard utility for modifying runpaths. However, there is no need to wait. I have written an unofficial test program I call 'rpath' that you can download and build. To build rpath, you will need a version of Solaris Nevada newer than build 61, or a recent version of OpenSolaris. To check your system, try:If your grep doesn't find DT_SUNW_STRPAD, your system lacks the necessary support.% grep DT_SUNW_STRPAD /usr/include/sys/link.h #define DT_SUNW_STRPAD 0x60000019 /* # of unused bytes at the */
To build rpath, unpack the compressed tar file and type 'make'. If you are using gcc, first edit the Makefile and uncomment the CC line:
rpath is used as follows:% gunzip < rpath.tgz | tar xvpf - % cd rpath % make
NAME
rpath - set/get runpath of ELF dynamic objects
SYNOPSIS
rpath [-dr] file [runpath]
DESCRIPTION
rpath can display, modify, or delete the runpath of a
dynamic ELF object.
If called without a runpath argument and without the -r
option, the current runpath, if any, is written to stdout.
If -r is specified, the existing runpath is removed. If run-
path is supplied, the runpath of the object is set to the
new value.
OPTIONS
The following options are supported:
-d Cause detailed ELF information about the ELF file and
the changes being made to it to be written to stderr.
-r Instead of adding or modifying the file runpath, rpath
removes any DT_RPATH or DT_RUNPATH entries from the
dynamic section of the file. This action completely
removes any existing from the file. When this option is
used, rpath does not allow the runpath argument.
Using rpath
Let's use rpath to look at its own runpath. We will see that it doesn't have one, something that can be verified using elfdump:
% rpath rpath
% elfdump -d rpath | egrep 'R*PATH|STRPAD'
[28] SUNW_STRPAD 0x200
% rpath rpath pointless:runpath
% rpath rpath
pointless:runpath
% elfdump -d rpath | egrep 'R*PATH|STRPAD'
[28] SUNW_STRPAD 0x1ee
[30] RUNPATH 0x33f pointless:runpath
We can observe the runtime linker looking in 'pointless' and 'runpath' as it loads rpath (note: output is edited for width):
% LD_DEBUG=libs ./rpath
13707:
13707: hardware capabilities - 0x25ff7 [ AHF SSE3 SSE2
SSE FXSR AMD_3DNowx AMD_3DNow AMD_MMX MMX CMOV
AMD_SYSC CX8 TSC FPU ]
13707:
13707:
13707: configuration file=/var/ld/ld.config: unable to
process file
13707:
13707:
13707: find object=libelf.so.1; searching
13707: search path=pointless:runpath (RUNPATH/RPATH
from file rpath)
13707: trying path=pointless/libelf.so.1
13707: trying path=runpath/libelf.so.1
13707: search path=/lib (default)
13707: search path=/usr/lib (default)
13707: trying path=/lib/libelf.so.1
13707:
13707: find object=libc.so.1; searching
13707: search path=pointless:runpath (RUNPATH/RPATH from
file rpath)
13707: trying path=pointless/libc.so.1
13707: trying path=runpath/libc.so.1
13707: search path=/lib (default)
13707: search path=/usr/lib (default)
13707: trying path=/lib/libc.so.1
13707:
13707: find object=libc.so.1; searching
13707: search path=/lib (default)
13707: search path=/usr/lib (default)
13707: trying path=/lib/libc.so.1
13707:
13707: 1:
13707: 1: transferring control: rpath
13707: 1:
usage: rpath [-dr] file [runpath]
13707: 1:
% rpath -r rpath
% rpath rpath
% elfdump -d rpath | egrep 'R*PATH|STRPAD'
[28] SUNW_STRPAD 0x1ee
On the plus side, you can always re-add the same runpath back without using any more space:
% rpath rpath pointless:runpath
% rpath rpath
pointless:runpath
% elfdump -d rpath | egrep 'R*PATH|STRPAD'
[28] SUNW_STRPAD 0x1ee
[30] RUNPATH 0x33f pointless:runpath
Conclusions
Our best advice has always been that the LD_LIBRARY_PATH environment variable should not be used to work around objects with bad or missing runpaths. It is best to rebuild such objects and set the runpath correctly. This hasn't changed, and you should always do so if you can.The problem with that advice is that there are times when all you have is the object, and no option to rebuild. In that case, LD_LIBRARY_PATH has been a necessary evil (and one that we've been glad to have). With the advent of objects that can have their runpaths modified, we now have a better answer, and the use of LD_LIBRARY_PATH for this purpose should be allowed to slowly fade away.
Technorati Tag:
OpenSolaris
Technorati Tag:
Solaris
Posted at 03:49PM Jun 12, 2007 by ali in Sun | Comments[3]
Which Solaris Files Are Stripped?
In my previous blog entry about the new .SUNW_ldynsym sections, I made the following statement:
It used to be common practice for system binaries to be stripped in order to save space. However, observability is a central tenet of the Solaris philosophy. Solaris objects and executables are therefore shipped in unstripped form, and have been for many years, in order to support such symbol lookups.
It turns out that this is is only partially true...
Brian Utterback posted a comment and pointed out that 490 of the 719 ELF binaries in /usr/bin on his Solaris 10 system are stripped. This shows that Solaris binaries have not been unstripped "for many years". I looked at /usr/bin on my desktop system, which is running a fairily recent Nevada build, and found that only 51 of the 815 files there are stripped. It appears that binaries are (mostly) stripped now. What changed between Solaris 10 and today? And, why "mostly"?
As I usually do in such situations, I sent mail to my fellow Linker Alien Rod Evans. I asked him for his recollection of what policies were used for stripping Solaris files in the past. Here is a summary of what he told me:
- For a very long time, the rule was that executables are stripped, and sharable libraries are not. The underlying idea was that people would not care to debug our executables, but certainly would debug their own programs that are linked to our libraries. We're not sure when this started, but are pretty sure that it covers most, if not all, of the Solaris 2.x era (we've no idea about the SunOS 4.x days).
- In the early years, enforcement of this rule was rather incomplete, and exceptions occured. Starting with Solaris 9, automated checks in the nightly builds tightened things up significantly.
- The policy was changed in September 2005 (2 months before I joined Sun)
to not strip any files.
The change took effect with Nevada build 24, with
5072038 binaries shouldn't be stripped
I imagine that the introduction of DTrace made complete symbol information in binaries more important than before. - Solaris is built by combining various "consolidations". The above comments only apply to the core ON consolidation, which consists of the OS and Networking parts of Solaris. The other consolidations are built according to their own rules, which can and do differ. So, you should not be surprised to find some stripped files, even on a current development build of Solaris, like the 51 files I found in /usr/bin on my system.
The new .SUNW_ldynsym sections reduce the need for everything to be unstripped, so we may end up relaxing our ON rule if there is a reason to do so. And if the other consolidations continue to strip their files, .SUNW_ldynsym will provide better observability for them.
Brian is absolutely right — we have not been shipping "Solaris objects and executables" in unstripped form for many years, only sharable libraries! I knew that libraries were unstripped, and that current builds don't strip either binaries, and those two facts misled me.
On the plus side, I have a better understanding of the issue now... :-)
Technorati Tag:
OpenSolaris
Technorati Tag:
Solaris
Posted at 03:12PM Feb 09, 2007 by ali in Sun |
What Is .SUNW_ldynsym?
Solaris ELF files have a new ELF symbol table. The section type is SHT_SUNW_LDYNSYM, and the section is named .SUNW_ldynsym. In the 20+ years in which the ELF standard has been in use, we have only needed two symbol tables (.symtab, and .dynsym) to support linking, so the addition of a third symbol table is a notable event for ELF cognoscenti. Even if you aren't one of those, you may encounter these sections, and wonder what they are for. I hope to explain that here.
Solaris has many tools that examine running processes or core files and generate stack traces. For example, consider the following call to pstack(1), made on an Xterm process currently running on my system:
In order to show you those function names, pstack (really the libproc library used by pstack) needs to map the addresses of functions on the stack to the ELF symbols that correspond to them. Usually, these symbols come from the symbol table (.symtab). If this symbol table has been removed with the strip(1) program, then the dynamic symbol table (.dynsym) will be used instead. As described in a previous blog entry, the .dynsym contains the subset of global symbols from .symtab that are needed by the runtime linker ld.so.1(1). This fallback allows us to map global functions to their names, but local function symbols are not available. Observability tools like pstack(1) will display the hexidecimal address of such local functions when a name is not available. This is better than nothing, but is not particularly helpful.% pstack 3094 3094: xterm -ls -geometry 80x51+0+175 fef4bea7 pollsys (8046600, 2, 0, 0) fef0767e pselect (5, 8400168, 84001e8, fef95260, 0, 0) + 19e fef0798e select (5, 8400168, 84001e8, 0, 0) + 7e 0805b250 in_put (10, 8416720, 0, fedd561e, 8416720, 0) + 1b0 08059b20 VTparse (84166a8, 8057acc, fed387c5, 8416720, 84166a8, 804688c) + 90 0805d1f1 VTRun (8046a28, 8046870, feffa7c0, 8046808, 8046858, 804685c) + 205 08057add main (0, 80468b4, 80468c8) + 945 08056eee _start (4, 8046a90, 0, 8046a9a, 8046aa4, 0) + 7a
It used to be common practice for system binaries to be stripped in order to save space. However, observability is a central tenet of the Solaris philosophy. Solaris objects and executables are therefore shipped in unstripped form, and have been for many years, in order to support such symbol lookups. For the most part, this has been a winning strategy, but there are still issues that come up from time to time:
- strip(1) removes much more than the symbol table. Usually the size of this extra data is not a significant concern, but there are certain very large programs where the space savings might be worthwhile. It would be great to strip those particular things, but losing the local function symbols and the ability to make accurate stack traces is a bitter pill to swallow. This has led to a number of proposed features to "strip everything except local function symbols". These ideas are reasonable, but complicated. We like the fact that "strip" is a simple straightforward operation, and want to avoid complicating the concept.
- We don't strip our files, but many Solaris users do. This becomes a problem when those applications misbehave, and they (or we, if you have a high end support contract) are trying to figure out why. Often, it is not possible to rebuild such applications in order to debug them. The ability to observe unmodified applications running in a production environment is another key Solaris virtue, as exemplified by DTrace.
I tried hard to avoid adding a new symbol table type, and instead tried several experiments in which the additional local function symbols were placed in the dynsym. The reason for wanting this was to avoid having to modify ELF utilities and debuggers to know about a new symbol table. If the added symbols are in the existing .dynsym, those tools will automatically see them, without needing modification. As detailed in the ARC case that I filed for this work (PSARC/2006/526), I tried many different permutations. In every case, I discovered undesirable backward compatibility issues that kept me from using that solution. It turns out that the layout of .dynsym, and the other ELF sections that interact with it, are completely constrained, and there is no 100% backward compatible way to add local symbols to it.
ELF was designed from the very beginning to make it possible to introduce new section types with full backward/forward compatibility. You can always safely add a new section, with a moderate amount of care, and it will work. More than anything, this ability to extend ELF accounts for its long life. Given that the .dynsym cannot be extended with local symbols, I made the obvious (in hindsight) decision to to introduce a new section type (SHT_SUNW_LDYNSYM), and add a new symbol table section named .SUNW_ldynsym to every Solaris file that has a .dynsym section. Once that decision was made, the implementation was straightforward, giving me confidence that it was the right way to go.
The .SUNW_ldynsym section can be thought of as the local initial part of the .dynsym that we wish to build, but can't. The Solaris linker ( ld(1)) takes care to actually place them side by side, so that the end of the .SUNW_ldynsym section leads directly into the start of the .dynsym section. The runtime linker ( ld.so.1(1)) takes advantage of this to treat them as a single table within the implementation of dladdr(3C). Note that this trick works for applications that mmap(2) the file and access it directly. If you are accessing an ELF file via libelf, as many utilities do, you can't make any assumptions about the relative positions of different sections.
As with .dynsym, .SUNW_ldynsym sections are allocable, meaning that they are part of the process text segment. This means that they are available at runtime for dladdr(3C). It also means that they cannot be stripped. Although you cannot strip .SUNW_ldynsym sections, you can prevent them from being generated by ld(1), by using the -znoldynsym linker option.
.SUNW_ldynsym sections consume a small amount of additional space. We found that for all of core Solaris (OS and Networking), the increase in size was on the order of 1.4%. This small increase pays off by letting our observability tools do a better job. Furthermore, the presence of .SUNW_ldynsym means that in many cases, you can strip programs that you might not have been willing to strip before.
Example
Let's use the following program to see how .SUNW_ldynsym sections improve Solaris observability of local functions:
/*
* Program to demonstrate SHT_SUNW_LDYNSYM sections. The
* global main program calls a local function named
* static_func(). static_func() uses printstack() to exercise
* the dladdr(3C) function provided by the runtime linker,
* and then deliberately causes a segfault. The resulting core
* file can be examined by pstack(1) or mdb(1).
*
* In all these cases, if a stripped binary of this program
* contains a .SUNW_ldynsym section, the static_func() function
* will be observable by name, and otherwise simply as an
* address.
*/
#include <ucontext.h>
static void
static_func(void)
{
/* Use dladdr(3C) to print a call stack */
printstack(1);
/*
* Write to address 0, killing the process and
* producing a core file.
*/
*((char *) 0) = 1;
}
int main(int argc, char *argv[])
{
static_func();
return (0);
}
The elfdump(1) command can be used to let us examine the three symbol tables contained in test_ldynsym. There is no need to examine this (large) output too carefully, but there are some interesting facts worth noticing:% cc -Wl,-znoldynsym test.c -o test_noldynsym % cc test.c -o test_ldynsym
- Every symbol in .SUNW_ldynsym or .dynsym is also found in .symtab, because .symtab is a superset of the other two tables. This is why it is always preferred to the other two, when available.
- .symtab is much larger than the other two tables combined, which leads to the temptation to strip it, along with the other things strip(1) removes.
- The symbols in .dynsym are strictly limited to those needed by the runtime linker.
- If you consider the .SUNW_ldynsym and .dynsym symbol tables as a single logical entity, you can see that the result follows the rules for ELF symbol table layout.
% elfdump -s test_ldynsym
Symbol Table Section: .SUNW_ldynsym
index value size type bind oth ver shndx name
[0] 0x00000000 0x00000000 NOTY LOCL D 0 UNDEF
[1] 0x00000000 0x00000000 FILE LOCL D 0 ABS test_ldynsym
[2] 0x00000000 0x00000000 FILE LOCL D 0 ABS crti.s
[3] 0x00000000 0x00000000 FILE LOCL D 0 ABS crt1.o
[4] 0x00000000 0x00000000 FILE LOCL D 0 ABS crt1.s
[5] 0x00000000 0x00000000 FILE LOCL D 0 ABS fsr.s
[6] 0x00000000 0x00000000 FILE LOCL D 0 ABS values-Xa.c
[7] 0x00000000 0x00000000 FILE LOCL D 0 ABS test.c
[8] 0x080507f0 0x00000019 FUNC LOCL D 0 .text static_func
[9] 0x00000000 0x00000000 FILE LOCL D 0 ABS crtn.s
Symbol Table Section: .dynsym
index value size type bind oth ver shndx name
[0] 0x00000000 0x00000000 NOTY LOCL D 0 UNDEF
[1] 0x08050668 0x00000000 OBJT GLOB D 0 .plt _PROCEDURE_LINKAGE_TABLE_
[2] 0x08060974 0x00000004 OBJT WEAK D 0 .data environ
[3] 0x0806088c 0x00000000 OBJT GLOB D 0 .dynamic _DYNAMIC
[4] 0x080609c0 0x00000000 OBJT GLOB D 0 .bssf _edata
[5] 0x08060990 0x00000004 OBJT GLOB D 0 .data ___Argv
[6] 0x08050868 0x00000000 OBJT GLOB D 0 .rodata _etext
[7] 0x0805082c 0x0000001b FUNC GLOB D 0 .init _init
[8] 0x00000000 0x00000000 NOTY GLOB D 0 ABS __fsr_init_value
[9] 0x08050810 0x00000019 FUNC GLOB D 0 .text main
[10] 0x08060974 0x00000004 OBJT GLOB D 0 .data _environ
[11] 0x08060868 0x00000000 OBJT GLOB P 0 .got _GLOBAL_OFFSET_TABLE_
[12] 0x080506b8 0x00000000 FUNC GLOB D 0 UNDEF printstack
[13] 0x080506a8 0x00000000 FUNC GLOB D 0 UNDEF _exit
[14] 0x08050864 0x00000004 OBJT GLOB D 0 .rodata _lib_version
[15] 0x08050698 0x00000000 FUNC GLOB D 0 UNDEF atexit
[16] 0x08050678 0x00000000 FUNC GLOB D 0 UNDEF __fpstart
[17] 0x0805076c 0x0000007b FUNC GLOB D 0 .text __fsr
[18] 0x08050688 0x00000000 FUNC GLOB D 0 UNDEF exit
[19] 0x080506c8 0x00000000 FUNC WEAK D 0 UNDEF _get_exit_frame_monitor
[20] 0x080609c0 0x00000000 OBJT GLOB D 0 .bss _end
[21] 0x080506e0 0x0000008b FUNC GLOB D 0 .text _start
[22] 0x08050848 0x0000001b FUNC GLOB D 0 .fini _fini
[23] 0x08060978 0x00000018 OBJT GLOB D 0 .data __environ_lock
[24] 0x0806099c 0x00000004 OBJT GLOB D 0 .data __longdouble_used
[25] 0x00000000 0x00000000 NOTY WEAK D 0 UNDEF __1cG__CrunMdo_exit_code6F_v_
Symbol Table Section: .symtab
index value size type bind oth ver shndx name
[0] 0x00000000 0x00000000 NOTY LOCL D 0 UNDEF
[1] 0x00000000 0x00000000 FILE LOCL D 0 ABS test_ldynsym
[2] 0x080500f4 0x00000000 SECT LOCL D 0 .interp
[3] 0x08050108 0x00000000 SECT LOCL D 0 .SUNW_cap
[4] 0x08050118 0x00000000 SECT LOCL D 0 .hash
[5] 0x080501fc 0x00000000 SECT LOCL D 0 .SUNW_ldynsym
[6] 0x0805029c 0x00000000 SECT LOCL D 0 .dynsym
[7] 0x0805043c 0x00000000 SECT LOCL D 0 .dynstr
[8] 0x080505c4 0x00000000 SECT LOCL D 0 .SUNW_version
[9] 0x080505f4 0x00000000 SECT LOCL D 0 .SUNW_dynsymso
[10] 0x08050630 0x00000000 SECT LOCL D 0 .rel.data
[11] 0x08050638 0x00000000 SECT LOCL D 0 .rel.plt
[12] 0x08050668 0x00000000 SECT LOCL D 0 .plt
[13] 0x080506e0 0x00000000 SECT LOCL D 0 .text
[14] 0x0805082c 0x00000000 SECT LOCL D 0 .init
[15] 0x08050848 0x00000000 SECT LOCL D 0 .fini
[16] 0x08050864 0x00000000 SECT LOCL D 0 .rodata
[17] 0x08060868 0x00000000 SECT LOCL D 0 .got
[18] 0x0806088c 0x00000000 SECT LOCL D 0 .dynamic
[19] 0x08060974 0x00000000 SECT LOCL D 0 .data
[20] 0x080609c0 0x00000000 SECT LOCL D 0 .bssf
[21] 0x080609c0 0x00000000 SECT LOCL D 0 .bss
[22] 0x00000000 0x00000000 SECT LOCL D 0 .symtab
[23] 0x00000000 0x00000000 SECT LOCL D 0 .strtab
[24] 0x00000000 0x00000000 SECT LOCL D 0 .comment
[25] 0x00000000 0x00000000 SECT LOCL D 0 .debug_info
[26] 0x00000000 0x00000000 SECT LOCL D 0 .debug_line
[27] 0x00000000 0x00000000 SECT LOCL D 0 .debug_abbrev
[28] 0x00000000 0x00000000 SECT LOCL D 0 .shstrtab
[29] 0x080609c0 0x00000000 OBJT LOCL D 0 .bss _END_
[30] 0x08050000 0x00000000 OBJT LOCL D 0 .interp _START_
[31] 0x00000000 0x00000000 FILE LOCL D 0 ABS crti.s
[32] 0x00000000 0x00000000 FILE LOCL D 0 ABS crt1.o
[33] 0x00000000 0x00000000 FILE LOCL D 0 ABS crt1.s
[34] 0x08060994 0x00000004 OBJT LOCL D 0 .data __get_exit_frame_monitor_ptr
[35] 0x08060998 0x00000004 OBJT LOCL D 0 .data __do_exit_code_ptr
[36] 0x00000000 0x00000000 FILE LOCL D 0 ABS fsr.s
[37] 0x080609a0 0x00000020 OBJT LOCL D 0 .data trap_table
[38] 0x00000000 0x00000000 FILE LOCL D 0 ABS values-Xa.c
[39] 0x08060974 0x00000000 NOTY LOCL D 0 .data Ddata.data
[40] 0x080609c0 0x00000000 NOTY LOCL D 0 .bss Bbss.bss
[41] 0x08050868 0x00000000 NOTY LOCL D 0 .rodata Drodata.rodata
[42] 0x00000000 0x00000000 FILE LOCL D 0 ABS test.c
[43] 0x080507f0 0x00000019 FUNC LOCL D 0 .text static_func
[44] 0x080609c0 0x00000000 OBJT LOCL D 0 .bss Bbss.bss
[45] 0x08060974 0x00000000 OBJT LOCL D 0 .data Ddata.data
[46] 0x08050864 0x00000000 OBJT LOCL D 0 .rodata Drodata.rodata
[47] 0x00000000 0x00000000 FILE LOCL D 0 ABS crtn.s
[48] 0x08050668 0x00000000 OBJT GLOB D 0 .plt _PROCEDURE_LINKAGE_TABLE_
[49] 0x08060974 0x00000004 OBJT WEAK D 0 .data environ
[50] 0x0806088c 0x00000000 OBJT GLOB D 0 .dynamic _DYNAMIC
[51] 0x080609c0 0x00000000 OBJT GLOB D 0 .bssf _edata
[52] 0x08060990 0x00000004 OBJT GLOB D 0 .data ___Argv
[53] 0x08050868 0x00000000 OBJT GLOB D 0 .rodata _etext
[54] 0x0805082c 0x0000001b FUNC GLOB D 0 .init _init
[55] 0x00000000 0x00000000 NOTY GLOB D 0 ABS __fsr_init_value
[56] 0x08050810 0x00000019 FUNC GLOB D 0 .text main
[57] 0x08060974 0x00000004 OBJT GLOB D 0 .data _environ
[58] 0x08060868 0x00000000 OBJT GLOB P 0 .got _GLOBAL_OFFSET_TABLE_
[59] 0x080506b8 0x00000000 FUNC GLOB D 0 UNDEF printstack
[60] 0x080506a8 0x00000000 FUNC GLOB D 0 UNDEF _exit
[61] 0x08050864 0x00000004 OBJT GLOB D 0 .rodata _lib_version
[62] 0x08050698 0x00000000 FUNC GLOB D 0 UNDEF atexit
[63] 0x08050678 0x00000000 FUNC GLOB D 0 UNDEF __fpstart
[64] 0x0805076c 0x0000007b FUNC GLOB D 0 .text __fsr
[65] 0x08050688 0x00000000 FUNC GLOB D 0 UNDEF exit
[66] 0x080506c8 0x00000000 FUNC WEAK D 0 UNDEF _get_exit_frame_monitor
[67] 0x080609c0 0x00000000 OBJT GLOB D 0 .bss _end
[68] 0x080506e0 0x0000008b FUNC GLOB D 0 .text _start
[69] 0x08050848 0x0000001b FUNC GLOB D 0 .fini _fini
[70] 0x08060978 0x00000018 OBJT GLOB D 0 .data __environ_lock
[71] 0x0806099c 0x00000004 OBJT GLOB D 0 .data __longdouble_used
[72] 0x00000000 0x00000000 NOTY WEAK D 0 UNDEF __1cG__CrunMdo_exit_code6F_v_
Running the version without a .SUNW_ldynsym section:% strip test_ldynsym test_noldynsym % file test_ldynsym test_noldynsym test_ldynsym: ELF 32-bit LSB executable 80386 Version 1, dynamically linked, stripped test_noldynsym: ELF 32-bit LSB executable 80386 Version 1, dynamically linked, stripped
Our program used the printstack(3C) function to display its own stack. Afterwards, we use the pstack command to view the same data from the core file. In both cases, the top line represents the call to the local function static_func(), a fact that we know from examining the source code, since the number and/or '????????' used to represent it are less than obvious to an external observer.% ./test_noldynsym /home/ali/test/test_noldynsym:0x6ca /home/ali/test/test_noldynsym:main+0xb /home/ali/test/test_noldynsym:_start+0x7a Segmentation Fault (core dumped) % pstack core core 'core' of 5041: ./test_noldynsym 080506d2 ???????? (804692c, 80467a4, 805062a, 1, 80467b0, 80467b8) 080506eb main (1, 80467b0, 80467b8) + b 0805062a _start (1, 8046994, 0, 80469a5, 80469bf, 8046a03) + 7a
Running the version with a .SUNW_ldynsym section, the system is able to put a name to the local function:
% ./test_ldynsym /home/ali/test/test_ldynsym:static_func+0xa /home/ali/test/test_ldynsym:main+0xb /home/ali/test/test_ldynsym:_start+0x7a Segmentation Fault (core dumped) % pstack core core 'core' of 5044: ./test_ldynsym 08050802 static_func (8046930, 80467a8, 805075a, 1, 80467b4, 80467bc) + 12 0805081b main (1, 80467b4, 80467bc) + b 0805075a _start (1, 8046998, 0, 80469a7, 80469c1, 8046a05) + 7a
Conclusions
Sometimes it is the little things that make a difference. I expect that the local dynamic symbol table will provide valuable information in difficult debugging situations where one is examining large stripped programs running in a production environment. The rest of the time, the additional data is small, and will have little or no impact on performance..SUNW_ldynsym sections have been part of the Solaris development (Nevada) builds since last fall, and are also available in OpenSolaris.
Technorati Tag:
OpenSolaris
Technorati Tag:
Solaris
Posted at 02:05PM Feb 07, 2007 by ali in Sun | Comments[7]
Inside ELF Symbol Tables
ELF files are full of things we need to keep track of for later access: Names, addresses, sizes, and intended purpose. Without this information, an ELF file would not be very useful. We would have no way to make sense of the impenetrable mass of octal or hexidecimal numbers.
Consider: When you write a program in any language above direct machine code, you give symbolic names to functions and data. The compiler turns these things into code. At the machine level, they are known only by their address (offset within the file) and their size. There are no names in this machine code. How then, can a linker combine multiple object files, or a symbolic debugger know what name to use for a given address? How do we make sense of these files?
Symbols are the way we manage this information. Compilers generate symbol information along with code. Linkers manipulate symbols, reading them in, matching them up, and writing them out. Almost everything a linker does is driven by symbols. Finally, debuggers use them to figure out what they are looking at and to provide you with a human readable view of that information.
It is therefore a rare ELF file that doesn't have a symbol table. However, most programmers have only an abstract knowledge that symbol tables exist, and that they loosely correspond to their functions and data, and some "other stuff". Protected by the abstractions of compiler, linker, and debugger, we don't usually need to know too much about the details of how a symbol table is organized. I've recently completed a project that required me to learn about symbol tables in great detail. Today, I'm going to write about the symbol tables used by the linker.
.symtab and .dynsym
Sharable objects and dynamic executables usually have 2 distinct symbol tables, one named ".symtab", and the other ".dynsym". (To make this easier to read, I am going to refer to these without the quotes or leading dot from here on.)The dynsym is a smaller version of the symtab that only contains global symbols. The information found in the dynsym is therefore also found in the symtab, while the reverse is not necessarily true. You are almost certainly wondering why we complicate the world with two symbol tables. Won't one table do? Yes, it would, but at the cost of using more memory than necessary in the running process.
To understand how this works, we need to understand the difference between allocable and a non-allocable ELF sections. ELF files contain some sections (e.g. code and data) needed at runtime by the process that uses them. These sections are marked as being allocable. There are many other sections that are needed by linkers, debuggers, and other such tools, but which are not needed by the running program. These are said to be non-allocable. When a linker builds an ELF file, it gathers all of the allocable sections together in one part of the file, and all of the non-allocable sections are placed elsewhere. When the operating system loads the resulting file, only the allocable part is mapped into memory. The non-allocable part remains in the file, but is not visible in memory. strip(1) can be used to remove certain non-allocable sections from a file. This reduces file size by throwing away information. The program is still runnable, but debuggers may be hampered in their ability to tell you what the program is doing.
The full symbol table contains a large amount of data needed to link or debug our files, but not needed at runtime. In fact, in the days before sharable libraries and dynamic linking, none of it was needed at runtime. There was a single, non-allocable symbol table (reasonably named "symtab"). When dynamic linking was added to the system, the original designers faced a choice: Make the symtab allocable, or provide a second smaller allocable copy. The symbols needed at runtime are a small subset of the total, so a second symbol table saves virtual memory in the running process. This is an important consideration. Hence, a second symbol table was invented for dynamic linking, and consequently named "dynsym".
And so, we have two symbol tables. The symtab contains everything, but it is non-allocable, can be stripped, and has no runtime cost. The dynsym is allocable, and contains the symbols needed to support runtime operation. This division has served us well over the years.
Types Of Symbols
Given how long symbols have been around, there are surprisingly few types:
- STT_NOTYPE
- Used when we don't know what a symbol is, or to indicate the absence of a symbol.
- STT_OBJECT / STT_COMMON
- These are both used to represent data. (The word OBJECT in this context should not interpreted as having anything to do with object orientation. STT_DATA might have been a better name.)
STT_OBJECT is used for normal variable definitions, while STT_COMMON is used for tentative definitions. See my earlier blog entry about tentative symbols for more information on the differences between them.
- STT_FUNC
- A function, or other executable code.
- STT_SECTION
- When I first started learning about ELF, and someone would say something about "section symbols", I thought they meant a symbol from some given section. That's not it though: A section symbol is a symbol that is used to refer to the section itself. They are used mainly when performing relocations, which are often specified in the form of "modify the value at offset XXX relative to the start of section YYY".
- STT_FILE
- The name of a file, either of an input file used to construct the ELF file, or of the ELF file itself.
- STT_TLS
- A third type of data symbol, used for thread local data. A thread local variable is a variable that is unique to each thread. For instance, if I declare the variable "foo" to be thread local, then every thread has a separate foo variable of their own, and they do not see or share values from the other threads. Thread local variables are created for each thread when the thread is created. As such, their number (one per thread) and addresses (depends on when the thread is created, and how many threads there are) are unknown until runtime. An ELF file cannot contain an address for them. Instead, a STT_TLS symbol is used. The value of a STT_TLS symbol is an offset, which is used to calculate a TLS offset relative to the thread pointer. You can read more about TLS in the Linker And Libraries Guide.
- STT_REGISTER
- The Sparc architecture has a concept known as a "register symbol". These symbols are used to validate symbol/register usage, and can also be used to initialize global registers. Other architectures don't use these.
In addition to symbol type, each symbols has other attributes:
- Name (Optional: Not all symbols need a name, though most do)
- Value
- Size
- Binding and Visibility
- ELF Section it references
Symbols Table Layout And Conventions
The symbols in a symbol table are written in the following order:- Index 0 in any symbol table is used to represent undefined symbols. As such, the first entry in a symbol table (index 0) is always completely zeroed (type STT_NOTYPE), and is not used.
- If the file contains any local symbols, the second entry (index 1) the symbol table will be a STT_FILE symbol giving the name of the file.
- Section symbols.
- Register symbols.
- Global symbols that have been reduced to local scope via a mapfile.
- For each input file that supplies local symbols, a STT_FILE symbol giving the name of the input file is put in the symbol table, followed by the symbols in question.
- The global symbols immediately follow the local symbols in the symbol table. Local and global symbols are always kept separate in this manner, and cannot be mixed together.
Next Time: Augmenting The Dynsym
One of the big advantages of Solaris relative to other operating systems is the extensive support for observability: The ability to easily look inside a running program and see what it is doing, in detail. To do that well requires symbols. The symbols in the dynsym may not be enough to do a really good job. For example, to produce a stack trace, we need to take each function address and match it up to its name. If we are looking at a stripped file, or referencing the file from within the process using it via dladdr(3C), we won't have any way to find names for the non-global functions, and will have to resort to displaying hex addresses. This is better than nothing, but not by much. The standard files in a Solaris distribution are not stripped for exactly this reason. However, many files found in production are stripped, and in-process inspection is still limited to the dynsym.Machines are much larger than they used to be. The memory saved by the symtab/dynsym division is still a good thing, but there are times when we wish that the dynsym contained a bit more data. This is harder than it sounds. The layout of dynsym interacts with the rest of an ELF file in ways that are set in stone by years of existing practice. Backward compatibility is a critical feature of Solaris. We try extremely hard to keep those old programs running. And yet, the needs of observability, spearheaded by important new features like DTrace, put pressure on us in the other direction.
This discussion is prelude to work I recently did to augment the dynsym to contain local symbols, while preserving full backward compatibility with older versions Solaris. I plan to cover that in a future blog entry. ELF is old, and much of how it works cannot be changed. Its original designers (our "Founding Fathers", as Rod calls them) anticipated that this would be the case, based no doubt on hard experience with earlier systems. The ELF design is therefore uniquely flexible, which explains why it has survived as long as it has. There is always a way to add something new. Sometimes, it takes several tries to find the best way.
Technorati Tag:
OpenSolaris
Technorati Tag:
Solaris
Posted at 02:22PM Sep 23, 2006 by ali in Sun |
What Are "Tentative" Symbols?
In the Linker and Libraries Guide, you will encounter discussion of tentative symbols. Based on the name, we might expect that such a symbol is missing something, but what? And why does the linker have to treat them as a special case?
A tentative symbol is a symbol used to track a global variable when we don't know its size or initial value. In other words, a symbol for which we have not yet assigned a storage address. They are also known as "common block" symbols, because they have their origins in the implementation of Fortran COMMON blocks. They are historical baggage — something that needs to work for compatibility with the past, but also something to avoid in new code.
Consider the following two C declarations, made at outer file scope:
int foo;
int foo = 0;
Superficially, these both appear to declare a global variable named
foo with an initial value of 0. However, the first definition is
tentative — it will have a value of 0 only if some other file
doesn't explicitly give it a different value. The outcome depends on
what else we link this file against.
To get a better handle on this, let's create two separate C files (t1.c, and t2.c) and experiment:
t1.c
#include <stdio.h> #ifdef TENTATIVE_FOO int foo; #else int foo = 0; #endif int main(int argc, char *argv[]) { printf("FOO: %d\n", foo); return (0); }
t2.c
int foo = 12;
First, we compile and link t1.c by itself, using both forms of declaration for variable foo:
% cc -DTENTATIVE_FOO t1.c; ./a.out
FOO: 0
% cc t1.c; ./a.out
FOO: 0
As expected, they give identical results. Now, lets add t2.c to the mix and see what happens:
% cc -DTENTATIVE_FOO t1.c t2.c; ./a.out
FOO: 12
% cc t1.c t2.c; ./a.out
ld: fatal: symbol `foo' is multiply-defined:
(file t1.o type=OBJT; file t2.o type=OBJT);
ld: fatal: File processing errors. No output written to a.out
./a.out: No such file or directory
As you can see, the two different ways of declaring foo are not
100% equivalent. The tentative declaration of foo in t1.c
took on the value provided by the declaration in t2.c. In contrast,
the linker was unwilling to merge the two non-tentative definitions of
foo that had different values, and instead issued a fatal link error.
Normal C rules say that a variable at file scope without an explicit value is assigned an initial value of 0. However, the existence of other global variables with the same name can change this. The C compiler is only able to see the code in the single file it is compiling, and cannot know how to handle this case. So, it marks it as tentative by giving the symbol a type of STT_COMMON, and leaves it for the linker to figure out. The linker is in a position to match up all of these symbols and merge them into a single instance. The linker has no insight into programmer intent though, and it cannot protect you from doing this by accident. The result usually works, but is fragile.
The other declaration form (with a value) causes a non-tentative symbol to be created (STT_OBJECT). In this case, the linker ensures that all the declarations agree. This is the right behavior if you care about robust and scalable code.
It is worth noting that you will never see a tentative symbol with local scope. It can only happen to global symbols, because global symbols in different files are the only way you can get this form of aliasing to occur.
History
Tentative symbols are bad software engineering. A declaration in one file should not be able to alter one in another file. The need for them dates from the early days of the Fortran language. In Fortran, you can declare a common block in more than one file, with each file independently specifying the number, types, and sizes of the variables. The linker then takes all of these blocks, allocates enough space to satisfy the largest one, and makes all them point at that space. This is a very crude form of a union (variant), and is therefore very useful (and dangerous) Fortran technique.Sadly, it didn't stop there. We still sometimes find this practice in C code. Two files will both declare:
int foo;
and then expect that they are both be referring to a single global
variable, with an initial value of 0. This is not necessary.
The proper solution has existed for decades. The safe way to do
the above is to have exactly one
declaration for the global variable in a single file. The other files
that need to access to it use the "extern" keyword
to let the compiler know what is going on. The statement
extern int foo;
is a reference, not a declaration, and it has a single unambiguous
interpretation.
Moral: Don't Do That!
Don't use common block binding in your code. It was a bad idea 40 years ago, and it hasn't improved with age. The necessity of backward compatibility is such that compilers and linkers must support common block binding. We are stuck with it, but we don't have to use it.You should always try to minimize or eliminate global variables. However, when you do use them:
- There should be exactly one declaration for each global variable, contained in a single file. When declaring that single instance, always give it an explicit value, even if that value is 0. The C language says that the value is 0 if you don't, but doing it explicitly ensures that you can't accidentally fall into the "tentative trap" if some other module should come along later and define it. Note that this only applies to global variables. Static variables declared at file scope can be safely assumed to have an initial value of 0.
- The module that declares the variable should supply a header file containing an extern statement for the variable. Furthermore, the module must #include its own header file. The compiler allows you to have a declaration and an extern statement for a variable in the same compilation scope, and it will check to make sure they agree. This ensures that your module can't export a bad extern definition to other code.
- Other modules that access the global variable must always #include the header file from the defining module, and must never supply their own explicit extern statement for the variable. This protects them from being stuck with an obsolete and incorrect definition if the variable should change later.
Technorati Tag:
OpenSolaris
Technorati Tag:
Solaris
Posted at 10:16PM Sep 22, 2006 by ali in Sun |
Settling An Old Score (Linker Division)
For years, I worked on an interactive language used by scientists to do data analysis and visualization. That program makes heavy use of sharable libraries. Solaris was my primary development platform, due to its superior facilities for observing and debugging software, so my usual strategy was to write and debug my code under Solaris, and then move the results to the many other Unix (and VMS, Windows, and Macintosh) platforms that we supported.
I became very familiar with one quirk of the Solaris linker that bit me many times over the years. The issue has to do with how ld(1) handles the situation where it needs to replace an existing output file. Historically, this was handled by truncating the existing file and rewriting it in place. This preserves the existing inode, and any hard links that may happen to be pointing to it. However, it has a very bad effect on any running process that happens to be using that file. For example, if the output file is a sharable library, creating a new version while a program that uses it is running will inevitably cause that program to die in an unplanned and unexpected way.
If you're not a developer of software that runs for unbounded amounts of time, then you probably have not seen this behavior. If you develop code that does however, then you've almost certainly hit it at some point. In my case, I'd hit it about once a year, usually while multitasking, flipping back and forth between several xterms. Usually, it was obvious what had happened, and I'd quickly recover. Sometimes though, if I was debugging the program for some other reason, the unexpected SIGSEGV or SIGBUS would send me off into the weeds, debugging a mysterious problem in a part of the program unrelated to where I expected the problem to lie. After a minute or so, I'd