Tuesday April 26, 2005
Tuesday April 26, 2005 My Relocations Don't Fit - Position Independence
A couple of folks have come across the following relocation error when running their applications on AMD64:
$ prog
ld.so.1: prog: fatal: relocation error: R_AMD64_32: file \
libfoo.so.1: symbol (unknown): value 0xfffffd7fff0cd457 does not fit
The culprit,
Shared objects are typically built using position independent code, using
compiler options such as
If a shared object is built from position-dependent code, the text segment can require modification at runtime. This modification allows relocatable references to be assigned to the location that the object has been loaded. The relocation of the text segment requires the segment to be remapped as writable. This modification requires a swap space reservation, and results in a private copy of the text segment for the process. The text segment is no longer sharable between multiple processes. Position-dependent code typically requires more runtime relocations than the corresponding position-independent code. Overall, the overhead of processing text relocations can cause serious performance degradation.
When a shared object is built from position-independent code, relocatable references are generated as indirections through data in the shared object's data segment. The code within the text segment requires no modification. All relocation updates are applied to corresponding entries within the data segment.
The runtime linker attempts to handle text relocations should these relocations exist. However, some relocations can not be satisfied at runtime.
The AMD64 position-dependent code sequence typically generates code which can only be loaded into the lower 32-bits of memory. The upper 32-bits of any address must all be zeros. Since shared objects are typically loaded at the top of memory, the upper 32-bits of an address are required. Position-dependent code within an AMD64 shared object is therefore insufficient to cope with relocation requirements. Use of such code within a shared object can result in runtime relocation errors cited above.
This situation differs from the default ABS64 mode that is used for 64-bit SPARCV9 code. This position-dependent code is typically compatible with the full 64-bit address range. Thus, position-dependent code sequences can exist within SPARCV9 shared objects. Use of either the ABS32 mode, or ABS44 mode for 64-bit SPARCV9 code, can still result in relocations that can not be resolved at runtime. However, each of these modes require the runtime linker to relocate the text segment.
Build all your shared objects using position independent code.
A Update - Wednesday March 21, 2007
If you believe you have compiled all the components of your shared object
using
$ elfdump -d library | fgrep TEXTREL
If this flag is found, then the link-editor thinks this file contains non-pic code. One explanation might be that you have include an assembler file in the shared library. Any assembler must be written using position-independent instructions. Another explanation is that you might have included objects from an archive library as part of the link-edit. Typically, archives are built with non-pic objects.
You can track down the culprit from the link-editors debugging capabilities. Build your shared object.
$ LD_OPTIONS=-Dreloc,detail cc -o library .... 2> dbg
The diagnostic output in dbg can be inspected to locate the non-pic relocation, and from this you can trace back to the input file that supplied the relocation as part of building your library.
(2005-04-26 09:40:09.0) Permalink Comments [1]
Wednesday February 02, 2005 Loading Relocatable Objects at Runtime - Very Expensive
The runtime linker, ld.so.1(1), is capable of loading relocatable objects. This capability arrived somewhat by accident, and as a side effect of some other projects related to the link-editor, ld(1). But, it was thought that this capability could prove useful within a development environment. A user can preload relocatable objects, and this technique might provide a quick prototyping turnaround instead of having to first combine the relocatable objects into some huge shared object.
However, this capability doesn't come cheap, and it was never thought to be useful for production software. Recently, a customer problem uncovered a regression in relocatable object processing. But, it was a surprise to see that this technique, in this case the dlopen(3c) of a relocatable object, was being used in production code.
The runtime linker only knows how to deal with dynamic objects, that is, dynamic executables and shared objects. When a relocatable object is encountered, the runtime linker first loads the link-editor. The link-editor then converts the relocatable object into a shared object memory image within the process. The runtime linker then processes this memory image as it would any other shared object.
The link-editor is really a family of libraries, and you can see these if you inspect the link-maps of a process that has triggered the loading of a relocatable object.
% mdb main
> ::bp exit
> :r
> ::Rt_maps
Link-map lists (dynlm_list): 0x8045f38
----------------------------------------------
Lm_list: 0xfeffa204 (LM_ID_BASE)
----------------------------------------------
Link_map* ADDR() NAME()
----------------------------------------------
0xfeffc8f4 0x08050000 main
0xfeffccc4 0xfeef0000 /lib/libc.so.1
----------------------------------------------
Lm_list: 0xfeffa1c8 (LM_ID_LDSO)
----------------------------------------------
0xfeffc590 0xfefc8000 /lib/ld.so.1
0xfeee06e4 0xfee60000 /lib/libld.so.2
0xfeee0af0 0xfed90000 /lib/libc.so.1
0xfed80068 0xfed50000 /lib/libelf.so.1
As you can see, the runtime linkers link-map list (LM_ID_LDSO) contains the runtime linker itself, and all the libraries that effectively make up the link-editor.
Besides loading all these support libraries, the shared object image adds additional overhead. Relocatable objects are typically non-pic, and thus require considerable more relocation than their pic (position independent) counterparts. This relocation overhead is passed on to the shared object memory image.
And, because the image is created in memory, each process that uses this technique creates it's own private image. There is no sharing of text pages from the shared object image, as is common for shared objects. This customers application forked off a number of processes that all dlopen()'ed the same relocatable object.
So, this technique may have some use within a development environment, but there are expenses:
the link-editor must be loaded into the process,
the relocatable object is then converted into a shared object memory image,
there are typically many more relocations to process in the memory image than there would be from a comparable pic shared object, and
the resulting shared object image isn't sharable between processes.
Personally, I'd recommend not using this technique in production software.
(2005-02-02 11:29:26.0) Permalink Comments [8]
Sunday January 02, 2005 Interface Creation - using the compilers
In a previous
posting,
I covered how the interface of
a dynamic object could be established by defining the interface
symbols within a
This
First, the link-editor only processes symbols in their mangled form. For example, even a simple interface such as:
void foo(int bar)
has a C++ symbolic representation of:
% elfdump -s foo.o
....
[2] .... FUNC GLOB D 0 .text __1cDfoo6Fi_v_
As no tool exists that can determine a symbols mangled name other
than the compilers themselves, trying to establish definitions of
this sort within a
The second issue is that some interfaces created by languages such
as C++, provide implementation details of the language itself. These
implementation interfaces often must remain global within a group of
similar dynamic objects, as one interface must interpose
on all the others for the correct execution of the application.
As users generally are not aware of what implementation symbols are
created, they can blindly demote these symbols to local when
applying any interface definitions with a
Thankfully, some recent ELF extension work carried out
with various Unix vendors has established a set
of visibility attributes that can be applied to ELF
symbol table entries. These attributes are maintained within
the symbol entries st_other field, and are fully
documented under "ELF Symbol Visibility" in the
"Object File Format" chapter of the
Linker and Libraries Guide.
The compilers, starting with Sun ONE Studio 8, are now capable
of describing symbol visibility. These definitions are then
encoded in the symbol table, and used by ld(1) in a similar manner
as reading definitions from a
As with any interface definition technique, this compilation
method can greatly reduce the number of symbols that would
normally be employed in runtime relocations. Given the number
and size of C++ symbols, this technique can produce runtime
relocation reductions that far exceed those that would be
found in similar C objects. In addition, as the compiler knows what
implementation symbols must remain global
within the final object, these symbols are given the
appropriate visibility attribute to insure their correct use.
Presently there are two recommendations for establishing an
objects interface. The first is to define all interface
symbols using the __global directive, and reduce all
other symbols to local using the -xldscope=hidden
compiler option. This model provides the most flexibility.
All global symbols are interposable, and allow for any copy
relocations1 to be processed correctly.
The second model is to define all interface symbols using the
__symbolic directive, and again reduce all other symbols
to local using the -xldscope=hidden
compiler option. Symbolic symbols (also termed protected),
are globally visible, but
have been internally bound to.
This means that these symbols do not
require symbolic runtime relocation,
but can not be interposed upon,
or have copy relocations against them.
In practice, I'd expect to see significant savings in the
runtime relocation of any modules that used either model.
However, the savings between using the __global
or the __symbolic model may be harder to measure.
In a nutshell, if you do not want a user to interpose upon
your interfaces, and don't export data items, you can probably
go with __symbolic. If in doubt, stick with the
more flexible use of __global.
The following examples uses C++ code that was furnished to me as
being representative of what users may develop.
All interface symbols have employed the __global
attribute. Compiling this module with -xldscope=hidden
reveals the following symbol table entries.
Notice that the first 4 local (LOCL) symbols would normally have
been defined as global without using the symbol definitions and
compiler option. This is a simple example, as implementations get
more complex, expect to see a larger fraction of symbols
demoted to locals.
For a definition of other related compiler options, at least how
they relate to C++, see
Linker Scoping2.
1
Copy relocations are a technique employed to allow references
from non-pic code to external data items, while maintaining the
read-only permission of a typical text segment. This relocations
use, and overhead, can be avoided by designing shared objects that
do not export data interfaces.
2
It's rumored the compiler folks are also working on __declspec
and GCC __attribute__ clause implementations. These should
aid porting code and interface definitions from other platforms.
Giri Mandalika has posted a very detailed
article
on this topic, including the __declspec implementation.
(2005-01-02 21:21:24.0)
Permalink
Comments [3]
Static Linking - where did it go?
With Solaris 10 you can no longer build a static executable.
It's not that ld(1) doesn't allow static linking, or using
archives, it's just that
We've been warning users against static linking for some time
now, and linking against
If an application is built against
In addition, implementation details of
One common failing we have discovered is that many folks built against
Because of this potential for distroying any application
binary interface guarantee, the 64-bit version of Solaris
never delivered any 64-bit system archives libraries. We figured
we nip that problem in the bud right away.
But why did we wait until Solaris 10 to stop delivering
32-bit system archives? It turns out the merge of
In earlier releases of Solaris, threaded applications needed
to build against
Now, if you built an application against
In Solaris 10,
The merger of
Therefore, to put to rest the consistent failure of
partially static applications from release to release,
we've now made it impossible to make such applications.
Some folks thought of static applications as being
a means of insulating themselves from system changes.
But as I explained above, they were not insulating
themselves from user/kernel interface changes.
Note that there is some flexibility even in a dynamic
linking environment. Although applications are built
to encode the interpretor /usr/lib/ld.so.1,
applications can be built to specify an alternative
interpretor:
Or, you can invoke an alternative interpretor directly:
But, even if you create an environment with an alternate
runtime linker and dynamic
See also this explanation.
(2004-12-06 15:49:03.0)
Permalink
Comments [2]
Filters are a class of shared objects that are available with Solaris.
These objects allow for the redirection of symbol bindings at runtime.
They have been employed to provide standards interfaces and to allow the
selection of optimized function implementations at runtime.
For those of you unfamiliar with filters, here's an introduction,
plus a glimpse of some new techniques available with Solaris 10.
Filters can exist in two forms, standard filters and auxiliary filters.
At runtime, these objects redirect bindings from themselves to
alternative shared objects, that are known as filtees. Shared objects
are identified as filters by using the link-editors -F and
-f options.
Standard filters provide no implementation for the interfaces they
define. Essentially they provide a symbol table. When used to build a
dynamic object they satisfy the symbol resolution requirements of the
link-editing process. However at runtime, standard filters redirect a
symbol binding from themselves to a filtee.
Standard filters have been used to implement
Auxiliary filters work in much the same way, however if a filtee
implementation can't be found, the symbol binding is satisfied by
the filter itself. Basically, auxiliary filters allow an interface to find a
better alternative. If an alternative is found it will be used, and
if not, the generic implementation provided by the filter provides a
fallback.
Auxiliary filters have been used to provide platform specific implementations.
Typically, these are implementations that provide a performance
improvement on various platforms.
You can observe that a symbol binding has been satisfied by a filtee by using
the runtime linkers
tracing.
The following output shows the symbol memset being searched
for in the application
Until now, a filter has been an identification applied to a whole shared
object. With Solaris 10, per-symbol filtering has been introduced.
This allows individual symbol table entries to identify themselves
as standard, or auxiliary filters. These filters provide greater flexibility,
together with less runtime overhead than whole object filters.
Individual symbols
are identified as filters using mapfile entries at the time
an object is built.
For example,
The above definitions provide a couple of advantages. First,
you no longer need to link against
Per-symbol filters have also proved useful in consolidating existing
interfaces. For example, for historic standards compliance,
The definition of a filtee has often employed runtime tokens
such as $PLATFORM.
These tokens are expanded to provide pathnames specific
to the environment in which the filter is found. A new
capability provided with Solaris 10 is the $HWCAP token.
This token is used to identify a directory in which one or more
hardware capability libraries can be found.
Shared objects can be built to record their hardware capabilities requirements.
Filtees can be constructed that use various hardware capabilities as
a means of optimizing their performance. These filtees can be
collected in a single directory. The $HWCAP token can then
be employed by the filter to provide the selection of the optimal
filtee at runtime:
The hardware capabilities of a platform are conveyed to the runtime linker
from the kernel. The runtime linker then matches these capabilities
against the requirements of each filtee. The filtees are then
sorted in descending order of their hardware capability values.
These sorted filtees are used to resolve symbols that are defined within
the filter.
This model of file identification provides greater flexibility than
the existing use of $PLATFORM, and is well suited to filtering
use.
As usual, examples and all the gory details of filters and the new
techniques outlined above can be found in the Solaris 10 Linker and Libraries guide.
Discussions on how to alter any hardware capabilities established by the compilers can be found
hear,
and here.
(2004-10-22 21:41:55.0)
Permalink
Comments [3]
Since Solaris 2.0, the link-editors have provided a mechanism
for tracing what they're doing. As this mechanism has been
around for so long, plus I've used some small examples in
previous postings, I figured most folks knew of its existence.
I was reminded the other day that this isn't the case. For
those of you unfamiliar with this tracing, here's an introduction,
plus a glimpse of a new analysis tool available with Solaris 10.
You can set the environment variable LD_DEBUG to
one or more pre-defined tokens. This setting causes the
runtime linker, ld.so.1(1), to display information regarding
the processing of any application that inherits this
environment variable.
The special token help provides
a list of token capabilities without executing any application.
One of the most common tracing selections reveals the binding
of a symbol reference to a symbol definition.
Those bindings that occur before transferring to
Another common tracing selection reveals what files are loaded.
This reveals initial dependencies that are loaded prior to transferring
control to
Note, the environment variable LD_DEBUG_OUTPUT can be
used to specify a file name to which diagnostics are written (the file name
gets appended with the pid). This is helpful to prevent the
tracing information from interfering with normal program output, or
for collecting large amounts of data for later processing.
In a previous
posting
I described how you could discover unused, or unreferenced dependencies.
You can also discover these dependencies at runtime.
Unused objects are determined prior to calling
Lastly, there are our old friends .init sections. Executing these
sections in an attempt to fulfill the expectations of modern languages
(I'm being polite here), and expected programming techniques, has been
shall we say, challenging. .init tracing is produced no matter
what debugging token you chose.
Note that in this example,
the topologically sorted order established to fire .init's
has been interrupted. We dynamically fire the .init of
The debugging library that provides these tracing diagnostics is also
available to the link-editor, ld(1). This debugging library provides a common
diagnostic format for tracing both linkers. Use the
link-editors -D option to obtain tracing info. As most
compilers have already laid claim to this option, the LD_OPTIONS
environment variable provides a convenient setting.
For example, to see all the gory details of the symbol resolution
undertaken to build an application, try:
and stand back ... the output can be substantial.
Although tracing a process at runtime can provide useful information
to help diagnose process bindings, the output can be substantial.
Plus, it only tells you what bindings have occurred. This information
lacks the
full symbolic interface data of each object involved, which in
turn can hide what you think should be occurring. In Solaris 10,
we added a new utility, lari(1), which provides the Link Analysis of
Runtime Interfaces.
This perl(1) script analyzes a debugging
trace, together with the symbol tables of each object involved in
a process. lari(1) tries to discover any interesting
symbol relationships. Interesting, typically means that a symbol
name exists in more than one dynamic object, and interposition is
at play. Interposition can be your friend, or your enemy - lari(1)
doesn't know which. But historically, a number of application failures
or irregularities have boiled down to some unexpected interposition
which at the time was hard to track down.
For example, a typical interposition might show up as:
Here, two versions of function
lari(1) also uncovers direct bindings or symbols that are defined with
protected visibility. These can result in multiple instances of a
symbol being bound to from different callers:
Again, perhaps this is what the user wants to achieve, perhaps not ...
but it is interesting.
There are many more permutations of symbol diagnostic that can be
produced by lari(1), including the identification of explicit
interposition (such as preloading, or objects built with -z interpose),
copy relocations, and dlsym(3c) requests. Plus, as lari(1) is
effectively discovering the interfaces used by each object within
a process, it can create
versioning
mapfiles that can be
used as templates to rebuild each object.
(2004-09-29 12:30:10.0)
Permalink
Relocations - careful with that debugging flag
I received an application from a customer the other day. It's quite
a big sucker, consisting of the application and over 70 shared
objects (that's besides the system objects that also get used).
The customer has complained that it takes a long time to load this
application. In particular, it takes a long time to verify their
objects using
Using
The set of shared objects supplied by the customer do not specify any
of their dependencies. In fact, the application seems responsible
for establishing all dependencies, not only those that the application
references, but also those needed to satisfy all dependencies. If
each shared object defined their dependencies, then
A quick poke around with
Looking a little deeper I found that around 2.3 million relocations
are RELATIVE relocations. These are relocations that simply
need the base offset of the object to be added to the relocation offset.
This is a simple operation, involving no symbol lookup, and is only
accounting for a few percent of the cost.
The rest of the startup cost stems from the symbolic relocations, of
which there are some 740,000 that needed processing with
I don't have the source for this set of objects to experiment with
not using -g. But I'm left concluding that the bulk of
the startup cost of this process is due to these $X....
symbols.
If you don't want fix-and-continue, be careful how you use the
compiler flags. This overhead in relocation processing probably
isn't what you want in production software.
Note, you can also build objects with the -zcombreloc
flag of
(2004-08-30 15:55:07.0)
Permalink
Comments [2]
For some time now, we've been versioning core system libraries.
You can display version definitions, and version requirements with
So, what do these versions provide? Shared object versioning has often
been established with various conventions of renaming the file itself
with different major or minor (or micro) version numbers. However, as
applications have become more complex, specifically because they are
constructed from objects that are asynchronously delivered from external
partners, this file naming convention can be problematic.
In developing the core Solaris libraries, we've been rather obsessed
with compatibility, and rather than expect customers to rebuild against
different shared object file names (i.e.,
Now you could maintain compatibility by retaining all existing public
interfaces, and only adding new interfaces, without the versioning
scheme. However, the version scheme has a couple of advantages:
consumers of the interface sets record their requirements on
the version name they reference.
establishing interface sets removes unnecessary interfaces from
the name-space.
the version sets provide a convenient means of policing
interface evolution.
When a consumer references a versioned shared object, the version
name representing the interfaces the consumer references are
recorded. For example, an application that references the
This version name requirement is verified at runtime. Therefore,
should this application be executed in an environment consisting
of an older
This verification might seem simplistic, and won't the application
be terminated anyway if a required interface can't be located?
Well yes, but function binding normally occurs
at the time the function is first called. And this call can
be some time after an application is started (think scientific
applications that can run for days or weeks). It is far better to
be informed that an interface can't be located when a library is
first loaded, that to be killed some time later when a specific
interface can't be found.
Defining a version typically results in the demotion of many other
global symbols to local scope. This localization can prevent unintended
symbol collisions.
For example, most shared objects are built
from many relocatable objects, each referencing one another. The interface that the developer wishes to export from the shared object is normally a
subset of the number of global symbols that would normally remain
visible.
Version definitions can be defined using a
The demotion of unnecessary global symbols to locals greatly reduces the
relocation requirements of the object at runtime, and can significantly
reduce the runtime startup cost of loading the object.
Of course, interface compatibility requires a disciplined approach to
maintaining interfaces. In the previous example, should the signature
of foo1() be changed, or foo2() be deleted, then the
use of a version name is meaningless. Any application that had
built against the original interfaces, will fail at runtime when the
new library is delivered, even though the version name verification
will have been satisfied.
With the core Solaris libraries we maintain compatibility as we
evolve through new releases by maintaining existing public interfaces
and only adding new version sets. Auditing of the version sets help
catch any mistaken interface deletions or additions. Yeah, we
fall foul of cut-and-paste errors too :-)
For more information on versioning refer to the
Versioning Quick Reference. Or for a detailed description refer to
Application Binary Interfaces and Versioning.
(2004-08-22 21:47:13.0)
Permalink
Lazy Loading - there's even a fall back
In my
previous posting, I described the use of lazy loading. Of course, when
we initially played with an implementation of this technology, a couple of
applications immediately fell over. It turns out that a fall back was
necessary.
Let's say an application developer creates an application with two
dependencies. The developer wishes to employ lazy loading for both
dependencies.
The application developer has no control over the dependency
The only reason this library has been successfully employed by any
application is because the application, or some other shared object within
the process, has made the
dependency
Now, suppose the application
When control is passed to bar(), the reference it makes to
its implicit dependency foo() is not going to be found,
because the shared object
Of course, there can be a down-side to this fall back. If
To prevent lazy loading from being compromised, always record those
dependencies you need (and nothing else).
(2004-08-01 20:05:07.0)
Permalink
Dependencies - perhaps they can be lazily loaded
In a
previous posting, I stated that you should only record those dependencies
you need, and nothing else. There's another step you can take to
reduce start-up processing overhead.
Dynamic objects need to resolve symbolic references from each other.
Function calls are typically implemented through an indirection
that allows the function binding to be deferred until the function
call is first made. See
When Relocations Are Performed.
Because of this deferral, it is also possible to cause the defining
dependency to be loaded when the function call is first made. This
model is referred to as
Lazy
Loading.
To establish lazy loading, you must pass the -z lazyload
option to
The lazy loading attribute of these dependencies can be displayed with
By default,
Once function relocations are processed, both dependencies are loaded
to resolve the function reference.
ldd(1) becomes a convenient tool for discovering whether lazy
loading might be applicable. Suppose we rebuilt
This has revealed that loading
Lazy loading can be observed at runtime using the runtime linkers
debugging capabilities (LD_DEBUG=files). For example, if
wally() was called with a zero argument, we'd see
Note, not only on does lazy loading have the potential of reducing the cost of
start-up processing, but if lazy loading references are never called, the dependencies
will never be loaded as part of the process.
(2004-07-27 17:25:59.0)
Permalink
Excellent, Mike has decided to join the party.
(2004-07-22 10:35:51.0)
Permalink
Dependencies - define what you need, and nothing else
I recently attended
Usenix, where
Bryan explained how
DTrace
had been used to uncover some excessive system load brought on
by the behavior of one application. A member of the audience asked
whether the application was uncovering a poorly implemented part of
the system. Bryan responded that in such cases the system will
always be analyzed to determine whether it could do better. But there
comes a point, that if an application requests an expensive service,
that's what it will get. Perhaps the application should be reexamined
to see if it needs the service in the first place?
This observation is very applicable to the runtime linking environment.
Over the years we've spent countless hours pruning the cost of
Think about the work the runtime linker
has to do to load an object. It has to find the object (sometimes
through a plethora of
LD_LIBRARY_PATH components), load the object, and relocate it.
The runtime linker then repeats this process for any dependencies of the loaded
object. That's a lot of work. So why do so many applications load
dependencies they don't need?
Perhaps it's sloppyness, too much
Note the use of the -r option. We want to force
The -u option uncovers totally unused objects, but there can
still be wasteful references. For example, the same application
reveals that a number of objects have wasteful dependencies:
Although the
To reduce system overhead, only record those dependencies you need,
and nothing else. As part of building the core OS, we run
scripts that perform actions such as
ldd(1)
-U in an attempt to prevent unnecessary dependency loading from
creeping in.
Note, you can also observe unused object processing using the runtime
linkers debugging capabilities (LD_DEBUG=unused). Or, you
can uncover unused objects during a link-edit using the same debugging
technique (LD_OPTIONS=-Dunused). Another way of pruning unwanted
dependencies is to use the
-z ignore option of
(2004-07-15 19:39:07.0)
Permalink
A recent email discussion reminded me of how fragile, and prevalent,
LD_LIBRARY_PATH use it.
Within a development environment, this variable is very useful.
I use it all the time to experiment with new libraries. But within a
production environment, use of this environment variable can be
problematic.
See
Directories Searched by the Runtime Linker for an overview of
LD_LIBRARY_PATH use at runtime.
People use this environment variable to establish search paths for applications whose dependencies do not reside in constant locations.
Sometimes wrapper scripts are employed to set this variable,
other times users maintain an LD_LIBRARY_PATH within their
If you have a large number of LD_LIBRARY_PATH components specified,
you'll see
Wrapper scripts attempt to compensate for inherited LD_LIBRARY_PATH
use. For example, a version of acroread reveals:
The script is prepending its LD_LIBRARY_PATH requirement to
any inherited definition. Although this provides the necessary environment
for acroread to execute, we're still wasting time looking for any
system libraries in the acroread sub-directories.
When 64-bit binaries came along, we had a bit of a dilemma with how to
interpret LD_LIBRARY_PATH. But, because of its popularity, it was
decided to leave it applicable to both class of binaries (64 and 32-bit),
even though its unusual for a directory to contain both 64 and 32-bit
dependencies. We also added LD_LIBRARY_PATH_64 and
LD_LIBRARY_PATH_32 as a means of specifying search paths that
are specific to a class of objects. These class specific environment
variables are used instead of any generic
LD_LIBRARY_PATH setting.
Which leads me back to the recent email discussion. Seems a customer
was setting both the _64 and _32 variables as part
of their startup script, because both 64 and 32 bit processes could be spawned. However, one spawned process was acroread. Its
LD_LIBRARY_PATH setting was being overridden by the
_32 variable, and hence it failed to execute. Sigh.
Is there a solution to this mess? I guess we could keep bashing
LD_LIBRARY_PATH into submission some way, but why not get
rid of the LD_LIBRARY_PATH requirement altogether? This
can be done. Applications and dependencies can be built to include
a runpath using
Is there a limitation to $ORIGIN use? Yes, as directed
by the security folks,
expansion of this token is not allowed for secure applications.
But then again, for secure applications, LD_LIBRARY_PATH components
are ignored for non-secure directories anyway. See
Security.
For a flexible mechanism of finding dependencies, use a runpath that
includes the $ORIGIN token, and try not to create secure
applications :-)
(2004-07-10 22:20:54.0)
Permalink
Comments [8]
So, blogs seem popular, and a couple of folks have suggested
I start one, so here it is. I've been at
Sun for 15 years,
most of that time developing and maintaining the linker-editors,
various related tools, documentation, and building lots of software.
The link-editors start with
I maintain all related manual pages, and the
Linker and
Libraries Guide. This is one of the few manuals written by
the engineers that maintain the code. The
DTrace
crew have followed this example, and have produced their own excellent
documentation.
For those looking for link-editor information, I suggest
starting with the latest and greatest, which at this point is
a version of the
Solaris 10 Linker and
Libraries Guide available off of
http://docs.sun.com.
A good starting point is
Appendix A - Link-Editor Quick Reference,
a cheat sheet of the various objects that are
created by the link-editor. From this section you can vector off
to all sort of gory details. If you're not runing Solaris 10
yet (which you could if you used
Solaris Express), don't worry.
Appendix D -
Linker and Libraries Updates and New Features
itemizes the major changes from release to
release, so you can always determine what is, or isn't available
for your release.
But you might be running something newer than you think. The
link-editors are delivered as part of the Solaris core OS, however
we're always providing new features that are required by other utilities,
such as the compliers.
And the compilers are delivered asynchronously with various OS
releases. Consequently we're always providing patches. And our
patches are a snapshot of some of the latest bits available at the time
the patch was created. Effectively, we only have one source
base for the link-editors. Changes are
made in one place, and integrated into the latest patches. Thus
a patch to Solaris 8 or Solaris 9, SPARC and Intel, will
be the same, and comprise of a snapshot of what's been integrated
into Solaris 10. Again, the best place to find documentation
is the
Solaris 10 Linker and
Libraries Guide.
So, that completes this introduction. Hopefully I'll follow
up with other postings, perhaps some clarification of
existing practice, some new cheat-sheets, or other items that
seem helpful. If you've got any comments,
questions or advice for improvements, let us know. The door
is always open, and we're always looking for ideas and feedback.
Oh yeah, you're supposed to get a little personal with this blog
stuff aren't you? When I'm not working, I'm on a bike (road and
mountain), or chasing my daughter around. And, having originated
from the British Isles, a passion for real beer remains :-)
(2004-07-07 11:49:43.0)
Permalink
% cat interface.h
class item {
protected:
item();
public:
virtual void method1() = 0;
virtual void method2() = 0;
virtual ~item();
};
extern item *make_item();
% cat implementation.cc
#include "interface.h"
class __global item; /* Ensures global linkage for any
implicitly generated members. */
item::item() { }
item::~item() { }
class item_impl : public item {
void method1();
void method2();
};
void item_impl::method1() { }
void item_impl::method2() { }
void helper_func() { }
__global item *make_item() {
helper_func();
return new item_impl;
}
% elfdump -CsN.symtab implementation.so.1
...
[31] .... FUNC LOCL H 0 .text void helper_func()
[32] .... OBJT LOCL H 0 .data item_impl::__vtbl
[35] .... FUNC LOCL H 0 .text void item_impl::method1()
[36] .... FUNC LOCL H 0 .text void item_impl::method2()
[54] .... OBJT GLOB D 0 .data item::__vtbl
[55] .... FUNC GLOB D 0 .text item::item()
[58] .... FUNC GLOB D 0 .text item::~item #Nvariant 1()
[59] .... FUNC GLOB D 0 .text item*make_item()
[61] .... OBJT GLOB D 0 .data _edata
[67] .... FUNC GLOB D 0 .text item::~item()
[77] .... FUNC GLOB D 0 .text item::item #Nvariant 1()
A Update - Sunday May 29, 2005
Monday December 06, 2004
% cp /usr/lib/ld.so.1 /local/safe/lib
% cp /usr/lib/libc.so.1 /local/safe/lib
...
% LD_OPTIONS=-I/local/safe/lib/ld.so.1 \
cc -o main main.c -R/local/safe/lib ...
% /local/safe/lib/ld.so.1 main ...
Friday October 22, 2004
% elfdump -d /usr/lib/libdl.so.1
Dynamic Section: .dynamic
index tag value
[0] SONAME 0x138 libdl.so.1
[1] FILTER 0x143 /usr/lib/ld.so.1
...
% elfdump -d /usr/lib/libc.so.1
Dynamic Section: .dynamic
index tag value
...
[3] SONAME 0x6280 libc.so.1
[4] AUXILIARY 0x628a /usr/platform/$PLATFORM/lib/libc_ psr.so.1
...
% LD_DEBUG=symbols,bindings date
.....
11055: symbol=memset; lookup in file=/usr/bin/date
11055: symbol=memset; lookup in file=/usr/lib/libc.so.1
11055: symbol=memset; lookup in file=/usr/platform/SUNW,Sun-Fire/lib/libc_psr.so.1
11055: binding file=/usr/lib/libc.so.1 to \
file=/usr/platform/SUNW,Sun-Fire/lib/libc_psr.so.1: symbol `memset'
.....
% cat mapfile
SUNW_1.22 {
global:
....
dlopen = FUNCTION FILTER /usr/lib/ld.so.1;
dlsym = FUNCTION FILTER /usr/lib/ld.so.1;
....
};
....
SUNW_0.7 {
global:
....
memcmp = AUXILIARY /platform/$PLATFORM/lib/libc_psr.so.1;
memcpy = AUXILIARY /platform/$PLATFORM/lib/libc_psr.so.1;
....
};
% elfdump -y /lib/libc.so.1
Syminfo Section: .SUNW_syminfo
index flgs bound to symbol
....
[93] F [1] libm.so.2 isnand
[95] F [1] libm.so.2 isnanf
....
% elfdump -H /opt/ISV/lib/hwcap/*
/opt/ISV/lib/hwcap/libfoo_hwcap1.so.1:
Hardware/Software Capabilities Section: .SUNW_cap
index tag value
[0] CA_SUNW_HW_1 0x869 [ SSE MMX CMOV SEP FPU ]
/opt/ISV/lib/hwcap/libfoo_hwcap2.so.1:
Hardware/Software Capabilities Section: .SUNW_cap
index tag value
[0] CA_SUNW_HW_1 0x1871 [ SSE2 SSE MMX CMOV AMD_SYSC FPU ]
% elfdump -d /opt/ISV/lib/libfoo.so.1
Dynamic Section: .dynamic
index tag value
...
[3] SONAME 0x138 libfoo.so.1
[4] AUXILIARY 0x4124 /opt/ISV/lib/hwcap/$HWCAP
...
Wednesday September 29, 2004
% LD_DEBUG=bindings main
.....
00966: binding file=main to file=/lib/libc.so.1 symbol `_iob'
.....
00966: binding file=/lib/libc.so.1 to file=main: symbol `_end'
.....
00966: 1: transferring control: main
.....
00966: 1: binding file=main to file=/lib/libc.so.1: symbol `atexit'
.....
00966: 1: binding file=main to file=/lib/libc.so.1: symbol `exit'
% LD_DEBUG=files main
.....
16763: file=libc.so.1; needed by main
16763: file=/lib/libc.so.1 [ ELF ]; generating link map
.....
16763: 1: transferring control: ./main
.....
16763: 1: file=/lib/libc.so.1; \
filter for /platform/$PLATFORM/lib/libc_psr.so.1
16763: 1: file=/platform/SUNW,Sun-Blade-1000/lib/libc_psr.so.1; \
filtered by /lib/libc.so.1
16763: 1: file=/platform/SUNW,Sun-Blade-1000/lib/libc_psr.so.1 [ ELF ]; \
generating link map
.....
16763: 1: file=libelf.so.1; dlopen() called from file=./main \
[ RTLD_LAZY RTLD_LOCAL RTLD_GROUP RTLD_WORLD ]
16763: 1: file=/lib/libelf.so.1 [ ELF ]; generating link map
% LD_DEBUG=unused main
.....
11143: 1: file=libWWW.so.1 unused: does not satisfy any references
11143: 1: file=libXXX.so.1 unused: does not satisfy any references
.....
11143: 1: transferring control: ./main
.....
% LD_DEBUG=basic main
.....
34561: 1: calling .init (from sorted order): libYYY.so.1
34561: 1: calling .init (done): libYYY.so.1
.....
34561: 1: calling .init (from sorted order): libZZZ.so.1
.....
34561: 1: calling .init (dynamically triggered): libAAA.so.1
34561: 1: calling .init (done): libAAA.so.1
.....
34561: 1: calling .init (done): libZZZ.so.1
% LD_OPTIONS=symbols,detail cc -o main $(OBJS) $(LIBS) ...
% lari main
[2:3]: foo(): /opt/ISV.I/lib/libfoo.so.1
[2:0]: foo(): /opt/ISV.II/lib/libbar.so.1
[2:4]: bar[0x80]: /opt/ISV.I/lib/libfoo.so.1
[2:0]: bar[0x100]: /opt/ISV.II/lib/libbar.so.1
% lari main
[2:1D]: foo(): ./libA.so
[2:1D]: foo(): ./libB.so
Monday August 30, 2004
% size -x main *.so
main: 2df35c + 2675a4 + 80918f8 = 0x85d81f8
libxxx.so: 64d4d9c + 9af9f6 + 19604ba = 0x87e4c4c
libyyy.so: 4db7aeb + 76aa4c + 32cc16c = 0x87ee6a3
libzzz.so: 3f347ce + d8ebb1 + 4642a3b = 0x9305dba
....
Sunday August 22, 2004
% pvs -d /lib/libelf.so.1
libelf.so.1;
SUNW_1.5;
SUNW_1.4;
....
SUNWprivate_1.1;
% cc -o main main.c -lelf
% pvs -r main
libelf.so.1 (SUNW_1.4);
% pvs -dn /lib/libelf.so.1
SUNW_1.3;
SUNWprivate_1.1;
% main
ld.so.1: ./main: fatal: libelf.so.1: version `SUNW_1.4' not found \
(required by file ./main)
% cat mapfile
ISV_1.1 {
global:
foo1();
foo2();
local:
*;
};
% cc -o libfoo.so.1 -G -Kpic -Mmapfile foo.c bar.c ...
% pvs -dos libfoo.so.1
libfoo.so.1 - ISV_1.1: foo1;
libfoo.so.1 - ISV_1.1: foo2;
Sunday August 01, 2004
% ldd main
foo.so => ./foo.so
bar.so => ./bar.so
...
% ldd -r bar.so
symbol not found: foo (./bar.so)
% LD_DEBUG=bindings,symbols,files main
.....
07683: 1: transferring control: ./main
.....
07683: 1: file=bar.so; lazy loading from file=./main: symbol=bar
.....
07683: 1: binding file=./main to file=./bar.so: symbol `bar'
07683: 1: symbol=foo; lookup in file=./main [ ELF ]
07683: 1: symbol=foo; lookup in file=./bar.so [ ELF ]
07683: 1:
07683: 1: rescanning for lazy dependencies for symbol: foo
07683: 1:
07683: 1: file=foo.so; lazy loading from file=./main: symbol=foo
.....
07683: 1: binding file=./bar.so to file=./foo.so: symbol `foo'
Tuesday July 27, 2004
% cat wally.c
extern void foo(), bar();
void wally(int who)
{
who ? foo() : bar();
}
% cc -o wally.so wally.c -G -Kpic -zdefs -zlazyload -R'$ORIGIN' foo.so bar.so
% elfdump -d wally.so | egrep "NEEDED|POSFLAG"
[0] POSFLAG_1 0x1 [ LAZY ]
[1] NEEDED 0x66 foo.so
[2] POSFLAG_1 0x1 [ LAZY ]
[3] NEEDED 0x6d bar.so
% ldd -Ld wally.so
%
% ldd -Lr wally.so
foo.so => ./foo.so
bar.so => ./bar.so
% cc -o wally.so wally.c -G -Kpic -zdefs -R'$ORIGIN' foo.so bar.so
% ldd -Ldu wally.so
foo.so => ./foo.so
bar.so => ./bar.so
unused object=./foo.so
unused object=./bar.so
% LD_DEBUG=files main
.....
25670: 1: transferring control: ./main
.....
25608: 1: file=bar.so; lazy loading from file=./wally.so: symbol=bar
.....
Thursday July 22, 2004
Thursday July 15, 2004
% ldd -u -r app
...
unused object=/lib/libmd5.so.1
% ldd -U -r app
...
unreferenced object=/usr/openwin/lib/libX11.so.4; unused dependency of app
unreferenced object=/usr/openwin/lib/libXt.so.4; unused dependency of app
unreferenced object=/lib/libw.so.1; unused dependency of app
Saturday July 10, 2004
% ldd -s /usr/bin/date
...
find object=libc.so.1; required by /usr/bin/date
search path=/opt/ISV/lib (LD_LIBRARY_PATH)
LD_LIBRARY_PATH="`prepend "$ACRO_INSTALL_DIR/$ACRO_CONFIG/lib:\
$ACRO_INSTALL_DIR/$ACRO_CONFIG/lib" "$LD_LIBRARY_PATH"`
Wednesday July 07, 2004
