Tuesday July 27, 2004

Dependencies - perhaps they can be lazily loaded

In a previous posting, I stated that you should only record those dependencies you need, and nothing else. There's another step you can take to reduce start-up processing overhead.

Dynamic objects need to resolve symbolic references from each other. Function calls are typically implemented through an indirection that allows the function binding to be deferred until the function call is first made. See When Relocations Are Performed. Because of this deferral, it is also possible to cause the defining dependency to be loaded when the function call is first made. This model is referred to as Lazy Loading.

To establish lazy loading, you must pass the -z lazyload option to ld(1) when you build your dynamic object. In addition, the association of a symbol reference to a dependency requires that the dependency is specified as part of the link-edit. It is recommended that you use the link-editors -z defs option to insure that all dependencies are specified when you build your dynamic object. The following example establishes lazy dependencies for the references foo() and bar().

    % cat wally.c
    extern void foo(), bar();

    void wally(int who)
    {
        who ? foo() : bar();
    }
    % cc -o wally.so wally.c -G -Kpic -zdefs -zlazyload -R'$ORIGIN' foo.so bar.so

The lazy loading attribute of these dependencies can be displayed with elfdump(1).

   % elfdump -d wally.so | egrep "NEEDED|POSFLAG"
        [0]  POSFLAG_1        0x1               [ LAZY ]
        [1]  NEEDED           0x66              foo.so
        [2]  POSFLAG_1        0x1               [ LAZY ]
        [3]  NEEDED           0x6d              bar.so

By default, ldd(1) displays all dependencies, in that it will force lazy loaded objects to be processed. To reveal lazy loading, use the -L option. For example, when a dynamic object is loaded into memory, all data relocations are performed before the object can gain control. Thus the following operation reveals that neither dependency is loaded.

    % ldd -Ld wally.so
    %

Once function relocations are processed, both dependencies are loaded to resolve the function reference.

    % ldd -Lr wally.so
        foo.so =>       ./foo.so
        bar.so =>       ./bar.so

ldd(1) becomes a convenient tool for discovering whether lazy loading might be applicable. Suppose we rebuilt wally.so without the -z lazyload option. And recall from my previous posting that the -u option can be used to discover unused dependencies.

    % cc -o wally.so wally.c -G -Kpic -zdefs -R'$ORIGIN' foo.so bar.so
    % ldd -Ldu wally.so
        foo.so =>        ./foo.so
        bar.so =>        ./bar.so

      unused object=./foo.so
      unused object=./bar.so

This has revealed that loading wally.so and relocating it as would occur at process startup, did not require the dependencies foo.so or bar.so to be loaded. This confirms that these two dependencies can be lazily loaded when reference to them is first made.

Lazy loading can be observed at runtime using the runtime linkers debugging capabilities (LD_DEBUG=files). For example, if wally() was called with a zero argument, we'd see bar.so lazily loaded.

   % LD_DEBUG=files main
   .....
   25670: 1: transferring control: ./main
   .....
   25608: 1: file=bar.so;  lazy loading from file=./wally.so: symbol=bar
   .....

Note, not only on does lazy loading have the potential of reducing the cost of start-up processing, but if lazy loading references are never called, the dependencies will never be loaded as part of the process.

(2004-07-27 17:25:59.0) Permalink

Thursday July 22, 2004

Linker Alien Spotting

Excellent, Mike has decided to join the party.

(2004-07-22 10:35:51.0) Permalink

Thursday July 15, 2004

Dependencies - define what you need, and nothing else

I recently attended Usenix, where Bryan explained how DTrace had been used to uncover some excessive system load brought on by the behavior of one application. A member of the audience asked whether the application was uncovering a poorly implemented part of the system. Bryan responded that in such cases the system will always be analyzed to determine whether it could do better. But there comes a point, that if an application requests an expensive service, that's what it will get. Perhaps the application should be reexamined to see if it needs the service in the first place?

This observation is very applicable to the runtime linking environment. Over the years we've spent countless hours pruning the cost of ld.so.1(1), only to see little improvement materialize with real applications. Alternatively, there's no better way of reducing the overhead of servicing a particular operation, than not requesting the operation in the first place :-)

Think about the work the runtime linker has to do to load an object. It has to find the object (sometimes through a plethora of LD_LIBRARY_PATH components), load the object, and relocate it. The runtime linker then repeats this process for any dependencies of the loaded object. That's a lot of work. So why do so many applications load dependencies they don't need?

Perhaps it's sloppyness, too much Makefile cut-and-pasting, or the inheritance of global build flags. Or, perhaps the developer doesn't realize a dependency isn't required. One way to discover dependency requirements is with ldd(1) and the -u option. For example, this application, nor any of its dependencies, make reference to libmd5.so.1:

    % ldd -u -r app
    ...
    unused object=/lib/libmd5.so.1

Note the use of the -r option. We want to force ldd(1) to bind all relocations, data and functions. However, here we're wastfully loading libmd5.so.1. This should be removed as a dependency.

The -u option uncovers totally unused objects, but there can still be wasteful references. For example, the same application reveals that a number of objects have wasteful dependencies:

    % ldd -U -r app
    ...
    unreferenced object=/usr/openwin/lib/libX11.so.4; unused dependency of app
    unreferenced object=/usr/openwin/lib/libXt.so.4; unused dependency of app
    unreferenced object=/lib/libw.so.1; unused dependency of app

Although the X libraries are used by some dependency within the process, they're not referenced by the application. There are data structures maintained by the runtime linker that track dependencies. If a dependency isn't required, we've wasted time creating these data structures. Also, should the object that requires the X libraries be redelivered in a form that no longer needs the X libraries, the application is still going to cause them to be wastefully loaded.

To reduce system overhead, only record those dependencies you need, and nothing else. As part of building the core OS, we run scripts that perform actions such as ldd(1) -U in an attempt to prevent unnecessary dependency loading from creeping in.

Note, you can also observe unused object processing using the runtime linkers debugging capabilities (LD_DEBUG=unused). Or, you can uncover unused objects during a link-edit using the same debugging technique (LD_OPTIONS=-Dunused). Another way of pruning unwanted dependencies is to use the -z ignore option of ld(1) when building your application or shared object.

(2004-07-15 19:39:07.0) Permalink

Saturday July 10, 2004

LD_LIBRARY_PATH - just say no

A recent email discussion reminded me of how fragile, and prevalent, LD_LIBRARY_PATH use it. Within a development environment, this variable is very useful. I use it all the time to experiment with new libraries. But within a production environment, use of this environment variable can be problematic. See Directories Searched by the Runtime Linker for an overview of LD_LIBRARY_PATH use at runtime.

People use this environment variable to establish search paths for applications whose dependencies do not reside in constant locations. Sometimes wrapper scripts are employed to set this variable, other times users maintain an LD_LIBRARY_PATH within their .profile. This latter model can often get out of hand - try running:

    % ldd -s /usr/bin/date
    ...
    find object=libc.so.1; required by /usr/bin/date
	search path=/opt/ISV/lib	 (LD_LIBRARY_PATH)

If you have a large number of LD_LIBRARY_PATH components specified, you'll see libc.so.1 being wastefully searched for, until it is finally found in /usr/lib. Excessive LD_LIBRARY_PATH components don't help application startup performance.

Wrapper scripts attempt to compensate for inherited LD_LIBRARY_PATH use. For example, a version of acroread reveals:

    LD_LIBRARY_PATH="`prepend "$ACRO_INSTALL_DIR/$ACRO_CONFIG/lib:\
	$ACRO_INSTALL_DIR/$ACRO_CONFIG/lib" "$LD_LIBRARY_PATH"`

The script is prepending its LD_LIBRARY_PATH requirement to any inherited definition. Although this provides the necessary environment for acroread to execute, we're still wasting time looking for any system libraries in the acroread sub-directories.

When 64-bit binaries came along, we had a bit of a dilemma with how to interpret LD_LIBRARY_PATH. But, because of its popularity, it was decided to leave it applicable to both class of binaries (64 and 32-bit), even though its unusual for a directory to contain both 64 and 32-bit dependencies. We also added LD_LIBRARY_PATH_64 and LD_LIBRARY_PATH_32 as a means of specifying search paths that are specific to a class of objects. These class specific environment variables are used instead of any generic LD_LIBRARY_PATH setting.

Which leads me back to the recent email discussion. Seems a customer was setting both the _64 and _32 variables as part of their startup script, because both 64 and 32 bit processes could be spawned. However, one spawned process was acroread. Its LD_LIBRARY_PATH setting was being overridden by the _32 variable, and hence it failed to execute. Sigh.

Is there a solution to this mess? I guess we could keep bashing LD_LIBRARY_PATH into submission some way, but why not get rid of the LD_LIBRARY_PATH requirement altogether? This can be done. Applications and dependencies can be built to include a runpath using ld(1), and the -R option. This path is used to search for the dependencies of the object in which the runpath is recorded. If the dependencies are not in a constant location, use the $ORIGIN token as part of the pathname.

Is there a limitation to $ORIGIN use? Yes, as directed by the security folks, expansion of this token is not allowed for secure applications. But then again, for secure applications, LD_LIBRARY_PATH components are ignored for non-secure directories anyway. See Security.

For a flexible mechanism of finding dependencies, use a runpath that includes the $ORIGIN token, and try not to create secure applications :-)

(2004-07-10 22:20:54.0) Permalink Comments [8]

Wednesday July 07, 2004

Hello there

So, blogs seem popular, and a couple of folks have suggested I start one, so here it is. I've been at Sun for 15 years, most of that time developing and maintaining the linker-editors, various related tools, documentation, and building lots of software.

The link-editors start with ld(1), which takes various input from the compilers, and typically spits out a dynamic executable or share object. Then there's ld.so.1(1), the runtime linker, which takes a dynamic executable and combines it with its dependencies as part of executing a process. The utilities that compliment these processes include, crle(1), elfdump(1), pvs(1), and a bunch of support libraries, auditors, etc.

I maintain all related manual pages, and the Linker and Libraries Guide. This is one of the few manuals written by the engineers that maintain the code. The DTrace crew have followed this example, and have produced their own excellent documentation.

For those looking for link-editor information, I suggest starting with the latest and greatest, which at this point is a version of the Solaris 10 Linker and Libraries Guide available off of http://docs.sun.com. A good starting point is Appendix A - Link-Editor Quick Reference, a cheat sheet of the various objects that are created by the link-editor. From this section you can vector off to all sort of gory details. If you're not runing Solaris 10 yet (which you could if you used Solaris Express), don't worry. Appendix D - Linker and Libraries Updates and New Features itemizes the major changes from release to release, so you can always determine what is, or isn't available for your release.

But you might be running something newer than you think. The link-editors are delivered as part of the Solaris core OS, however we're always providing new features that are required by other utilities, such as the compliers. And the compilers are delivered asynchronously with various OS releases. Consequently we're always providing patches. And our patches are a snapshot of some of the latest bits available at the time the patch was created. Effectively, we only have one source base for the link-editors. Changes are made in one place, and integrated into the latest patches. Thus a patch to Solaris 8 or Solaris 9, SPARC and Intel, will be the same, and comprise of a snapshot of what's been integrated into Solaris 10. Again, the best place to find documentation is the Solaris 10 Linker and Libraries Guide.

So, that completes this introduction. Hopefully I'll follow up with other postings, perhaps some clarification of existing practice, some new cheat-sheets, or other items that seem helpful. If you've got any comments, questions or advice for improvements, let us know. The door is always open, and we're always looking for ideas and feedback.

Oh yeah, you're supposed to get a little personal with this blog stuff aren't you? When I'm not working, I'm on a bike (road and mountain), or chasing my daughter around. And, having originated from the British Isles, a passion for real beer remains :-)

(2004-07-07 11:49:43.0) Permalink