Nitro Burning Builds

Want to Cut Your Build Time in Half?

Thanks to work done by Sasha Kolbasov and Jonathan Chew on Project Nitro, your Solaris OS/Networking (ON) consolidation build time could be cut in half.

Build times are cut most drastically for SPARC builds, but x64/x86 users benefit as well. Use b85 or later. For details, search for "parallel" in the changelog for b85. More improvement is coming.

The April SVOSUG meeting (Silicon Valley OpenSolaris User Group) was pretty lively. People who do not build ON regularly have experience with other large builds. Everyone shared ideas about how to speed up a build. People were very interested in the details of what Sasha and Jonathan have done to speed up the ON build, and almost as interested in the tools they invented to visualize where improvements were made and where more improvements might be made. There was much discussion at the meeting of precompiled header files and recursive make. Sasha and Jonathan discovered issues with precompiled header files in the ON build. The ON make system does use recursive make invocations, but make is difficult to parallelize well since all dependencies are not known at the top level.

The greatest reduction in build time was acheived by increasing parallelism in the build. This revealed interesting issues with missing build dependencies and race conditions. Some advice:

• Express dependencies correctly. Dependencies should be explicit and complete. Do not use .WAIT to manage dependencies. Use .WAIT to separate big chunks of work, but not to manage dependencies.
• Use .PARALLEL as much as possible.
• Look for parts of the build that have common prerequisites and build those prerequisites up front.
• Convert pattern-matching rules to explicit rules in Makefiles.

DTrace can help you understand what targets are built and why. DTrace is an extremely useful took for analyzing complex workloads. Use gnuplot to visualize the data. Testing under extreme load uncovers some races. Debugging race conditions takes a lot of time. Modern machines can do a lot in parallel. Flat Makefiles might provide better parallelism, but they are difficult to write.

Some of the tasks of this project:

• Eliminate duplicate work
• Eliminate unnecessary dependencies
• Manage dependencies to minimize incremental build time
• Enable more parallelism (multiple threads, cores, sockets, machines)
• Optimize tools

Some results of this project:

• Huge improvement from using the newest Sun Studio 11 lint(1B). Even bigger improvement on x86 than on SPARC.
• Smaller improvement from building with source and objects in /tmp for building debug and non-debug sequentially and in parallel.
• Big improvement using ZFS (instead of UFS) with 32 GB of RAM on x86. Much smaller improvement on SPARC.
• Small improvement on x86 from using the ccache free and open source tool.
• Small improvement on x86 from using the distcc free and open source tool.

Hopefully the slide set for this presentation will show up on the Project Nitro page soon.

Posted at 04:26PM May 12, 2008 by alta in General  |  Comments[0]