Kool Aid Served Daily

My builds are taking too long. At NetApp, I never worried about my PATH variable - it never changed pretty much the entire time I was there. /usr/local/bin always had the right compiler - remember NetApp does servers only. And they cross-compiled everything they could. Sun does clients and no real cross-compiling that I can see.

So my full build was taking 7-9 hours, which was 5-6 hours too long. I thought the nightly script ignored my PATH and built its own. I was wrong. So I took out some WAN links in my PATH and got down to about 4 hours, which is still longer than other people. Now, I could just copy a PATH statement from a colleague, but where is the fun in that I ask you?

I decided to play with perl to see if I could find out which directory components were remote. Here is my first hack at a script:

#!/usr/local/bin/perl

$envPath = `printenv PATH`;

chomp($envPath);

@paths = split(":", $envPath);

foreach $key (@paths) {
        `df -k $key`;
        print $key . "\n";
}

Okay, I know the output isn't very pretty, but lets try this puppy out:

[th199096@warlock ~]> perl ./dfpath.pl
/ws/onnv-tools/SUNWspro/SOS10/bin
/opt/teamware/bin
/ws/onnv-tools/teamware/bin
/opt/onbld/bin
/opt/onbld/bin/i386
/ws/onnv-tools/onbld/bin
/ws/onnv-tools/onbld/bin/i386
/usr/dt/bin
/usr/openwin/bin
/usr/ccs/bin
/opt/sfw/sbin
/opt/sfw/bin
/usr/sfw/sbin
/usr/sfw/bin
/usr/local/bin
df: (/usr/local/sbin) not a block device, directory or mounted resource
/usr/local/sbin
/usr/bin
/usr/sbin
/bin
/sbin
/usr/ucb
/usr/X11R6/bin
/export/home/th199096/bin
/usr/games

This is on my home system, which has a slightly different PATH than the build machines up at work. I'll move on to that system in just a minute. First I want to see what is going on with /usr/local/sbin.

[th199096@warlock ~]> df -k /usr/local/sbin
df: (/usr/local/sbin) not a block device, directory or mounted resource
[th199096@warlock ~]> df -k /usr/local
Filesystem            kbytes    used   avail capacity  Mounted on
/dev/dsk/c1t0d0s0    63168555 19722622 42814248    32%    /
[th199096@warlock ~]> cd /usr/local/sbin
/usr/local/sbin: No such file or directory.
[th199096@warlock ~]> ls -la /usr/local/
total 8
drwxr-xr-x   4 root     root         512 Jan 24 18:38 .
drwxr-xr-x  39 root     sys         1024 Jan 24 23:28 ..
drwxr-xr-x   2 root     bin          512 Jan 25 00:45 bin
drwxr-xr-x   3 root     sys         1024 Jan 24 18:40 etc

I like to pretend I'm going to share my .tcshrc across different platforms. Lets look at the relevant parts

# Set a reasonable route through the file system, paths etc.
set path=(/usr/local/{bin,sbin} /usr/{bin,sbin} /{bin,sbin})

if ( -x /usr/ucb ) then
        set path=($path /usr/ucb)
endif

if ( -x /usr/j2sdk1.4.207/bin ) then
        set path=(/usr/j2sdk1.4.207/bin $path)
endif

if ( -x /usr/sfw/bin ) then
        set path=(/usr/sfw/bin $path)
endif

if ( -x /usr/sfw/sbin ) then
        set path=(/usr/sfw/sbin $path)
endif

if ( -x /opt/sfw/bin ) then
        set path=(/opt/sfw/bin $path)
endif

if ( -x /opt/sfw/sbin ) then
        set path=(/opt/sfw/sbin $path)
endif

if ( -x /usr/ccs/bin ) then
        set path=(/usr/ccs/bin $path)
endif

if ( -x /usr/openwin/bin ) then
        set path=(/usr/openwin/bin $path)
endif

if ( -x /usr/dt/bin ) then
        set path=(/usr/dt/bin $path)
endif

if ( -x /ws/onnv-tools/onbld/bin/`uname -p` ) then
        set path=(/ws/onnv-tools/onbld/bin/`uname -p` $path)
endif

if ( -x /ws/onnv-tools/onbld/bin ) then
        set path=(/ws/onnv-tools/onbld/bin $path)
endif

if ( -x /opt/onbld/bin/`uname -p` ) then
        set path=(/opt/onbld/bin/`uname -p` $path)
endif

if ( -x /opt/onbld/bin ) then
        set path=(/opt/onbld/bin $path)
endif

if ( -x /ws/onnv-tools/teamware/bin ) then
        set path=(/ws/onnv-tools/teamware/bin $path)
endif

if ( -x /opt/teamware/bin ) then
        set path=(/opt/teamware/bin $path)
endif

if ( -x /ws/onnv-tools/SUNWspro/SOS10/bin ) then
        set path=(/ws/onnv-tools/SUNWspro/SOS10/bin $path)
endif

#if ( -x /opt/onnv-gate/public/bin ) then
#       set path=(/opt/onnv-gate/public/bin $path)
#endif

# Path for X
if ( -x /usr/X11R6/bin ) then
        set path=($path /usr/X11R6/bin)
else if ( -x /usr/X11/bin ) then
        set path=($path /usr/X11/bin)
endif

# Other possibilities to put in the path
if( -x ~/bin ) then
        set path=($path ~/bin)
endif

if( -x ~/scripts ) then
        set path=($path ~/scripts)
endif

if( -x /usr/share/bin ) then
        set path=($path /usr/share/bin)
endif

if( -x /usr/contrib/bin ) then
        set path=($path /usr/contrib/bin)
endif

if( -x /usr/local/scripts ) then
        set path=($path /usr/local/scripts)
endif

if( -x /usr/games ) then
        set path=($path /usr/games)
endif

if( -x /Applications ) then
        set path=($path /Applications ~/Applications)
endif

I need to make this part:

# Set a reasonable route through the file system, paths etc.
set path=(/usr/local/{bin,sbin} /usr/{bin,sbin} /{bin,sbin})

Be a little smarter. As a matter of fact, I need to stop the cut and paste action above. I'll try a little trick I picked up from sr. But before I do that, note that on the work build machines, I already have:

# Optimized for Solaris
set path=(/usr/local/{bin,sbin} /usr/{bin,sbin} /sbin)

I'm going to cut out some fluff and use sr's trick here:

# Make sure this is last
set path=(.)

set cpath = (/opt/onbld/bin /opt/onbld/bin/`/usr/bin/uname -p` \
                /usr/ccs/bin /opt/SUNWspro/bin /opt/teamware/bin \
                /usr/sfw/bin /opt/sfw/sbin /opt/sfw/bin /usr/dt/bin  \
                /usr/openwin/bin /usr/local/{bin,sbin} /usr/{bin,sbin} \
                /sbin /usr/ucb ~/bin)
foreach dir (${cpath})
        if ( -x ${dir} ) then
                set path=(${dir} ${path})
        endif
end

Which yields:

[th199096@warlock ~]> perl ~/dfpath.pl
/export/home/th199096/bin
/usr/ucb
/sbin
/usr/sbin
/usr/bin
/usr/local/bin
/usr/openwin/bin
/usr/dt/bin
/opt/sfw/bin
/opt/sfw/sbin
/usr/sfw/bin
/opt/teamware/bin
/usr/ccs/bin
/opt/onbld/bin/i386
/opt/onbld/bin
.

Which is backwards from what I want. I want to be able to write it correctly, so lets try:

# Make sure this is last
set path=(.)

set cpath = (/opt/onbld/bin /opt/onbld/bin/`/usr/bin/uname -p` \
                /usr/ccs/bin /opt/SUNWspro/bin /opt/teamware/bin \
                /usr/sfw/bin /opt/sfw/sbin /opt/sfw/bin /usr/dt/bin  \
                /usr/openwin/bin /usr/local/{bin,sbin} /usr/{bin,sbin} \
                /sbin /usr/ucb ~/bin)
set dpath=""
foreach dir (${cpath})
        set dpath=(${dir} ${dpath})
end

foreach dir (${dpath})
        if ( -x ${dir} ) then
                set path=(${dir} ${path})
        endif
end

unset cpath dir dpath

Which yields:

[th199096@warlock ~]> perl ~/dfpath.pl
/opt/onbld/bin
/opt/onbld/bin/i386
/usr/ccs/bin
/opt/teamware/bin
/usr/sfw/bin
/opt/sfw/sbin
/opt/sfw/bin
/usr/dt/bin
/usr/openwin/bin
/usr/local/bin
/usr/bin
/usr/sbin
/sbin
/usr/ucb
/export/home/th199096/bin
.

I've made my paths easier to maintain and I've also already removed some dead wood. Now, time to work on the perl script. If we tweak the df slightly, we get:

print `df -k $key`;

Which yields:

[th199096@warlock ~]> perl ~/dfpath.pl | head
Filesystem            kbytes    used   avail capacity  Mounted on
/dev/dsk/c1t0d0s0    63168555 19722640 42814230    32%    /
/opt/onbld/bin
Filesystem            kbytes    used   avail capacity  Mounted on
/dev/dsk/c1t0d0s0    63168555 19722640 42814230    32%    /
/opt/onbld/bin/i386
Filesystem            kbytes    used   avail capacity  Mounted on
/dev/dsk/c1t0d0s0    63168555 19722640 42814230    32%    /
/usr/ccs/bin
Filesystem            kbytes    used   avail capacity  Mounted on

Okay, we need to get rid of the first line:

        print `df -k $key | grep -v Filesystem`;

Which yields:

[th199096@warlock ~]> perl ~/dfpath.pl | head
/dev/dsk/c1t0d0s0    63168555 19722640 42814230    32%    /
/opt/onbld/bin
/dev/dsk/c1t0d0s0    63168555 19722640 42814230    32%    /
/opt/onbld/bin/i386
/dev/dsk/c1t0d0s0    63168555 19722640 42814230    32%    /
/usr/ccs/bin
/dev/dsk/c1t0d0s0    63168555 19722640 42814230    32%    /
/opt/teamware/bin
/dev/dsk/c1t0d0s0    63168555 19722640 42814230    32%    /
/usr/sfw/bin

But why use Perl at all if we continue down this pipe path? I thought I had to use Perl to split the ':' apart, but it turns out the $path keeps the list without the ':'. So the following tcsh script works:

#!/usr/bin/tcsh -f

foreach i ( $path )
        set j = `df -k ${i} | grep -v Filesystem | grep -v ^/dev`
        if ( "${j}" != "" ) then
                echo "${i} is not local"
        endif
end

This will tell me all filesystems in my path which are not local. Note I have switched to a work system.

[th199096@haulass ~]> ./dftab.tcsh
/opt/onbld/bin is not local
/opt/onbld/bin/i386 is not local
/opt/SUNWspro/bin is not local
/opt/teamware/bin is not local
/opt/sfw/bin is not local
/usr/local/bin is not local
/usr/local/sbin is not local
/home/th199096/bin is not local
. is not local

It seems weird that a NFS engineer is concerned with network traffic - i.e., why not just make the protocol faster. I'm actually looking to see which paths are WAN. A little tweak to the script yields:

[th199096@haulass ~]> ./dftab.tcsh
/opt/onbld/bin is not local
tools-i386:/export/tools.i386/onnv-tools/onbld 7150861 5940637 1138716 84% /opt/onbld
/opt/onbld/bin/i386 is not local
tools-i386:/export/tools.i386/onnv-tools/onbld 7150861 5940637 1138716 84% /opt/onbld
/opt/SUNWspro/bin is not local
tools-i386:/export/tools.i386/on10-tools/SUNWspro/SOS8/bin 7150861 5940637 1138716 84% /opt/SUNWspro/bin
/opt/teamware/bin is not local
tools-i386:/export/tools.i386/on10-tools/SUNWspro/SOS8 7150861 5940637 1138716 84% /opt/teamware
/opt/sfw/bin is not local
boulder-local1-27,boulder-local2-27,boulder-local1-26,boulder-local2-26,boulder-local1-25,boulder-local2-25:/export/opt.sfw.i386/opt/sfw 24932348 21650384 3032641 88% /opt/sfw
/usr/local/bin is not local
boulder-local1-27,boulder-local2-27,boulder-local1-26,boulder-local2-26,boulder-local1-25,boulder-local2-25:/export/local.i386 24932348 21650384 3032641 88% /usr/local
/usr/local/sbin is not local
boulder-local1-27,boulder-local2-27,boulder-local1-26,boulder-local2-26,boulder-local1-25,boulder-local2-25:/export/local.i386 24932348 21650384 3032641 88% /usr/local
/home/th199096/bin is not local
hera-home1.central:/export/home1/41/th199096 175540457 111466051 60563597 65% /home/th199096
. is not local
hera-home1.central:/export/home1/41/th199096 175540457 111466051 60563597 65% /home/th199096

So all of them check out as being local - i.e., in the sense they are not going out across the WAN. I had to login to tools-i386 to verify this for that machine. What I really need to do now is reduce the number of path components. I've already removed /bin because it is a symlink to /usr/bin. This is just a hack, but here I reduce my entries in my PATH for a build:

[th199096@haulass ~]> more concise.tcsh
set path=""

set cpath = (/opt/onbld/bin /opt/onbld/bin/`/usr/bin/uname -p` \
                /usr/ccs/bin /opt/SUNWspro/bin /opt/teamware/bin \
                /usr/sfw/bin /usr/{bin,sbin} /sbin )
set dpath=""
foreach dir (${cpath})
        set dpath=(${dir} ${dpath})
end

foreach dir (${dpath})
        if ( -x ${dir} ) then
                set path=(${dir} ${path})
        endif
end

unset cpath dir dpath
[th199096@haulass ~]> source concise.tcsh
[th199096@haulass ~]> printenv PATH
/opt/onbld/bin:/opt/onbld/bin/i386:/usr/ccs/bin:/opt/SUNWspro/bin:/opt/teamware/bin:/usr/sfw/bin:/usr/bin:/usr/sbin:/sbin
[th199096@haulass ~]> ./dftab.tcsh
/opt/onbld/bin is not local
tools-i386:/export/tools.i386/onnv-tools/onbld 7150861 5940637 1138716 84% /opt/onbld
/opt/onbld/bin/i386 is not local
tools-i386:/export/tools.i386/onnv-tools/onbld 7150861 5940637 1138716 84% /opt/onbld
/opt/SUNWspro/bin is not local
tools-i386:/export/tools.i386/on10-tools/SUNWspro/SOS8/bin 7150861 5940637 1138716 84% /opt/SUNWspro/bin
/opt/teamware/bin is not local
tools-i386:/export/tools.i386/on10-tools/SUNWspro/SOS8 7150861 5940637 1138716 84% /opt/teamware

I'm going to kick off a full build to see if that made a difference.

Here was my build time when I had WAN links in my PATH:

[th199096@haulass havok]> more speed.txt
[th199096@haulass havok]>
[1]    Done                          env -i `which nightly` -n ./clean.env
Time spent in user mode   (CPU seconds) : 16610.72s
Time spent in kernel mode (CPU seconds) : 4789.76s
Total time                              : 7:46:51.40s
CPU utilisation (percentage)            : 76.3%

And here was my build time when I took out the WAN links and rearranged the ordering. I.e., our CUE environment rewrote my PATH for me. :-< :

[th199096@haulass havok]>
[1]    Done                          env -i `which nightly` -n ./clean.env
Time spent in user mode   (CPU seconds) : 15498.05s
Time spent in kernel mode (CPU seconds) : 4439.35s
Total time                              : 4:02:55.12s
CPU utilisation (percentage)            : 136.7%

So changing the paths saved 3 hours and 45 minutes.

I'm off to play with the family, check back later for the real results.

Update The results were worse. My ssh session was disconnected, but it took 5 hours.

With WAN:                       7:46:40
Without WAN and stupid paths:   4:02:50
With hacks in here:             5:00:58

I can't swear the build didn't get longer, there was a major putback. I should try this on a Sparc. End Update