Kool Aid Served Daily

In The Cake is a Lie, I gushed over being able to buy Steam games online. And once again, I decided to try to register my Half Life game with Steam. And once again I was disappointed.

I'm a software developer and I understand the need for licensing. I own two copies of every game my son and I play together. (I actually own 3 of Impossible Creatures because he misplaced one and I wasn't going to use some tool to load a CD image to fool his computer.) I have as many copies of WinXP as I have machines running it.

But, ID, Relic, Lucas Arts, even Microsoft, all work with the principle that if you have the physical media and the UPC, you own the game. I've almost never kept my receipt past 30 days for these games. I've gone and bought $9 new copies of Impossible Creatures from CompUSA years after the game was popular and I've had no problem with the licensing keys.

The way that Steam and Valve's support differs from this model is that they assume you are a thief - you are guilty until proven innocent. If I want to activate my Half Life game with Steam, mainly because I think it would be neat more than anything else, I've got to send them $10. It doesn't matter that I've owned the game for over 5 years. The assumption is that either I bought the game on eBay or I'm trying to steal the keys.

Even if I had my original receipt, the fact that it is over 90 days old would probably force me to pay the $10 fee anyway. I don't know, I wasn't smart enough to try and forge a receipt.

It might be the case that I registered the games with Sierra Online. It might be the case that I registered the game with Steam and I lost that account for some reason. It just doesn't matter. I'm assumed to be the one who is trying to pull a fast one. They will not tell you who has the keys registered. You can't see if it was an old account name you used to have.

The reality is that someone could have used a license key generator to steal my key. Or someone could have gotten the keys when they were in the retail store. Or duplicate keys may have been issued to retail boxes. But Valve and Steam are not willing to listen to you, work with you, or do anything other than insist that you pay $10 to get your keys.

It isn't the $10 which is sticking in my throat. It is the fact that if I give them that money, I have in fact admitted to being a thief. And at that point, they will have stolen something from me that I can never get back.

After the dotcom meltdown, tech companies started looking at ways to save money and a fat low hanging saver was employee vacation time. One way to encourage employees to take vacations was a mandatory shutdown of offices in the US (legal ramifications at overseas sites). The two prime periods were the 4th of July week and Christmas week. It gave the illusion that the employee was getting more bang for their buck.

I went through this at my last company. It wasn't nice, but we all understood it was either that or lay more people off. The directors at the time all stopped accruing vacation time as a sign of support. Things got better and we stopped having to do this trick. By the way, one of the resistances of employees is that they accrue time off as a sort of severance package. If you get laid off and have 6 weeks of vacation time, you have a cushion to help you land a new job.

When I left that company, I left the highest vacation accrual rate and started over at Sun at the bottom. I knew that they had just done a week off at Christmas, but I had no idea if it was paid or a shutdown. When June rolled around and I found out that there was a July 4th shutdown, I was bummed out. It wiped out my balance and it kept me from going anywhere over the Christmas holidays. It turned out that the Christmas week time was however a paid time off and not a shutdown.

You can imagine the joy I felt when it was announced this year that we would not be having a July 4th shutdown. What it told me was that the company was doing well and this OpenSolaris thing was a step in the right direction. I'm not sure if management realized how big of an employee morale boost this was for us.

I actually took the week off and went to Maui. I had the time saved up, I was confident in the company, and I wanted to help reduce the overall vacation balance - I want us to make the numbers in the quarter and the tear. Plus, I wanted to try Maui in the summer as opposed to the winter. I still have time left over in case we want to do something around Christmas.

In contrast, I found out a week before the break that my last company had to have a shutdown the week of the 4th of July. And my guess is that caused a morale problem in their ranks. I'll go back to my mood last year at this time versus that I have right now - I'm mellow from a well deserved vacation and not steamed from being forced to take time off. I know both events play an important part in a company's well being - if the only way to make the quarter or fiscal year is a shutdown, then you have to do it. But I have to wonder if taking a voluntary week off is better for a company's morale (which can lead to an impact to its bottom line) over the savings of a shutdown.

I'm not making this up - one of the skins in Jedi Knight: Jedi Academy by Lucas Arts is an old boss of mine - Paul Crozier. If you know Paul, you would have to agree:

The fun part will be passing this on to his current direct reports and letting them speculate on whether he is on the Light or Dark Side.

In reading one of Stephen's older blog entries a steaming pile of p.o.o., he has the hackergotchis for the planet.opensolaris.org contributers.

I found this web tutorial: my head, so I tried to follow it to get my very own hackergotchis:

As you can see, I didn't get the drop shadow done correctly, so I decided not to do it.

Your results:
You are Spider-Man

Spider-Man 80% Hulk 70% Green Lantern 65% Iron Man 60% Catwoman 60% Superman 45% The Flash 45% Batman 45% Robin 42% Supergirl 35% Wonder Woman 25%

You are intelligent, witty, a bit geeky and have great power and responsibility.

Anyone who knows me right now would have picked Green Lantern. :>

I haven't blogged much about company culture and the shock I had coming to Sun. I guess I felt I was integrated. I was wrong. In many ways, I am still an outsider. Day in and out, I sit in my office at home, disconnected from the company hivemind. I sometimes get asked by people outside Sun what the plans are and I feel they get offended when I say I don't know. I'm not holding out, I literally do not know.

Sure, I could look at past actions and what I know about the business to make predictions. But at best, they are just as likely to be correct as an industry analyst. It is like Battlestar Galactica, I don't know the writers/producers, but I can predict where the story arcs are going. Can I be wrong? You betcha, which is why I watch the show anyway.

One of the things which is really exciting me about Sun right now are the UI improvements being introduced between builds 50-55 of nevada. They might have started creeping in before then, but I really started to notice along then. The difference is drastic from Solaris 9 and even Solaris 10. I'm going to write about what I think is going on, but please remember, I'm not representing Sun or even the development engineers.

We had the great browser war and before that we had the great windowing system war. Microsoft seemed to win both. Netscape Navigator won on technical merits, but the masses adopted Internet Explorer. As for the Windows versus X battle, I hated it when I was at NetApp and everyone pretty much had to use Windows and Outlook. We needed the extra internal test cycles on hosting Exchange databases, so we converted desktops to Win 2000 and then WinXP.

I think Sun lost the desire to prove it was the better desktop. Either all the engineers got old enough or turned their attention to other matters. Let the young Linux upstarts do it. And they tried, but they never won. Full disclosure, my home terminal is a WinXP box running PuTTY: a free telnet/ssh client and my backup is Fedora Core 4 Linux. My work terminal is a Sun Ray 1G connected to a Solaris Nevada b55 w2100z box. I went back to WinXP because of games and I'm very happy with PuTTY's interface to the Solaris and Linux servers I have at home.

But a funny thing happened, Apple and MAC OS X showed that you could have a BSD core and pretty UI and win people over from WinXP. The installation of new hardware, the interaction between software, etc, was so easy that non-technical people could do it. I know of a lot of diehard Solaris developers who flocked to the Mac Book Pro. And the remains of Netscape reared their head in the form of Firefox to show that the browser war wasn't over either. Suddenly, it became viable to have a non-Microsoft OS which seamlessly handled multimedia and the Internet.

Now I don't think anyone at Sun woke up and said, "You know, we could grab the desktop." I'm not even sure such secret plans exist. But just as Sun has been pouring Star Office, NFSv4, dtrace, and zfs contributions down the open source well, they have been drawing out the innovations made in Gnome, Thunderbird (the mail client which got me off of mutt by the way), Firefox, etc. And you know what, the Solaris desktop experience is pretty fun. Put a dead quiet Sun Ray thin client on someone's desk and it becomes really pleasant.

I've found that I don't have to turn from my Sun Ray 1G to my WinXP box to go look at something on the web. Or to get a crisp terminal session going. As a matter of fact, I want to retire that box to just being a game box. Or even better, I'll have it dual boot WinXP and some OpenSolaris variant. Or perhaps get Xen or vmware running on it to have it serve up a bunch of OSes. It really doesn't matter, as long as I can kick off a game whenever my son feels like pitting his skills against mine.

I was walking across the Santa Clara campus Thursday morning and it was still wet - either from rain, dew, or watering. As I walked down one pavement, I noticed a snail in front of me. I immediately thought I should help it get off to the grass. After all, it could easily be stepped on. But sometimes a snail is fragile, and definitly yucky, so I passed on. I then counted about 10 snails, all trying to get to the grass. I realized that snails are just like customers. And I decided to take some pictures to prove my point.

First of all, I went back to get that first snail, if I had helped him, I would have been trapped helping them all. That happened to me at NetApp when I started helping NGS on the customer mailing lists. They came to expect my help and if I didn't respond fast enough, they sought me out. They would bypass policy and procedure to call me up at home, at the pool, after I left NetApp, etc.

That picture didn't come out very well. Perhaps I was blocking the pain.

I also wanted to take a picture of two snails close together. One had to be larger than the other. The point was if two customers come to you at the same time, which do you help? At NetApp, the answer might be both. I got 4 escalations in a 12 hour period one weekend. But do you help the bigger one? Perhaps the smaller one is a reference account and no one told you how important they are to the company. You can't trust the sales guys to tell you how important the customer is to your bottom line. After 2 months of doing escalations, I had to yawn when I was told an account was worth $2 million. Yawn, I was supposed to drop a $20 million account to help a $2 million one?

Anyway, I almost threw out that picture as well. But it proves a point:

Sometimes you can't help a customer because you can't figure out what is going on. It could be that they've supplied you with bad information, your customer support organization has held onto the case for too long, or there is too much going on.

Another way to look at this is in this image, where at just the right magnification, the problem looks clear:

But as you get closer, as you peel the onion (a term I hate thanks to 4 months of startup hell), you find that things aren't really that clear:

Sometimes you might find a customer like this snail. They've been working with you so long to find a problem that they are going in circles: (If you look at the snail trail, you can see it has been circling the wagons.)

Sometimes you really need to avoid being pulled into this mess. No matter what you do, the customer will have no faith that you have helped them solve their problem. Also, you might fix the current problem and expose the next problem for some other department to fix. And then the customer comes back to you. The upside is if you really do fix their problem, your customer support and sales teams will remember that and come straight to you with the next tough one.

I want to close with a fascinating one:

This one is crystal clear and the customer is safe at home. If you help out, you might make the problem worse. (By the way, in reality, this snail was moving fast. I tried to get a movie, but my batteries were too low.)

My final thought was I should step on one and show you what happens when a customer gets really bad support. But I don't do that to snails, even to prove a good point. Perhaps that also says I wouldn't do that to a customer.

By the way, I understand NetApp now has a Level 3 customer support organization in place and is making head roads into reducing the strain on development engineers.

My builds are taking too long. At NetApp, I never worried about my PATH variable - it never changed pretty much the entire time I was there. /usr/local/bin always had the right compiler - remember NetApp does servers only. And they cross-compiled everything they could. Sun does clients and no real cross-compiling that I can see.

So my full build was taking 7-9 hours, which was 5-6 hours too long. I thought the nightly script ignored my PATH and built its own. I was wrong. So I took out some WAN links in my PATH and got down to about 4 hours, which is still longer than other people. Now, I could just copy a PATH statement from a colleague, but where is the fun in that I ask you?

I decided to play with perl to see if I could find out which directory components were remote. Here is my first hack at a script:

#!/usr/local/bin/perl

$envPath = `printenv PATH`;

chomp($envPath);

@paths = split(":", $envPath);

foreach $key (@paths) {
        `df -k $key`;
        print $key . "\n";
}

Okay, I know the output isn't very pretty, but lets try this puppy out:

[th199096@warlock ~]> perl ./dfpath.pl
/ws/onnv-tools/SUNWspro/SOS10/bin
/opt/teamware/bin
/ws/onnv-tools/teamware/bin
/opt/onbld/bin
/opt/onbld/bin/i386
/ws/onnv-tools/onbld/bin
/ws/onnv-tools/onbld/bin/i386
/usr/dt/bin
/usr/openwin/bin
/usr/ccs/bin
/opt/sfw/sbin
/opt/sfw/bin
/usr/sfw/sbin
/usr/sfw/bin
/usr/local/bin
df: (/usr/local/sbin) not a block device, directory or mounted resource
/usr/local/sbin
/usr/bin
/usr/sbin
/bin
/sbin
/usr/ucb
/usr/X11R6/bin
/export/home/th199096/bin
/usr/games

This is on my home system, which has a slightly different PATH than the build machines up at work. I'll move on to that system in just a minute. First I want to see what is going on with /usr/local/sbin.

[th199096@warlock ~]> df -k /usr/local/sbin
df: (/usr/local/sbin) not a block device, directory or mounted resource
[th199096@warlock ~]> df -k /usr/local
Filesystem            kbytes    used   avail capacity  Mounted on
/dev/dsk/c1t0d0s0    63168555 19722622 42814248    32%    /
[th199096@warlock ~]> cd /usr/local/sbin
/usr/local/sbin: No such file or directory.
[th199096@warlock ~]> ls -la /usr/local/
total 8
drwxr-xr-x   4 root     root         512 Jan 24 18:38 .
drwxr-xr-x  39 root     sys         1024 Jan 24 23:28 ..
drwxr-xr-x   2 root     bin          512 Jan 25 00:45 bin
drwxr-xr-x   3 root     sys         1024 Jan 24 18:40 etc

I like to pretend I'm going to share my .tcshrc across different platforms. Lets look at the relevant parts

# Set a reasonable route through the file system, paths etc.
set path=(/usr/local/{bin,sbin} /usr/{bin,sbin} /{bin,sbin})

if ( -x /usr/ucb ) then
        set path=($path /usr/ucb)
endif

if ( -x /usr/j2sdk1.4.207/bin ) then
        set path=(/usr/j2sdk1.4.207/bin $path)
endif

if ( -x /usr/sfw/bin ) then
        set path=(/usr/sfw/bin $path)
endif

if ( -x /usr/sfw/sbin ) then
        set path=(/usr/sfw/sbin $path)
endif

if ( -x /opt/sfw/bin ) then
        set path=(/opt/sfw/bin $path)
endif

if ( -x /opt/sfw/sbin ) then
        set path=(/opt/sfw/sbin $path)
endif

if ( -x /usr/ccs/bin ) then
        set path=(/usr/ccs/bin $path)
endif

if ( -x /usr/openwin/bin ) then
        set path=(/usr/openwin/bin $path)
endif

if ( -x /usr/dt/bin ) then
        set path=(/usr/dt/bin $path)
endif

if ( -x /ws/onnv-tools/onbld/bin/`uname -p` ) then
        set path=(/ws/onnv-tools/onbld/bin/`uname -p` $path)
endif

if ( -x /ws/onnv-tools/onbld/bin ) then
        set path=(/ws/onnv-tools/onbld/bin $path)
endif

if ( -x /opt/onbld/bin/`uname -p` ) then
        set path=(/opt/onbld/bin/`uname -p` $path)
endif

if ( -x /opt/onbld/bin ) then
        set path=(/opt/onbld/bin $path)
endif

if ( -x /ws/onnv-tools/teamware/bin ) then
        set path=(/ws/onnv-tools/teamware/bin $path)
endif

if ( -x /opt/teamware/bin ) then
        set path=(/opt/teamware/bin $path)
endif

if ( -x /ws/onnv-tools/SUNWspro/SOS10/bin ) then
        set path=(/ws/onnv-tools/SUNWspro/SOS10/bin $path)
endif

#if ( -x /opt/onnv-gate/public/bin ) then
#       set path=(/opt/onnv-gate/public/bin $path)
#endif

# Path for X
if ( -x /usr/X11R6/bin ) then
        set path=($path /usr/X11R6/bin)
else if ( -x /usr/X11/bin ) then
        set path=($path /usr/X11/bin)
endif

# Other possibilities to put in the path
if( -x ~/bin ) then
        set path=($path ~/bin)
endif

if( -x ~/scripts ) then
        set path=($path ~/scripts)
endif

if( -x /usr/share/bin ) then
        set path=($path /usr/share/bin)
endif

if( -x /usr/contrib/bin ) then
        set path=($path /usr/contrib/bin)
endif

if( -x /usr/local/scripts ) then
        set path=($path /usr/local/scripts)
endif

if( -x /usr/games ) then
        set path=($path /usr/games)
endif

if( -x /Applications ) then
        set path=($path /Applications ~/Applications)
endif

I need to make this part:

# Set a reasonable route through the file system, paths etc.
set path=(/usr/local/{bin,sbin} /usr/{bin,sbin} /{bin,sbin})

Be a little smarter. As a matter of fact, I need to stop the cut and paste action above. I'll try a little trick I picked up from sr. But before I do that, note that on the work build machines, I already have:

# Optimized for Solaris
set path=(/usr/local/{bin,sbin} /usr/{bin,sbin} /sbin)

I'm going to cut out some fluff and use sr's trick here:

# Make sure this is last
set path=(.)

set cpath = (/opt/onbld/bin /opt/onbld/bin/`/usr/bin/uname -p` \
                /usr/ccs/bin /opt/SUNWspro/bin /opt/teamware/bin \
                /usr/sfw/bin /opt/sfw/sbin /opt/sfw/bin /usr/dt/bin  \
                /usr/openwin/bin /usr/local/{bin,sbin} /usr/{bin,sbin} \
                /sbin /usr/ucb ~/bin)
foreach dir (${cpath})
        if ( -x ${dir} ) then
                set path=(${dir} ${path})
        endif
end

Which yields:

[th199096@warlock ~]> perl ~/dfpath.pl
/export/home/th199096/bin
/usr/ucb
/sbin
/usr/sbin
/usr/bin
/usr/local/bin
/usr/openwin/bin
/usr/dt/bin
/opt/sfw/bin
/opt/sfw/sbin
/usr/sfw/bin
/opt/teamware/bin
/usr/ccs/bin
/opt/onbld/bin/i386
/opt/onbld/bin
.

Which is backwards from what I want. I want to be able to write it correctly, so lets try:

# Make sure this is last
set path=(.)

set cpath = (/opt/onbld/bin /opt/onbld/bin/`/usr/bin/uname -p` \
                /usr/ccs/bin /opt/SUNWspro/bin /opt/teamware/bin \
                /usr/sfw/bin /opt/sfw/sbin /opt/sfw/bin /usr/dt/bin  \
                /usr/openwin/bin /usr/local/{bin,sbin} /usr/{bin,sbin} \
                /sbin /usr/ucb ~/bin)
set dpath=""
foreach dir (${cpath})
        set dpath=(${dir} ${dpath})
end

foreach dir (${dpath})
        if ( -x ${dir} ) then
                set path=(${dir} ${path})
        endif
end

unset cpath dir dpath

Which yields:

[th199096@warlock ~]> perl ~/dfpath.pl
/opt/onbld/bin
/opt/onbld/bin/i386
/usr/ccs/bin
/opt/teamware/bin
/usr/sfw/bin
/opt/sfw/sbin
/opt/sfw/bin
/usr/dt/bin
/usr/openwin/bin
/usr/local/bin
/usr/bin
/usr/sbin
/sbin
/usr/ucb
/export/home/th199096/bin
.

I've made my paths easier to maintain and I've also already removed some dead wood. Now, time to work on the perl script. If we tweak the df slightly, we get:

print `df -k $key`;

Which yields:

[th199096@warlock ~]> perl ~/dfpath.pl | head
Filesystem            kbytes    used   avail capacity  Mounted on
/dev/dsk/c1t0d0s0    63168555 19722640 42814230    32%    /
/opt/onbld/bin
Filesystem            kbytes    used   avail capacity  Mounted on
/dev/dsk/c1t0d0s0    63168555 19722640 42814230    32%    /
/opt/onbld/bin/i386
Filesystem            kbytes    used   avail capacity  Mounted on
/dev/dsk/c1t0d0s0    63168555 19722640 42814230    32%    /
/usr/ccs/bin
Filesystem            kbytes    used   avail capacity  Mounted on

Okay, we need to get rid of the first line:

        print `df -k $key | grep -v Filesystem`;

Which yields:

[th199096@warlock ~]> perl ~/dfpath.pl | head
/dev/dsk/c1t0d0s0    63168555 19722640 42814230    32%    /
/opt/onbld/bin
/dev/dsk/c1t0d0s0    63168555 19722640 42814230    32%    /
/opt/onbld/bin/i386
/dev/dsk/c1t0d0s0    63168555 19722640 42814230    32%    /
/usr/ccs/bin
/dev/dsk/c1t0d0s0    63168555 19722640 42814230    32%    /
/opt/teamware/bin
/dev/dsk/c1t0d0s0    63168555 19722640 42814230    32%    /
/usr/sfw/bin

But why use Perl at all if we continue down this pipe path? I thought I had to use Perl to split the ':' apart, but it turns out the $path keeps the list without the ':'. So the following tcsh script works:

#!/usr/bin/tcsh -f

foreach i ( $path )
        set j = `df -k ${i} | grep -v Filesystem | grep -v ^/dev`
        if ( "${j}" != "" ) then
                echo "${i} is not local"
        endif
end

This will tell me all filesystems in my path which are not local. Note I have switched to a work system.

[th199096@haulass ~]> ./dftab.tcsh
/opt/onbld/bin is not local
/opt/onbld/bin/i386 is not local
/opt/SUNWspro/bin is not local
/opt/teamware/bin is not local
/opt/sfw/bin is not local
/usr/local/bin is not local
/usr/local/sbin is not local
/home/th199096/bin is not local
. is not local

It seems weird that a NFS engineer is concerned with network traffic - i.e., why not just make the protocol faster. I'm actually looking to see which paths are WAN. A little tweak to the script yields:

[th199096@haulass ~]> ./dftab.tcsh
/opt/onbld/bin is not local
tools-i386:/export/tools.i386/onnv-tools/onbld 7150861 5940637 1138716 84% /opt/onbld
/opt/onbld/bin/i386 is not local
tools-i386:/export/tools.i386/onnv-tools/onbld 7150861 5940637 1138716 84% /opt/onbld
/opt/SUNWspro/bin is not local
tools-i386:/export/tools.i386/on10-tools/SUNWspro/SOS8/bin 7150861 5940637 1138716 84% /opt/SUNWspro/bin
/opt/teamware/bin is not local
tools-i386:/export/tools.i386/on10-tools/SUNWspro/SOS8 7150861 5940637 1138716 84% /opt/teamware
/opt/sfw/bin is not local
boulder-local1-27,boulder-local2-27,boulder-local1-26,boulder-local2-26,boulder-local1-25,boulder-local2-25:/export/opt.sfw.i386/opt/sfw 24932348 21650384 3032641 88% /opt/sfw
/usr/local/bin is not local
boulder-local1-27,boulder-local2-27,boulder-local1-26,boulder-local2-26,boulder-local1-25,boulder-local2-25:/export/local.i386 24932348 21650384 3032641 88% /usr/local
/usr/local/sbin is not local
boulder-local1-27,boulder-local2-27,boulder-local1-26,boulder-local2-26,boulder-local1-25,boulder-local2-25:/export/local.i386 24932348 21650384 3032641 88% /usr/local
/home/th199096/bin is not local
hera-home1.central:/export/home1/41/th199096 175540457 111466051 60563597 65% /home/th199096
. is not local
hera-home1.central:/export/home1/41/th199096 175540457 111466051 60563597 65% /home/th199096

So all of them check out as being local - i.e., in the sense they are not going out across the WAN. I had to login to tools-i386 to verify this for that machine. What I really need to do now is reduce the number of path components. I've already removed /bin because it is a symlink to /usr/bin. This is just a hack, but here I reduce my entries in my PATH for a build:

[th199096@haulass ~]> more concise.tcsh
set path=""

set cpath = (/opt/onbld/bin /opt/onbld/bin/`/usr/bin/uname -p` \
                /usr/ccs/bin /opt/SUNWspro/bin /opt/teamware/bin \
                /usr/sfw/bin /usr/{bin,sbin} /sbin )
set dpath=""
foreach dir (${cpath})
        set dpath=(${dir} ${dpath})
end

foreach dir (${dpath})
        if ( -x ${dir} ) then
                set path=(${dir} ${path})
        endif
end

unset cpath dir dpath
[th199096@haulass ~]> source concise.tcsh
[th199096@haulass ~]> printenv PATH
/opt/onbld/bin:/opt/onbld/bin/i386:/usr/ccs/bin:/opt/SUNWspro/bin:/opt/teamware/bin:/usr/sfw/bin:/usr/bin:/usr/sbin:/sbin
[th199096@haulass ~]> ./dftab.tcsh
/opt/onbld/bin is not local
tools-i386:/export/tools.i386/onnv-tools/onbld 7150861 5940637 1138716 84% /opt/onbld
/opt/onbld/bin/i386 is not local
tools-i386:/export/tools.i386/onnv-tools/onbld 7150861 5940637 1138716 84% /opt/onbld
/opt/SUNWspro/bin is not local
tools-i386:/export/tools.i386/on10-tools/SUNWspro/SOS8/bin 7150861 5940637 1138716 84% /opt/SUNWspro/bin
/opt/teamware/bin is not local
tools-i386:/export/tools.i386/on10-tools/SUNWspro/SOS8 7150861 5940637 1138716 84% /opt/teamware

I'm going to kick off a full build to see if that made a difference.

Here was my build time when I had WAN links in my PATH:

[th199096@haulass havok]> more speed.txt
[th199096@haulass havok]>
[1]    Done                          env -i `which nightly` -n ./clean.env
Time spent in user mode   (CPU seconds) : 16610.72s
Time spent in kernel mode (CPU seconds) : 4789.76s
Total time                              : 7:46:51.40s
CPU utilisation (percentage)            : 76.3%

And here was my build time when I took out the WAN links and rearranged the ordering. I.e., our CUE environment rewrote my PATH for me. :-< :

[th199096@haulass havok]>
[1]    Done                          env -i `which nightly` -n ./clean.env
Time spent in user mode   (CPU seconds) : 15498.05s
Time spent in kernel mode (CPU seconds) : 4439.35s
Total time                              : 4:02:55.12s
CPU utilisation (percentage)            : 136.7%

So changing the paths saved 3 hours and 45 minutes.

I'm off to play with the family, check back later for the real results.

Update The results were worse. My ssh session was disconnected, but it took 5 hours.

With WAN:                       7:46:40
Without WAN and stupid paths:   4:02:50
With hacks in here:             5:00:58

I can't swear the build didn't get longer, there was a major putback. I should try this on a Sparc. End Update

NetApp stock jumped a couple of dollars today and I sold off my remaining stock options. I went 5 years at NetApp with the stock underwater. But they were still Golden Handcuffs. I didn't make the millions I was slated to before Dot Bomb, but I finally got some reward for all of the hard work I poured into that company.

Sun has a policy, which I'm sure most companies also do, that you can't have a major investment in another company. Especially one which can be viewed as a competitor. They do have an escape clause for prior options, but I am glad to be rid of any appearance of conflict. It won't change how I view either company. Neither will it change how I interact with NetApp or their employees.

Now I just have to learn to watch Sun's stock and make sure I contribute to making it rise.

I've had 5 plastic and 2 cloth bracelets on my left wrist. I started out with the Armstrong foundation one, way back before most people knew what it was all about. Over time, I've added ones for soccer, baseball, team work (for a 8U baseball team), and the Sooners. I even branched one over to my right hand when I was in Maui.

I'm down in Austin for a group get together and one of the guys presented me with a lime green CoolThreads (tm) bracelet. When I get my hands on a camera, I'll post a picture. I know he might have done it to be nice, but I really view it as a great example of team building.

What it told me was that he was willing to give up something which was his to make a team mate happy. He might have been ribbing me as well, not many business people wear as much plastic as I sport. But when a team is working well, they tend to share, even with new guys.

I was once a manager (by the way, you can get that 1/2 of your brain back when you revert to an individual contributer) and got to distribute swag. A vendor had given us say 10 glasses. The only problem was that there were 11 of us. The solution was quite clear, almost as clear as the nice goblet. I did without.

I'm not trying to toot my own horn here. Yes, this was another example of team building. I never told anyone that I never got one of the goblets.

No, the real point of this revelation is the reaction one of the people had when I gave them the glass: "What, only one? What am I going to do with just one? Get me another." Now this person is still a friend of mine and can be very generous - the reaction was more a product of the times. This occurred right when Dot-Bomb exploded and morale was low.

Both of the objects were cheap, but the two actions illustrate how an individual's attitude and response to swag can effect company morale. The goblet incident was another straw in the camel's back which drove me away from management and the Bay Area. I stayed with that company, NetApp, due to a remarkable display of leadership by two directors. I'll talk about that some day.

The repercussions from the bracelet have obviously not had enough time to be analysed. Right now though, they confirm my choice to join not only a group of people I already knew, but a company whose vision I wanted to share.

I'm normally not very impressed by football analysts. I hate the phrases like General, chess match, etc. I believe football is a game of wits as much as physicality, but I don't like the battle/intellectual analogies.

I was however very impressed with Sean Salisbury's analysis after the Superbowl. Before I delve into that, this is the guy who basically does a wrestling match with John Clayton every week during the season. Sean acts tough, he was the jock and former pro quarterback. And John is very analytical, by the numbers, he plays a professor. But it is an act - sometimes they almost can't contain their laughter at the situation.

What Sean did after the Superbowl is something he tries to push in all of his game analysis - his love of the individual subsuming their ego for their teammates. Sean was a role player in his stint with the NFL - he had to be prepared to go in at any time, but the focus was not on him. I'm sure he had a lot of talent and made many sacrifices just for the chance to go in when the star got hurt.

While he brings out these types of points every week, what I thought was phenomenal was his taking advantage of what would later be dubbed a mediocre Superbowl and spreading his message to everyone watching the postgame analysis. Everyone wanted to hear him rag the officials, which he did do a little bit. But when he talked about Hines Ward, Jerome Bettis, and Matt Hasselbeck - he used them as fodder for his lessons of teamwork and humility.

With Hines, he talked about his commitment to the win rather than the stats. Sean was very blunt, there are wide receivers who if the team wins but they do not get 10 catches, 100 yards, some touchdowns, etc, they count it as a lose. He lectured that Hines would rather have 1 catch and a victory than the individual statistics. Another aspect that he pointed out was that Hines was willing to celebrate touchdowns with his teammates - he shared the glory and attention. Sean isn't very impressed with typical wide receiver self focus.

With Jerome, Sean focused on an aging star who did not complain when his role diminished. Jerome didn't whine about number of touches, other players getting the attention, etc. If you remember last year, Jerome was considered over the hill and Staley was the new guy. You might also remember that Jerome had a stellar year stepping up when Staley got injured. He didn't tear the team apart with "Get me the damn ball!", instead he kept his thoughts hidden from the public and led the team by example. Perhaps Jerome was steaming inside about his demotion, perhaps he was ready to quit, but the point is that at no time was this conducted in the media forum. Jerome may not have had the physical impact this year, but you can not ignore the emotional impact and leadership he provided to his younger team mates.

You don't have to buy into "Win one for the Bus," the point is that the Steelers as a team bought into it. They were able to deflect some of the media attention by focusing on Jerome.

With Matt, Sean focused on the loosing team may have had the best individual contributer. Sean didn't berate Matt's performance because of the loss. Nor did he raise Ben's (Roethlisberger) performance because of the win. Sean looked straight at the camera and claimed that Matt was the best athlete on the field that day. I.e., even if you face adversity, you have to give your best effort. At work today, the buzz was that if a couple of calls had been different, the Seahawks would have won the game. Can Matt afford to think that? I don't believe so.

It is evident that Sean loves the game of football and it hurts him to see individuals put themselves over their team mates. On occasion, he does take the opportunity to expose this type of behavior in his analysis segments. I thought it was daring to take it on during prime post game coverage. But it is just Sean practicing what he is preaching - instead of dwelling on the negatives of the officiating, the irregular play of the teams, etc, he found it in himself to point out how team work is supposed to behave. Even if it is now counter-culture in modern football.

I'm on the board of directors (BOD) for the local chapter of Little League and no, my son has never made the all-star team. We had tryouts for the remaining major league division spots today. I coach a 9U team, so I went out to help keep the lines moving. One of the things I'm supposed to do is keep my mouth shut - I can't help a kid in the middle of a skill test. Either I help them all before it starts or it is unfair to start in the middle.

A typical trial is for there to be two lines, say at short stop and first base. The SS fields a hit ball by a coach and throws it to first. I was struck by how many kids did not make sure they tagged the base after they made the catch. Sure, if the throw was right to them, they tagged the base. But otherwise, if they came off the base, they would throw the ball home without the tag.

I was happy that the kids at least came off the base to protect the ball from going out of bounds. But the lack of attention to detail drove me nuts. The saying isn't that practice makes perfect, but that perfect practice makes perfect.

I asked one of the coaches scouting the kids what was he looking for in the kids, athleticism? He said no and then he showed me his analog spreadsheet of players and skills.

Thinking back on the day, I have to compare how I interview candidates versus the tryouts. When I interview, I never concentrate on programming skills - I let others do that. What I focus on is the composure of the candidate if I ask uncomfortable questions. By uncomfortable, I don't mean inappropriate, I mean picking on detail in the resume and seeing if they squirm.

As an example, we had a candidate who had gotten his masters degree three months earlier. He had a gap between then and now. I asked him what he had been doing. He said he had been back to India. I said okay, that is one month, what did you do for the other two? He then said there had been family issues. I let it go for a bit, I told him that in the Valley, such a gap normally meant either a failed startup or a failed relationship with startup management.

After a bit, I asked him why should I hire him if he has family problems? He replied that he said issues and not problems. I told him I knew, but issues could be problems. I never asked what the issues were, that would have been inappropriate. Instead, I just pushed him, trying to fluster him. He didn't flinch and he kept a grin on his face. I told him what I had done and why, he didn't let me see if his was irate. We hired him.

Engineering departments can be highly demanding of your time. If you can't handle the stress, even if you can do the work, you can make your group dysfunctional. What this candidate showed me was that by doing things right in a practice round, the interview, I could have faith that he would also handle himself correctly in a real life crisis.

One of my colleagues talks about how stressful the interview with me was for him - I picked on something no one else ever talked about. But you know what, he was neck and neck with another candidate. His cool reaction in that interview got him the job and he was a great addition to that group.

Just as in a baseball tryout, you draft the candidate with the best raw potential. The only real difference is that you don't look at the parents and wonder if taking the kid is worth the aggravation of dealing with them.

Technorati Tags: Company Culture Culture

I've been told by managers that I don't self-promote myself very well. I've done things which weren't in my job description, but since I never went around and told everyone, I never got credit for it. I am of the mindset that you do your job right and people will notice.

With my last company, the people who did notice were individual contributors. I'd get Christmas cards from escalation engineers, I'd be the first name on the list of people to call when the problem looked like NFS or Unix, I'd have plenty of friends at company events, etc. But when the focal review process started, I had no documentation of what I had done and those individual contributers had not much to do with the review process.

With Sun, I've decided to blog about things which concern me, and right now that it quality and company culture. I've always been intrigued by both and my fascination might come across as if I never had either. :->

The way this all ties together, the blogging and the shameless self-promotion is that as I write about life at Sun and OpenSolaris, I am going to be recording what I have been doing correctly. For example, my article last night on Learning to deliver quality over content, I've obviously made such decisions in the past. Perhaps the outcome was not what I wanted and I certainly do not want my management to say, "Hey, wait a minute, we just hired a guy who has never delivered an on-time product with any quality." I also don't want a reader to thing, "Hey, wait a minute, he was last at NetApp, their stuff must stink." Neither is true.

But what I want to capture here are milestones I've reached which show insight as to how I'm improving. For example, there were many reasons why I wanted that article to get published and it would have been real easy to deliver a substandard product. I did not - now I've still got the original article to write and a request for a second one based on my response to the editor.

Where I'm improving is in leadership skills. I've been a professor and a manager - I do better at leading from the bottom than the top. At NetApp, when I was a developer for the filesystems group, I was also the lab manager for a group of 18. I was also half a continent away. It didn't take being able to go into a lab to string cables - it took being able to connect a developer with a need to a system administrator. I volunteered for the additional work because I could do it effectively.

At NetApp, I chased that elusive job title of Senior Engineer. I probably focused more on the title than the responsibilities. How many times have you gotten a resume to review and the candidate had 4 years of experience and 4 prior jobs all with the title of Senior Engineer? At one point, my friends went about calling themselves Junior System Administrators. The guys with the Senior label might have had more years in, but didn't know how to do simple Unix tasks.

But, they were really good at self-promotion - they were the first to proclaim what a great job they did on every little task. I think I've come full circle as to another reason why I don't toot my own horn. :->

Anyway, one of the other themes I am exploring is what constitutes leadership from below? How can I help promote others without trying to promote myself? Can I knock out any hidden agenda and truly help others? I got burnt out on helping students who didn't want to try. "Gimme an A!" was their rallying cry.

So I'm going to use my blog both to publicize what I do and to document quality, leadership, and culture. By forcing myself to organize my thoughts on these subjects, I'm going to improve the quality of that product known as myself.

Technorati Tags: Sun OpenSolaris Solaris NetApp Network Appliance Company Culture Culture

I've just faced my first major cultural hurdle at Sun - I basically chose to not ship a product because the quality was not sufficient for a customer. If you have been paying attention, then you know that after I left NetApp and before I started with Sun, I started working in OpenSolaris with an Opteron box called wont. It got its name because it won't work. I picked a combination of cutting edge hardware which resulted in a system which would boot as long as I didn't have the SATA drives connected. But I needed them to play with ZFS.

The reason I needed the system, even before I joined Sun, was because I had made a proposal for an article about building a 1TB NAS server out of commodity parts and OpenSolaris. The proposal got accepted and I started cranking on getting the system to work. Along the way, I joined Sun, and I've been learning to develop and build remotely (I live in Tulsa and my group is mainly in both Broomfield and Austin). One of the problems of ramping up off site is learning the things everyone takes for granted. I've got a great mentor for this, but if I can't express a question, it is hard for him to provide me an answer, especially if it has become ingrained in his daily habits.

So, I have the internal bits which are supposed to provide the fix for the SATA bug I have been pounding my head against. I finally got them built tonight, but I haven't tried to install them yet. For all I know, a crucial step was missed and I'll have a brick pretty soon. But that article is due, real soon now.

In grad school, and as a professor, I ate this type of pressure up and played the game. I got papers accepted to conferences and might have been editing the thing right up until 10 minutes before FedEx closed the night before. I always managed to hit the LPU (least publishable unit), but I didn't always deliver the best product. As a matter of fact, the best conference paper ever was one finished before the CFP went out. I wrote it myself and had my colleagues at other universities critic it - it got accepted to the AAAI conference.

At other companies, we called it a business decision to ship a partially implemented feature and accepted the risk of bugs being found by customers. I've actually heard someone relate that for a major project they delivered at another company, upper management made a decision to push quality testing from the QA organization to the customer.

The point is that either revenue rode on shipping on time or the developer was proud of their new code and wanted the world to see it. And if you as a company either have a set of customers who understand that risk or you are willing to dedicate the support resources, that is a decision you can make and live with the consequences. For example, I've seen my share of oil and gas modelers - they value computational speed over ease of management. If your box delivers 10x over the competition, well, they can handle a clunky interface or the occasional hung system. But if the competition starts catching up, then you better deliver stability.

What I like about Sun is the commitment to quality and the extended release model. The train leaves on time, but you only get coupled to it if you can show you are not going to derail it. The default is not that you are added to the train and pulled off if you cause problems. No, the default is that you better prove that you've been on a clone of the train, you've taken it on the same route, and if a crash occurred, well, you started over.

Another aspect of Sun's development model is that each functional group owns their quality. For example, when NFSv4 was being developed, it was done out of a shared code branch. The put back (to the main gate) was not a bunch of different minor deltas over a period of time. Instead, it was a group tested and cohesive whole chunk of love. As an individual, you had to prove that your changes did not break the group branch. As a group, they had to show that the sum of their changes did not break main.

So back to my quandary - I might be able to get a working system and crank out a decent coverage of getting a 1TB NAS box. At the very least, I could suck in all of the ZFS documentation, how-tos, white pages, etc, and simulate that I had indeed built the system and show it worked. I committed to my management that I was doing this and it would bring glory to the group - okay, to me. It might not get me a raise this year, but it would lay the groundwork for one next year.

As the title of this entry suggests, I didn't do either of the above deceptions. It was only after I contacted the editor and explained my dilemma that I realized I had been caught in a new cultural shift. (And if you think it is easy for me to admit failure, well, you don't know me very well.) I've been stressing out over this big time - I've pushed off working on the text. I could have gotten all of the accompanying material processed and ready for the meat, even though I wasn't sure I would get to the meat. Instead, I've been focusing on getting my remote build environment up and running. I knew I was stalling, I rationalized I was an old pro at delivering content at the last minute. I didn't even get mad when my wife asked that I go to a birthday party this afternoon. She knew I had a deadline, but I knew she needed someone to help drive for her on the long route.

I don't feel like I've let my manager down or jeopardized my job. I strongly feel that I demonstrated a new skill - one which was hard for me not to learn, but to act on. In the past, I've relied on passing the decision of quality onto either my manager or the QA group. In a top down organization, that is easy to let slip away. But Sun prides itself on being bottom up (another cultural nuance for me to absorb) and this is also causing me to learn a new set of behavior.

One of the ways I've always stood out is being able to get the job done on time - where done means acceptable. I've always produced code which passes all of the unit testing I can devise, but management does not always understand that is not sufficient. The reason you have an empowered QA organization is that a developer is too close to the code and makes assumptions that will bite someone else using the code. Have a project which has to be done in a month, no problem, let me drop this other deliverable and get cranking on this interrupt.

I've always had a hard time in telling someone "No." At NetApp, we had a term for this, it was the SuperHero effect. The cultural thing to do was to drop whatever you were doing and give all of your effort to the problem at hand. At the end, I'd wear a tshirt with a Superman logo whenever I went up Sunnyvale to mingle with my team. I had it bad, I was on all of the major customer support mailing lists and they all knew my name and numbers. Different managers would have suggestions of waiting X minutes before replying - it didn't work. I had to help and I had to deliver product all of the time.

At various times, I would grab control of quality and say "No." But if I wasn't proactive, my power would slip away. After all, in a top down environment, you are ultimately assigned responsibility. I'm finding out how rewarding a bottom up culture can be - I've decided I own both my results and their quality. It can't be assigned away from me, even inadvertently.

I feel liberated that I didn't compromise quality on this deliverable. Okay, the article actually failed my unit testing - whatever. I feel I can show my new management that I understand Sun's commitment to quality and appreciate the cultural tools which are in place and in turn facilitate me being able to contribute to the overall quality. Remember, quality isn't just something you put in a deliverable and tangible product. If it doesn't permeate every aspect of your job, then you deliver an inferior product.

Technorati Tags: Sun OpenSolaris Solaris NetApp Network Appliance Company Culture Culture

A friend sent me a script he wanted tested on an OpenSolaris system. I ran it and right away it barfed on something. We've sent some emails back and forth.

I finally offered him access onto my home network. I expect he will decline. But I did so with the trust that he would not abuse the system. I've had students who had to be locked into chroot hells during exams and they still installed viruses, etc.

Security starts by establishing trust. At NetApp, all the engineers had access to root privileges. It was scary and rife with potential for abuse. The most malicious thing I knew of in 6 years was the user, we never found out who, that changed the color of the main internal web page every day. I.e., there was a trust established that the engineers would not really screw up things.

But as a company grows, it is hard to maintain that trust. Eventually there are enough people that you don't know and enough people who inadvertantly do stupid things. We went in, took away an evil setuid program, and gave them access to sudo. This act allowed IT to log all uses and eventually to start locking down critical servers.

Once I left IT at NetApp, I didn't want to know privileged account passwords any more. I wanted them to change them, because with great power comes great responsibility. I.e., if I just happened to be logged in at 2AM Central, I could solve an issue for them instead of them driving into the site. If that caused more problems, well, I didn't want to hear about it on the engineering mailing lists.

At Sun, either you know the root password or you don't have the privilege. sudo is on the companion software CD, but I've yet to find it on a production machine. Not that I'm trying, but it can be easy to type a command in the wrong window.

Another difference between the two companies is that NetApp has a single signon server for most IT apps and Sun does not. That translates to you entering your password a lot more at Sun.

NetApp is moving closer to removing casual root privileges and Sun is moving closer to a single signon.

Back to my friend - I'm happy to extend my trust because I know he doesn't want the responsibility of introducing a problem. I know he would feel bound to help solve any problems he caused. And I know he would expect the same of me.

And he's logged in right now. :->

I'll have to ask him how to lock down sudo rights on the machines he is on. That is a big :->.

Technorati Tags: Sun OpenSolaris Solaris Company Culture Culture