Kool Aid Served Daily

Okay, I added a StatCounter to my blog. I've been seeing emails go by about doing it, I had some down time, so I learned how to do it. Basically, just go to StatCounter to get set up with the minimal counter. It will give you some HTML to add to your web pages.

I added it to the '_day' template. I got there via 'login' -> 'Preferences' -> 'Templates' -> '_day'. I added the link there, it isn't where I'd like it, but that is a nit. Also, I decided to add the javascript code to the 'Weblog' template to keep it up in the 'head'.

I figured this out via the User Guide link at the bottom of the page. Sounds easy, but I was Google crazy for a bit.

When tracking, you do get to see the IP (but you need my account name and password to get in) but that is it. Anyway, it also reports how the entry was linked to. I found an entry which came from http://www.google.com/ig. Okay, I clicked on it and evidently Google allows you to have a home page - basically it is the search engine with your own content. Now, I still can't tell who was on this page either - you have to login.

Well, gmail is my primary web email (i.e., for when I can't ssh past my firewall), so I logged in. I added weather for Tulsa, stock info for NTAP and SUNW. By the way, I guess I'll stop using Yahoo! for my finances now. :->

It also allowed you to add your own content - I thought I would create a section with my favorite humor:

• Dilbert


• User Friendly


• Get Fuzzy


• News of the Weird


• BOFH

Okay, I mainly listed them to see if I could create a list in a blog.

But, I couldn't do that. I could however search for content, so I searched for NFSv4. One of the returned items was my blog! Okay, I added that and it was really, really neat. Until I get a screenshot, it basically showed the title of the Blog as a section header and had the last 3 blog entries as linked news items. Sweet.

Okay, here is the image, which some stuff zapped for privacy:

Note that you can drag and drop the sections to different locations.

Technorati Tags: Google

I have a desktop which tunnels into work via a vpn. I'm trying to get a clean build of Nevada across the WAN. My builds were taking a day, when they should take 2 hours. As I had a really messed-up partition scheme from the last install, I decided to fix that and also just reset the image. I made sure I had all of the stuff I needed stored away on an USB hard drive.

I love everything USB - it really reminds me of the days of IPC and IPX towers. I'm convinced my son will view the CPU as just another USB device. Anyway, I've had problems with the drive with different versions of Nevada builds, so the underlying filesystem is FAT. This fact is crucial to everything which follows.

The system comes back up, I add users and I get my homedir restored. When I saved my homedir, I just did a gzipped tarball to avoid having bunches of small files floating on the HD. Everything there came up green. I also reinstalled the VPN software and was connected in no time. The previous time, I hadn't backed up my configuration info, which cost me several hours waiting for the servers to propagate the new keys.

I installed my local copy (again a tarball) of my workspace and even got it synced up with a bringover. OKay, I went to the gate README and found out how I wanted to configure my automounts. Sun really uses the automounter and I'm a novice at it. So, I copied over my individual files from the HD and restarted autofs:

svcadm restart autofs

Now, I didn't bring over auto_home or auto_master. I could get into systems via /net, but no luck for the entries in /ws. In retrospect, I've given you all of the clues necessary to solve this problem. But, that would mean zipping ahead two hours of learning in my life. Nope, I'm going to reproduce the issue and make you suffer through it. Really. Of course, the meat of the reproduction is helping you debug automounter issues.

Okay, to reproduce this, I'm going to setup some shares on sandman:

# share -F nfs -o rw -d "home dirs" /export/home
# share -F nfs -o ro -d "etc" /etc
# share -F nfs -o ro -d "logs" /var

And, since ultralord is our victim, lets see how it looks at first:

ultralord:> showmount -e sandman
export list for sandman:
/export/home (everyone)
/etc         (everyone)
/var         (everyone)
ultralord:> showmount -e ultralord
showmount: ultralord: RPC: Program not registered
ultralord:> uname -a
SunOS ultralord 5.10 Generic sun4u sparc SUNW,Ultra-5_10

Hmm, running stock Solaris 10, but that will not matter. Okay, lets make some automount maps:

ultralord:> more auto_master
#
# Copyright 2003 Sun Microsystems, Inc.  All rights reserved.
# Use is subject to license terms.
#
# ident "@(#)auto_master        1.8     03/04/28 SMI"
#
# Master map for automounter
#
+auto_master
/net            -hosts          -nosuid,nobrowse
/home           auto_home       -nobrowse
+auto_ws
ultralord:> more auto_ws
/ws_nc  auto_junk
/ws     -vers=3,fstype=cachefs,cachedir=/cache,backfstpe=nfs,snr auto_junk
ultralord:> more auto_junk
sandman         -noquota,retry=5,timeo=600,nosuid \
                /home   sandman:/export/home \
                /etc    sandman:/etc \
                /var    sandman:/var

ultralord       -noquota,retry=5,timeo=600,nosuid \
                /home   ultralord:/export/home \
                /etc    ultralord:/etc \
                /var    ultralord:/var
ultralord:> ls -la /ws
/ws: No such file or directory

Note that /ws does not yet exist and that we are automounting from ultralord, even though it is not exporting anything via NFS. We restart autofs:

# svcadm restart autofs
# ls -la /ws
/ws: No such file or directory

It should have created the directory. Okay, anything in the normal log files?

# grep auto /var/adm/messages /var/log/syslog
# ps -ef | grep automount
    root  3929     1   0 12:07:33 ?           0:00 /usr/lib/autofs/automountd
    root  3936  3931   0 12:09:22 pts/2       0:00 grep automount
#

Not even a notice that the service had been restarted. Okay, how do we make automount be chatty? The man page states that you use '-v'. Hmm, I've checked and checked, I can't see a way to pass parameters in via svcadm. The trick is that you need to edit /etc/default/autofs:

# Verbose mode.  Notifies of autofs mounts, unmounts, or other
# non-essential events.  This equivalent to the "-v" argument.
#AUTOMOUNT_VERBOSE=FALSE

# Verbose.  Log status messagess to the console.
# This is equivalent to the "-v" argument.
#AUTOMOUNTD_VERBOSE=FALSE

# Turn on or off browsing for all autofs mount points.
# This is equivalent to the "-n" argument.
#AUTOMOUNTD_NOBROWSE=FALSE

# Trace.  Expand each RPC call and display it on standard output.
# This is equivalent to the "-T" argument.
#AUTOMOUNTD_TRACE=0

Time to flip on AUTOMOUNT_VERBOSE and AUTOMOUNTD_VERBOSE. The others do not look right. And again we restart autofs.

 
# svcadm restart autofs
# grep auto /var/adm/messages /var/log/syslog
#

So, what is the deal? Well it turns out that the logs are stored in /var/svc/log. And we want system-filesystem-autofs:default.log:

# tail system-filesystem-autofs:default.log
[ Jan 25 12:13:10 Method "stop" exited with status 0 ]
[ Jan 25 12:13:10 Executing start method ("/lib/svc/method/svc-autofs start") ]
automount: /net mounted
automount: /home mounted
automount: /ws_nc mounted
automount: /ws mounted
automount: no unmounts
[ Jan 25 12:13:10 Method "start" exited with status 0 ]
[ Jan 25 12:13:10 Rereading configuration. ]
[ Jan 25 12:13:10 No 'refresh' method defined.  Treating as :true. ]
# ls -la /ws
total 3
dr-xr-xr-x   1 root     root           1 Jan 25 12:13 .
drwxr-xr-x  28 root     root         512 Jan 25 12:07 ..
# ls -la /ws_nc
total 3
dr-xr-xr-x   1 root     root           1 Jan 25 12:13 .
drwxr-xr-x  28 root     root         512 Jan 25 12:07 ..

Hey, something created the mount points for me. Lets try and see something:

# cd /ws
# ls -la
total 3
dr-xr-xr-x   1 root     root           1 Jan 25 12:13 .
drwxr-xr-x  28 root     root         512 Jan 25 12:07 ..
# cd etc
etc: does not exist
# cd sandman
sandman: does not exist

Oh well, at least it got logged, right? No.

# tail system-filesystem-autofs:default.log
[ Jan 25 12:13:10 Method "stop" exited with status 0 ]
[ Jan 25 12:13:10 Executing start method ("/lib/svc/method/svc-autofs start") ] automount: /net mounted
automount: /home mounted
automount: /ws_nc mounted
automount: /ws mounted
automount: no unmounts
[ Jan 25 12:13:10 Method "start" exited with status 0 ]
[ Jan 25 12:13:10 Rereading configuration. ]
[ Jan 25 12:13:10 No 'refresh' method defined.  Treating as :true. ]

Okay, I'll cut out an hour of simulated time and revist /etc/default/autofs:

# Trace.  Expand each RPC call and display it on standard output.
# This is equivalent to the "-T" argument.
#AUTOMOUNTD_TRACE=0
AUTOMOUNTD_TRACE=10

At this point, I do not care if it is a boolean or I'm increasing the verbosity level.

# tail system-filesystem-autofs:default.log
automount: /home mounted
automount: /ws_nc mounted
automount: /ws mounted
automount: no unmounts
[ Jan 25 12:13:10 Method "start" exited with status 0 ]
[ Jan 25 12:13:10 Rereading configuration. ]
[ Jan 25 12:13:10 No 'refresh' method defined.  Treating as :true. ]
[ Jan 25 12:19:29 Stopping because service restarting. ]
[ Jan 25 12:19:29 Executing stop method ("/lib/svc/method/svc-autofs stop 134") ]
umount: /ws busy
# tail system-filesystem-autofs:default.log
[ Jan 25 12:19:34 Method "stop" exited with status 0 ]
[ Jan 25 12:19:35 Executing start method ("/lib/svc/method/svc-autofs start") ]
t1        maxthreads: 64 rlim_max: 0 rlim_cur: 512
automount: /net mounted
automount: /home mounted
automount: /ws_nc mounted
automount: no unmounts
[ Jan 25 12:19:35 Method "start" exited with status 0 ]
[ Jan 25 12:19:35 Rereading configuration. ]
[ Jan 25 12:19:35 No 'refresh' method defined.  Treating as :true. ]

Ahem, needed to get out of /ws in my other window. Okay, lets see what happens.

# cd /ws
# ls -la
total 3
dr-xr-xr-x   1 root     root           1 Jan 25 12:13 .
drwxr-xr-x  28 root     root         512 Jan 25 12:07 ..

Yields:

t1      READDIR REQUEST : -vers=3,fstype=cachefs,cachedir=/cache,backfstpe=nfs,snr @ 100000
t1      map -vers=3,fstype=cachefs,cachedir=/cache,backfstpe=nfs,snr not found, adding...
t1      getmapkeys_files -vers=3,fstype=cachefs,cachedir=/cache,backfstpe=nfs,snr

t1      READDIR REPLY   : status=2

Sweet, we are seeing something. More!

# cd sandman
sandman: does not exist

Yields:

t1      LOOKUP REQUEST: Wed Jan 25 12:21:47 2006
t1        name=sandman[] map=-vers=3,fstype=cachefs,cachedir=/cache,backfstpe=nfs,snr opts= path=/ws direct=0
t1        do_lookup1: action=2 wildcard=FALSE error=2
t1      LOOKUP REPLY    : status=2
t2      -vers=3,fstype=cachefs,cachedir=/cache,backfstpe=nfs,snr freeing cache

Hmm, how about on the loopback device? (Note I switched to /ws_nc)

# cd ultralord
ultralord: No such file or directory.

which does:

t2      auto_junk cache still valid
t1      LOOKUP REQUEST: Wed Jan 25 12:27:30 2006
t1        name=ultralord[] map=auto_junk opts= path=/ws_nc direct=0
t1        PUSH /etc/auto_junk
t1              Executable map: map=/etc/auto_junk key=ultralord
t1              read_execout: forking .....
t1              read_execout: map=/etc/auto_junk key=ultralord line=
t1        POP /etc/auto_junk
t1        do_lookup1: action=2 wildcard=FALSE error=2
t1      LOOKUP REPLY    : status=2
t4      LOOKUP REQUEST: Wed Jan 25 12:27:30 2006
t4        name=ultralord[] map=auto_junk opts= path=/ws_nc direct=0
t4        PUSH /etc/auto_junk
t4              Executable map: map=/etc/auto_junk key=ultralord
t4              read_execout: forking .....
t4              read_execout: map=/etc/auto_junk key=ultralord line=
t4        POP /etc/auto_junk
t4        do_lookup1: action=2 wildcard=FALSE error=2
t4      LOOKUP REPLY    : status=2

And for grins, what do with see in /net?:

# cd /net
# cd sandman
# ls -la
total 5
dr-xr-xr-x   4 root     root           4 Jan 25 12:29 .
dr-xr-xr-x   2 root     root           2 Jan 25 12:19 ..
dr-xr-xr-x   1 root     root           1 Jan 25 12:29 etc
dr-xr-xr-x   1 root     root           1 Jan 25 12:29 export
dr-xr-xr-x   1 root     root           1 Jan 25 12:29 var

gives

t1      LOOKUP REQUEST: Wed Jan 25 12:29:30 2006
t1        name=sandman[] map=-hosts opts=nosuid,nobrowse path=/net direct=0
t1        mapline: -hosts
t1        do_mapent_hosts: host sandman
t1              ping: sandman timeout=15 request vers=4 min=2
t1        pingnfs: Trying ping via "circuit_v"
t1              pingnfs OK: nfs version=4
t1        do_mapent_hosts: host sandman OK
t1
        do_mapent_hosts:(return)
t1        (nfs,nfs)      /sandman/etc -nosuid,nobrowse
                sandman:/etc
t1              me->map_fsw=
t1               mntlevel=-1    modify=FALSE    faked=FALSE err=0
t1        (nfs,nfs)      /sandman/var -nosuid,nobrowse
                sandman:/var
t1              me->map_fsw=
t1               mntlevel=-1    modify=FALSE    faked=FALSE err=0
t1        (nfs,nfs)      /sandman/export/home -nosuid,nobrowse
                sandman:/export/home
t1              me->map_fsw=
t1               mntlevel=-1    modify=FALSE    faked=FALSE err=0
t1
        hierarchical_sort:
        (,  ,)
                (etc, -1, nosuid,nobrowse)
                (var, -1, nosuid,nobrowse)
                (export,  ,)
                        (home, -1, nosuid,nobrowse)
t1        No rootnode, travpath=
t1        node mntpnt=/etc       travpath=/etc
t1        node mntpnt=/var       travpath=/var
t1
        set_and_fake_mapent_mntlevel
        (,  ,)
                (etc, 1, nosuid,nobrowse)
                (var, 1, nosuid,nobrowse)
                (export, 1, nosuid,nobrowse,direct)
                        (home, -1, nosuid,nobrowse)
t1
        modify_mapents:
t1        (autofs,autofs)        /sandman/etc -nosuid,nobrowse,direct
                :-hosts
t1              me->map_fsw=
t1               mntlevel=1     modify=TRUE     faked=FALSE err=0
t1        (autofs,autofs)        /sandman/var -nosuid,nobrowse,direct
                :-hosts
t1              me->map_fsw=
t1               mntlevel=1     modify=TRUE     faked=FALSE err=0
t1        (autofs,autofs)        /sandman/export -nosuid,nobrowse,direct
                :-hosts
t1              me->map_fsw=
t1               mntlevel=1     modify=FALSE    faked=TRUE err=0
t1        do_lookup1: action=2 wildcard=FALSE error=0
t1      LOOKUP REPLY    : status=0
t5      MOUNT REQUEST:   Wed Jan 25 12:29:30 2006
t5        name=sandman[] map=-hosts opts=nosuid,nobrowse path=/net direct=0
t5        mapline: -hosts
t5        do_mapent_hosts: host sandman
t5        do_mapent_hosts: host sandman OK
t5
        do_mapent_hosts:(return)
t5        (nfs,nfs)      /sandman/etc -nosuid,nobrowse
                sandman:/etc
t5              me->map_fsw=
t5               mntlevel=-1    modify=FALSE    faked=FALSE err=0
t5        (nfs,nfs)      /sandman/var -nosuid,nobrowse
                sandman:/var
t5              me->map_fsw=
t5               mntlevel=-1    modify=FALSE    faked=FALSE err=0
t5        (nfs,nfs)      /sandman/export/home -nosuid,nobrowse
                sandman:/export/home
t5              me->map_fsw=
t5               mntlevel=-1    modify=FALSE    faked=FALSE err=0
t5
        hierarchical_sort:
        (,  ,)
                (etc, -1, nosuid,nobrowse)
                (var, -1, nosuid,nobrowse)
                (export,  ,)
                        (home, -1, nosuid,nobrowse)
t5        No rootnode, travpath=
t5        node mntpnt=/etc       travpath=/etc
t5        node mntpnt=/var       travpath=/var
t5
        set_and_fake_mapent_mntlevel
        (,  ,)
                (etc, 1, nosuid,nobrowse)
                (var, 1, nosuid,nobrowse)
                (export, 1, nosuid,nobrowse,direct)
                        (home, -1, nosuid,nobrowse)
t5
        modify_mapents:
t5        (autofs,autofs)        /sandman/etc -nosuid,nobrowse,direct
                :-hosts
t5              me->map_fsw=
t5               mntlevel=1     modify=TRUE     faked=FALSE err=0
t5        (autofs,autofs)        /sandman/var -nosuid,nobrowse,direct
                :-hosts
t5              me->map_fsw=
t5               mntlevel=1     modify=TRUE     faked=FALSE err=0
t5        (autofs,autofs)        /sandman/export -nosuid,nobrowse,direct
                :-hosts
t5              me->map_fsw=
t5               mntlevel=1     modify=FALSE    faked=TRUE err=0
t5        do_mount1:
t5        (autofs,autofs)       /net/sandman/etc
                -nosuid,nobrowse,direct
                :-hosts penalty=0
t5        (autofs,autofs)       /net/sandman/var
                -nosuid,nobrowse,direct
                :-hosts penalty=0
t5        (autofs,autofs)       /net/sandman/export
                -nosuid,nobrowse,direct
                :-hosts penalty=0
t5        root=/net/sandman      next_subdir=/etc
t5        mount_autofs -hosts on /net/sandman/etc
t5      rel_mntpnt = ./etc
t5        root=/net/sandman      next_subdir=/var
t5        mount_autofs -hosts on /net/sandman/var
t5      rel_mntpnt = ./var
t5        root=/net/sandman      next_subdir=/export
t5        mount_autofs -hosts on /net/sandman/export
t5      rel_mntpnt = ./export
t5      MOUNT REPLY    : status=0, AUTOFS_ACTION
t5      freeing action list
t2      auto_junk cache still valid

So, automount is working, but really, it isn't. It was about this time last night that I finally figured out what was going on. If you look at the debug output, you can clearly see Executable map:. When I copied the map files back to the USB drive, it was to a FAT filesystem - and the 'x' bits were set.

Lets see if we can clear this up with these commands:

# cd
# ls -la /etc/auto_junk /etc/auto_ws
-rwxr-xr-x   1 root     root         244 Jan 25 12:02 /etc/auto_junk
-rwxr-xr-x   1 root     root          78 Jan 25 12:04 /etc/auto_ws
# chmod -x  /etc/auto_junk /etc/auto_ws
# ls -la /etc/auto_junk /etc/auto_ws
-rw-r--r--   1 root     root         244 Jan 25 12:02 /etc/auto_junk
-rw-r--r--   1 root     root          78 Jan 25 12:04 /etc/auto_ws
# svcadm restart autofs
# ls -la /ws_nc/sandman
total 5
dr-xr-xr-x   4 root     root           4 Jan 25 12:33 .
dr-xr-xr-x   2 root     root           2 Jan 25 12:33 ..
dr-xr-xr-x   1 root     root           1 Jan 25 12:33 etc
dr-xr-xr-x   1 root     root           1 Jan 25 12:33 home
dr-xr-xr-x   1 root     root           1 Jan 25 12:33 var

Do not forget to clean up your /etc/default/autofs file.

By the way, my Nevada build is under 2 hours now. It is still messed up, I suspect because I am on a clean install of a Nevada system.