Kool Aid Served Daily

I've had wont off for some time, it gets hot in my office. Anyway, I started it back up and I can't ssh into the box. I can't restart ssh:

# svcadm restart ssh
# ps -ef | grep ssh
    root   250   232   0 10:45:28 console     0:00 grep ssh
# cd /var/svc/log
# tail network-ssh:default.log
[ Apr  2 10:42:55 Stopping because service disabled. ]
[ Apr  2 10:42:55 Executing stop method (:kill) ]
[ Apr  2 10:51:37 Executing start method ("/lib/svc/method/sshd start") ]
[ Apr  2 10:51:44 Method "start" exited with status 0 ]
[ Apr  3 12:46:02 Stopping because service disabled. ]
[ Apr  3 12:46:02 Executing stop method (:kill) ]
[ Apr  3 12:55:53 Executing start method ("/lib/svc/method/sshd start") ]
[ Apr  3 12:56:00 Method "start" exited with status 0 ]
[ Apr 13 22:37:28 Stopping because service disabled. ]
[ Apr 13 22:37:28 Executing stop method (:kill) ]

Thinking I wasn't doing the right ps options:

# ps -ef
     UID   PID  PPID   C    STIME TTY         TIME CMD
    root     0     0   0 10:34:00 ?           0:34 sched
    root     1     0   0 10:34:03 ?           0:00 /sbin/init
    root     2     0   0 10:34:03 ?           0:00 pageout
    root     3     0   0 10:34:03 ?           0:00 fsflush
    root     7     1   0 10:34:05 ?           0:03 /lib/svc/bin/svc.startd
    root     9     1   0 10:34:06 ?           0:05 /lib/svc/bin/svc.configd
    root   197     1   0 10:34:21 ?           0:00 /usr/lib/utmpd
    root   110     1   0 10:34:17 ?           0:00 /usr/lib/power/powerd
    root    84     1   0 10:34:17 ?           0:00 /usr/lib/sysevent/syseventd
    root   111     1   0 10:34:17 ?           0:00 /usr/lib/picl/picld
    root   192     7   0 10:34:20 console     0:00 -sh
    root    99     1   0 10:34:17 ?           0:00 /usr/sbin/nscd
    root   253   232   0 10:48:15 console     0:00 ps -ef
    root   232   192   0 10:37:08 console     0:00 tcsh
  daemon   115     1   0 10:34:18 ?           0:00 /usr/lib/crypto/kcfd

Not much at all is running!

Okay, I rebooted to see if anything came up on the console. (Note, I tend to reboot when I don't understand something. It seems to drive other people crazy.) And I see:

cannot mount '/zoo': directory is not empty
use legacy mountpoint to allow this behavior, or use the -O flag
svc:/system/filesystem/local:default: WARNING: /usr/sbin/zfs mount -a failed: exit status 1
Apr 21 10:34:20 svc.startd[7]: svc:/system/filesystem/local:default: Method "/lib/svc/method/fs-local" failed with exit status 95.

Right away, I should check /var/svc/log/system-filesystem-local:default.log, but I'll take my sweet time getting there. :>

First, I find that svcs -x will tell me info:

# svcs -x
svc:/system/filesystem/local:default (local file system mounts)
 State: maintenance since Fri Apr 21 10:34:20 2006
Reason: Start method exited with $SMF_EXIT_ERR_FATAL.
   See: http://sun.com/msg/SMF-8000-KS
   See: /var/svc/log/system-filesystem-local:default.log
Impact: 28 dependent services are not running.  (Use -v for list.)

svc:/network/rpc/gss:default (Generic Security Service)
 State: uninitialized since Fri Apr 21 10:34:07 2006
Reason: Restarter svc:/network/inetd:default is not running.
   See: http://sun.com/msg/SMF-8000-5H
   See: gssd(1M)
Impact: 14 dependent services are not running.  (Use -v for list.)

svc:/network/rpc/smserver:default (removable media management)
 State: uninitialized since Fri Apr 21 10:34:08 2006
Reason: Restarter svc:/network/inetd:default is not running.
   See: http://sun.com/msg/SMF-8000-5H
   See: rpc.smserverd(1M)
Impact: 3 dependent services are not running.  (Use -v for list.)

svc:/application/print/server:default (LP print server)
 State: disabled since Fri Apr 21 10:34:07 2006
Reason: Disabled by an administrator.
   See: http://sun.com/msg/SMF-8000-05
   See: lpsched(1M)
Impact: 1 dependent service is not running.  (Use -v for list.)

If you do a 'svcs -xv' you can get more detailed info. That lead me to looking at http://sun.com/msg/SMF-8000-5H, which in turn had me restarting services. That did not work. So I then went to the link to http://sun.com/msg/SMF-8000-KS. This lead me to look at the log files.

# tail system-filesystem-local:default.log
[ Apr 13 22:37:39 Executing stop method (null) ]
[ Apr 19 11:21:21 Executing start method ("/lib/svc/method/fs-local") ]
WARNING: /usr/sbin/zfs mount -a failed: exit status 1
[ Apr 19 11:21:22 Method "start" exited with status 95 ]
[ Apr 21 10:14:41 Executing start method ("/lib/svc/method/fs-local") ]
WARNING: /usr/sbin/zfs mount -a failed: exit status 1
[ Apr 21 10:14:42 Method "start" exited with status 95 ]
[ Apr 21 10:34:19 Executing start method ("/lib/svc/method/fs-local") ]
WARNING: /usr/sbin/zfs mount -a failed: exit status 1
[ Apr 21 10:34:20 Method "start" exited with status 95 ]

Sweet, what happens if I try '/usr/sbin/zfs mount -a' manually?

# /usr/sbin/zfs mount -a
cannot mount '/zoo': directory is not empty
use legacy mountpoint to allow this behavior, or use the -O flag
# ls -la /zoo
total 6
drwxr-xr-x   3 root     root         512 Apr  6 21:46 .
drwxr-xr-x  42 root     root        1024 Mar 29 12:33 ..
dr-xr-xr-x   2 root     root         512 Apr  6 21:46 isos

Now, did I manage to, sometime in the recent past, create a directory directly in /zoo? Or is there an 'isos' filesystem?

# zfs list
NAME                   USED  AVAIL  REFER  MOUNTPOINT
zoo                   24.3G   110G  10.5K  /zoo
zoo/home               623M  9.39G  16.5K  /export/zfs
zoo/home/coach        12.5K  9.39G  12.5K  /export/zfs/coach
zoo/home/haynest      12.5K  9.39G  12.5K  /export/zfs/haynest
zoo/home/kanigix         8K  9.39G     8K  /export/zfs/kanigix
zoo/home/loghyr       12.5K  9.39G  12.5K  /export/zfs/loghyr
zoo/home/morgan       12.5K  9.39G  12.5K  /export/zfs/morgan
zoo/home/mrx          12.5K  9.39G  12.5K  /export/zfs/mrx
zoo/home/nfsv2        12.5K  9.39G  12.5K  /export/zfs/nfsv2
zoo/home/nfsv3        12.5K  9.39G  12.5K  /export/zfs/nfsv3
zoo/home/nfsv4         362K  9.39G   264K  /export/zfs/nfsv4
zoo/home/nfsv4@monday  97.5K      -   108K  -
zoo/home/spud         12.5K  9.39G  12.5K  /export/zfs/spud
zoo/home/stacy        12.5K  9.39G  12.5K  /export/zfs/stacy
zoo/home/tdh           622M  9.39G   622M  /export/zfs/tdh
zoo/home/thomas       12.5K  9.39G  12.5K  /export/zfs/thomas
zoo/isos              20.9G   110G  20.9G  /zoo/isos
zoo/local                9K   110G     9K  /zoo/local
zoo/scratch           43.7M   110G  43.7M  /zoo/scratch
zoo/x86               2.78G   110G  2.78G  /zoo/x86

Okay, there is a filesystem. But, if we look at the directory entry again, it is quite recent and I think much newer than when I created the zoo pool:

# ls -la /zoo
total 6
drwxr-xr-x   3 root     root         512 Apr  6 21:46 .
drwxr-xr-x  42 root     root        1024 Mar 29 12:33 ..
dr-xr-xr-x   2 root     root         512 Apr  6 21:46 isos
# ls -la /zoo/isos/
total 4
dr-xr-xr-x   2 root     root         512 Apr  6 21:46 .
drwxr-xr-x   3 root     root         512 Apr  6 21:46 ..

Lets delete the directory and see what happens:

# rm -rf /zoo/isos/
# /usr/sbin/zfs mount -a
#

And lets reboot to get a clean slate:

# ps -ef |

*****************************************************************************
*
* Starting Desktop Login on display :0...
*
* Wait for the Desktop Login screen before logging in.
*
*****************************************************************************
grep ssh
    root   575   217   0 11:00:18 console     0:00 grep ssh
    root   408     1   0 11:00:02 ?           0:00 /usr/lib/ssh/sshd

And does ssh work?

[tdh@adept log]> ssh wont
Password:
Last login: Tue Apr 11 12:41:41 2006 from adept.internal.
Sun Microsystems Inc.   SunOS 5.11      snv_36  October 2007

Sweet!