Ganesh Hiregoudar's Weblog

SMF troubleshooting techniques

Logfiles, Manifests and Methods
===============================

SMF logfiles can be found in the following locations:
    /var/svc/log
    /etc/svc/volatile

The SMF manifests and methods may be found here:
    /var/svc/manifest/*
    /lib/svc/method/*

To see the log file location for a given service use:   
$ svcs -l <FMRI> | grep logfile

(<FMRI> - Fault Management Resource Identifier. See the svcs(1) manual page for
details of how to specify an FMRI)

Daemons, Configuration and Recovery
===================================

The primary daemons for SMF are svc.startd and svc.configd.  At boot time the
kernel will start init, which in turn starts svc.startd and that starts
svc.configd.  This latter daemon (svc.configd) will read information from the
repository in /etc/svc/repository.db which is then used by svc.startd to start
the services.

If there is a problem booting the system you should try:

ok boot -m milestone=none

Then try to reach milestone "single-user" by running:

# svcadm milestone svc:/milestone/single-user:default

Observe the services as they start, look at any messages on screen and consult
the error logs for more information. If the repository is corrupt you may need
to restore from a previous saved copy, or reseed it.

The /lib/svc/bin/restore_repository script will guide you through this process.

=================================================

If a service fails to start, you should check:
* The state of the service instance
  - If a service is in maintenance mode use:
       svcs -xv or svcs -l to see why it failed
       inspect the log files for further information
       svcadm clear <FMRI> once the fault has been repaired
* The dependencies for the service
  - Are they optional or required?
       svcs -d shows the dependencies for a service 
       svcs -l shows whether the dependencies are optional or required
* The error log for the instance
       svcs -l <FMRI> | grep logfile tells you where the logfile is
* The properties for the service
       svcprop -p start <FMRI> shows the start properties for a service
* Have any changes been made to the repository?
  - How many services are affected?
  - Have any snapshots been created?
  - Can you resort to a previous snapshot?
* The start method for the service
  - What type of service is it?  (transient, child, contract)
  - Look at the start method
  - Did the service get killed?  Why?
  - Check the start method in the manifest and compare it to the repository    
    entry
  - If you need to truss the service starting, you could try something like:
      # svcprop -p start/exec network/inetd     ! Use to figure out start method
      # svccfg -s network/inetd setprop 'start/exec = “truss -f -a -o 
        /tmp/truss.out
         /usr/lib/inet/inetd %m”'       ! Use the method obtained from svcprop 
      # svcadm refresh network/inetd
      # svcadm restart network/inetd
      Remember to restore the original start method when finished!

======================================================

If there are problems booting the system, you should:
* Capture the error message(s)
* Try to boot to an alternative milestone, eg:
        ok boot -m milestone=none
        ok boot -s
* If successful and you can login, try to start the next milestone
* Check the error logs and dependencies
* Can you boot the system in verbose mode, eg:
        ok boot -m verbose
NOTE: This changes the boot behaviour from parallel to serial
* If the repository is beyond repair, try restoring from an earlier copy
or in the worst case you may need to reseed it.
The utility for this is /lib/svc/bin/restore_repository

Posted at 11:17PM Jul 31, 2005 by G S Hiregoudar in Solaris  |  Comments[0]