Wednesday September 22, 2004

SMF/ Predictive Self Healing: svcs(1)

Perhaps the most often used tool in the SMF world is svcs. It's the master "observational" tool, used for querying the state and properties of all the services on your machine.

Used with no options, it simply lists all services that are enabled. Enabled means that the administrator wishes these services to be running. They may not be, because their dependencies are not met, they failed to start correctly, or some other reason. But they're the services that should be running.

# svcs
STATE          STIME    FMRI
legacy_run     Sep_17   lrc:/etc/rcS_d/S10pfil
legacy_run     Sep_17   lrc:/etc/rcS_d/S29wrsmcfg
...
legacy_run     Sep_17   lrc:/etc/rc2_d/S72autoinstall
legacy_run     Sep_17   lrc:/etc/rc2_d/S72directory
...
legacy_run     Sep_17   lrc:/etc/rc3_d/S84patchserver
legacy_run     Sep_17   lrc:/etc/rc3_d/S90samba
online         Sep_17   svc:/system/svc/restarter:default
online         Sep_17   svc:/network/loopback:default
online         Sep_17   svc:/network/physical:default
online         Sep_17   svc:/system/filesystem/root:default
...
online         Sep_17   svc:/network/ssh:default
online         Sep_17   svc:/system/coreadm:default
...
online         Sep_17   svc:/milestone/single-user:default
online         Sep_17   svc:/system/system-log:default
online         Sep_17   svc:/system/utmp:default
online         Sep_17   svc:/system/filesystem/local:default
online         Sep_17   svc:/milestone/name-services:default
online         Sep_17   svc:/network/inetd:default

Now, I've edited this list quite a bit. As it stands now, a freshly installed Solaris machine will come up with 108 running services. I'm sure this number will change a bit before we ship.

What can we see above? First, we see the legacy services, which I've mentioned in a previous post. These are the scripts that still exist in the rcX directories. For example, you can see above that /etc/rc3.d/S90samba was started. Since these services haven't been converted to SMF services, they won't be automatically restarted, but they'll be started, just like they were in previous incarnations of Solaris. If you have your own personal scripts, this is where you'll see them.

Below this you start to see online SMF services, such as ssh and inetd. You'll also see "milestones", which are services that are simply lists of dependencies that represent a system state, such as "single-user", or "local filesystems" as being available.

Let's take a look at some of the command line options that make svcs so powerful and useful.

-a

The -a option means "all". It shows all the services on a machine, whether they're enabled or not.

# svcs -a
STATE          STIME    FMRI
legacy_run     Sep_17   lrc:/etc/rcS_d/S10pfil
legacy_run     Sep_17   lrc:/etc/rcS_d/S29wrsmcfg
...
disabled       Sep_17   svc:/application/print/server:default
disabled       Sep_17   svc:/network/nfs/server:default
disabled       Sep_17   svc:/network/time:default
disabled       Sep_17   svc:/network/talk:default
online         Sep_17   svc:/system/svc/restarter:default
online         Sep_17   svc:/network/loopback:default
online         Sep_17   svc:/network/physical:default
...

This listing will show you all of the services seen in svcs, plus all the disabled services. Solaris 10 ships with most networking services turned off, for security purposes. You'll see above things like being an nfs server or a print server are also off by default. They can be enabled simply using svcadm, which will be the subject of a later post.

Now we'll start to take a look at how using svcs can be useful in analyzing a service.

-d

The -d option to svcs shows which services this service depends on. Let's take a real world example. Say that for some reason, inetd isn't running on your machine, and you want to look at what it depends on.

# svcs -d network/inetd
STATE          STIME    FMRI
online         Sep_17   svc:/network/loopback:default
online         Sep_17   svc:/network/physical:default
disabled       Sep_17   svc:/network/rpc/bind:default
online         Sep_17   svc:/milestone/single-user:default
online         Sep_17   svc:/system/filesystem/local:default
online         Sep_17   svc:/milestone/name-services:default

You can see above that inetd depends on networking, being in single user mode, local files, and name services. And also on rpcbind, which is disabled. You'll know that you need to enable rpcbind to get inetd to run. And since we have dynamic dependency checking with SMF, as soon as you enable rpcbind, inetd will come online.

-D

The -D option to svcs shows all the services that depend ON a given service. Again, for a real world example, let's take a look at rpcbind. Say you were considering disabling rpcbind, and you wanted to know what effect this would have on your system.

# svcs -D network/rpc/bind
STATE          STIME    FMRI
disabled       Sep_17   svc:/network/nis/server:default
disabled       Sep_17   svc:/network/nfs/client:default
disabled       Sep_17   svc:/network/rpc/bootparams:default
disabled       Sep_17   svc:/network/nfs/server:default
...
online         Sep_17   svc:/network/rpc/keyserv:default
online         Sep_17   svc:/network/inetd:default
online         Sep_17   svc:/network/nis/client:default
online         Sep_17   svc:/milestone/multi-user:default
online          0:42:34 svc:/network/nfs/cbd:default
online          0:42:34 svc:/network/nfs/mapid:default
online          1:26:47 svc:/network/nfs/nlockmgr:default
online          1:26:47 svc:/network/nfs/status:default
...

Again, I've removed several lines for brevity, but you can see that there are several disabled services that depend on rpcbind, and quite a few online services. Including inetd, your ability to be a nis server, the multi-user milestone, and others. So you can see that disabling rpcbind would have a significant effect on your machine, and you're able to know *exactly* how your system will be affected.

-l

Say you want to know a general "overview" of one of your services. svcs -l (list) shows the relevent stats on a service, including who its restarter is, who it depends on, what its state is, etc.

# svcs -l network/inetd:default
fmri         svc:/network/inetd:default
enabled      true
state        online
next_state   none
restarter    svc:/system/svc/restarter:default
contract_id  43 
dependency   require_all/none svc:/milestone/single-user (online)
dependency   optional_all/error svc:/network/rpc/bind (online)
dependency   optional_all/error svc:/network/physical (online)
dependency   require_all/error svc:/system/filesystem/local (online)
dependency   require_all/error svc:/network/loopback (online)

# svcs -l network/telnet:default
fmri         svc:/network/telnet:default
enabled      true
state        online
next_state   none
restarter    svc:/network/inetd:default
contract_id  128

The two examples above show inetd and telnet. You can see that they're both enabled and online. You can also see that inetd is restarted by the master restarter, but telnet is restarted by inetd. inetd has several dependencies, while telnet has none (other than inetd running).

You'll notice that inetd has two different types of dependencies, optional and require. While I'll go into types of dependencies in another post, it's helpful to note that "require" means that that dependency must be online, while "optional" means that it has to be online only if it's enabled.

-p

Sometimes you might want to know what processes are controlled by a service. The -p (process) option gives you the pid, start time, and name of all the processes started by a service. For example, sendmail:

# svcs -p network/smtp:sendmail
STATE          STIME    FMRI
online         Sep_17   svc:/network/smtp:sendmail
               Sep_17        452 sendmail
               Sep_17        453 sendmail

You can see above that sendmail has two processes on my machine, both started on Sept 17th.

One last example I'll show is how svcs can be useful for scripting purposes.

-H, -o

The -H option means "don't show column headings", and -o is used to pick output columns. Say you wanted to write a perl script that took services in a certain state, and performed an action upon them.

[straylight] % svcs -H -o state,fmri
legacy_run     lrc:/etc/rcS_d/S10pfil
legacy_run     lrc:/etc/rcS_d/S29wrsmcfg
legacy_run     lrc:/etc/rcS_d/S55fdevattach
...
online         svc:/system/svc/restarter:default
online         svc:/network/loopback:default
online         svc:/network/physical:default
online         svc:/system/filesystem/root:default
...

This way, you're telling svcs to output all the enabled services, showing only their state and FMRI, without the header. This can be fed into a script that looks for services that are "degraded" or in "maintenance" mode, and emails the administrator, for example.

Next time, we'll take a look at svcadm, and how to administer a system by enabling and disabling services. As always, comments and questions are welcome.