Infinite Probability Menno Lageman's Weblog

Monitoring zone boot and shutdown using DTrace

Several people have expressed a desire for a way to monitor zone state transitions such as zone boot or shutdown events. Currently there is no way to get notified when a zone is booted or shutdown. One way would be to run zoneadm list -p at regular intervals and parse the output, but this has some drawbacks that make this solution less ideal:

• it is inefficient because you are polling for events,
• you will probably start at least two processes for each polling cycle (zoneadm(1M) and nawk(1)),
• more importantly, you could miss transitions if your polling interval is too large. Since a zone reboot might take only seconds, you would need to poll often in order not to miss a state change.

A better, much more efficient solution can be built using DTrace, the 'Swiss Army knife of system observability'. As mentioned in this message on the DTrace forum, the zone_boot() function looks like a promising way to get notifications when a zone is booted. Listing all FBT probes with the string 'zone_' in their name (dtrace -l fbt|grep zone_) turns up another interesting function: zone_shutdown(). To verify that these probes are fired when a zone is either booted or shutdown, let's enable both probes:

# dtrace -n 'fbt:genunix:zone_boot:entry, fbt:genunix:zone_shutdown:entry {}'
dtrace: description 'fbt:genunix:zone_boot:entry, fbt:genunix:zone_shutdown:entry ' matched 2 probes

When zoneadm -z zone1 boot is executed we see that the zone_boot:entry probe fires:

CPU     ID                    FUNCTION:NAME
  0   6722                  zone_boot:entry

The zone_shutdown:entry probe fires when the zone is shutdown (either by zoneadm -z zone1 halt or using init 0 from within the zone):

  0   6726              zone_shutdown:entry

This gives us the basic 'plumbing' for the monitoring script. By instrumenting the zone_boot() and zone_shutdown() functions with the FBT provider we can wait for zone boot and shutdown with almost zero overhead. Now what is left is finding out the name of the zone that was booted or shutdown. This requires some knowledge of the implementation and access to the source (anyone interested can take a look at the source after OpenSolaris is launched, so stay tuned).

A quick look at the source shows that we can get the zone name by instrumenting a third function, zone_find_all_by_id() that is called by both zone_boot() and zone_shutdown(). This function returns a pointer to a zone_t structure (defined in /usr/include/sys/zone.h). The DTrace script below uses a common DTrace idiom: in the :entry probe we set a thread-local variable trace that is used as a predicate in the :return probes (the :return probes have the information we're after). The FBT provider :return probe stores the function return value in args[1] so we can access the zone name as args[1]->zone_name in fbt:genunix:zonefind_all_by_id:return and save it for later use in fbt:genunix:zone_boot:return and fbt:genunix:zone_shutdown:return.

#!/usr/sbin/dtrace -qs

self string name;

fbt:genunix:zone_boot:entry
{
        self->trace = 1;
}

fbt:genunix:zone_boot:return
/self->trace && args[1] == 0/
{
        printf("Zone %s booted\n", self->name);
        self->trace = 0;
        self->name = 0;
}

fbt:genunix:zone_shutdown:entry
{
        self->trace = 1;
}

fbt:genunix:zone_shutdown:return
/self->trace && args[1] == 0/
{
        printf("Zone %s shutdown\n", self->name);
        self->trace = 0;
        self->name = 0;
}

fbt:genunix:zone_find_all_by_id:return
/self->trace/
{
        self->name = stringof(args[1]->zone_name);
}

Starting the script and booting and shutting down some Zones gives the following result:

# ./zonemon.d
Zone aap booted
Zone noot booted
Zone noot shutdown
Zone noot booted
Zone aap shutdown

So there you have it, a simple DTrace script that will efficiently wait for zone boot and shutdown events. Enjoy.

Technorati Tag: Solaris

Technorati Tag: DTrace

( May 25 2005, 12:30:38 PM CEST ) Permalink Comments [5]