Eric Kustarz's Weblog

e-street

Tuesday May 03, 2005

dscript to get active NFS clients / nfstop

Have you ever wondered which client(s) we're banging away when your NFS server got a little sluggish? Well with the power of D, it becomes quite simple. Here's a dscript to do exactly that:

#!/usr/sbin/dtrace -FCs

#define AF_INET 2
#define AF_INET6 26

fbt::common_dispatch:entry
{
self->ca = (struct sockaddr *)(args[0]->rq_xprt->xp_xpc.xpc_rtaddr.buf);
self->sin_addr = (uchar_t *)&((struct sockaddr_in *)self->ca)->sin_addr;

self->in = 1;
}

fbt::common_dispatch:return
/self->in && self->ca->sa_family == AF_INET/
{
self->sin_addr = (uchar_t *)&((struct sockaddr_in *)self->ca)->sin_addr;
@hosts[self->sin_addr[0], self->sin_addr[1], self->sin_addr[2],
self->sin_addr[3]] = count();

self->in = 0;
self->ca = 0;
self->sin_addr = 0;
}

fbt::common_dispatch:return
/self->in && self->ca->sa_family == AF_INET6/
{
self->sin6 = (uchar_t *)&((struct sockaddr_in6 *)self->ca)->sin6_addr;

@hosts6[self->sin6[0], self->sin6[1], self->sin6[2], self->sin6[3],
   self->sin6[4], self->sin6[5], self->sin6[6], self->sin6[7],
   self->sin6[8], self->sin6[9], self->sin6[10], self->sin6[11],
   self->sin6[12], self->sin6[13], self->sin6[14], self->sin6[15]]
   = count();

self->in = 0;
self->ca = 0;
self->sin6 = 0;
}

END
{
printa("\nhost: %d.%d.%d.%d num nfs calls: %@d", @hosts);
printa("\nhost: %x%x:%x%x:%x%x:%x%x:%x%x:%x%x:%x%x:%x%x num nfs calls: %@d", @hosts6);
}

And here's an example of the output:

# ./get_nfs_clients.d
dtrace: script './get_nfs_clients.d' matched 4 probes
^C
CPU FUNCTION 7 | :END
host: 1.1.1.1 num nfs calls: 2
host: 3.2.2.2 num nfs calls: 2
host: 3.2.2.3 num nfs calls: 2
host: 3.2.2.4 num nfs calls: 5
host: 1.3.3.5 num nfs calls: 6
host: 2.4.4.6 num nfs calls: 10
host: 1.3.3.7 num nfs calls: 10
host: 1.5.3.8 num nfs calls: 25
host: fe80:00:00:00:00:00:00:xx num nfs calls: 2

#

Sorted automatically from least to most active... with ipv4 and ipv6 hosts being separate sorts (due to being separate aggregates).

very cool! aaaah, but Richard, being the DE as he is, one upped me to create a 'nfstop' of sorts by replacing the above 'END' with:

tick-3s
{
trunc (@hosts, 20);
trunc (@hosts6, 20);
printf("\033[H\033[2J");
printa("\nhost: %d.%d.%d.%d num nfs calls: %@d", @hosts);
printa("\nhost: %x%x:%x%x:%x%x:%x%x:%x%x:%x%x:%x%x:%x%x num nfs calls: %@d", @hosts6);
trunc (@hosts);
trunc (@hosts6);
}

Now we're talking!

(2005-11-14 17:03:12.0/2005-05-03 09:41:19.0) Permalink Comments [4]
Trackback: http://blogs.sun.com/erickustarz/en_US/entry/dscript_to_retrieve_active_nfs

Trackback URL: http://blogs.sun.com/erickustarz/en_US/entry/dscript_to_retrieve_active_nfs

Comments:

[Trackback] Ein nützliches Dtrace-Script, um festzustellen, was auf einzelnen NFS-Verbindungen los ist: dscript to get active NFS clients / nfstop...

Posted by www.c0t0d0s0.org on May 03, 2005 at 12:37 PM PDT #

Eric, thanks a most concrete example of how dtrace can be used.

Now that you've stimulated my interest in dtrace, some points/questions:

how would I as a Solaris 10 user know that a trace point called common_dispatch exists in the kernel? (well given that that's a surboutine name, at least was in S8, *I* knew that, but that's cheating since I reviewed the code for the guy who wrote common_dispatch() :-).
how would I know what dtrace-able variables exist in common_dispatch(), as well as what they do?
how could I as a kernel developer (or someone using the stuff in opensolaris.org) add dtrace probes to my own NFS server? :-)

Posted by Mike Eisler on May 09, 2005 at 05:53 PM PDT #

Excellent question! (and one that i had when i first heard of dtrace)

So for sure, knowing about common_dispatch() is impossible unless you know/have access to the code. The good thing is that when opensolaris happens, everyone will access to the code. The bad thing is that its not practical to have every admin start looking through the code to try and pry out the info they want.

So i think the future answer is a NFS provider... check out:
Dtrace tracing guide and the vminfo provider, for example.

The Dtrace tracing guide in particular is aces, chock full of useful examples. When we move to a legit NFS provider, then the documentation will be there to know that the probes in fact exist, and how to use them.

As for creating your own, i would first look at SDT (statically defined tracing) .

So one last thing that i think should be mentioned is the lockstat provider. The lockstat(1M) command is actually just a user of the lockstat provider. This is a similar path we could take with retrieving useful NFS information - give the admin/user a command instead of a dscript to run, and they don't even have to know they're using dtrace.

Posted by eric kustarz on May 10, 2005 at 03:50 PM PDT #

I've added an entry over at my blog that expands this answer :)

Posted by Robert Gordon on May 24, 2005 at 08:04 PM PDT #

Eric Kustarz's Weblog

e-street

Languages: