vi-curious Gary J Little's Weblog

Oi! who said you could use that CPU

I just happened to stumble across a very useful switch to prstat, which I either forgot about or never knew existed. The magic switch is -m which displays microstate accounting information. Woo hoo you say, WTF does that mean? Well it means that you can see which processes are running in USR and which are using SYS e.g.

   PID USERNAME USR SYS TRP TFL DFL LCK SLP LAT VCX ICX SCL SIG PROCESS/NLWP
    50 garyli   2.4  13 0.0 0.0 0.0 0.0  74 9.7  1K 169  9K   0 find/1
 26632 garyli   0.5 0.9 0.0 0.0 0.0 0.0  99 0.0   2   6 550   0 prstat/1
    47 garyli   0.4 0.6 0.0 0.5 0.0 0.0  98 0.8   4   7 544   0 prstat/1
 22201 garyli   0.5 0.4 0.0 0.0 0.0 0.0  95 4.4 251   4 745   0 acroread/1
 21140 garyli   0.0 0.0 0.0 0.1 0.0  17  83 0.1 130  25 210   0 netscape-bin/6
 24289 garyli   0.0 0.0 0.0 0.0 0.0 0.0 100 0.0   1   0   9   0 dtterm/1
    52 garyli   0.0 0.0 0.0 0.0 0.0 0.0 100 0.1   5   2  10   0 dtscreen/1
 13896 garyli   0.0 0.0 0.0 0.0 0.0  53  46 0.2 215   1 189   0 java/15
   169 daemon   0.0 0.0  -   -   -   -  100  -    0   0   0   0 statd/2
   170 daemon   0.0 0.0  -   -   -   -  100  -    0   0   0   0 lockd/2
 24299 garyli   0.0 0.0 0.0 0.0 0.0 0.0 100 0.0   0   0   0   0 ksh/1
 21221 garyli   0.0 0.0 0.0 0.0 0.0 0.0 100 0.0   0   0   0   0 dtcalc/1
 10102 garyli   0.0 0.0 0.0 0.0 0.0 0.0 100 0.0   0   0   0   0 ksh/1
   212 root     0.0 0.0  -   -   -   -  100  -    0   0   0   0 automountd/3
   141 root     0.0 0.0  -   -   -   -  100  -    0   0   0   0 ypbind/1
Total: 165 processes, 263 lwps, load averages: 0.47, 0.34, 0.18

So we see that find(1) is spending 13% of its time in sys. Hmmm tarzan say sys is bad - better log a bug. But wait, if we look at the SCL column we see it's made 9,000 system calls so no wonder it spent a lot if time in sys. Every time we make a system call, we are doing stuff in sys - so that explains that. If we really wanted to we could do a truss -c and see which syscalls are made, and we'd see stuff like chdir() lstat() fstat() and so on. Mystery solved.

So now you can drill down through your vmstat->mpstat->prstat to see who is using all that CPU. Of course it's pretty useless for short lived processes but you could always use regular accounting for that.