Wednesday June 29, 2005

Using prstat to estimate memory slow-downs

There are many indicators which show when Solaris has a memory shortage, but when you get in this situation wouldn't it be nice if you could tell who is being affected, and to what degree?

Here's a simple method: use the microstate measurement facility in prstat(1M). Using the memory-stall counts, you can observe the percentage of wall clock time spent waiting in data faults.

The microstates show a timeline for 100% of the wall clock time of a thread in between samples. The time is broken down into eight categories; on cpu in USR/SYS, in a software trap (TRP), in memory stalls (DFL and TFL), in user locks (LCK), sleeping on I/O or something else (SLP), and the time spent waiting on the run queue (LAT). Of particular interest here is the DFL column. It shows the percentage of time spent waiting for data faults to be serviced. When an application is stalled waiting to be paged in from the swap device, time accumulates in this column.

The In the following example we shows a severe memory shortage situation. The system was running short of memory, and each thread in filebench is waiting for memory approximately 90% of the time.

sol8$ prstat -mL
   PID USERNAME USR SYS TRP TFL DFL LCK SLP LAT VCX ICX SCL SIG PROCESS/LWPID
 15625 rmc      0.1 0.7 0.0 0.0  95 0.0 0.9 3.2  1K 726  88   0 filebench/2
 15652 rmc      0.1 0.7 0.0 0.0  94 0.0 1.8 3.6  1K  1K  10   0 filebench/2
 15635 rmc      0.1 0.7 0.0 0.0  96 0.0 0.5 3.2  1K  1K   8   0 filebench/2
 15626 rmc      0.1 0.6 0.0 0.0  95 0.0 1.4 2.6  1K 813  10   0 filebench/2
 15712 rmc      0.1 0.5 0.0 0.0  47 0.0  49 3.8  1K 831 104   0 filebench/2
 15628 rmc      0.1 0.5 0.0 0.0  96 0.0 0.0 3.1  1K 735   4   0 filebench/2
 15725 rmc      0.0 0.4 0.0 0.0  92 0.0 1.7 5.7 996 736   8   0 filebench/2
 15719 rmc      0.0 0.4 0.0 0.0  40  40  17 2.9  1K 708 107   0 filebench/2
 15614 rmc      0.0 0.3 0.0 0.0  92 0.0 4.7 2.4 874 576  40   0 filebench/2
 15748 rmc      0.0 0.3 0.0 0.0  94 0.0 0.0 5.5 868 646   8   0 filebench/2
 15674 rmc      0.0 0.3 0.0 0.0  86 0.0 9.7 3.2 888 571  62   0 filebench/2
 15666 rmc      0.0 0.3 0.0 0.0  29  46  23 2.1 689 502 107   0 filebench/2
 15682 rmc      0.0 0.2 0.0 0.0  24  43  31 1.9 660 450 107   0 filebench/2

( Jun 29 2005, 04:40:32 PM PDT ) Permalink Comments [2]