Monitoring ZFS Statistic on the system loaded with Oracle database
Nothing happens without a reason. My previous post about Monitoring ZFS Statistic is not exception. This week I had T5240 (128 threads, 64GB memory, Solaris 10 10/08 aka U6) with 6140 attached and not loaded with other my projects. So I decided to load the boxes with Oracle on ZFS (binaries, datafiles, logs) and run some tests. Not a benchmarking. Just to check if this can live and breathe. I also wanted to check the battle for the memory between Oracle SGA (set to 32GB), 3800 processes connecting to database (taking approx. 5MB not shared memory each) and ZFS.
On storage I have made single 1TB RAID1 LUN and put ZFS on it. No data/logs of ZFS separation. Keeping it simple.
root@rac05#zpool list NAME SIZE USED AVAIL CAP HEALTH ALTROOT oracle 1016G 243G 773G 23% ONLINE -
Arcstat script was modified a little further (lines 82, 177, 229 and 230) so I don't have too many zeros in statistics by going up from bytes to MB. Download new arcstat script and description file for dimstat.
81 my $version = "0.1"; 82 my $mega = 1024 * 1024; 83 my $cmd = "Usage: arcstat.pl [-hvx] [-f fields] [-o file] [interval [count]]\n"; ... 171 sub prettynum { 172 my @suffix=(' ','K', 'M', 'G', 'T', 'P', 'E', 'Z'); 173 my $num = $_[1] || 0; 174 my $sz = $_[0]; 175 my $index = 0; 176 return sprintf("%s", $num) if not $num =~ /^[0-9\.]+$/; 177 return sprintf("*%d", $sz, $num); 178 while ($num > 1000 and $index < 8) { 179 $num = $num/1000; 180 $index++; 181 } 182 return sprintf("%*d", $sz, $num) if ($index == 0); 183 return sprintf("%*d%s", $sz - 1, $num,$suffix[$index]); 184 } ... 200 sub calculate { ... 229 $v{"arcsz"} = $cur{"size"}/$mega; 230 $v{"c"} = $cur{"c"}/$mega; ... 238 }
On the picture you may see results of monitoring:

At 1:01 test was started. It took 1 hour until all processes become ready (1 process was started each 1 second). Then Oracle database was busy between 2:04 and 2:53.
As you can see ZFS is taking more memory if it can, but it stays "polite" by freeing this memory for other processes when it needed.
I will be glad to hear from you if you have any thoughts and experience running Oracle Database on ZFS.
Thanks to Valery for his Perl Syntax highlighting plugin for Netbeans.


While single fs for Oracle is easier to maintain I would at least separate indexes and data files and match a zfs recordsize to Oracle's db_block_size on these filesystem.
Posted by Robert Milkowski on December 16, 2008 at 06:34 AM PST #
Thanks Robert. You are absolutely right that if someone want to try Oracle on ZFS then he should pay attention to data separation and record size matching. This is starting point to very long road of tuning. Here I just wanted to give another tool for ZFS monitoring.
Posted by Roman Ivanov on December 16, 2008 at 07:34 AM PST #