Jonathan Adams's Weblog
Wednesday Nov 23, 2005
Coverage testing
A couple years back, I wrote up a description of how to use the Sun Studio compiler's coverage testing features to test userland code. Now that OpenSolaris is here, I thought it might come in handy for a larger audience. Here's goes:
How do I do coverage analysis on my user-level code?
The Sun Workshop compilers we use have some pretty good profiling and test analysis tools built in to them. One of the more useful for user-space code is Coverage Analysis, which gives you a measure of how complete your testing is.Coverage analysis annotates each "block" of straight-line code with a count of the number of times it has executed. For testing, what is usually more interesting is which lines were never executed, and the "Coverage", or percentage of blocks in your program or library that were exercised in your testing. For more information, see tcov(1), in /opt/SUNWspro/man.
Compilation and Linking
Coverage analysis requires a special compilation of your program or library. Each .c file needs to be compiled with -xprofile=tcov, and the final link (either to executable or shared library) also needs -xprofile=tcov.
Setting:
CFLAGS += -xprofile=tcov CFLAGS64 += -xprofile=tcov DYNFLAGS += -xprofile=tcov (shared libraries only)in the appropriate Makefiles, then make clean; make install is sufficient.
Generating Profile Data
The -xprofile=tcov version of your binary will generate profile information every time the executable is run (or, in the case of a shared library, any executable which links against it is run) and exits normally. The output is placed (by default) in ./progname.profile/, which will build up data from all executions as they exit. It will even join up 32-bit and 64-bit data sets.
The tcov output location is controlled by two environment variables, SUN_PROFDATA_DIR (default '.'), and SUN_PROFDATA (default 'progname.profile'). So if you are testing libfoo.so, and want to join the data from a bunch of executions into /tmp/libfoo.profile, you would set:
sh: % SUN_PROFDATA_DIR=/tmp % SUN_PROFDATA=libfoo.profile % export SUN_PROFDATA_DIR SUN_PROFDATA csh: % setenv SUN_PROFDATA_DIR /tmp % setenv SUN_PROFDATA libfoo.profilebefore your runs.
Processing the profile data
Once you have finished gathering data, you can use the tcov(1) command, located in /opt/SUNWspro/bin (or wherever you keep your compilers) to analyze it. It's syntax is pretty straightforward:
% tcov -x profile_dir sourcefile...For example, to analyze the previous libfoo example, you might: (here I use a seperate directory for my tcov analysis)
% cd usr/src/lib/libfoo % mkdir tcov % cd tcov % tcov -x /tmp/libfoo.profile ../common/*.c ../sparc/*.c ../sparcv9/*.cAnalyzing the data
Nota Bene: The counts tcov uses to generate its output are updated without holding locks. For multi-threaded programs only, this means that some counts may be lower than expected. Nevertheless, if a block has been executed at least once, its count will be non-zero.
For each source file you pass in on the command line, tcov will generate a .tcov file (for example, ../common/foo.c -> foo.c.tcov). Each file contains the original source, annotated with execution counts. Each line that starts a "basic block" is prefixed with either '##### ->', indicating that it has not been executed, or 'count ->', indicating how many times it was executed.
After the annotated source, there is a summary of the file, including things like total blocks, number executed, % coverage, average executions per block, etc.
I've written a tool, tcov_summarize, which takes the tcov files in the current directory and displays a summary of the current state. The columns are "total blocks", "executed blocks", and "% executed" (or % coverage).
Command example: cpio
% cd usr/src/cmd/cpio
% grep tcov Makefile
CFLAGS += -xprofile=tcov
% make
... (made cpio) ...
% mkdir tcov
% cd tcov
% ../cpio
cpio: One of -i, -o or -p must be specified.
USAGE:
cpio -i[bcdfkmrstuv@BSV6] [-C size] [-E file] [-H hdr] [-I file [-M msg]] [-R id] [patterns]
cpio -o[acv@ABLV] [-C size] [-H hdr] [-O file [-M msg]]
cpio -p[adlmuv@LV] [-R id] directory
% ls
cpio.profile/
% tcov -x cpio.profile ../*.c
% ls
cpio.c.tcov cpio.profile/ cpiostat.c.tcov
% tcov_summarize
1818 32 1.76 cpio.c
2 0 0.00 cpiostat.c
1820 32 1.76 total
% find . | ../cpio -ocB > /dev/null
590 blocks
% tcov -x cpio.profile ../*.c
% tcov_summarize
1818 326 17.93 cpio.c
2 0 0.00 cpiostat.c
1820 326 17.91 total
%
Library example: libumem
% cd usr/src/lib/libumem
% grep tcov Makefile.com
CFLAGS += -v $(LOCFLAGS) -I$(CMNDIR) -xprofile=tcov
CFLAGS64 += -v $(LOCFLAGS) -I$(CMNDIR) -xprofile=tcov
DYNFLAGS += -M $(MAPFILE) -z interpose -xprofile=tcov
% make
... (made libumem) ...
% mkdir tcov
% cd tcov
% SUN_PROFDATA_DIR=`pwd`
% SUN_PROFDATA=libumem.profile
% export SUN_PROFDATA_DIR SUN_PROFDATA
% LD_PRELOAD=../sparc/libumem.so.1 LD_PRELOAD_64=../sparcv9/libumem.so.1
% export LD_PRELOAD LD_PRELOAD_64
% ls
% ls
libumem.profile/
% tcov -x libumem.profile ../common/*.c ../sparc/*.c
% /home/jwadams/bin/tcov_summarize
75 44 58.67 envvar.c
10 7 70.00 getpcstack.c
72 22 30.56 malloc.c
78 27 34.62 misc.c
592 255 43.07 umem.c
1 0 0.00 umem_agent_support.c
315 167 53.02 vmem.c
13 10 76.92 vmem_base.c
20 0 0.00 vmem_mmap.c
35 17 48.57 vmem_sbrk.c
1211 549 45.33 total
% tcov -x libumem.profile ../common/*.c ../sparc/*.c
% /home/jwadams/bin/tcov_summarize
77 45 58.44 envvar.c
10 7 70.00 getpcstack.c
72 28 38.89 malloc.c
78 27 34.62 misc.c
592 314 53.04 umem.c
1 0 0.00 umem_agent_support.c
315 192 60.95 vmem.c
13 10 76.92 vmem_base.c
20 0 0.00 vmem_mmap.c
35 17 48.57 vmem_sbrk.c
1213 640 52.76 total
%
(Note that running tcov gave us more coverage, since the library is being preloaded underneath it)
Tags: [ OpenSolaris, Solaris ]
Posted at 02:04PM Nov 23, 2005 by jwadams in Solaris | Comments[15]
Wednesday Nov 16, 2005
An initial encounter with ZFS
After ZFS became available in onnv_27, I immediately upgraded my desktop system to the newly minted bits. After some initial setup, I've been happily using ZFS for all of my non-root, non-NFSed data. I'm getting about 1.7x my storage due to ZFS's compression, and have new-found safety, since my data is now mirrored.
During the initial setup, my intent was to use only the slices I'd already set up to do the transfer. What I did not plan for was the fact that ZFS does not currently allow you to remove a non-redundant slice from a storage pool without destroying the pool; here's what I did, as well as what I should have done:
My setup
Before I began, my system layout was fairly simple:
c0t0d0: Total disk cylinders available: 24620 + 2 (reserved cylinders) Part Tag Flag Cylinders Size Blocks 0 root wm 1452 - 7259 8.00GB (5808/0/0) 16779312 1 unassigned wm 0 0 (0/0/0) 0 2 backup wm 0 - 24619 33.92GB (24620/0/0) 71127180 3 swap wu 0 - 1451 2.00GB (1452/0/0) 4194828 4 unassigned wm 0 0 (0/0/0) 0 5 unassigned wm 0 0 (0/0/0) 0 6 unassigned wm 0 0 (0/0/0) 0 7 aux0 wm 7260 - 24619 23.91GB (17360/0/0) 50153040 c0t1d0: Total disk cylinders available: 24620 + 2 (reserved cylinders) Part Tag Flag Cylinders Size Blocks 0 altroot wm 0 - 5807 8.00GB (5808/0/0) 16779312 1 unassigned wm 0 0 (0/0/0) 0 2 backup wm 0 - 24619 33.92GB (24620/0/0) 71127180 3 swap wu 5808 - 7259 2.00GB (1452/0/0) 4194828 4 unassigned wm 0 0 (0/0/0) 0 5 unassigned wm 0 0 (0/0/0) 0 6 unassigned wm 0 0 (0/0/0) 0 7 aux1 wm 7260 - 24619 23.91GB (17360/0/0) 50153040That is, two 34GB hard disks, partitioned identically. There are four slices of major interest:
c0t0d0s0 / 8G root filesystem c0t0d0s7 /aux0 24G data needing preserving c0t1d0s0 /altroot 8G alternate root, currently empty c0t1d0s7 /aux1 24G some data (/opt) which needed preservingMy goal was to create a 24Gig mirrored ZFS pool, using the underlying slices of /aux0 and /aux1. /altroot would be my initial stepping stone.
The process
Without any prior experience setting up a ZFS pool, I did the following steps:
... remove /altroot from /etc/vfstab ... # zpool create mypool c0t1d0s0 invalid vdev specification use '-f' to override the following errors: /dev/dsk/c0t1d0s0 contains a ufs filesystem # zpool create -f mypool c0t1d0s0 # zpool list NAME SIZE USED AVAIL CAP HEALTH ALTROOT mypool 7.94G 32.5K 7.94G 0% ONLINE - # zfs set compression=yes mypool # zfs create pool/opt # zfs set mountpoint=/new_opt mypool/opt ... copy data from /aux1/opt to /new_opt, clear out /aux1 ... ... remove /aux1 from vfstab, and remove the /opt symlink ... # zfs set mountpoint=/opt mypool/opt # df -h /opt Filesystem size used avail capacity Mounted on mypool/opt 7.9G 560M 7.4G 6% /opt #I now had all of the data I needed off of /aux1, and I wanted to add it to the storage pool. This is where I made a mistake; zfs, in its initial release, cannot remove a non-redundant device from a pool (this is being worked on). I did:
# zpool add -f mypool c0t1d0s7 (*MISTAKE*)
# zpool status mypool
pool: mypool
state: ONLINE
scrub: none requested
config:
NAME STATE READ WRITE CKSUM
mypool ONLINE 0 0 0
c0t1d0s0 ONLINE 0 0 0
c0t1d0s7 ONLINE 0 0 0
# zfs create mypool/aux0
# zfs create mypool/aux1
# zfs set mountpoint=/new_aux0 mypool/aux0
# zfs set mountpoint=/aux1 mypool/aux1
... move data from /aux0 to /new_aux0 ...
... remove /aux0 from /etc/vfstab ...
# zfs set mountpoint=/aux0 mypool/aux0
And now I was stuck; I wanted to end up with a configuration of:
mypool
mirror
c0t0d0s7
c0t1d0s7
but there was no way to get there without removing c0t1d0s0 from the
pool, which ZFS doesn't allow you to do directly. I ended up creating a
new pool, "pool" with c0t0d0s7 in it, copying all of my data *again*,
destroying "mypool", then mirroring c0t0d0s7 by doing:
# zpool attach pool c0t0d0s7 c0t1d0s7 #
The right way
If I'd planned this all better, the right way to build the pool I wanted would have been to do:
# zpool create pool c0t1d0s0
... create /opt_new, move data to it ...
# zpool replace pool c0t1d0s0 c0t1d0s7
... create /aux0_new, move data to it ...
# zpool attach pool c0t1d0s7 c0t0d0s7
... clean up attributes, wait for sync to complete ...
# zpool iostat -v
capacity operations bandwidth
pool used avail read write read write
------------ ----- ----- ----- ----- ----- -----
pool 15.0G 8.83G 0 1 1.47K 6.99K
mirror 15.0G 8.83G 0 1 1.47K 6.99K
c0t1d0s7 - - 0 1 2.30K 7.10K
c0t0d0s7 - - 0 1 2.21K 7.10K
------------ ----- ----- ----- ----- ----- -----
#
The real lesson here is to do a little planning if you have to juggle
slices around; missteps can take some work to undo. Read about "zpool replace"
and "zpool attach"; they are very useful for this kind of juggling.
Once I got everything set up, everything just works; despite having cut my available storage in /aux0 and /aux1 in half (48GB -> 24GB, due to mirroring), compression is giving me back a substantial fraction of the loss (~70%, give or take, and assuming the ratio holds steady):
# zfs get -r compressratio pool NAME PROPERTY VALUE SOURCE pool compressratio 1.70x - pool/aux0 compressratio 1.54x - pool/aux1 compressratio 2.01x - pool/opt compressratio 1.68x - #and ZFS itself is quite zippy. I hope this is a useful lesson, and that you enjoy ZFS!
Tags: [ OpenSolaris, ZFS ]
Posted at 09:00AM Nov 16, 2005 by jwadams in Solaris | Comments[4]