Jonathan Adams's Weblog

Coverage testing

A couple years back, I wrote up a description of how to use the Sun Studio compiler's coverage testing features to test userland code. Now that OpenSolaris is here, I thought it might come in handy for a larger audience. Here's goes:

How do I do coverage analysis on my user-level code?

The Sun Workshop compilers we use have some pretty good profiling and test analysis tools built in to them. One of the more useful for user-space code is Coverage Analysis, which gives you a measure of how complete your testing is.

Coverage analysis annotates each "block" of straight-line code with a count of the number of times it has executed. For testing, what is usually more interesting is which lines were never executed, and the "Coverage", or percentage of blocks in your program or library that were exercised in your testing. For more information, see tcov(1), in /opt/SUNWspro/man.

Compilation and Linking
Coverage analysis requires a special compilation of your program or library. Each .c file needs to be compiled with -xprofile=tcov, and the final link (either to executable or shared library) also needs -xprofile=tcov.

Setting:

CFLAGS += -xprofile=tcov
CFLAGS64 += -xprofile=tcov
DYNFLAGS += -xprofile=tcov (shared libraries only)

in the appropriate Makefiles, then make clean; make install is sufficient.

Generating Profile Data
The -xprofile=tcov version of your binary will generate profile information every time the executable is run (or, in the case of a shared library, any executable which links against it is run) and exits normally. The output is placed (by default) in ./progname.profile/, which will build up data from all executions as they exit. It will even join up 32-bit and 64-bit data sets.

The tcov output location is controlled by two environment variables, SUN_PROFDATA_DIR (default '.'), and SUN_PROFDATA (default 'progname.profile'). So if you are testing libfoo.so, and want to join the data from a bunch of executions into /tmp/libfoo.profile, you would set:

sh:
% SUN_PROFDATA_DIR=/tmp
% SUN_PROFDATA=libfoo.profile
% export SUN_PROFDATA_DIR SUN_PROFDATA
csh:
% setenv SUN_PROFDATA_DIR /tmp
% setenv SUN_PROFDATA libfoo.profile

before your runs.

Processing the profile data
Once you have finished gathering data, you can use the tcov(1) command, located in /opt/SUNWspro/bin (or wherever you keep your compilers) to analyze it. It's syntax is pretty straightforward:

% tcov -x profile_dir sourcefile...

For example, to analyze the previous libfoo example, you might: (here I use a seperate directory for my tcov analysis)

% cd usr/src/lib/libfoo
% mkdir tcov
% cd tcov
% tcov -x /tmp/libfoo.profile ../common/*.c ../sparc/*.c ../sparcv9/*.c

Analyzing the data
Nota Bene: The counts tcov uses to generate its output are updated without holding locks. For multi-threaded programs only, this means that some counts may be lower than expected. Nevertheless, if a block has been executed at least once, its count will be non-zero.

For each source file you pass in on the command line, tcov will generate a .tcov file (for example, ../common/foo.c -> foo.c.tcov). Each file contains the original source, annotated with execution counts. Each line that starts a "basic block" is prefixed with either '##### ->', indicating that it has not been executed, or 'count ->', indicating how many times it was executed.

After the annotated source, there is a summary of the file, including things like total blocks, number executed, % coverage, average executions per block, etc.

I've written a tool, tcov_summarize, which takes the tcov files in the current directory and displays a summary of the current state. The columns are "total blocks", "executed blocks", and "% executed" (or % coverage).

Command example: cpio

% cd usr/src/cmd/cpio
% grep tcov Makefile
CFLAGS += -xprofile=tcov
% make
... (made cpio) ...
% mkdir tcov
% cd tcov
% ../cpio
cpio: One of -i, -o or -p must be specified.
USAGE:
        cpio -i[bcdfkmrstuv@BSV6] [-C size] [-E file] [-H hdr] [-I file [-M msg]] [-R id] [patterns]
        cpio -o[acv@ABLV] [-C size] [-H hdr] [-O file [-M msg]]
        cpio -p[adlmuv@LV] [-R id] directory
% ls
cpio.profile/
% tcov -x cpio.profile ../*.c
% ls
cpio.c.tcov       cpio.profile/     cpiostat.c.tcov
% tcov_summarize
 1818    32   1.76 cpio.c
    2     0   0.00 cpiostat.c
 1820    32   1.76 total
% find . | ../cpio -ocB > /dev/null
590 blocks
% tcov -x cpio.profile ../*.c
% tcov_summarize
 1818   326  17.93 cpio.c
    2     0   0.00 cpiostat.c
 1820   326  17.91 total
%

Library example: libumem

% cd usr/src/lib/libumem   
% grep tcov Makefile.com   
CFLAGS +=       -v $(LOCFLAGS) -I$(CMNDIR) -xprofile=tcov
CFLAGS64 +=     -v $(LOCFLAGS) -I$(CMNDIR) -xprofile=tcov
DYNFLAGS +=     -M $(MAPFILE) -z interpose -xprofile=tcov
% make
... (made libumem) ...
% mkdir tcov   
% cd tcov   
% SUN_PROFDATA_DIR=`pwd`   
% SUN_PROFDATA=libumem.profile   
% export SUN_PROFDATA_DIR SUN_PROFDATA   
% LD_PRELOAD=../sparc/libumem.so.1 LD_PRELOAD_64=../sparcv9/libumem.so.1
% export LD_PRELOAD LD_PRELOAD_64   
% ls   
% ls   
libumem.profile/
% tcov -x libumem.profile ../common/*.c ../sparc/*.c   
% /home/jwadams/bin/tcov_summarize   
   75    44  58.67 envvar.c
   10     7  70.00 getpcstack.c
   72    22  30.56 malloc.c
   78    27  34.62 misc.c
  592   255  43.07 umem.c
    1     0   0.00 umem_agent_support.c
  315   167  53.02 vmem.c
   13    10  76.92 vmem_base.c
   20     0   0.00 vmem_mmap.c
   35    17  48.57 vmem_sbrk.c
 1211   549  45.33 total
% tcov -x libumem.profile ../common/*.c ../sparc/*.c   
% /home/jwadams/bin/tcov_summarize   
   77    45  58.44 envvar.c
   10     7  70.00 getpcstack.c
   72    28  38.89 malloc.c
   78    27  34.62 misc.c
  592   314  53.04 umem.c
    1     0   0.00 umem_agent_support.c
  315   192  60.95 vmem.c
   13    10  76.92 vmem_base.c
   20     0   0.00 vmem_mmap.c
   35    17  48.57 vmem_sbrk.c
 1213   640  52.76 total
%

(Note that running tcov gave us more coverage, since the library is being preloaded underneath it)

Tags: [ OpenSolaris, Solaris ]

Posted at 02:04PM Nov 23, 2005 by jwadams in Solaris  |  Comments[15]