A recent article (and part 2) documented the author's attempts to benchmark MySQL performance on a variety of operating systems. He noted that a popular MySQL benchmark called Super-Smack doesn't compile on Solaris. I hate when this happens-- inevitably this tells the reader that Solaris isn't really a serious platform for MySQL (how could it be, if the benchmark doesn't work?). But from the other benchmarks the author provides, we can see that Solaris performs respectably; and I suspect that MySQL has itself not received as much tuning under Solaris as, for example, under Linux. With the help of performance analysis tools like DTrace, trapstat, etc. we (or you, gentle reader) can fix that.

First, I'd like to clarify one claim in the article: Solaris 10 is bundled with a compiler. That wasn't true in the beta build the author used; but it is true as of the FCS build. So, the benchmark will compile without installing any additional software.

I decided that it was time to get Super-Smack working under Solaris. The first task was to get the program to compile. The configure script ran OK, and I elected to just use the MySQL included in Solaris 10. For more serious benchmarking, I would study this to decide whether to build MySQL myself. So:

$ PATH=/usr/bin:/usr/sbin:/usr/sfw/bin:/usr/ccs/bin
$ export PATH
$ ./configure --with-mysql \
     --with-mysql-lib=/usr/sfw/lib/ \
     --with-mysql-include=/usr/sfw/include/mysql \
     --prefix=/home/dp/super-smack

I then had to make a few edits: src/Makefile needs to link the benchmark with the additional libraries -lsocket -lnsl. A proper autoconf setup should detect this, but... no problem. A few minor edits to C files were also needed:

• Added #include <strings.h> to engines.cc for bzero.
• In query.cc, replaced calls to flock() with calls to fcntl(3c):

    -  flock(1, LOCK_EX);
    +  fcntl(1, F_SETLK, F_WRLCK); 
  

It turns out that flock is used only sparingly, at the end of the benchmark run, so we don't need to pay attention to any performance implications.

So now, it built cleanly! Hooray. Next, I muddled my way through getting mysqld started. Once I did, I had to cope with one more problem: Super-Smack, by way of libmysqlclient, seems to want to access the mysql database via a UNIX domain socket at /var/lib/mysql/mysql.sock. However, the database seems to put that socket in /tmp/mysql.sock. I wasn't sure why, and I decided to investigate that discrepancy out later. I hacked things up by putting an appropriate symlink in /var/lib/mysql to work around the problem.

Next, I ran Super-Smack as instructed in the article, and things went somewhat haywire. A quick look revealed that Super-Smack has a fairly conventional design: A parent process forks a bunch of children, which do benchmark activities. When these are finished, they write information back to the parent. I received a variety of error messages, and after applying truss to some abbreviated runs, Jonathan and I decided that the parent super-smack process was exiting prematurely, and failing to collect the data being sent to it by its children. A quick scan of the source code led me to this innocuous looking line of code:

      pid_t pid = wait4(-1, 0, 0, NULL);

This is where the master super-smack process waits for its children. My brokenness-sense was tingling. That -1 just looks wrong. And, for Solaris, it is. This first argument, -1, is the pid to wait for. In Linux's wait4, this is implemented as follows (excerpted from the linux man page):

    < -1
        which means to wait for any child process whose process group
        ID is equal to the absolute value of pid. 
    -1
        which means to wait for any child process; this is equivalent
        to calling wait3. 
     0
        which means to wait for any child process whose process group ID
        is equal to that of the calling process. 
    > 0
        which means to wait for the child whose process ID is equal
        to the value of pid.

So now we know what the author meant: Wait for any child process. While not fully documented (which is a bug), Solaris implements a slightly different ruleset:

     < 0
        which means to wait for any child process whose process group
        ID is equal to the absolute value of pid. 

     0
        which means to wait for any child process; this is equivalent
        to calling wait3. 

    > 0
        which means to wait for the child whose process ID is equal
        to the value of pid.

So, on Solaris, wait4(-1, ...) instructs the OS to wait for any child process whose process group ID is 1, while on Linux, it does a wait3(). damn. A final note is that wait4() is not defined by POSIX, the Single Unix Spec, or any standards body I could find. Please, write portable code! One wonders why wait3() wasn't used in the first place. Quickly changing the code fixes the problem.

At this point, I have what appears to be a working Super-Smack on Solaris, and some initial results. I'll intentionally not mention what hardware this was run on, since I've not bothered to perform even rudimentary performance analysis:

    $ super-smack /smacks/select-key.smack 10 10000 
    Query Barrel Report for client smacker1
    connect: max=330ms  min=6ms avg= 59ms from 10 clients 
    Query_type      num_queries     max_time        min_time        q_per_s
    select_index    200000          0               0               8590.39

mpstat(1m) shows that this benchmarks spends a lot of time abusing the system call path, and twiddling bits in userland; it's not clear whether this is really a good test of MySQL performance, since the test client and the database have to fight for CPU resources... All in all, not a terrible night's work. I owe Jonathan a big thanks for his help!
(2005-04-06 11:30:01.0) Permalink Comments [2]
Trackback: http://blogs.sun.com/dp/entry/smacking_super_smack_into_shape