Thursday May 14, 2009

I like unit tests - but running them can be painful. Commit-validation tests which bring up UI are obviously annoying, but even simple unit tests that get in the way. Does this look familiar?



The above menubar should look familiar to anyone on a Mac who's run unit tests for client side Java code. Not necessarily GUI tests, just any test where GUI libraries are loaded.

For every testcase, the test runner fires up a new process, which tells OSX that "I'm graphical, give me focus!", and this steals the focus from the user. The test finishes quickly thereafter, the process quits - and then the test runner goes to the next test and the whole cycle starts over.

This is really painful because I like to have lots of tests. The Python editor support for Netbeans had 600+ tests; the counts for the JavaScript support was higher, and the Ruby support even higher than that. Whenever I run tests, I basically have to fight with my computer to get focus. Forget trying to write anything - every second or so my keystrokes get stolen as the next test grabs focus - so I've gotten in the habit of using the time for browsing, since I'm mostly reading, and I can handle clicking a link a second time if the first click got lost. But every now and then somebody will ping me on instant messaging - and it's maddening trying to respond while this is going on.

If this is sounding painfully familiar to you, I have good news. I've finally figured out a setup where this is no longer a problem!

The key discovery was that I can run my tests from another account on the system. With OSX' fast user switching, I can switch to the alternate account, launch the unit tests, and return back to my regular account where the tests won't interfere with display focus. In order to let me run my tests from that other account, I just open a terminal there and su -l tor in the shell to run all the commands (or NetBeans) as myself.

This was a huge improvement since it removes the 10-30 minutes testrun downtime. But it had some disadvantages - first, I don't like running tests from a shell, and second, it's hard to know when things finish - and switching back and forth to check is annoying since I always have passwords on my accounts so the machine isn't open.

So the second step was to set up Hudson (a continuous integration server that is trivial to setup, and has a huge number of plugins which makes graphing code coverage, unit tests, findbugs results, integrating with version control systems etc trivial. And it's not just for Java developers.)

Instead of logging into the other account for each test run, I log into the other account once, and start up Glassfish with Hudson running. From now on I can access, configure and launch builds right from my own browser in my primary account. The key step here is that Glassfish was started from the secondary account, so its primary display is associated with the builder account. When my build in Glassfish gets to the test stage, it's actually doing the display connection just as before, and if I log into the secondary account, I get the annoying focus flashing just as before. Look - the tested process is a user visible application in the dock:

Another improvement which really helped is the "File System" version control plugin for Hudson. I want my Hudson builds to build my current working copy. I don't want to check my code into Mercurial (even a local repository) just so that Hudson can grab the code and build it. I want Hudson to build my current changes - my current edits. After all, I'm trying to test them before checking in! And I discovered that there is a plugin which will let me do that - it's just a "file system" version control wrapper - which means you just point Hudson to your local directory, and off it goes. When the build starts, it makes a quick disk-copy of the source tree. Even though Mercurial cloning is pretty fast, this is even faster. The disk copy also lets me specify a filter of files to exclude, so I had it ignore *.class files. The diskcopy only takes 10 seconds or so before the build kicks off, and it's building a snapshot of my current in-progress, edited working copy! (It can also just update the copy based on changed filestamps - that's even faster, but it didn't seem to correctly delete removed files, so I let it start from scratch each time.)



(Note - this plugin isn't in the catalog that you can browse right from within the Manage Plugins page within Hudson; I downloaded the .hpi file from http://wiki.hudson-ci.org/display/HUDSON/File+System+SCM and installed it in the Advanced tab.)

The final ingredient is the new Hudson support in NetBeans 6.7. I don't even have to go to the browser to kick off build jobs. I just point NetBeans to my Hudson server once, and from then on I have full Hudson integration. When I want to run my tests I just select Launch job:



I get notified if there's a problem:



I can look at failures and logs:



I can see build logs etc. directly in the output window, and hyperlinks warp to directly to files - to the files as they were in the build, not the current state:

So to recap - with this setup, as I'm editing my code and I want to check the tests, I just right click on a node and say "Start Job" - and off it goes without bothering me at all - no more focus interruptions, and no more GUI windows popping up from interactive tests. It's trivial to check the results. And it's even added one more level of convenience: I have multiple projects, each with unit tests, and from the IDE I couldn't have them all run with a single gesture. My build job does that.

I'm really stoked! I was at one point able to do this when I was working on Linux and Solaris by setting my $DISPLAY variable and doing tricks with VNC. But that still required my tests to run in a console - which made interpreting the results sucky.

If you haven't played with Hudson, try it - it's unbelievably easy to set up. Just download the Glassfish appserver and install it, download the Hudson .war file, and drop the hudson.war into the autodeploy directory of the appserver, and browse to localhost:8080/hudson. Once you're there you can install plugins (under Manage hudson), point to your local installations of the JDK, ant, etc., and configure your build jobs by running scripts, launching maven scripts, writing ruby scripts, or obviously running ant scripts.

Some final miscellaneous tips:

1. I don't want Time Machine to back up my builds trees, or Spotlight to index data in these directories, so I went to the TimeMachine preferences and had it exclude the ~/.hudson/jobs/ directory.


2. I did the same thing for Spotlight - but unlike the Time Machine preferences, there was no checkbox to "Display Invisible Files" (e.g. files that start with a dot, such as .hudson) in its file chooser. Here's a tip I didn't learn until recently: When a Mac filechooser has focus, you can press slash (/) - and this will open a text field where you can directly type the path it should jump to. I typed /Users/tor/.hudson and from there I was able to select the jobs directory to exclude.


3. You might be tempted to skip the "filesystem version control" plugin and just have your build symlink to your working copy. Be careful; if Hudson is configured to delete older builds you might find yourself without your source code. I'm not saying it will follow your symlinks - Java has support for symlinks now - but I haven't tried it, and I have been bitten by ant in the past where it decided to follow symlinks in its zeal to delete recursively!


4. I recently discovered that you can reorder the build steps in a Hudson job. The little graphic to the left of a build task is a handle you can just drag to reorder!