..about what?! tm's weblog

Lots of folks in NetBeans started to use Hudson as a tool for continuous integration, or just for building NetBeans from various clones, or to build test, whatever you can imagine. Just take a look at Deadlock machine and you will see a plenty of NetBeans related jobs. We (NetBeans QE) recently got an idea to utilize Hudson as our testing infrastructure. Yes, NetBeans have a lot of automated tests that are being ran daily and test results are evaluated for potential problems. So you have tests, you have machines, but you still need an infrastructure to connect this together. We chose Hudson, because it's a great tool, and what's really cool - is extensible by various plugins (available on Hudson's web, or you can write your own if you wish). The basic idea was to use Hudson as "master-slave" architecture, where you simply set up on master what do you want to run, and master distributes the work to the slaves, that execute the jobs. More detailed approach to this - configure on master which tests do you want to run on which platforms, and they will get executed on appropriate slaves. And getting one level lower - to achieve this, you have to have something we called generic Ant script, which will provide all the stuff needed to execute test:

1. Download IDE - the IDE build which will be tested
2. Download NetBeans Test Distribution - Test Distribution contains binaries of all tests from NetBeans repo.
3. Run selected tests - Specify which tests from Test Distribution will run in tested IDE

So i wrote and Ant script with plenty of useful targets to provide this functionality (Yeah, by this I learned a lot how to write Ant scrips:-) ). This Ant script is generic, and all the stuff like where to take IDE build, where to take Test Distribution, which tests run, what to do with results etc. are controlled by passing corresponding properties. Once we have this working, the only thing we have to do, is to distribute the Ant script to machine we want to run tests on, and provide some better UI than command line for the main properties, which control the test execution. Therefore me and Max Sauer extended Hudson by Test Run Plugin (sources are not public right now), that enables Hudson to copy our generic Ant script on target slave machines and extends job configuration with our panel for setting properties for testrun. With all this stuff working, people can simply log in on master, select one or more slave machines on which will tests run, configure which build will be tested by which test (through nice GUI :-) ), run the job and take a nap while watching the progress of tests execution on target machines. When test run is finished, you get a nice overview of test results (yes, built-in feature of Hudson).
The main benefits we take from Hudson (aka features we really like):

• Master-slave architecture - you can have various machine with various architectures/OS added into testing farm.
• JUnit results processing - nice HTML, results trends, charts,...
• Scheduler - triggering of testrun by another job (which can listen e.g. if new production build is available), or "cron-like" settings.
• Email notification - send email if tests do not pass.
• SCP plugin - we can send test results to remote machine that can parse them and store into DB.
• Authentication - simply configure and point Hudson to your LDAP server, really out of box functionality.
• Multiple JDK installations - specify location where are JDKs located and voila - you can run whole matrix of tests - one axis is JDKs, second is platform.
• ...many many others:-)

And conclusion? Hudson is not only contentious integration tool, it can be almost whatever you like.