Thursday April 13, 2006

Jini: 300,000 test-executions strong

Testing the Jini Technology Starter Kit is a lot of fun. It's also a lot of work.

Since version 2.0 of the starter kit (when a new plug-able implementation of the Java RMI programming model was added -- named JERI), the test matrix has really exploded! For instance, here is a sampling of the different options that we test:

Operating Systems: 1 Mac, 3 Linux, 2 Windows, and 4 Solaris
JDKs: Java 1.4.2 and Java 5.0
Hotspot VM: client and server
RMI Transport: default, JRMP, JERI, HTTP, HTTP thru a proxy, HTTPS, HTTPS thru a proxy, JSSE, and Kerberos
Persistence: activatable, persistent, and transient modes of all the starter kit services
Assertions: enabled and disabled
Logging: default level and finest level
NIO Usage in JERI: enabled and disabled
Mode: local host execution and distributed execution across many hosts

Taking the cross product of the above options together with the 1600 or so unique tests that we have, we end up running almost 300,000 tests over a full two week test cycle (roughly 25,000 tests per day). Since the entire test process is automated, we simply repeat this test cycle every two weeks. For all the gory details, check out the test matrix for Jini Technology Starter Kit v2.1 (released 10/2005).

Our test lab is comprised of roughly 100 machines with 2 or 3 different operating systems installed on each machine. With a distributed system of this magnitude, the test scheduling software is (of course) based on Jini itself...but that's another blog entry.

Finally, there are a couple challenges presented by such a large volume of regularly repeated tests. First, the size of the test results from a single day can be many gigabytes. Storing and archiving this amount of data is burdensome. We've been able to decrease this to megabytes by throwing away logs for tests that pass. Second, many of the tests wait for a specific amount of time for some event to happen or not happen (we're testing a distributed, event drive system after all). These timing dependencies in the tests can be (too) sensitive to network load, test host load, a specific RMI transport, and many other factors. The test failures caused by these timing issues are a real nuisance. A lot of time can be wasted chasing down these false negatives.

The Final Word...
Why write something in five days that you can spend five years automating?
- Terence Parr

Posted by nigequality ( Apr 13 2006, 04:00:00 PM EDT ) Permalink Comments [0]

Wednesday April 05, 2006

Compute Server: locally debug your Sun Grid app

If you're using Compute Server and you're not already a test driven developer, then this blog is for you! Time to roll up your sleeves and test the Compute Server project that you've created. Besides, it will save you from needlessly spending $$$ debugging your project on the Sun Grid Compute Utility.

Assuming you've already downloaded the latest version and installed the NetBeans 5.0 plug-in from the Compute Server project, here's a quick run down on debugging your app within NetBeans before submitting it to the Sun Grid Compute Utility.

The plug-in contains a Compute Server execution environment that can run on your local machine. This execution environment approximates the Compute Server execution environment on Sun Grid, with the exception that only one task will be executed at a time.

Once you have written your Compute Server application, you are ready to debug it locally. This is done by right-clicking on the project name in the Projects pane and then selecting Debug Project as shown below (alternatively, if your project is set as the main project, you can click the Debug Main Project toolbar icon).

Debugging a Compute Server project will automatically cause a build of any classes that have changed since the last build or debugging session of the project. In addition, the project will be packaged using the values taken from the project's property sheet.

When the debug session commences, an execution environment will be setup on the local machine that approximates the Compute Server execution environment on Sun Grid, with the exception that only one task can be executed at a time. As you would expect, the session will stop at break points set on any project source file in the NetBeans IDE.

On completion of the run, the run output is compressed into a ZIP file just as it would be on Sun Grid. Then, if an output processing class has been defined, it's main method is automatically called, passing the ZIP file as the first argument.

Clearly, lots of details have been left out here. To understand the Compute Server programming model, including tasks and output processing classes, refer to these information sources:

• My previous blog entry on java.net
• Online help within NetBeans (see the Compute Server Projects topic)
• Javadoc found in the Compute Server project download file
• The users@computeserver.developer.network.com mailing list

Happy debugging!

The Final Word...
Quality Assurance: because CPUs deserve better!

Posted by nigequality ( Apr 05 2006, 05:37:43 AM EDT ) Permalink Comments [2]