Thursday April 13, 2006

Jini: 300,000 test-executions strong

Testing the Jini Technology Starter Kit is a lot of fun. It's also a lot of work.

Since version 2.0 of the starter kit (when a new plug-able implementation of the Java RMI programming model was added -- named JERI), the test matrix has really exploded! For instance, here is a sampling of the different options that we test:

Operating Systems: 1 Mac, 3 Linux, 2 Windows, and 4 Solaris
JDKs: Java 1.4.2 and Java 5.0
Hotspot VM: client and server
RMI Transport: default, JRMP, JERI, HTTP, HTTP thru a proxy, HTTPS, HTTPS thru a proxy, JSSE, and Kerberos
Persistence: activatable, persistent, and transient modes of all the starter kit services
Assertions: enabled and disabled
Logging: default level and finest level
NIO Usage in JERI: enabled and disabled
Mode: local host execution and distributed execution across many hosts

Taking the cross product of the above options together with the 1600 or so unique tests that we have, we end up running almost 300,000 tests over a full two week test cycle (roughly 25,000 tests per day). Since the entire test process is automated, we simply repeat this test cycle every two weeks. For all the gory details, check out the test matrix for Jini Technology Starter Kit v2.1 (released 10/2005).

Our test lab is comprised of roughly 100 machines with 2 or 3 different operating systems installed on each machine. With a distributed system of this magnitude, the test scheduling software is (of course) based on Jini itself...but that's another blog entry.

Finally, there are a couple challenges presented by such a large volume of regularly repeated tests. First, the size of the test results from a single day can be many gigabytes. Storing and archiving this amount of data is burdensome. We've been able to decrease this to megabytes by throwing away logs for tests that pass. Second, many of the tests wait for a specific amount of time for some event to happen or not happen (we're testing a distributed, event drive system after all). These timing dependencies in the tests can be (too) sensitive to network load, test host load, a specific RMI transport, and many other factors. The test failures caused by these timing issues are a real nuisance. A lot of time can be wasted chasing down these false negatives.

The Final Word...
Why write something in five days that you can spend five years automating?
- Terence Parr

Posted by nigequality ( Apr 13 2006, 04:00:00 PM EDT ) Permalink Comments [0]

Wednesday April 05, 2006

Compute Server: locally debug your Sun Grid app

If you're using Compute Server and you're not already a test driven developer, then this blog is for you! Time to roll up your sleeves and test the Compute Server project that you've created. Besides, it will save you from needlessly spending $$$ debugging your project on the Sun Grid Compute Utility.

Assuming you've already downloaded the latest version and installed the NetBeans 5.0 plug-in from the Compute Server project, here's a quick run down on debugging your app within NetBeans before submitting it to the Sun Grid Compute Utility.

The plug-in contains a Compute Server execution environment that can run on your local machine. This execution environment approximates the Compute Server execution environment on Sun Grid, with the exception that only one task will be executed at a time.

Once you have written your Compute Server application, you are ready to debug it locally. This is done by right-clicking on the project name in the Projects pane and then selecting Debug Project as shown below (alternatively, if your project is set as the main project, you can click the Debug Main Project toolbar icon).

Debugging a Compute Server project will automatically cause a build of any classes that have changed since the last build or debugging session of the project. In addition, the project will be packaged using the values taken from the project's property sheet.

When the debug session commences, an execution environment will be setup on the local machine that approximates the Compute Server execution environment on Sun Grid, with the exception that only one task can be executed at a time. As you would expect, the session will stop at break points set on any project source file in the NetBeans IDE.

On completion of the run, the run output is compressed into a ZIP file just as it would be on Sun Grid. Then, if an output processing class has been defined, it's main method is automatically called, passing the ZIP file as the first argument.

Clearly, lots of details have been left out here. To understand the Compute Server programming model, including tasks and output processing classes, refer to these information sources:

• My previous blog entry on java.net
• Online help within NetBeans (see the Compute Server Projects topic)
• Javadoc found in the Compute Server project download file
• The users@computeserver.developer.network.com mailing list

Happy debugging!

The Final Word...
Quality Assurance: because CPUs deserve better!

Posted by nigequality ( Apr 05 2006, 05:37:43 AM EDT ) Permalink Comments [2]

Thursday March 23, 2006

Compute Server: grid computing for the rest of us!

What makes distributed computing so darn hard? Well, to start with, so much can go wrong. Networks change. Network speeds fluctuate. Networks fail. Nodes fail. And more. Are you testing all these scenarios? Time has shown that we tend to make a number of assumptions when building and testing distributed applications that, in the long run, turn out to be false. Ignoring the falsity of these assumptions leads to unreliable systems.

Well, reliable distributed computing on the grid just got a whole lot easier for Java developers. Today we released an early access version of Compute Server. So instead of fretting over all the details of reliably distributing work to different nodes, coordinating processes, monitoring and reacting to network and node failures, Compute Server takes care of these details for you so you can focus on your computations. From the description of our project:

Sun's Compute Server technology aims to enable Java developers to easily and efficiently use the Sun Grid Compute Utility as a platform for the distributed execution of parallel computations.

So what kinds of computations? Of course there's the usual stuff -- scientific, financial, image processing, etc. But I'm a tester -- so computations could also be test suites, right? Right. But that's a future blog entry.

Using the included NetBeans 5.0 plug-in, Compute Server enables you to split your computations (execution of large test suites in my case) into discrete tasks and farm those tasks out to the CPUs on the Sun Grid Compute Utility. Three simple steps to get started:

   1. Get NetBeans 5.0
   2. Get Java 5.0
   3. Get Compute Server 0.1 (Early Access)

The documentation in the Compute Server download ZIP file should take you the rest of the way (if it doesn't then please let us know). I suggest starting with one of the included example NetBeans projects. Again, this is early access -- lots more features to come.

Oh, and above all, send us your feedback to help shape its future (mailing list subscription required).

The Final Word...
Too few of us question our assumptions; I unquestioningly assume that every assumption should be questioned.
-- David Parnas

Posted by nigequality ( Mar 23 2006, 08:00:00 AM EST ) Permalink Comments [0]

Wednesday March 22, 2006

Sun Grid: the ultimate test bed

It's a great day! It was just announced that the Sun Grid Compute Utility is open to the public.

As a software quality engineer for a number of advanced technologies for distributed systems, I can't think of a better test bed for qualifying my distributed applications! Just think of all the mundane things I don't need to do anymore...

... I don't need to submit and wait for large purchase order approvals
... I don't need to order, receive, or track test equipment assets
... I don't need to install and patch operating systems on that test equipment
... I don't need to troubleshoot problems or service test equipment
... I don't need to manage power, a/c, or space requirements for test equipment

In a matter of a couple hours or less, I can have access to hundreds of CPUs. Obtaining resources for meaningful stress tests, performance tests -- even unit tests -- have never been so quick and easy!

Granted, it's not as simple as just uploading my application-under-test, my JUnit test suites, and letting it rip...but stay tuned! Something might be coming to make it just that easy...

Posted by nigequality ( Mar 22 2006, 11:00:00 AM EST ) Permalink Comments [1]