RoboGeek

Testing Java GUI applications & multiple platforms

As I said in earlier blog entries, I work in the Java SQE (J2SE) team. It is the SQE team that develops functional tests of the Java functionality. These tests are outside the scope of the JCK tests the Java customers use to validate their Java implementations.

My specialty is with tools for automating GUI oriented testing. For example I am partially responsible for the java.awt.Robot class.

This background has given me an interesting conclusion to share. This conclusion is likely different than our marketers want to have said. So I should say that this is purely my opinion.

What is this conclusion?

It's very simple - the presence of (commercial) GUI test automation tools have a large impact on the potential success for an operating system.

Why? It has to do with the relative testability of applications built for that operating system.

Why? The rule of thumb for GUI application testing is "Testing is never finished, only abandoned". This means that any application development team has a certain budget for testing, and once that budget is exhausted they won't test further. The effect of automated GUI tests is to amplify the amount of testing that can be done with a given budget.

Here is the core of my contention. Windows is the only platform with commercial GUI automation tools. This makes for an advantage to Windows since application developers can automate their tests, and thereby do more testing with their budget. Similarly, on a platform lacking commercial GUI automation tools the application developers won't be able to automate their tests, and hence their application won't be quite as well tested.

Java, however, is in a different situation.

Java has the java.awt.Robot class, and there are several GUI automation tools for Java. I have a javapedia page here which goes over those tools. This makes a Java application testable across platforms, even when there is the general lack of GUI automation tools.

That's a good thing, yes? Mostly. Turns out there is a limitation in that the Java GUI automation tools rely on being able to use java.awt.Component.getLocationOnScreen(). But what if you want to interact with another application besides the Java application?

Like, what if you want to test an applet, and some of your scenarios require interacting with the web browser containing the applet? Bzzzzt!! No luck, because that browser is a foreign application. The Java GUI automation tool is unable to "see" the browser components.

What if you went to the native operating system calls and, through the magic of JNI, brought information about the GUI components into Java?

On an X11 system you can write some native code using XQueryTree, and thereby find locations of components. However XQueryTree only gives you a set of rectangles, and doesn't give you any clue of what those rectangles are. Does anybody have a time machine so we can go back to the mid-80's and beat someone up? If you don't know what those rectangles are, you don't have any ability to know which rectangle to click on.

On a Win32/64 system native code (EnumWindows and EnumChildren) is able to give you more information. It's not perfect, but it's better than nothing.

But the cruelest twist is that on every platform there are several GUI toolkits which do not form their components in the traditional way. The Mozilla Application Object Model is a great example of this, where inside their toolkit they draw their own widgets purely as graphics. This means not even the Windowing System is going to know where Mozilla application components are. Even if one were to write a JNI module to get native windowing system data into Java, that module would also be unable to see the native components.

The bottom line for Java application developers is that ... so long as your application exists within a single Java VM, and doesn't interact (much) with native applications, you'll be able to automate tests against it. But if your application interacts with native applications you're stuck with manual testing for those scenarios.