What problems are faced in Java GUI automation? To help the Java Quality team understand the hurdles faced in
automating Java GUI tests, I wrote up a paper detailing the
problems. This was a couple years ago, and I want to summarize it
here. The purpose was to document the challenges we face in
automating the Java GUI tests, and guide us in choosing the tools we
now use in the automation work.
What I came up with was a list of attributes ... each attribute
contributing its own automation challenge. I kept the attributes
as orthogonal as possible, to let them be considered independantly
where possible.
Has a GUI: While this might seem nonsensical, not all "client side" testing requires a GUI. Therefore, if a test application does have a GUI, then the challenge is to have an automation tool.
Graphics
rendering destinations: In Java there are at least
three destinations upon which Java2D can render. To test Java 2D,
one must test all those destinations. What are these
destinations? They are: a) on-screen via a Component, b)
off-screen to a BufferedImage, c)off-screen to a printer. Of
course, for each there are several alternatives depending on how you
configure the destination
The automation challenge here is to capture the output for each rendering destination.
Keyboard and Mouse interaction with GUI: Having to interact with the GUI is a step up in complexity from Has a GUI,
since a test application could have a GUI, but you don't interact with
that GUI. For example, a test application could be a simple
container for a rendering test, and you simply run the test application
and do a screen capture.
The automation challenge is to have an automation tool capable of
sending keyboard and mouse events. I've listed several on this page.
One of the big concerns in this area is to ensure the events sent by
the automation tool are indistinguishable from keyboard or mouse events
sent when a human bangs around on their computer.
In Java (AWT, Swing) it is quite possible to inject Event objects into
the AWT Event queue. A Swing application will respond to those
events, and in some test tools they claim it is a big feature to be
able to test a Swing application this way. But this is an
incomplete test.
Why? Because the events haven't traversed the normal code path
used when humans interact with the application. There is a lot of
processing of incoming events done by AWT, and these events eventually
turn into the objects that go through the EventQueue. If you
inject events into the EventQueue, that code in AWT is not traversed,
and you are missing out on possible bugs inside AWT.
Complexity of the GUI:
Obviously, the more complex a GUI is, the more difficult it will be to
automate it. A highly complex GUI presents automation challenges
all over the place.
However, complexity actually hides in the open. For example, if
you want to click on the arrow button in a scroll bar, how do you do
it? Every automation tool depends on being able to answer this
question:
For things which appear in the AWT Component tree (traversed by calling Component.getChildren) it is trivial to determine their location, because you can call Component.getLocationOnScreen. However some objects you want to interact with do not appear on the Component tree, such as the scroll bars.
Another complexity comes when you parent someting in a Viewport (JScrollPane) component. How do you know which parts of the child component are actually visible? How do you know how far to scroll the viewport to make something visible?
The text components like JTextArea present an interesting challenge. What if you want to use the mouse to select a specific range of text? These components give you no clue the x,y coordinate of the text they've drawn on-screen.
The AWT MenuComponent's do not offer a getLocationOnScreen method, meaning you don't know where they are.
In AWT there are a couple native dialogs, which are opaque objects. If you want to automate interactions with those dialogs, the only choice is to write some JNI native code to call the native windowing system and discover the component location that way.
For testing Applet's running in a browser, several types of tests require interactions with the browser. One cannot know where the browser is, unless you go to the trouble of writing a tool such as the one I discussed yesterday. If you have more than one Applet you want to test together, then that offers its own automation difficulty, because while the Applets are all running in the same JVM, they are in separate AppContext's. There are a few security walls one must breach to discover component locations in another AppContext.
A multiple-window application is more difficult to test than a single-window one. So long as the multiple windows are in the same Java VM then it's relatively easy, but like I said for the Applet/Browser case if the application involves native windows then you do not know where the native windows are.
If you are testing several Java Applications together, each in their own JVM, this is a challenge similar to the native window challenge. A test can only "see" the Components in its own JVM, and is just as blind to Components in other JVM's as it is to the Components of native applications.
Drag & Drop is relatively simple to automate, but presents a few challenges. One biggie is, again, often you want to drag something to a different application. e.g. Dragging a URL into the Location Bar of a browser.
Requires Visual Verification: You can do a lot of testing without verifying that the application looks right while running. There's a risk here, of course, that you might miss rendering problems. However, visual verification is tough.
The simplistic approach is to take screen shots, and then verify those screen shots against later builds of your product. But ... how do you organize the screen shots making it easy to maintain them? And easy to update them when needed? The maintanence of a set of golden images, as we call them in the Java team, is actually very expensive.
Then, which platform do you take the screen shots on? All platforms? The more screen shots you have, the more expensive is the problem of maintaining the golden image repository. Also, you may find subtle differences between graphics cards, even on the same OS/CPU.
What about when (not if) the application GUI changes. You have to reshoot all the golden images, adding to the maintenance cost. If the appliction is being actively developed, it might be changing weekly, meaning you have to recapture all the images each week. The more often you recapture the images, the more expensive, and the harder it is to get a return on investment.
Testing your application under different look&feel's is an obvious use for visual verification.
One thing you cannot test under visual verification is whether you have the correct cursor. Robot cannot capture the cursor image.
Graphics Complexity: If you are doing visual verification, there are added complexities that can arise.
The most obvious are animations. The problem is, where in the animation cycle do you do the screen capture, and how can you predict which of the possible images you will capture? You can't, in the general sense. Though in some specific cases you can change the animation algorithm allowing you to single-step it, and then do screen captures at each/most steps.
Text, fonts, international languages, oh my: Here be dragons (from China). This problem is of exponential proportions.
The niaive approach is to take a given character string (e.g. "the lazy brown fox jumped over the lazy dog") and render it in each font, in each font size, in each styling, with each Java2D perturbation you can think of, on each operating system, in each locale/language, in each color, and in each phase of the moon. Or something like that. The point is that you quickly get to huge numbers of tests, because of combinatorial mathematics.
What to do? Be smart, and test the most strategically significant combinations.
Sound rendering accuracy: Some applications make bleeps and bloops as sound cues. For example, Quicken makes a cash register sound every time you enter a transaction. Other applications play audio as its main purpose (e.g. an MP3 player).
If you want to test the accuracy of sound playback, well, good luck. You have several possible innacuracies in the mix. The first is that the sound card is going to translate the digital sound instructions into analog signals. The second is when the analog signal is played on a speaker, the speakers have varying accuracy. The third is when you go to digitize the sound, you're doing it through a microphone and an audio digitizer, each of which have their own innacuracies. To verify the test, you'd have to play the sound, digitize it, and "compare" the two files, but with all the innacuracies you've got a tough row to hoe in doing the comparison.
You can bypass some of the innacuracies by connecting the line output of the sound card to the line input of another computer. But there are still innacuracies.
(2005-04-13 16:08:00.0) Permalink Comments [0]
