A huge project like OpenOffice.org requires quite a lot of testing. The more complex it gets the higher is the risk that a simple change to the code breaks other seemingly unrelated parts. Our approach here at Sun Engineering was to make a number of tests, the so called "Release tests", mandatory for each Childworkspace (CWS) that is going through our QA.

This has proven to be quite useful, the number of regressions introduced especially in times of mass-integrations (close to UI freezes/feature freezes) has decresed significantly. However, this approach also has a serious drawback. And that is time.

The challenge 

Even though we've decided only to run a "small" selection of our tests, mostly the so called "update tests" which focus on resources, we end up having a running time of more than 18 hours. This time is spent on each CWS for at least two platforms. And that is too much for overnight testing.

This becomes even more problematic when considering that the QA Automation is outnumbered by a factor of 10  which can lead to a large number of Childworkspaces being handed over from development and thus turning QA into a nasty bottleneck as we calculate with an average of about 4 days stay in the QA for each CWS to get from "Ready for QA" to "Approved by QA".

Part of the solution is to significantly speed up the time it takes to complete the automated testing.

Possible solutions

One solution can always be to just upgrade the hardware stack the tests are run on. Use more machines, use powerful machines. Unfortunately this does not scale. First of all there is the human factor - you cannot keep track of 20 simultaneous sessions every day and live. Especially not if you have to do the manual testing for the single issues at the same time. And then there are other technical constraints like network performance, a broken switch here and there.

Fast hardware is nice but there is one area where there's much more to gain: The test scripts themselves.

The OpenOffice.org QA has been using automated tests for many years now. During this time the environment has evolved. The Testtool has been enhanced by a number of features and the new Testtool  (based on a m226 or later) again brings a number of features to make the tests faster and more reliable.

So first of all: Why do the tests not run at maximum speed by default? The answer is simple enough: Many dialogs or actions are executed asynchronously, multi threaded - whatever. This means that an action is triggered and from the triggers point of view it is immediately done. The spell checking might be a good example for this: You have a huge document, enable the spell checking which starts to mark all the unknown words in your text. While this happens you can go on writing at the end of your text and at some point the spell checking reaches your current position.

This has of course impact on the Testtool as it cannot determine when such a background task is completed. The trivial solution in the past was to use the BASIC wait() instruction, or - because the test script developers knew that this was a shaky solution - the wrapper sub sleep( seconds ).

The drawback is obvious. At the time those tests were written we had very slow hardware compared to what we use today so the delays were chosen long enough for the slow machines. But this time is a fixed loss of time. Just looking at the test "firs.bas" reveals that more than 53% of the time is spent waiting.

Using method .exists() and the new built-in command WaitSlot()

In the past our code looked pretty much like this:

MyDialogButton.click()
sleep( 2 )
kontext "MyDialog"
MyButtonOnDialog.click()
sleep( 10 )
MyDialog.close() 

Using the .exists() method already saves up to two seconds:

MyDialogButton.click()
Kontext "MyDialog"
if ( MyDialog.exists( 2 ) ) then

MyButtonOnDialog.click()
sleep( 10 )
MyDialog.close() 

else 

...

endif 

Just using .exists( 2 ) makes the Testtool wait until "MyDialog" becomes available but for a maximum of 2 seconds. So the best case is that you save 2 seconds because the dialog is available almost instantaneous. An additional benefit is that - according to almost any programming guide in the world - an if() statement is always followed by an else statement which gives you a wonderful opportunity to do a warnlog( "MyDialog is not open after clicking on MyDialogButton" ) - whoever runs your test and runs into this warning is going to be grateful because the alternative is an errormessage like "error: Non existing object".

New with the latest Testtool is WaitSlot(). The function is defined as follows:

int WaitSlot( time in milliseconds )

The default time is 1000 milliseconds. To get a status two helper constants have been introduced as well

WSFinished (slot finished) and

WSTimeout (slot blocked)

so a normal usage could be:

if ( WaitSlot( 2000 ) <> WSFinished ) then

 warnlog( "Slot not finished within 2000 milliseconds" )

endif

Please note that WaitSlot() does not take any control or dialog to be connected to but it merely waits for any slot to finish. This means our sample could now look like:

MyDialogButton.click()

Kontext "MyDialog"

if ( MyDialog.exists( 2 ) ) then

MyButtonOnDialog.click()

if ( WaitSlot( 10000 ) <> WSFinished ) ) the

warnlog( "MyBottonOnDialog did not finish within 10 seconds" )

else

printlog( "MyButtonOnDialog executed ok" )

endif

MyDialog.close() 

else 

... 

endif 

What WaitSlot() and .exists() have in common is that they only wait until the slot is completed or the dialog has become accessible. So the possible time gain is about 12 seconds in this tiny sample.

Where to gain big time

These functions have been applied to many places within the framework module by now. And - in combination with some minor fixes to declarations (some dialogs have faulty entries in the .win lists) i was able to speed up  things considerably. Alone the basic_formcontrols.bas test is down from about 50 minutes to 5!

The most effective place to apply those mechanisms is certainly the global module. I've done some tests on the function hMenuSelectNr() and hOpenMenu() - they cause an unnecessary delay of at least five seconds each time called. Using WaitSlot() removed the delay without sacrificing stability. Five seconds doesn't sound like much. But these are only two frequently used functions and it sums up.

Ok, let's do it everywhere and we're fast as lightning!

Unfortunately no.

I've already discovered a couple of areas where neither functions are of any help. First of all: Toolbars. When you return from any dialog to a document and access a toolbar directly, chances are that your button is disabled when you click on it. You cannot use WaitSlot() here because the toolbars have a life of their own - the items are enabled one by one in a random order and you never know for sure whether you button is ready. So currently a small hard coded delay appears to be appropriate - sleep( 1 ) or maybe just a wait( 100 ) should be sufficient, wrap it in a try ... catch ... endcatch block if unsure.

Another place is treelists. Take the Script Selectors as an example. When opening this dialog we have a list with all macros/scripts known to the office program presented in a tree. Some of the scripts are written in Java, others are BASIC code. To populate the list Java is triggered and takes time to be loaded. So when you open the dialog it might easily take up to 5 seconds before the list is fully populated. This cannot be covered by WaitSlot() either.

Loading/Saving documents is the third big area where we cannot use the new functions. This is especially sad because these are the most frequently used functions within our tests. We've been trying to solve that for a long time now but no really good solution has emerged yet. But stay tuned, changes are coming in this area as well.

tags: qa