Automated testing of recent OOO320_m1-OOO320_m2 builds is finished. All Cat0 tests were run on English builds by automation team and have been completed. On OOO320m2 there is 1 warning only in automated tests (105967) except on MacOS we have 2 errors and 3 warnings (generated by issue 105275). Issue 105967 is handled in CWS automation320m2 and issue 105275 in CWS impress178. Green state of automated cat0 tests is expected with integration of those CWS'.

The high VTTDI-value is due to an issue 105962. This hinders f_first.bas to be added to QUASTe which is a regression in automated test and is expected to be fixed with integration of CWS automation320m2 also.

See trend created from testresults stored in QUASTe

Stay tuned for updates on testresults.....

tags: automated_tests qa

With availability of Milestone m1 on branch OOO320 (OpenOffice.org 3.2) automation team - as always on a way to a release - starts to test each milestone based on a matrix.
Autotests were made more stable in the past weeks and are now prepared to run 'Greenstate' for this release. In case of errors found by test scripts beginning with m1 they have to be close analyzed as they may be failures in OpenOffice.org. Please contact script owners if an error was found and you need help to analyze. For all potential errors in test scripts a CWS will be created to fix them and have fixes available in following milestone. Of course main goal is to assure OpenOffice.org remains free of issues for QA automation purposes.
All results of automated tests are collected in QUASTe[1] and are available to everyone to see what has been tested. An overview of what should be tested one can take a look on the matrix[2]

English installsets are tested with automated tests of category 0 by SUN automation team[3] with tests of Category 1-3[4] by module owners[3] on selected platforms and languages.

All community members are invited to run automated tests on platforms and languages of their choice. This should be tests of category 0 at first. See matrix what selection fits best to you.

If you have any questions please contact dev@qa.openoffice.org or responsible testers.

Thanks for your support.

tags: automated_tests qa

tags: automated_tests qa quality

With "normal" setup installation it is currently not possible to have more than 1 OpenOffice.org installed on the same windows system in parallel. But for testing purposes it is really necessary to have more than 1 installation available to e.g. check for regressions. For those who looked over the readme.txt coming with the installation set might have been stumbled over the so called 'Administrative installation'. This kind of installation is used to put an installation on a network drive for example to let it be reached within a network. This of course can also be used to 'install' an OpenOffice.org to your local drives. The only disadvantage is you don't have any system integration like registry or menu entries or quickstarter enabled. But if you plan not to test or use those features it isn't required to have a full system integration and also not if you plan to run automated or manual tests based on office features.

Get started:

Installation:

Lets say you've downloaded the (currently) latest OpenOffice.org in version OOO310m9

• Simply extract the archive to a location of your choice (use for example "C:\OpenOffice.org 3.1")
• After all files have been unpacked the 'Installation Wizard' starts => Please cancel it
• Click Start->Run and enter "C:\OpenOffice.org 3.1\setup.exe" /a
  
• Press 'OK' button to execute 
• Now that the 'Installation Wizard' started again please press 'Next'
• On second page please enter the target location ( use for example "C:\test\OOO310m9" )
• click 'next' and wait until installation has been finished

To have installation more comfortable without any user interaction one can use 'msiexec' to install OpenOffice.org. Simply use the following syntax for an administrative installation into a directory of your choice: "msiexec.exe /a  {OpenOffice.org-sourcedir}\openofficeorg31.msi /quiet /qb TARGETDIR={OpenOffice.org-targetdir}"

Tipp: To manage your installations easily one could create some links to different installation directory's on your desktop for example.

Adapt directory for user settings

Each installation must use a different directory for storing user-settings to avoid unmeant effects. This can easily be done by adapting an ini-file called 'bootstrap.ini'
(to be found in directory "C:\test\OOO310m9\OpenOffice.org 3\program\" of your previous made installation)
Simply change the string found for key 'User Installation' in area [Bootstrap] to a path of your choice. In this example we use "C:\test\OOO310m9".

If you now start the OpenOffice.org by executing "C:\test\OOO310m9\OpenOffice.org 3\program\soffice.exe" it uses it's own user-directory.

Adapt changes for automated testing

If you plan to use an installation for automated testing with VCLTestTool please check Wiki-page to learn what settings must be made. To let VCLTestTool find the administrative installation it is required to adapt an option named 'OOoProgramDir'. Simply change this to the program-dir of your OpenOffice.org Installation like "C:\test\OOO310m9\OpenOffice.org 3\program" in this example

Enjoy !

tags: automated_tests qa

Automated testing of recent OOO310_m9 builds is finished. All Cat0 tests were run on English builds by automation team and have been completed. 1 warning on Solaris was submitted. This shows up because of issue #100780 This warnlog will be removed on next milestone ( #100809 ) because it'll be fixed not until OpenOffice.org 3.2

See the trend created from testresults stored in QUASTe

Cat 1-3 testing on release candidates

With availability of milestone OOO310m9 automation team starts testing of Cat1-3 tests on english language followed by updated release matrix. As always all members of localization teams with skills in automated testing are invited to test the localized builds and add their results to QUASTe. Thanks for your support.

tags: automated_tests qa quaste

Automated testing of recent OOO310_m8 builds is finished. All Cat0 tests were run on English builds by automation team and have been completed. All automated tests created the same results as in previous milestone except on MacOS. Here a stopper (Issue 100667) has been found and lead to incomplete testresults. This will be fixed within next milestone.

See the trend created from testresults stored in QUASTe

tags: automated_tests qa quaste

Automated testing of recent OOO310_m6 builds is finished. All Cat0 tests were run on English builds by automation team and have been completed. The trend is absolutely clear the OOO310 version is getting much better and stable from the view of  automated tests. Currently we only have 1 problem on linux and 1 problem on MAC in each case. These problems are identified in testscripts and a fix is available.

See the trend created from testresults stored in QUASTe
VTTDI-Index

Let's keep up the trend.....

tags: automated_tests qa quaste

A new page has been introduced recently that shows only unique errors and warnings for each test. Unique errors (or warnings) are those only occurred on a CWS and not on MWS of same milestone. The list of testresults can be viewed optionally as a tabpage list (default) grouped by each application or as a complete list.

There are 3 different values:

1. Same test created different results on Master workspace:
   The testresults created on CWS differ from testresults created on MWS. Testresults created on CWS should be evaluated accurately and should be fixed in CWS if possible.

2. Same test created the same results on Master workspace:

   The testresults created for CWS are the same like on MWS. If those issues are not fixed within the CWS they can be ignored.

3. Same test successfully passed on Master workspace:
   If the same test succesfully passed on MWS and not on CWS the testresults should be evaluated exactly as those are potential regressions which should not be integrated in MWS.
   In this case autotests or issues in OpenOffice.org must be fixed within this CWS !
   
   

The page can be reached with selecting 'CWS testresults' => 'Platform' and on the following page click 'Analyze errors'.

See QUASTe Homepage for details

tags: automated_tests qa quaste

There is a lot going on currently, all small steps to further improve the reliability of the tests as we are slowly moving toward testbots. Let me start with a few things of more general scope, a special OOo port, a few coding best practies and finally I go into some detail about slots.

Automatic creation of hid.lst

Thorsten Bosbach and Gregor Hartman are currently working on getting the hid.lst generated automatically during the build process. This sounds like nothing new as the hid.lst has been delivered along with the build for a long time. However, there have always been a number of entries that were missing (namely those found in the file cons.txt) and thus it was required to manually fix the hid.lst each time UI changes were done.

Speedup dialog testing on Solaris

OpenOffice.org has a somewhat special way of handling dialogs on Solaris. From a VCL Testtool point of view it looks like the dialogs are available almost instantly when opened. But the controls on the dialog might not be ready yet – there is some sort of lazy loading going on which causes the VCL Testtool to fail when clicking on a button or some other control. In the past we have littered the code with sleep(n) statements which leads to a major waste of time.

Thorsten Bosbach and Gregor Hartmann are currently working on a mechanism to find out whether a dialog is really ready – with all of its controls – or not. As soon as a solution has been found we can begin eliminating those sleep() statements and get considerably faster.

eComStation anyone?

After IBM officially stopped their support for the OS/2 platform in 2006 quite a number of former OS/2 customers switched to eComStation. eComStation can be regarded as an incremental update to OS/2 and has been in development for some years. Today, with version 2.0 nearing completion, eComStation offers support for current hardware and most state of the art technologies. eComStation is maintained by a company located in the Netherlands called Mensys.

And of course there is OpenOffice.org. The eComStation and OS/2 users around the world have been enjoying their OOo versions since 2.4, always just lagging behind a few weeks, sometimes days, after the vanilla OOo releases. And again they are gearing up for the 3.1 release which is expected to be available quite close to the 3.1 launch.

But there still is a lot to be done. The OS/2 and eComStation community is a comparably small one and doesn't hang around on the OOo mailing lists very much. They keep a low profile which results in their CWS usually being ditched last minute before the next release causing the developers a lot of extra work doing the resyncing locally over a longer period of time. The smart question that always comes up and is used as justification for not integrating those CWS is “Who uses OS/2?”. For an Open Source project this is probably the wrong question. Let's hope that this gets better over time.

Who volunteers to do the build for Unununium?

QUASTe

Did anyone notice the recent changes to QUASTe done by Helge Delfs? Helge is constantly enhancing the interface with functionality that really makes life easier for automated testers. You can now compare all your tests for a platform on just one page.

Think positive!

In some cases we – the test script writers – run into unexpected dialogs. Most of them are known and handled. But in many cases the test developer decided to use following snippet of code to handle an unexpected dialog:

if ( not myDialog.exists( 5 ) ) then ...

which is a bad approach. Think positive and use the classic form of

if ( myDialog.exists( 5 ) ) then ...

This is both easier to understand and allows for more clear, simple coding. You usually need an “else” statement anyway.

Refactoring

We – the Sun QA-Automation – are constantly working on the test script code doing smaller and bigger refactoring tasks with the goal to unify out coding style and thereby improve readability and maintainability to lower the barrier of entry for volunteers. Go ahead! Write issues for those tests you find especially poorly written or don't understand at all. But be nice to me and don't look into the framework module. Please! ;-)

One target for refactoring are the possible names for gApplication.  We have quite a number of different names to set the application we are working on. Switching to a Master-Document can be done with gApplication = "MasterDOC", "DOCUMENTMASTER" or "globaldokument" in upper, lower and mixed case. This is annoying and forces script writers to check for all combinations which at least results in three cases to be checked each time one wants to make sure that the current application really is a Master-Document. So in the near future we're going to reduce the number of allowed names.

0.“BACKGROUND”

1.“WRITER”

2.“CALC”

3.“IMPRESS”

4.“DRAW”

5.“MATH”

6.“MASTERDOCUMENT”

7.“HTML”

8.“DATABASE”

9.“CHART”

If – for some reason – someone needs to cycle over applications for a test (which is the case at least in framework) please keep the order of the items. It should be the order already fixed in the file global/input/applications.txt. “BACKGROUND” is a pseudo gApplication used within Framework and its use is discouraged.

Along with these changes we also modify the interface of the global function hToolsOptions() so you can use the function with gApplication directly without the need to code the matching from "WRITER" to "TEXTDOCUMENT".

Slottery 

When dealing with slots in BASIC test scripts every script developer tends to have a facial expression like Mel Gibson in Shakespeare's Hamlet when citing “To be or not to be – that is the question”. Some look like Klaus Kinski under normal conditions.
Slots can be synchronous and asynchronous. And “Ay, there's the rub”. Synchronous slots are easy to handle – they're just fired off and return at some point and you can just go on with the test. Not a big deal, really.
But some of the more prominent slots like “EditCopy”, “EditCut”, EditPaste” are asynchronous.
This frequently results in quite a number of WaitSlot() and Sleep(5) statements before and after the call to the slot.
In this case it is a good idea to live with failure and use a try...catch block within a for...next loop with timeout. This might look as follows:

dim iExitCounter as integer

for iExitCounter = 1 to 10

try

EditCopy

exit for

catch

wait( 100 )

endcatch

next iExitCounter

if ( iExitCounter = 10 ) then

WarnLog( “Failed to copy string” )

endif

In CWS automation002 (DEV300m42) I have added two functions called hUseAsyncSlot() and hClickButton() to handle these problematic slots and buttons. Use them if you have to deal with async slots, they should work in most cases with the exception of toolbars.

tags: automated_tests

In the ongoing effort to speed up the automated tests i had a look at the global filters which are loaded during each test initialization. These filters are defined in gvariabl.inc, called from master.inc and filled in t_filters.inc.

Some time ago we switched from UI filternames to API filternames which resulted in using the API calls FileSaveAs() and FileOpen() instead of opening the file dialogs each time. The result was a considerable speed gain. You might want to read my prior blogs about this subject:

• Faster File Operations
• Filternames For All Locales And Brands

Thorsten Bosbach reworked large parts of the initialization code switching as much as possible to API calls instead of accessing the controls/settings via UI. While doing so we noticed that establishing a UNO connection to the office can be a quite time consuming process as well (though still much faster than UI actions).

So I started profiling and noticed that retrieving the filter names via GetDefaultFilterNames() took somewhere near 15 seconds because the filters were retrieved with API calls one by one.

The first optimization i did was to get the whole bunch of filters in one go from the API reducing the time to some 6 to 12 seconds - still a lot (about 20% of the test case initialization time).

I discussed the matter with my fellow QA colleagues and we decided to further pursue the matter.

The result was: Only one test really needs all filters (f_first.bas) and some filters were used in a grand total of 11 .inc files.

So i suggested removing the call to GetDefaultFilterNames() from master.inc. The details of the modification is fully documented in issue 96341 which is resolved in CWS qascripts02 to be integrated into DEV300m36.

The speed gain is at least 6 seconds per test file of which we have 225 for the time being. Additionally i was able to trash on unused function, moved two functions to the /includes/optional/, a nice little cleanup benefit. If you need the current UI filter names you just have to include the file in your .bas file:

sub LoadIncludeFiles
[...] 

use "global/tools/includes/optional/t_ui_filters.inc"
call GetUseFiles()

end sub

tags: automated_tests