EIS just got a couple of new features.What´s new is that there are now a couple of tests that can be done on a OpenOffice.org ChildWorkspace which each have a status which is used to caclulate an overall status for the OpenOffice.org ChildWorkspace. Tests are either short test which are run automatically by EIS each time the user visits the new "Tests" tabpage of a ChildWorkspace or are longer running tests which are queued for execution on dedicated test computers. The short automatic tests do test such things like if all tasks at the ChildWorkspace are already fixed or if the release set at the ChildWorkspace is allowed for the MasterWorkspace the ChildWorkspace is based on. Longer running tests queued for execution on dedicated computers are AutomationCAT-0 tests, ConvWatch-Tests and Performance Test. Additionally also a couple of buildbots have been integrated which do now report their status of the build back to EIS. For tests being queued to dedicated computers to be run on this does mean that you might have to wait some time more for the results because tests on other ChildWorkspaces might be before yours in the queue. On the other hand using dedicated machines for the tests does make tests better comparable and does eliminate some problems we currently have with running automated tests on different hardware.

AutomationCAT-0 tests

these tests are run on dedicated computes, they need approximately 8 hours and use the VCLTesttool with a couple of selected test scripts (called the CAT-0 tests) to check the quality of an OpenOffice.org installation. At the end of the test the status is in EIS is set to "finished needs review" and a link to QUASTe is being provided where the QA Rep can look at the details of the many testscript results and after review can finally set the status for the test in EIS to "ok" or "failed".

 Performance Tests

these tests are also run on dedicated computers, they need round about 15 minutes and do compare loading of some sample test documents and startup of  OOo modules without opening a document for an OpenOffice.org of the current ChildWorkspace with the result of the same tests for an OpenOffice.org of  the corresponding MasterWorkspace. At the end of the test the status is reported back to EIS and also an attachment has been save on the ChildWorkspace which contains details about the tests. This is done to find potential regressions in performance on a ChildWorkspace. The test has some tolerantable difference in performance that is allowed and could as well be cause by other factors than those in responsibility of the source code changes on the ChildWorkspace. AutomationCAT-0, ConvWatch and Performance tests currently need to be able to find installation sets for the CWS on Suns internal network. But if you are a Sun-external QARep of a ChildWorkspace owned by a Sun developer you can now also start these tests. We are in the process of considering how a integration of ChildWorkspaces where the owner is not a Sun developer could look like, stay tuned.

ConvWatch Tests

these test are also run on dedicated computers and  need between 2 and a half and 3 hours to run. Here a couple of test documents are loaded, than printed to a postscript file and the postscript file is than converted to a graphics file. This is done for the OpenOffice.org of the ChildWorkspace and the corresponding MasterWorkspace. Graphics from the MasterWorkspace OpenOffice.org are than compared to those from the ChildWorkspace OpenOffice.org to find potential regressions in the layout engines and import filters.

Sample test tabpage of a ChildWorkspace in EIS

As you can see in the picture below the new "Tests" tabpage in EIS contains the overall result and a table with results for individual tests and buttons to start tests. 

 Known Issues

there is an automatic test for wether a tool called cwslocalize has been called on a ChildWorkspace to calculate translation word count for the ChildWorkspace. This tool is not yet available to Sun external developers. This is being worked on. 

See Also

The OOo Wiki contains some more details and there is also a "TestBots a first steps" announcement on the QA Mailinglist from Jörg Jahnke.

tags: automation cat0 development eis openoffice.org qa test testtool tools webapplication