As you could read with the last developer milestone ( DEV300m60) we reached the date to branch the code line for the OOo 3.2 release. The strings for translation were extracted from this milestone and were integrated into Pootle. Also the last features were integrated. So all teams can start to do their work to get released a full localized and stable build of OOo 3.2 at the end of November '09.

Some weeks ago I announced here that I will finalize the feature-list for this release soon after the last build with the last features is ready for use. I am ready and you can check the wiki page, if you are interested in all feature announcements for OOo 3.2. The list includes the issues numbers, the feature announcement, the specification, the test case specification and the CWS name. As you can see this isn't the 'What's New Guide'. This list should help testers and translators to identify the new features and check if it works as estimated and/or as specified. So please take a look at :

http://wiki.services.openoffice.org/wiki/Feature_Freeze_Testing_3.2

To get a deeper look at some features I collected some blogs about the new stuff in OOo 3.2 :

• Introduction of the new UNO AWT Grid Control
• Security and Privacy Feature Improvements in upcoming OpenOffice.org 3.2
• Graphite Support
• Community work in the framework project – Part 1
• Saving sheets separately: Preliminary results
• The framework project, OpenOffice.org 3.2 and Windows 7
• Chart performance
• Unified ODF Icons (approved to integrate in OOO320m1)

As I could read in the meeting minutes of the Release Status Meeting from yesterday it was approved to release a Beta version in some days. It should be the build OOO320m2, where some show-stopper issues are fixed. If you are interested to give feedback you can use the mailing list releases@openoffice.org. If you want to promote a critical issue for the final release do not hesitate to communicate it also on that mailing list – as soon as possible –. If you are interested in currently known show-stoppers you can check the list of issue 99999.

Some days ago there started a discussion about this topic in a German mailing list on OOo. It wasn't really this question in the thread. But I want to pick up this discussion in this blog and want to take this provocative question to highlight some quality metrics and actions which are in progress.

Back to the question - Does OpenOffice.org 3.x have a general quality issue? From my perspective a clear NO! Why?

Problem statements

1. OpenOffice.org is software and software isn't error-free. It's the same for OOo. If you search in IssueTracker (the bug tracking system for OOo) you can find the open issues. There are a mass, but are they all relevent for the general quality of the product? Nearly the have of the issues are wishes for new features and enhancements.
   But what do could tell us a numeber here about the general quality of OOo?
   In the thread mentioned above I have to read, that some experienced community members have the feeling, that some areas in OOo are broken – one feature which was named is the Mail Merge Wizard. Others wrote about some bugs in other special areas. But does this mean, that OOo is broken in general. I don't think so. The download numbers of OOo 3.x are amazing. Nearly 60 million for OOo 3.0 (in the past 8-9 months) and near to 13 million for OOo 3.1 (in the past 7-8 weeks). Do these people have so much trouble with the office suite? We will not get their feedback. So I can talk about my experience only. Sometimes I run into trouble with the office suite and I am angry about each of the bugs. But I can work in general with the office on a high quality level. I know (because I got on each of my blogs comments like – my bug xxxx isn't fixed until now, this hinder me in using the office → please fix it soon) that some of the issues hinder somebody to work with our product. But in general?
   I would say, these numbers do not show a general bad quality of our product. But it shows, that we shouldn't forget the older functionality. We have to work still on stabilizing the existing functionality instead of concentrating on newer functionality only.

2. In the releases mailing list a lot of stopper issues were reported in the past month. The numbers are higher than in the past releases and we got such reports near after the release. When I got this information I have to say YES - OOo has a problem with the quality.

   
   

   But the handling of stopper issues was changed in the past releases. So lets take a look at the very next release of OOo 3.1.1. Some days ago there were 35 bugs registered at the stopper issue. The first request for a stopper came in the time-frame when we were releasing OOo 3.1. Since then we got continuously issues reported. But this is what the release managers wanted. They want to hear which are the most annoying and important bugs for the next bug fix release. In the past the teams worked silently and created child work spaces with a number of issues, without telling why these issues are important for a release. The release team wants more stopper issues to have a better priority by the users and they got them.
   So these numbers doesn't tell us if the quality is worse than in the past. => So a clear No! This number does not show that OOo has a quality issue!
   
   

3. What's with the numbers of reported issues in general? Will this number tell us, if the product is broken? Let's have a look.
   
   
   
   The graph shows that the numbers go down. Currently less than 900 issues reports came in per month. In 2005 the number was nearly the twice (~1700). But does this number means that the product is healthy or which number of incoming reports will show this?
   I do not want to say that the product is healthy with such numbers. Each bug report is an error in usability or functionality and someone has a problem to use the product. So the number has to be reduced more and more. But I am realistic. Zero issues per month cannot and shouldn't be the goal. Software is error-prone and it cannot be error-free. If OOo gets less than 100 issues per month, the project is nearly dead instead of healthy, I think! So this numbers show only that we get less reports, but on a high level.
   
   
4. Another number can be interested. Do we get more and more regressions in the last product releases? Sometimes it looks like. Because most of the stopper issues are regressions and especially short before the release of OOo 3.1 there were a lot of such reports. But let's take a look on the general numbers. 
   
   

   These numbers show that we get nearly the same number of regressions over the past years. If you calculate the numbers per month, there were ~80 issues per month which are marked as a regression. In my opinion this is a high number, but does it tell us that OOo is broken? Because most of the issues are fixed before the release of the final version.
   The most critical point here is, that each regression is a regression in general functionality or usability which works in an older version. A usage-scenario breaks inside the developer milestones. But this isn't new, as we can see by the numbers. And the QA teams found the issues mostly in time.
   But perhaps a closer look at the end of the release of OOo 3.1 could help to understand what could goes on.
   Short before the release we got a lot of regressions. An analysis by the development team shows that in a short time frame when this regression were integrated into the code line more than 150 CWS with nearly 1000 issues were integrated. But most of the problems came in with 5 CWS. One of it break the references and other functionality in Spreadsheet nearly completely. Why these CWSs were integrated with such problems!? One reason was wrong/missing communication because of missing knowledge about all the dependencies in such CWS. The result was that the QA responsible checked the wrong area for regressions. Another CWS were well tested, but introduced a mass of conflicts at the integration into the master code line (more than 150!). Some of the conflicts were solved incorrectly. As I heard from Release Engineering team we had such CWS never before.
   This analysis shows that we are working well in general - most of the 150 CWS do not integrated regressions -, but in some cases we have to increase the carefulness.
   

What else can be an indicator for a general quality issue? User feedback or user satisfaction. But how to get this information in an open source product? Feedback on mailing lists are always negative, because the users want to tell about their problems. The users which are satisfied with the product will not write it in the lists. We got positive feedback in the OOo Surves and  in press, but other press articles doesn't vote our product so good. So positive feedback isn't so easy to collect.

Resolution

So let's take a look, what is, will or can be addressed in the near future.

Perhaps you read the blog about Tests in EIS by Bernd Eilers. It's a start to get more and easier full automated testing on a CWS. Some tooling needs more testing and enhancements. Some tools aren't available for external CWSs (outside Sun). But to have such tests in one tool is a step forward to get the focus more on finding the regressions. Perhaps we have to reconsider some test scenarios and should bring more effort in code quality, code testing and more test scripts for automated GUI testing etc. In parts we started this, but it could be increased. In my opinion we should take more time for testing (in all parts of the development cycle) instead of bringing a mass of new features. We did in OOo 3.1, but it should be increased.

To address regressions too – in another direction – there are discussions about changing integration modules for CWS and Master builds. Perhaps you read the blogs about continuous integration by Mathias Bauer or Martin Hollmichel. When parts of these ideas could be realized it can be a step forward for a better code quality in general.

What about the Renaissance project for a new UI of OOo, can it help? I haven't seen the data of the usage tracking tool which were integrated in OOo 3.1 until now. But I heard that we get tons of reports with information about the usage of the Office. Perhaps a look into these data can help to get a prioritization of working areas in the Office. If we know bugs in the high prioritized area we should fix them first (when they are valid enough). Effort in bug-fixing in non/less-used areas could be reduced.

Another good point which I took from the discussion in the German mailing list is to get a general prioritization of features and bug fixing areas for each release. I saw some steps forward for this in the past weeks inside Sun. Perhaps we can get prioritization lists for the whole office for the next releases.

Conclusion

In my opinion OpenOffice.org does not have a quality issue in general. There are a mass of defects in the product and most of them will never be addressed. We got a lot of stopper issues in the past and for the next releases but nearly all these issues were and will be fixed until the products are final. Only the regressions which cannot be addressed in time is the critical mass. We have to concentrate on minimizing these bugs earlier. Also we have to prioritize the work on old bug fixing areas. Only fixing a lot of issues will costs time, but will not help in general to make the product better for all users. The quality management of OOo should be overworked to address missing processes or other things, which can help to improve the quality of OOo.

If you want to discuss topics like this, you are invited to discuss it on the QA mailing list – dev@qa.openoffice.org.

Last week on 7 ^th of May OpenOffice.org 3.1 was released and more than 2 million users downloaded that version until now. OOo 3.1 is the first feature release after the successful OOo 3.0 release 7 months ago. What was done in that version. Here you can find the numbers beside the features described in the “ what's new guide".

General numbers :

321 CWS with nearly 1750 issues were integrated in the code line for OOo 3.1. These are the numbers which are for the 3.1 code line only. In between there was release OOo 3.0.1 and CWSs which came in late in OOo 3.0 are integrated in this code line. The overall numbers are : 454 CWS with 2189 issues.

More details on the CWS numbers :

In the following graph you can see how many CWS were integrated in which developer milestone. A special milestone was dev300m32, because at this point the migration to Subversion was done and therefore a mass of CWS were integrated directly before and after that time. But there wasn't integrated any CWS in m32.

Other high rate of integrations are for Feature Freeze (round about m38) and Code Freeze and the split to the release code line (round about m1).

80% of all CWS were approved by the Sun QA team. Outside Sun only 7% of all CWS were 'approved by QA'.

The graph for the integration of fixed issues with these CWS is nearly the same.

The biggest integration with more than 300 fixed issues was round about the feature freeze date. And you can see that after splitting the release code line the stabilization of the version was done. The number of integrated fixes went down.

What about the “time to market” numbers for a CWS. How long does it take to get the CWSs approved by the QA teams.

It takes round about 7 days setting a CWS the last time to 'ready for QA' and the nomination for integration. And it is always the same, when a lot of CWS are ready for testing, it takes more time for approving them.

Issue numbers / types :

Nearly 1750 issues were fixed and integrated in this release (only for OOo 3.1).

220 of the integrated issues are marked as features or enhancements. Nearly all of them we got until Feature Freeze. But as in the past releases the Release Status Meeting had to decide about exceptions. But in general it works well to get them all in time. In which component we got most of the features you can see in the next graph.

1166 fixed defects were integrated into OOo 3.1. As in all past releases the most issues were fixed for Writer (word processor).

Also 287 patches are integrated.

This week there were some discussions in mailing lists on OOo, if the right issues were fixed. There were discussed if it is best to fix old issues first. This brought me to do another statistics => when the integrated issues were reported.

So most of the features and defects were submitted in 2008. But also issues which are older than 3 years were addressed in this release. So the development doesn't concentrate on newer features and defects only. They identify also features and defects which are knowing for a long time.

I think, it's good to know these numbers!