Some days ago there started a discussion about this topic in a German mailing list on OOo. It wasn't really this question in the thread. But I want to pick up this discussion in this blog and want to take this provocative question to highlight some quality metrics and actions which are in progress.

Back to the question - Does OpenOffice.org 3.x have a general quality issue? From my perspective a clear NO! Why?

Problem statements

1. OpenOffice.org is software and software isn't error-free. It's the same for OOo. If you search in IssueTracker (the bug tracking system for OOo) you can find the open issues. There are a mass, but are they all relevent for the general quality of the product? Nearly the have of the issues are wishes for new features and enhancements.
   But what do could tell us a numeber here about the general quality of OOo?
   In the thread mentioned above I have to read, that some experienced community members have the feeling, that some areas in OOo are broken – one feature which was named is the Mail Merge Wizard. Others wrote about some bugs in other special areas. But does this mean, that OOo is broken in general. I don't think so. The download numbers of OOo 3.x are amazing. Nearly 60 million for OOo 3.0 (in the past 8-9 months) and near to 13 million for OOo 3.1 (in the past 7-8 weeks). Do these people have so much trouble with the office suite? We will not get their feedback. So I can talk about my experience only. Sometimes I run into trouble with the office suite and I am angry about each of the bugs. But I can work in general with the office on a high quality level. I know (because I got on each of my blogs comments like – my bug xxxx isn't fixed until now, this hinder me in using the office → please fix it soon) that some of the issues hinder somebody to work with our product. But in general?
   I would say, these numbers do not show a general bad quality of our product. But it shows, that we shouldn't forget the older functionality. We have to work still on stabilizing the existing functionality instead of concentrating on newer functionality only.

2. In the releases mailing list a lot of stopper issues were reported in the past month. The numbers are higher than in the past releases and we got such reports near after the release. When I got this information I have to say YES - OOo has a problem with the quality.

   
   

   But the handling of stopper issues was changed in the past releases. So lets take a look at the very next release of OOo 3.1.1. Some days ago there were 35 bugs registered at the stopper issue. The first request for a stopper came in the time-frame when we were releasing OOo 3.1. Since then we got continuously issues reported. But this is what the release managers wanted. They want to hear which are the most annoying and important bugs for the next bug fix release. In the past the teams worked silently and created child work spaces with a number of issues, without telling why these issues are important for a release. The release team wants more stopper issues to have a better priority by the users and they got them.
   So these numbers doesn't tell us if the quality is worse than in the past. => So a clear No! This number does not show that OOo has a quality issue!
   
   

3. What's with the numbers of reported issues in general? Will this number tell us, if the product is broken? Let's have a look.
   
   
   
   The graph shows that the numbers go down. Currently less than 900 issues reports came in per month. In 2005 the number was nearly the twice (~1700). But does this number means that the product is healthy or which number of incoming reports will show this?
   I do not want to say that the product is healthy with such numbers. Each bug report is an error in usability or functionality and someone has a problem to use the product. So the number has to be reduced more and more. But I am realistic. Zero issues per month cannot and shouldn't be the goal. Software is error-prone and it cannot be error-free. If OOo gets less than 100 issues per month, the project is nearly dead instead of healthy, I think! So this numbers show only that we get less reports, but on a high level.
   
   
4. Another number can be interested. Do we get more and more regressions in the last product releases? Sometimes it looks like. Because most of the stopper issues are regressions and especially short before the release of OOo 3.1 there were a lot of such reports. But let's take a look on the general numbers. 
   
   

   These numbers show that we get nearly the same number of regressions over the past years. If you calculate the numbers per month, there were ~80 issues per month which are marked as a regression. In my opinion this is a high number, but does it tell us that OOo is broken? Because most of the issues are fixed before the release of the final version.
   The most critical point here is, that each regression is a regression in general functionality or usability which works in an older version. A usage-scenario breaks inside the developer milestones. But this isn't new, as we can see by the numbers. And the QA teams found the issues mostly in time.
   But perhaps a closer look at the end of the release of OOo 3.1 could help to understand what could goes on.
   Short before the release we got a lot of regressions. An analysis by the development team shows that in a short time frame when this regression were integrated into the code line more than 150 CWS with nearly 1000 issues were integrated. But most of the problems came in with 5 CWS. One of it break the references and other functionality in Spreadsheet nearly completely. Why these CWSs were integrated with such problems!? One reason was wrong/missing communication because of missing knowledge about all the dependencies in such CWS. The result was that the QA responsible checked the wrong area for regressions. Another CWS were well tested, but introduced a mass of conflicts at the integration into the master code line (more than 150!). Some of the conflicts were solved incorrectly. As I heard from Release Engineering team we had such CWS never before.
   This analysis shows that we are working well in general - most of the 150 CWS do not integrated regressions -, but in some cases we have to increase the carefulness.
   

What else can be an indicator for a general quality issue? User feedback or user satisfaction. But how to get this information in an open source product? Feedback on mailing lists are always negative, because the users want to tell about their problems. The users which are satisfied with the product will not write it in the lists. We got positive feedback in the OOo Surves and  in press, but other press articles doesn't vote our product so good. So positive feedback isn't so easy to collect.

Resolution

So let's take a look, what is, will or can be addressed in the near future.

Perhaps you read the blog about Tests in EIS by Bernd Eilers. It's a start to get more and easier full automated testing on a CWS. Some tooling needs more testing and enhancements. Some tools aren't available for external CWSs (outside Sun). But to have such tests in one tool is a step forward to get the focus more on finding the regressions. Perhaps we have to reconsider some test scenarios and should bring more effort in code quality, code testing and more test scripts for automated GUI testing etc. In parts we started this, but it could be increased. In my opinion we should take more time for testing (in all parts of the development cycle) instead of bringing a mass of new features. We did in OOo 3.1, but it should be increased.

To address regressions too – in another direction – there are discussions about changing integration modules for CWS and Master builds. Perhaps you read the blogs about continuous integration by Mathias Bauer or Martin Hollmichel. When parts of these ideas could be realized it can be a step forward for a better code quality in general.

What about the Renaissance project for a new UI of OOo, can it help? I haven't seen the data of the usage tracking tool which were integrated in OOo 3.1 until now. But I heard that we get tons of reports with information about the usage of the Office. Perhaps a look into these data can help to get a prioritization of working areas in the Office. If we know bugs in the high prioritized area we should fix them first (when they are valid enough). Effort in bug-fixing in non/less-used areas could be reduced.

Another good point which I took from the discussion in the German mailing list is to get a general prioritization of features and bug fixing areas for each release. I saw some steps forward for this in the past weeks inside Sun. Perhaps we can get prioritization lists for the whole office for the next releases.

Conclusion

In my opinion OpenOffice.org does not have a quality issue in general. There are a mass of defects in the product and most of them will never be addressed. We got a lot of stopper issues in the past and for the next releases but nearly all these issues were and will be fixed until the products are final. Only the regressions which cannot be addressed in time is the critical mass. We have to concentrate on minimizing these bugs earlier. Also we have to prioritize the work on old bug fixing areas. Only fixing a lot of issues will costs time, but will not help in general to make the product better for all users. The quality management of OOo should be overworked to address missing processes or other things, which can help to improve the quality of OOo.

If you want to discuss topics like this, you are invited to discuss it on the QA mailing list – dev@qa.openoffice.org.

Last week on 7 ^th of May OpenOffice.org 3.1 was released and more than 2 million users downloaded that version until now. OOo 3.1 is the first feature release after the successful OOo 3.0 release 7 months ago. What was done in that version. Here you can find the numbers beside the features described in the “ what's new guide".

General numbers :

321 CWS with nearly 1750 issues were integrated in the code line for OOo 3.1. These are the numbers which are for the 3.1 code line only. In between there was release OOo 3.0.1 and CWSs which came in late in OOo 3.0 are integrated in this code line. The overall numbers are : 454 CWS with 2189 issues.

More details on the CWS numbers :

In the following graph you can see how many CWS were integrated in which developer milestone. A special milestone was dev300m32, because at this point the migration to Subversion was done and therefore a mass of CWS were integrated directly before and after that time. But there wasn't integrated any CWS in m32.

Other high rate of integrations are for Feature Freeze (round about m38) and Code Freeze and the split to the release code line (round about m1).

80% of all CWS were approved by the Sun QA team. Outside Sun only 7% of all CWS were 'approved by QA'.

The graph for the integration of fixed issues with these CWS is nearly the same.

The biggest integration with more than 300 fixed issues was round about the feature freeze date. And you can see that after splitting the release code line the stabilization of the version was done. The number of integrated fixes went down.

What about the “time to market” numbers for a CWS. How long does it take to get the CWSs approved by the QA teams.

It takes round about 7 days setting a CWS the last time to 'ready for QA' and the nomination for integration. And it is always the same, when a lot of CWS are ready for testing, it takes more time for approving them.

Issue numbers / types :

Nearly 1750 issues were fixed and integrated in this release (only for OOo 3.1).

220 of the integrated issues are marked as features or enhancements. Nearly all of them we got until Feature Freeze. But as in the past releases the Release Status Meeting had to decide about exceptions. But in general it works well to get them all in time. In which component we got most of the features you can see in the next graph.

1166 fixed defects were integrated into OOo 3.1. As in all past releases the most issues were fixed for Writer (word processor).

Also 287 patches are integrated.

This week there were some discussions in mailing lists on OOo, if the right issues were fixed. There were discussed if it is best to fix old issues first. This brought me to do another statistics => when the integrated issues were reported.

So most of the features and defects were submitted in 2008. But also issues which are older than 3 years were addressed in this release. So the development doesn't concentrate on newer features and defects only. They identify also features and defects which are knowing for a long time.

I think, it's good to know these numbers!

Beside this event 5 other releases were done for OOo. So there were a lot work for the QA and release teams last year. 6 releases are more than in the past years!

The 6 product releases for OOo in 2008

• OOo 2.4
  • released end of March 2008 (with a delay of round about one month)
  • was the last but biggest feature release on OOo 2.x code line
• OOo 3.0 Beta 1
  • released mid of April 2008
  • nearly all features for OOo 3.0 were integrated
• OOo 2.4.1
  • released mid of June 2008
  • was the last bigger regular bug fix release on OOo 2.x code line (the next updates on OOo 2.x code line will fix security and very critical issues only)
• OOo 3.0 Beta 2
  • released begin of July 2008
  • final feature set for OOo 3.0 were integrated and stabilization of the code base were done
• OOo 3.0 Final
  • released mid of October 2008 (at 8 ^th anniversary of OOo - 13 ^th of October)
  • numbers about that release
• OOo 2.4.2
  • released on 28 ^th of October 2008
  • currently the last bug fix release on OOo 2.x code line, were ~25 critical issues were fixed

In January 2009 (the very next days) OOo 3.0.1 will be released. So stay tuned for the first bug fix release on OOo 3.x code line. Also Feature Freeze for OOo 3.1 was done end of December.

What other numbers can be interested for the last year?

• nearly 900 child work spaces (CWS) were integrated in the code lines (duplicate integrations in different code lines aren't counted)
• more than 4300 issues (features, enhancements, bug fixes etc.) were integrated in the code lines (duplicate integrations in different code line aren't counted)
• more than 12750 issues were reported in IssueTracker

What does this mean in detail and according to the past years?

Child Work Spaces (CWS) and integrated issues

894 CWS with 4360 issues were integrated in 2008. 83 CWSs with 741 issues of them include features and enhancements. The full allocation over the months can be seen in the next graphs.

How many CWSe and Issues were integrated in the years before is shown in the next graphs.

I asked myself, why we had so much integrations of CWS and issues in 2005 and also in 2006. The answer I found is, OOo worked in that time frame on bug fixing only. In October 2005 OOo 2.0 was released. Before this release features and bug fixes were done in parallel for a long time. And after the betas a mass of bug fixes had to be done. With OOo 2.0.1 in December 2005 and OOo 2.0.2 in February 2006 more than 3500 fixed issues – mostly bug fixes - were integrated. This rate OOo never reached again. Beside this the release model changed in 2007 with the toggle of Feature release with Bug-Fix releases. So since mid of 2006 the teams are working more and more on integration of new features than in 2005.

Incoming Issues in IssueTracker increased in 2008

The high interest on OOo 3.0 was noticeable also in IssueTracker (the issue tracking system for OOo). The number of issues increased before first Beta in March and was on a high level in Beta phases of OOo 3.0 until the final release in October 08.

In 2007 the duration of incoming issues per month were under 1000 issues. But in 2008 the number increased on nearly the same level as in 2006.

These numbers say that 25% of all reported defects and 52% of all reported features and enhancements aren't addressed until now. In sum more than 3000 issues (~27%) aren't addressed. If this is a good or a bad number, I cannot say. I do not have the comparison with other bigger products until now. But I know that with OOo 3.0.1 and OOo 3.1 the quality will and should be increased. So stay tuned for these releases. As I wrote above OOo 3.0.1 will be out in the very soon.

On the other side more than 50% of all issues are done and 12% are addressed for the very next releases. So the general rate for issue handling isn't bad, I think. But I always know, there are still issues which hinder people on using OOo in their environment. It isn't possible to work on all that issues.

My resume

Because of the mass of releases – more than in the past years – and the big pressure to release OOo 3.0 it was a lot of work for all community teams. In my opinion we, the whole OOo community, did more than in the past years. Perhaps the code contribution in lines of code wasn't as high as in the past years (I haven't checked this by myself), but there were many other things to do, that these releases could be realized in time – nearly in time.

Last week Eric Bachard wrote about the number of contributors to OOo and Jim Parkinson, VP at Sun Microsystems, wrote that Sun will continue to support OOo. Also the contribution of Extensions for OOo started and increased dramatically in 2008. Everybody who want to contribute his/her new feature without build environment of OOo could use this method. And a lot of users/contributors did this.

But we need a lot of more people in all projects on OOo to increase the success of OOo in future. Some people talk about code contribution only. I talk about we need more people in all areas and projects – User Experience, QA, L10N, Development or any other project on OOo – to get dreams fulfilled.

So let 2009 be also a successful year and come to OOo – as community member or as user only.

Patches are code contributions mostly used by external developers (community members outside Sun). A developer find an bug/enhancement and a solution. Without using build environment ( ChildWorkSpace) etc. the code snippet can be contributed as patch in IssueTracker. The issue will be taken over from someone who is willing to bring in the patches into a CWS and the QA approval processes. On this way 300 issues with patches are integrated in OOo 3.0. See more details about OOo 3.0 in my last blog.

To get an overview about patch contribution on OOo, let's take a look into the issues which are flagged as patch and which were submitted in 2008 (until October '08).

When were these patches reported?

From January to October 2008 489 issues with type 'patch' are submitted in IssueTracker. The following graph show the allocation over the months.

How many patches are untouched?

The biggest part (69%) of the patches are fixed. Only 20% of the patches are in state 'new' or 'started'. Unconfirmed issues are only 12 issues (2,5%). The rest (8%) are closed as duplicate, invalid or another status.

In which version the fixed patches are integrated?

Most of the patches (63%) were integrated in the last release – OOo 3.0. The next mass of issues (24%) are integrated or are planned to integrate in OOo 3.1. This means most of the patches are integrated early for the next feature releases.

Which components get the most patches?

Most of the patches are submitted for the build environment, tools etc. It's different as for general bug fixing. Here are the applications like word processing and spreadsheet at the first ranks.

Who submitted most patches?

Most of the patches - nearly 32% of all submitted patches - were contributed by cmc. Beside the top10 it isn't easy to find out, if the issue is reported by the patch submitter or the issue was submitted by someone who found the bug. So it isn't so easy to find out, how many different patch contributors there were.

How many patches were submitted the past years?

To can compare the ratio of the submitted patches for the past years, the numbers are related to the patches submitted per month.

The number over the past years were simply the same. In the past 9 months of 2008 the number decreased by 7-8 patches per month.

What does the decrease identify? Difficult to say. The allocation in which components the patches were delivered changed over the years. From 2005-2007 more than 26% of all patches were delivered for the porting project – in 2008 only 5%. The same for L10N components. From 2005-2008 nearly 12% of all patches were for that project – in 2008 only 2%. So perhaps contributors went away from OOo or the projects are more stable as in the past and less patches are needed in such projects.

How many patches are still open / backlog from the past years?

In sum nearly ~3950 issues which are marked as patches were submitted for OOo in the past years. Only 137 (3.5%) of them are marked as 'new', 70 (2%) marked as 'started' and 18 (0.5%) are unconfirmed. Most of the issues, 3225 or 82% are in state fixed. So the backlog isn't high.

Resume

Patches seems to be still a good way to bring in code snippets (bug fixes, translation, enhancements, fixes for tooling, fixes for build environment etc.) from external developers into the code line of OOo without using the processes of ChildWorkSpace handling and QA. If the patches the right way to increase the code contribution on OOo, I do not know.

As you still know I am the manager of the Quality Assurance Team at Sun Microsystems for StarOffice and OpenOffice.org and everyday I work and do my daily business with each developer milestone. I do the same with the ones of the OOo 3.0 code line. Why I do not have any objections? Why I am not afraid and why I am not a friend of writing 'please do not use the Beta for your daily work, you can destroy your documents'? Please read why I am so confident.

In the past years since OOo 2.0 Beta the development of OOo did a major step to increase the stability and quality of our product. Processes and guidelines were integrated to get a stable version every day. Some groups in the community say, these hinder the contribution to the project. I do understand this. But the great benefit for all of us is, that every contributor and each user of OOo can get stable releases every 3 months. And with the change to 'more frequent feature releases' every 6 months also new features are integrated in the new versions.

The number of users and interested people on OOo increased the past years dramatically. In some weeks more than 1 million downloads per week were registered. From these numbers the project was fare fare away when starting with OOo 2.0 Beta begin 2005. Beside the increased numbers of users of and contributors to OOo the number of incoming issues in IssueTracker decreased. In 2005 round about 1800 issues were reported each month. Last year the submission rate was at 1050 issues and in the past months before the announcement of OOo 2.4 and OOo 3.0 Beta the numbers decreased to 950 issues per month. Now with the announcement of OOo 3.0 Beta the numbers increased a little bit. But in general the numbers are fare fare away from the numbers at OOo 2.0 Beta release.

 

Since the release of OOo 3.0 Beta some more than 300 issues are reported on this version. But this does not increase the numbers for May in general. So I have to think the users switch to the Beta and report their issues on that version and do not report it on an older release. Currently less than 5 issues are identified as show stopper issue for the Beta Refresh end of June.

What was done to get the higher quality since OOo 2.0?

First of all the concentration on developing with Child Work Spaces (CWS) are very important. To bring in new features and changes only when they are tested in a separate build (the CWS). Only when it is finished the QA approval will be given and the CWS can be integrated into the master code line. This hinder to bring in unfinished features and perhaps broken functionality. Regressions are still there on the master code line. This couldn't be reduced to zero. But it could be minimized.

The second quality initiative was to concentrate on stability in OOo 2.0.3 and 2.0.4. With the Crash Reporter feature all most critical crashes were identified and fixed. This tooling is used for each release and all critical issues from the last releases are fixed in the next release. In 2.x time-frame nearly 400 issues reported over the crash reporter tooling by the user of OOo could be fixed. These 400 issues represent nearly 85% of all reported crashes.

Other quality initiatives in the OOo 2.x time frame were 'Warning Free Code' and check the code with Valgrind. This increased the general quality of the code. Memory leaks, programming errors, build errors etc. were identified and this represent the general stability of the product.

The most important point is, that OOo 3.0 based on the stable version of 2.4. More than 2.5 million downloads of that versions in the past 4 weeks shows a high credit in that version.

All these information – the number of incoming issues decrease, the incoming crash reports for newer versions decrease, the good processes for quality assurance on CWSs, the stable 2.x code line and ... – give me the confidence to do my daily business with every new build of OOo 3.0 (or sometimes with StarOffice).

I know that I have to use sometimes the latest version of OOo 2.x. This I do only when I have to create business critical documents and when I know the end users will not have an office which work with ODF 1.2 (like OOo 3.0). In some cases incompatibilities can occur with when features from 3.0 code-line are used, which aren't in 2.x. But you will get a warning in OOo 2.x, when you are opening a file with ODF 1.2.

When you are now more interested in using OOo 3.0 Beta, download it at http://download.openoffice.org/3.0beta/

If you are interested in general feedback on OOo 3.0 Beta, you can find it in OOo Wiki http://wiki.services.openoffice.org/wiki/OOo_3_Beta_feedback
If you think a critical bug should be fixed until Beta Refresh or the Final version of OOo write it on that page and let discuss it on the dev@qa.openoffice.org mailing list.

I understand that users think that OOo 3.0 isn't a big step forward when they are always using the newest versions of OOo 2.x. The feature set increased like each other Update and some more bug fixes are integrated – that's all. But with opening a new code line and integrated some major changes in the structure of OOo it will be possible to bring in the changes which are voted often on OOo mailing lists and also on the Beta Feedback side.

I am confident and looking promising to the updates on OOo 3.x code line.

Last month I was frightened about the high number of unconfirmed defects. At that time we had 993 defects open. Now I have to say, the number increased again.

Here are the current numbers (in brackets is the comparison to the last count) :

• 1092 defects (+99 or +10%) are in state 'unconfirmed'

• these are the areas with unconfirmed defects
  • framework : 262 (-18 or -6%)
  • word processor / writer : 320 (+53 or 20%)
  • data base : 75 (+8 or +12%)
  • spreadsheet / calc : 101 (+17 or +20%)
  • graphics / presentation / drawing : 66 (+30 or +83%)
  • others : 258 (+2 or +1%)
• the submit date of the 'unconfirmed' defects
  • 194 until last announced statistics
  • 415 (-26 or -7%) until November '07 (last time the link was wrong)
  • 156 (-65 or -29% ) in first half of 2007
  • 142 (-13 or -8%) in second half of 2006
  • 80 (+/-0) in first half of 2006
  • 86 (-4 or -4%) are older than 2006

As you can see most of the open unconfirmed defects are submitted in the last half year. It seams it is still hard to confirm all new incoming issues quickly. But these are the most important ones. If OpenOffice.org has a regression between releases often the issues are written short after the release. I think that most important regressions are found, verified and fixed from one release to the next (release cycle is every 3 months), but do we have all of them identified. Nobody can tell, when more than 600 defects are unverified from the last half year.

To get a feeling, if these numbers are too critical. Here are some more numbers of the last half year.

• 3863 defects were submitted since 1 ^st July 2007
• 1062 (28%) defects were fixed and they are integrated in the past releases (OOo 2.3.1) or in the current developer snapshots
• 1158 (30%) defects were marked as invalid, duplicate or couldn't reproduced by a QA person or a developer
• 890 (23%) defects are in state 'new'

• 173 (4%) are in state 'started'

• 560 (15%) defects are 'unconfirmed'
  

If you want to work on these issues and also want to work in the QA Teams on OOo. Please take a look at the QA-project on OOo. First steps can be found at http://qa.openoffice.org/ooQAReloaded/Docs/QA-Reloaded-HowToStart.html.

All informations about issue handling can be found on the QA sites .
General questions should be discussed on the mailing list dev@qa.openoffice.org