Some days ago there started a discussion about this topic in a German mailing list on OOo. It wasn't really this question in the thread. But I want to pick up this discussion in this blog and want to take this provocative question to highlight some quality metrics and actions which are in progress.

Back to the question - Does OpenOffice.org 3.x have a general quality issue? From my perspective a clear NO! Why?

Problem statements

1. OpenOffice.org is software and software isn't error-free. It's the same for OOo. If you search in IssueTracker (the bug tracking system for OOo) you can find the open issues. There are a mass, but are they all relevent for the general quality of the product? Nearly the have of the issues are wishes for new features and enhancements.
   But what do could tell us a numeber here about the general quality of OOo?
   In the thread mentioned above I have to read, that some experienced community members have the feeling, that some areas in OOo are broken – one feature which was named is the Mail Merge Wizard. Others wrote about some bugs in other special areas. But does this mean, that OOo is broken in general. I don't think so. The download numbers of OOo 3.x are amazing. Nearly 60 million for OOo 3.0 (in the past 8-9 months) and near to 13 million for OOo 3.1 (in the past 7-8 weeks). Do these people have so much trouble with the office suite? We will not get their feedback. So I can talk about my experience only. Sometimes I run into trouble with the office suite and I am angry about each of the bugs. But I can work in general with the office on a high quality level. I know (because I got on each of my blogs comments like – my bug xxxx isn't fixed until now, this hinder me in using the office → please fix it soon) that some of the issues hinder somebody to work with our product. But in general?
   I would say, these numbers do not show a general bad quality of our product. But it shows, that we shouldn't forget the older functionality. We have to work still on stabilizing the existing functionality instead of concentrating on newer functionality only.

2. In the releases mailing list a lot of stopper issues were reported in the past month. The numbers are higher than in the past releases and we got such reports near after the release. When I got this information I have to say YES - OOo has a problem with the quality.

   
   

   But the handling of stopper issues was changed in the past releases. So lets take a look at the very next release of OOo 3.1.1. Some days ago there were 35 bugs registered at the stopper issue. The first request for a stopper came in the time-frame when we were releasing OOo 3.1. Since then we got continuously issues reported. But this is what the release managers wanted. They want to hear which are the most annoying and important bugs for the next bug fix release. In the past the teams worked silently and created child work spaces with a number of issues, without telling why these issues are important for a release. The release team wants more stopper issues to have a better priority by the users and they got them.
   So these numbers doesn't tell us if the quality is worse than in the past. => So a clear No! This number does not show that OOo has a quality issue!
   
   

3. What's with the numbers of reported issues in general? Will this number tell us, if the product is broken? Let's have a look.
   
   
   
   The graph shows that the numbers go down. Currently less than 900 issues reports came in per month. In 2005 the number was nearly the twice (~1700). But does this number means that the product is healthy or which number of incoming reports will show this?
   I do not want to say that the product is healthy with such numbers. Each bug report is an error in usability or functionality and someone has a problem to use the product. So the number has to be reduced more and more. But I am realistic. Zero issues per month cannot and shouldn't be the goal. Software is error-prone and it cannot be error-free. If OOo gets less than 100 issues per month, the project is nearly dead instead of healthy, I think! So this numbers show only that we get less reports, but on a high level.
   
   
4. Another number can be interested. Do we get more and more regressions in the last product releases? Sometimes it looks like. Because most of the stopper issues are regressions and especially short before the release of OOo 3.1 there were a lot of such reports. But let's take a look on the general numbers. 
   
   

   These numbers show that we get nearly the same number of regressions over the past years. If you calculate the numbers per month, there were ~80 issues per month which are marked as a regression. In my opinion this is a high number, but does it tell us that OOo is broken? Because most of the issues are fixed before the release of the final version.
   The most critical point here is, that each regression is a regression in general functionality or usability which works in an older version. A usage-scenario breaks inside the developer milestones. But this isn't new, as we can see by the numbers. And the QA teams found the issues mostly in time.
   But perhaps a closer look at the end of the release of OOo 3.1 could help to understand what could goes on.
   Short before the release we got a lot of regressions. An analysis by the development team shows that in a short time frame when this regression were integrated into the code line more than 150 CWS with nearly 1000 issues were integrated. But most of the problems came in with 5 CWS. One of it break the references and other functionality in Spreadsheet nearly completely. Why these CWSs were integrated with such problems!? One reason was wrong/missing communication because of missing knowledge about all the dependencies in such CWS. The result was that the QA responsible checked the wrong area for regressions. Another CWS were well tested, but introduced a mass of conflicts at the integration into the master code line (more than 150!). Some of the conflicts were solved incorrectly. As I heard from Release Engineering team we had such CWS never before.
   This analysis shows that we are working well in general - most of the 150 CWS do not integrated regressions -, but in some cases we have to increase the carefulness.
   

What else can be an indicator for a general quality issue? User feedback or user satisfaction. But how to get this information in an open source product? Feedback on mailing lists are always negative, because the users want to tell about their problems. The users which are satisfied with the product will not write it in the lists. We got positive feedback in the OOo Surves and  in press, but other press articles doesn't vote our product so good. So positive feedback isn't so easy to collect.

Resolution

So let's take a look, what is, will or can be addressed in the near future.

Perhaps you read the blog about Tests in EIS by Bernd Eilers. It's a start to get more and easier full automated testing on a CWS. Some tooling needs more testing and enhancements. Some tools aren't available for external CWSs (outside Sun). But to have such tests in one tool is a step forward to get the focus more on finding the regressions. Perhaps we have to reconsider some test scenarios and should bring more effort in code quality, code testing and more test scripts for automated GUI testing etc. In parts we started this, but it could be increased. In my opinion we should take more time for testing (in all parts of the development cycle) instead of bringing a mass of new features. We did in OOo 3.1, but it should be increased.

To address regressions too – in another direction – there are discussions about changing integration modules for CWS and Master builds. Perhaps you read the blogs about continuous integration by Mathias Bauer or Martin Hollmichel. When parts of these ideas could be realized it can be a step forward for a better code quality in general.

What about the Renaissance project for a new UI of OOo, can it help? I haven't seen the data of the usage tracking tool which were integrated in OOo 3.1 until now. But I heard that we get tons of reports with information about the usage of the Office. Perhaps a look into these data can help to get a prioritization of working areas in the Office. If we know bugs in the high prioritized area we should fix them first (when they are valid enough). Effort in bug-fixing in non/less-used areas could be reduced.

Another good point which I took from the discussion in the German mailing list is to get a general prioritization of features and bug fixing areas for each release. I saw some steps forward for this in the past weeks inside Sun. Perhaps we can get prioritization lists for the whole office for the next releases.

Conclusion

In my opinion OpenOffice.org does not have a quality issue in general. There are a mass of defects in the product and most of them will never be addressed. We got a lot of stopper issues in the past and for the next releases but nearly all these issues were and will be fixed until the products are final. Only the regressions which cannot be addressed in time is the critical mass. We have to concentrate on minimizing these bugs earlier. Also we have to prioritize the work on old bug fixing areas. Only fixing a lot of issues will costs time, but will not help in general to make the product better for all users. The quality management of OOo should be overworked to address missing processes or other things, which can help to improve the quality of OOo.

If you want to discuss topics like this, you are invited to discuss it on the QA mailing list – dev@qa.openoffice.org.