As you could read with the last developer milestone ( DEV300m60) we reached the date to branch the code line for the OOo 3.2 release. The strings for translation were extracted from this milestone and were integrated into Pootle. Also the last features were integrated. So all teams can start to do their work to get released a full localized and stable build of OOo 3.2 at the end of November '09.

Some weeks ago I announced here that I will finalize the feature-list for this release soon after the last build with the last features is ready for use. I am ready and you can check the wiki page, if you are interested in all feature announcements for OOo 3.2. The list includes the issues numbers, the feature announcement, the specification, the test case specification and the CWS name. As you can see this isn't the 'What's New Guide'. This list should help testers and translators to identify the new features and check if it works as estimated and/or as specified. So please take a look at :

http://wiki.services.openoffice.org/wiki/Feature_Freeze_Testing_3.2

To get a deeper look at some features I collected some blogs about the new stuff in OOo 3.2 :

• Introduction of the new UNO AWT Grid Control
• Security and Privacy Feature Improvements in upcoming OpenOffice.org 3.2
• Graphite Support
• Community work in the framework project – Part 1
• Saving sheets separately: Preliminary results
• The framework project, OpenOffice.org 3.2 and Windows 7
• Chart performance
• Unified ODF Icons (approved to integrate in OOO320m1)

As I could read in the meeting minutes of the Release Status Meeting from yesterday it was approved to release a Beta version in some days. It should be the build OOO320m2, where some show-stopper issues are fixed. If you are interested to give feedback you can use the mailing list releases@openoffice.org. If you want to promote a critical issue for the final release do not hesitate to communicate it also on that mailing list – as soon as possible –. If you are interested in currently known show-stoppers you can check the list of issue 99999.

Some days ago there started a discussion about this topic in a German mailing list on OOo. It wasn't really this question in the thread. But I want to pick up this discussion in this blog and want to take this provocative question to highlight some quality metrics and actions which are in progress.

Back to the question - Does OpenOffice.org 3.x have a general quality issue? From my perspective a clear NO! Why?

Problem statements

1. OpenOffice.org is software and software isn't error-free. It's the same for OOo. If you search in IssueTracker (the bug tracking system for OOo) you can find the open issues. There are a mass, but are they all relevent for the general quality of the product? Nearly the have of the issues are wishes for new features and enhancements.
   But what do could tell us a numeber here about the general quality of OOo?
   In the thread mentioned above I have to read, that some experienced community members have the feeling, that some areas in OOo are broken – one feature which was named is the Mail Merge Wizard. Others wrote about some bugs in other special areas. But does this mean, that OOo is broken in general. I don't think so. The download numbers of OOo 3.x are amazing. Nearly 60 million for OOo 3.0 (in the past 8-9 months) and near to 13 million for OOo 3.1 (in the past 7-8 weeks). Do these people have so much trouble with the office suite? We will not get their feedback. So I can talk about my experience only. Sometimes I run into trouble with the office suite and I am angry about each of the bugs. But I can work in general with the office on a high quality level. I know (because I got on each of my blogs comments like – my bug xxxx isn't fixed until now, this hinder me in using the office → please fix it soon) that some of the issues hinder somebody to work with our product. But in general?
   I would say, these numbers do not show a general bad quality of our product. But it shows, that we shouldn't forget the older functionality. We have to work still on stabilizing the existing functionality instead of concentrating on newer functionality only.

2. In the releases mailing list a lot of stopper issues were reported in the past month. The numbers are higher than in the past releases and we got such reports near after the release. When I got this information I have to say YES - OOo has a problem with the quality.

   
   

   But the handling of stopper issues was changed in the past releases. So lets take a look at the very next release of OOo 3.1.1. Some days ago there were 35 bugs registered at the stopper issue. The first request for a stopper came in the time-frame when we were releasing OOo 3.1. Since then we got continuously issues reported. But this is what the release managers wanted. They want to hear which are the most annoying and important bugs for the next bug fix release. In the past the teams worked silently and created child work spaces with a number of issues, without telling why these issues are important for a release. The release team wants more stopper issues to have a better priority by the users and they got them.
   So these numbers doesn't tell us if the quality is worse than in the past. => So a clear No! This number does not show that OOo has a quality issue!
   
   

3. What's with the numbers of reported issues in general? Will this number tell us, if the product is broken? Let's have a look.
   
   
   
   The graph shows that the numbers go down. Currently less than 900 issues reports came in per month. In 2005 the number was nearly the twice (~1700). But does this number means that the product is healthy or which number of incoming reports will show this?
   I do not want to say that the product is healthy with such numbers. Each bug report is an error in usability or functionality and someone has a problem to use the product. So the number has to be reduced more and more. But I am realistic. Zero issues per month cannot and shouldn't be the goal. Software is error-prone and it cannot be error-free. If OOo gets less than 100 issues per month, the project is nearly dead instead of healthy, I think! So this numbers show only that we get less reports, but on a high level.
   
   
4. Another number can be interested. Do we get more and more regressions in the last product releases? Sometimes it looks like. Because most of the stopper issues are regressions and especially short before the release of OOo 3.1 there were a lot of such reports. But let's take a look on the general numbers. 
   
   

   These numbers show that we get nearly the same number of regressions over the past years. If you calculate the numbers per month, there were ~80 issues per month which are marked as a regression. In my opinion this is a high number, but does it tell us that OOo is broken? Because most of the issues are fixed before the release of the final version.
   The most critical point here is, that each regression is a regression in general functionality or usability which works in an older version. A usage-scenario breaks inside the developer milestones. But this isn't new, as we can see by the numbers. And the QA teams found the issues mostly in time.
   But perhaps a closer look at the end of the release of OOo 3.1 could help to understand what could goes on.
   Short before the release we got a lot of regressions. An analysis by the development team shows that in a short time frame when this regression were integrated into the code line more than 150 CWS with nearly 1000 issues were integrated. But most of the problems came in with 5 CWS. One of it break the references and other functionality in Spreadsheet nearly completely. Why these CWSs were integrated with such problems!? One reason was wrong/missing communication because of missing knowledge about all the dependencies in such CWS. The result was that the QA responsible checked the wrong area for regressions. Another CWS were well tested, but introduced a mass of conflicts at the integration into the master code line (more than 150!). Some of the conflicts were solved incorrectly. As I heard from Release Engineering team we had such CWS never before.
   This analysis shows that we are working well in general - most of the 150 CWS do not integrated regressions -, but in some cases we have to increase the carefulness.
   

What else can be an indicator for a general quality issue? User feedback or user satisfaction. But how to get this information in an open source product? Feedback on mailing lists are always negative, because the users want to tell about their problems. The users which are satisfied with the product will not write it in the lists. We got positive feedback in the OOo Surves and  in press, but other press articles doesn't vote our product so good. So positive feedback isn't so easy to collect.

Resolution

So let's take a look, what is, will or can be addressed in the near future.

Perhaps you read the blog about Tests in EIS by Bernd Eilers. It's a start to get more and easier full automated testing on a CWS. Some tooling needs more testing and enhancements. Some tools aren't available for external CWSs (outside Sun). But to have such tests in one tool is a step forward to get the focus more on finding the regressions. Perhaps we have to reconsider some test scenarios and should bring more effort in code quality, code testing and more test scripts for automated GUI testing etc. In parts we started this, but it could be increased. In my opinion we should take more time for testing (in all parts of the development cycle) instead of bringing a mass of new features. We did in OOo 3.1, but it should be increased.

To address regressions too – in another direction – there are discussions about changing integration modules for CWS and Master builds. Perhaps you read the blogs about continuous integration by Mathias Bauer or Martin Hollmichel. When parts of these ideas could be realized it can be a step forward for a better code quality in general.

What about the Renaissance project for a new UI of OOo, can it help? I haven't seen the data of the usage tracking tool which were integrated in OOo 3.1 until now. But I heard that we get tons of reports with information about the usage of the Office. Perhaps a look into these data can help to get a prioritization of working areas in the Office. If we know bugs in the high prioritized area we should fix them first (when they are valid enough). Effort in bug-fixing in non/less-used areas could be reduced.

Another good point which I took from the discussion in the German mailing list is to get a general prioritization of features and bug fixing areas for each release. I saw some steps forward for this in the past weeks inside Sun. Perhaps we can get prioritization lists for the whole office for the next releases.

Conclusion

In my opinion OpenOffice.org does not have a quality issue in general. There are a mass of defects in the product and most of them will never be addressed. We got a lot of stopper issues in the past and for the next releases but nearly all these issues were and will be fixed until the products are final. Only the regressions which cannot be addressed in time is the critical mass. We have to concentrate on minimizing these bugs earlier. Also we have to prioritize the work on old bug fixing areas. Only fixing a lot of issues will costs time, but will not help in general to make the product better for all users. The quality management of OOo should be overworked to address missing processes or other things, which can help to improve the quality of OOo.

If you want to discuss topics like this, you are invited to discuss it on the QA mailing list – dev@qa.openoffice.org.

I have collected with some colleagues the best practices for writing software test code. A good design of test cases can help you to reduce the maintainance effort, make the test runs faster and more efficient, etc. Please read and comment the summary here in my blog.

Beside this event 5 other releases were done for OOo. So there were a lot work for the QA and release teams last year. 6 releases are more than in the past years!

The 6 product releases for OOo in 2008

• OOo 2.4
  • released end of March 2008 (with a delay of round about one month)
  • was the last but biggest feature release on OOo 2.x code line
• OOo 3.0 Beta 1
  • released mid of April 2008
  • nearly all features for OOo 3.0 were integrated
• OOo 2.4.1
  • released mid of June 2008
  • was the last bigger regular bug fix release on OOo 2.x code line (the next updates on OOo 2.x code line will fix security and very critical issues only)
• OOo 3.0 Beta 2
  • released begin of July 2008
  • final feature set for OOo 3.0 were integrated and stabilization of the code base were done
• OOo 3.0 Final
  • released mid of October 2008 (at 8 ^th anniversary of OOo - 13 ^th of October)
  • numbers about that release
• OOo 2.4.2
  • released on 28 ^th of October 2008
  • currently the last bug fix release on OOo 2.x code line, were ~25 critical issues were fixed

In January 2009 (the very next days) OOo 3.0.1 will be released. So stay tuned for the first bug fix release on OOo 3.x code line. Also Feature Freeze for OOo 3.1 was done end of December.

What other numbers can be interested for the last year?

• nearly 900 child work spaces (CWS) were integrated in the code lines (duplicate integrations in different code lines aren't counted)
• more than 4300 issues (features, enhancements, bug fixes etc.) were integrated in the code lines (duplicate integrations in different code line aren't counted)
• more than 12750 issues were reported in IssueTracker

What does this mean in detail and according to the past years?

Child Work Spaces (CWS) and integrated issues

894 CWS with 4360 issues were integrated in 2008. 83 CWSs with 741 issues of them include features and enhancements. The full allocation over the months can be seen in the next graphs.

How many CWSe and Issues were integrated in the years before is shown in the next graphs.

I asked myself, why we had so much integrations of CWS and issues in 2005 and also in 2006. The answer I found is, OOo worked in that time frame on bug fixing only. In October 2005 OOo 2.0 was released. Before this release features and bug fixes were done in parallel for a long time. And after the betas a mass of bug fixes had to be done. With OOo 2.0.1 in December 2005 and OOo 2.0.2 in February 2006 more than 3500 fixed issues – mostly bug fixes - were integrated. This rate OOo never reached again. Beside this the release model changed in 2007 with the toggle of Feature release with Bug-Fix releases. So since mid of 2006 the teams are working more and more on integration of new features than in 2005.

Incoming Issues in IssueTracker increased in 2008

The high interest on OOo 3.0 was noticeable also in IssueTracker (the issue tracking system for OOo). The number of issues increased before first Beta in March and was on a high level in Beta phases of OOo 3.0 until the final release in October 08.

In 2007 the duration of incoming issues per month were under 1000 issues. But in 2008 the number increased on nearly the same level as in 2006.

These numbers say that 25% of all reported defects and 52% of all reported features and enhancements aren't addressed until now. In sum more than 3000 issues (~27%) aren't addressed. If this is a good or a bad number, I cannot say. I do not have the comparison with other bigger products until now. But I know that with OOo 3.0.1 and OOo 3.1 the quality will and should be increased. So stay tuned for these releases. As I wrote above OOo 3.0.1 will be out in the very soon.

On the other side more than 50% of all issues are done and 12% are addressed for the very next releases. So the general rate for issue handling isn't bad, I think. But I always know, there are still issues which hinder people on using OOo in their environment. It isn't possible to work on all that issues.

My resume

Because of the mass of releases – more than in the past years – and the big pressure to release OOo 3.0 it was a lot of work for all community teams. In my opinion we, the whole OOo community, did more than in the past years. Perhaps the code contribution in lines of code wasn't as high as in the past years (I haven't checked this by myself), but there were many other things to do, that these releases could be realized in time – nearly in time.

Last week Eric Bachard wrote about the number of contributors to OOo and Jim Parkinson, VP at Sun Microsystems, wrote that Sun will continue to support OOo. Also the contribution of Extensions for OOo started and increased dramatically in 2008. Everybody who want to contribute his/her new feature without build environment of OOo could use this method. And a lot of users/contributors did this.

But we need a lot of more people in all projects on OOo to increase the success of OOo in future. Some people talk about code contribution only. I talk about we need more people in all areas and projects – User Experience, QA, L10N, Development or any other project on OOo – to get dreams fulfilled.

So let 2009 be also a successful year and come to OOo – as community member or as user only.

The video of my presentation at the OOoCon2008 is live since some days. If you want to know more about the “The complex world of Quality Assurance on OOo” take a look at the video and the presentation file.

Thanks to Peter Junge and John McCreesh for releasing all videos of the conference.

I'm back in Germany from the OOoCon 2008 in Beijing, China. For me it was an amazing and interested conference. Not only because I found some new friends, I think. No, it was amazing, because some companies came to me and asked me, if I can help them to bring quality into their products which based on OOo. Wow, last year in Barcelona I was contacted only when somebody want to talk about code quality!

In my eyes quality is and will be more important for software products as in the past years. And I think, the QA community on OOo is doing a tremendous job!

At my presentation and the other presentations about Quality Assurance I mentioned in my last blog, there were round about 40-50 interested people in the meeting room. I was gladly surprised about this feedback.

For example in and after my presentation I was asked by people from IBM, how they can adapt Child Work Space (CWS) handling in their development process and how they can use the VCL TestTool for making general Quality Assurance on Lotus Symphony. In another discussion I was asked if the crash reporter tooling can be used also for other branded versions of OOo to get analyzed instabilities in their builds.

I also meet very often with employees by RedFlag 2000. They are also interested in changing their development process and how they can use the TestTool to bring in the needed quality in their products. The most interested number I have heard at the conference was for me, that RedFlag 2000 has 65 person for testing and QA. They aren't working all for OOo, but such a big QA department for software was a surprise to me. I hope I and the members of my team could give them useful tips and tricks.

It's the right time to work on quality for all of us, I think. Especially when other big companies are willing to contribute their changes to OOo and bring in QA resources to the QA project.

Beside the mass of interested discussions it was interested for me, that some guys are thinking the guys from QA are bad guys, aren't we? But ... perhaps sometimes I like to be the bad guy! - smile -

If you are interested in what's the complex world of QA on OOo is, here you can find my presentation file with many links in it and perhaps the video of my presentation will be on-line soon too. Then I will send the link in a comment to this blog.

Beside the business I could also say, that the trip to the conference was an impressive experience for me. The Chinese people I met and all members of the RedFlag 2000 and IBM team which organized the conference, were so friendly and open-hearted. Only the mass of the Chinese people in the streets were strange for me.

For example I do not want to drive car in Beijing. To drive taxi is like you are driving in a computer game. I never saw the “game over” but sometimes it was near that. In Germany every second it will crash thousand times in one street, when every driver cross the lines like a taxi driver in Beijing. So when you are in Beijing you have to take a taxi – which is very cheap for European measure – and you will find out, how it is possible to drive in totally chaos. But it works!

As pedestrian you are the fair game in the road traffic. Everybody is trying to overrun you! When you want to cross the road and your traffic light is green, the cars do not stop for you when they want to turn to one's right or left. Sometimes you find a crossing guard with a flag in his hand. But he only stops the cars, bicycles and other pedestrians when a foreigner like me try to cross the road. Then the Chinese people do not know, why they have to stop. So they do not stop or they are looking all on you. Nice feeling!

The food and meals I got were all very well and some were amazing – except the food in the canteen of the Peking University we got at the first conference day (only the banana I could eat). Also the culture to share all meals with everybody on the table assists to be more risky with the food. So I tried out nearly everything and sometime I do not want to know what is was. Only the chicken legs I didn't taste. And the best is to go out for food with some kindly Chinese people. So thank you Lihua and Meiying for ordering the soup without the chicken head and foots etc. I enjoyed our meal on Suturday.

Thanks to everybody for the successful OOoCon 2008 in Beijing.