Martin Hollmichel's Weblog: Martin Hollmichel's Weblog

OpenOffice.org now in transition to distributed source code control (mercurial)

The transition of the source code repository from CVS via subversion to mercurial is now almost complete. It was a long way to go, there was the agreement for a long time that an migration from CVS to a more modern source code control is desired. The major obstacle was the lack of native merge capability support, simulating the merge tracking with supplemental scripting was not that handy and snappy as it should be. Also more and more people expressed their desire for a modern distributed SCM which had much more better merge support at that time. At that time of discussing all the reviewed distributed systems (git, mercurial, bazaar) were still at heavy development so that we decided to do step in between: to change to subversion as it also seems to be the perfect step in between for the data and history migration. Unfortunately we found out quite late, the the subversion merge tracking was far from being mature so that we still suffered performance issues with that system.

We than had a deeper look at the latest developments of git, mercurial (and partially bazaar) and found that both systems seem to fulfill OpenOffice.org requirements. Within the ESC the discussion about the pro's were done passionately, "unfortunately" there were no serious cons (beside bzr at that time) so that the discussion look like a vi vs. emacs war, but of course we need to have just one system in the end. So finally, Thorsten and Michael M. made clear during the meeting that there were disappointed about the final outcome but again there was an agreement that mercurial will able to do the job.

ODF Icons

 We also had a brief discussion about a new proposal of ODF Document Icons:

goal is to connect the artists of the various teams (Gnome, KDE, StarOffice, etc.) to get a common look and feel for these icons and take care of the different themes and color schemes of the various desktops at the same time (see also here for earlier proposals)

greetings from Orvieto,

A lot of stuff is going on for OpenOffice.org 3.2 release. What are your favorites ?

 I'm sorry that I not listed all contributions to OpenOffice.org being done during the last month. For a full list of contribution please visit http://eis.services.openoffice.org  and do your favorite query.

Recently I had to make a tough decision for the upcoming OpenOffice.org 3.1 release. A blog entry claiming that rules are not followed triggered the following explanation. I really appreciate all the feedback and sometime also the criticism within the release status meeting. Fridrich, you have been and are invited in that meeting on regular basis. With your particpation we could have avoided an offensive sounding blog one can't comment on the site.

While the release candidate 1 went out a security issue was raised in the OpenOffice.org bugtracking system. Members of the OpenOffice.org community already observed for longer time the efforts to do an update of the very old version of python (2.3.x) to a recent version. Since this update was driven by a member of the OpenOffice.org community (Joerg) in his spare time the progress was not that fast as desired but work was ongoing.

At the time I got noticed about the reported security issues I already knew that that we need to do a second release candidate and asked some of my Sun colleagues if we can do anything to help Joerg to get the update done so that the issues is resolved. The policy established by the OpenOffice.org security team was that we try to address any security issue before the final rc actually is in the process of being released.

So we started help working on the child workspace (mainly the support of Mac, Solaris and Windows platform, the work for Linux, BSD was almost done) did QA and the tinderboxes run and finally decided that the cws was ready to integrate before the next release so that we complied with all the rules of OpenOffice.org development. Although I have to admit that this was a tough decision because exchanging code of that size always is with some remaining risk. Knowing that in OpenOffice.org only very limited usage of python code is there and we are confident that this code was covered during QA I finally decided to nominate this community owned cws before the last release candidate.

Some time ago I stumbled across an article about continuous integration (http://martinfowler.com/articles/continuousIntegration.html) and that this now had become a mainstream technique for software development. Been there, done that was the first thought. In ancient times of OpenOffice.org development we already had that model: we maintained a single source repository and everyone commits every day. The result was not as good as desired. During heavy refactoring of the code for new big features we ran regularly in the situation that the active development code line was broken. Broken means the source code built fine but the product itself had major bugs that made QA difficult or impossible. That why the concept of child workspaces has been invented (also know as private workspace, private system build and private versioning in SCM pattern language (http://www.scmpatterns.com/book/pattern-summary.html ). With this concept we were able to do complex feature development on private branch and keeping it in sync with the latest changes on the development codeline. There we some child workspaces living for three years before finally integrated, e.g. aw024, aw033).

But although in child workspaces verifications builds and automated tests are done there are still bugs which will be found by manual testing. One have too find the balance between running (and extend the test cases) the test suite for longer time and integrate early into the development code line to enable a broader group of testers for giving feedback.

With the current integration model we're doing 0.5 up to 3 builds a week. So isn't this already something like continuous integration ? There's one problem with it: In theory, once you have updated your branch (cws) to the HEAD of the development codeline there are no conflicts to resolve when merging then those changes back, it would be possible to automate this step. Since a OpenOffice.org build for all the languages and platforms will typically last more than 1 day it has become practice that a whole bunch of cws get integrated before the next full build is started. Thankfully the Release Engineering takes care of the integrations then by sequentializing them and coordinate with the developers about potential conflict resolution. This leads to the situation that it can take several days the a build of the integrated child workspace finally is available. I'm not sure if this is a problem at all but I started to wonder about drawing a picture that would like more continuous integration.

I finally got this picture:

The assumption in this picture is that it is possible to do a minimal verifications of the build significantly faster than within a day, e.g. by reducing the build configuration, less languages, less platforms, reduced feature set. If it would be possible to do several builds a day in this way, this would be the possibility to resync their branch to the latest know buildable HEAD of the development codeline. Integration in to HEAD and also the revert back in case of breaking the stuff should then be doable by pure machine power without human interaction. Once a day a farm of build machines would be able to provide a nightly build which could be the basis for manual and automated testing:

I'm not sure if this picture make a big difference and if it is practical at all. At least this would look like more continuous integration, I'm interested in how about others think about it.

Earlier this week there was again a meeting of the Engineering Steering Committee . Most interesting discussion was about the upcoming move to a distributed sourcecode configuration management (SCM) tool. Since the past transition from CVS to subversion it turned out that the new merge tracking feature introduced with subversion 1.5 works not that well as expected we now want to hurry to do the final move to a system which is doing the merge tracking very well. We know that modern distributed SCM like bazaar, git and mercurial are able to do this very well. Unfortunately bzr was not able to do an import of the existing (already down-strapped history) svn repository so that Heiner's evaluation covered git and mercurial. It confirmed my observations that it almost an equal race between both. Like vi vs. emacs. We expect that both will do their job. Unless nothing surprising will reveal in the OOo DSCM survey I think I will support to start pilot by using mercurial. Why ? Other big projects (sponsored by Sun) like OpenJDK, OpenSolaris and Kenai also do use mercurial. The upcoming OpenOffice.org release 3.1 already got a delay caused by the move from CVS to subversion, I don't want to do the 3.2 release based on subversion again, we need to start work on the transition to a DSCM now so don't expect me to do long discussion about some feature details of git vs. hg.

A happy vi user :-)

When watching a tennis match together with members of my tennis team, it is often fun to comment on the match, comment on mistakes of the players and what have could have done better to win the point. So it will not take that long until somebody says that if you really play as good in practice as you're able in theory, you will win even the Wimbledon championship ! Everybody who ever saw my tennis play in practice is pretty sure that I will never win any championship.
But why I'm telling such a simple story ? I was wondering for quite some time why some people are very shy to do many of their technical discussion open in the public. So far I have heard the answer, that doing work in small, closed teams is much more effective than doing this is big team. And I wasn't able to prove the opposite. The story of commenting a tennis match gave me finally an idea for the reason: Just commenting a match has almost no effect. I'm not the person on the playground being responsible for the result of the match. Although my comments and advises are correct, I am finally not able to give the right response because 1) I'm not the player and 2) even if I would be the player, I would not be able to give the right answers - in practice, because I have not the power or the resources for the right answers. Even worse, if I would hear in that situation some comments on how to give better answers, this would make my play worse because I would get angry about these "advices". Even if these advises are correct and given with the best wishes for me, they will be destructive for my play.
I guess the same applies for some of the related discussion on OpenOffice.org, people don't want to hear any advise if the originator of that advise does not have the power to help with the appropriate action.
So my lessons learned are:

• I only will participate in discussions where I'm ready to help in an active manner, if I'm only able to give an advise without sustaining help, I will shut up. I will assume that the actors also have read the right books and doing the right things that there are able to do.
• I will do as much discussion as possible do in the public, but I will use also as much filtering as I can for those people which are generating noise without really providing help, but I will carefully listening to people which are 'literally' helpful.
• A person who is "responsible" (in German "verantwortlich", not "zustaendig" how many Germans are sometimes misinterpreting the term "responsibility") for something should be able to give answers to any (maybe silly) question, if a person being in charge of a certain project will act in a closed environment, he will never be able to be responsible !

So in short:

• Be open, otherwise you will not be able to get new, unexpected help and you will never be responsible for your project !
• Only give advise if you are able to help !

A first developer snapshot of the PDF-Import Extension is now available on the Extensions Website.
Imported a couple of PDF files laying around on my disk. I'm quite surprised how many authors seen to think that PDF can't be edited anymore. Using the security settings (restrict permission) should help to set the expectations right.
Also new with the PDF Import extension for OpenOffice.org is the possibility to create hybrid files (PDF containing ODF). This allows the editing of those document with the original document format.

On Slashdot some comments based on the review by Bruce Byfield of Linux.com.

I looked today into the search suggestion list of various search engines.

	google	ask.com	yahoo
1	download	tutorial	download
2	mac	download	2.2
3	templates	templates	2.0
4	.com	.com	software
5	software	base	suite
6	docx	impress

For me it was unexpected that .com was that high in the various lists. Google provides an interface for looking up this data, e.g. http://google.com/complete/search?q=openoffice&output=toolbar

The last childworkspaces containing new UI or Feature has been integrated now. With one exception: onlineupdate03 will be a little bit delayed. Now waiting for build m177.

What does UI Freeze mean: any strings changes are not longer allowed to do. This deadline ensure that translators will have some time before the release to do their work. Strings changes done after this deadline will remain untranlated. UI Freeze is also an important milestone for documentation puposes: builds after this milestones can be used to do screenshots and writing documentation about new features.

Feature Freeze: new functionality is not allowed any more. This will give some time for feedback on new feature or new code. That said, I also consider major rewrites/refactored peaces of code as new features.

The OpenOffice.org Snapshot builds are available on regular basis to give feedback.

In the past weeks there were some discussions about reviewing open patches for OpenOffice.org and about the possibilities to integrate them in the main code line.

Quick review of patches

Patches should be reviewed quickly. There is now in agreement of reviewers that patches do need have a target milestone in the near future. If a patch can not applied as is there will be some comments on it in the issue. Or the patch will be refused because the patch breaks things and can't be fixed.

Specifications

On the OpenOffice.org wiki better is prepared to make the Guidelines for writing OpenOffice.org Specification more clear. There is a new Specification Template out which provides a "guided tour". A self-reviewable, lightweight process for reviewing a specification by the community is desirable. Specification for implementing obvious and non-controversial features may even don't require a full blown specification document but only some documentation at a centralized location.

QA on Child Workspaces

A similar problem as for Specifications applies for QA. The QA Team (Andre Schnabel, Joost Andrae and other) are currently looking on how interested people can be included more in feature (cws) and localization testing than before.

automated builds and tests

Some people already took a deeper look on how to implement something like a Tinderbox compile and test farm and are creating a new subproject for it.

I think having all of the above in place will make it for both, patch submitter and reviewers, more easy to get contributions into the OpenOffice.org code base. New contributors will get faster feedback if spec and QA Guidelines are more transparent. With a tinderbox farm many aspects of the review will be filtered out early.

The OpenOffice.org Roadmap draw the picture to a more continuous development compared to OOo 2.0. It allows new functionality to be integrated when it is complete. Future 2.x releases will not only be minor bugfix releases but will also contains enhancements on the road to OOo 3.0.
This will benefit both: Developers and Users of OpenOffice.org:

• Developer will be more flexible wrt feature planning: there's no big drama anymore if a new feature will miss a major release deadline, it will be integrated into the code line once it is ready and stable.
• User will sooner get access to feature, he has not to wait for the next major release in every case

Last bot not least there will be less problems with slipping (forward :) ) a major release.

With OpenOffice.org 2.0.1 we have the first new features a micro release. Unfortunately we also got the first problems that we missed the UI Freeze for that feature and missed translation timeline, this result in English string for localized version (see screenshot below) in the main insert menu "Formatting Mark" and it's submenu. Please help fix this issue by providing the translation for your favorite lang into issue 58172 until Monday noon (GMT) for 2.0.1 release.

What's going on in OpenOffice.org development

As you might have already heard, we are still planning for the next releases of OpenOffice.org. Although there is an OOo 3.0 target in the IssueTracker, there is not yet a concrete plan for 3.0. OOo 3.0 currently is the synonym for the next major release of OpenOffice.org.

OOo 2.0 was a major release where we have shrinked down the amount of issues from our original plan to make the release finally happen. For 2.0.1, which is coming soon, we had many items on the list which we didn't managed to do for 2.0. Only few minor new features like "Context menu for numberings" or "Add shortcut to jump to recently used slide and view" will make it into 2.0.1 release.

For OOo 2.0.2 I expect that we will incorporate feedback from 2.0 release to increase the stability further by doing bug fixes. I think we will be able to manage a release about every three to six months, so we're going to plan next releases on this basis. Since we are now able to add new UI into minor releases we can add new features into the product even before the next major release. From my experience in 2.0.1 release, we should have a feature freeze at least 6 weeks before the planned release date, to enable all local communities to do their localization. It's also obvious that this is only doable UI changes or enhancements with a limited amount of strings to be translated.

Changes like bigger UI-changes, enhancement of API's or any other complex changes which needs some more time for implementation and review (rewrite of code, new concepts, major incompatible changes) should be targeted to a next major release.

For this reason I would like to ask OpenOffice.org contributors about their plans for the next time, so that we are able to compile a plan for the time until the next major release. Good starting points for this might be:

• a revisit of the PCD plan of 2.0, what have we accomplished for 2.0, what's left ?
• Publish your plans like Malte already did for performance area.
• Put stalled items like the rewrite of the Chart module onto the agenda.

For 2.0.2, planned for end of February, there are already some interesting items at the horizon, such as the Icon Theme Switching , new implementation of Spellchecking hunspell , Quattro Pro 6 Import Filter for Calc .

Contributing code to the OpenOffice.org repository is not easy, there are a lot of rules to follow. In the past there were again questions about these rules like:

• Have really all child workspaces passed this rules ?
• Do I have to comply with all these rules, I only have just a small change to make ?

We have to keep in mind that we want to achieve to major objectives with these rules:

• keep the quality of the build, meaning we are everytime able to release
• keep integration of new or changed code smooth, don't break the build

Since we introduced these rules we made major steps forward: we now have less build breakages than before and we also experiences less P1 issues when a build has been completed. But we still experience faults when integrating code back to upstream. If we look into these issues, we soon discover two categories: issues which could have been avoided if all rules would have been followed and those, which simply haven't been covered by test cases or where the cost-value ratio is too high. We have now this situation:

• These rules can't ensure that no failures will happen at reasonable cost
• Understanding the objectives of these rules help to avoid integration failures

This leads to the practice that we look at the cost-value ratio for many child workspaces: workspace for enhancing build environment, localization or build fixes will often get a check by code review to get integrated fast, others with feature enhancements will get even more attention by our GateKeeper (Joerg Sievers mark those cws with a pin). I as the Release Manager for the current code line look into many child workspaces and look if a owner of a child workspace needs help or even more control. The good news is that most people involved in this are pretty good in understanding our goals and apply to the rules as needed. If there are questions left please don't hesitate to ask your project leads for guidance or help to get your child workspace finally integrated. So the answer to the two above questions is a "no, but ...".