Martin Hollmichel's Weblog: Martin Hollmichel's Weblog

OpenOffice.org now in transition to distributed source code control (mercurial)

The transition of the source code repository from CVS via subversion to mercurial is now almost complete. It was a long way to go, there was the agreement for a long time that an migration from CVS to a more modern source code control is desired. The major obstacle was the lack of native merge capability support, simulating the merge tracking with supplemental scripting was not that handy and snappy as it should be. Also more and more people expressed their desire for a modern distributed SCM which had much more better merge support at that time. At that time of discussing all the reviewed distributed systems (git, mercurial, bazaar) were still at heavy development so that we decided to do step in between: to change to subversion as it also seems to be the perfect step in between for the data and history migration. Unfortunately we found out quite late, the the subversion merge tracking was far from being mature so that we still suffered performance issues with that system.

We than had a deeper look at the latest developments of git, mercurial (and partially bazaar) and found that both systems seem to fulfill OpenOffice.org requirements. Within the ESC the discussion about the pro's were done passionately, "unfortunately" there were no serious cons (beside bzr at that time) so that the discussion look like a vi vs. emacs war, but of course we need to have just one system in the end. So finally, Thorsten and Michael M. made clear during the meeting that there were disappointed about the final outcome but again there was an agreement that mercurial will able to do the job.

ODF Icons

 We also had a brief discussion about a new proposal of ODF Document Icons:

goal is to connect the artists of the various teams (Gnome, KDE, StarOffice, etc.) to get a common look and feel for these icons and take care of the different themes and color schemes of the various desktops at the same time (see also here for earlier proposals)

greetings from Orvieto,

A lot of stuff is going on for OpenOffice.org 3.2 release. What are your favorites ?

 I'm sorry that I not listed all contributions to OpenOffice.org being done during the last month. For a full list of contribution please visit http://eis.services.openoffice.org  and do your favorite query.

Recently I had to make a tough decision for the upcoming OpenOffice.org 3.1 release. A blog entry claiming that rules are not followed triggered the following explanation. I really appreciate all the feedback and sometime also the criticism within the release status meeting. Fridrich, you have been and are invited in that meeting on regular basis. With your particpation we could have avoided an offensive sounding blog one can't comment on the site.

While the release candidate 1 went out a security issue was raised in the OpenOffice.org bugtracking system. Members of the OpenOffice.org community already observed for longer time the efforts to do an update of the very old version of python (2.3.x) to a recent version. Since this update was driven by a member of the OpenOffice.org community (Joerg) in his spare time the progress was not that fast as desired but work was ongoing.

At the time I got noticed about the reported security issues I already knew that that we need to do a second release candidate and asked some of my Sun colleagues if we can do anything to help Joerg to get the update done so that the issues is resolved. The policy established by the OpenOffice.org security team was that we try to address any security issue before the final rc actually is in the process of being released.

So we started help working on the child workspace (mainly the support of Mac, Solaris and Windows platform, the work for Linux, BSD was almost done) did QA and the tinderboxes run and finally decided that the cws was ready to integrate before the next release so that we complied with all the rules of OpenOffice.org development. Although I have to admit that this was a tough decision because exchanging code of that size always is with some remaining risk. Knowing that in OpenOffice.org only very limited usage of python code is there and we are confident that this code was covered during QA I finally decided to nominate this community owned cws before the last release candidate.

Some time ago I stumbled across an article about continuous integration (http://martinfowler.com/articles/continuousIntegration.html) and that this now had become a mainstream technique for software development. Been there, done that was the first thought. In ancient times of OpenOffice.org development we already had that model: we maintained a single source repository and everyone commits every day. The result was not as good as desired. During heavy refactoring of the code for new big features we ran regularly in the situation that the active development code line was broken. Broken means the source code built fine but the product itself had major bugs that made QA difficult or impossible. That why the concept of child workspaces has been invented (also know as private workspace, private system build and private versioning in SCM pattern language (http://www.scmpatterns.com/book/pattern-summary.html ). With this concept we were able to do complex feature development on private branch and keeping it in sync with the latest changes on the development codeline. There we some child workspaces living for three years before finally integrated, e.g. aw024, aw033).

But although in child workspaces verifications builds and automated tests are done there are still bugs which will be found by manual testing. One have too find the balance between running (and extend the test cases) the test suite for longer time and integrate early into the development code line to enable a broader group of testers for giving feedback.

With the current integration model we're doing 0.5 up to 3 builds a week. So isn't this already something like continuous integration ? There's one problem with it: In theory, once you have updated your branch (cws) to the HEAD of the development codeline there are no conflicts to resolve when merging then those changes back, it would be possible to automate this step. Since a OpenOffice.org build for all the languages and platforms will typically last more than 1 day it has become practice that a whole bunch of cws get integrated before the next full build is started. Thankfully the Release Engineering takes care of the integrations then by sequentializing them and coordinate with the developers about potential conflict resolution. This leads to the situation that it can take several days the a build of the integrated child workspace finally is available. I'm not sure if this is a problem at all but I started to wonder about drawing a picture that would like more continuous integration.

I finally got this picture:

The assumption in this picture is that it is possible to do a minimal verifications of the build significantly faster than within a day, e.g. by reducing the build configuration, less languages, less platforms, reduced feature set. If it would be possible to do several builds a day in this way, this would be the possibility to resync their branch to the latest know buildable HEAD of the development codeline. Integration in to HEAD and also the revert back in case of breaking the stuff should then be doable by pure machine power without human interaction. Once a day a farm of build machines would be able to provide a nightly build which could be the basis for manual and automated testing:

I'm not sure if this picture make a big difference and if it is practical at all. At least this would look like more continuous integration, I'm interested in how about others think about it.

Earlier this week there was again a meeting of the Engineering Steering Committee . Most interesting discussion was about the upcoming move to a distributed sourcecode configuration management (SCM) tool. Since the past transition from CVS to subversion it turned out that the new merge tracking feature introduced with subversion 1.5 works not that well as expected we now want to hurry to do the final move to a system which is doing the merge tracking very well. We know that modern distributed SCM like bazaar, git and mercurial are able to do this very well. Unfortunately bzr was not able to do an import of the existing (already down-strapped history) svn repository so that Heiner's evaluation covered git and mercurial. It confirmed my observations that it almost an equal race between both. Like vi vs. emacs. We expect that both will do their job. Unless nothing surprising will reveal in the OOo DSCM survey I think I will support to start pilot by using mercurial. Why ? Other big projects (sponsored by Sun) like OpenJDK, OpenSolaris and Kenai also do use mercurial. The upcoming OpenOffice.org release 3.1 already got a delay caused by the move from CVS to subversion, I don't want to do the 3.2 release based on subversion again, we need to start work on the transition to a DSCM now so don't expect me to do long discussion about some feature details of git vs. hg.

A happy vi user :-)

There was an interesting posting from Thorsten Bosbach some time ago about setting up a minimal work environment.

In the meantime the configure scripts moved from the subdirectory config_office to the root of the OpenOffice.org source tree, so some minor adoptions need to be done for getting the minimal work environment set up again. Since in subversion there in no chance anymore to have single file under revision control we need to use the "svn export" workaround to get the new top level configure scripts. For a detailed list of files needed to set up the environment please have a look into my script for automating this.

The OpenOffice.org configure script is designed to set up the build environment of all the OpenOffice.org stuff needed for generated to complete product and not all of these settings a currently switchable by the configure script. For Unix, the X11 Development Header (and libraries) still needed to be installed, including such libs as freetype. But if these are available, just the Archive::Zip modules for perl is needed at the moment. For Windows, the situation is worse. There are many prerequisites, e.g. msi tools, macro assembler, platform SDK and many more, which can't be disabled by configure at the moment, so a minimal set up of the build environment is difficult to achieve.

Since the bash now have a new >>& redirection operator, which appends the standard output and standard error to the named file and also the parser now understands `|&' as a synonym for `2>&1 |', which redirects the standard error for a command through a pipe we really should use bash as the default shell to make the "--with-use-shell=bash" switch obsolete.

No surprise, this minimal build environment can't only be used for the testautomation module, but also for all the dictionaries provided as extensions (svn co svn://svn.services.openoffice.org/ooo/trunk/dictionaries; cd dictionaries; build) and the extras modules (example documents and templates).

When watching a tennis match together with members of my tennis team, it is often fun to comment on the match, comment on mistakes of the players and what have could have done better to win the point. So it will not take that long until somebody says that if you really play as good in practice as you're able in theory, you will win even the Wimbledon championship ! Everybody who ever saw my tennis play in practice is pretty sure that I will never win any championship.
But why I'm telling such a simple story ? I was wondering for quite some time why some people are very shy to do many of their technical discussion open in the public. So far I have heard the answer, that doing work in small, closed teams is much more effective than doing this is big team. And I wasn't able to prove the opposite. The story of commenting a tennis match gave me finally an idea for the reason: Just commenting a match has almost no effect. I'm not the person on the playground being responsible for the result of the match. Although my comments and advises are correct, I am finally not able to give the right response because 1) I'm not the player and 2) even if I would be the player, I would not be able to give the right answers - in practice, because I have not the power or the resources for the right answers. Even worse, if I would hear in that situation some comments on how to give better answers, this would make my play worse because I would get angry about these "advices". Even if these advises are correct and given with the best wishes for me, they will be destructive for my play.
I guess the same applies for some of the related discussion on OpenOffice.org, people don't want to hear any advise if the originator of that advise does not have the power to help with the appropriate action.
So my lessons learned are:

• I only will participate in discussions where I'm ready to help in an active manner, if I'm only able to give an advise without sustaining help, I will shut up. I will assume that the actors also have read the right books and doing the right things that there are able to do.
• I will do as much discussion as possible do in the public, but I will use also as much filtering as I can for those people which are generating noise without really providing help, but I will carefully listening to people which are 'literally' helpful.
• A person who is "responsible" (in German "verantwortlich", not "zustaendig" how many Germans are sometimes misinterpreting the term "responsibility") for something should be able to give answers to any (maybe silly) question, if a person being in charge of a certain project will act in a closed environment, he will never be able to be responsible !

So in short:

• Be open, otherwise you will not be able to get new, unexpected help and you will never be responsible for your project !
• Only give advise if you are able to help !

A first developer snapshot of the PDF-Import Extension is now available on the Extensions Website.
Imported a couple of PDF files laying around on my disk. I'm quite surprised how many authors seen to think that PDF can't be edited anymore. Using the security settings (restrict permission) should help to set the expectations right.
Also new with the PDF Import extension for OpenOffice.org is the possibility to create hybrid files (PDF containing ODF). This allows the editing of those document with the original document format.

On Slashdot some comments based on the review by Bruce Byfield of Linux.com.

I looked today into the search suggestion list of various search engines.

	google	ask.com	yahoo
1	download	tutorial	download
2	mac	download	2.2
3	templates	templates	2.0
4	.com	.com	software
5	software	base	suite
6	docx	impress

For me it was unexpected that .com was that high in the various lists. Google provides an interface for looking up this data, e.g. http://google.com/complete/search?q=openoffice&output=toolbar

OpenOffice.org 2.0.4 comes now with a new online update feature which will automagically notifiy the user about new updates.

It is available as own module during the installation

and its settings can be changed via Tools->Options dialog

A few days ago there was a meeting of some OpenOffice.org project leads who agreed on writing more blogs for their projects to get more visibility for their project. I was asked why we want use one more medium to work with on regular basis for communication.

Here's my answer: Please think about how to get best your target audience involved and then choose the appropriate channel which fits the purpose and you are able to deal with.

Think first about your problem you want to solve with your communication and choose the right channel. In our OpenOffice.org world we have to deal with different content we have to communicate and various communication start and end points. Examples for content may be:

• documentation
• announcements
• presentations
• questions and answers

In some of these cases it may be desired to have the possibility to get feedback. Or it may be even required, e.g. if you need some guidance by an experienced developer during a debug session. Examples for communication end points are:

• Individuals
• Employees of a company participating the OOo Project
• Members of an OpenOffice.org Project
• all members of all OpenOffice.org Communities
• OpenOffice.org Users
• the rest of the world
• and many more like managers, committees, etc.

In most case you think of communication on OpenOffice.org you are yourself the communication start point. This is the easy one.

So it might be obvious if you seek advise during your debugging session that you do not choose the channel "TV" but a personal meeting or a phone call. Or the other extreme, if you want to do some advertising to increase the amount of OpenOffice.org users, you will probably do not choose the channel "phone calls" but a "TV Spot". It not too difficult to put a list with good examples together:

• Reference Documentation: -> Web Pages
• Guides and FAQs: -> Wiki
• Proposals and Discussion about them: -> Mail/Newsgroup
• Questions: -> mail/irc
• write something intesting with unclear communication end point: -> blog
• ask for more money: -> personal meeting with your manager
• ... endless list

I often heard that people tend to prefer privat communication instead of doing this in public. They want to have quick confirmation for doing things right and it is much faster to get this in a personal talk than writing a mail and waiting for responses. In many case you will not even get response on mailing lists. I think it is understandable but not the point. But it is not the point, we need to get all stakeholders involved for any issue regarding the specific project. So the primary issue is not that we need to communicate almost all communication on public channels, but to involve all concerned parties. This usually will result that this is somehow done on public channels but the main point is to do this on the right channel.

Our starting point was "why we want to use one more medium (blogs) ?". There was the question how we can attract more developers for participating in the OOo project. One of the answers was that we need to make OOo more interesting in technical aspects. We found that project specific articles might increase the common interest in OOo. Doing this in blogs seems a appropriate way for doing this: You are able to reach not already subscribed people and the effort for doing this via blog is reasonable low.

Further readings: Social Software

Pavel found it first: "Looks like after logging at the issue page, we are now back on that issue page." This will save a lot "go back and reload" key strokes :) (see issue 34822 )

The last childworkspaces containing new UI or Feature has been integrated now. With one exception: onlineupdate03 will be a little bit delayed. Now waiting for build m177.

What does UI Freeze mean: any strings changes are not longer allowed to do. This deadline ensure that translators will have some time before the release to do their work. Strings changes done after this deadline will remain untranlated. UI Freeze is also an important milestone for documentation puposes: builds after this milestones can be used to do screenshots and writing documentation about new features.

Feature Freeze: new functionality is not allowed any more. This will give some time for feedback on new feature or new code. That said, I also consider major rewrites/refactored peaces of code as new features.

The OpenOffice.org Snapshot builds are available on regular basis to give feedback.

In the past weeks there were some discussions about reviewing open patches for OpenOffice.org and about the possibilities to integrate them in the main code line.

Quick review of patches

Patches should be reviewed quickly. There is now in agreement of reviewers that patches do need have a target milestone in the near future. If a patch can not applied as is there will be some comments on it in the issue. Or the patch will be refused because the patch breaks things and can't be fixed.

Specifications

On the OpenOffice.org wiki better is prepared to make the Guidelines for writing OpenOffice.org Specification more clear. There is a new Specification Template out which provides a "guided tour". A self-reviewable, lightweight process for reviewing a specification by the community is desirable. Specification for implementing obvious and non-controversial features may even don't require a full blown specification document but only some documentation at a centralized location.

QA on Child Workspaces

A similar problem as for Specifications applies for QA. The QA Team (Andre Schnabel, Joost Andrae and other) are currently looking on how interested people can be included more in feature (cws) and localization testing than before.

automated builds and tests

Some people already took a deeper look on how to implement something like a Tinderbox compile and test farm and are creating a new subproject for it.

I think having all of the above in place will make it for both, patch submitter and reviewers, more easy to get contributions into the OpenOffice.org code base. New contributors will get faster feedback if spec and QA Guidelines are more transparent. With a tinderbox farm many aspects of the review will be filtered out early.