When I came across Martin Hollmichel's blog about “Continuous integration”, it immediately attracted my attention because I'm thinking about the same thing for some weeks now. On a first glance, and with the way we are doing our milestone and release builds now in mind, you might think that Martin is a loony. If that was true, it was quite unfortunate for me, as it would make me a loony too because I share his ideas. Sure, taking the current way of doing builds it's hard to believe that this integration model could work. But let's look on it from another side: what is it what Martin (if I understand him correctly!) and I want to achieve and what must be changed to make that possible?

Before you read on – please be aware that I just describe a (possible) goal. Fortunately it can be reached in several steps, so don't get discouraged by the fact that of course not everything can be done at once and now. But I hope that you can discover the advantages that “continuous integration” can bring to us.

Here are the problems I would like to see solved:

1. The integration of several CWS in one step that we do currently is prone to inter-CWS conflicts and can lead to a source tree

   • with merge conflicts

   • without conflicts but with build errors

   • that builds but has bugs not present in any of the integrated CWS

   Avoiding this will make integration smoother and will improve the quality of the master builds.

2. As a new milestone build usually is started only after a certain number of CWS has been integrated, the “time to master” for a CWS can be several days or even weeks. A worst case scenario is that an integrated CWS created a huge number of conflicts that delays the build and any further integrations and needs developers to fix them, preferably on Friday evening. Faster “ time to master” is important for developers, as they need it for the creation of follow-up CWS, but in an indirect way also for quality: the earlier a CWS is integrated, the more it can get real life testing by users. Users only rarely use CWS builds, but at least some of them give our milestone builds a try.

3. Integration of CWS needs too much work done by release engineers that either could be automated or done by developers. IMHO conflicts always should be fixed by the CWS developers, not by anybody else. But this should never hinder the integration process, so it must happen outside of the integration step. Doing everything else in an automated way enables us to work on integrations 24/7. And the release engineers will have more time for other things.

The problems described in (1) can't happen if a CWS to be integrated is resynced to the current master and has got sufficient testing (working tools and processes assumed).

A “Deluxe” solution that addresses all points could be the following:

• Take the first CWS.

• Resync it to the most current master on the desired code line.

• If that doesn't work, set the CWS back to “new”.

• If the resync worked fine, integrate the CWS; now merge conflicts shouldn't happen.

• The CWS is backed out again and reset to “new” if one of the following things happens:

  • the build breaks on at least one platform,

  • the smoke test is not passed on at least one platform,

  • from a to be defined number of other tests (could be performance, test tool, convwatch etc., everything we want) at least one has not been passed on at least one platform.

• If everything went fine, the current source tree is made a milestone, the new most current master on this code line.

• If either we have a new milestone or the CWS has been backed out, the next CWS is taken and the process starts again.

• There are other things to be discussed, especially for the people working in the build environment in Hamburg (like e.g. for which milestones we would want to have pre-compiled binaries), but that shouldn't bother the rest of the community.

This process would give us much more control over the quality of the master builds and it would shorten the “time to master” for all well-behaving CWS considerably. And it also addresses the third point: each step is basically an automatic one and, if no problems happen, everything can be carried out without manual interaction and it can be done “around the clock”. Problems are kept away from the master and are moved into the CWS. All CWS that don't create any problems can be integrated continuously, the “naughty” ones are skipped and queued again after fixing the problems.

It depends on the speed of each single step how fast this integration moves forward and if the overall speed is acceptable. Let's have a look on the current performance and let's also do a rough estimation for what we might need.

Currently resyncing this is awfully slow. With svn it would be even faster to extract a patch from the CWS to be integrated, create a new CWS on the current master and apply and commit the patch to it, treat this CWS according to my list and either then merge it into the master (what should work fine) or throw it away (no reason to back anything out). Maybe we could do it that way, but my hope is that our planned move to a DSCM will make resyncing considerably faster, so that the process could be done without creating an intermediate CWS. This needs verification.

Building the source and packing installation sets takes too long nowadays; I think that it can be made faster by

• building on faster hardware and environment, if necessary on a local source tree,

• building only the absolutely necessary stuff (needs changes in packaging of course),

• not building full installation sets but using the “ PKGFORMAT=installed” output as this is enough to run tests afterwards,

• speeding up the build environment.

Of course there will be a few CWS where the points 2 and 3 don't apply (e.g. changes in localization, extensions, installation, binfilter), but we can integrate these CWS in dedicated milestones. We can do a complete build with complete packaging as today after integration of 10 CWS or so. Then we will have a comparable output of “complete” milestones as today, but with many intermediate steps in between.

As everything described is a more or less automatic process, we are not bound to release engineers being available for integration and can do it 24/7. So for a feasibility study we shouldn't ask how many CWS per day a release engineer can integrate today, but rather how fast we can make our processes to achieve the necessary integration rate if we assume that there will always be at least one CWS that can be integrated.

According to Thorsten Ziehm we have integrated between 50 and 125 CWS per month in 2008, on average 75 per month. That would be approximately 2.5 CWS per day (in 24/7 mode!). These numbers look promising enough to consider it doable, especially if we can assume that building and testing can be accelerated as described. Without having exact numbers I expect that the number of nominated CWS per day has large peaks – but I think that can be overcome, mainly by better planing in development.

So we could get the same number of CWS integrated, but with more control over quality, shorter time to master for individual CWS and with less manual work. And the whole process can be implemented incrementally, something I always consider to be important if changes are to be applied. So we can start with the first step (always resync CWS, don't let them in the master if that conflicts, otherwise integrate) and look for ways to make this an automatic process. Next step would be to look for possible ways to do builds and tests after each integration in a reasonable time frame, but until we have found a way to manage the resources, nothing needs to be changed. I guesstimate that now at least all tests except the smoke test will take too long, I don't know if there are some possible steps to allow at least for building and smoke testing after each integration step. Perhaps this is something we can implement in a not so distinct future, so we can leave this up to a reality check.

So far I have talked about the “every day” work. Whether we can follow the same strategy when we approach the release date or if we need more complete builds then is open for discussion.

So please don't see Martin's blog and this one as strange musings of some first class loonies, they are just a look on things from a completely different viewing angle, to solve problems that we know but never tried to fix. I think we described worthwile goals and I hope that instead of comments why this can't work we get a help to make that happen. I outlined a possible goal. And each journey towards a goal starts with the first step.