Kelly O'Hair's Weblog (blogs.sun.com)
Saturday Feb 28, 2009
What Continuous Build Systems Got Wrong
Pearls Before Swine had a great comic recently, not that I have any issue with people that don't agree with me:
I have always been a big fan of regular automated build and testing during development. With every project I have ever been on, if there wasn't some kind of nightly build and test of the team sources, or my own sources, I set one up. Anyone working in software development knows or should know the importance of having this constant monitoring of the product status. So don't get me wrong, I think that all the continuous build systems out there are great to have.
BUT... They all seem to have grown up around the single central source repository model, and I think they have developed some bad habits. The ideal to me is that a changeset or change to the source repository is built and tested BEFORE it ever sees a shared team area, and the continuous build of the team area should be only a final verification. The build should never be broken, we should be able to have golden (or more golden?) changesets all the time. With a Distributed SCM like Mercurial, this is actually quite possible. With a Distributed SCM a changeset can be created, then the repository can be merged with the parent repository, and everything can be fully built and tested before the changeset is ever seen by other developers.
So what we need is a continuous verification build system.
Clint comments about Git and Continuous Integration and also refers to another article on Agile Continuous Integration with Git.. These comments apply to Mercurial too.
I know that the ElectricCloud ElectricCommander product supposedly has a 'trial run' type of feature I was told about, which is a step in the right direction. And I have heard that the Hudson system may have some 'test build' plugin someday.
-kto
Posted at 01:00PM Feb 28, 2009 by kto in Java | Comments[8]
Subscribe!
A project that I have been wanting to tackle could be a helpful step in this direction. The most useful part is that it tests the build in a neutral environment - not one controlled by the developer.
http://mgerdts.blogspot.com/2008/06/opensolaris-build-service-proposal.html
Posted by Mike Gerdts on February 28, 2009 at 03:31 PM PST #
Hi Kelly,
In the context of Hudson, see:
http://wiki.hudson-ci.org/display/HUDSON/Git+Plugin
Something similar could be done for the Mercurial plugin too.
Best,
Ismael
Posted by Ismael Juma on February 28, 2009 at 03:46 PM PST #
Buildbot has a built-in "try" mechanism: http://djmitche.github.com/buildbot/docs/0.7.10 which tests patches before committing.
Mozilla has a hacked version which allows specifying an mercurial repo (since they use both Buildbot and mercurial).
Posted by Robert Helmer on February 28, 2009 at 05:21 PM PST #
Jetbrains' TeamCity product has a feature called "delayed commit" which prevents people from breaking the build:
http://www.jetbrains.com/teamcity/delayed_commit.html
Posted by Kjetil Ødegaard on March 01, 2009 at 02:56 AM PST #
I assume that you're suggesting something more than making sure that developers run the local build before they check in?
Socializing that idea to one's teams can have a huge impact on the green-ness of their build.
If a team manages to reach that level of discipline and still needs something else, how about being able to create little CI builds on the fly in CI servers? I'm thinking of a self-service model. Even though that will put me out of a job :)
Posted by Julian Simpson on March 01, 2009 at 08:44 AM PST #
Many thanks to the people posting comments, all valuable information.
-kto
Posted by Kelly Ohair on March 01, 2009 at 01:23 PM PST #
The folks at SavaJe developed a pretty good system that has the properties you want. It originally was on BitKeeper but was adapted to Mercurial.
Briefly, developers would push changes to the "train" repo. The automated build system would build the train repo and, if successful, run some tests. If these all passed, the changesets that had just built and passed tests would get pulled into the "integration" repo. Developers would always pull from the "integration" repo so they'd always get stable, tested bits.
(The "train" and "integration" terms aren't particularly descriptive, but that's what they were called.)
Posted by Stuart Marks on March 02, 2009 at 02:11 PM PST #
Back when I was the tl integrator, i worked on the "never-ever-broken" model that Kelly alludes to. JDK development improves on traditional software development by having a minimal level of quality of the MASTER (always passes integrator tests; only integrators can commit to the MASTER) but I would go further and add another level, let's call it GOLDEN-MASTER, which only gets bits that have actually passed *all* of the tests, not just the ones the previous integrator happened to run. This would keep hotspot developers from breaking java.util.concurrent, as has been known to happen in the past.
Posted by Martin Buchholz on March 13, 2009 at 01:17 PM PDT #