Jyri Virkki

Endless Night (Take Three)

With a mixture of sadness (because it hurts OpenSolaris adoption) and great amusement (because, really, how can they still be doing this!) it is now once again time for my biannual SFW build statistics update.

You may recall my original article on unconsolidating back in December 2007 where I pointed out all the problems with this peculiar practice and the inevitability of it collapsing under the weight of its own build time and size.

Later in June 2008 I posted an update on SFW build times. To summarize, at the time SFW was up to 205 packages and a build took about three and a half hours and 10GB of disk space. (Refer to the previous articles for more info on the simplifying assumptions behind the numbers.)

Fast forward to February 2009, where are we? I ran an SFW build overnight on the current bits (build 108 closed last night) and it took seven and a half hours and 12.3GB of disk space (on the same machine as I've run the previous two build tests).

A few observations...

• The build time has increased faster than the previous linear prediction. Using the current time/pkgs ratio, we'd be looking at 124 hours build time at 5000 packages (and 496 hours for 20,000 packages).
• The build disk size usage has increased slower than the previous linear prediction. Using current size/pkgs ratio, we'd be looking at 203GB to build 5000 packages (and 815GB to build 20,000 packages).
• The build is now up to a full working day. So any developer working on integrating (or fixing or updating) open source applications into OpenSolaris gets one shot per day of getting a clean nightly build (a requirement for integration).

Here is a graph of the data points so far. The dotted boxes at 5000 and 20000 packages show the range of predicted future build times.

And here is the data for the build size. While 12GB doesn't seem too bad given modern disk sizes, in practice it is also a big problem.

In my Web Stack engineering group we have several shared lab servers for doing development work and we are chronically running out of disk space (to the point that often builds fail due to lack of space, which isn't too amusing given the build took all day). With a handful of developers all of whom have a handful of workspaces (for different integration projects) going at the same time, 12GB at a time adds up surprisingly fast.

On the positive side, there has been some good news since my previous update. With the contrib repo up and running, there is an alternative to SFW (and of course, Web Stack project has its own Web Stack project repository but this is only for web tier components and not for general purpose components).

Sadly, this doesn't really help as much as it could because it is only being leveraged for packages which are considered unimportant and/or unsupported. So we continue to stuff most packages into SFW.

As before, it continues to be inevitable that SFW will collapse, it is only a matter of when it becomes so painful that it will no longer be possible to ignore the problem.

Any bets on the timeline?

Posted at 04:55PM Feb 03, 2009 by jyri in WebStack  |  Comments[4]