Jyri Virkki

« Previous page | Main | Next page »
http://blogs.sun.com/jyrivirkki/date/20090211 Wednesday February 11, 2009

Updated varnish and nginx packages

Tonight I updated the nginx and varnish packages in the Web Stack project repository to their respective latest stable releases (nginx-0.6.35 and varnish-2.0.2).

There are no enhancements to the packages this time other than the version so both are still rough (refer to my previous entries on each for their current state). These continue to be experimental components, so if you'd like to play with them please do and let webstack-discuss@opensolaris.org know how it works for you and any suggestions for improvement. If there is interest in these (or other) components, we can look into making the OpenSolaris integration stronger!


Posted at 02:07AM Feb 11, 2009 by jyri in WebStack  | 

http://blogs.sun.com/jyrivirkki/date/20090210 Tuesday February 10, 2009

Support for Web Stack now available...

If you've been following blogs.sun.com at all today you've surely seen all the entries about the Glassfish portfolio announcements so I won't repeat all that here.

For us in Web Stack, the important part of this is that Sun is now selling support for the Web Stack components (and all other Glassfish portfolio components). So for those of you who've asked if or when can you get production support for Web Stack, now you can. Head over to http://www.sun.com/software/webstack/ and click on the "Contact Sales" link for more.

That's my ad for the week! Meanwhile, I'm pushing out some interesting updates to OpenSolaris Web Stack, I'll blog about those shortly....


Posted at 06:19PM Feb 10, 2009 by jyri in WebStack  |  Comments[3]

http://blogs.sun.com/jyrivirkki/date/20090203 Tuesday February 03, 2009

Endless Night (Take Three)

With a mixture of sadness (because it hurts OpenSolaris adoption) and great amusement (because, really, how can they still be doing this!) it is now once again time for my biannual SFW build statistics update.

You may recall my original article on unconsolidating back in December 2007 where I pointed out all the problems with this peculiar practice and the inevitability of it collapsing under the weight of its own build time and size.

Later in June 2008 I posted an update on SFW build times. To summarize, at the time SFW was up to 205 packages and a build took about three and a half hours and 10GB of disk space. (Refer to the previous articles for more info on the simplifying assumptions behind the numbers.)

Fast forward to February 2009, where are we? I ran an SFW build overnight on the current bits (build 108 closed last night) and it took seven and a half hours and 12.3GB of disk space (on the same machine as I've run the previous two build tests).

A few observations...

  • The build time has increased faster than the previous linear prediction. Using the current time/pkgs ratio, we'd be looking at 124 hours build time at 5000 packages (and 496 hours for 20,000 packages).
  • The build disk size usage has increased slower than the previous linear prediction. Using current size/pkgs ratio, we'd be looking at 203GB to build 5000 packages (and 815GB to build 20,000 packages).
  • The build is now up to a full working day. So any developer working on integrating (or fixing or updating) open source applications into OpenSolaris gets one shot per day of getting a clean nightly build (a requirement for integration).

Here is a graph of the data points so far. The dotted boxes at 5000 and 20000 packages show the range of predicted future build times.

And here is the data for the build size. While 12GB doesn't seem too bad given modern disk sizes, in practice it is also a big problem.

In my Web Stack engineering group we have several shared lab servers for doing development work and we are chronically running out of disk space (to the point that often builds fail due to lack of space, which isn't too amusing given the build took all day). With a handful of developers all of whom have a handful of workspaces (for different integration projects) going at the same time, 12GB at a time adds up surprisingly fast.

On the positive side, there has been some good news since my previous update. With the contrib repo up and running, there is an alternative to SFW (and of course, Web Stack project has its own Web Stack project repository but this is only for web tier components and not for general purpose components).

Sadly, this doesn't really help as much as it could because it is only being leveraged for packages which are considered unimportant and/or unsupported. So we continue to stuff most packages into SFW.

As before, it continues to be inevitable that SFW will collapse, it is only a matter of when it becomes so painful that it will no longer be possible to ignore the problem.

Any bets on the timeline?


Posted at 04:55PM Feb 03, 2009 by jyri in WebStack  |  Comments[4]

http://blogs.sun.com/jyrivirkki/date/20090116 Friday January 16, 2009

Notes on Web Server Open Sourcing

Brian Aker wrote about the open sourcing of our web server and it got picked up on slashdot today.

I was reading through the comments and figured I'd throw in a few notes about what this code is and is not...

(I worked directly on the Web Server product for some years and while it is not my day job today, I'm still very closely affiliated with the group who works on the commercial version of this product inside Sun.)

  • First, the released code is not a snapshot of the Netscape Enterprise Server from the 90's!
  • What it is, is a snapshot of the very latest source code for JES Web Server 7.0 (with some non-core parts removed, such as the administration infrastructure - see full list of differences here.
  • The commercial version of this product is actively maintained and sold by Sun (note it is free to download and use, however - so feel free to download both the source and the commercial binaries and try/compare both, if you wish).
  • That said, the code is indeed a direct descendant of the Netscape Enterprise Server. The marketing name changes over the years have not marked rewrites of the core code, it's been the same code all along.
  • While the revision history is not part of the open sourced snapshot (sorry), I can mention that in the internal repository of this code I see cvs comments dating back to 1995.
  • With over ten years of development and bug fixing a lot has changed, naturally. On the other hand, if you were involved with the original product way back then, you'll definitely find some familiar bits and pieces here and there. As with any mature software product, there are always some parts which have not changed in ages.
  • So, while not a mummified snapshot, the code is indeed interesting as a piece of Internet history. Furthermore, it is also interesting as a modern living product.
  • Extreme scalability in multi-CPU (or multi-core) hardware is perhaps the most interesting angle from which to look at the code. (Funnily enough, with the rise of parallelism in modern hardware, maybe the code is becoming more interesting these days instead of less!)
  • As to who or why be interested, that doesn't really have any one answer. If you find it interesting or useful for either reason (or some other of your own), enjoy! Being under BSD license, there are many ways to take advantage of it.


Posted at 05:47PM Jan 16, 2009 by jyri in WebStack  |  Comments[1]

http://blogs.sun.com/jyrivirkki/date/20090113 Tuesday January 13, 2009

Announcing Open Source Web Server

I'm happy to announce that our Web Server product (about which I've been writing here for a few years now) is now open sourced and available as part of the OpenSolaris Web Stack community!

Well, technically it is not exactly the Web Server product, since the open sourced code does not include some of the value-add components such as the administration framework. But it is the real deal, the massively scalable web server core which is used in the JES Web Server 7.0 product is now all open source!

This marks another milestone in the very long history of this web server. Back in the 90's this was the Netscape Enterprise Server, which later morphed into the iPlanet Web Server during the Sun|Netscape Alliance. After some years it was renamed the SunONE Web Server and most recently renamed again to the JES Web Server (Sun just like to keep you confused, thus the constant renaming of the product!)

The code is placed under BSD license, this should allow for good cross pollination with other web tier projects.

Enjoy!

Source code is available via: 

% hg clone ssh://anon@hg.opensolaris.org/hg/webstack/webserver

Build instructions are here: http://wikis.sun.com/display/wsFOSS/Build+Instructions

(The code itself is highly portable as you can see based on the supported platforms of the commercial product. Building on other platforms is a bit more involved due to dependencies so the build instructions only cover the more flexible platforms.)

(edit: adding link to top level info page)

More info here: http://wikis.sun.com/display/wsFOSS/Open+Web+Server

Posted at 11:42AM Jan 13, 2009 by jyri in WebStack  |  Comments[10]

http://blogs.sun.com/jyrivirkki/date/20081222 Monday December 22, 2008

Sun Web Stack 1.4

I'm happy to announce Sun Web Stack 1.4 is now available for download!

Despite the confusing version number this is the first release of this new product. If you've been following my articles on OpenSolaris Web Stack then this product will seem quite familiar. In fact, Sun Web Stack 1.4 is a port of the primary OpenSolaris Web Stack components from 2008.11 to Solaris 10 (and RedHat Enterprise Linux).

So now you can take advantage of the work delivered by the Web Stack project on your existing Solaris 10 production systems and not only on OpenSolaris... Cool!

Speaking of cool, the strange version number (for a first release) is due to the fact that Sun Web Stack takes over where CoolStack left off. Since the final CoolStack release was 1.3, we decided to continue the version numbering as a nod to that continuity. Unlike CoolStack though, Sun Web Stack is a full-fledged Sun product, which means you'll have the option of purchasing support for it.

The packages are freely available of course. Here's some useful links:

I want to stress that Sun Web Stack is a port of the OpenSolaris Web Stack work. That means that future component upgrades and feature additions all go to OpenSolaris first and then will get promptly backported to Solaris 10 and RedHat under the Sun Web Stack banner. This also means that if you want to discuss future Web Stack directions, the best place is still webstack-discuss@opensolaris.org. The support forum I listed above is specifically for support questions related to using Sun Web Stack binaries on Solaris 10 or RedHat Linux. As the architect for both OpenSolaris Web Stack and Sun Web Stack, I'll continue covering both in this blog so stay tuned for more!

The initial design of this project started in September and it was mostly completed towards the end of last month. In conjunction with the OpenSolaris 2008.11 work, that made for very busy three months! So the timing of this release is perfect, just in time for the holiday break!

In short, if you've been following Web Stack work but are still on Solaris 10, go check out Sun Web Stack, give it a try and let us know what else you'd like to see included in the future.

Early in 2009 we will be planning on what to enhance and add to Web Stack for the next release of OpenSolaris (and thus, the next release of Sun Web Stack as well) so this is a good time to get your Web Stack wishes in!


Posted at 11:44PM Dec 22, 2008 by jyri in WebStack  |  Comments[3]

http://blogs.sun.com/jyrivirkki/date/20081215 Monday December 15, 2008

Web Server in OpenSolaris

Web Server 7.0u4 has been released and of particular interest is that it is now formally supported in OpenSolaris... In conjunction with the OpenSolaris Web Stack project, another piece of the puzzle falls in place for scalable web tier solutions on OpenSolaris!

Posted at 05:38PM Dec 15, 2008 by jyri in WebServer  |  Comments[5]

http://blogs.sun.com/jyrivirkki/date/20081201 Monday December 01, 2008

Web Stack Python Packages

Thanks to Brian, the OpenSolaris Web Stack project IPS repository now has a number of new packages, all of them related to Python. I've published updates to the public repo tonight containing:

  • python-configobj
  • python-simplejson
  • python-paste-deploy
  • python-turbocheetah
  • python-turbojson
  • python-turbokid
  • genshi
  • python-cheetah
  • python-paste

As with all components in the Web Stack project repository, give them a try and if you find them useful (or if you don't find them useful for some reason!) let the project team know your thoughts at webstack-discuss@opensolaris.org. If you have any favorite web tier components or frameworks which you'd like to see available for OpenSolaris let us know that as well!


Posted at 08:44PM Dec 01, 2008 by jyri in WebStack  |  Comments[6]

http://blogs.sun.com/jyrivirkki/date/20081126 Wednesday November 26, 2008

Web Stack Updates

It has been an impossibly busy 3 months, something had to drop so blogging was it. Time to pick it up again!

Since it's been a while, I'll revisit what has been going on with OpenSolaris Web Stack in the last few months. This blog may have been quiet but things have been happening!

  • Lighttpd is now included in Web Stack! Amanda has some entries on lighttpd integration and scalability
  • Ruby has been updated to 1.8.6-p287
  • Ruby also integrates some new performance improvements on SPARC
  • Tomcat has been updated to 5.5.27
  • Apache httpd is now at 2.2.9
  • As part of the Apache 2.2.9 work, APR and APR-util have been split into separate packages. These libraries are consumed by some other apps, not just Apache so now OpenSolaris users don't end up installing all of Apache if they only needed APR.
  • Also on the Apache front, while not integrated yet, check out Nick's recent work on Apache and privileges
  • MySQL updated to 5.0.67. Previously we were hoping to go straight to 5.1 instead of doing another 5.0.* update but 5.1 release moved out a bit, to the point where it wasn't possible to get it into 2008.11 so we ended up going to 5.0.67 to benefit from accumulated bug fixes for 2008.11. The plan is to have 5.1 in Web Stack (including dtrace support!) very soon after it gets released.
  • PHP memcached extension has been upgraded to 2.2.4
  • PHP has support for imap/imap-ssl now
  • PHP xdebug extension updated to 2.0.3
  • PHP suhosin upgraded to 0.9.27
  • PHP tcpwrap is now 1.1.3
  • PHP also now has support for PEAR/PECL
  • And finally on PHP, the packaging has been refactored a bit to be more sensible. Instead of being named after the full version (SUNWphp524) they are now just SUNWphp52. This allows bugfix updates without constantly renaming packages. Some unnecessarly splits were also combined, although that improvement is more visible to Nevada users than OpenSolaris users (because IPS combines some packages during post-production anyway).

So there's your whirlwind review of recent Web Stack changes...

You can get access to all these and the rest of Web Stack by updating your OpenSolaris installation to the latest, a good way to spend a long Thanksgiving weekend ;-0

(And OpenSolaris 2008.11 should be out real soon with many other improvements beyond the world of Web Stack, but if you're mainly interested in Web Stack components all of the above updates are already there in build 101 so you don't have to wait.)


Posted at 05:49PM Nov 26, 2008 by jyri in WebStack  |  Comments[6]

http://blogs.sun.com/jyrivirkki/date/20080806 Wednesday August 06, 2008

Web Stack at Immersion Week

Thanks to those of you who attended the Web Stack session at Immersion Week (Sun field/sales meeting) this morning! I'm glad to see the drupal-demo package was well received.

As I mentioned, for any comments/feedback please contact us at webstack-discuss@opensolaris.org. Thanks!


Posted at 08:30PM Aug 06, 2008 by jyri in WebStack  | 

http://blogs.sun.com/jyrivirkki/date/20080731 Thursday July 31, 2008

IPS updates

Good news on the IPS front, the bug which prevented pkg from installing from additional repositories such as Web Stack has been fixed and with the recent availability of build 94 bits for OpenSolaris the IPS fix is now available.

So as soon as you upgrade your OpenSolaris image to b94 or later, it will no longer be necessary to use the workaround I documented earlier in order to install any of our Web Stack packages.


Posted at 01:02AM Jul 31, 2008 by jyri in WebStack  |  Comments[4]

http://blogs.sun.com/jyrivirkki/date/20080730 Wednesday July 30, 2008

Drupal demo package

I decided to tweak the delivery of the evaluation/demo/quickstart script included in the Drupal package a bit. Since the script (/usr/bin/drupal_evaluation_init) is for various reasons not suitable for setting up a real production site I wasn't very happy that it was in the 'drupal' package. After all, it shouldn't be installed on a production site. So I created a new package called 'drupal-demo' which now delivers the quickstart script (and a manpage for it).

There's another benefit to splitting it out like this; I made 'drupal-demo' have a dependency on the MySQL package and removed that dependency from the core 'drupal' package. This makes more sense: for a production site you can install only 'drupal' package and since you're more likely to point it at a MySQL instance on a different server there is no need to forcefully install MySQL locally. If you want to play around, test or demonstrate Drupal you can install 'drupal-demo' which gets you not only the quickstart script but also MySQL on localhost which is perfect for test/demo purposes.

(Unfortunately this benefit is only theoretical at the moment. That's because all of MySQL, including the client libraries, is in one big package right now. This means that for PHP to have access to the MySQL libraries it needs (which is required for Drupal), PHP ends up bringing in all of MySQL locally anyway. However, we consider this a bug and there should be some refactoring of the MySQL package coming up.)

I'll be doing the same for other packages which include this kind of demo/quickstart setup (right now, phpmyadmin) both to separate the non-production files and to manage the dependencies better.

If you have any preference on the naming of these packages let me know. I'm not entirely happy with the *-demo suffix but it's the most concise one I could think of at the moment.


Posted at 07:18PM Jul 30, 2008 by jyri in WebStack  |  Comments[1]

http://blogs.sun.com/jyrivirkki/date/20080724 Thursday July 24, 2008

Web Stack source packages

In a previous article I mentioned how easy it was to build one component in Debian by just grabbing the source package and its dependencies.

Earlier this week I was cleaning up the script which builds and produces the Web Stack repository packages and it occurred to me it'd be really easy to enhance it to also publish source packages, so why not. If you've looked at the repository in the last couple days you may have already noticed several *-source packages.

So let's say you'd like to build nginx on OpenSolaris yourself instead of using the nginx package from the Web Stack repository (maybe you want to change a build flag or something). So now you have one more option on how to do it: 

% pfexec pkg install pkg://webstack/nginx-source
% pfexec zsh
# cd /opt/webstack-source/nginx
# make install

If you have your own local IPS repo running you can also easily publish the freshly rebuilt package into it by simply: 

# export PKG_REPO=http://myips.example.com:10000
# make publish

(where, of course, the value you set for PKG_REPO is actually the URL to your own local IPS repository)

Then you (or others in your group) could simply try out your rebuild package by installing it from your $PKG_REPO.

Notice that the *-source package sets all its dependencies such that you can simply type 'make' right away and all the packages (such as compilers and needed libraries) are already in place. You don't have to go through any READMEs to figure out what other packages you need before compiling. Pretty cool! ;-)

Now, admittedly, in some ways this is just a gimmick due to IPS limitations today. For one thing, I had to pick an install location (so I chose /opt/webstack-source/$PKGNAME) because unlike apt, pkg doesn't know abot source packages so there's no convenient way to drop the source into "." which is probably where you really want it. So it also means you then had to become root to build under /opt/webstack-source/ (I might change this to use a dedicated user but still it's not optimal). (Installing the source package into a user image might seem like a workaround but it won't do what you want since then all the dependencies get installed in the wrong place.)

You could of course simply check out the source directly with mercurial and get the same content with the benefit of placing the source tree wherever you want and owned by your uid. The drawback is that in that case you will need to figure out and manuall install all dependencies needed to actually build the source.

So, there's benefits and drawbacks either way. Nonetheless, an interesting experiment!


Posted at 12:29PM Jul 24, 2008 by jyri in WebStack  | 

http://blogs.sun.com/jyrivirkki/date/20080723 Wednesday July 23, 2008

nginx package updates

Tonight was a moment to play with the nginx package which has been in the Web Stack repository for a while now. I added a working default configuration, smf support and a stub man page. While I was there I also updated the version to 0.6.32.

So you know how it goes.. give it a try and send suggestions on how to improve it for OpenSolaris to webstack-discuss@opensolaris.org! 

% pfexec pkg set-authority -O http://pkg.opensolaris.org/webstack webstack
% pfexec pkg refresh
% pfexec pkg install pkg://webstack/nginx
% man nginx
% pfexec svccfg import /var/svc/manifest/network/http-nginx.xml

(the above is just to work around a bug in IPS, it does 
not yet import the manifest automatically)

% svcadm enable nginx


Posted at 11:08PM Jul 23, 2008 by jyri in WebStack  | 

http://blogs.sun.com/jyrivirkki/date/20080722 Tuesday July 22, 2008

Updated phpMyAdmin package

As most of you know the Web Stack repository has had a phpMyAdmin package for a long while now. It was in fact the first package to show up on the project repository. It was very much an experiment and it delivered its files in the simplest way possible, by simply dumping everything under the Apache htdocs directory.

With the release of the drupal OpenSolaris package recently, I decided to go back and rearrange the phpMyAdmin package to follow the same model and I just pushed it out to the repository. Changes include:

1. File layout similar to drupal (and also similar to the upcoming phpPgAdmin package): static files live under /usr/phpmyadmin/ and the editable configuration is in /etc/phpmyadmin/

2. The package delivers sample Apache configuration in /etc/apache2/2.2/samples-conf.d/phpmyadmin.conf

3. A stub man page is included (man phpmyadmin) and a quick-start evaluation script is also included (/usr/bin/phpmyadmin_evaluation_init).

Also, not specific to phpMyAdmin, I improved the package publishing so now the version numbers are preserved, so you'll find the package list more informative than before. For example by looking at "pkg:/phpmyadmin@2.11.7,5.11-1:20080722T221304Z" you can tell I updated the version of phpMyAdmin to 2.11.7.

Finally, while bug 2253 has recently been fixed, unless you're running a handbuilt version of IPS you'll still need to use the workaround in order to install Web Stack packages: 

% pfexec pkg set-authority -O http://pkg.opensolaris.org/webstack webstack
% pfexec pkg refresh
% pfexec pkg install pkg://webstack/phpmyadmin
% man phpmyadmin

As always, please try it out and send your feedback to webstack-discuss@opensolaris.org. Better yet, if you're inspired to improve it, let me know.


Posted at 11:10PM Jul 22, 2008 by jyri in WebStack  |